Information processing device, information processing method, and program

Abstract

Improve convenience for distributors. [Solution] The information processing device includes a display control unit that controls the display of the raw video, an image acquisition unit that acquires a display image that displays content distributed from a server device, and a support identification unit that identifies the content of support from the viewer from the display mode of the support on the display image, and the display control unit changes the display mode of the virtual model depending on the content of the support.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, a technology for displaying a virtual model in a virtual space and live streaming the virtual space has been known. In live streaming, a broadcaster can broadcast a video using a video distribution platform provided by a distribution server. In addition, in such a broadcast, viewers who watch the broadcast can provide support to the broadcaster in the form of tips, gifts, etc.

[0003] Patent document 1 discloses an information processing system having a computer processor, which includes a distribution unit that distributes information about a video including animation of a character object of a broadcasting user to a viewing user terminal, a reception unit that receives a display request for a specific gift sent from the viewing user terminal, an object display unit that displays a specific gift object corresponding to the specific gift in the video in accordance with the display request received by the reception unit, a judgment unit that determines whether behavior information of the broadcasting user or other gift objects satisfies predetermined conditions associated with the specific object, and an expression unit that changes at least one of the movement or shape of the specific object if the judgment unit determines that the predetermined conditions are met. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-108105 Summary of the Invention [Problem to be solved by the invention]

[0005] In live streaming using a virtual model VM, there is a demand for controlling the content of streaming in accordance with the support provided by the viewing user. However, in conventional systems such as that disclosed in Patent Document 1, the support provided by the viewer is usually managed by the device that distributes the video. In such a system, in order for the broadcaster to obtain the support content, the broadcaster must obtain information from the distribution server, which leaves room for improvement in terms of convenience for the broadcaster.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to improve the convenience for distributors in distribution using virtual models. [Means for solving the problem]

[0007] According to one aspect of the present invention, an information processing device configured to be able to communicate with a server device that distributes content including a raw video including a virtual model and support content provided by a viewer includes a display control unit that controls the display of the raw video, an image acquisition unit that acquires a display image displaying the content distributed from the server device, and a support identification unit that identifies the content of support from the viewer from the display mode of the support on the display image, and the display control unit changes the display mode of the virtual model depending on the content of the support.

[0008] According to one aspect of the present invention, an information processing method includes steps of acquiring a display image displaying content distributed from a server device including a raw video including a virtual model and content of support from a viewer for the distribution, identifying the content of the support from the display mode of the support on the display image, and performing display control to change the display mode of the virtual model according to the content of the support.

[0009] According to one aspect of the present invention, the program causes a computer to execute the steps of acquiring a display image displaying content distributed from a server device including a raw video including a virtual model and content of support from viewers regarding the distribution, identifying the content of the support from the display mode of the support on the display image, and performing display control to change the display mode of the virtual model according to the content of the support. [Effects of the Invention]

[0010] According to the present invention, convenience for distributors in distribution using virtual models is improved. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing a configuration of an information processing system according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing a configuration of a server device according to a first embodiment of the present invention. [Figure 3] FIG. 2 is an image diagram showing an example of a display screen of content provided by a viewer page according to the first embodiment of the present invention. [Figure 4] 1 is a block diagram showing a configuration of an information processing device according to a first embodiment of the present invention. [Figure 5] FIG. 3 is a diagram illustrating an example of information stored in a storage unit according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a flowchart illustrating a processing procedure for display control by the information processing device according to the first embodiment of the present invention. [Figure 7] FIG. 10 is a schematic diagram showing the configuration of an information processing system according to a second embodiment of the present invention. [Figure 8] FIG. 10 is a block diagram showing the configuration of an information processing device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] (First embodiment) An information processing device 100, a program, and an information processing system 1000 according to a first embodiment of the present invention will be described below with reference to the drawings.

[0013] First, the overall configuration of an information processing system 1000 will be described with reference to FIG.

[0014] The information processing system 1000 is a system for live-streaming content including a virtual model VM (see FIG. 3) generated by a distributor to viewers. In addition, in the information processing system 1000, viewers of the live stream can support the distributor through a distribution platform provided by a server device 150 via a user terminal 160.

[0015] Support refers to all elements provided by viewers to streamers, including tips, gifts, Super Chats, items, and bits. Support includes not only money itself but also objects with monetary value. Below, support is explained in the form of gifts given (thrown) to streamers on the streaming platform.

[0016] A publicly known system for support (tipping, gifting) on ​​a distribution platform can be adopted. For example, viewers can charge money (Japanese yen) on the distribution platform, convert it into the platform's own currency (coins, tokens), and use this unique currency to purchase gifts (items) of various values. Viewers can give gifts to streamers during live streaming as support money, assistance, or as part of so-called "oshikatsu" (supporting a streamer). The streamer can receive compensation from the provider of the distribution platform based on the gifts given by viewers.

[0017] As shown in FIG. 1, the information processing system 1000 includes a server device 150 (video distribution server), a user terminal 160 used by a viewer, and an information processing device 100 (distributor terminal) used by a distributor.

[0018] The server device 150 is configured by a computer including a CPU 151 (Central Processing Unit), a storage device 152 configured with a ROM (Read Only Memory) and a RAM (Random Access Memory), and an I / O interface 153 (Input / Output Interface). The RAM stores data for processing by the CPU 151, the ROM stores control programs and the like for the CPU 151 in advance, and the I / O interface 153 is used for inputting and outputting information to and from connected devices. The CPU 151 executes the control program to perform various processes of the information processing device 100 described in this specification. Note that the server device 150 may be configured as a single device, or may be divided into multiple devices and configured so that each control is distributed among the multiple devices.

[0019] The server device 150 is configured to be able to communicate with the information processing device 100 and the user terminal 160 via a network NW. The server device 150 provides an application that realizes a distribution platform that distributes videos provided by distributors via the information processing device 100 to viewers via the user terminal 160. The distribution platform provided by the server device 150 allows distribution to multiple viewers.

[0020] Distributors and viewers can distribute and view videos from a web page provided by server device 150 as part of the distribution platform, via information processing device 100 and user terminal 160, respectively. Specifically, in response to a request from a distributor via information processing device 100, server device 150 outputs a web page (hereinafter also referred to as a "web page for distributor") for outputting a video to be distributed (hereinafter also referred to as a "stock video") to information processing device 100. In addition, in response to a request from a viewer via user terminal 160, server device 150 outputs a web page (hereinafter also referred to as a "web page for viewer") for viewing content including the stock video to user terminal 160.

[0021] 2, the server device 150 has, as its functional configuration, a material acquisition unit 155 that acquires stock videos provided by distributors through the information processing device 100, a distribution unit 156 that distributes (provides) content including the stock videos to the user terminal 160, and an input acquisition unit 157 that acquires comments and support from viewers through the user terminal 160. Each component of the server device 150 shown in FIG. 2 represents each function of the server device 150 as a virtual unit, and does not necessarily mean that it physically exists.

[0022] When a request signal is received from a distributor via information processing device 100, material acquisition unit 155 displays a distributor webpage for outputting material video from information processing device 100 on display device 130 of information processing device 100, which will be described later, and accepts the provision of material video. When material video is output from information processing device 100 via the distributor webpage, material acquisition unit 155 acquires the material video. The acquired material video is input to distribution unit 156.

[0023] When a request signal is received from a viewer via user terminal 160, distribution unit 156 displays a web page for the viewer to view the content on display device 164 of user terminal 160 (described later), as shown in Fig. 3, and distributes the content to the viewer. The content includes not only the raw video but also comments and support from the viewer acquired by input acquisition unit 157 (described later).

[0024] As shown in Figure 3, the content display screen provided through the viewer webpage includes a video display field Fm for displaying raw video, a comment field Fc for displaying comments and support, and an input field for viewers to enter comments and support content.

[0025] In this embodiment, the comment field Fc and the input field are arranged superimposed on the video display field Fm. In other words, the video display field Fm displays a video in which comments and support are superimposed on a raw video. Note that the comment field Fc and the input field are not limited to being superimposed on the video display field Fm, and may be displayed separately from the video display field Fm on the display device 164 of the user terminal 160. The dashed lines in FIG. 3 indicate imaginary boundaries that separate the comment field Fc.

[0026] In the comment field Fc, when a new comment is entered, the new comment is displayed from the bottom of the screen, and the comments and support that are already displayed are displayed so that they move toward the top. There is a limit to the number of comments that can be displayed in the comment field Fc, and if a new comment is entered when the number of comments already entered has reached the limit, the oldest comments will be deleted from the comment field Fc.

[0027] The input acquisition unit 157 accepts comments and support from viewers through input fields on the viewer webpage. The input fields include a comment input field Fi, a support selection field Fg, and a send button B1. The comment input field Fi is configured to allow input of text indicating a comment. The support selection field Fg is configured to allow selection of pre-set support content, specifically gifts with different monetary values ​​(amounts required for purchase) (e.g., gifts A to E), and "none," which means no gift will be sent. Note that, as described above, support may be money itself rather than a gift, and when money itself is sent as support, the amount of support may be selected from pre-set options, or the viewer may be able to set an arbitrary amount.

[0028] When the send button B1 is selected, the text entered in the comment input field Fi and the support content selected in the support selection field Fg are output from the user terminal 160 to the server device 150, and the comment and support content are acquired by the input acquisition unit 157. The acquired comment and support content are input to the distribution unit 156. Note that when the send button B1 is operated with "none" selected as the support content, only the comment is input to the information processing device 100.

[0029] 1, the user terminal 160 is configured by a computer including a CPU 161 (Central Processing Unit), a storage device 162 configured with a ROM (Read Only Memory) and a RAM (Random Access Memory), and an I / O interface 163 (Input / Output Interface). The RAM stores data for processing by the CPU 161, the ROM stores control programs and the like for the CPU 161 in advance, and the I / O interface 163 is used for inputting and outputting information to and from connected devices. The CPU 161 executes the control programs to perform various processes of the user terminal 160 described in this specification. The user terminal 160 may be configured as a single device, or may be divided into multiple devices and configured so that each control is distributed among the multiple devices.

[0030] User terminal 160 is a terminal used by each viewer who watches the content distributed by server device 150, and may be, for example, a mobile terminal such as a smartphone, or a personal computer (PC). Server device 150 can distribute content to multiple user terminals 160, but only a single user terminal 160 is shown in the drawing.

[0031] The user terminal 160 further includes a display device 164 that displays information, and an input device 165 that accepts operational inputs from the viewer. The viewer can check the content (or the viewer web page) distributed from the server device 150 through the display device 164, and can output comments and support to the server device 150 by operating the input device 165.

[0032] The information processing device 100 is a terminal used by a distributor who distributes moving images, and is configured to be able to communicate with the server device 150 via a network NW. The information processing device 100 creates and controls the moving images to be distributed, and transmits the created moving images to the server device 150 for live distribution. The information processing device 100 distributes moving images using a fictitious virtual model VM.

[0033] The information processing device 100 includes a control device 110 that executes various processes, an imaging device 120 that captures images, a display device 130 that displays information, and an input device 131 through which a broadcaster performs operational input. The control device 110 is connected to the imaging device 120, the display device 130, and the input device 131 by wire or wirelessly so as to be able to communicate with them. The information processing device 100 may be a so-called desktop computer in which the control device 110, the imaging device 120, the display device 130, and the input device 131 are configured separately. Alternatively, the information processing device 100 may be a laptop computer in which the control device 110, the imaging device 120, the display device 130, and the input device 131 are configured integrally, or a mobile terminal such as a smartphone or tablet.

[0034] The control device 110 is configured by a computer including a CPU 111 (Central Processing Unit), a storage device 112 configured with a ROM (Read Only Memory) and a RAM (Random Access Memory), and an I / O interface 113 (Input / Output Interface). The RAM stores data for processing by the CPU 111, the ROM stores control programs and the like for the CPU 111 in advance, and the I / O interface 113 is used for inputting and outputting information to and from connected devices. The CPU 111 executes the control programs to perform various processes described in this specification. The control device 110 may be configured as a single device, or may be divided into multiple devices and configured so that each control is distributed among the multiple devices.

[0035] The control device 110 is configured to be able to communicate with the server device 150 via the network NW.

[0036] The imaging device 120 includes a camera 125 for capturing an image and a controller 121 for controlling the operation of the camera 125. The imaging device 120 can be configured as a so-called AI camera that integrally includes the camera 125 (image sensor) and the controller 121 (edge ​​computer) that can perform image processing. The imaging device 120 may also be configured as a mobile terminal such as a smartphone that integrates a computer, a display device, an input device, and a camera. The imaging device 120 may also be configured such that the controller 121 and the camera 125 are provided as separate entities and connected to each other.

[0037] The controller 121 is configured by a computer including a CPU 122 (Central Processing Unit), a storage device 123 configured with a ROM (Read Only Memory) and a RAM (Random Access Memory), and an I / O interface 124 (Input / Output Interface). The RAM stores data for processing by the CPU 122, the ROM stores control programs and the like for the CPU 122 in advance, and the I / O interface 124 is used for inputting and outputting information to and from connected devices. The CPU 122 executes the control program to perform the various processes described in this specification.

[0038] The controller 121 executes predetermined processing and controls the operation of the camera 125 based on a control command from the control device 110 .

[0039] The display device 130 is a so-called monitor, and displays information based on a signal input from the control device 110. The input device 131 is a device operated by the distributor, and the distributor's operation input via the input device 131 is input to the control device 110.

[0040] In the information processing device 100, the control device 110 creates and controls material images and outputs them to the server device 150, and also receives content distributed from the server device 150 and displays it on the display device 130. The imaging device 120 continuously captures images of the content (hereinafter also referred to as "display images") displayed on the display device 130 to acquire a moving image. The imaging device 120 identifies the content of the support provided by the viewer from the acquired display images and outputs the identified content of the support to the control device 110. The control device 110 changes the display mode of the material video according to the content of the acquired support.

[0041] Fig. 4 is a diagram showing the functional configuration of the information processing device 100. Each component of the information processing device 100 shown in Fig. 4 is shown as a virtual unit representing each function of the information processing device 100, and does not necessarily mean that it physically exists.

[0042] 4, the information processing device 100 has, as its functional configuration, a material generation unit 10 that generates material videos, a delivery request unit 11 that requests delivery from the server device 150, a display control unit 12 that controls the display of the material videos, an image acquisition unit 21 that acquires a display image that displays content delivered from the server device 150 to the user terminal 160, a support identification unit 22 that identifies the content of support from the viewer from the display mode of the support on the display image, and a memory unit 13 that stores information. The material generation unit 10, the delivery request unit 11, the display control unit 12, and the memory unit 13 are implemented in the control device 110. The image acquisition unit 21 and the support identification unit 22 are implemented in a controller 121 of the imaging device 120.

[0043] The material generation unit 10 generates a three-dimensional humanoid virtual model VM (see FIG. 3). The virtual model VM can be generated by any method, such as using a generation AI that generates a three-dimensional model by providing text or images as input, creating a new virtual model VM based on a real person or an existing humanoid object (avatar) (a method using so-called deep fake technology), or manually creating a completely new three-dimensional model. In this embodiment, the virtual model VM is humanoid, but the invention is not limited to this.

[0044] The material generation unit 10 generates a plurality of patterns of appearance (display mode) such as facial expressions and gestures for the virtual model VM. Of the generated patterns of display mode of the virtual model VM (hereinafter also simply referred to as "display pattern"), one pattern is set as a standard (default) display pattern. Hereinafter, for the virtual model VM, the standard display mode pattern will be referred to as a "standard pattern", and other display mode patterns will be referred to as "changed patterns". The same method as for generating the virtual model VM can be used to generate the patterns of display mode of the virtual model VM. Each pattern of the display mode of the virtual model VM generated by the material generation unit 10 is stored in advance in the storage unit 13.

[0045] The standard pattern can be set as a display pattern in which the virtual model VM stands upright with a reference facial expression and blinks at predetermined time intervals, for example.

[0046] The change pattern can be set as a display pattern that differs in appearance from the standard pattern, such as facial expressions such as smiling, sad, angry, or expressionless, actions such as raising one's arms in the air, waving, folding one's arms, or lip-syncing to utter a predetermined phrase, or other predetermined external effects. Also, a specific sound (sound effect, voice, music) may be generated in association with the display pattern.

[0047] The display control unit 12 creates a source video including the generated virtual model VM and performs display control to change the display mode of the source video. Specifically, the display control unit 12 creates a video in which a standard pattern virtual model VM arranged in three-dimensional space is viewed from a predetermined angle. The display control unit 12 also changes the virtual model VM based on predetermined rules. The display control by the display control unit 12 will be described in detail later. The source video created and the display mode of which has been controlled by the display control unit 12 in this manner is input to the distribution request unit 11.

[0048] The distribution request unit 11 outputs a distribution request for distributing a moving image and the raw moving image controlled by the display control unit 12 to the server device 150.

[0049] The image acquisition unit 21 acquires a display image through the camera 125 that captures the content displayed on the display device 130. The acquired display image is input to the support specification unit 22.

[0050] The support identification unit 22 identifies the content of the support from the support display Gf (see FIG. 3) in the comment field Fc of the display image. Specifically, the support identification unit 22 identifies the content of the support included in the display image by object recognition using a trained model that has learned the relationship between the display of an object in the image and the content (classification) of the object. Since a known object recognition technology can be adopted, a detailed description will be omitted. The trained model may be one that is provided in the information processing device 100, or one that is provided in a device external to the information processing device 100 and executed by being called by the image acquisition unit 21. The identified content of the support is input to the display control unit 12.

[0051] The storage unit 13 is configured by, for example, a database, and stores information necessary for processing executed by the information processing device 100. Specifically, as shown in Fig. 5, the storage unit 13 stores the support content, the display pattern of the virtual model VM, and the selection probability that each display pattern is used by display control, in association with each other.

[0052] In this embodiment, a plurality of change patterns for the virtual model VM are set for one support content, except for "none". A selection probability for selection by display control when the corresponding support is provided is set for each change pattern. In this embodiment, the selection probability is set to be equal for each support content. Note that one change pattern may be set that does not change the display mode (in other words, the same as the standard pattern). Furthermore, even when the support is "none", the display pattern may be changed based on a predetermined selection probability.

[0053] Next, a description will be given of the operations of the information processing device 100 and the information processing system 1000. Note that although each process executed by the information processing device 100 and the server device 150 is executed by each functional unit of the information processing device 100 and the server device 150, the following description will be given assuming that the process is executed by the information processing device 100 or the server device 150.

[0054] A method of video distribution using a video distribution platform performed by the information processing system 1000 will be described below.

[0055] First, in preparation for distribution by the distributor, the control device 110 of the information processing device 100 controls the raw video including the virtual model VM, and is set to display the content distributed from the server device 150 on the display device 130. In addition, the imaging device 120 is set so that the camera 125 captures the content displayed on the display device 130.

[0056] When a distributor distributes a video, a material video including a virtual model VM is created by the control device 110 of the information processing device 100, and a distribution request and the material video are output to the server device 150. Specifically, the control device 110 calls up the virtual model VM stored in the storage unit 13 and creates a material video based on a standard pattern. Since known techniques can be used to create the material video, detailed explanations will be omitted, but the material video can be created, for example, by the distributor operating the input device 131 and executing a program.

[0057] When server device 150 receives a distribution request from control device 110 of information processing device 100, it displays content including the raw video and a comment field Fc that displays comments received from viewers on a web page for viewers (see FIG. 3). Furthermore, in response to a request from a viewer via user terminal 160, server device 150 provides user terminal 160 with a web page for viewers that displays the content.

[0058] Viewers can view broadcasts by broadcasters by accessing a viewer webpage provided by the server device 150. Viewers can also provide comments and support for the broadcast through a comment input field Fi and a support selection field Fg on the viewer webpage. Comments and support from viewers entered into the server device 150 are displayed in real time in the comment field Fc. In this way, videos are distributed on the distribution platform.

[0059] Next, a method for controlling a material video (information processing method) by information processing device 100 will be described with reference to FIG. Fig. 6 is a flowchart of processing executed by information processing device 100. When distribution by a distributor starts, information processing device 100 controls the material video by executing the processing shown in Fig. 6 at predetermined time intervals.

[0060] 6, in step S10, a display image of the distributed content is acquired from a video captured by camera 125 on display device 130. Step S10 corresponds to a step of acquiring a display image that displays the content distributed from server device 150.

[0061] In step S11, the information processing device 100 performs processing (image processing) to recognize a display Gf of support in the display image. Note that in step S11, processing is performed using a tracking (object tracking) technique that recognizes the identity of an object (in other words, tracks an object in the image) so that a display Gf of support recognized in a previous process is not recognized as new support.

[0062] In step S12, it is determined whether support was recognized in step S11, and if there is no support, the process ends. In step S12, if support from a viewer is recognized in the displayed image, the process proceeds to step S13.

[0063] In step S13, the content of the recognized support is identified. Step S13 corresponds to a step of identifying the content of the support from the display mode of the support on the display image.

[0064] In step S14, one change pattern to be used for display control is selected from the change patterns corresponding to the support content identified in step S13 based on the selection probability. The change pattern can be selected using a set selection probability and a random number. For example, as shown in Figure 3, if support for Gift C is provided, one change pattern is selected from the change patterns corresponding to support for Gift C based on the selection probability (5% in Figure 5).

[0065] In step S15, display control is performed to change the display mode of the virtual model VM on the source video according to the change pattern selected in step S14. The display time of the selected change pattern may be set in advance for each display pattern and stored in the storage unit 13, or may be set to a uniform arbitrary time. When the display time of the change pattern has elapsed, the process proceeds to step S16. Step S15 corresponds to a step of performing display control to change the display mode of the virtual model VM according to the content of the support.

[0066] In this way, by changing the display mode of the virtual model VM in response to support from the viewer, an interactive user experience can be provided to the viewer, improving viewer satisfaction and increasing their willingness to support.

[0067] The viewer may provide multiple supports with the same content at approximately the same time (within a relatively short predetermined time range) (two cases will be explained below as examples). In such cases, it is preferable to control the reactions to the supports (the display modes of the virtual model VM) so that they do not overlap.

[0068] Specifically, when two supports are recognized in step S11, they are identified as having the same content in steps S12 and S13. Then, in step S14, one change pattern is selected from a plurality of change patterns for each support based on the selection probability. At this time, if the selected change pattern overlaps between the two supports, a process is executed to reselect the change pattern for either one so that it does not overlap. Then, the display mode of the virtual model VM is changed according to each of the two change patterns selected corresponding to the two supports, and a different reaction is made to each of the two supports. By executing such a process, it is possible to improve viewer satisfaction.

[0069] This process of avoiding overlapping change patterns can also be applied when the same support is provided in successive processing flows. For example, each time the processing flow shown in FIG. 6 is executed, the content of the support and the selected change pattern are at least temporarily stored. Then, when the same support is provided in a new processing flow executed within a predetermined time or a predetermined number of processing executions after a certain processing flow, the change pattern used for display control is made different. This also improves viewer satisfaction.

[0070] In step S16, display control is performed to return the display mode of the virtual model VM to the standard pattern, and the process ends.

[0071] The information processing system 1000 and the information processing device 100 according to this embodiment have been described above, but the series of processes in the information processing device 100 described above may be provided as a program for causing a computer to execute the processes.

[0072] For example, the program according to the above embodiment is a program configured to be executable by the information processing device 100 as a computer, and causes the computer to execute the program.

[0073] Furthermore, a program for executing the above-described series of processes is provided by a storage medium readable by the information processing device 100. The program may also be provided to the information processing device 100 via a network NW. Furthermore, part of the processes performed by the information processing device 100 described in the above embodiment may be executed by the user terminal 160 or the server device 150.

[0074] According to the present embodiment described above, the following advantageous effects are achieved.

[0075] The information processing device 100 can independently identify the support content based on the displayed image and perform control according to the support content. Therefore, the distributor can control the display of the virtual model VM in response to support from the viewer without receiving information about the support content from the server device 150, thereby improving convenience for the distributor.

[0076] Here, as a modified example of display control by the information processing device 100, when support is provided, it is also possible to change the display mode of the virtual model VM in real time using a trained model (generative AI) that generates the display mode of the virtual model VM. For example, support (and comments) provided by a viewer can be input into the trained model, and the display mode of the virtual model VM can be generated and controlled in real time according to the level of support (e.g., amount of money). In such a case, it may be possible to control the virtual model VM in a display mode that is more appropriate in response to the support provided by the viewer, making it possible to further attract viewers.

[0077] In contrast, in this embodiment, the change pattern to be used for control is selected from change patterns stored in advance in the storage unit 13. That is, in this embodiment, the display mode of the virtual model VM is changed by control using a predetermined algorithm or so-called rule-based control. This makes control easier and reduces the computer resources required for control compared to the modified example in which a change pattern for the virtual model VM is generated in real time by a generation AI. Furthermore, compared to the modified example, this embodiment does not have hallucinations that occur in the generation AI, so it is possible to achieve highly predictable and intended control.

[0078] In this embodiment, the probability of selecting a change pattern is equal for each support content. That is, the change pattern used in display control is selected randomly. As a result, the greater the number of change patterns, the less bias occurs in the pattern selection, preventing viewer boredom and improving satisfaction.

[0079] Furthermore, in this embodiment, the information processing device 100 includes a control device 110 that controls the display of the virtual model VM, and an imaging device 120 that captures display images and identifies support content. In this way, by distributing the display control of the virtual model VM and the acquisition and image processing of display images by different terminals, the processing load is distributed compared to when these are executed on a single terminal, and therefore high processing power (specifications) is not required for each terminal. Therefore, the distributor can easily execute the information processing method according to this embodiment.

[0080] (Second embodiment) Next, an information processing device 200 according to a second embodiment will be described with reference to Fig. 7. The following description will focus on differences from the first embodiment, and the same components as those in the first embodiment will be denoted by the same reference numerals and descriptions thereof will be omitted as appropriate.

[0081] The information processing device 100 according to the first embodiment includes two computers: a control device 110 and a controller 121 of the imaging device 120, and the processing executed by the information processing device 100 is distributed and processed by the control device 110 and the controller 121 of the imaging device 120.

[0082] In contrast, the information processing device 200 according to the second embodiment includes a control device 210, which is a single computer, and processes are executed by the single computer (control device 210). The hardware configuration of the control device 210 (CPU 201, storage device 202, I / O interface 203) is the same as that of the control device 110 according to the first embodiment, and therefore a description thereof will be omitted.

[0083] 7, a control device 210 of an information processing device 200 according to the second embodiment is equipped with a material generation unit 10, a distribution request unit 11, a display control unit 12, an image acquisition unit 21, a support specification unit 22, and a storage unit 13. Furthermore, the information processing device 200 does not necessarily have to include the imaging device 120 as in the first embodiment.

[0084] In the second embodiment, the control device 210 is configured to be able to communicate with the server device 150 via the network NW. The control device 210 does not output the content (in other words, a web page for the viewer) distributed from the server device 150 to the display device 130, but instead acquires a display image of the content using the image acquisition unit 21. The support identification unit 22 acquires the display image from the image acquisition unit 21 and identifies the content of the support from the viewer from the display image. That is, in the second embodiment, the information processing device 200 directly acquires the display image from the data distributed from the server device 150 and identifies the content of the support from the viewer, without going through the steps of displaying the content on the display device 130 and capturing an image of the display device 130 using the imaging device 120.

[0085] According to the second embodiment, as in the first embodiment, the display of the virtual model VM can be controlled in response to support from the viewer without receiving information regarding the content of the support from the server device 150, thereby improving convenience for the broadcaster.

[0086] Furthermore, in the second embodiment, the information processing device 200 is configured by a single computer (control device 210), so that processing can be completed within the information processing device 200, and the device configuration can be simplified.

[0087] (Third embodiment) Next, an information processing device according to a third embodiment will be described. The following description will focus on differences from the first and second embodiments, and the same components as those in the first or second embodiment will be denoted by the same reference numerals and descriptions thereof will be omitted as appropriate.

[0088] In the first embodiment, the information processing device 100 has two computers: the control device 110 and the controller 121 of the imaging device 120. The controller 121 of the imaging device 120 is equipped with an image acquisition unit 21 and a support specification unit 22.

[0089] In the second embodiment, the information processing device 200 includes one computer (control device 210). In the information processing device 200 of the second embodiment, the image acquisition unit 21 directly acquires the content distributed from the server device 150.

[0090] In contrast, the information processing device according to the third embodiment is configured by one computer as in the second embodiment, and acquires the content display screen by an imaging device 120 that captures an image of a display device 130 as in the first embodiment. In other words, the third embodiment is a combination of the configurations of the first and second embodiments. The information processing device according to the third embodiment is similar to the first embodiment in that it includes a control device, an imaging device, a display device, and an input device (see FIG. 1), and its functional configuration is similar to that of the second embodiment (see FIG. 8).

[0091] The third embodiment differs from the first embodiment in that the functional configuration of the information processing device is implemented in a single computer. The imaging device in the third embodiment is not limited to a terminal equipped with a computer, such as an AI camera or a mobile terminal such as a smartphone, but may also be a device such as a digital camera that does not have a controller that performs image processing. The operation of the imaging device, acquisition of the display image, and identification of the assistance content (image processing) are performed by a control device of the information processing device, not by a computer in the imaging device.

[0092] According to the third embodiment, as in the first and second embodiments, the display of the virtual model VM can be controlled in response to support from the viewer without receiving information regarding the content of the support from the server device 150, thereby improving convenience for the broadcaster.

[0093] Furthermore, in the third embodiment, since the imaging device 120 is provided, it is possible to easily obtain a display image that displays the content.

[0094] Next, modifications of each embodiment will be described.

[0095] (First Modification) In each of the above embodiments, the display control by the display control unit 12 changes the display mode of the virtual model VM in the source video in accordance with the content of support from the viewer. In addition, the display mode of the background in the source video can also be changed.

[0096] Specifically, similar to the display control of the display mode of the virtual model VM in each embodiment, a plurality of background change patterns corresponding to the content of support are stored in the storage unit 13, and a background change pattern to be used for display control is selected from the plurality of change patterns depending on the content of support. In this way, by changing the display mode of the background in addition to the virtual model VM, it is possible to easily increase the variety of viewer reactions to support, and improve user satisfaction.

[0097] Note that a non-changing pattern (same as the standard pattern) may be set as the background change pattern. The background change pattern may be selected randomly as in the virtual model VM in each of the above embodiments, or may be configured to be displayed in a corresponding background change pattern only when the virtual model VM is displayed in a specific change pattern.

[0098] (Second Modification) In each of the above embodiments, the support identification unit 22 identifies the content of support from the display image by object recognition using a trained model. However, the identification of the content of support is not limited to using object recognition using a trained model, as long as the configuration is such that the content of support is identified from the display image.

[0099] For example, the content of the assistance may be identified by applying rule-based image processing to the display image without using a trained model. Even in such a modified example, the same effects as those of the above-described embodiments can be achieved.

[0100] (Third Modification) In each of the above embodiments, the selection probability of a change pattern is set to be equal for each support content. However, there may be a bias in the selection probability of change patterns for the same support content. In other words, among multiple change patterns for a certain support content, one or more change patterns that are part of the multiple change patterns may be set to have a lower selection probability than the other change patterns. Also, in this case, the selection probability of each change pattern can be set individually.

[0101] According to such a modified example, by setting a change pattern with a lower selection probability than other change patterns, it is possible to add a gambling element to the viewer's experience of seeing whether a rare change pattern will appear, thereby improving the viewer's experience value (user engagement).

[0102] (Fourth Modification) In each of the above embodiments, a change pattern for the virtual model VM corresponding to the content of the support is selected. However, it is also possible to select a change pattern according to the content of the support and the comment, which is text information input together with the support, and perform display control.

[0103] For example, if a comment entered together with support contains a predetermined phrase, the system can select one or more preset change patterns to be used for display control from among a plurality of change patterns corresponding to the support content, with a selection probability also set in advance.

[0104] Alternatively, the input comment may be classified using a trained model, and a change pattern to be used for display control may be selected according to the classification. Alternatively, a reply to the input comment may be generated using a trained model (a so-called large-scale language model), and the display mode may be changed based on the generated reply (for example, lip-syncing the virtual model VM to utter the generated reply).

[0105] In this way, by controlling the display mode of the virtual model VM based on the content of the comment in addition to the support, it is possible to respond more appropriately to the viewer's input, thereby further improving viewer satisfaction.

[0106] (Other variations) In the above embodiments, the source video includes a virtual model VM in a three-dimensional space. However, the source video is not limited to three-dimensional content and may be expressed in two dimensions.

[0107] Furthermore, the material video is not limited to one in which the virtual model VM is viewed from a predetermined angle, but may be one in which the viewpoint of the virtual model VM is switched as appropriate.

[0108] The effects of the above-described embodiments will be described below.

[0109] An information processing device 100, 200 configured to be able to communicate with a server device 150 that distributes content including a raw video including a virtual model VM and support content provided by a viewer includes a display control unit 12 that controls the display of the raw video, an image acquisition unit 21 that acquires a display image that displays the content distributed from the server device 150, and a support identification unit 22 that identifies the support content from the viewer from the display mode of the support on the display image, and the display control unit 12 changes the display mode of the virtual model VM according to the support content.

[0110] The information processing method by the information processing device 100, 200 includes the steps of acquiring a display image displaying distribution content from the server device 150 including a raw video including a virtual model VM and content of support from viewers regarding the distribution, identifying the content of the support from the display mode of the support on the display image, and performing display control to change the display mode of the virtual model VM according to the content of the support.

[0111] The program, which is executable by the information processing device 100, 200, which is a computer, causes the computer to perform the following steps: acquiring a display image that displays the content distributed from the server device 150, which includes a raw video including a virtual model VM and the content of support from the viewer for the distribution; identifying the content of the support from the display mode of the support on the display image; and performing display control that changes the display mode of the virtual model VM according to the content of the support.

[0112] According to these configurations, the distributor can independently identify the support content based on the displayed image and perform control according to the support content. Therefore, the distributor can control the display of the virtual model VM in response to support from the viewer without receiving information about the support content from the server device 150, thereby improving convenience for the distributor.

[0113] Furthermore, in the information processing devices 100 and 200, the source video includes a background showing an area other than the area occupied by the virtual model VM, and the display control unit 12 changes the display mode of the background depending on the content of the support.

[0114] This configuration increases the variety of display modes of the material video, thereby improving viewer satisfaction.

[0115] In addition, the information processing devices 100, 200 further include a memory unit 13 that pre-stores multiple change patterns that change the display mode of the raw video depending on the content of the support, and the display control unit 12 selects a change pattern to be used for display control from the multiple change patterns that correspond to the content of the support.

[0116] In this configuration, display control is performed based on change patterns stored in advance in the storage unit 13, which has the advantages of realizing easy control, reducing computer resources, and realizing stable control with high predictability.

[0117] Furthermore, in the information processing devices 100 and 200, the display control unit 12 selects a change pattern to be used for display control from a plurality of change patterns corresponding to support, based on a selection probability set for each change pattern.

[0118] In this configuration, the change pattern is selected randomly or irregularly, which can prevent viewers from getting bored (losing interest) and increase their motivation to provide support.

[0119] Furthermore, in the information processing devices 100 and 200, a predetermined change pattern among a plurality of change patterns is set to have a lower selection probability than other change patterns.

[0120] In this configuration, the change in the display mode of the virtual model VM can be made to have a kind of game-like quality, which can improve the experience value of the viewer and their motivation to help.

[0121] Furthermore, in the information processing devices 100 and 200, the support includes a comment, which is text information, and the display control unit 12 selects a change pattern to be used for display control based on the content of the comment.

[0122] In this configuration, by controlling the display mode of the virtual model VM based on the content of the comment in addition to the support, it is possible to respond more appropriately to the viewer's input, thereby further improving viewer satisfaction.

[0123] Furthermore, in the information processing devices 100 and 200, the support identification unit 22 identifies the content of the support by object recognition using a trained model that has learned the relationship between the display mode of the support on the display image and the content of the support.

[0124] In this configuration, even if information is not provided from the server device 150, the content of the support can be identified from the display image showing the content.

[0125] The information processing device 100 further includes a display device 130 that displays content and an imaging device 120 that captures an image of the display screen of the display device 130, and the image acquisition unit 21 acquires the image of the display screen captured by the imaging device 120 as a display image.

[0126] In this configuration, a display image that displays the content can be easily acquired.

[0127] In addition, the information processing device 100 further includes a control device 110 that is communicatively connected to the imaging device 120, and the display control unit 12 is implemented in the control device 110, and the image acquisition unit 21 and the support identification unit 22 are implemented in the imaging device 120.

[0128] In this configuration, the processing can be distributed, reducing the processing load on each device. As a result, the performance required of each device can be reduced. Since the imaging device 120 can be a publicly known mobile terminal (e.g., a smartphone), the information processing device 100 can be easily constructed.

[0129] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments. [Explanation of symbols]

[0130] 100 Information processing device 200 Information processing device 150 Server device 12 Display control unit 13 Storage section 21 Image acquisition unit 22 Support Specification Department 110 Control device 120 Imaging device 121 Controller 130 Display device VM Virtual Model

Claims

1. An information processing device configured to be able to communicate with a server device that distributes content including a raw video including a virtual model and support content provided by a viewer, a display control unit that controls the display of the material video; an image acquisition unit that acquires a display image that displays the content distributed from the server device; a support identification unit that identifies the content of the support from the viewer from a display mode of the support on the display image, the display control unit changes a display mode of the virtual model according to the content of the support; the support identification unit identifies the content of the support by object recognition using a trained model that has learned a relationship between a display mode of the support on the display image and the content of the support; Information processing device.

2. 2. The information processing device according to claim 1, the material video includes a background showing an area other than an area occupied by the virtual model, the display control unit changes the display mode of the background depending on the content of the support. Information processing device.

3. 2. The information processing device according to claim 1, a storage unit that stores in advance a plurality of change patterns for changing the display mode of the raw video according to the content of the support; the display control unit selects the change pattern to be used in the display control from a plurality of change patterns corresponding to the content of the assistance. Information processing device.

4. 4. The information processing device according to claim 3, the display control unit selects the change pattern to be used for the display control from the plurality of change patterns corresponding to the support, based on a selection probability set for each of the change patterns. Information processing device.

5. 5. The information processing device according to claim 4, a predetermined change pattern among the plurality of change patterns is set to have a lower selection probability than other change patterns; Information processing device.

6. 4. The information processing device according to claim 3, The support includes a comment that is text information, the display control unit selects the change pattern to be used for the display control based on the content of the comment. Information processing device.

7. 2. The information processing device according to claim 1, a display device for displaying the content; an imaging device that captures an image of the display screen of the display device, the image acquisition unit acquires an image of the display screen captured by the imaging device as the display image; Information processing device.

8. 8. The information processing device according to claim 7, a control device configured to be able to communicate with the imaging device; the display control unit is implemented in the control device, the image acquisition unit and the support specification unit are implemented in the imaging device; Information processing device.

9. acquiring a display image showing content distributed from a server device, the content including a raw video including a virtual model and support from a viewer for the distribution; identifying the content of the support from a display mode of the support on the display image; performing a display control step of changing a display mode of the virtual model according to the content of the support; and identifying the content of the support by object recognition using a trained model that has learned the relationship between the display mode of the support on the display image and the content of the support. Information processing methods.

10. A program configured to be executable by a computer, acquiring a display image showing content distributed from a server device, the content including a raw video including a virtual model and support from a viewer for the distribution; identifying the content of the support from a display mode of the support on the display image; performing a display control step of changing a display mode of the virtual model according to the content of the support; and identifying the content of the assistance by object recognition using a trained model that has learned the relationship between the display mode of the assistance on the display image and the content of the assistance. program.

Images