Video processing device, method, and program

Abstract

To enable presentation of visual information that enhances excitement of a user viewing remotely distributed content.SOLUTION: A video processing device according to an embodiment comprises an acquisition unit configured to acquire information indicative of content of an operation performed by a user viewing content including video and audio on a display screen of the content, and a generation unit configured to generate visual information that can be superimposed on the content displayed on the display screen on the basis of a relationship between periodic audio content included in the content and content of the operation indicated by the information acquired by the acquisition unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a video processing device, a method, and a program. [Background technology]

[0002] When participating in online events such as watching live music or sporting events, remote spectator users are forced to one-sidedly view content including the streamed video and audio of the event, making it difficult for them to get excited by applauding or cheering like on-site spectators.

[0003] Therefore, efforts are being made to create two-way communication using text chat (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Yuki Abe, Myung-Keun Choi, Daisuke Sakamoto, and Tetsuo Ono, "Experimental Verification of Input Performance of a Translucent Double-Flick Keyboard for Stream Live Chat Input," Proceedings of IPSJ Interaction 2022, pp.125-134 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above text chat requires the user to input characters, which causes a problem in that the user's concentration on the distributed content is interrupted.

[0006] One method for solving this problem is to present a visual effect, such as a heart mark, by a simple action of the user tapping on the content display screen.

[0007] When the content is a live music performance, there are two patterns in which a user taps on the display screen: one is to tap in time with the rhythm of the song being streamed, and the other is to tap continuously at short intervals regardless of the rhythm of the song in order to express the user's excitement or support for the event.

[0008] However, in the current method, the tap operation by the user and the visual effect presented by this tap operation have a 1:1 correspondence, for example, one heart mark is presented with one tap operation. Therefore, the above two patterns behind the tap operation are not taken into consideration, and there is no feedback from the content distribution system to increase the user's excitement according to the type of the pattern.

[0009] Moreover, it is desirable that the information fed back to the user from the distribution system be as simple as possible so as not to break the user's concentration on the content. Therefore, it is not desirable to increase the types of the above-mentioned feedback information, such as characters, other marks, or effects for emphasis.

[0010] The present invention has been made in light of the above-mentioned circumstances, and its object is to provide a video processing device, method, and program that are capable of presenting visual information that will increase the excitement of users viewing content distributed remotely. [Means for solving the problem]

[0011] A video processing device according to one embodiment of the present invention includes an acquisition unit that acquires information indicating the content of operations performed on a display screen of content including video and audio by a user watching the content, and a generation unit that generates visual information that can be superimposed on the content and displayed on the display screen based on the relationship between the content of periodic audio included in the content and the content of the operations indicated in the information acquired by the acquisition unit.

[0012] A video processing method according to one embodiment of the present invention is a method performed by a video processing device, and includes acquiring, by an acquisition unit of the video processing device, information indicating the content of operations performed on a display screen of content including video and audio by a user watching the content, and generating, by a generation unit of the video processing device, visual information that can be superimposed on the content and displayed on the display screen, based on the relationship between the content of periodic audio included in the content and the content of the operations indicated in the information acquired by the acquisition unit. Effect of the Invention

[0013] According to the present invention, it is possible to present visual information that enhances the excitement of a user viewing content distributed from a remote location. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an application example of a content delivery system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of a procedure for generating visual effect information by a content distribution system according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram showing an example of a hardware configuration of a user terminal according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an application example of a content delivery system according to an embodiment of the present invention. A content delivery system according to an embodiment of the present invention controls the number of visual effects, which are visual information superimposed on content and presented on a display screen of a terminal used by a user viewing delivery content including video and audio delivered from a remote location, particularly live broadcast delivery content of an event that gathers a large number of spectators on-site, such as a live music concert or sporting event, based on the content of the tap operation and the content of the content. This provides dynamic feedback from the delivery system to the user without increasing the types of information superimposed on the content and presented on the display screen, and presents visual effects that increase the user's excitement.

[0016] 1, a content delivery system 100 according to the present embodiment includes a user terminal 10 and a content delivery device 20, which are connectable to each other via a communication network. The user terminal 10 and the content delivery device 20 can function as video processing devices or visual information processing devices.

[0017] The user terminal 10 is a terminal, such as a smartphone, used by a user who wants to view distributed content.

[0018] The content distribution device 20 is a device that provides distribution content to the user terminal 10 via a communication network, and includes a content distribution unit 21, a performance generation unit 22, and a venue performance unit .

[0019] Each component of the user terminal 10 will now be described. The user action acquisition unit 11 of the user terminal 10 acquires user action information indicating the content of a tap operation, which is an operation of touching a display screen of a distribution content by a user using the user terminal 10 with a finger, for example, the interval and timing of the tap operation. This user action acquisition unit 11 can be realized, for example, by a touch display function of a smartphone.

[0020] The control unit 12 generates visual effect information, for example a heart mark, that can be superimposed on the delivered content displayed on a display device (not shown) of the user terminal 10, for example, a display screen of a smartphone, based on the content of the delivered content, here the content of the periodic sound in the content, and the content of the tap operation indicated by the user operation information. The generation of the visual effect will be described later in detail. In addition, the control unit 12 transmits the above user action information to the effect generation unit 22 of the content distribution device 20. Here, we will explain an example in which the function of generating visual effect information based on user action information and the function of transmitting user action information to the content distribution device 20 are implemented as functions of the control unit 12 of the user terminal 10, for example, as functions of an application program (sometimes referred to as an app) built into a smartphone. However, the present invention is not limited to this example, and the function may be implemented as a function of a Web application program that runs on a server device (not shown) that is connectable to the user terminal 10 and the content distribution device 20 via a communication network.

[0021] The display control unit 13 superimposes the visual effects transmitted from the control unit 12 on the video, audio, and meta information of the distributed content on the display device, and displays the visual effects on the display device.

[0022] Next, each unit of the content distribution device 20 will be described. The content distribution unit 21 of the content distribution device 20 transmits the video, audio, and meta information of the distribution content to the user terminal 10. This meta information includes BPM (Beats Per Minute) information, which is the number of beats per unit time related to periodic audio in the distribution content, i.e., the frequency of beats, and information indicating the production content of the local event.

[0023] The performance generation unit 22 generates visual effect information based on each user's tap operation based on the time interval and timing of each user's tap operation indicated in user operation information transmitted from the user terminal 10 used by each of the multiple users, and outputs this generated visual effect information to the content distribution unit 21 and the venue performance unit 23.

[0024] The content distribution unit 21 superimposes this visual effect information on the video of the content being distributed and transmits it to each user's user terminal 10. This allows the tap operations of each user watching remotely to be reflected as visual effects in a virtual event venue and fed back to each user's user terminal 10, recreating a visual effect as if multiple remote spectators were gathering at a single event venue.

[0025] Furthermore, venue production section 23 reflects the visual effect information output from production generation section 22 as the production at the event venue in order to be responsible for the production at the local event venue. As an example, the venue production unit 23 displays the total number of taps by remote spectators, i.e., each user using the user terminal 10, on the display screen of a venue display, here a large display installed at an event venue and displaying images for local spectators. As another example, the venue production unit 23 displays virtual spectator seats on the venue display, and superimposes visual effects associated with the tapping operations on the virtual spectator seats, thereby presenting the presence of remote spectators to local spectators.

[0026] Next, the generation of the visual effect information will be described in detail. In this embodiment, the visual effect information generated by the control unit 12 of the user terminal 10 and the effect generation unit 22 of the content distribution device 20 is classified into the following two types.

[0027] The first visual effect information is a "rhythm effect" that corresponds to a tap operation on the display screen of the user terminal 10 by the user using the user terminal 10 in accordance with the periodic sound in the distributed content, for example, the rhythm of a song played in a music live or the rhythm of an athlete's entrance music or cheering song played in a sports venue. This rhythm effect can be expressed by musical note marks or heart marks, etc.

[0028] The second visual effect information is a "cheering effect" that corresponds to a continuous tap operation on the display screen of the user terminal 10 by the user using the user terminal 10 at short intervals compared to the rhythm effect, as a substitute for clapping or cheering for the distributed content.

[0029] As described above, in the present embodiment, the number of visual effects, which are visual information presented on the display screen superimposed on the content, is controlled based on the content of the tap operation and the content of the content.

[0030] FIG. 2 is a diagram showing an example of a procedure for generating visual effect information by a content distribution system according to an embodiment of the present invention. Here, generation by the control unit 12 of the user terminal 10 will be described as an example. First, when the tap frequency [F] indicated by the user operation information acquired by the user operation acquisition unit 11, that is, the number of tap operations per unit time based on the above tap cycle, and the BPM [b] of the periodic sound of the distributed content are substantially the same, and here, when the threshold value is W, when the condition of "|b - F| < W)" is satisfied (Yes in S11), the control unit 12 of the user terminal 10 generates N (N>1) rhythm effects in response to one tap operation by the user, (S12), visual effect information corresponding to this rhythm effect is generated (S16), and is output to the display control unit 13.

[0031] The generated visual effect information corresponding to the N rhythm effects is superimposed on the delivery content as described above, for example, N heart marks are displayed at once on the display screen of the delivery content on the user terminal 10, or one heart mark is displayed N times in succession at continuous timing, thereby providing the user with feedback that the rhythm of the audio of the delivery content matches the rhythm of the user's tap operation without increasing the types of visual effect information. In other words, when the user is well into the rhythm of the audio of the delivery content, a more comfortable number of marks can be presented to the user.

[0032] Furthermore, when the above tap frequency [F] indicated by the user action information acquired by the user action acquisition unit 11 is somewhat higher than the above BPM [b], in this case when the condition "F>b+V" is satisfied where the threshold value is V (Yes in S13), the control unit 12 of the user terminal 10 generates M (M>1) cheering effects in response to one tap operation by the user (S14), and visual effect information in response to these cheering effects is generated (S16) and output to the display control unit 13.

[0033] The visual effect information corresponding to the generated M cheering effects is superimposed on the distribution content as described above, for example, M heart marks are displayed at once on the display screen of the distribution content on the user terminal 10, or one heart mark is displayed M times in succession at continuous timing, etc., thereby providing the user with feedback that the user is performing a relatively fast tap operation on the display screen to cheer on a performer or player in the distribution content or to get excited about the content of the distribution content without increasing the types of visual effect information, thereby enhancing the user's emotions. In other words, the user's cheering or excitement can be strengthened.

[0034] Also, when the tap frequency [F] indicated by the user operation information acquired by the user operation acquisition unit 11 is somewhat lower than the above BPM [b], here, when the condition of "F < b - W" is satisfied with the threshold value being W (No in S13), the control unit 12 of the user terminal 10 generates one rhythm effect corresponding to one tap operation by the user (S15), visual effect information corresponding to this rhythm effect is generated (S16), and is output to the display control unit 13.

[0035] In the present embodiment, by controlling the number of visual effect information presented in response to a tap operation using the minimum operation information of a tap operation by the user and the periodic voice information of the distribution content, visual effect information that enhances the excitement of the user is dynamically presented.

[0036] The above control includes, for example, control to adjust the number of visual effect information in real time according to the change in BPM during the distribution music when the distribution content is a music live, and control related to the generation of visual effect information according to a rhythm effect or a cheering effect by determining the pattern of the relationship between the frequency of the tap operation and the rhythm of the voice of the distribution content.

[0037] In the present embodiment, when the tap operation by the user matches the rhythm of the music of the distribution content, for example, feedback is given to the user by increasing the number of visual effects. Also, in the present embodiment, when the tap operation by the user is an operation related to cheering for the distribution content, feedback is given to the user by increasing the number of visual effects.

[0038] Thereby, with a simple visual expression that does not increase the types of visual effect information, it is possible to give feedback that contributes to the excitement of the user without interrupting the concentration of the user watching the distribution content. Also, since the above feedback naturally enhances the effect that occurs with the user's operation, it is expected to make it difficult for the user to recognize the feedback operation and to enhance the excitement unconsciously.

[0039] FIG. 3 is a block diagram showing an example of a hardware configuration of a user terminal according to an embodiment of the present invention. In the example shown in Fig. 3, the user terminal 10 of the content distribution system according to the embodiment is configured by a personal computer or a mobile information terminal, for example, a smartphone, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to the hardware processor 111A via a bus 115. The content distribution device 20 shown in Fig. 1 is configured by a server computer or a personal computer, and has each unit shown in Fig. 3. The configuration of each unit will be described below by taking the user terminal 10 as an example.

[0040] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0041] An input device 200 and an output device 300 that are attached to the user terminal 10 and used by a user or the like are connected to the input / output interface 113 . The input / output interface 113 takes in operation data input by a user or the like through an input device 200 such as a keyboard, a touch panel, a touchpad, a mouse, etc., and outputs output data to an output device 300 including a display device using liquid crystal or organic EL (Electro Luminescence), etc., for display. The input device 200 and the output device 300 may be devices built into the user terminal 10, or may be input devices and output devices of other information terminals that can communicate with the user terminal 10 via the network NW.

[0042] The program memory 111B is a non-transient tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and stores programs necessary for executing various control processes, etc. according to one embodiment.

[0043] The data memory 112 is a tangible storage medium, for example, a combination of the above-mentioned non-volatile memory and a volatile memory such as RAM (Random Access Memory), and is used to store various data acquired and created in the course of various processes performed by the user terminal 10.

[0044] A user terminal 10 according to an embodiment of the present invention can be configured as a video processing device having a processing function unit implemented by software. The storage areas used as work memories by the various components of the user terminal 10 can be configured by using the data memory 112 shown in Fig. 3. However, these configured storage areas are not essential components within the user terminal 10, and may be areas provided in an external storage medium such as a Universal Serial Bus (USB) memory, or in a storage device such as a database server located in the cloud.

[0045] The above-mentioned processing function unit can be realized by reading and executing a program stored in the program memory 111B by the above-mentioned hardware processor 111A. Note that the processing function unit may be realized in various other forms, including an integrated circuit such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0046] The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disc (CD-ROM, DVD, MO, etc.), or a semiconductor memory (ROM, RAM, flash memory, etc.), and may be distributed by transmitting the program via a communication medium. The programs stored on the medium include a setting program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes the present device reads the program recorded on the recording medium, and in some cases, configures the software means by the setting program, and executes the above-mentioned processing by controlling the operation of the software means. The recording medium in this specification is not limited to a recording medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory provided inside the computer or in a device connected via a network.

[0047] The present invention is not limited to the above-mentioned embodiment, and can be modified in various ways without departing from the gist of the present invention. The embodiments may be combined as appropriate, and in that case, the combined effect can be obtained. Furthermore, the above-mentioned embodiment includes various inventions, and various inventions can be extracted by combinations selected from the multiple components disclosed. For example, if the problem can be solved and the effect can be obtained even if some components are deleted from all the components shown in the embodiment, the configuration from which the components are deleted can be extracted as an invention. [Explanation of symbols]

[0048] 100…Content distribution system 10...User terminal 11...User action acquisition unit 12...Control section 13...Display control unit 20...Content distribution device 21…Content Distribution Department 22…Direction generation section 23…Venue Production Department

Claims

1. an acquisition unit that acquires information indicating the details of an operation performed on a display screen of a content including video and audio by a user viewing the content; a generation unit that generates visual information that can be superimposed on the content and displayed on the display screen, based on a relationship between a periodic sound included in the content and a content of an operation indicated by the information acquired by the acquisition unit; A video processing device comprising:

2. The generation unit is generating visual information indicating that an operation on the display screen is in sync with the periodic sound when a predetermined condition is satisfied in which the frequency of the operation on the display screen indicated by the information acquired by the acquisition unit is equal to or close to a frequency of beats related to the periodic sound included in the content, the visual information being generated in response to one operation on the display screen in a plurality of pieces; The video processing device according to claim 1 .

3. The generation unit is generating visual information indicating that an operation on the display screen is an operation related to assistance for the content when a predetermined condition is satisfied, the visual information being a plurality of pieces of visual information corresponding to one operation on the display screen, the visual information indicating that the operation on the display screen is an operation related to assistance for the content, the plurality of pieces of visual information being corresponding to one operation on the display screen when a predetermined condition is satisfied, the plurality of pieces of visual information being corresponding to one operation on the display screen, the plurality of pieces of visual information being corresponding to one operation on the display screen when a predetermined condition is satisfied, the plurality of pieces of visual information indicating that the operation on the display screen is an operation related to assistance for the content, the plurality of pieces of visual information being corresponding to one operation on the display screen when a predetermined condition is satisfied ... The video processing device according to claim 1 .

4. The generation unit is generating one piece of the visual information for one operation on the display screen when a predetermined condition is satisfied, the frequency of the operation on the display screen being lower than a frequency of beats related to a periodic sound included in the content, the frequency being indicated by the information acquired by the acquisition unit; The video processing device according to claim 1 .

5. A method performed by a video processing device, comprising: acquiring information indicating details of an operation performed on a display screen of a content including video and audio by a user viewing the content, using an acquisition unit of the video processing device; generating, by a generating unit of the video processing device, visual information that can be superimposed on the content and displayed on the display screen, based on a relationship between a content of a periodic sound included in the content and a content of an operation indicated by the information acquired by the acquiring unit; A video processing method comprising:

6. A video processing program that causes a processor to function as each unit of the video processing device according to claim 1 .

Images