Information processing system, information processing device, and control program

Abstract

The system enhances entertainment value by displaying viewer reactions on the viewing screen. [Solution] The information processing system (for example, group 5) includes: a first receiving means (for example, a first image display operation receiving unit 72) that receives a first operation performed on the first device (for example, a mobile terminal 3) by a first user who is viewing a viewing target displayed on the first screen (for example, screen 3a) of the first device, in order to add a first element to the viewing target; a second receiving means (for example, a first image display operation receiving unit 72) that receives a second operation performed on the second device (for example, a mobile terminal 3) by a second user who is viewing the viewing target displayed on the second screen (for example, screen 3a) of the second device, in order to add a second element to the viewing target; and a first control means (for example, an image display control unit 70 of the first device) that causes a third element based on a combination of the first operation and the second operation to be displayed on the first screen.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, and a control program.

Background Art

[0002] There is known a mechanism for causing viewers of video content being distributed to conduct a group chat about the content of the video content. For example, Patent Document 1 discloses a client device that displays a fixed message regarding the content of video content on a chat screen in which avatars corresponding to users participating in the group chat are arranged around the video content, according to a button selected by the user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The client device described in Patent Document 1 displays a fixed message selected by the user in a balloon near the avatar representing the user. In such a display method of the fixed message, as the number of users viewing the video content increases, the number of avatars also increases, but the entertainment value of the users does not improve.

[0005] The problem to be solved by the present invention is to improve the entertainment value in a mechanism for displaying the reaction of viewers on the display screen of the viewing target.

Means for Solving the Problems

[0006] To solve the above problems, the present invention displays a third element on the first screen of the first device, based on a combination of a first operation performed on the first device by a first user who is viewing a viewing target displayed on the first screen of the first device in order to add a first element to the viewing target, and a second operation performed on the second device by a second user who is viewing the viewing target displayed on the second screen of the second device in order to add a second element to the viewing target. [Effects of the Invention]

[0007] According to the present invention, the entertainment value can be improved in a mechanism that displays the viewer's reaction on the display screen of the content being viewed. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating the configuration of a video streaming system. (Example) [Figure 2] Block diagram showing the electrical configuration of the server. (Example) [Figure 3] Block diagram showing the electrical configuration of a mobile device. (Example) [Figure 4] This diagram shows the display configuration of a mobile device screen. (Example) [Figure 5] This figure shows a screen displaying onomatopoeic images. (Example) [Figure 6] This is a diagram showing the screen displaying the second image. (Example) [Figure 7] This is a timing chart showing when a combined image is displayed. (Example) [Figure 8] This is a timing chart showing the case where the second image is replaced. (Example) [Figure 9] This is a functional block diagram related to video streaming on the server. (Example) [Figure 10] This is a functional block diagram related to onomatopoeia display on mobile devices. (Example) [Figure 11] This is a flowchart illustrating the flow of onomatopoeia display processing performed on a mobile device. (Example) [Figure 12] This is a flowchart showing the flow of onomatopoeia display processing executed on the server. (Example) [Figure 13] This is a flowchart showing the flow of onomatopoeia display processing executed on the server. (Example) [Figure 14] This is a timing chart showing when combined images are displayed. (Variation 2) [Modes for carrying out the invention]

[0009] [1. Embodiments] [1-1. Main Features] This embodiment relates to a mechanism for displaying viewer reactions on the display screen of the object being viewed. In this embodiment, in order to improve entertainment value, a third element is displayed on the first screen based on a combination of a first operation performed on the first device by a first user who is viewing the object being viewed on the first screen of the first device in order to add a first element to the object being viewed, and a second operation performed on the second device by a second user who is viewing the object being viewed on the second screen of the second device in order to add a second element to the object being viewed.

[0010] The viewing target refers to video content that is perceived at least visually by the user. The video content may also include audio content perceived aurally by the user. The video content is distributed, for example, from a distribution device to an information processing device. Suitable distribution methods for the video content include, for example, streaming distribution and progressive download distribution. The video content is, for example, a video. The viewing target is displayed in a basic area, which is part of the screen of the information processing device.

[0011] A user (viewer) viewing the target content can add a predetermined element (additional element) to the target content through a predetermined operation (additional operation). The predetermined operation is, for example, an input element. It is a pointing operation on an (operation element). The input element is, for example, a button displayed within a screen. The pointing operation is, for example, a combination of a touch operation on a touch panel and a positioning operation and a click operation of a pointer displayed within the screen. A predetermined element is, for example, an image representing a user's reaction to a viewing target. The user's reaction is, for example, the user's emotion. An image representing the user's emotion towards a viewing target may be called an emotion image. The emotion image is, for example, an image in which onomatopoeia (also referred to as sound words, sound imitation words, mimetic words, etc.) indicating the user's emotion is designed. An image in which onomatopoeia is designed may be called an onomatopoeia image. Adding an element to a viewing target means associating the display timing of the element with the progress of the viewing target. When the display timing is associated with the progress, the display timing is specified at a specific position on the timeline. The element added to the viewing target can be displayed on the screen at the associated display timing during the progress of the viewing target (i.e., during playback display).

[0012] The first user is the user (the person himself / herself) who uses the first device (information processing device). A predetermined operation (the first operation) by the first user is, for example, received by the first device. The second user is another user (another person) who uses the second device (another information processing device). A predetermined operation (the second operation) by the second user is, for example, received by the second device. A predetermined element (the first element) based on the first operation is displayed in a first range that at least partially overlaps the basic area. A predetermined element (the second element) based on the second operation is displayed in a second range that does not overlap the basic area. The first range may have a part (the first part) that does not overlap the basic area in addition to the part (the second part) that overlaps the basic area. The first range may further have a part (the third part) that overlaps the second range.

[0013] Based on the timings of the first operation and the second operation, a predetermined element (third element) based on the combination of the first operation and the second operation is displayed. Specifically, when the timings match or are close, the predetermined element is displayed. That the timings are close means that they are close in real time or the positions on the timeline of the viewing target are close. That they are close in real time means that when the first user and the second user are viewing the viewing target almost simultaneously, the timings of the first operation and the second operation in real time are close. That the positions on the timeline of the viewing target are close means that when the first user and the second user are viewing the viewing target almost simultaneously or at different times, the timings on the timeline of the first operation and the second operation are close. Being close means, for example, that the time difference is within a predetermined threshold. The predetermined element based on the combination of the first operation and the second operation is preferably displayed in the first range.

[0014] [1-2. Supplementary Note] The present invention at least includes the following forms from [A] to [D] below.

[0015] [A] First reception means for receiving a first operation performed on a first device by a first user who is viewing a viewing target displayed on a first screen of the first device to add a first element to the viewing target; second reception means for receiving a second operation performed on a second device by a second user who is viewing the viewing target displayed on a second screen of the second device to add a second element to the viewing target; and first control means for causing the first screen to display a third element based on a combination of the first operation and the second operation.

[0016] [B] Reception means for receiving a first operation performed by a first user who is viewing a viewing target to add a first element to the viewing target; and display control means for displaying a third element based on a combination of the first operation and a second operation performed on another information processing device by a second user who is viewing the viewing target on the other information processing device.

[0017] [C] A control program that enables the computer of an information processing device to implement a reception function that receives a first operation performed by a first user who is viewing the viewing target in order to add a first element to the viewing target, and a display control function that displays a third element based on a combination of the first operation and a second operation performed on the other information processing device by a second user who is viewing the viewing target on the other information processing device in order to add a second element to the viewing target. The control program may be recorded non-temporarily on a recording medium.

[0018] [D] A control method for an information processing device, comprising: a reception step of receiving a first operation performed by a first user who is viewing the viewing object to add a first element to the viewing object; and a display control step of displaying a third element based on a combination of the first operation and a second operation performed on the other information processing device by a second user who is viewing the viewing object on the other information processing device to add a second element to the viewing object.

[0019] The following technical limitations may be added to the "information processing system" in [A] above. Similar technical limitations may also be added to the "information processing device" in [B] above, the "control program" in [C] above, and the "control method for the information processing device" in [D] above, respectively.

[0020] (1) The first control means causes the third element to be displayed on the first screen based on a combination of the first operation and the second operation performed during a predetermined period based on the time the first operation was received on the timeline of the viewing target. The predetermined period based on the time of reception is, for example, a predetermined period with the time of reception as the start point, end point, or midpoint, or a predetermined period including the time of reception. The predetermined period may be a fixed value or a variable value. The variable value may vary, for example, depending on the number of viewing users or the number of predetermined elements added. As a result, the third element is displayed when the first operation and the second operation are close together on the timeline.

[0021] (2) The system further comprises a first providing means that provides operation information for the first operation to a first instruction means that outputs a first instruction to display the third element, and a second providing means that provides operation information for the second operation to the first instruction means, wherein the first control means displays the third element based on the first instruction on the first screen when it receives the first instruction output from the first instruction means within the predetermined period based on a combination of the first operation and the second operation corresponding to the provided operation information. The first instruction means is, for example, a server device that can be connected to the first and second devices. The first instruction can be output at any time while viewing the content to be viewed. This allows the third element based on a combination of the first and second operations to be displayed when the first user and the second user are viewing at the same time.

[0022] (3) The system further comprises a third receiving means that receives a second instruction output from a second instruction means that outputs a second instruction to display the second element based on the second operation at a timing corresponding to the second operation, and if the first control means receives the second instruction to display the second element at a timing included in the predetermined period before the first operation is performed, it displays the third element based on a combination of the first operation and the second operation related to the second instruction on the first screen. The second instruction means is, for example, a server device that can be connected to the first and second devices. The second instruction may be output all at once when viewing of the viewing target begins. This allows the third element based on a combination of the first and second operations to be displayed when a second user is viewing before the first user.

[0023] (4) The system further comprises at least one of the following: a second control means for displaying the first element on the first screen based on the first operation, except when the third element is displayed in response to the first operation; and a third control means for displaying the second element on the first screen based on the second instruction, except when the third element is displayed in response to the second instruction. This ensures that when the third element is displayed, at least one of the first and second elements is not displayed. [Examples]

[0024] [2. Examples] The following describes embodiments of the present invention with reference to the drawings. This embodiment relates to a mechanism for displaying images representing the user's (viewer's) emotions (images representing reactions) on the display screen of a video (viewing target). In this embodiment, the shared information processing device is described as a server, and the information processing devices (first device, second device) are described as mobile terminals, so-called smartphones. Note that the mobile terminal is not limited to a smartphone, but may also be a tablet terminal or a laptop computer. The information processing device may also be a desktop computer. The programs according to this embodiment are stored in the shared information processing device or the information processing device, but are not limited to these, and may also be stored in storage media such as optical discs such as CD-ROMs (Compact Disk Read Only Memory) or portable semiconductor memory such as USB (Universal Serial Bus) flash memory.

[0025] [2-1. Features] In this embodiment, operation information of an addition operation (first operation) performed on the first mobile terminal (first device) to add a first image (first element) to a video (viewing target) displayed on the screen (first screen) of the first mobile terminal (first device), and operation information of an addition operation (second operation) performed on the second mobile terminal to add a first image (second element) to a video displayed on the screen (second screen) of the second mobile terminal (second device) are provided from each mobile terminal to the server (first instruction means, shared information processing device). In order to improve entertainment value, the server displays a combination image (third element) based on the combination of the first and second operations corresponding to the operation information received on the first screen and the second screen, respectively.

[0026] Furthermore, in this embodiment, in order to suppress sudden increases in load on mobile devices, two or more (a predetermined number) second images are identified as an image group (group of elements) from among the second images (shared elements) associated with the video, based on the proximity of display timing (timing) and the commonality (additional attributes) between the subject and object, while the timing at which the video is displayed on the mobile device of the user watching the video is during playback. A replacement process is then performed in which the second images included in the image group are invalidated, and one (a number less than a predetermined number) new second images (alternative elements) that inherit the additional attributes are added to the video. The subject refers to the user who added the second image to the video, and commonality among subjects means that the subjects belong to the same user group. Each user belongs to at least one of the user groups. The object refers to the second image added to the video. This refers to the types of elements, and commonality among objects means that the objects belong to the same group of elements. Each element belongs to at least one of the element groups.

[0027] [2-2. Video Streaming System] The server in the embodiment is included in the video distribution system and distributes video to mobile devices. The mobile device in the embodiment is included in the video distribution system and receives video from the server.

[0028] [2-2-1. Overall configuration of the video distribution system] Figure 1 is a schematic diagram of a video distribution system 1 according to an embodiment. The video distribution system 1 consists of a communication line 2, multiple mobile terminals 3, and a server 4.

[0029] Communication line 2 forms a computer network and is, for example, a wide-area communication line provided by an electric power company.

[0030] The mobile terminal 3 is, for example, an information processing terminal such as a smartphone, tablet, or laptop computer. The mobile terminal 3 is equipped with a touch panel display 3a for displaying images, a speaker 3b for outputting sound, a microphone 3c for receiving sound input, and a camera 3d for capturing images of subjects. To display videos distributed from the server 4, the mobile terminal 3 is fitted with, for example, dedicated application software (hereinafter referred to as the "video viewing app").

[0031] Server 4 is a shared information processing device that distributes content to multiple mobile terminals 3 via communication line 2. The content in this embodiment is, as an example, video. Videos are not particularly limited; they may include animations, dramas, or other content with a storyline, or music videos, live music videos, or various sports videos. The video in this embodiment is, as an example, animation. Furthermore, the content distributed from the server is not limited to videos; it may also be other services, such as a series of still images, as long as it is viewable by the user.

[0032] Furthermore, "viewing" as used herein means that a user perceives the content through at least one of their senses, either visually or aurally. In addition to the above, the content may also include, for example, a video without sound, or sound without images.

[0033] Multiple mobile devices 3 from which the video is streamed are registered to server 4 as group 5, for example. (Hereinafter referred to as "group registration") is performed. Server 4 simultaneously distributes the video to the group-registered mobile devices 3. That is, the multiple group-registered mobile devices 3 display the video distributed from Server 4 on their touch panel displays 3a at the same time, in other words, in sync. As a result, each user of the group-registered mobile devices 3 will watch the video at the same time.

[0034] The video may be distributed at a predetermined date and time, or at any time chosen by each of the 5 groups. If the video is distributed at any time chosen, for example, the host mobile device 3 will request the server 4 to distribute the video, and the server 4, upon receiving the distribution request, will distribute the video to the mobile devices 3 that make up the 5 group.

[0035] In Figure 1, Group 5 is shown as consisting of four mobile devices 3, but this is just an example; the number of mobile devices 3 in Group 5 is not limited as long as it is composed of multiple devices 3. Furthermore, the types of mobile devices 3 that make up Group 5 are not limited; smartphones, tablets, laptops, etc., may be mixed together.

[0036] [2-2-2. Server Configuration] Figure 2 is a block diagram showing the electrical configuration of server 4 according to the embodiment.

[0037] The server 4 according to this embodiment includes a CPU (Central Processing Unit) 20, which is the main control unit that manages the operation of the entire server 4; a ROM (Read Only Memory) 22 in which various programs and data are pre-stored; a RAM (Random Access Memory) 24 used as a work area when the CPU 20 executes various programs; and an HDD (Hard Disk Drive) 26 as a storage means for storing various programs and data.

[0038] HDD26 stores information related to the registered group 5, videos (video data) to be distributed to the mobile terminal 3, and various other information. Note that the storage means is not limited to HDD26; other storage media such as semiconductor memory like flash memory may also be used.

[0039] Furthermore, the server 4 includes an input unit 28 consisting of a keyboard and mouse, etc., which accepts input for various operations; a monitor 30 such as a liquid crystal display device that displays various images; and an external interface 32 that is connected to other information processing devices such as a mobile terminal 3 via a communication line 2 and transmits and receives various data with other information processing devices.

[0040] The CPU 20, ROM 22, RAM 24, HDD 26, operation input unit 28, monitor 30, and external interface 32 are electrically connected to each other via the system bus 34. Therefore, the CPU 20 can access the ROM 22, RAM 24, and HDD 26, understand the operation status of the operation input unit 28, display images on the monitor 30, and send and receive various types of data with other information processing devices via the external interface 32.

[0041] [2-2-3. Electrical Configuration of Mobile Devices] Figure 3 is a functional block diagram showing the electrical configuration of the mobile terminal 3. In addition to the configuration shown in Figure 1, the mobile terminal 3 includes a main control unit 40, a main memory unit 42, an auxiliary memory unit 44, a communication unit 46, and operation buttons 48.

[0042] The main control unit 40 is, for example, a CPU, microprocessor, or DSP (Digital Signal Processor), and controls the overall operation of the mobile terminal 3.

[0043] The main memory unit 42 is composed of, for example, RAM or DRAM (Dynamic Random Access Memory), and is used as a work area when the main control unit 40 executes processing based on various programs.

[0044] The auxiliary storage unit 44 is a non-volatile memory such as flash memory, and stores various data such as images and programs used for processing by the main control unit 40. The programs stored in the auxiliary storage unit 44 include, for example, the OS (Operating System) for realizing the basic functions of the mobile terminal 3, drivers for controlling various hardware, programs for realizing email, web browsing, and various other functions. In addition, the auxiliary storage unit 44 has pre-stored application software such as a video viewing application for playing videos distributed from the server 4 and displaying them on the touch panel display 3a.

[0045] The communication unit 46 is, for example, a NIC (Network Interface Controller) and has the function of connecting to the communication line 2. Alternatively, the communication unit 46 may have functions to connect to a wireless LAN (Local Area Network), a wireless WAN (Wide Area Network), short-range wireless communication such as Bluetooth®, and infrared communication, either in place of or together with the NIC.

[0046] The operation buttons 48 are located on the side of the mobile terminal 3 and include a power button for starting or stopping the mobile terminal 3, and a volume control button for the sound output by the speaker 3b.

[0047] The main control unit 40, main memory unit 42, auxiliary memory unit 44, communication unit 46, touch panel display 3a, speaker 3b, microphone 3c, camera 3d, and operation buttons 48 are electrically connected to each other via a system bus 49. Therefore, the main control unit 40 can access the main memory unit 42 and auxiliary memory unit 44, display images on the touch panel display 3a, understand the user's operation status of the touch panel display 3a and operation buttons 48, input sound to the microphone 3c, output sound from the speaker 3b, control the camera 3d, and access various communication networks and other information processing devices via the communication unit 46.

[0048] [2-3. Onomatopoeia Display] The mobile terminal 3 according to this embodiment can display videos distributed from the server 4 on the touch panel display 3a, while also displaying images (onomatopoeia images, which will be described in detail later) that respond to user operations. In the following description, the function of displaying onomatopoeia images on the touch panel display 3a is referred to as the onomatopoeia display function, and the touch panel display 3a is also referred to as the screen 3a.

[0049] The onomatopoeia display function is, for example, one of the functions of the video viewing application according to the embodiment. However, for example, it is not possible to display onomatopoeia for all videos displayed on screen 3a using the video viewing application; the videos for which onomatopoeia display is possible are predetermined.

[0050] [2-3-1. Screen Layout] Figures 4 to 6 show the display mode of screen 3a of a mobile terminal 3 running the video viewing application according to the embodiment. Screen 3a is configured with a streaming video display area 50A, a first image display area 50B, and a second image display area 50C.

[0051] The video streaming display area 50A is the area for displaying the video streamed from the server 4. The first image display area 50B is the area for displaying the first image 51 (see Figure 5) in response to user operations. The second image display area 50C is the area for displaying the second image 52 (see Figure 6) based on the first image 51 displayed on another mobile terminal 3 by another user's operation. As will be described in detail later, the second image 52 is a reduced image of the first image 51 displayed in the first image display area 50B of the other mobile terminal 3. Furthermore, the first image display area 50B and the second image display area 50C in this embodiment are, as an example, provided at positions opposite each other with the video streaming display area 50A in between.

[0052] Here, the first image 51 is an image that includes an emotional image showing the emotions of a user watching a video. The emotional image in this embodiment is, as an example, an image that depicts onomatopoeia (also called sound-imitation or mimetic words) that express the user's emotions, in other words, an onomatopoeia image. Examples of onomatopoeia images include "Awesome," "Incredible," and "Bravo." Note that stylization means, for example, that the characters representing onomatopoeia are depicted in a decorative way.

[0053] Furthermore, the first image 51 may include various other images besides onomatopoeia images, such as character images showing characters appearing in the video, dialogue images showing characters' lines, text entered by the user, and so on.

[0054] Furthermore, the first image 51 may vary depending on the scene in the video. For example, depending on whether it is a cheerful scene, a sad scene, or a scene of characters confronting each other, onomatopoeic images or character images that the user is likely to choose based on their empathy with the scene may be displayed.

[0055] Furthermore, the displayable first image 51 may differ depending on the user. For example, the types of displayable first image 51 may increase by obtaining a specific item through a service such as a game related to the distributed video. Also, for example, the types of displayable first image 51 may increase on the condition that a predetermined consideration (e.g., monetary value such as currency, virtual currency, or points) is paid.

[0056] In the first image display area 50B according to the embodiment, a number of display buttons 53A to 53D are displayed, which are operation units operated by the user and each corresponds to a different first image 51. As a result, the user can display the first image 51 in the first image display area 50B by operating the display buttons 53A to 53D while watching a video. Operation of the display buttons 53A to 53D is, for example, by touching (pressing) the display buttons 53A to 53D with an object. In the embodiment, the object is, for example, the user's finger. In the following description, if the display buttons 53A to 53D are not described separately, the alphabetical notation at the end will be omitted.

[0057] In the examples shown in Figures 4 to 6, four display buttons 53A to 53D are displayed in a horizontal row at the bottom of the first image display area 50B. Display button 53A corresponds to an onomatopoeic image depicting "Sugee" (amazing), display button 53B corresponds to an onomatopoeic image depicting "Yabai" (incredible), and display button 53C corresponds to an onomatopoeic image depicting "Bravo". Furthermore, display button 53D corresponds to a character image. The display buttons 53 are designed to allow users to intuitively recognize the corresponding onomatopoeic images and character images.

[0058] In this way, by displaying multiple display buttons 53 in the first image display area 50B, the user can easily display the first image 51 on screen 3a that best matches their reaction (emotion) to the video. Furthermore, by including an onomatopoeic image that represents the user's emotion in the first image 51, the user can display their own emotions after watching the video on screen 3a.

[0059] In the examples shown in Figures 4 to 6, four types of display buttons 53 are displayed in the first image display area 50B, but this is just an example, and the number and types of display buttons 53 are not limited to these. Furthermore, the display buttons 53 may be switched and displayed as appropriate to enable the display of different first images 51 depending on the video scene, etc.

[0060] Furthermore, the arrangement of the display buttons 53 is just one example; for example, they may be displayed together on the right or left side of the first image display area 50B, or they may be displayed in two horizontal columns at the bottom of the first image display area 50B. In addition, for example, by the user swiping left or right from the bottom of the screen 3a, new display buttons 53 for displaying different types of first images 51 may be displayed.

[0061] [2-3-2. Displaying the first image] Referring to Figure 5, the display mode of the first image 51 will be explained in detail. In the example in Figure 5, the user operates the display button 53C, and an onomatopoeic image of "Bravo" is displayed on screen 3a.

[0062] The onomatopoeia display function according to this embodiment accepts user input to display an onomatopoeia image (first image 51) on screen 3a while a video is displayed on screen 3a. That is, while watching the video, the user operates the display button 53 to display an onomatopoeia image on screen 3a that corresponds to the emotion they felt while watching the video.

[0063] The onomatopoeia image according to the embodiment is displayed on screen 3a showing the video, changing according to how the user operates the display button 53. As an example, the onomatopoeia image according to the embodiment is displayed so as to extend from the first image display area 50B to the streamed video display area 50A, depending on how the user operates the display button 53.

[0064] In this way, users can change the onomatopoeic image displayed on screen 3a by operating the mobile device 3 while watching the video. Therefore, users can easily display the emotions they experience while watching the video on the screen.

[0065] Furthermore, in the example shown in Figure 5, the onomatopoeia image is displayed superimposed on the streaming video display area 50A, extending beyond the first image display area 50B. Thus, the onomatopoeia display function according to this embodiment superimposes the onomatopoeia image onto the streaming video display area 50A, extending beyond the first image display area 50B, according to the user's operation. This makes it easier for the user to empathize with the video they are watching, thereby enhancing the entertainment value of watching the video.

[0066] In Figure 5, the onomatopoeia image is superimposed not only on the streaming video display area 50A but also on the second image display area 50C. However, this is just one example, and the onomatopoeia image may not be superimposed on the second image display area 50C.

[0067] Next, we will explain in detail the onomatopoeia images that change depending on how the display button 53 is operated.

[0068] Image 51 of the embodiment shows, as an example, a display that changes depending on the duration of the user's operation of the display button 53. The duration of operation of the display button 53 is, for example, the time the display button 53 is continuously pressed (hereinafter referred to as "long press time").

[0069] In the embodiment, the onomatopoeia display function starts displaying the onomatopoeia image from the vicinity of the display button 53 when the user presses the display button 53. Displaying the onomatopoeia image from the vicinity of the display button 53 means, for example, that the onomatopoeia image is displayed so that its lower edge approximately coincides with the position of the display button 53.

[0070] Then, the onomatopoeia image expands from the first image display area 50B toward the streamed video display area 50A (hereinafter referred to as "expanded display") depending on the length of time the display button 53 is pressed.

[0071] The stretched display is, for example, a display in which an onomatopoeic image is made elongated or enlarged. The stretched display may also involve the onomatopoeic image being stretched while changing its form, such as the number of characters or the type of characters. For example, if the onomatopoeic image is "Bravo," it may be stretched so that "a" increases between "bra" and "bo" depending on the length of the press, or it may be stretched so that "o" increases between "bravo" and "-" depending on the length of the press.

[0072] In other words, if the user feels a stronger emotion towards the video they are watching, they continue to press the display button 53 to display an onomatopoeic image corresponding to that emotion. As a result, the onomatopoeic image is extended from the first image display area 50B toward the streaming video display area 50A, and is displayed on screen 3a in a more prominent manner.

[0073] In this way, the onomatopoeia images included in the first image 51 change according to the user's actions, allowing the user to display their own reactions (emotions) to the content on screen 3a in a more entertaining manner.

[0074] Furthermore, the first image 51 is displayed so as to extend from the first image display area 50B toward the streamed video display area 50A, depending on the user's operation, allowing the user to display their reaction (emotions) to the content on screen 3a in a more entertaining way.

[0075] When the user finishes operating the display button 53 (by pressing and holding it), the display of the onomatopoeia image on screen 3a will stop. The display of the onomatopoeia image may stop in any way, for example, by the displayed onomatopoeia image disappearing from screen 3a instantaneously, or by the enlarged onomatopoeia image shrinking towards the display button 53 and disappearing from screen 3a.

[0076] Furthermore, in the example shown in Figure 5, the area around the display button 53C being operated by the user is highlighted in order to display the onomatopoeia image. This allows the user to easily recognize the display button 53C they are operating and enhances the entertainment value of the onomatopoeia display.

[0077] Furthermore, while character images and the like, which are first images 51 other than the onomatopoeia images in the embodiment, are displayed on screen 3a without changing, the invention is not limited to this, and first images 51 other than the onomatopoeia images may also be displayed on screen 3a showing the video while changing according to the user's operation of the display button 53.

[0078] [2-3-3. Displaying the second image] Referring to Figure 6, the display mode of the second image 52 will be explained in detail. As mentioned above, the second image 52 is a reduced image of the first image 51 displayed on another mobile terminal 3 by another user's operation.

[0079] The onomatopoeia display function according to this embodiment displays the second image 52 in the second image display area 50C with virtually no time delay from the moment another user displays the first image 51 on another mobile device 3. This allows the user to know the reactions of other users who are watching the same video while watching the video.

[0080] Furthermore, if the second image 52 is superimposed on the streaming video display area 50A, as with the first image 51, users may find the second image 52 distracting and annoying while watching the video. Therefore, the second image 52 is not superimposed on the streaming video display area 50A beyond the second image display area 50C, regardless of how other users are operating. This allows users to see other users' reactions to the video they are watching without feeling annoyed.

[0081] As mentioned above, the first image 51 can also be superimposed onto the streaming video display area 50A, but this is done at the user's discretion, so the user will not feel any inconvenience from the superimposition of the first image 51 onto the streaming video display area 50A.

[0082] Furthermore, since screen 3a is configured such that the first image display area 50B and the second image display area 50C are different areas, the first image 51 displayed by the user and the second image 52 displayed by another user will not be displayed together. As a result, the user can easily distinguish and recognize the first image 51 that they displayed and the second image 52 displayed by another user.

[0083] Furthermore, the other users in this embodiment are, for example, other users registered in the same group as the user. That is, if the user is watching the video without being registered in a group, the second image 52 will not be displayed in the second image display area 50C. However, this is not limited to this, and the second image 52 may be displayed in the second image display area 50C due to the actions of all other users watching the video, regardless of whether they are registered in a group or not.

[0084] Furthermore, in the second image display area 50C according to the embodiment, multiple second images 52 are displayed based on the operations of multiple other users. In the second image display area 50C according to the embodiment, for example, the position where the second image 52 is displayed is not predetermined for each other user. That is, multiple second images 52 from the operations of multiple other users are displayed in a random mix of positions in the second image display area 50C. Moreover, the second image 52 may be displayed while moving up, down, left, and right within the second image display area 50C. However, this is not limited to this, and the position where the second image 52 is displayed may be predetermined for each other user within the second image display area 50C.

[0085] Furthermore, the second image 52 is displayed adjacent to an identification image (hereinafter referred to as "user icon") 54 that indicates another user who has displayed the first image 51, which is designated as the second image 52, on their mobile device 3. This allows the user to recognize which user created the second image 52 displayed in the second image display area 50C. In the example in Figure 6, the user icon 54 is a single letter of the alphabet displayed in the center of a circle, but this is just one example, and the user icon 54 may be an image set for each user, or it may be the user's initials, and the design of the user icon 54 is not limited.

[0086] Furthermore, although the second image 52 shown in Figure 6 displays the onomatopoeia image horizontally, this is set to correspond to the size (aspect ratio) of the second image display area 50C, and the second image 52 may also display the onomatopoeia image vertically.

[0087] Furthermore, although the second image 52 is not superimposed on the streaming video display area 50A, it may be displayed within the second image display area 50C while changing (e.g., stretching) in response to the operation patterns of other users. If the second image 52 is not superimposed on the streaming video display area 50A, it may be superimposed from the second image display area 50C onto the first image display area 50B.

[0088] [2-3-4. Display of combined images] The mobile terminal 3 according to this embodiment has a combination image display function that displays a combination image on the screen 3a corresponding to a combination of an operation by the user to display the first image 51 and an operation by another user to display the first image 51.

[0089] This allows mobile device 3 to display user and other users' reactions to videos in a more entertaining way. Furthermore, images that are not displayed by a single user's actions can be displayed through a combination of actions by other users, enabling mobile device 3 to encourage more users to view the content.

[0090] Furthermore, the combined image is displayed superimposed on the video shown on screen 3a. In other words, the combined image is displayed superimposed not only on the first image display area 50B, but also on the streamed video display area 50A and the second image display area 50C. This allows users to easily recognize that a combined image has been displayed, and enhances the entertainment value of watching the video.

[0091] Furthermore, if a combined image is displayed on screen 3a, it will be displayed in a way that allows the user to recognize that it is a combined image. For example, the user ID of another user may be displayed on screen 3a along with the combined image so that it can be recognized that another user performed the display operation at the same time.

[0092] Figure 7 is a timing chart showing the case where the combined image is displayed on screen 3a.

[0093] The combined image display function according to this embodiment displays a combined image on screen 3a when the timing of a user's operation and another user's operation falls within a predetermined time period (hereinafter referred to as the "combination period"). In other words, if multiple users perform the operation to display the first image 51 within the combination period, it is understood that this is a user reaction resulting from viewing the same scene. Therefore, since the combined image display function displays a combined image based on the reactions of multiple users in response to scenes in the video, the entertainment value for users from watching the video is further enhanced.

[0094] The start of the combination period is the moment when a user initiates the display operation of the first image 51. That is, as shown in Figure 7(A), if user B performs a display operation within the combination period which begins at the moment user A initiates the display operation, the combined image will be displayed on the screen 3a of both users A and B's mobile devices 3.

[0095] The display period for the combined image is, for example, until the user finishes displaying it. In the example in Figure 7(A), the combined image is displayed on the screen 3a of both users A and B's mobile devices 3 from the time user B starts displaying it until it finishes.

[0096] In the example in Figure 7(B), although it is within the combination period, the display operation by user C is performed after the display operation by user A, which initiated the combination, has finished. In the example in Figure 7(B), the combination image is displayed on the screen 3a of the mobile terminals 3 of users A, B, and C from the time user B's display operation starts until user C's display operation finishes.

[0097] Note that the display configuration of the combined images shown in Figure 7 is just one example and is not limited to this. For example, there is no concept of a combination period, and the combined image may be displayed on screen 3a at the timing when a user's display operation and another user's display operation overlap. Alternatively, a display time may be predetermined for each combined image, and the display of the combined image may end when that display time has elapsed.

[0098] Furthermore, the combined image according to the embodiment is displayed on screen 3a in place of the first image 51. In the example shown in Figure 7(A), first image 51, corresponding to user A's display operation, is initially displayed on screen 3a of user A's mobile terminal 3. However, as user B begins a display operation, the display of the first image 51 ends, and the combined image is displayed on screen 3a of user A's mobile terminal 3. On the other hand, the first image 51 that user B intended to display is not displayed on screen 3a of user B's mobile terminal 3; instead, the combined image is displayed. In this way, because the combined image is displayed on screen 3a in place of the first image 51, a different image from the first image 51 that the user intended to display is displayed on screen 3a, thus enhancing the entertainment value for users watching the video.

[0099] Furthermore, the combined image according to the embodiment is displayed on screen 3a when the first image 51 selected by the user and the first image 51 selected by another user are pre-set images. As a result, the entertainment value of watching the video is enhanced because the user may see a combined image that they themselves did not expect, depending on the image they selected.

[0100] For example, a combined image is displayed on screen 3a when the first image 51 selected by one user is the same as the first image 51 selected by another user. In other words, when users watching a video select the same first image 51, there is a high probability that they are experiencing similar emotions. Therefore, the display of a combined image allows users to recognize that they have empathized with other users through watching the video. More specifically, for example, if the first image 51 selected by one user is the same as the first image 51 selected by another user, an enlarged version of that first image 51 is displayed on screen 3a as a combined image. However, this is not the only example; special images that are not displayed by operating the display button 53 may also be displayed as combined images.

[0101] Furthermore, even if the first image 51 selected by the user is different from the first image 51 selected by another user, if the combination is a pre-set combination, the combined image may be displayed. In this case, for example, a special image that is not displayed by operating the display button 53 as a combined image will be displayed.

[0102] In the combined image display function according to the embodiment, the second image based on the combined image displayed on another user's screen 3a will not be displayed in the second image display area 50C of the mobile terminal 3 of a user whose combined image is not displayed. However, the invention is not limited to this, and the second image based on the combined image displayed on another user's screen 3a may be displayed in the second image display area 50C of the mobile terminal 3 of a user whose combined image is not displayed.

[0103] [2-4. Replacement Process] The server 4 in this embodiment has a replacement processing function that identifies two second images 52 whose display timings are close together from among the second images as an image group, and repeats a unit processing that replaces the image group with one new second image 52 until no image groups can be identified. As a result, the number of second images 52 displayed on the mobile terminal 3 during video playback is reduced, and sudden increases in load on the mobile terminal 3 can be suppressed.

[0104] Figure 8 is a timing chart showing the case where the second image is replaced.

[0105] As a premise, the video timeline is divided into multiple sections. The length of each section is, for example, the same as the length of the combined period. The replacement processing function according to the embodiment identifies two second images 52 that have commonalities between the subject and object from among multiple second images 52 that are each included in a section with the same start time of the display period, and performs a unit processing that invalidates the two second images 52 included in the image group and associates one new second image 52 with the video timeline, repeating this until no more image groups can be identified.

[0106] The second image 52 has attributes (second image display information) attached to it that can identify the subject (the user ID of the user who added the second image to the video and the user group to which that user belongs), the object (the type of the second image and the element group to which that type belongs), the display period, the number of times it has been composited, its status, etc. The type is determined by the object and the number of times it has been composited. The pattern of the second image 52 is determined by the type. Preferably, the appearance of the pattern reflects the number of times it has been composited, such as by displaying a number indicating the number of times it has been composited. As shown in Figure 8(A), the start of the display period of the second image A and the start of the display period of the second image B may both fall within the nth interval (n is an appropriate natural number) on the timeline. Furthermore, the type of the second image A and the object of the second image B are assumed to be the same. In this case, the replacement processing function identifies the second image A and the second image B as an image group, disables the status of the second image A and the second image B respectively, and associates a new second image C with the video's timeline.

[0107] The substituted second image 52 (second image C) inherits the additional attributes of the original second image 52 (second image A and second image B). The substituted second image 52, for example, inherits the subject and object as they are. The number of times the substituted second image 52 is synthesized is, for example, the sum of the number of times each of the original second image 52 is synthesized. The display period of the substituted second image 52 is, for example, the period from the earlier start date to the later end date. As shown in Figure 8(A), the display period of second image C corresponds to the period from the start date of second image A to the end date of second image B.

[0108] The original second image 52 is invalidated and its association with the video is broken. On the other hand, the replaced second image 52 can form an image group with other second images 52 and be replaced by a new second image 52. As shown in Figure 8(B), the start of the display period for second image C and the start of the display period for second image D are both included in the nth interval (n is an appropriate natural number) on the timeline. The user who added second image C and the user who added second image D belong to the same user group, and the type of second image C and the type of second image D are the same. In this case, the replacement function identifies second image C and second image D as an image group, invalidates the status of second image C and second image D respectively, and associates a new second image E with the video's timeline. The display period for second image E is the period from the start of second image C to the end of second image D.

[0109] [2-5. Functional Blocks] Next, the combined image display and replacement processing according to the embodiment will be explained using a functional block diagram. Note that the combined image display function in the mobile terminal 3 is included in the onomatopoeia display function, and the combined image display and replacement processing functions in the server 4 are each included in the video distribution function.

[0110] [2-5-1. Functional block related to the server's video streaming function] Figure 9 is a functional block diagram relating to the video distribution function of Server 4 according to the embodiment.

[0111] The CPU 20 of server 4 includes a group registration unit 60, a video distribution unit 61, a first image display information reception control unit 62, a second image display information generation unit 63, a second image display information transmission control unit 64, a combined image information generation unit 65, a combined image information transmission control unit 66, and a replacement processing unit 67. Each function of the CPU 20 is realized by a program stored in the HDD 26.

[0112] The group registration unit 60 accepts registration of a group 5 consisting of multiple mobile terminals 3. For example, group 5 registration is performed by registering a user ID.

[0113] The video distribution unit 61 distributes video to the mobile terminal 3 in response to a request from the mobile terminal 3.

[0114] The first image display information receiving control unit 62 receives the first image display information received from the mobile terminal 3 via the external interface 32, associates the first image display information with the corresponding video, and stores it in the HDD 26.

[0115] The first image display information is information about the first image 51 displayed on the screen 3a of the mobile terminal 3, and is transmitted from the mobile terminal 3 to the server 4 at the time the first image 51 is displayed on the mobile terminal 3. The first image display information includes, for example, the type of first image 51 displayed, the user ID indicating the user who displayed the first image 51, the type of user operation performed to display the first image 51, and the elapsed time from the start of the video when the first image 51 was displayed.

[0116] The second image display information generation unit 63 generates second image display information based on the received first image display information, associates the second image display information with the corresponding video, and stores it in the HDD 26.

[0117] The second image display information is information about the second image 52 to be displayed in the second image display area 50C, and includes the type of the second image 52, the user ID of the user who displayed the first image 51, which is the source of the second image 52, on their mobile terminal 3, the display period, the number of times it has been synthesized, the status, etc.

[0118] The second image display information transmission control unit 64 controls the external interface 32 to transmit the second image display information to the mobile terminal 3. The destination of the second image display information is a different mobile terminal 3 from the mobile terminal 3 to which the first image display information was transmitted.

[0119] The combination image information generation unit 65 performs a combination determination based on the received first image display information. Then, the combination image information generation unit 65 generates combination image information indicating the combination image according to the result of the combination determination and stores the combination image information in the HDD 26.

[0120] The combined image information is information about the combined image to be displayed on screen 3a, and includes the type of combined image, the user IDs of multiple users who performed the display operation at the same time, etc.

[0121] Furthermore, the combination determination is performed for each piece of first image display information, and it is determined whether the display operation of the first image 51 indicated by the first image display information falls within the combination period based on the start timing of the display operation of the first image 51 by another user.

[0122] The combined image information transmission control unit 66 controls the external interface 32 to transmit the combined image information to the mobile terminal 3. The mobile terminal 3 to which the combined image information is transmitted are multiple mobile terminals 3 corresponding to the user IDs included in the combined image information.

[0123] The replacement processing unit 67 executes the replacement process.

[0124] [2-5-2. Functional block related to onomatopoeia display on mobile devices] Figure 10 is a functional block diagram relating to onomatopoeia display in the mobile terminal 3 according to the embodiment.

[0125] The main control unit 40 of the mobile terminal 3 comprises an image display control unit 70, a display area setting unit 71, a first image display operation reception unit 72, an operation state determination unit 73, a first image display information generation unit 74, a first image display information transmission control unit 75, a second image display information reception control unit 76, and a combined image information reception control unit 77. Each function of the main control unit 40 is realized by a program stored in the auxiliary storage unit 44.

[0126] The image display control unit 70 controls the touch panel display 3a to display an image based on image data. In this embodiment, the image display control unit 70 displays a video distributed from the server 4 in the distributed video display area 50A, displays a first image 51 in the first image display area 50B according to user operation, and displays a second image 52 in the second image display area 50C based on the first image 51 displayed by another user on another mobile terminal 3. Furthermore, in this embodiment, the image display control unit 70 displays the first image 51 superimposed on the distributed video display area 50A beyond the first image display area 50B according to the user operation.

[0127] The display area setting unit 71 sets the position and size (length and width) of the streamed video display area 50A, the first image display area 50B, and the second image display area 50C as the display mode of screen 3a when video is streamed from server 4. Although these positions and sizes are predetermined, the user may set them arbitrarily.

[0128] The first image display operation reception unit 72 receives the user's operation of the display button 53 while the screen 3a is displaying a video. That is, the first image display operation reception unit 72 recognizes the display button 53 that the user has operated from among the multiple display buttons 53, and displays the first image 51 corresponding to that display button 53 on the screen 3a via the image display control unit 70. The first image display operation reception unit 72 displays the first image 51 on the screen 3a such that the display mode of the first image 51 changes according to the determination result of the operation state determination unit 73.

[0129] The operation state determination unit 73 determines the operation state of the display button 53 by the user. The operation state here refers to the operation mode, including the duration of the long press on the display button 53.

[0130] The first image display information generation unit 74 generates first image display information corresponding to the first image 51 displayed on screen 3a by user operation.

[0131] The first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

[0132] The second image display information receiving control unit 76 outputs the second image display information received from the server 4 via the communication unit 46 to the image display control unit 70.

[0133] The combined image information receiving control unit 77 outputs the combined image information received from the server 4 via the communication unit 46 to the image display control unit 70. The image display control unit 70 then displays the combined image on the screen 3a displaying the video. In this embodiment, the image display control unit 70 displays the combined image on the screen 3a in place of the first image 51, which is displayed in response to the user's operation of the display button 53.

[0134] [2-6. Processing Flow] Next, the flow of onomatopoeia display processing in the embodiment of the mobile terminal 3 and server 4 will be explained using a flowchart.

[0135] [2-6-1. Flowchart for displaying onomatopoeia on mobile devices] Figure 11 is a flowchart showing the flow of onomatopoeia display processing performed by the main control unit 40 of the mobile terminal 3. The program for performing onomatopoeia display processing is pre-stored in a predetermined area of ​​the auxiliary storage unit 44. For example, onomatopoeia display processing starts when the mobile terminal 3 starts playing a video distributed from the server 4, and ends when the video playback ends.

[0136] In step S100, the occurrence of a trigger is detected. In this embodiment, the trigger is either a first trigger (reception of user operation of display button 53) or a second trigger (reception of second image display information). The mobile terminal 3 in this embodiment updates the screen display content each time a trigger is detected. The first and second triggers may occur in parallel. Therefore, while processing in response to the occurrence of one trigger is being executed, processing in response to the occurrence of the other trigger may be executed. In addition, multiple second triggers may occur in parallel. Therefore, while processing in response to the occurrence of one second trigger is being executed, processing in response to the occurrence of another second trigger may be executed.

[0137] In step S102, if the detected trigger is the first trigger, the process proceeds to step S106; if the detected trigger is the second trigger, the process proceeds to step S104.

[0138] In step S104, the image display control unit 70 displays the second image 52, indicated by the second image display information, in the second image display area 50C, and the process ends.

[0139] In step S106, the image display control unit 70 displays the first image 51 corresponding to the display button 53 operated by the user in the first image display area 50B. The first image 51 may be stored in the mobile terminal 3 in advance, or it may be downloaded from the server 4 each time the display button 53 is operated by the user.

[0140] In step S108, the first image display information generation unit 74 generates first image display information corresponding to the first image 51 displayed in the first image display area 50B.

[0141] In step S110, the first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

[0142] In step S112, the operation status determination unit 73 determines whether the user has finished operating the display button 53. If the determination is positive, the process proceeds to step S120; if the determination is negative, the process proceeds to step S114.

[0143] In step S114, the combined image information reception control unit 77 determines whether or not combined image information has been received via the communication unit 46. If the determination is positive, the process proceeds to step S118; otherwise, the process proceeds to step S116.

[0144] In step S116, the image display control unit 70 displays the first image 51 (onomatopoeia image) in an extended format until the user finishes operating the display button 53, and then returns to step S112.

[0145] In step S118, the image display control unit 70 displays the combined image indicated by the combined image information on screen 3a, and then returns to step S112.

[0146] In step S120, the image display control unit 70 finishes displaying the first image 51 and terminates the process.

[0147] In the flowchart shown in Figure 11, the first image display information is generated along with the display of the first image 51, and the first image display information is sent to the server 4. However, the system is not limited to this, and the first image display information may be sent to the server 4 after the display of the first image 51 has finished. Furthermore, the first image display information transmission control unit 75 may send first image display information indicating the start of the display of the first image 51 to the server 4 at the timing when the display of the first image 51 begins, and send first image display information indicating the end of the display of the second image 52 to the server 4 at the timing when the display of the first image 51 has finished. This allows the server 4 to recognize that the first image 51 (onomatopoeia image) is being displayed in an expanded form during the interval between receiving the first image display information indicating the start of the display of the first image 51 and the first image display information indicating the end of the display.

[0148] [2-6-2. Flowchart for displaying onomatopoeia on the server] Figure 12 is a flowchart showing the flow of onomatopoeia display processing performed by the CPU 20 of server 4. The program for performing onomatopoeia display processing is pre-stored in a predetermined area of ​​HDD 26. For example, onomatopoeia display processing starts when server 4 is distributing video to mobile terminal 3 and ends when video distribution ends.

[0149] In step S200, the first image display information reception control unit 62 determines whether or not it has received the first image display information from the mobile terminal 3. If the determination is positive, the process proceeds to step S202; otherwise, the process enters a waiting state.

[0150] In step S202, the first image display information receiving control unit 62 stores the received first image display information in the HDD 26.

[0151] In step S204, the combined image information generation unit 65 performs a combination determination. If the determination is positive, the process proceeds to step S208; if the determination is negative, the process proceeds to step S206. Specifically, if the display operation of the first image 51 to be combined is performed by multiple users within the combination period, the determination is positive and the process proceeds to step S208. On the other hand, if the display operation of the first image 51 to be combined is not performed by multiple users within the combination period, the determination is negative and the process proceeds to step S206.

[0152] In step S206, the second image display information generation unit 63 generates second image display information based on the received first image display information and stores it in the HDD 26.

[0153] In step S208, the combination image information generation unit 65 generates combination image information that indicates a combination image corresponding to the display operation of the first image 51 that is the subject of the combination.

[0154] In step S210, the combined image information transmission control unit 66 controls the external interface 32 to transmit the combined image information and returns to step S200. The destinations for the combined image information are the multiple mobile terminals 3 that transmitted the first image display information to be combined, i.e., the mobile terminals 3 corresponding to the multiple user IDs included in the combined image information.

[0155] Figure 13 is a flowchart showing the flow of onomatopoeia display processing performed by the CPU 20 of server 4. The flow shown in Figure 13 targets the second image display information whose display period start date falls within the interval at the end of each interval in the video timeline.

[0156] In step S300, the replacement processing unit 67 makes a determination. If the determination is positive, the process proceeds to step S302; if the determination is negative, the process proceeds to step S304. Specifically, if there is second image display information that has commonalities between the subject and the object, the determination is positive and the process proceeds to step S302. On the other hand, if there is no second image display information that has commonalities between the subject and the object, the determination is negative and the process proceeds to step S304.

[0157] In step S302, the replacement processing unit 67 performs replacement processing. In the unit processing of the replacement processing, the replacement processing unit 67 identifies two second image display information that have commonalities between the subject and the object as an image group, invalidates the status of the two second image display information included in the image group, and generates one new second image display information. In step S302, this unit processing is repeated until there are no more second image display information that have commonalities between the subject and the object.

[0158] In step S304, the second image display information transmission control unit 64 controls the external interface 32 to transmit all of the second image display information included in the target section to a different mobile terminal 3 from the mobile terminal 3 that transmitted the first image display information. Therefore, the display period of the second image 52 based on the second image display information on the mobile terminal 3 will be delayed compared to the display period associated with the video.

[0159] As described above, in the embodiment, the mobile terminal 3 provides the server 4 with operation information for an addition operation performed on the first mobile terminal 3 by a first user who is watching a video displayed on the screen 3a of the first mobile terminal 3 in order to add a first image 51 to the video, and operation information for an addition operation performed on the second mobile terminal 3 by a second user who is watching the same video displayed on the screen 3a of the second mobile terminal 3 in order to add a first image 51 to the video. In order to improve entertainment value, the server 4 displays a combination image based on the combination of the addition operation by the first user and the addition operation by the second user on the screen 3a of the first mobile terminal 3 and the screen 3a of the second mobile terminal 3, respectively. As a result, images that would not be displayed by an operation by one user are displayed by a combination of operations by other users, so the mobile terminal 3 can encourage more users to watch the video.

[0160] Furthermore, the server 4 in this embodiment replaces two or more second images 52 associated with the video that have close display timings and share common subject and object characteristics with one new second image 52 during video playback. As a result, the number of second images 52 displayed on the mobile terminal 3 is reduced, allowing the server 4 to suppress sudden increases in load on the mobile terminal 3.

[0161] [3. Variant] The technical scope of the present invention is not limited to the scope described in the above embodiments. Various modifications or improvements can be made to the above embodiments without departing from the spirit of the invention, and such modified or improved forms are also included in the technical scope of the present invention.

[0162] Furthermore, the various processing flows described in the above embodiments are merely examples, and unnecessary steps may be deleted, new steps added, or the processing order rearranged without departing from the spirit of the present invention.

[0163] [3-1. Variation 1 (Sharing images without using a server)] In the above embodiment, a server 4 receives first image display information from a mobile terminal 3, transmits second image display information generated based on the first image display information to another mobile terminal 3, and the other mobile terminal 3 displays the second image 52 indicated by the second image display information in the second image display area 50C. However, the present invention is not limited to this embodiment.

[0164] For example, multiple mobile terminals 3 may send and receive first image display information to each other without going through server 4, and each mobile terminal 3 may display the second image 52 in the second image display area 50C. That is, when mobile terminal 3 receives first image display information from another mobile terminal 3, it reads information about the first image 51 displayed on the other mobile terminal 3 from the first image display information and displays the second image 52 based on that information in the second image display area 50C.

[0165] [3-2. Variation 2 (Watching the video independently)] The above embodiment illustrates a case where images expressing emotions towards a video are shared among multiple users simultaneously watching the video. In contrast, an image expressing the emotions of a user who watched a video first can be shared with a user who watches the video later. This makes it possible to display combined images according to the combination of additional actions taken by the first user who watched the video first and the second user who watched it later.

[0166] In the following, we consider the case where the timing at which a first user using a first mobile terminal 3 (first device) views a certain video (viewing target) is later than the timing at which a second user using a second mobile terminal 3 (second device) views the same video (Figure 13(A)). In this case, the additional operations performed by the first user and the additional operations performed by the second user may not be in close proximity in real time. However, on the video's timeline, they are in close proximity (Figure 13(B)). The first user and the second user may be the same person and someone else, or the same person in the past and the same person in the present.

[0167] In the modified example 2, the operation information of the addition operation (second operation) performed on the second mobile terminal 3a by a second user who is watching a video displayed on the screen 3a (second screen) of the second mobile terminal 3 in order to add a first image (second element) to the video is transmitted from the second mobile terminal 3a to the server 4. The information is provided to the (second instruction means, shared information processing device). The server 4 pre-transmits second image display information (second instruction), which causes the second image (second element) to be displayed at a timing corresponding to an additional operation by the second user, to the first mobile terminal 3 used by the first user who wants to watch the same video.

[0168] In order to improve entertainment value, the first mobile terminal 3, upon receiving an addition operation (first operation) performed on the first mobile terminal 3a by a first user who is watching a video displayed on the screen 3a (first screen) of the first mobile terminal 3 to add a first image (first element) to the video, displays a combined image (third element) based on the combination of the received addition operation (first operation) and the addition operation (second operation) related to the second image display information on the screen 3a of the first mobile terminal 3. In modified example 2, since the timing of the second operation performed on the second mobile terminal 3 is known in advance, it is also possible to display the combined image (third element) based on the combination of the received addition operation (first operation) and the addition operation (second operation) related to the second image display information on the screen 3a of the first mobile terminal 3 without delay after receiving the first operation on the first mobile terminal 3.

[0169] [3-3. Variation 3 (Do not display the first image when a combined image is displayed)] In the above embodiment, the first image is displayed even when a combined image is displayed. Alternatively, the first image may be prevented from being displayed when a combined image is displayed.

[0170] Specifically, the image display control unit 70 (second control means) of the mobile terminal 3 waits for the reception of combined image information until a predetermined period (combination period) has elapsed since the additional operation was received by the first image display operation reception unit 72 (first reception means). If the image display control unit 70 receives combined image information within the predetermined time, it displays a combined image (third element) based on the received combined image information. If it does not receive combined image information within the predetermined period, it displays a first image (first element) based on the received additional operation.

[0171] [3-4. Variation 4 (Display the second image even when a combined image is displayed)] In the above embodiment, the second image is not displayed when the combined image is displayed. However, the second image may be displayed even when the combined image is displayed. Specifically, when the server 4 receives the first image information, it may generate second image display information regardless of whether or not a combined display operation has been performed, and send it to the mobile terminal 3.

[0172] [3-5. Modification 5 (Limiting the activation of the substitution process)] In the above embodiment, the unit processing is repeated at predetermined intervals (combination periods), and multiple second image display information to be replaced is replaced with one second image display information. However, in situations where sudden increases in load are unlikely to occur on the mobile terminal 3, the replacement process may be omitted.

[0173] Specifically, the replacement processing unit 67 (identification means) of server 4 omits the unit processing (replacement processing) that identifies two (a predetermined number) of second image display information included in a group of adjacent element information (shared elements) whose display timings are close together as an image group (element group) if the number of elements in the adjacent element group does not satisfy the activation condition. The activation condition is, for example, that the number of elements is equal to or greater than a predetermined threshold. The activation condition may be flexibly varied according to the processing capacity of the mobile terminal 3. For example, a predetermined threshold may be set according to the processing capacity of the terminal with the lower capacity among the multiple mobile terminals 3 belonging to group 5. Processing capacity refers to, for example, the display processing capacity (DSP clock frequency) or reception processing capacity (communication line speed) of the mobile terminal 3. [Explanation of symbols]

[0174] 1. Video distribution system 2. Communication lines 3. Mobile terminals (examples of "device 1", "device 2", and "information processing device") 3a Screen (Examples of "Screen 1" and "Screen 2") 4. Server (an example of "first instruction means," "second instruction means," and "shared information processing device") 5. Group (An example of an "information processing system") 50A Streaming video display area 50B 1st image display area 50C 2nd image display area 51 Image 1 52. Second image (an example of "shared elements" and "alternative elements") 53 Display button 60 Group Registration Department 61 Video Streaming Department 62 First Image Reception Information Reception Control Unit 63 Second image display information generation section 64 Second Image Display Information Transmission Control Unit 65 Combined Image Information Generation Unit 66 Combined Image Information Transmission Control Unit 67. Substitution Processing Unit (Example of "Specific Means" and "Substitution Means") 70 Image display control unit (an example of "first control means", "second control means", "third control means", and "display control means") 71 Display area setting section 72 First image display operation reception unit (an example of "first reception means" and "second reception means") 73 Operation status determination unit 74 First image display information generation section 75 First Image Display Information Transmission Control Unit (an example of the "First Providing Means" and "Second Providing Means") 76 Second image display information receiving control unit (an example of the "third receiving means") 77 Combined Image Information Receiving and Control Unit

Claims

1. The processor is instructed to display a video featuring characters, delivered from the server, on the touch panel display of the first terminal at a predetermined date and time. The processor causes the first button to be displayed on the touch panel display of the first terminal. The processor causes the first image to be displayed on the touch panel display of the first terminal based on the touch operation of the first button by the user of the first terminal. The processor causes the touch panel display of the first terminal to display an image on the first terminal showing another user operating the second terminal who is registered in the same group as the user of the first terminal. The processor receives the second image display information generated by the server based on the operation of the second terminal by the other user. The processor is instructed to display the second image on the touch panel display of the first terminal based on the second image display information. The aforementioned first image includes an image that can be displayed by paying with points. program.

2. The first image above includes images related to emotions, The program according to claim 1.

3. Equipped with one or more processors, The one or more processors mentioned above are: At a predetermined date and time, a video featuring the character is displayed on the touch panel display of the first terminal. The system displays a video featuring characters on the touch panel display of a second terminal operated by another user registered in the same group as the user of the first terminal. The first button is displayed on the touch panel display of the first terminal. Based on the touch operation of the first button by the user of the first terminal, the first image is displayed on the touch panel display of the first terminal. The second button is displayed on the touch panel display of the second terminal. Based on the user of the second terminal touching the second button, the second image is displayed on the touch panel display of the second terminal. The second image and an image identifying the user of the second terminal are displayed on the touch panel display of the first terminal. The aforementioned first image includes an image that can be displayed by paying points, The second image mentioned above includes an image that can be displayed by paying points. The first image and the second image that can be displayed depending on the point payment of the user of the first terminal and the user of the second terminal are different. Information processing device.

4. The processor is instructed to display a video featuring a character on the touch panel display of a first terminal at a predetermined date and time, and to display a video featuring a character on the touch panel display of a second terminal operated by another user registered in the same group as the user of the first terminal. The processor causes the first button to be displayed on the touch panel display of the first terminal. The processor causes the first image to be displayed on the touch panel display of the first terminal based on the touch operation of the first button by the user of the first terminal. The processor causes the second button to appear on the touch panel display of the second terminal. The processor causes the user of the second terminal to touch the second button, and to display the second image on the touch panel display of the second terminal, and to display the second image and an image identifying the user of the second terminal on the touch panel display of the first terminal. The aforementioned first image includes an image that can be displayed by paying points, The second image mentioned above includes an image that can be displayed by paying points. The first image and the second image that can be displayed depending on the point payment of the user of the first terminal and the user of the second terminal are different. method.

5. Equipped with a server and multiple terminals, The aforementioned server, At a predetermined date and time, a video featuring the character is displayed on the touch panel display of the first terminal. The system displays a video featuring characters on the touch panel display of a second terminal operated by another user registered in the same group as the user of the first terminal. The first button is displayed on the touch panel display of the first terminal. Based on the touch operation of the first button by the user of the first terminal, the first image is displayed on the touch panel display of the first terminal. The second button is displayed on the touch panel display of the second terminal. Based on the user of the second terminal touching the second button, the second image is displayed on the touch panel display of the second terminal. The second image and an image identifying the user of the second terminal are displayed on the touch panel display of the first terminal. The aforementioned first image includes an image that can be displayed by paying points, The second image mentioned above includes an image that can be displayed by paying points. The first image and the second image that can be displayed depending on the point payment of the user of the first terminal and the user of the second terminal are different. system.

Images