Information processing system, information processing device and control program

Abstract

To improve entertainment properties in a system for displaying reactions of viewers on a display screen of a viewing target.SOLUTION: An information processing system (e.g., group 5) includes: first reception means (e.g., a first image display operation reception unit 72) that receives first operation performed on a first device (e.g., a portable terminal 3) by a first user who is viewing a viewing target displayed on a first screen (e.g., a screen 3a) of the first device in order to add a first element to the viewing target; second reception means (e.g., a first image display operation reception unit 72) that receives second operation performed on a second device (e.g., a portable terminal 3) by a second user who is viewing the viewing target displayed on a second screen (e.g., a screen 3a) of the second device in order to add a second element to the viewing target; and first control means (e.g., an image display control unit 70 of the first device) that displays a third element on the first screen based on a combination of the first operation and the second operation.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing device, and a control program. [Background technology]

[0002] There is known a mechanism for allowing viewers of video content being distributed to participate in a group chat about the content of the video content. For example, Patent Document 1 discloses a client device that displays a fixed message related to the content of the video content in response to a button selected by a user on a chat screen where avatars corresponding to the users participating in the group chat are arranged around the video content. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 026170 Summary of the Invention [Problem to be solved by the invention]

[0004] The client device described in Patent Document 1 displays a standard message selected by the user in a speech bubble next to an avatar representing the user. With this method of displaying standard messages, the number of avatars increases as the number of users watching video content increases, but the entertainment value for users does not improve.

[0005] The problem to be solved by the present invention is to improve the entertainment value of a system for displaying the viewer's reaction on the display screen of the viewing target. [Means for solving the problem]

[0006] In order to solve the above problem, the present invention displays a third element on a first screen of a first device based on a combination of a first operation performed on a first device by a first user viewing a viewing object displayed on a first screen of the first device to add a first element to the viewing object, and a second operation performed on a second device by a second user viewing the viewing object displayed on a second screen of the second device to add a second element to the viewing object. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the entertainment value of a system in which the viewer's reaction is displayed on the display screen of the viewing target. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating the configuration of a video distribution system (embodiment). [Figure 2] 1 is a block diagram showing the electrical configuration of a server (embodiment); [Figure 3] 1 is a block diagram showing the electrical configuration of a mobile terminal (embodiment); [Figure 4] 10 is a diagram showing a display mode of a screen of a mobile terminal (embodiment); [Figure 5] FIG. 10 is a diagram showing a screen displaying an onomatopoeia image (Example). [Figure 6] 10 is a diagram showing a screen displaying a second image (Example). [Figure 7] 10 is a timing chart showing a case where a combined image is displayed (Example). [Figure 8] 10 is a timing chart showing a case where the second image is replaced (Example). [Figure 9] FIG. 1 is a functional block diagram of a server for video distribution (embodiment); [Figure 10] FIG. 1 is a functional block diagram relating to onomatopoeia display on a mobile terminal (embodiment); [Figure 11] 10 is a flowchart showing the flow of an onomatopoeia display process executed on a mobile terminal (embodiment). [Figure 12] 10 is a flowchart showing the flow of an onomatopoeia display process executed by the server (embodiment). [Figure 13] 10 is a flowchart showing the flow of an onomatopoeia display process executed by the server (embodiment). [Figure 14] 10 is a timing chart showing a case where a combined image is displayed (Modification 2). DETAILED DESCRIPTION OF THE INVENTION

[0009] [1. Embodiment] [1-1. Main Features] This embodiment relates to a mechanism for displaying viewer responses on a display screen of a viewing target. In this embodiment, in order to enhance entertainment value, a third element based on a combination of a first operation performed on a first device by a first user viewing a viewing target displayed on a first screen of the first device to add a first element to the viewing target, and a second operation performed on a second device by a second user viewing the viewing target displayed on a second screen of the second device to add a second element to the viewing target is displayed on the first screen of the first device.

[0010] The viewing object refers to video content that is at least visually perceived by the user. The video content may also include audio content that is audibly perceived by the user. The video content is distributed, for example, from a distribution device to an information processing device. The video content is preferably distributed, for example, by streaming distribution or progressive download distribution. The video content is, for example, a moving image. The viewing object is displayed in a basic area that is a part of the screen of the information processing device.

[0011] A user (viewer) viewing a viewing target can add a predetermined element (additional element) to the viewing target by performing a predetermined operation (additional operation). The predetermined operation can be, for example, an input element The input element is, for example, a pointing operation on an operation element. The input element is, for example, a button displayed on the screen. The pointing operation is, for example, a combination of a touch operation on a touch panel and a positioning operation and a click operation of a pointer displayed on the screen. The predetermined element is, for example, an image representing the user's reaction to the viewing object. The user's reaction is, for example, the user's emotion. An image representing the user's emotion to the viewing object is sometimes called an emotional image. The emotional image is, for example, an image that depicts onomatopoeia (also called onomatopoeic words, mimetic words, or mimetic words) that expresses the user's emotion. An image that depicts onomatopoeia is sometimes called an onomatopoeia image. Adding an element to a viewing object means associating the display timing of the element with the progress of the viewing object. When the display timing is associated with the progress, the display timing is specified at a specific position on the timeline. The element added to the viewing object can be displayed on the screen at the associated display timing while the viewing object is being played (i.e., while it is being played and displayed).

[0012] The first user is a user (the user himself / herself) who uses a first device (information processing device). A predetermined operation (first operation) by the first user is accepted, for example, by the first device. The second user is another user (another person) who uses a second device (another information processing device). A predetermined operation (second operation) by the second user is accepted, for example, by the second device. A predetermined element (first element) based on the first operation is displayed in a first range that at least partially overlaps the basic region. A predetermined element (second element) based on the second operation is displayed in a second range that does not overlap the basic region. The first region may have a portion (first portion) that does not overlap the basic region in addition to a portion (second portion) that overlaps the basic region. The first region may further have a portion (third portion) that overlaps the second region.

[0013] Based on the timing of the first operation and the second operation, a predetermined element (third element) based on the combination of the first operation and the second operation is displayed. Specifically, when the timings match or are close, the predetermined element is displayed. "Close in timing" refers to close in real time or close in position on the timeline of the viewing target. "Close in real time" refers to close in real time the timing of the first operation and the second operation when a first user and a second user are viewing the viewing target at approximately the same time. "Close in position on the timeline of the viewing target" refers to close in timing on the timeline of the first operation and the second operation when a first user and a second user are viewing the viewing target at approximately the same time or at different times. "Close" refers to, for example, the time difference being within a predetermined threshold. The predetermined element based on the combination of the first operation and the second operation is preferably displayed in the first range.

[0014] [1-2. Notes] The present invention includes at least the following embodiments [A] to [D].

[0015] [A] An information processing system comprising: a first receiving means for receiving a first operation made on a first device by a first user viewing a viewing object displayed on a first screen of the first device to add a first element to the viewing object; a second receiving means for receiving a second operation made on a second device by a second user viewing the viewing object displayed on a second screen of the second device to add a second element to the viewing object; and a first control means for displaying a third element based on a combination of the first operation and the second operation on the first screen.

[0016] [B] An information processing device comprising: a receiving means for receiving a first operation performed by a first user viewing an object to be viewed to add a first element to the object to be viewed; and a display control means for displaying a third element based on a combination of the first operation and a second operation performed by a second user viewing the object to another information processing device to add a second element to the object to be viewed.

[0017] [C] A control program that causes a computer of an information processing device to implement a reception function that receives a first operation performed by a first user viewing a viewing target to add a first element to the viewing target, and a display control function that displays a third element based on a combination of the first operation and a second operation performed by a second user viewing the viewing target on another information processing device to add a second element to the viewing target. Note that the control program may be non-temporarily recorded on a recording medium.

[0018] [D] A control method for an information processing device, including: a reception step of receiving a first operation performed by a first user viewing an object to add a first element to the object to be viewed; and a display control step of displaying a third element based on a combination of the first operation and a second operation performed by a second user viewing the object on another information processing device to add a second element to the object to be viewed.

[0019] The following technical limitations may be added to the "information processing system" in [A] above. Similar technical limitations may also be added to the "information processing device" in [B] above, the "control program" in [C] above, and the "control method for an information processing device" in [D] above.

[0020] (1) The first control means displays on the first screen the third element based on a combination of the first operation and the second operation performed within a predetermined period based on the time of reception of the first operation on the timeline of the viewing target. The predetermined period based on the time of reception is, for example, a predetermined period with the time of reception as its start point, end point, or midpoint, or a predetermined period including the time of reception. The predetermined period may be a fixed value or a variable value. The variable value may vary depending on, for example, the number of users viewing and the number of added predetermined elements. As a result, the third element is displayed when the first operation and the second operation are close to each other on the timeline.

[0021] (2) The device further includes a first providing means for providing operation information of the first operation to a first instruction means for outputting a first instruction to display the third element, and a second providing means for providing operation information of the second operation to the first instruction means. When the first control means receives the first instruction output from the first instruction means based on a combination of the first operation and the second operation corresponding to the received operation information within the predetermined period, the device displays the third element based on the first instruction on the first screen. The first instruction means is, for example, a server device connectable to the first device and the second device. The first instruction can be output at any time while the viewing target is being viewed. As a result, when a first user and a second user are viewing at the same time, the third element based on the combination of the first operation and the second operation can be displayed.

[0022] (3) The device further includes a third receiving means for receiving a second instruction output from a second instruction means for outputting a second instruction to display the second element based on the second operation at a timing corresponding to the second operation. When the first control means receives the second instruction to display the second element at a timing included in the predetermined period before the first operation is performed, the third element based on a combination of the first operation and the second operation related to the second instruction is displayed on the first screen. The second instruction means is, for example, a server device connectable to the first device and the second device. The second instruction may be output collectively when viewing of the viewing target begins. As a result, when the second user is viewing before the first user, the third element based on the combination of the first operation and the second operation may be displayed.

[0023] (4) The display device further includes at least one of a second control means for displaying the first element based on the first operation on the first screen except when the third element is displayed in response to the first operation, and a third control means for displaying the second element based on the second instruction on the first screen except when the third element is displayed in response to the second instruction, whereby at least one of the first element and the second element is not displayed when the third element is displayed. [Example]

[0024] 2. Working Example An embodiment of the present invention will be described below with reference to the drawings. This embodiment relates to a mechanism for displaying an image (image showing a reaction) representing the emotion of a user (viewer) on a display screen of a video (viewing target). In this embodiment, the shared information processing device is a server, and the information processing devices (first device, second device) are mobile terminals, so-called smartphones. Note that the mobile terminal is not limited to a smartphone, but may also be a tablet terminal, a laptop computer, or the like. The information processing device may also be a desktop personal computer. Each program according to the embodiment is stored in the shared information processing device or the information processing device, but is not limited thereto and may also be stored in a storage medium such as an optical disk such as a CD-ROM (Compact Disk Read Only Memory) or a portable semiconductor memory such as a USB (Universal Serial Bus) flash memory.

[0025] [2-1. Features] In this embodiment, operation information of an addition operation (first operation) performed by a first user (owner) viewing a video (viewing target) displayed on a screen (first screen) of a first mobile terminal (first device) to add a first image (first element) to the video, and operation information of an addition operation (second operation) performed by a second user (another person) viewing the video displayed on a screen (second screen) of a second mobile terminal (second device) to add a first image (second element) to the video are provided from each mobile terminal to a server (first instruction means, shared information processing device).To improve entertainment value, the server displays a combination image (third element) based on a combination of the first operation and the second operation according to the provided operation information on the first screen and the second screen, respectively.

[0026] Furthermore, in this embodiment, in order to suppress a sudden increase in load on a mobile terminal, two or more (a predetermined number) second images (shared elements) associated with a video, the timing of which is to be displayed on the mobile terminal of a user watching the video while the video is being played, are identified as an image group (element group) based on the proximity (timing) of the display timing and the commonality (additional attribute) of the subject and object, and a replacement process is executed in which the second images included in the image group are invalidated and one (a number less than the predetermined number) new second image (alternative element) that inherits the additional attribute is added to the video. The subject refers to the user who added the second image to the video, and having commonality among the subjects means that the subjects belong to the same user group. Each user belongs to at least one of the user groups. The object refers to the second image added to the video. A commonality refers to the type of (element), and objects have commonality if they belong to the same element group. Each element belongs to at least one element group.

[0027] [2-2.Video distribution system] A server according to an embodiment is included in a video distribution system and distributes video to a mobile terminal.A mobile terminal according to an embodiment is included in a video distribution system and receives video from the server.

[0028] [2-2-1. Overall configuration of video distribution system] 1 is a schematic diagram of a video distribution system 1 according to an embodiment. The video distribution system 1 includes a communication line 2, a plurality of mobile terminals 3, and a server 4.

[0029] The communication line 2 forms a computer network, and is, for example, a wide area communication line provided by an electric utility company.

[0030] The mobile terminal 3 is, for example, an information processing terminal such as a smartphone, tablet terminal, or laptop computer. The mobile terminal 3 includes a touch panel display 3a for displaying images, a speaker 3b for outputting sound, a microphone 3c for inputting sound, and a camera 3d for capturing images of a subject. In order to display videos distributed from the server 4, for example, dedicated application software (hereinafter referred to as a "video viewing app") is installed on the mobile terminal 3.

[0031] The server 4 is a shared information processing device that distributes content to multiple mobile terminals 3 via a communication line 2. The content according to the embodiment is, for example, a video. The video may be, for example, an animation or a drama that has a story and progresses along the story, or a music video, a live music video, or a variety of sports videos, and is not particularly limited. Note that, for example, the video according to the embodiment is an animation. Furthermore, the content distributed from the server is not limited to a video, and may be other services such as a collection of consecutive still images, as long as the content is viewable by the user.

[0032] Furthermore, viewing here means that the user perceives the content using at least one of sight and hearing, and the content may, in addition to the above, be, for example, a video without sound, or sound without images.

[0033] For example, a plurality of mobile terminals 3 to which video is distributed are registered in the server 4 as a group 5. (Hereinafter referred to as "group registration"). The server 4 simultaneously distributes videos to the mobile terminals 3 registered in the group. That is, the multiple mobile terminals 3 registered in the group display the videos distributed from the server 4 at the same time, in other words, synchronized, on their touch panel displays 3a. This allows each user of the mobile terminals 3 registered in the group to watch the videos at the same time.

[0034] The video may be distributed at a predetermined date and time, or at any timing for each group 5. When the video is distributed at any timing, for example, the host mobile terminal 3 requests the server 4 to distribute the video, and the server 4, upon receiving the distribution request, distributes the video to the mobile terminals 3 that make up the group 5.

[0035] 1, group 5 is made up of four mobile terminals 3, but this is just an example, and there is no limit to the number of mobile terminals 3 in group 5 as long as group 5 is made up of multiple mobile terminals 3. There is also no limit to the types of mobile terminals 3 that make up group 5, and a mixture of smartphones, tablet terminals, laptop computers, etc. may be present.

[0036] [2-2-2. Server configuration] FIG. 2 is a block diagram showing the electrical configuration of the server 4 according to the embodiment.

[0037] The server 4 according to the embodiment includes a CPU (Central Processing Unit) 20, which is the main control unit that controls the operation of the entire server 4, a ROM (Read Only Memory) 22 in which various programs and various data are pre-stored, a RAM (Random Access Memory) 24 that is used as a work area when the CPU 20 executes various programs, and an HDD (Hard Disk Drive) 26 as a storage means for storing various programs and various data.

[0038] The HDD 26 stores information about the registered group 5, moving images (moving image data) to be distributed to the mobile terminal 3, and various other information. Note that the storage means is not limited to the HDD 26, and may be, for example, another storage medium such as a semiconductor memory, such as a flash memory.

[0039] Furthermore, the server 4 is equipped with an operation input unit 28 consisting of a keyboard, mouse, etc., which accepts input of various operations, a monitor 30 such as a liquid crystal display device which displays various images, and an external interface 32 which is connected to other information processing devices such as a mobile terminal 3 via the communication line 2 and sends and receives various data to and from the other information processing devices.

[0040] The CPU 20, ROM 22, RAM 24, HDD 26, operation input unit 28, monitor 30, and external interface 32 are electrically connected to one another via a system bus 34. Therefore, the CPU 20 can access the ROM 22, RAM 24, and HDD 26, grasp the operation status of the operation input unit 28, display images on the monitor 30, and transmit and receive various data to and from other information processing devices, etc. via the external interface 32.

[0041] [2-2-3. Electrical configuration of mobile terminal] Fig. 3 is a functional block diagram showing the electrical configuration of the mobile terminal 3. In addition to the configuration shown in Fig. 1, the mobile terminal 3 includes a main control unit 40, a main memory unit 42, an auxiliary memory unit 44, a communication unit 46, and operation buttons 48.

[0042] The main control unit 40 is, for example, a CPU, a microprocessor, a DSP (Digital Signal Processor), or the like, and controls the overall operation of the mobile terminal 3.

[0043] The main storage unit 42 is configured with, for example, RAM or DRAM (Dynamic Random Access Memory), and is used as a work area when the main control unit 40 executes processes based on various programs.

[0044] The auxiliary storage unit 44 is, for example, a non-volatile memory such as a flash memory, and stores various data such as images, as well as programs used in processing by the main control unit 40. The programs stored in the auxiliary storage unit 44 include, for example, an operating system (OS) for implementing the basic functions of the mobile terminal 3, drivers for controlling various hardware, and programs for implementing email, web browsing, and various other functions. The auxiliary storage unit 44 also stores in advance application software such as a video viewing app for playing videos distributed from the server 4 and displaying them on the touch panel display 3a.

[0045] The communication unit 46 is, for example, a NIC (Network Interface Controller), and has a function of connecting to the communication line 2. Note that the communication unit 46 may have, instead of or together with the NIC, a function of connecting to a wireless LAN (Local Area Network), a function of connecting to a wireless WAN (Wide Area Network), a function of enabling short-range wireless communication such as Bluetooth (registered trademark), infrared communication, etc.

[0046] The operation buttons 48 are provided on the side of the mobile terminal 3, and include a power button for starting or stopping the mobile terminal 3, a volume adjustment button for adjusting the volume of the sound output by the speaker 3b, and the like.

[0047] The main control unit 40, main memory unit 42, auxiliary memory unit 44, communication unit 46, touch panel display 3a, speaker 3b, microphone 3c, camera 3d, and operation buttons 48 are electrically connected to one another via a system bus 49. Therefore, the main control unit 40 can access the main memory unit 42 and auxiliary memory unit 44, display images on the touch panel display 3a, grasp the operation status of the touch panel display 3a and operation buttons 48 by the user, input sound to the microphone 3c, output sound from the speaker 3b, control the camera 3d, and access various communication networks and other information processing devices via the communication unit 46.

[0048] [2-3. Onomatopoeia display] The mobile terminal 3 according to the embodiment is capable of displaying images (onomatopoeia images, the details of which will be described later) in response to user operations while displaying videos distributed from the server 4 on the touch panel display 3a. In the following description, the function of displaying onomatopoeia images on the touch panel display 3a is referred to as an onomatopoeia display function, and the touch panel display 3a is also referred to as a screen 3a.

[0049] The onomatopoeia display function is, for example, one of the functions of the video viewing application according to the embodiment. Note that, for example, not all videos displayed on the screen 3a using the video viewing application can display onomatopoeia, and only videos that can display onomatopoeia are predetermined.

[0050] [2-3-1. Screen Configuration] 4 to 6 show display modes of a screen 3a of a mobile terminal 3 on which a video viewing application according to the embodiment is running. A distribution video display area 50A, a first image display area 50B, and a second image display area 50C are set on the screen 3a.

[0051] The distributed video display area 50A is an area for displaying videos distributed from the server 4. The first image display area 50B is an area for displaying a first image 51 (see FIG. 5) in response to a user's operation. The second image display area 50C is an area for displaying a second image 52 (see FIG. 6) based on the first image 51 displayed on another mobile terminal 3 by another user's operation. As will be described in detail later, the second image 52 is a reduced image of the first image 51 displayed in the first image display area 50B of the other mobile terminal 3. In addition, the first image display area 50B and the second image display area 50C according to the embodiment are, for example, provided at positions facing each other with the distributed video display area 50A in between.

[0052] Here, the first image 51 is an image including an emotional image that shows the emotion of the user watching the video. As an example, the emotional image according to the embodiment is an image in which an onomatopoeic word (also called an onomatopoeic word or mimetic word) that shows the emotion of the user is illustrated, in other words, an onomatopoeic image. Examples of onomatopoeic images include "Awesome," "Awesome," and "Bravo." The stylized image is, for example, a decorative drawing of letters that show the onomatopoeic word.

[0053] In addition to onomatopoeia images, the first image 51 may also include various other images, such as character images showing characters appearing in the video, dialogue images showing the lines of the characters, text entered by the user, and the like.

[0054] The first image 51 may also vary depending on the scene of the video. For example, an onomatopoeia image or a character image that is likely to be selected by the user in response to a happy scene, a sad scene, a scene of a confrontation between characters, or the like may be displayed.

[0055] The first images 51 that can be displayed may differ depending on the user. For example, the types of first images 51 that can be displayed may increase by acquiring a specific item through a service such as a game related to the distributed video. Furthermore, the types of first images 51 that can be displayed may increase, for example, on the condition that a predetermined amount of compensation (for example, monetary value such as currency, virtual currency, or points) is paid.

[0056] In the first image display area 50B according to the embodiment, a plurality of display buttons 53A to 53D are displayed as operation units that are operated by the user and each correspond to a different first image 51. As a result, the user can display the first image 51 in the first image display area 50B by operating the display buttons 53A to 53D while watching a video. Note that, for example, operating the display buttons 53A to 53D means touching (pressing) the display buttons 53A to 53D with a pointer. For example, the pointer according to the embodiment is the user's finger. In the following description, when the display buttons 53A to 53D are not to be distinguished from one another, the alphabetical notation at the end of the name will be omitted.

[0057] In the examples of FIGS. 4 to 6, four display buttons 53A to 53D are displayed in a row horizontally at the bottom of the first image display area 50B. Display button 53A corresponds to an onomatopoeia image that is a stylized version of "SUGE," display button 53B corresponds to an onomatopoeia image that is a stylized version of "YABAI," and display button 53C corresponds to an onomatopoeia image that is a stylized version of "BRAVO." Furthermore, display button 53D corresponds to a character image. Note that the display buttons 53 are designed so that the user can intuitively recognize the corresponding onomatopoeia image, character image, etc.

[0058] In this way, by displaying a plurality of display buttons 53 in the first image display area 50B, the user can easily display on the screen 3a the first image 51 that best matches the user's reaction (emotion) to the video. Furthermore, by including an onomatopoeia image that indicates the user's emotion in the first image 51, the user can display on the screen 3a the emotion that the user felt when watching the video.

[0059] 4 to 6, four types of display buttons 53 are displayed in the first image display area 50B, but this is just an example, and the number and types of display buttons 53 are not limited to these. Furthermore, the display buttons 53 may be switched as appropriate so that different first images 51 can be displayed depending on the scene of the video, etc.

[0060] The arrangement of the display buttons 53 is also an example, and for example, the display buttons 53 may be displayed together on the right or left side of the first image display area 50B, or may be displayed in two horizontal rows at the bottom of the first image display area 50B. Furthermore, for example, when the user swipes the bottom of the screen 3a to the left or right, new display buttons 53 for displaying different types of first images 51 may be displayed.

[0061] [2-3-2. Display of the first image] The display mode of the first image 51 will be described in detail with reference to Fig. 5. In the example of Fig. 5, the user operates the display button 53C, and an onomatopoeia image in which "Bravo" is stylized is displayed on the screen 3a.

[0062] The onomatopoeia display function according to the embodiment accepts a user's operation for displaying an onomatopoeia image (first image 51) on the screen 3 a while a video is displayed on the screen 3 a. That is, the user operates the display button 53 while watching the video, thereby displaying on the screen 3 a an onomatopoeia image corresponding to the emotion felt by watching the video.

[0063] The onomatopoeia image according to the embodiment is displayed on the screen 3a displaying the video, changing according to the manner in which the user operates the display button 53. As an example, the onomatopoeia image according to the embodiment is displayed so as to extend from the first image display area 50B to the distribution video display area 50A according to the manner in which the user operates the display button 53.

[0064] In this way, while watching a video, the user can change the onomatopoeia image displayed on the screen 3a by operating the mobile terminal 3. Therefore, the user can easily display the emotions that they have felt while watching the video on the screen while watching the video.

[0065] 5, the onomatopoeia image is displayed beyond the first image display area 50B and superimposed on the distributed video display area 50A. In this way, the onomatopoeia display function according to the embodiment superimposes the onomatopoeia image beyond the first image display area 50B onto the distributed video display area 50A in accordance with the user's operation mode, which makes it easier for the user to empathize with the video they are watching, and makes watching the video more entertaining.

[0066] 5, the onomatopoeia image is superimposed on the second image display area 50C beyond the distribution video display area 50A. However, this is just one example, and the onomatopoeia image may not be superimposed on the second image display area 50C.

[0067] Next, the onomatopoeia images that change depending on the manner in which the display button 53 is operated will be described in detail.

[0068] As an example, the display mode of the first image 51 according to the embodiment changes depending on the duration of the user's operation of the display button 53. The duration of the operation of the display button 53 is, for example, the time during which the display button 53 is continuously pressed (hereinafter referred to as "long press time").

[0069] When a user presses the display button 53, the onomatopoeia display function according to the embodiment starts displaying an onomatopoeia image starting from a position near the display button 53. Displaying an onomatopoeia image starting from a position near the display button 53 means, for example, that the onomatopoeia image is displayed so that the bottom edge of the onomatopoeia image is approximately aligned with the position of the display button 53.

[0070] The onomatopoeia image is extended from the first image display area 50B toward the streaming moving image display area 50A according to the length of time the display button 53 is pressed (hereinafter referred to as "extended display").

[0071] The elongated display may be, for example, a display in which the onomatopoeia image is elongated or enlarged. Note that the elongated display may be a display in which the onomatopoeia image is elongated while changing the form, such as the number of characters or the type of characters. For example, if the onomatopoeia image is "Bravo," the image may be elongated so that an "a" is added between "Bra" and "Bo" depending on the long press time, or the image may be elongated so that an "o" is added between "Bravo" and "-" depending on the long press time.

[0072] That is, if the user feels a stronger emotion toward the video being viewed, the user continues to press display button 53 to display the onomatopoeia image corresponding to that emotion. As a result, the onomatopoeia image is displayed extended from first image display area 50B toward distribution video display area 50A, and the onomatopoeia image is displayed on screen 3a so that it stands out more.

[0073] In this way, the onomatopoeia image included in the first image 51 changes in accordance with the manner of operation by the user, allowing the user to display his or her own reaction (emotion) to the content on the screen 3a in a more entertaining manner.

[0074] In addition, the first image 51 is displayed so as to extend from the first image display area 50B toward the distributed video display area 50A depending on the user's operation, allowing the user to display their own reaction (emotions) to the content on the screen 3a in a more entertaining manner.

[0075] Note that the display of the onomatopoeia image on the screen 3a is stopped when the user finishes operating (long press) the display button 53. To stop the display of the onomatopoeia image, for example, the displayed onomatopoeia image may disappear from the screen 3a instantly, or the expanded onomatopoeia image may shrink toward the display button 53 and disappear from the screen 3a.

[0076] 5, the periphery of the display button 53C operated by the user to display the onomatopoeia image is highlighted, which allows the user to easily recognize the display button 53C that the user is operating, and also makes the onomatopoeia display more entertaining.

[0077] Furthermore, in the embodiment, the first image 51 other than the onomatopoeia image, such as a character image, is displayed on the screen 3a without changing, but this is not limited to this, and the first image 51 other than the onomatopoeia image may also be displayed on the screen 3a displaying the video while changing depending on the manner in which the user operates the display button 53.

[0078] [2-3-3. Display of the second image] The display mode of the second image 52 will be described in detail with reference to Fig. 6. As described above, the second image 52 is a reduced image of the first image 51 displayed on another mobile terminal 3 by an operation of another user.

[0079] The onomatopoeia display function according to the embodiment displays the second image 52 in the second image display area 50C with almost no time delay from the timing when another user displays the first image 51 on another mobile terminal 3. This allows the user to know the reactions of other users who are watching the same video while watching the video.

[0080] Furthermore, if second image 52 is displayed superimposed on distributed moving image display area 50A like first image 51, the user may find second image 52 to be a nuisance, interfering with viewing the moving image. Therefore, second image 52 is not displayed superimposed on distributed moving image display area 50A beyond second image display area 50C, regardless of the operation mode of other users. This allows the user to check other users' reactions to the moving image they are viewing without feeling annoyed.

[0081] As mentioned above, the first image 51 can also be superimposed on the distribution video display area 50A, but this is done at the user's own discretion, so the user does not feel bothered by the superimposition of the first image 51 on the distribution video display area 50A.

[0082] Furthermore, the screen 3a is set so that the first image display area 50B and the second image display area 50C are different areas, so that the first image 51 displayed by the user and the second image 52 displayed by another user are not displayed together. This allows the user to easily distinguish between the first image 51 displayed by the user and the second image 52 displayed by another user.

[0083] Further, the other users according to the embodiment are, for example, other users who are registered in the same group as the user. That is, when a user is watching a video without being registered in a group, the second image 52 is not displayed in the second image display area 50C. However, this is not limited to this, and the second images 52 generated by the operations of all other users watching the video may be displayed in the second image display area 50C regardless of whether or not the user is registered in a group.

[0084] Furthermore, the second image display area 50C according to the embodiment displays a plurality of second images 52 created by operations by a plurality of other users. In the second image display area 50C according to the embodiment, as an example, the position at which the second image 52 is displayed is not determined for each of the other users. That is, the second image display area 50C displays a plurality of second images 52 created by operations by the plurality of other users in a mixed manner at random positions. Furthermore, the second images 52 may be displayed while moving up, down, left, or right within the second image display area 50C. However, this is not a limitation, and the position at which the second image 52 is displayed within the second image display area 50C may be determined for each of the other users.

[0085] Additionally, an identification image (hereinafter referred to as a "user icon") 54 indicating the other user who caused the first image 51 designated as the second image 52 to be displayed on the user's mobile terminal 3 is displayed adjacent to the second image 52. This allows the user to recognize which user created the second image 52 displayed in the second image display area 50C. Note that, in the example of FIG. 6, the user icon 54 is a circle with a single alphabet character displayed in the center, but this is just one example, and the user icon 54 may be an image set for each user or the user's initials, and the design of the user icon 54 is not limited.

[0086] In addition, the second image 52 shown in FIG. 6 displays the onomatopoeia image horizontally, but this is set in accordance with the size (aspect ratio) of the second image display area 50C, and the second image 52 may also display the onomatopoeia image vertically.

[0087] Furthermore, although the second image 52 is not superimposed on the streaming moving image display area 50A, it may be displayed while changing (e.g., expanding) in the second image display area 50C in response to the operation mode by another user. Note that the second image 52 may be superimposed from the second image display area 50C onto the first image display area 50B as long as it is not superimposed on the streaming moving image display area 50A.

[0088] [2-3-4. Combined image display] The mobile terminal 3 in the embodiment has a combination image display function that displays on the screen 3a a combination image corresponding to a combination of an operation by a user to display a first image 51 and an operation by another user to display a first image 51.

[0089] This allows the mobile terminal 3 to display the user's and other users' reactions to the video on the screen in a more entertaining manner. Also, because images that would not be displayed by an operation by one user are displayed in combination with operations by other users, the mobile terminal 3 can encourage more users to view the content.

[0090] Furthermore, the combined image is displayed superimposed on the video displayed on the screen 3a. That is, the combined image is displayed superimposed not only on the first image display area 50B but also on the distributed video display area 50A and the second image display area 50C. This allows the user to easily recognize that the combined image is being displayed, and makes watching the video even more entertaining.

[0091] When the combined image is displayed on the screen 3a, it is displayed so that the user can recognize that it is a combined image. For example, the user IDs of other users may be displayed on the screen 3a together with the combined image so that the user can recognize that other users have performed display operations at the same time.

[0092] FIG. 7 is a timing chart showing a case where a combined image is displayed on the screen 3a.

[0093] The combination image display function according to the embodiment displays a combination image on the screen 3a when the timing of a user's operation and another user's operation is within a predetermined time (hereinafter referred to as the "combination period"). In other words, when multiple users perform a display operation on the first image 51 within the combination period, it is interpreted as the users' reactions to viewing the same scene. Therefore, the combination image display function displays a combination image based on the reactions of multiple users according to the scenes in the video, which further enhances the entertainment value for users when viewing the video.

[0094] The start of the combination period is the timing when a certain user starts a display operation of the first image 51. That is, as shown in Fig. 7(A), when user B performs a display operation within the combination period starting from the timing when user A starts a display operation, a combined image is displayed on the screens 3a of the mobile terminals 3 of users A and B.

[0095] The display period of the combined image is, for example, until the display operation by the user is completed. In the example of Fig. 7(A), the combined image is displayed on the screen 3a of the mobile terminal 3 of the user A and the user B from the start of the display operation by the user B until the end of the display operation.

[0096] The example of Fig. 7(B) shows a case where a display operation by user C is performed within the combination period after the display operation by user A, which is the start timing of the combination period, has ended. In the example of Fig. 7(B), a combined image is displayed on the screens 3a of the mobile terminals 3 of users A, B, and C from the start of the display operation by user B until the end of the display operation by user C.

[0097] 7 is an example, and is not limited thereto. For example, there may be no concept of a combination period, and the combination image may be displayed on the screen 3a at the timing when a display operation by a user and a display operation by another user overlap. Also, a display time may be predetermined for each combination image, and the display of the combination image may end at the timing when the display time has elapsed.

[0098] Furthermore, the combined image according to the embodiment is displayed on the screen 3a in place of the first image 51. In the example of FIG. 7(A), first, the first image 51 according to the display operation of user A is displayed on the screen 3a of user A's mobile terminal 3, but as user B's display operation is started, the display of the first image 51 ends and the combined image is displayed on the screen 3a of user A's mobile terminal 3. On the other hand, on user B's mobile terminal 3, the first image 51 that user B intended to display is not displayed on the screen 3a, but the combined image is displayed. In this way, by displaying the combined image on the screen 3a in place of the first image 51, an image different from the first image 51 that the user intended to display is displayed on the screen 3a, which further enhances the entertainment value for users watching the video.

[0099] Furthermore, the combined image according to the embodiment is displayed on the screen 3a when the first image 51 selected by the user and the first image 51 selected by another user are preset images. This may result in a combined image that the user himself / herself does not expect being displayed in accordance with the image selection made by the user himself / herself, thereby further increasing the entertainment value of watching the video.

[0100] For example, the combined image is displayed on the screen 3a when the first image 51 selected by the user and the first image 51 selected by another user are the same. In other words, when two users watching a video select the same first image 51, it is highly likely that the users are experiencing similar emotions. For this reason, by displaying the combined image, the user can recognize that they have empathized with the other user by watching the video. More specifically, for example, when the first image 51 selected by the user and the first image 51 selected by the other user are the same first image 51, an enlarged image of the first image 51 is displayed on the screen 3a as the combined image. However, this is not limiting, and a special image that is not displayed by operating the display button 53 may also be displayed as the combined image.

[0101] Furthermore, even if the first image 51 selected by the user and the first image 51 selected by another user are different first images 51, if the combination is a preset combination, a combination image may be displayed. In this case, for example, a special image that is not displayed by operating the display button 53 is displayed as the combination image.

[0102] In the combined image display function according to the embodiment, a second image based on the combined image displayed on the screen 3a of another user is not displayed in the second image display area 50C of the portable terminal 3 of the user on whom the combined image is not displayed. However, the present invention is not limited to this, and a second image based on the combined image displayed on the screen 3a of another user may be displayed in the second image display area 50C of the portable terminal 3 of the user on whom the combined image is not displayed.

[0103] [2-4. Replacement process] The server 4 according to the embodiment has a replacement processing function that identifies, as an image group, two second images 52 that are close in display timing from among the second images, and repeats a unit process of replacing the image group with one new second image 52 until no image groups are identified. This reduces the number of second images 52 displayed during playback of a video on the mobile terminal 3, and can suppress sudden increases in the load on the mobile terminal 3.

[0104] FIG. 8 is a timing chart showing a case where the second image is replaced.

[0105] As a premise, the timeline of the video is divided into multiple sections. The length of each section is, for example, the same as the length of the combined period. For example, the replacement processing function according to the embodiment identifies, as an image group, two second images 52 that share a common subject and object from among a plurality of second images 52 each included in the same section whose display period starts, and then invalidates each of the two second images 52 included in the image group while associating one new second image 52 with the timeline of the video. This unit processing is repeated until no more image groups are identified.

[0106] The second image 52 is assigned attributes (second image display information) that can identify the subject (the user ID of the user who added the second image to the video and the user group to which the user belongs), the object (the type of the second image and the element group to which that type belongs), the display period, the number of times it is combined, the status, etc. The type is determined by the object and the number of times it is combined. The design of the second image 52 is determined by the type. It is preferable that the appearance of the design reflects the number of times it is combined, for example by displaying a number indicating the number of times it is combined. As shown in FIG. 8(A), the start of the display period of second image A and the start of the display period of second image B may both be within the nth interval (n is an appropriate natural number) on the timeline. It is assumed that the type of second image A and the object of second image B are the same. In this case, the replacement processing function identifies second image A and second image B as an image group, invalidates the status of second image A and second image B, and associates a new second image C with the video timeline.

[0107] The second image 52 after replacement (second image C) inherits the additional attributes of the second image 52 before replacement (second image A and second image B). The second image 52 after replacement, for example, inherits the subject and object as they are. The number of times the second image 52 after replacement is combined is, for example, the total number of times each second image 52 before replacement was combined. The display period of the second image 52 after replacement is, for example, the period from the earlier start time to the later end time. As shown in FIG. 8(A), the display period of second image C corresponds to the period from the start time of second image A to the end time of second image B.

[0108] The second image 52 before replacement is invalidated and its association with the video is released. Meanwhile, the second image 52 after replacement forms an image group with other second images 52 and can be replaced with a new second image 52. As shown in FIG. 8(B), the start of the display period of second image C and the start of the display period of second image D are both included in the nth interval (n is an appropriate natural number) on the timeline. Assume that the user who added second image C and the user who added second image D belong to the same user group, and the type of second image C and the type of second image D are the same. In this case, the replacement processing function identifies second image C and second image D as an image group, invalidates the status of second image C and second image D, and associates one new second image E with the video timeline. The display period of second image E is the period from the start of second image C to the end of second image D.

[0109] [2-5. Functional Blocks] Next, the combined image display and replacement process according to the embodiment will be described using a functional block diagram. Note that the function related to the combined image display in the mobile terminal 3 is included in the onomatopoeia display function, and the function related to the combined image display and replacement process in the server 4 is included in the video distribution function.

[0110] [2-5-1. Functional block related to server video distribution function] FIG. 9 is a functional block diagram of the video distribution function of the server 4 according to the embodiment.

[0111] The CPU 20 of the server 4 includes a group registration unit 60, a video distribution unit 61, a first image display information reception control unit 62, a second image display information generation unit 63, a second image display information transmission control unit 64, a combination image information generation unit 65, a combination image information transmission control unit 66, and a replacement processing unit 67. Each function of the CPU 20 is realized by a program stored in the HDD 26.

[0112] The group registration unit 60 accepts registration of a group 5 made up of a plurality of mobile terminals 3. Note that the registration of the group 5 is performed by registering a user ID, for example.

[0113] The video distribution unit 61 distributes video to the mobile terminal 3 in response to a request from the mobile terminal 3.

[0114] The first image display information reception control unit 62 accepts the first image display information received from the mobile terminal 3 via the external interface 32, and stores the first image display information in the HDD 26 in association with the corresponding moving image.

[0115] The first image display information is information relating to the first image 51 displayed on the screen 3a of the mobile terminal 3, and is transmitted from the mobile terminal 3 to the server 4 at the time the first image 51 is displayed on the mobile terminal 3. The first image display information includes, for example, the type of the displayed first image 51, the user ID indicating the user who caused the first image 51 to be displayed, the manner of operation performed by the user to display the first image 51, the elapsed time from the start of the video when the first image 51 was displayed, etc.

[0116] The second image display information generating unit 63 generates second image display information based on the received first image display information, and stores the second image display information in the HDD 26 in association with the corresponding moving image.

[0117] The second image display information is information about the second image 52 to be displayed in the second image display area 50C, and includes the type of second image 52, the user ID of the user who displayed the first image 51 that is the source of the second image 52 on their own mobile terminal 3, the display period, the number of times it is synthesized, the status, etc.

[0118] The second image display information transmission control unit 64 controls the external interface 32 to transmit the second image display information to the mobile terminal 3. The destination of the second image display information is another mobile terminal 3 different from the mobile terminal 3 that transmitted the first image display information.

[0119] The combination image information generation unit 65 performs combination determination based on the received first image display information. Then, the combination image information generation unit 65 generates combination image information indicating a combination image according to the result of the combination determination, and stores the combination image information in the HDD 26.

[0120] The combination image information is information about the combination image to be displayed on the screen 3a, and includes the type of combination image, the user IDs of multiple users who performed display operations at the same time, and the like.

[0121] In addition, the combination determination is performed for each first image display information, and it is determined whether the display operation of the first image 51 indicated by the first image display information is within a combination period based on the start timing of the display operation of the first image 51 by another user.

[0122] The combination image information transmission control unit 66 controls the external interface 32 to transmit the combination image information to the mobile terminal 3. The mobile terminals 3 to which the combination image information is transmitted are the multiple mobile terminals 3 corresponding to the user IDs included in the combination image information.

[0123] The substitution processing unit 67 executes the substitution processing.

[0124] [2-5-2. Functional block for displaying onomatopoeia on mobile devices] FIG. 10 is a functional block diagram relating to onomatopoeia display of the mobile terminal 3 according to the embodiment.

[0125] The main control unit 40 included in the mobile terminal 3 includes an image display control unit 70, a display area setting unit 71, a first image display operation receiving unit 72, an operation state determination unit 73, a first image display information generation unit 74, a first image display information transmission control unit 75, a second image display information reception control unit 76, and a combination image information reception control unit 77. Each function of the main control unit 40 is realized by a program stored in the auxiliary storage unit 44.

[0126] The image display control unit 70 controls the touch panel display 3a to display an image based on the image data. The image display control unit 70 according to the embodiment displays a video distributed from the server 4 in the distributed video display area 50A, displays a first image 51 in the first image display area 50B in response to a user operation, and displays a second image 52 based on the first image 51 displayed on another mobile terminal 3 by another user in the second image display area 50C. Furthermore, the image display control unit 70 according to the embodiment displays the first image 51 beyond the first image display area 50B so as to be superimposed on the distributed video display area 50A in response to a user operation mode.

[0127] The display area setting unit 71 sets the positions and sizes (length and width) of the distributed moving image display area 50A, the first image display area 50B, and the second image display area 50C as the display mode of the screen 3a when a moving image is distributed from the server 4. Note that these positions and sizes are predetermined, but may also be set arbitrarily by the user.

[0128] The first image display operation accepting unit 72 accepts an operation of the display button 53 by the user while the screen 3a is displaying a moving image. That is, the first image display operation accepting unit 72 recognizes the display button 53 operated by the user among the plurality of display buttons 53, and causes the first image 51 corresponding to the display button 53 to be displayed on the screen 3a via the image display control unit 70. The first image display operation accepting unit 72 displays the first image 51 on the screen 3a such that the display mode of the first image 51 changes depending on the determination result by the operation state determination unit 73.

[0129] The operation state determination unit 73 determines the operation state of the user's display button 53. The operation state here refers to an operation mode including, for example, the time the display button 53 is pressed and held.

[0130] The first image display information generating unit 74 generates first image display information according to the first image 51 displayed on the screen 3a in response to a user operation.

[0131] The first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

[0132] The second image display information reception control unit 76 outputs the second image display information received from the server 4 via the communication unit 46 to the image display control unit 70.

[0133] The combination image information reception control unit 77 outputs the combination image information received from the server 4 via the communication unit 46 to the image display control unit 70. As a result, the image display control unit 70 displays the combination image on the screen 3a displaying the video. Note that the image display control unit 70 according to the embodiment displays the combination image on the screen 3a instead of the first image 51 in response to the user's operation of the display button 53.

[0134] [2-6. Processing flow] Next, the flow of the onomatopoeia display process according to the embodiment in the mobile terminal 3 and the server 4 will be described using a flowchart.

[0135] [2-6-1. Flow for displaying onomatopoeia on mobile devices] 11 is a flowchart showing the flow of onomatopoeia display processing executed by the main control unit 40 included in the mobile terminal 3. A program for executing the onomatopoeia display processing is stored in advance in a predetermined area of ​​the auxiliary storage unit 44. As an example, the onomatopoeia display processing starts when the mobile terminal 3 starts playing a video distributed from the server 4, and ends when the playback of the video ends.

[0136] In step S100, occurrence of a trigger is detected. The trigger in this embodiment is either a first trigger (acceptance of a user's operation of the display button 53) or a second trigger (reception of second image display information). The mobile terminal 3 in this embodiment updates the display content on the screen every time occurrence of a trigger is detected. The first trigger and the second trigger may occur in parallel. Therefore, while processing is being executed in response to the occurrence of one trigger, processing is sometimes executed in response to the occurrence of the other trigger. Furthermore, multiple second triggers may occur in parallel. Therefore, while processing is being executed in response to the occurrence of one second trigger, processing is sometimes executed in response to the occurrence of another second trigger.

[0137] In step S102, if the detected trigger is the first trigger, the process proceeds to step S106, and if the detected trigger is the second trigger, the process proceeds to step S104.

[0138] In step S104, the image display control unit 70 displays the second image 52 indicated by the second image display information in the second image display area 50C, and the process ends.

[0139] In step S106, the image display control unit 70 displays the first image 51 corresponding to the display button 53 operated by the user in the first image display area 50B. The first image 51 may be stored in the mobile terminal 3 in advance, or may be downloaded from the server 4 each time the display button 53 is operated by the user.

[0140] In step S108, the first image display information generating unit 74 generates first image display information corresponding to the first image 51 displayed in the first image display area 50B.

[0141] In step S110, the first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

[0142] In step S112, the operation state determination unit 73 determines whether or not the user has finished operating the display button 53. If the determination is affirmative, the process proceeds to step S120, and if the determination is negative, the process proceeds to step S114.

[0143] In step S114, the combined image information reception control unit 77 determines whether or not combined image information has been received via the communication unit 46, and if the determination is affirmative, the process proceeds to step S118, and if the determination is negative, the process proceeds to step S116.

[0144] In step S116, the image display control unit 70 displays the first image 51 (onomatopoeia image) in an expanded state until the user finishes operating the display button 53, and then the process returns to step S112.

[0145] In step S118, the image display control unit 70 displays the combination image indicated by the combination image information on the screen 3a, and the process returns to step S112.

[0146] In step S120, the image display control unit 70 ends the display of the first image 51, and the process ends.

[0147] 11, the first image display information is generated together with the display of the first image 51 and transmitted to the server 4, but this is not limiting, and the first image display information may be transmitted to the server 4 after the display of the first image 51 has ended. Furthermore, the first image display information transmission control unit 75 may transmit to the server 4 first image display information indicating the start of display of the first image 51 at the timing when the display of the first image 51 starts, and may transmit to the server 4 first image display information indicating the end of display of the second image 52 at the timing when the display of the first image 51 ends. This allows the server 4 to recognize that the first image 51 (onomatopoeia image) is displayed in an elongated manner during the interval between receiving the first image display information indicating the start of display of the first image 51 and the first image display information indicating the end of display.

[0148] [2-6-2. Flow for displaying onomatopoeia on the server] 12 is a flowchart showing the flow of the onomatopoeia display process executed by the CPU 20 of the server 4. A program for executing the onomatopoeia display process is pre-stored in a predetermined area of ​​the HDD 26. As an example, the onomatopoeia display process starts when the server 4 is distributing a video to the mobile terminal 3, and ends when the video distribution ends.

[0149] In step S200, the first image display information reception control unit 62 determines whether or not the first image display information has been received from the mobile terminal 3. If the determination is affirmative, the process proceeds to step S202, and if the determination is negative, the process enters a waiting state.

[0150] In step S202, the first image display information reception control unit 62 stores the received first image display information in the HDD .

[0151] In step S204, the combination image information generation unit 65 performs a combination determination, and if the determination is affirmative, the process proceeds to step S208, and if the determination is negative, the process proceeds to step S206. That is, if the display operation of the first image 51 to be combined is performed by multiple users within the combination period, the determination is affirmative, and the process proceeds to step S208. On the other hand, if the display operation of the first image 51 to be combined is not performed by multiple users within the combination period, the determination is negative, and the process proceeds to step S206.

[0152] In step S206, the second image display information generating unit 63 generates second image display information based on the received first image display information, and stores the second image display information in the HDD .

[0153] In step S208, the combination image information generating unit 65 generates combination image information indicating a combination image according to the display operation of the first images 51 to be combined.

[0154] In step S210, the combination image information transmission control unit 66 controls the external interface 32 to transmit the combination image information, and the process returns to step S200. The destinations of the combination image information are the multiple mobile terminals 3 that have transmitted the first image display information to be combined, i.e., the mobile terminals 3 that correspond to the multiple user IDs included in the combination image information.

[0155] Fig. 13 is a flowchart showing the flow of onomatopoeia display processing executed by the CPU 20 of the server 4. Each time the end of a section on the timeline of the video is reached, the flow shown in Fig. 13 targets the second image display information whose display period starts within that section.

[0156] In step S300, the replacement processing unit 67 makes a determination, and if the determination is affirmative, the process proceeds to step S302, and if the determination is negative, the process proceeds to step S304. That is, if there is second image display information having commonality between the subject and the object, the determination is affirmative and the process proceeds to step S302. On the other hand, if there is no second image display information having commonality between the subject and the object, the determination is negative and the process proceeds to step S304.

[0157] In step S302, the replacement processing unit 67 performs replacement processing. In a unit process of the replacement processing, the replacement processing unit 67 identifies two pieces of second image display information that have a common subject and object as an image group, invalidates the status of each of the two pieces of second image display information included in the image group, and generates one new piece of second image display information. In step S302, this unit process is repeated until there are no more pieces of second image display information that have a common subject and object.

[0158] In step S304, the second image display information transmission control unit 64 controls the external interface 32 to transmit all second image display information included in the target section to a mobile terminal 3 different from the mobile terminal 3 that transmitted the first image display information. Therefore, the display period of the second image 52 on the mobile terminal 3 based on the second image display information will be delayed from the display period associated with the video.

[0159] As described above, in the mobile terminal 3 according to the embodiment, operation information of an additional operation performed on the first mobile terminal 3 by a first user watching a video displayed on the screen 3a of the first mobile terminal 3 to add a first image 51 to the video, and operation information of an additional operation performed on the second mobile terminal 3 by a second user watching the video displayed on the screen 3a of the second mobile terminal 3 to add a first image 51 to the video, are provided from each mobile terminal 3 to the server 4. To improve entertainment value, the server 4 displays a combination image based on a combination of an additional operation by the first user and an additional operation by the second user on the screen 3a of the first mobile terminal 3 and the screen 3a of the second mobile terminal 3, respectively. In this way, an image that would not be displayed by an operation by one user is displayed in combination with an operation by another user, and the mobile terminal 3 can encourage more users to watch videos.

[0160] Furthermore, the server 4 according to the embodiment replaces, from among second images 52 associated with a video and to be displayed on the mobile terminal 3 of a user watching the video while the video is being played, two or more second images 52 that are close in display timing and have commonality in subject and object with one new second image 52. This reduces the number of second images 52 to be displayed on the mobile terminal 3, allowing the server 4 to suppress a sudden increase in load on the mobile terminal 3.

[0161] [3. Modifications] The technical scope of the present invention is not limited to the scope described in the above embodiments. Various modifications and improvements can be made to the above embodiments without departing from the spirit of the invention, and such modifications and improvements are also included in the technical scope of the present invention.

[0162] Furthermore, the various processing flows described in the above embodiments are also examples, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the present invention.

[0163] [3-1. Variation 1 (Sharing images without going through a server)] In the above embodiment, the server 4 receives first image display information from the mobile terminal 3, transmits second image display information generated based on the first image display information to another mobile terminal 3, and the other mobile terminal 3 displays the second image 52 indicated by the second image display information in the second image display area 50C, but the present invention is not limited to this.

[0164] For example, the mobile terminal 3 may display the second image 52 in the second image display area 50C by transmitting and receiving first image display information among multiple mobile terminals 3 without going through the server 4. That is, when the mobile terminal 3 receives first image display information from another mobile terminal 3, it reads information about the first image 51 displayed on the other mobile terminal 3 from the first image display information, and displays the second image 52 based on that information in the second image display area 50C.

[0165] [3-2. Variation 2 (Watching the video alone)] The above embodiment has been described as an example in which an image expressing the emotion of a video is shared among multiple users who are simultaneously watching the video. In contrast, an image expressing the emotion of a user who first watched a video can be shared with a user who watches the video later. This allows a combined image to be displayed according to a combination of an additional operation by a first user who watched the video earlier and an additional operation by a second user who watches the video later.

[0166] In the following, we consider a case where a first user using a first mobile terminal 3 (first device) watches a video (viewing target) after a second user using a second mobile terminal 3 (second device) watches the video (FIG. 13(A)). In this case, the additional operation by the first user and the additional operation by the second user may not be in a proximate relationship in real time. However, the two are in a proximate relationship on the progression axis (timeline) of the video (FIG. 13(B)). The first user and the second user may be the same person and another person, or may be the same person in the past and the same person in the present.

[0167] In the second modification, operation information of an addition operation (second operation) performed on the second mobile terminal 3a by a second user who is watching a video displayed on the screen 3a (second screen) of the second mobile terminal 3 to add a first image (second element) to the video is transmitted from the second mobile terminal 3a to the server 4. (second instruction means, shared information processing device). The server 4 transmits in advance to the first mobile terminal 3 used by the first user who intends to watch the same video, second image display information (second instruction) for displaying the second image (second element) at a timing according to the additional operation by the second user.

[0168] To improve entertainment value, when the first mobile terminal 3 receives an additional operation (first operation) performed on the first mobile terminal 3a by a first user who is watching a video displayed on the screen 3a (first screen) of the first mobile terminal 3 to add a first image (first element) to the video, the first mobile terminal 3 displays a combined image (third element) based on a combination of the received additional operation (first operation) and an additional operation (second operation) related to the second image display information on the screen 3a of the first mobile terminal 3. Note that in the second modification example, since the timing of the second operation performed on the second mobile terminal 3 is known in advance, it is also possible to display a combined image (third element) based on a combination of the received additional operation (first operation) and an additional operation (second operation) related to the second image display information on the screen 3a of the first mobile terminal 3 without delay after the first operation is received on the first mobile terminal 3.

[0169] [3-3. Modification 3 (Do not display the first image when a combined image is displayed)] In the above embodiment, the first image is also displayed when the combined image is displayed. However, it is also possible to prevent the first image from being displayed when the combined image is displayed.

[0170] Specifically, the image display control unit 70 (second control means) of the mobile terminal 3 waits to receive combination image information until a predetermined period (combination period) has elapsed since the first image display operation receiving unit 72 (first receiving means) received the additional operation. If the image display control unit 70 receives the combination image information within the predetermined period, it displays the combination image (third element) based on the received combination image information, and if the image display control unit 70 does not receive the combination image information within the predetermined period, it displays the first image (first element) based on the received additional operation.

[0171] [3-4. Variation 4 (displaying the second image even when a combined image is displayed)] In the above embodiment, the second image is not displayed when the combined image is displayed. However, the second image may be displayed even when the combined image is displayed. Specifically, when the server 4 receives the first image information, it may generate second image display information and transmit it to the mobile terminal 3 regardless of whether a display operation to combine the first image information has been performed.

[0172] [3-5. Variation 5 (Limiting the activation of replacement processing)] In the above embodiment, the unit process is repeated for each predetermined period (combination period), and multiple pieces of second image display information to be replaced are replaced with one piece of second image display information. However, in a situation where a sudden increase in load on the mobile terminal 3 is unlikely to occur, the replacement process may be omitted.

[0173] Specifically, when the number of elements in a proximity element group including multiple pieces of second image display information (shared elements) whose display timings are close to each other does not satisfy the activation condition, the replacement processing unit 67 (identification means) of the server 4 omits a unit process (replacement process) of identifying two (a predetermined number) pieces of second image display information included in the proximity element group as an image group (element group). The activation condition is, for example, that the number of elements is equal to or greater than a predetermined threshold. The activation condition may be flexibly changed depending on the processing capabilities of the mobile terminals 3. For example, it is advisable to set the predetermined threshold according to the processing capabilities of the mobile terminals 3 with the lower capabilities among the multiple mobile terminals 3 belonging to group 5. The processing capabilities may, for example, be the display processing capabilities (DSP clock frequency) or reception processing capabilities (communication line speed) of the mobile terminals 3. [Explanation of symbols]

[0174] 1. Video distribution system 2. Communication lines 3. Mobile terminal (an example of the "first device," "second device," or "information processing device") 3a Screen (Example of "1st screen" and "2nd screen") 4. Server (an example of the "first instruction means," "second instruction means," and "shared information processing device") 5 Groups (Example of "Information Processing Systems") 50A Streaming video display area 50B 1st image display area 50C 2nd image display area 51 First image 52 Second image (an example of a "shared element" or "alternative element") 53 Display button 60 Group Registration Department 61 Video Distribution Department 62 First image reception information reception control section 63 Second image display information generation section 64 Second image display information transmission control section 65 Combined image information generation unit 66 Combined image information transmission control section 67 Replacement processing unit (an example of "identification means" and "replacement means") 70 Image display control unit (an example of the "first control means," "second control means," "third control means," or "display control means") 71 Display area setting section 72 first image display operation receiving unit (an example of the “first receiving means” and the “second receiving means”) 73 Operation status determination unit 74 First image display information generation section 75 First image display information transmission control unit (an example of the "first providing means" and the "second providing means") 76 Second image display information reception control unit (an example of the “third reception means”) 77 Combined image information reception control section

Claims

1. one or more processors, receiving a first operation on a first device on which a viewing target is displayed on a screen; receiving a second operation on a second device on which the viewing target is displayed on a screen; When the second operation is performed within a predetermined period based on the time when the first operation is received, display elements based on the first operation and the second operation are added to the viewing target and displayed. Information processing device.

2. when an option identical to an option selected by the first operation is selected from a plurality of options by the second operation, the one or more processors display the display element obtained by processing the identical option; The information processing device according to claim 1 .

3. one or more processors receiving a first operation on a first device on which a viewing target is displayed on a screen; one or more processors receive a second operation on a second device on which the viewing target is displayed on a screen; When the second operation is performed within a predetermined period based on the time when the first operation is received, one or more processors add display elements based on the first operation and the second operation to the viewing target and display the display elements. Information processing methods.

4. one or more processors, a function of receiving a first operation on a first device on which a viewing target is displayed on a screen; a function of receiving a second operation on a second device on which the viewing target is displayed on a screen; a function of adding and displaying a display element based on the first operation and the second operation to the viewing target when the second operation is performed within a predetermined period based on the time of receiving the first operation; A program that executes the following.

5. A server, a first terminal, and a second terminal, receiving a first operation on a first terminal on which a viewing target is displayed on a screen; receiving a second operation on a second terminal on which the viewing target is displayed on a screen; the server, when the second operation is performed within a predetermined period based on the time of receiving the first operation, adds a display element based on the first operation and the second operation to the viewing target and displays the display element; system.

Images