Shared information processing device and control program

Abstract

To prevent unexpected workload from increasing in a terminal for displaying elements, concerning a system where the elements are shared among a plurality of users viewing each of common subjects to be viewed.SOLUTION: A shared information processing device (e.g., a server 4) representing one embodiment of the present invention comprises: identifying means (e.g., a replacement processing unit 67) for, among shared elements in which timings to be displayed in a user terminal (e.g., a mobile terminal 3) viewing subjects to be viewed are associated with the subjects to be viewed, identifying the shared elements in a prescribed count as an element group based on the timings; and replacement means (e.g., a replacement processing unit 67) for replacing the element group identified by the identifying means with substitute elements fewer than the prescribed count.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to a mechanism for sharing elements among a plurality of users who respectively view a common viewing target.

Background Art

[0002] There is known a communication system that shares comments input according to reproduced content among a plurality of users. For example, in the communication system described in Patent Document 1, a user terminal that has acquired from a comment server comments to be displayed together with the content before the start of reproduction of the content excludes old comments from display targets so that the number of displayed comments in each of a plurality of time segments in which the timeline is divided becomes a predetermined number or less.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the communication system described in Patent Document 1, processing for leveling the temporal display number of comments displayed together with the content is executed before the start of reproduction of the content. Therefore, it is impossible to cope with a sudden increase in load caused by comments added during the reproduction of the content.

[0005] The problem to be solved by the present invention is to suppress a sudden increase in load on a terminal that displays an element in a mechanism for sharing elements among a plurality of users who respectively view a common viewing target.

Means for Solving the Problems

[0006] To solve the above problems, the present invention identifies a predetermined number of shared elements as a group of elements from among the shared elements associated with the viewing target, based on the timing at which the viewing target is displayed on the terminal of the user viewing the viewing target, and replaces the group of elements with a number of substitute elements that is less than the predetermined number.

[0007] A shared information processing device, in one embodiment of the present invention, comprises: an identification means for identifying a predetermined number of shared element groups based on the timing at which a viewing target is displayed on the terminal of a user viewing the viewing target; and a replacement means for replacing the element groups identified by the identification means with a number of substitute elements less than the predetermined number. [Effects of the Invention]

[0008] According to the present invention, sudden increases in load on terminals displaying shared elements can be suppressed. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the configuration of a video streaming system. (Example) [Figure 2] Block diagram showing the electrical configuration of the server. (Example) [Figure 3] Block diagram showing the electrical configuration of a mobile device. (Example) [Figure 4] This diagram shows the display configuration of a mobile device screen. (Example) [Figure 5] This figure shows a screen displaying onomatopoeic images. (Example) [Figure 6] This is a diagram showing the screen displaying the second image. (Example) [Figure 7] This is a timing chart showing when a combined image is displayed. (Example) [Figure 8] This is a timing chart showing the case where the second image is replaced. (Example) [Figure 9] This is a functional block diagram related to video streaming on the server. (Example) [Figure 10] This is a functional block diagram related to onomatopoeia display on mobile devices. (Example) [Figure 11] This is a flowchart illustrating the flow of onomatopoeia display processing performed on a mobile device. (Example) [Figure 12] This is a flowchart showing the flow of onomatopoeia display processing executed on the server. (Example) [Figure 13] This is a flowchart showing the flow of onomatopoeia display processing executed on the server. (Example) [Figure 14] This is a timing chart showing when combined images are displayed. (Variation 2) [Modes for carrying out the invention]

[0010] [1. Embodiments] [1-1. Main Features] This embodiment relates to a mechanism for sharing elements among multiple users who are each viewing a common viewing target. In this embodiment, in order to suppress a sudden increase in load on the terminal that displays the shared elements, the timing at which the viewing target is displayed on the terminal of the user viewing the viewing target identifies a predetermined number of the shared elements associated with the viewing target as a group of elements based on that timing, and replaces the group of elements with a number of substitute elements that is less than the predetermined number.

[0011] "Viewing object" refers to an audiovisual object that a user can perceive through at least one of their senses: sight or hearing. An audiovisual object is a concept that encompasses images, sounds, and combinations thereof. Audiovisual objects include both objects that are not signaled and objects that are reproduced based on signals. The signals of reproduced objects can be, for example, recorded on a recording medium or carried by a carrier wave. Reproduced objects are, for example, reproduced as at least one of images or sounds by a device with playback capabilities.

[0012] The "progression" of the viewing target refers to the change of the viewing target over time. The change is a concept that encompasses invariability and change. Invariability means that visual information and auditory information continue to be constant. Change means that visual information and auditory information are not constant. Change is, for example, irregular change, regular change, periodic repetition, temporary disappearance, or combinations thereof. When the viewing target is an object that is not signalized, the timing (time point) during progression can be specified by time information. When the viewing target is an object that is reproduced based on a signal, the time point during progression can be specified by time information or the elapsed time from the start point of progression (i.e., the position on the timeline).

[0013] The "viewing" of the viewing target refers to grasping the progression of the viewing target by at least either vision or hearing. When the viewing target is an object that is not signalized, the said viewing target is viewed almost simultaneously with the progression. The viewing of a non-signalized object is, for example, viewing a sports game at a stadium, viewing natural phenomena (e.g., solar eclipse, lunar eclipse, aurora, sunrise, sunset, sunset glow, rare meteorological phenomena, etc.) at an appropriate location, viewing an event (e.g., festival, street speech, etc.) at an appropriate location. When the viewing target is an object that is reproduced based on a signal, the said viewing target is viewed when it is reproduced. The viewing of a reproduced object is, for example, viewing a television broadcast, viewing a radio broadcast, viewing an online-distributed video, viewing an online-distributed music.

[0014] A user (viewer) who views a viewing target can associate a predetermined element (shared element) with the viewing target by a predetermined operation (additional operation). The predetermined operation is, for example, a pointing operation on an input element (operation element). The input element is, for example, a button displayed within a screen. The pointing operation is, for example, a combination of a touch operation on a touch panel, a positioning operation, and a click operation of a pointer displayed within the screen. When a predetermined element is associated with a viewing target, the timing for displaying the shared element during the progress of the viewing target is associated with the viewing target. The element associated with the viewing target can be displayed on the screen at the associated display timing during the progress of the viewing target.

[0015] The predetermined element is a visual element representing a user's reaction to the viewing target. The user's reaction is, for example, the user's emotion. The visual element is, for example, a still image, an animated image, a moving image, text, or a combination thereof. An image representing the user's emotion towards the viewing target may be called an emotion image. The emotion image is, for example, an image in which onomatopoeia (also referred to as mimetic words, sound words, or mimicry words) indicating the user's emotion is designed. An image in which onomatopoeia is designed may be called an onomatopoeia image. Further, the predetermined element can be shared among a plurality of users who view a common viewing target. The element that can be shared is transmitted and received online and displayed on the screen of a terminal. The terminal having a screen on which the element that can be shared is displayed may be the same device as the device that reproduces the object that is the viewing target, or may be a different device.

[0016] When identified "based on timing," a predetermined number of shared elements whose display timings are in close proximity can be identified as a group of elements. Two shared elements are said to have display timings in close proximity if the time difference between when one is displayed on the screen and when the other is displayed on the screen is less than or equal to a predetermined threshold. Three or more shared elements are said to have display timings in close proximity if two shared elements, arbitrarily selected from the three or more shared elements, have display timings in close proximity. When determining whether or not there is a proximity relationship in display timing, a single point in time between the start and end of the display of each shared element is adopted as the display timing. The points in time adopted as the display timing are, for example, the start point, the end point, and a predetermined reference point.

[0017] A "substitute element" refers to visual information associated with the viewer in place of a group of elements (the replacement target) that contains two or more shared elements. Substitute elements may be visual information common to the replacement target, or visual information whose display manner differs from the replacement target. Display manners include, for example, display time, display size, display color, presence or absence of additional visual information, and presence or absence of movement. The number of duplicate elements may be indicated to the user by changing the display manner. Specifically, it is good to change the display manner so that it becomes more flashy (richer) as the number of duplicates increases. Flashiness may include, for example, a longer display time, a larger display size, a darker display color, a larger amount of additional visual information, or greater movement. The number of duplicate elements may also be indicated to the user by changing the display manner. Specifically, it is good to add visual information indicating the number of duplicates to the substitute element. Visual information indicating the number of duplicates may be, for example, an icon that displays the number corresponding to the number of duplicates, or an icon with a number corresponding to the number of duplicates.

[0018] [1-2. Addendum] The present invention encompasses at least the embodiments described in [A] to [C] below.

[0019] [A] A shared information processing device comprising: an identification means for identifying a predetermined number of shared elements as a group of elements from among the shared elements associated with the viewing target, based on the timing at which the viewing target is displayed on the terminal of a user viewing the viewing target; and a replacement means for replacing the group of elements identified by the identification means with a number of substitute elements less than the predetermined number.

[0020] [B] A control program that causes the computer of an information processing device to implement the following functions: a specification function that identifies a predetermined number of shared elements as a group of elements from among the shared elements associated with the viewing target, based on the timing at which the viewing target is displayed on the terminal of the user viewing the viewing target; and a replacement function that replaces the group of elements identified by the specification function with a number of substitute elements less than the predetermined number. The control program may be recorded non-temporarily on a recording medium.

[0021] [C] A control method for an information processing device, comprising: an identification step of identifying a predetermined number of shared elements as a group of elements from among the shared elements associated with the viewing target, based on the timing at which the viewing target is displayed on the terminal of a user viewing the viewing target; and a replacement step of replacing the group of elements identified in the identification step with a number of substitute elements less than the predetermined number.

[0022] The following technical limitations may be added to the "information processing device" in [A] above. Similar technical limitations may also be added to the "control program" in [B] above and the "control method for the information processing device" in [C] above, respectively.

[0023] (1) The identification means identifies the group of elements based on the additional attributes attached to the shared elements, and the replacement means adds the replacement element as the shared element to the viewing target, which inherits the additional attributes attached to at least one of the multiple shared elements included in the group of elements. When identified "based on additional attributes," a predetermined number of shared elements having common additional attributes can be identified as a group of elements. Having commonalities means being identical or highly related. As a result, the replacement element can be further replaced by another replacement element.

[0024] (2) The identifying means identifies the group of elements based on the additional attribute that can identify at least one of the user group to which the user who associated the shared element with the viewing target belongs and the element group to which the shared element associated with the viewing target belongs. As a result, multiple shared elements that are likely to overlap in time are replaced with alternative elements.

[0025] Furthermore, users who can associate shared elements with viewing content can belong to at least one user group. Each user can voluntarily join any of the user groups. In addition, each user may be automatically joined to any of the user groups depending on their attributes and circumstances. Shared elements associated with viewing content can belong to at least one element group. Each shared element is pre-assigned to at least one element group.

[0026] (3) The identification means identifies the element group based on the additional attribute that can identify both the user group and the element group. As a result, multiple shared elements that are likely to overlap in time are replaced with alternative elements.

[0027] (4) The replacement means replaces the group of elements identified by the identification means with the substitute elements corresponding to the number of overlaps of the specified type if the predetermined number of shared elements included in the group of elements identified by the identification means are of the same type. This provides a visual effect to prevent the load from increasing excessively.

[0028] (5) The identifying means identifies the set only when the predetermined activation conditions are met. This ensures that the substitution with the alternative element is activated only in situations where sudden increases in load are likely to occur.

[0029] An "activation condition" refers to a condition for determining whether or not to activate a substitution process (a process that replaces a predetermined number of shared elements with a predetermined number of substitute elements). Preferably, the activation condition is set as a condition relating to the number of elements in a group to which adjacent shared elements belong. The condition relating to the number of elements is, for example, whether or not the number of elements has reached or exceeded a predetermined upper limit. The predetermined upper limit may be a fixed value or a variable value. Preferably, the variable value is set according to the processing capacity of the terminal. The processing capacity of the terminal may be the processing capacity for displaying information or the processing capacity for receiving information. For example, the processing capacity for displaying information is the clock frequency of the terminal's control processing unit. For example, the processing capacity for receiving information is the speed of the terminal's communication line.

[0030] (6) The identifying means identifies a predetermined number of shared elements included in a group of adjacent elements, unless the number of elements in the adjacent element group, which includes a plurality of shared elements whose timings are close together, does not satisfy the activation conditions. This prevents the replacement with alternative elements from being activated in situations where sudden increases in load are unlikely to occur.

[0031] (7) The identifying means identifies a predetermined number of shared elements included in the proximity element group as the element group, unless the activation conditions which vary according to the processing capacity of the terminal are not met. This allows for flexible determination of whether a situation is unlikely to occur where a sudden increase in load is likely to occur. [Examples]

[0032] [2. Examples] The following describes embodiments of the present invention with reference to the drawings. This embodiment relates to a mechanism for displaying images representing the user's (viewer's) emotions (images representing reactions) on the display screen of a video (viewing target). In this embodiment, the shared information processing device is described as a server, and the information processing devices (first device, second device) are described as mobile terminals, so-called smartphones. Note that the mobile terminal is not limited to a smartphone, but may also be a tablet terminal or a laptop computer. The information processing device may also be a desktop computer. The programs according to this embodiment are stored in the shared information processing device or the information processing device, but are not limited to these, and may also be stored in storage media such as optical discs such as CD-ROMs (Compact Disk Read Only Memory) or portable semiconductor memory such as USB (Universal Serial Bus) flash memory.

[0033] [2-1. Features] In this embodiment, operation information of an addition operation (first operation) performed on the first mobile terminal (first device) to add a first image (first element) to a video (viewing target) displayed on the screen (first screen) of the first mobile terminal (first device), and operation information of an addition operation (second operation) performed on the second mobile terminal to add a first image (second element) to a video displayed on the screen (second screen) of the second mobile terminal (second device) are provided from each mobile terminal to the server (first instruction means, shared information processing device). In order to improve entertainment value, the server displays a combination image (third element) based on the combination of the first and second operations corresponding to the operation information received on the first screen and the second screen, respectively.

[0034] Furthermore, in this embodiment, in order to suppress sudden increases in load on mobile devices, two or more (a predetermined number) second images are identified as an image group (group of elements) from among the second images (shared elements) associated with the video, based on the proximity of display timing (timing) and the commonality (additional attributes) between the subject and object, while the timing at which the video is displayed on the mobile device of the user watching the video is during playback. A replacement process is then performed in which the second images included in the image group are invalidated, and one (a number less than a predetermined number) new second images (alternative elements) that inherit the additional attributes are added to the video. The subject refers to the user who added the second image to the video, and commonality among subjects means that the subjects belong to the same user group. Each user belongs to at least one of the user groups. The object refers to the second image added to the video. This refers to the type of element, and commonality among objects means that the objects belong to the same group of elements. Each element belongs to at least one of the element groups.

[0035] [2-2. Video Streaming System] The server in the embodiment is included in the video distribution system and distributes video to mobile devices. The mobile device in the embodiment is included in the video distribution system and receives video from the server.

[0036] [2-2-1. Overall configuration of the video distribution system] Figure 1 is a schematic diagram of a video distribution system 1 according to an embodiment. The video distribution system 1 consists of a communication line 2, multiple mobile terminals 3, and a server 4.

[0037] Communication line 2 forms a computer network and is, for example, a wide-area communication line provided by an electric power company.

[0038] The mobile terminal 3 is, for example, an information processing terminal such as a smartphone, tablet, or laptop computer. The mobile terminal 3 is equipped with a touch panel display 3a for displaying images, a speaker 3b for outputting sound, a microphone 3c for receiving sound input, and a camera 3d for capturing images of subjects. To display videos distributed from the server 4, the mobile terminal 3 is fitted with, for example, dedicated application software (hereinafter referred to as the "video viewing app").

[0039] Server 4 is a shared information processing device that distributes content to multiple mobile terminals 3 via communication line 2. The content in this embodiment is, as an example, video. Videos are not particularly limited; they may include animations, dramas, or other content with a storyline, or music videos, live music videos, or various sports videos. The video in this embodiment is, as an example, animation. Furthermore, the content distributed from the server is not limited to videos; it may also be other services, such as a series of still images, as long as it is viewable by the user.

[0040] Furthermore, "viewing" as used herein means that a user perceives the content through at least one of their senses, either visually or aurally. In addition to the above, the content may also include, for example, a video without sound, or sound without images.

[0041] Multiple mobile devices 3 from which the video is streamed are registered to server 4 as group 5, for example. (Hereinafter referred to as "group registration") is performed. Server 4 simultaneously distributes the video to the group-registered mobile devices 3. That is, the multiple group-registered mobile devices 3 display the video distributed from Server 4 on their touch panel displays 3a at the same time, in other words, in sync. As a result, each user of the group-registered mobile devices 3 will watch the video at the same time.

[0042] The video may be distributed at a predetermined date and time, or at any time chosen by each of the 5 groups. If the video is distributed at any time chosen, for example, the host mobile device 3 will request the server 4 to distribute the video, and the server 4, upon receiving the distribution request, will distribute the video to the mobile devices 3 that make up the 5 group.

[0043] In Figure 1, Group 5 is shown as consisting of four mobile devices 3, but this is just an example; the number of mobile devices 3 in Group 5 is not limited as long as it is composed of multiple devices 3. Furthermore, the types of mobile devices 3 that make up Group 5 are not limited; smartphones, tablets, laptops, etc., may be mixed together.

[0044] [2-2-2. Server Configuration] Figure 2 is a block diagram showing the electrical configuration of server 4 according to the embodiment.

[0045] The server 4 according to this embodiment includes a CPU (Central Processing Unit) 20, which is the main control unit that manages the operation of the entire server 4; a ROM (Read Only Memory) 22 in which various programs and data are pre-stored; a RAM (Random Access Memory) 24 used as a work area when the CPU 20 executes various programs; and an HDD (Hard Disk Drive) 26 as a storage means for storing various programs and data.

[0046] HDD26 stores information related to the registered group 5, videos (video data) to be distributed to the mobile terminal 3, and various other information. Note that the storage means is not limited to HDD26; other storage media such as semiconductor memory like flash memory may also be used.

[0047] Furthermore, the server 4 includes an input unit 28 consisting of a keyboard and mouse, etc., which accepts input for various operations; a monitor 30 such as a liquid crystal display device that displays various images; and an external interface 32 that is connected to other information processing devices such as a mobile terminal 3 via a communication line 2 and transmits and receives various data with other information processing devices.

[0048] The CPU 20, ROM 22, RAM 24, HDD 26, operation input unit 28, monitor 30, and external interface 32 are electrically connected to each other via the system bus 34. Therefore, the CPU 20 can access the ROM 22, RAM 24, and HDD 26, understand the operation status of the operation input unit 28, display images on the monitor 30, and send and receive various types of data with other information processing devices via the external interface 32.

[0049] [2-2-3. Electrical Configuration of Mobile Devices] Figure 3 is a functional block diagram showing the electrical configuration of the mobile terminal 3. In addition to the configuration shown in Figure 1, the mobile terminal 3 includes a main control unit 40, a main memory unit 42, an auxiliary memory unit 44, a communication unit 46, and operation buttons 48.

[0050] The main control unit 40 is, for example, a CPU, microprocessor, or DSP (Digital Signal Processor), and controls the overall operation of the mobile terminal 3.

[0051] The main memory unit 42 is composed of, for example, RAM or DRAM (Dynamic Random Access Memory), and is used as a work area when the main control unit 40 executes processing based on various programs.

[0052] The auxiliary storage unit 44 is a non-volatile memory such as flash memory, and stores various data such as images and programs used for processing by the main control unit 40. The programs stored in the auxiliary storage unit 44 include, for example, the OS (Operating System) for realizing the basic functions of the mobile terminal 3, drivers for controlling various hardware, programs for realizing email, web browsing, and various other functions. In addition, the auxiliary storage unit 44 has pre-stored application software such as a video viewing application for playing videos distributed from the server 4 and displaying them on the touch panel display 3a.

[0053] The communication unit 46 is, for example, a NIC (Network Interface Controller) and has the function of connecting to the communication line 2. Alternatively, the communication unit 46 may have functions to connect to a wireless LAN (Local Area Network), a wireless WAN (Wide Area Network), short-range wireless communication such as Bluetooth®, and infrared communication, either in place of or together with the NIC.

[0054] The operation buttons 48 are located on the side of the mobile terminal 3 and include a power button for starting or stopping the mobile terminal 3, and a volume control button for the sound output by the speaker 3b.

[0055] The main control unit 40, main memory unit 42, auxiliary memory unit 44, communication unit 46, touch panel display 3a, speaker 3b, microphone 3c, camera 3d, and operation buttons 48 are electrically connected to each other via a system bus 49. Therefore, the main control unit 40 can access the main memory unit 42 and auxiliary memory unit 44, display images on the touch panel display 3a, understand the user's operation status of the touch panel display 3a and operation buttons 48, input sound to the microphone 3c, output sound from the speaker 3b, control the camera 3d, and access various communication networks and other information processing devices via the communication unit 46.

[0056] [2-3. Onomatopoeia Display] The mobile terminal 3 according to this embodiment can display videos distributed from the server 4 on the touch panel display 3a, while also displaying images (onomatopoeia images, which will be described in detail later) that respond to user operations. In the following description, the function of displaying onomatopoeia images on the touch panel display 3a is referred to as the onomatopoeia display function, and the touch panel display 3a is also referred to as the screen 3a.

[0057] The onomatopoeia display function is, for example, one of the functions of the video viewing application according to the embodiment. However, for example, it is not possible to display onomatopoeia for all videos displayed on screen 3a using the video viewing application; the videos for which onomatopoeia display is possible are predetermined.

[0058] [2-3-1. Screen Layout] Figures 4 to 6 show the display mode of screen 3a of a mobile terminal 3 running the video viewing application according to the embodiment. Screen 3a is configured with a streaming video display area 50A, a first image display area 50B, and a second image display area 50C.

[0059] The video streaming display area 50A is the area for displaying the video streamed from the server 4. The first image display area 50B is the area for displaying the first image 51 (see Figure 5) in response to user operations. The second image display area 50C is the area for displaying the second image 52 (see Figure 6) based on the first image 51 displayed on another mobile terminal 3 by another user's operation. As will be described in detail later, the second image 52 is a reduced image of the first image 51 displayed in the first image display area 50B of the other mobile terminal 3. Furthermore, the first image display area 50B and the second image display area 50C in this embodiment are, as an example, provided at positions opposite each other with the video streaming display area 50A in between.

[0060] Here, the first image 51 is an image that includes an emotional image showing the emotions of a user watching a video. The emotional image in this embodiment is, as an example, an image that depicts onomatopoeia (also called sound-imitation or mimetic words) that express the user's emotions, in other words, an onomatopoeia image. Examples of onomatopoeia images include "Awesome," "Incredible," and "Bravo." Note that stylization means, for example, that the characters representing onomatopoeia are depicted in a decorative way.

[0061] Furthermore, the first image 51 may include various other images besides onomatopoeia images, such as character images showing characters appearing in the video, dialogue images showing characters' lines, text entered by the user, and so on.

[0062] Furthermore, the first image 51 may vary depending on the scene in the video. For example, depending on whether it is a cheerful scene, a sad scene, or a scene of characters confronting each other, onomatopoeic images or character images that the user is likely to choose based on their empathy with the scene may be displayed.

[0063] Furthermore, the displayable first image 51 may differ depending on the user. For example, the types of displayable first image 51 may increase by obtaining a specific item through a service such as a game related to the distributed video. Also, for example, the types of displayable first image 51 may increase on the condition that a predetermined consideration (e.g., monetary value such as currency, virtual currency, or points) is paid.

[0064] In the first image display area 50B according to the embodiment, a number of display buttons 53A to 53D are displayed, which are operation units operated by the user and each corresponds to a different first image 51. As a result, the user can display the first image 51 in the first image display area 50B by operating the display buttons 53A to 53D while watching a video. Operation of the display buttons 53A to 53D is, for example, by touching (pressing) the display buttons 53A to 53D with an object. In the embodiment, the object is, for example, the user's finger. In the following description, if the display buttons 53A to 53D are not described separately, the alphabetical notation at the end will be omitted.

[0065] In the examples shown in Figures 4 to 6, four display buttons 53A to 53D are displayed in a horizontal row at the bottom of the first image display area 50B. Display button 53A corresponds to an onomatopoeic image depicting "Sugee" (amazing), display button 53B corresponds to an onomatopoeic image depicting "Yabai" (incredible), and display button 53C corresponds to an onomatopoeic image depicting "Bravo". Furthermore, display button 53D corresponds to a character image. The display buttons 53 are designed to allow users to intuitively recognize the corresponding onomatopoeic images and character images.

[0066] In this way, by displaying multiple display buttons 53 in the first image display area 50B, the user can easily display the first image 51 on screen 3a that best matches their reaction (emotion) to the video. Furthermore, by including an onomatopoeic image that represents the user's emotion in the first image 51, the user can display their own emotions after watching the video on screen 3a.

[0067] In the examples shown in Figures 4 to 6, four types of display buttons 53 are displayed in the first image display area 50B, but this is just an example, and the number and types of display buttons 53 are not limited to these. Furthermore, the display buttons 53 may be switched and displayed as appropriate to enable the display of different first images 51 depending on the video scene, etc.

[0068] Furthermore, the arrangement of the display buttons 53 is just one example; for example, they may be displayed together on the right or left side of the first image display area 50B, or they may be displayed in two horizontal columns at the bottom of the first image display area 50B. In addition, for example, by the user swiping left or right from the bottom of the screen 3a, new display buttons 53 for displaying different types of first images 51 may be displayed.

[0069] [2-3-2. Displaying the first image] Referring to Figure 5, the display mode of the first image 51 will be explained in detail. In the example in Figure 5, the user operates the display button 53C, and an onomatopoeic image of "Bravo" is displayed on screen 3a.

[0070] The onomatopoeia display function according to this embodiment accepts user input to display an onomatopoeia image (first image 51) on screen 3a while a video is displayed on screen 3a. That is, while watching the video, the user operates the display button 53 to display an onomatopoeia image on screen 3a that corresponds to the emotion they felt while watching the video.

[0071] The onomatopoeia image according to the embodiment is displayed on screen 3a showing the video, changing according to how the user operates the display button 53. As an example, the onomatopoeia image according to the embodiment is displayed so as to extend from the first image display area 50B to the streamed video display area 50A, depending on how the user operates the display button 53.

[0072] In this way, users can change the onomatopoeic image displayed on screen 3a by operating the mobile device 3 while watching the video. Therefore, users can easily display the emotions they experience while watching the video on the screen.

[0073] Furthermore, in the example shown in Figure 5, the onomatopoeia image is displayed superimposed on the streaming video display area 50A, extending beyond the first image display area 50B. Thus, the onomatopoeia display function according to this embodiment superimposes the onomatopoeia image onto the streaming video display area 50A, extending beyond the first image display area 50B, according to the user's operation. This makes it easier for the user to empathize with the video they are watching, thereby enhancing the entertainment value of watching the video.

[0074] In Figure 5, the onomatopoeia image is superimposed not only on the streaming video display area 50A but also on the second image display area 50C. However, this is just one example, and the onomatopoeia image may not be superimposed on the second image display area 50C.

[0075] Next, we will explain in detail the onomatopoeia images that change depending on how the display button 53 is operated.

[0076] Image 51 of the embodiment shows, as an example, a display that changes depending on the duration of the user's operation of the display button 53. The duration of operation of the display button 53 is, for example, the time the display button 53 is continuously pressed (hereinafter referred to as "long press time").

[0077] In the embodiment, the onomatopoeia display function starts displaying the onomatopoeia image from the vicinity of the display button 53 when the user presses the display button 53. Displaying the onomatopoeia image from the vicinity of the display button 53 means, for example, that the onomatopoeia image is displayed so that its lower edge approximately coincides with the position of the display button 53.

[0078] Then, the onomatopoeia image expands from the first image display area 50B toward the streamed video display area 50A (hereinafter referred to as "expanded display") depending on the length of time the display button 53 is pressed.

[0079] The stretched display is, for example, a display in which an onomatopoeic image is made elongated or enlarged. The stretched display may also involve the onomatopoeic image being stretched while changing its form, such as the number of characters or the type of characters. For example, if the onomatopoeic image is "Bravo," it may be stretched so that "a" increases between "bra" and "bo" depending on the length of the press, or it may be stretched so that "o" increases between "bravo" and "-" depending on the length of the press.

[0080] In other words, if the user feels a stronger emotion towards the video they are watching, they continue to press the display button 53 to display an onomatopoeic image corresponding to that emotion. As a result, the onomatopoeic image is extended from the first image display area 50B toward the streaming video display area 50A, and is displayed on screen 3a in a more prominent manner.

[0081] In this way, the onomatopoeia images included in the first image 51 change according to the user's actions, allowing the user to display their own reactions (emotions) to the content on screen 3a in a more entertaining manner.

[0082] Furthermore, the first image 51 is displayed so as to extend from the first image display area 50B toward the streamed video display area 50A, depending on the user's operation, allowing the user to display their reaction (emotions) to the content on screen 3a in a more entertaining way.

[0083] When the user finishes operating the display button 53 (by pressing and holding it), the display of the onomatopoeia image on screen 3a will stop. The display of the onomatopoeia image may stop in any way, for example, by the displayed onomatopoeia image disappearing from screen 3a instantaneously, or by the enlarged onomatopoeia image shrinking towards the display button 53 and disappearing from screen 3a.

[0084] Furthermore, in the example shown in Figure 5, the area around the display button 53C being operated by the user is highlighted in order to display the onomatopoeia image. This allows the user to easily recognize the display button 53C they are operating and enhances the entertainment value of the onomatopoeia display.

[0085] Furthermore, while character images and the like, which are first images 51 other than the onomatopoeia images in the embodiment, are displayed on screen 3a without changing, the invention is not limited to this, and first images 51 other than the onomatopoeia images may also be displayed on screen 3a showing the video while changing according to the user's operation of the display button 53.

[0086] [2-3-3. Displaying the second image] Referring to Figure 6, the display mode of the second image 52 will be explained in detail. As mentioned above, the second image 52 is a reduced image of the first image 51 displayed on another mobile terminal 3 by another user's operation.

[0087] The onomatopoeia display function according to this embodiment displays the second image 52 in the second image display area 50C with virtually no time delay from the moment another user displays the first image 51 on another mobile device 3. This allows the user to know the reactions of other users who are watching the same video while watching the video.

[0088] Furthermore, if the second image 52 is superimposed on the streaming video display area 50A, as with the first image 51, users may find the second image 52 distracting and annoying while watching the video. Therefore, the second image 52 is not superimposed on the streaming video display area 50A beyond the second image display area 50C, regardless of how other users are operating. This allows users to see other users' reactions to the video they are watching without feeling annoyed.

[0089] As mentioned above, the first image 51 can also be superimposed onto the streaming video display area 50A, but this is done at the user's discretion, so the user will not feel any inconvenience from the superimposition of the first image 51 onto the streaming video display area 50A.

[0090] Furthermore, since screen 3a is configured such that the first image display area 50B and the second image display area 50C are different areas, the first image 51 displayed by the user and the second image 52 displayed by another user will not be displayed together. As a result, the user can easily distinguish and recognize the first image 51 that they displayed and the second image 52 displayed by another user.

[0091] Furthermore, the other users in this embodiment are, for example, other users registered in the same group as the user. That is, if the user is watching the video without being registered in a group, the second image 52 will not be displayed in the second image display area 50C. However, this is not limited to this, and the second image 52 may be displayed in the second image display area 50C due to the actions of all other users watching the video, regardless of whether they are registered in a group or not.

[0092] Furthermore, in the second image display area 50C according to the embodiment, multiple second images 52 are displayed based on the operations of multiple other users. In the second image display area 50C according to the embodiment, for example, the position where the second image 52 is displayed is not predetermined for each other user. That is, multiple second images 52 from the operations of multiple other users are displayed in a random mix of positions in the second image display area 50C. Moreover, the second image 52 may be displayed while moving up, down, left, and right within the second image display area 50C. However, this is not limited to this, and the position where the second image 52 is displayed may be predetermined for each other user within the second image display area 50C.

[0093] Furthermore, the second image 52 is displayed adjacent to an identification image (hereinafter referred to as "user icon") 54 that indicates another user who has displayed the first image 51, which is designated as the second image 52, on their mobile device 3. This allows the user to recognize which user created the second image 52 displayed in the second image display area 50C. In the example in Figure 6, the user icon 54 is a single letter of the alphabet displayed in the center of a circle, but this is just one example, and the user icon 54 may be an image set for each user, or it may be the user's initials, and the design of the user icon 54 is not limited.

[0094] Furthermore, although the second image 52 shown in Figure 6 displays the onomatopoeia image horizontally, this is set to correspond to the size (aspect ratio) of the second image display area 50C, and the second image 52 may also display the onomatopoeia image vertically.

[0095] Furthermore, although the second image 52 is not superimposed on the streaming video display area 50A, it may be displayed within the second image display area 50C while changing (e.g., stretching) in response to the operation patterns of other users. If the second image 52 is not superimposed on the streaming video display area 50A, it may be superimposed from the second image display area 50C onto the first image display area 50B.

[0096] [2-3-4. Display of combined images] The mobile terminal 3 according to this embodiment has a combination image display function that displays a combination image on the screen 3a corresponding to a combination of an operation by the user to display the first image 51 and an operation by another user to display the first image 51.

[0097] This allows mobile device 3 to display user and other users' reactions to videos in a more entertaining way. Furthermore, images that are not displayed by a single user's actions can be displayed through a combination of actions by other users, enabling mobile device 3 to encourage more users to view the content.

[0098] Furthermore, the combined image is displayed superimposed on the video shown on screen 3a. In other words, the combined image is displayed superimposed not only on the first image display area 50B, but also on the streamed video display area 50A and the second image display area 50C. This allows users to easily recognize that a combined image has been displayed, and enhances the entertainment value of watching the video.

[0099] Furthermore, if a combined image is displayed on screen 3a, it will be displayed in a way that allows the user to recognize that it is a combined image. For example, the user ID of another user may be displayed on screen 3a along with the combined image so that it can be recognized that another user performed the display operation at the same time.

[0100] Figure 7 is a timing chart showing the case where the combined image is displayed on screen 3a.

[0101] The combined image display function according to this embodiment displays a combined image on screen 3a when the timing of a user's operation and another user's operation falls within a predetermined time period (hereinafter referred to as the "combination period"). In other words, if multiple users perform the operation to display the first image 51 within the combination period, it is understood that this is a user reaction resulting from viewing the same scene. Therefore, since the combined image display function displays a combined image based on the reactions of multiple users in response to scenes in the video, the entertainment value for users from watching the video is further enhanced.

[0102] The start of the combination period is the moment when a user initiates the display operation of the first image 51. That is, as shown in Figure 7(A), if user B performs a display operation within the combination period which begins at the moment user A initiates the display operation, the combined image will be displayed on the screen 3a of both users A and B's mobile devices 3.

[0103] The display period for the combined image is, for example, until the user finishes displaying it. In the example in Figure 7(A), the combined image is displayed on the screen 3a of both users A and B's mobile devices 3 from the time user B starts displaying it until it finishes.

[0104] In the example in Figure 7(B), although it is within the combination period, the display operation by user C is performed after the display operation by user A, which initiated the combination, has finished. In the example in Figure 7(B), the combination image is displayed on the screen 3a of the mobile terminals 3 of users A, B, and C from the time user B's display operation starts until user C's display operation finishes.

[0105] Note that the display configuration of the combined images shown in Figure 7 is just one example and is not limited to this. For example, there is no concept of a combination period, and the combined image may be displayed on screen 3a at the timing when a user's display operation and another user's display operation overlap. Alternatively, a display time may be predetermined for each combined image, and the display of the combined image may end when that display time has elapsed.

[0106] Furthermore, the combined image according to the embodiment is displayed on screen 3a in place of the first image 51. In the example shown in Figure 7(A), first image 51, corresponding to user A's display operation, is initially displayed on screen 3a of user A's mobile terminal 3. However, as user B begins a display operation, the display of the first image 51 ends, and the combined image is displayed on screen 3a of user A's mobile terminal 3. On the other hand, the first image 51 that user B intended to display is not displayed on screen 3a of user B's mobile terminal 3; instead, the combined image is displayed. In this way, because the combined image is displayed on screen 3a in place of the first image 51, a different image from the first image 51 that the user intended to display is displayed on screen 3a, thus enhancing the entertainment value for users watching the video.

[0107] Furthermore, the combined image according to the embodiment is displayed on screen 3a when the first image 51 selected by the user and the first image 51 selected by another user are pre-set images. As a result, the entertainment value of watching the video is enhanced because the user may see a combined image that they themselves did not expect, depending on the image they selected.

[0108] For example, a combined image is displayed on screen 3a when the first image 51 selected by one user is the same as the first image 51 selected by another user. In other words, when users watching a video select the same first image 51, there is a high probability that they are experiencing similar emotions. Therefore, the display of a combined image allows users to recognize that they have empathized with other users through watching the video. More specifically, for example, if the first image 51 selected by one user is the same as the first image 51 selected by another user, an enlarged version of that first image 51 is displayed on screen 3a as a combined image. However, this is not the only example; special images that are not displayed by operating the display button 53 may also be displayed as combined images.

[0109] Furthermore, even if the first image 51 selected by the user is different from the first image 51 selected by another user, if the combination is a pre-set combination, the combined image may be displayed. In this case, for example, a special image that is not displayed by operating the display button 53 as a combined image will be displayed.

[0110] In the combined image display function according to the embodiment, the second image based on the combined image displayed on another user's screen 3a will not be displayed in the second image display area 50C of the mobile terminal 3 of a user whose combined image is not displayed. However, the invention is not limited to this, and the second image based on the combined image displayed on another user's screen 3a may be displayed in the second image display area 50C of the mobile terminal 3 of a user whose combined image is not displayed.

[0111] [2-4. Replacement Process] The server 4 in this embodiment has a replacement processing function that identifies two second images 52 whose display timings are close together from among the second images as an image group, and repeats a unit processing that replaces the image group with one new second image 52 until no image groups can be identified. As a result, the number of second images 52 displayed on the mobile terminal 3 during video playback is reduced, and sudden increases in load on the mobile terminal 3 can be suppressed.

[0112] Figure 8 is a timing chart showing the case where the second image is replaced.

[0113] As a premise, the video timeline is divided into multiple sections. The length of each section is, for example, the same as the length of the combined period. The replacement processing function according to the embodiment identifies two second images 52 that have commonalities between the subject and object from among multiple second images 52 that are each included in a section with the same start time of the display period, and performs a unit processing that invalidates the two second images 52 included in the image group and associates one new second image 52 with the video timeline, repeating this until no more image groups can be identified.

[0114] The second image 52 has attributes (second image display information) attached to it that can identify the subject (the user ID of the user who added the second image to the video and the user group to which that user belongs), the object (the type of the second image and the element group to which that type belongs), the display period, the number of times it has been composited, its status, etc. The type is determined by the object and the number of times it has been composited. The pattern of the second image 52 is determined by the type. Preferably, the appearance of the pattern reflects the number of times it has been composited, such as by displaying a number indicating the number of times it has been composited. As shown in Figure 8(A), the start of the display period of the second image A and the start of the display period of the second image B may both fall within the nth interval (n is an appropriate natural number) on the timeline. Furthermore, the type of the second image A and the object of the second image B are assumed to be the same. In this case, the replacement processing function identifies the second image A and the second image B as an image group, disables the status of the second image A and the second image B respectively, and associates a new second image C with the video's timeline.

[0115] The substituted second image 52 (second image C) inherits the additional attributes of the original second image 52 (second image A and second image B). The substituted second image 52, for example, inherits the subject and object as they are. The number of times the substituted second image 52 is synthesized is, for example, the sum of the number of times each of the original second image 52 is synthesized. The display period of the substituted second image 52 is, for example, the period from the earlier start date to the later end date. As shown in Figure 8(A), the display period of second image C corresponds to the period from the start date of second image A to the end date of second image B.

[0116] The original second image 52 is invalidated and its association with the video is broken. On the other hand, the replaced second image 52 can form an image group with other second images 52 and be replaced by a new second image 52. As shown in Figure 8(B), the start of the display period for second image C and the start of the display period for second image D are both included in the nth interval (n is an appropriate natural number) on the timeline. The user who added second image C and the user who added second image D belong to the same user group, and the type of second image C and the type of second image D are the same. In this case, the replacement function identifies second image C and second image D as an image group, invalidates the status of second image C and second image D respectively, and associates a new second image E with the video's timeline. The display period for second image E is the period from the start of second image C to the end of second image D.

[0117] [2-5. Functional Blocks] Next, the combined image display and replacement processing according to the embodiment will be explained using a functional block diagram. Note that the combined image display function in the mobile terminal 3 is included in the onomatopoeia display function, and the combined image display and replacement processing functions in the server 4 are each included in the video distribution function.

[0118] [2-5-1. Functional block related to the server's video streaming function] Figure 9 is a functional block diagram relating to the video distribution function of Server 4 according to the embodiment.

[0119] The CPU 20 of server 4 includes a group registration unit 60, a video distribution unit 61, a first image display information reception control unit 62, a second image display information generation unit 63, a second image display information transmission control unit 64, a combined image information generation unit 65, a combined image information transmission control unit 66, and a replacement processing unit 67. Each function of the CPU 20 is realized by a program stored in the HDD 26.

[0120] The group registration unit 60 accepts registration of a group 5 consisting of multiple mobile terminals 3. For example, group 5 registration is performed by registering a user ID.

[0121] The video distribution unit 61 distributes video to the mobile device 3 in response to a request from the mobile device 3.

[0122] The first image display information receiving control unit 62 receives the first image display information received from the mobile terminal 3 via the external interface 32, associates the first image display information with the corresponding video, and stores it in the HDD 26.

[0123] The first image display information is information about the first image 51 displayed on the screen 3a of the mobile terminal 3, and is transmitted from the mobile terminal 3 to the server 4 at the time the first image 51 is displayed on the mobile terminal 3. The first image display information includes, for example, the type of first image 51 displayed, the user ID indicating the user who displayed the first image 51, the type of user operation performed to display the first image 51, and the elapsed time from the start of the video when the first image 51 was displayed.

[0124] The second image display information generation unit 63 generates second image display information based on the received first image display information, associates the second image display information with the corresponding video, and stores it in the HDD 26.

[0125] The second image display information is information about the second image 52 to be displayed in the second image display area 50C, and includes the type of the second image 52, the user ID of the user who displayed the first image 51, which is the source of the second image 52, on their mobile terminal 3, the display period, the number of times it has been synthesized, the status, etc.

[0126] The second image display information transmission control unit 64 controls the external interface 32 to transmit the second image display information to the mobile terminal 3. The destination of the second image display information is a different mobile terminal 3 from the mobile terminal 3 to which the first image display information was transmitted.

[0127] The combination image information generation unit 65 performs a combination determination based on the received first image display information. Then, the combination image information generation unit 65 generates combination image information indicating the combination image according to the result of the combination determination and stores the combination image information in the HDD 26.

[0128] The combined image information is information about the combined image to be displayed on screen 3a, and includes the type of combined image, the user IDs of multiple users who performed the display operation at the same time, etc.

[0129] Furthermore, the combination determination is performed for each piece of first image display information, and it is determined whether the display operation of the first image 51 indicated by the first image display information falls within the combination period based on the start timing of the display operation of the first image 51 by another user.

[0130] The combined image information transmission control unit 66 controls the external interface 32 to transmit the combined image information to the mobile terminal 3. The mobile terminal 3 to which the combined image information is transmitted are multiple mobile terminals 3 corresponding to the user IDs included in the combined image information.

[0131] The replacement processing unit 67 executes the replacement process.

[0132] [2-5-2. Functional block related to onomatopoeia display on mobile devices] Figure 10 is a functional block diagram relating to onomatopoeia display in the mobile terminal 3 according to the embodiment.

[0133] The main control unit 40 of the mobile terminal 3 comprises an image display control unit 70, a display area setting unit 71, a first image display operation reception unit 72, an operation state determination unit 73, a first image display information generation unit 74, a first image display information transmission control unit 75, a second image display information reception control unit 76, and a combined image information reception control unit 77. Each function of the main control unit 40 is realized by a program stored in the auxiliary storage unit 44.

[0134] The image display control unit 70 controls the touch panel display 3a to display an image based on image data. In this embodiment, the image display control unit 70 displays a video distributed from the server 4 in the distributed video display area 50A, displays a first image 51 in the first image display area 50B according to user operation, and displays a second image 52 in the second image display area 50C based on the first image 51 displayed by another user on another mobile terminal 3. Furthermore, in this embodiment, the image display control unit 70 displays the first image 51 superimposed on the distributed video display area 50A beyond the first image display area 50B according to the user operation.

[0135] The display area setting unit 71 sets the position and size (length and width) of the streamed video display area 50A, the first image display area 50B, and the second image display area 50C as the display mode of screen 3a when video is streamed from server 4. Although these positions and sizes are predetermined, the user may set them arbitrarily.

[0136] The first image display operation reception unit 72 receives the user's operation of the display button 53 while the screen 3a is displaying a video. That is, the first image display operation reception unit 72 recognizes the display button 53 that the user has operated from among the multiple display buttons 53, and displays the first image 51 corresponding to that display button 53 on the screen 3a via the image display control unit 70. The first image display operation reception unit 72 displays the first image 51 on the screen 3a such that the display mode of the first image 51 changes according to the determination result of the operation state determination unit 73.

[0137] The operation state determination unit 73 determines the operation state of the display button 53 by the user. The operation state here refers to the operation mode, including the duration of the long press on the display button 53.

[0138] The first image display information generation unit 74 generates first image display information corresponding to the first image 51 displayed on screen 3a by user operation.

[0139] The first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

[0140] The second image display information receiving control unit 76 outputs the second image display information received from the server 4 via the communication unit 46 to the image display control unit 70.

[0141] The combined image information receiving control unit 77 outputs the combined image information received from the server 4 via the communication unit 46 to the image display control unit 70. The image display control unit 70 then displays the combined image on the screen 3a displaying the video. In this embodiment, the image display control unit 70 displays the combined image on the screen 3a in place of the first image 51, which is displayed in response to the user's operation of the display button 53.

[0142] [2-6. Processing Flow] Next, the flow of onomatopoeia display processing in the embodiment of the mobile terminal 3 and server 4 will be explained using a flowchart.

[0143] [2-6-1. Flowchart for displaying onomatopoeia on mobile devices] Figure 11 is a flowchart showing the flow of onomatopoeia display processing performed by the main control unit 40 of the mobile terminal 3. The program for performing onomatopoeia display processing is pre-stored in a predetermined area of ​​the auxiliary storage unit 44. For example, onomatopoeia display processing starts when the mobile terminal 3 starts playing a video distributed from the server 4, and ends when the video playback ends.

[0144] In step S100, the occurrence of a trigger is detected. In this embodiment, the trigger is either a first trigger (reception of user operation of display button 53) or a second trigger (reception of second image display information). The mobile terminal 3 in this embodiment updates the screen display content each time a trigger is detected. The first and second triggers may occur in parallel. Therefore, while processing in response to the occurrence of one trigger is being executed, processing in response to the occurrence of the other trigger may be executed. In addition, multiple second triggers may occur in parallel. Therefore, while processing in response to the occurrence of one second trigger is being executed, processing in response to the occurrence of another second trigger may be executed.

[0145] In step S102, if the detected trigger is the first trigger, the process proceeds to step S106; if the detected trigger is the second trigger, the process proceeds to step S104.

[0146] In step S104, the image display control unit 70 displays the second image 52, indicated by the second image display information, in the second image display area 50C, and the process ends.

[0147] In step S106, the image display control unit 70 displays the first image 51 corresponding to the display button 53 operated by the user in the first image display area 50B. The first image 51 may be stored in the mobile terminal 3 in advance, or it may be downloaded from the server 4 each time the display button 53 is operated by the user.

[0148] In step S108, the first image display information generation unit 74 generates first image display information corresponding to the first image 51 displayed in the first image display area 50B.

[0149] In step S110, the first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

[0150] In step S112, the operation status determination unit 73 determines whether the user has finished operating the display button 53. If the determination is positive, the process proceeds to step S120; if the determination is negative, the process proceeds to step S114.

[0151] In step S114, the combined image information reception control unit 77 determines whether or not combined image information has been received via the communication unit 46. If the determination is positive, the process proceeds to step S118; otherwise, the process proceeds to step S116.

[0152] In step S116, the image display control unit 70 displays the first image 51 (onomatopoeia image) in an extended state until the user finishes operating the display button 53, and then returns to step S112.

[0153] In step S118, the image display control unit 70 displays the combined image indicated by the combined image information on screen 3a, and then returns to step S112.

[0154] In step S120, the image display control unit 70 finishes displaying the first image 51 and terminates the process.

[0155] In the flowchart shown in Figure 11, the first image display information is generated along with the display of the first image 51, and the first image display information is sent to the server 4. However, the system is not limited to this, and the first image display information may be sent to the server 4 after the display of the first image 51 has finished. Furthermore, the first image display information transmission control unit 75 may send first image display information indicating the start of the display of the first image 51 to the server 4 at the timing when the display of the first image 51 begins, and send first image display information indicating the end of the display of the second image 52 to the server 4 at the timing when the display of the first image 51 has finished. This allows the server 4 to recognize that the first image 51 (onomatopoeia image) is being displayed in an expanded form during the interval between receiving the first image display information indicating the start of the display of the first image 51 and the first image display information indicating the end of the display.

[0156] [2-6-2. Flowchart for displaying onomatopoeia on the server] Figure 12 is a flowchart showing the flow of onomatopoeia display processing performed by the CPU 20 of server 4. The program for performing onomatopoeia display processing is pre-stored in a predetermined area of ​​HDD 26. For example, onomatopoeia display processing starts when server 4 is distributing video to mobile terminal 3 and ends when video distribution ends.

[0157] In step S200, the first image display information reception control unit 62 determines whether or not it has received the first image display information from the mobile terminal 3. If the determination is positive, the process proceeds to step S202; otherwise, the process enters a waiting state.

[0158] In step S202, the first image display information receiving control unit 62 stores the received first image display information in the HDD 26.

[0159] In step S204, the combined image information generation unit 65 performs a combination determination. If the determination is positive, the process proceeds to step S208; if the determination is negative, the process proceeds to step S206. Specifically, if the display operation of the first image 51 to be combined is performed by multiple users within the combination period, the determination is positive and the process proceeds to step S208. On the other hand, if the display operation of the first image 51 to be combined is not performed by multiple users within the combination period, the determination is negative and the process proceeds to step S206.

[0160] In step S206, the second image display information generation unit 63 generates second image display information based on the received first image display information and stores it in the HDD 26.

[0161] In step S208, the combination image information generation unit 65 generates combination image information that indicates a combination image corresponding to the display operation of the first image 51 that is the subject of the combination.

[0162] In step S210, the combined image information transmission control unit 66 controls the external interface 32 to transmit the combined image information and returns to step S200. The destinations for the combined image information are the multiple mobile terminals 3 that transmitted the first image display information to be combined, i.e., the mobile terminals 3 corresponding to the multiple user IDs included in the combined image information.

[0163] Figure 13 is a flowchart showing the flow of onomatopoeia display processing performed by the CPU 20 of server 4. The flow shown in Figure 13 targets the second image display information whose display period start date falls within the interval at the end of each interval in the video timeline.

[0164] In step S300, the replacement processing unit 67 makes a determination. If the determination is positive, the process proceeds to step S302; if the determination is negative, the process proceeds to step S304. Specifically, if there is second image display information that has commonalities between the subject and the object, the determination is positive and the process proceeds to step S302. On the other hand, if there is no second image display information that has commonalities between the subject and the object, the determination is negative and the process proceeds to step S304.

[0165] In step S302, the replacement processing unit 67 performs replacement processing. In the unit processing of the replacement processing, the replacement processing unit 67 identifies two second image display information that have commonalities between the subject and the object as an image group, invalidates the status of the two second image display information included in the image group, and generates one new second image display information. In step S302, this unit processing is repeated until there are no more second image display information that have commonalities between the subject and the object.

[0166] In step S304, the second image display information transmission control unit 64 controls the external interface 32 to transmit all of the second image display information included in the target section to a different mobile terminal 3 from the mobile terminal 3 that transmitted the first image display information. Therefore, the display period of the second image 52 based on the second image display information on the mobile terminal 3 will be delayed compared to the display period associated with the video.

[0167] As described above, in the embodiment, the mobile terminal 3 provides the server 4 with operation information for an addition operation performed on the first mobile terminal 3 by a first user who is watching a video displayed on the screen 3a of the first mobile terminal 3 in order to add a first image 51 to the video, and operation information for an addition operation performed on the second mobile terminal 3 by a second user who is watching the same video displayed on the screen 3a of the second mobile terminal 3 in order to add a first image 51 to the video. In order to improve entertainment value, the server 4 displays a combination image based on the combination of the addition operation by the first user and the addition operation by the second user on the screen 3a of the first mobile terminal 3 and the screen 3a of the second mobile terminal 3, respectively. As a result, images that would not be displayed by an operation by one user are displayed by a combination of operations by other users, so the mobile terminal 3 can encourage more users to watch the video.

[0168] Furthermore, the server 4 in this embodiment replaces two or more second images 52 associated with the video that have close display timings and share common subject and object characteristics with one new second image 52 during video playback. As a result, the number of second images 52 displayed on the mobile terminal 3 is reduced, allowing the server 4 to suppress sudden increases in load on the mobile terminal 3.

[0169] [3. Variant] The technical scope of the present invention is not limited to the scope described in the above embodiments. Various modifications or improvements can be made to the above embodiments without departing from the spirit of the invention, and such modified or improved forms are also included in the technical scope of the present invention.

[0170] Furthermore, the various processing flows described in the above embodiments are merely examples, and unnecessary steps may be deleted, new steps added, or the processing order rearranged without departing from the spirit of the present invention.

[0171] [3-1. Variation 1 (Sharing images without using a server)] In the above embodiment, a server 4 receives first image display information from a mobile terminal 3, transmits second image display information generated based on the first image display information to another mobile terminal 3, and the other mobile terminal 3 displays the second image 52 indicated by the second image display information in the second image display area 50C. However, the present invention is not limited to this embodiment.

[0172] For example, multiple mobile terminals 3 may send and receive first image display information to each other without going through server 4, and each mobile terminal 3 may display the second image 52 in the second image display area 50C. That is, when mobile terminal 3 receives first image display information from another mobile terminal 3, it reads information about the first image 51 displayed on the other mobile terminal 3 from the first image display information and displays the second image 52 based on that information in the second image display area 50C.

[0173] [3-2. Variation 2 (Watching the video independently)] The above embodiment illustrates a case where images expressing emotions towards a video are shared among multiple users simultaneously watching the video. In contrast, an image expressing the emotions of a user who watched a video first can be shared with a user who watches the video later. This makes it possible to display combined images according to the combination of additional actions taken by the first user who watched the video first and the second user who watched it later.

[0174] In the following, we consider the case where the timing at which a first user using a first mobile terminal 3 (first device) views a certain video (viewing target) is later than the timing at which a second user using a second mobile terminal 3 (second device) views the same video (Figure 13(A)). In this case, the additional operations performed by the first user and the additional operations performed by the second user may not be in close proximity in real time. However, on the video's timeline, they are in close proximity (Figure 13(B)). The first user and the second user may be the same person and someone else, or the same person in the past and the same person in the present.

[0175] In the modified example 2, the operation information of the addition operation (second operation) performed on the second mobile terminal 3a by a second user who is watching a video displayed on the screen 3a (second screen) of the second mobile terminal 3 in order to add a first image (second element) to the video is transmitted from the second mobile terminal 3a to the server 4. The information is provided to the (second instruction means, shared information processing device). The server 4 pre-transmits second image display information (second instruction), which causes the second image (second element) to be displayed at a timing corresponding to an additional operation by the second user, to the first mobile terminal 3 used by the first user who wants to watch the same video.

[0176] In order to improve entertainment value, the first mobile terminal 3, upon receiving an addition operation (first operation) performed on the first mobile terminal 3a by a first user who is watching a video displayed on the screen 3a (first screen) of the first mobile terminal 3 to add a first image (first element) to the video, displays a combined image (third element) based on the combination of the received addition operation (first operation) and the addition operation (second operation) related to the second image display information on the screen 3a of the first mobile terminal 3. In modified example 2, since the timing of the second operation performed on the second mobile terminal 3 is known in advance, it is also possible to display the combined image (third element) based on the combination of the received addition operation (first operation) and the addition operation (second operation) related to the second image display information on the screen 3a of the first mobile terminal 3 without delay after receiving the first operation on the first mobile terminal 3.

[0177] [3-3. Variation 3 (Do not display the first image when a combined image is displayed)] In the above embodiment, the first image is displayed even when a combined image is displayed. Alternatively, the first image may be prevented from being displayed when a combined image is displayed.

[0178] Specifically, the image display control unit 70 (second control means) of the mobile terminal 3 waits for the reception of combined image information until a predetermined period (combination period) has elapsed since the additional operation was received by the first image display operation reception unit 72 (first reception means). If the image display control unit 70 receives combined image information within the predetermined time, it displays a combined image (third element) based on the received combined image information. If it does not receive combined image information within the predetermined period, it displays a first image (first element) based on the received additional operation.

[0179] [3-4. Variation 4 (Display the second image even when a combined image is displayed)] In the above embodiment, the second image is not displayed when the combined image is displayed. However, the second image may be displayed even when the combined image is displayed. Specifically, when the server 4 receives the first image information, it may generate second image display information regardless of whether or not a combined display operation has been performed, and send it to the mobile terminal 3.

[0180] [3-5. Modification 5 (Limiting the activation of the substitution process)] In the above embodiment, the unit processing is repeated at predetermined intervals (combination periods), and multiple second image display information to be replaced is replaced with one second image display information. However, in situations where sudden increases in load are unlikely to occur on the mobile terminal 3, the replacement process may be omitted.

[0181] Specifically, the replacement processing unit 67 (identification means) of server 4 omits the unit processing (replacement processing) that identifies two (a predetermined number) of second image display information included in a group of adjacent element information (shared elements) whose display timings are close together as an image group (element group) if the number of elements in the adjacent element group does not satisfy the activation condition. The activation condition is, for example, that the number of elements is equal to or greater than a predetermined threshold. The activation condition may be flexibly varied according to the processing capacity of the mobile terminal 3. For example, a predetermined threshold may be set according to the processing capacity of the terminal with the lower capacity among the multiple mobile terminals 3 belonging to group 5. Processing capacity refers to, for example, the display processing capacity (DSP clock frequency) or reception processing capacity (communication line speed) of the mobile terminal 3. [Explanation of Symbols]

[0182] 1. Video distribution system 2. Communication lines 3. Mobile terminals (examples of "device 1", "device 2", and "information processing device") 3a Screen (Examples of "Screen 1" and "Screen 2") 4. Server (an example of "first instruction means," "second instruction means," and "shared information processing device") 5. Group (An example of an "information processing system") 50A Streaming video display area 50B 1st image display area 50C 2nd image display area 51 Image 1 52. Second image (an example of "shared elements" and "alternative elements") 53 Display button 60 Group Registration Department 61 Video Streaming Department 62 First Image Reception Information Reception Control Unit 63 Second image display information generation section 64 Second Image Display Information Transmission Control Unit 65 Combined Image Information Generation Unit 66 Combined Image Information Transmission Control Unit 67. Substitution Processing Unit (Example of "Specific Means" and "Substitution Means") 70 Image display control unit (an example of "first control means", "second control means", "third control means", and "display control means") 71 Display area setting section 72 First image display operation reception unit (an example of "first reception means" and "second reception means") 73 Operation status determination unit 74 First image display information generation section 75 First Image Display Information Transmission Control Unit (an example of the "First Providing Means" and "Second Providing Means") 76 Second image display information receiving control unit (an example of the "third receiving means") 77 Combined Image Information Receiving and Control Unit

Claims

1. The processor comprises, Information is obtained regarding a first operation in which a first element selected from multiple options is added to the viewing target on the first terminal of the first user who is viewing the viewing target. Information is obtained regarding a second operation in which a second element is added to the aforementioned viewing target, selected from multiple options, on the second terminal of a second user who is viewing the aforementioned viewing target. If the first and second operations are received within a predetermined time on the timeline of the object to be viewed, and the combination of the first element and the second element is a pre-set combination, the combination element, which is a combination of the first element and the second element and is not included in the selection, is displayed on the first terminal and the second terminal. Information processing device.

2. The processor obtains information on a first operation in which a first element selected from multiple options is added to the viewing target on the first terminal of a first user who is viewing the viewing target. The processor obtains information on a second operation in which a second element is added to the viewing target selected from multiple options on the second terminal of a second user who is viewing the viewing target. If the processor receives the first and second operations within a predetermined time on the timeline of the object to be viewed, and the combination of the first and second elements is a pre-set combination, it will display the combination element, which is a combination of the first and second elements and is not included in the selection, on the first and second terminals. Information processing methods.

3. Information is received from the first user who is viewing the target content, regarding a first operation to add a first element to the target content selected from multiple options. Information is received from a second user who is viewing the aforementioned viewing target, regarding a second operation to add a second element to the viewing target selected from multiple options. If the first and second operations are received within a predetermined time on the timeline of the object to be viewed, and the combination of the first element and the second element is a pre-set combination, the combination element formed by combining the first element and the second element, which is not included in the selection, is displayed on the display. A program that instructs the processor to perform a task.

4. The system comprises a server, a first terminal, and a second terminal, and the server is Information is obtained regarding a first operation in which a first element selected from multiple options is added to the viewing target on the first terminal of the first user who is viewing the viewing target. Information is obtained regarding a second operation in which a second element is added to the aforementioned viewing target, selected from a plurality of options, on the second terminal of a second user who is viewing the aforementioned viewing target. A system that, when the first and second operations are received within a predetermined time on the timeline of the object to be viewed, and the combination of the first element and the second element is a pre-set combination, displays a combination element, which is a combination of the first element and the second element and is not included in the selection, on the first terminal and the second terminal.

Images