Information processing device and moving image display method

Abstract

A moving image display unit 116 reproduces a moving image. A thumbnail display unit 118 displays a thumbnail of moving images related to the displayed image. The thumbnail display unit 118 changes the thumbnail of moving images to be displayed, in accordance with the image displayed by the moving image display unit 116.

Description

Information processing device and video display method

[0001] The present disclosure relates to a technique for allowing a user to watch a video.

[0002] Videos of games played by various players are uploaded to the video sharing server. When a user enters a string into the search box of a video sharing site, the video sharing server provides the user with game videos that match the entered string, and the user plays the provided game videos.

[0003] While watching a video, thumbnails of other videos, called related videos, may be displayed around the currently playing video. Users can view the related videos by selecting the thumbnail of the related videos. However, the currently playing video and the related videos are often not very related. Therefore, the present disclosure provides a mechanism for presenting videos that are highly related to the currently playing video.

[0004] An aspect of the present disclosure is an information processing device having a function of playing videos, the information processing device including a video display unit that plays the videos and a thumbnail display unit that displays thumbnails of videos related to a displayed image, the thumbnail display unit changing the thumbnail of the video to be displayed depending on the image displayed by the video display unit.

[0005] Another aspect of the present invention is a method for displaying a video on an information processing device, comprising the steps of playing the video, displaying a thumbnail of the video associated with the displayed image, and, when the displayed image is changed, displaying a thumbnail of the video associated with the changed image.

[0006] Any combination of the above components, and conversion of the present disclosure into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present disclosure.

[0007] 1 is a diagram illustrating an information processing system according to an embodiment; FIG. 2 is a diagram illustrating a hardware configuration of an information processing device; FIG. 3 is a diagram illustrating an example of functional blocks of an information processing device; FIG. 4 is a diagram illustrating an example of a game screen; FIG. 5 is a diagram illustrating an example of functional blocks of a video sharing server; FIG. 6 is a diagram illustrating an example of letters and numbers extracted from a game image; FIG. 7 is a diagram illustrating an example of a divided game video; FIG. 8 is a diagram illustrating an example of a home screen of a video sharing site; FIG. 9 is a diagram illustrating an example of a selection screen for a plurality of candidate videos; FIG. 10 is a diagram illustrating an example of a playback screen for a game video; FIG. 11 is a diagram illustrating another example of a playback screen for a game video.

[0008] In the embodiment, an information processing device, which is a user terminal device, executes a game program. When an event occurs in the game, the game program outputs event information identifying the event to the system software. An event occurs when there is a change in the progress of game play or a change in the behavior of a player character controlled by a player. The game program incorporates multiple play units, each of which has at least a set end condition.

[0009] A game program includes activities, which are a type of unit of play and have set start and end conditions. Activities may be composed of stages, quests, missions, tournaments, sessions, etc. Activities are set appropriately by the game maker, and one quest that appears in the course of the game may constitute one activity, or multiple quests may constitute one activity. When multiple quests constitute one activity, completing all of the multiple quests is the end condition for that activity.

[0010] When an activity starts, the game program outputs to the system software an activity identifier (activity ID) that identifies the activity and event information that includes information indicating the start of the activity. When the activity ends, the game program outputs to the system software event information that includes the activity ID and information indicating the end of the activity. When the activity ends, the game program may include the result of the activity (success or failure) in the event information.

[0011] The system software generates event data by adding a user identifier (user account) that identifies the user, a game identifier (game ID) that identifies the game, and time information (timestamp) to the event information output from the game program, and transmits the event data to an external server device. The server device collects the event data transmitted from multiple information processing devices operated by multiple players and analyzes play trends related to the activity. For example, the server device may statistically calculate the average play time of an activity based on the event data transmitted from the multiple information processing devices.

[0012] In the embodiment, the server device acquires video of a game played by multiple players and analyzes the game video. The server device performs image analysis of the game video to acquire state values ​​in the game. The state values ​​may be values ​​indicating the state of a scene in the game, or the static and / or dynamic state of a player character controlled by a player. The state value of a player character may be the HP value of the player character or an action name indicating an action performed by the player character. The server device associates the acquired state values ​​with the time at which the state values ​​occurred and sets them as metadata for the game video. The server device may embed the metadata related to the state values ​​in the game video, or may store it in a database.

[0013] The server device of the embodiment provides a video sharing service that distributes game videos uploaded by multiple players to users. When a user enters a string of characters in a search box of a video sharing site, the video sharing server searches for game videos that match the entered string and provides the searched game videos to the user.

[0014] FIG. 1 illustrates an information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 of the embodiment is a system in which a user can watch game videos, and includes an information processing device 10 operated by the user and a server device 5. An access point (hereinafter referred to as "AP") 8 has the functions of a wireless access point and a router, and the information processing device 10 connects to the AP 8 wirelessly or via a wired connection to be able to communicate with the server device 5 on a network 3. While FIG. 1 illustrates one user and one information processing device 10, in the information processing system 1, multiple information processing devices 10 operated by multiple users are connected to the server device 5 via the network 3.

[0015] The information processing device 10 is connected wirelessly or via a wire to an input device 6 operated by a user, and the input device 6 outputs information operated by the user to the information processing device 10. When the information processing device 10 receives operation information from the input device 6, it reflects the information in the processing of system software and game programs, and outputs the processing results from the output device 4. In the information processing system 1, the information processing device 10 is a game device (game console) that executes a game, and the input device 6 is a device such as a game controller that supplies user operation information to the information processing device 10. Note that the input device 6 may also be an input interface such as a keyboard or a mouse.

[0016] The auxiliary storage device 2 is a large-capacity storage device such as an HDD (hard disk drive) or an SSD (solid state drive), and may be a built-in storage device or an external storage device connected to the information processing device 10 via a USB (universal serial bus) or the like. The output device 4 may be a television having a display for outputting images and speakers for outputting sounds. The output device 4 may be connected to the information processing device 10 by a wired cable or wirelessly.

[0017] The camera 7, which is an imaging device, is provided near the output device 4 and captures images of the space around the output device 4. While Fig. 1 shows an example in which the camera 7 is attached to the top of the output device 4, it may be placed on the side or bottom of the output device 4, and in either case, it is placed in a position where it can capture an image of a user positioned in front of the output device 4. The camera 7 may be a stereo camera.

[0018] The server device 5 of the embodiment includes a management server 12 and a video sharing server 14. In the following description, the management server 12 and the video sharing server 14 are described as separate servers, but the management server 12 and the video sharing server 14 may be integrated into one server.

[0019] The management server 12 provides network services to users of the information processing system 1. The management server 12 manages user identifiers (user accounts) that identify each user, and each user signs in to the network services provided by the management server 12 using their user account. By signing in to the network service from their information processing device 10, users can register game save data and trophies, which are virtual rewards acquired during game play, on the management server 12. By registering save data and trophies on the management server 12, users can synchronize save data and trophies even if they use an information processing device other than the information processing device 10.

[0020] The management server 12 collects event data from multiple information processing devices 10 operated by multiple players (users). The management server 12 records the event data by linking it to user accounts. The management server 12 may perform statistical processing on the play times of activities from the collected event data to calculate the average play time of the activities, etc.

[0021] The video sharing server 14 provides a video sharing service that distributes video of games played by multiple players. The video sharing server 14 of the embodiment has a function of analyzing game videos uploaded by players and generating metadata for the game videos.

[0022] The video sharing server 14 performs image analysis of the game video to acquire game status values. The video sharing server 14 may have an optical character recognition (OCR) function and extract the number of combos achieved in battle scenes and the player character's HP (hit points) from the game video to acquire them as game status values. The video sharing server 14 may also have an object detection function and detect enemy characters that appear and items that are obtained to acquire them as game status values. The video sharing server 14 may also have an action recognition function and detect the actions of the player character to acquire them as game status values.

[0023] The video sharing server 14 associates the acquired state value with the time when the state value occurred and sets the associated state value as metadata for the game video. The video sharing server 14 may embed the metadata in the game video or store it in a database. Note that analysis of the game video may be performed by the management server 12 or another server, rather than by the video sharing server 14.

[0024] 2 shows the hardware configuration of the information processing device 10. The information processing device 10 includes a main power button 20, a power ON LED 21, a standby LED 22, a system controller 24, a clock 26, a device controller 30, a media drive 32, a USB module 34, a flash memory 36, a wireless communication module 38, a wired communication module 40, a subsystem 50, and a main system 60.

[0025] The main system 60 includes a main CPU (Central Processing Unit), a memory and memory controller serving as a main storage device, a GPU (Graphics Processing Unit), etc. The GPU is primarily used for processing game programs. The main CPU has the function of starting up system software and executing game programs installed in the auxiliary storage device 2 in the environment provided by the system software. The subsystem 50 includes a sub-CPU, a memory and memory controller serving as a main storage device, etc., but does not include a GPU.

[0026] While the main CPU has the function of executing a game program installed in the auxiliary storage device 2, the sub-CPU does not have such a function. However, the sub-CPU does have the function of accessing the auxiliary storage device 2 and the function of sending and receiving data to and from the server device 5. The sub-CPU is configured with only these limited processing functions, and therefore can operate with less power consumption than the main CPU. These functions of the sub-CPU are executed when the main CPU is in standby mode.

[0027] The main power button 20 is an input unit that receives operational input from the user, is provided on the front of the housing of the information processing device 10, and is operated to turn on or off the power supply to the main system 60 of the information processing device 10. The power ON LED 21 lights up when the main power button 20 is turned on, and the standby LED 22 lights up when the main power button 20 is turned off. The system controller 24 detects that the main power button 20 has been pressed by the user.

[0028] The clock 26 is a real-time clock that generates current date and time information and supplies it to the system controller 24 , the subsystem 50 , and the main system 60 .

[0029] The device controller 30 is configured as an LSI (Large-Scale Integrated Circuit) that, like a southbridge, transfers information between devices. As shown in the figure, devices such as the system controller 24, media drive 32, USB module 34, flash memory 36, wireless communication module 38, wired communication module 40, subsystem 50, and main system 60 are connected to the device controller 30. The device controller 30 absorbs differences in the electrical characteristics and data transfer speeds of each device and controls the timing of data transfer.

[0030] The media drive 32 is a drive device that is driven by loading a ROM medium 44 on which application software such as games and license information are recorded, and that reads programs, data, and the like from the ROM medium 44. The ROM medium 44 is a read-only recording medium such as an optical disk, a magneto-optical disk, or a Blu-ray disk.

[0031] The USB module 34 is a module that connects to external devices via a USB cable. The USB module 34 may also be connected to the auxiliary storage device 2 and the camera 7 via USB cables. The flash memory 36 is an auxiliary storage device that constitutes internal storage. The wireless communication module 38 wirelessly communicates with the input device 6 using a communication protocol such as Bluetooth (registered trademark) or IEEE802.11. The wired communication module 40 communicates with external devices via a wired connection and connects to the network 3 via the AP 8.

[0032] 3 shows an example of functional blocks of the information processing device 10. The information processing device 10 of the embodiment includes a processing unit 100 and a communication unit 102. The processing unit 100 includes a game execution unit 110, a game image generation unit 112, a display processing unit 114, an event information acquisition unit 120, an event data transmission unit 122, a game video acquisition unit 130, a game video transmission unit 132, a reception unit 140, and an acquisition unit 142. The display processing unit 114 includes a video display unit 116 and a thumbnail display unit 118.

[0033] The functionality of the components in the information processing device 10 may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, configured or programmed to perform the functions described herein. A processor is considered to be circuitry or processing circuitry including transistors and other circuits. A processor may also be a programmed processor that executes a program stored in a memory.

[0034] In this specification, a circuit, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0035] If the hardware is a processor that is considered to be a type of circuitry, the circuitry, means, or unit may be a combination of hardware and software used to configure the hardware and / or processor.

[0036] The communication unit 102 has the functions of both the wireless communication module 38 and the wired communication module 40. While the user is playing a game, the communication unit 102 receives information (game operation information) relating to the user's operation of the input device 6 and provides the information to the game execution unit 110.

[0037] The game execution unit 110 executes a game in accordance with user operations. While the user is playing the game, the game execution unit 110 executes a game program based on game operation information and performs calculations to move a player character within a virtual space. The game image generation unit 112 includes a GPU (Graphics Processing Unit) and receives the results of calculations in the virtual space, generating game images from a viewpoint (virtual camera) within the virtual space and game sounds at the viewpoint within the virtual space. The video display unit 116 outputs the game images and game sounds generated by the game image generation unit 112 from the output device 4.

[0038] 4 shows an example of a game screen displayed on the output device 4. The user progresses through the game by operating the input device 6 while viewing the game images and game sounds output from the output device 4. Hereinafter, for convenience of explanation, the game images and game sounds may also be simply referred to as "game video."

[0039] When an activity is started during the progress of the game, the game program outputs event information indicating the occurrence of an activity start event, and when the activity is ended, the game program outputs event information indicating the occurrence of an activity end event. When the event information acquisition unit 120 acquires event information from the game program, it provides the event information to the game video acquisition unit 130. The event information acquisition unit 120 also generates event data by adding a user account, a game ID, and time information (timestamp) indicating the time the event occurred to the acquired event information, and provides the generated event data to the event data transmission unit 122.

[0040] A game developer may incorporate various activities into a game. For example, if a battle activity against an enemy boss is incorporated into a game, the game program outputs, at the start of the battle, event information including an activity ID that identifies the battle activity and information indicating the start of the battle activity. When the player wins the battle against the enemy boss, the game program outputs event information including an activity ID that identifies the battle activity, information indicating the end of the battle activity, and information indicating the success of the activity.

[0041] The event data transmission unit 122 transmits the event data provided from the event information acquisition unit 120 to the management server 12 via the communication unit 102. In the information processing system 1, the event data transmission process is performed by all information processing devices 10 connected to the management server 12, and the management server 12 acquires event data for various games from the multiple information processing devices 10. Upon acquiring the event information from the game program, the event information acquisition unit 120 preferably immediately generates event data to which a user account and a timestamp are added, and the event data transmission unit 122 transmits the event data to the management server 12.

[0042] In the embodiment, the game video acquisition unit 130 has a ring buffer and has the function of automatically recording game videos generated by the game image generation unit 112 into the ring buffer in the background. The ring buffer logically connects the end and beginning of the ring buffer to enable cyclical use of memory space, and may be generated by setting the start and end addresses of the memory space in the auxiliary storage device 2. The game video acquisition unit 130 records game video data in the ring buffer in real time from the start address in a predetermined address order. Once the game video data reaches the end address, it returns to the start address and overwrites the previous data, repeating this process. For example, the ring buffer can record up to 60 minutes of game video. Therefore, the ring buffer records game video played by the user over the last 60 minutes along with time information (timestamps).

[0043] During game play, the user can generate user-generated content by cutting out at least a portion of the game video recorded in the ring buffer. When the user operates a predetermined button on input device 6 during game play, a content generation screen is superimposed on the game screen, and the user can select a time period for saving the most recent game play from multiple options (e.g., the last 30 seconds, the last minute, the last 5 minutes, the last 15 minutes, the last 60 minutes, etc.). When the user selects the saving time, game video acquisition unit 130 cuts out the game video for the time period selected by the user from the ring buffer and stores the cut-out game video in game video recording unit 154 as user-generated content.

[0044] As another method of generating content, the user may specify a start point of the content in real time while playing a game, and then specify an end point of the content in real time. When the end point is specified, game video acquisition unit 130 extracts the game video from the ring buffer from the start point to the end point specified by the user, and stores the extracted game video in game video recording unit 154 as user-generated content.

[0045] As described above, the game video acquisition unit 130 receives event information from the event information acquisition unit 120. The game video acquisition unit 130 associates the received event information with time information (timestamps) indicating the times at which the events occurred and records them in the ring buffer. When cutting out game video from the ring buffer, the game video acquisition unit 130 adds (embeds) event information and timestamps that occurred between the first time of cutting (start time) and the last time of cutting (end time) to the game video as metadata. Furthermore, the game video acquisition unit 130 adds (embeds) a user account and a game ID to the game video as metadata. The game video acquisition unit 130 saves the game video with the embedded metadata in the game video recording unit 154 as user-generated content.

[0046] A user can distribute user-generated content to an unspecified number of users by uploading the user-generated content to video sharing server 14. When a user operates input device 6 to input an instruction to upload user-generated content to information processing device 10, game video transmission unit 132 transmits the user-generated content (game video) stored in game video recording unit 154 to video sharing server 14.

[0047] 5 shows functional blocks of the video sharing server 14 according to the embodiment. The video sharing server 14 includes a processing unit 200, a communication unit 202, and a recording device 240. The processing unit 200 includes a game video acquisition unit 210, an analysis unit 212, a metadata setting unit 214, a video division unit 216, a vector setting unit 218, a search request reception unit 220, a search unit 222, a selection reception unit 224, a related video identification unit 226, a video provision unit 228, and a thumbnail provision unit 230. The recording device 240 includes a video recording unit 242, a metadata recording unit 244, and a vector recording unit 246.

[0048] The functionality of the components in the video hosting server 14 may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuitry, and / or combinations thereof, configured or programmed to perform the functions described herein. A processor is considered to be circuitry or processing circuitry that includes transistors and other circuits. A processor may also be a programmed processor that executes a program stored in a memory.

[0049] In this specification, a circuit, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0050] If the hardware is a processor that is considered to be a type of circuitry, the circuitry, means, or unit may be a combination of hardware and software used to configure the hardware and / or processor.

[0051] The game video acquisition unit 210 acquires videos of games played by multiple players from multiple information processing devices 10, and records the videos in the video recording unit 242. The video recording unit 242 stores the game videos by linking them to the user identifiers (user accounts) of the players.

[0052] The analysis unit 212 includes an AI (artificial intelligence) with an image analysis function. The analysis unit 212 performs image analysis of game videos using a trained model prepared for each game title, acquires game state values, and sets the state values ​​as metadata for the game videos. The analysis unit 212 may have various image analysis functions, as described below.

[0053] <Character Recognition Function> The analysis unit 212 may have an optical character recognition (OCR) function. The analysis unit 212 recognizes text included in the game video and extracts character data. For example, letters and numbers are displayed in the upper left corner of the game screen shown in Figure 4, and the analysis unit 212 uses the character recognition function to extract the letters and numbers included in the game screen.

[0054] FIG. 6 shows examples of letters and numbers extracted from a game image by the analysis unit 212. The analysis unit 212 sets a coordinate system for the game image, with the horizontal axis being the X axis and the vertical axis being the Y axis, and identifies the coordinates of the recognized letters and numbers. In the example shown in FIG. 6, the analysis unit 212 extracts the number "150" at the position (x1, y3), the letters "HP" at the position (x1, y4), the letters "COMBO" at the position (x2, y1), and the number "13" at the position (x2, y2). The analysis unit 212 derives the center of gravity of the letter or number as the coordinate of the letter or number.

[0055] The analysis unit 212 identifies the object indicated by the number from the relative positions of the letters and numbers. In this example, "HP" is displayed near the number displayed at the (x1, y3) position, and "combo" is displayed near the number displayed at the (x2, y2) position. Therefore, the analysis unit 212 identifies that the number displayed at the (x1, y3) position represents "HP (hit points)" and the number displayed at the (x2, y2) position represents "combo number." Therefore, the analysis unit 212 recognizes that the player character's HP at this point is 150 and the combo number is 13.

[0056] In this way, the number displayed at the position (x1, y3) is a status value indicating the HP, and the metadata setting unit 214 records the number displayed at the position (x1, y3) together with time information (time stamp) as metadata in the metadata recording unit 244. By recording the number displayed at the position (x1, y3) together with the time information, the transition of the HP is recorded in the metadata recording unit 244.

[0057] The number displayed at the position (x2, y2) is a status value indicating the combo number, and the metadata setting unit 214 records the number displayed at the position (x2, y2) together with time information as a status value indicating the combo number in the metadata recording unit 244. By recording the number displayed at the position (x2, y2) together with time information, the transition of the combo number is recorded in the metadata recording unit 244.

[0058] <Object Detection Function> The analysis unit 212 may have an object detection function and detect objects in game images. For example, the analysis unit 212 detects items obtained by the player character. When the analysis unit 212 detects an item obtained by the player character, it records the name of the detected item and time information indicating the time of detection in the metadata recording unit 244 as a status value indicating the obtained item.

[0059] The analysis unit 212 may also detect an enemy character that the player character will fight against. When the analysis unit 212 detects an enemy character that the player character will fight against, the analysis unit 212 records the name of the detected enemy character and time information indicating the time of detection in the metadata recording unit 244 as a status value indicating the enemy character.

[0060] <Behavior Recognition Function> The analysis unit 212 may have a behavior recognition function and detect the behavior of the player character in the game video. The analysis unit 212 detects the behavior of the player character in the game video using a trained model prepared for each game title.

[0061] The trained model may be an action detection model that has been trained by machine learning on game videos of multiple players. The action detection model performs machine learning using training game videos and names of actions performed by player characters in the game videos as training data. For the machine learning, a type of deep learning such as CNN, RNN, or LSTM may be used. The trained action detection model is trained to input game videos and output the names of detected actions and the times at which the actions occurred.

[0062] For example, an action detection model trained on game videos of the game title "Battle XZ" detects one of the following actions from the player character's movements: "punch," "kick," "guard," "jump," "run," or "jog." Because the types of player character actions vary depending on the game title, it is preferable to train an action detection model for each game title. Therefore, an action detection model trained on game videos of other game titles may detect different types of actions from those in the game title "Battle XZ." Training an action detection model for each game title makes it possible to detect actions in game videos with high accuracy. The metadata setting unit 214 records the action names and action occurrence times extracted by image analysis of the game videos in the metadata recording unit 244 as state values ​​indicating the actions.

[0063] As described above, the metadata setting unit 214 sets the metadata of the game video. The video sharing server 14 of the embodiment has a function of dividing one game video and expressing the characteristics of the divided multiple videos as vectors.

[0064] The video dividing unit 216 divides the game video. The video dividing unit 216 may divide the game video into predetermined unit times. For example, the video dividing unit 216 divides the game video into 20-second units.

[0065] 7 shows an example of dividing a game video. The game video shown in FIG. 7 is a four-minute video A in which player A plays "Battle XZ." The video dividing unit 216 divides video A into 20-second segments. Therefore, video A is divided into 12 divided videos A1 to A12.

[0066] The vector setting unit 218 has a function of expressing the features of the divided videos as vectors. The vector setting unit 218 may express the features of the divided videos as sentences (text) and vectorize the sentences to express the features of the divided videos as vectors. Specifically, the vector setting unit 218 expresses the features of the divided videos as sentences (text) using metadata of the divided videos and converts the sentences into text vectors using a known conversion technique. A known method such as doc2vec may be used to vectorize the text. By vectorizing the features of the divided videos, it becomes possible to calculate the similarity between the features of two divided videos, and therefore to evaluate the relevance of the two divided videos.

[0067] A method for setting the feature vectors of divided moving images will be described below. (Feature Vectors of Divided Moving Image A1) First, the vector setting unit 218 extracts event information and state values ​​associated with time information (timestamps) between 0:00 and 0:20 from the metadata embedded in moving image A and the metadata recorded in the metadata recording unit 244. The vector setting unit 218 expresses the features of divided moving image A1 in text from the extracted event information and state values.

[0068] For example, if there is no change in HP between 0:00 and 0:20 and HP remains at the same value, the vector setting unit 218 will detect that the player character has not received any damage in the divided video A1. Also, if the number of punches is 15, the number of kicks is 0, the number of guards is 0, and the enemy character is zombie Z, the vector setting unit 218 will detect that the player character has thrown a punch and attacked zombie Z. Therefore, from these metadata, the vector setting unit 218 will convert the characteristics of the divided video A1 into text as follows: "The player character threw 15 punches and attacked zombie Z without receiving any damage."

[0069] The vector setting unit 218 vectorizes the description of the divided video A1 to generate a feature vector A1. The vector setting unit 218 records the generated feature vector A1 in the vector recording unit 246.

[0070] (Feature Vector of Divided Moving Image A2) Next, the vector setting unit 218 extracts event information and state values ​​associated with time information (timestamps) between 0:20 and 0:40 from the metadata embedded in moving image A and the metadata recorded in the metadata recording unit 244. The vector setting unit 218 expresses the features of divided moving image A2 in text from the extracted event information and state values.

[0071] For example, if the HP is reduced to half between 0:20 and 0:40, the vector setting unit 218 will detect that the player character has received significant damage in the divided video A2. If the number of punches is 0, the number of kicks is 0, the number of guards is 6, and the enemy character is zombie Z, the vector setting unit 218 will detect that the player character has been attacked unilaterally by zombie Z and has been unable to counterattack. Therefore, the vector setting unit 218 will convert the characteristics of the divided video A2 from this metadata into text as follows: "The player character managed to avoid the attack from zombie Z by guarding, but his HP was significantly reduced and he was unable to counterattack."

[0072] The vector setting unit 218 vectorizes the description of the divided video A2 to generate a feature vector A2. The vector setting unit 218 records the generated feature vector A2 in the vector recording unit 246.

[0073] The vector setting unit 218 also performs the above vectorization process on the remaining divided videos A3 to A12, and records the feature vectors A3 to A12 of the divided videos A3 to A12 in the vector recording unit 246. In this way, the feature vectors A1 to A12 of the divided videos A1 to A12 obtained by dividing the video A are recorded in the vector recording unit 246. In this embodiment, the video dividing unit 216 divides all of the game videos recorded in the video recording unit 242, and the vector setting unit 218 generates feature vectors of the divided videos (divided videos), and records them in the vector recording unit 246.

[0074] By generating feature vectors for the segmented videos in this manner, the relevance of the two segmented videos can be evaluated based on the similarity between the two feature vectors. The similarity between the two feature vectors may be calculated, for example, using cosine similarity. Cosine similarity may be calculated for all combinations of the two feature vectors recorded in the vector recording unit 246, and a data set indicating the correspondence between the combinations of the two segmented videos and the cosine similarity may be recorded in the vector recording unit 246.

[0075] In the above example, the vector setting unit 218 generates a description indicating the characteristics of the divided video using the metadata embedded in video A and the metadata recorded in the metadata recording unit 244. However, in another example, the vector setting unit 218 may use a VLM (Vision and Language Model) to generate a description indicating the characteristics of the divided video, and vectorize the description generated by the VLM.

[0076] <Provision of Video Sharing Service> As described above, the video sharing server 14 records a plurality of game videos uploaded by a plurality of players in the video recording unit 242, and records feature vectors of a plurality of divided videos constituting the game videos in the vector recording unit 246. A user can operate the information processing device 10 to access the video sharing server 14 and watch the game videos uploaded to the video sharing server 14. The processing procedure when a user watches a game video recorded on the video sharing server 14 will be described below.

[0077] 8 shows an example of a home screen of a video sharing site. A user inputs a string of characters indicating a game video they wish to watch into a search box 300. In this example, the user inputs "video of playing Battle XZ" into the search box 300, and the receiving unit 140 receives the string of characters input by the user. When the user presses a search button 302, the acquiring unit 142 transmits a search request including the search string to the video sharing server 14.

[0078] In the video sharing server 14, the search request receiving unit 220 receives a search request from the information processing device 10, acquires a search string included in the search request, and supplies the acquired string to the search unit 222. The search unit 222 searches for game videos that match the search string from among a plurality of game videos recorded in the video recording unit 242. At this time, the search unit 222 refers to the metadata embedded in the game videos and searches for game videos that include the game ID of the game title "Battle XZ" as metadata.

[0079] When the search unit 222 searches for game videos and identifies multiple candidate videos, the video providing unit 228 transmits thumbnail images of the multiple candidate videos to the information processing device 10. The information processing device 10 displays the multiple thumbnail images on the output device 4 in a selectable manner.

[0080] FIG. 9 shows an example of a selection screen for a plurality of candidate videos. All of these game videos are videos of playing "Battle XZ." The user selects one game video on the selection screen, and the reception unit 140 receives the selection of the game video. It is assumed that the user has selected the game video of player A on the selection screen shown in FIG. 9. When the user selects the game video of player A, the acquisition unit 142 notifies the video sharing server 14 that the game video of player A has been selected.

[0081] In the video sharing server 14, the selection receiving unit 224 receives a selection of a game video from player A. When the selection receiving unit 224 receives the selection of a game video from player A, the video providing unit 228 provides the game video from player A to the information processing device 10. It is assumed that the game video from player A is the four-minute video A shown in FIG. 7 .

[0082] At this time, the related video identification unit 226 identifies the divided video to be played on the information processing device 10, and identifies a game video that includes a divided video that is related to the divided video being played. The divided video that is first played on the information processing device 10 is divided video A1, which is located at the beginning of video A (see FIG. 7). Therefore, the related video identification unit 226 identifies a game video that includes a divided video that is related to divided video A1.

[0083] The vector recording unit 246 records feature vectors of multiple divided videos. The related video identification unit 226 calculates the cosine similarity between the feature vector A1 of the divided video A1 and the feature vectors of other divided videos recorded in the vector recording unit 246, and identifies the feature vectors of a predetermined number of other divided videos with relatively high calculated cosine similarities. For example, the vector recording unit 246 may identify three feature vectors in descending order of cosine similarity with the feature vector A1 of the divided video A1. Note that if the vector recording unit 246 records a dataset indicating the correspondence between all combinations of feature vectors of two divided videos and cosine similarities, the related video identification unit 226 may identify the top three feature vectors with highest cosine similarities for the feature vector A1.

[0084] Here, the cosine similarity between the feature vector A1 and the feature vector H3 of the divided video H3 in video H is the highest, the cosine similarity between the feature vector A1 and the feature vector K7 of the divided video K7 in video K is the second highest, and the cosine similarity between the feature vector A1 and the feature vector S2 of the divided video S2 in video S is the third highest. From the calculation results of the cosine similarities, the related video identification unit 226 recognizes that videos H, K, and S include divided videos that are highly related to the divided video A1. When the related video identification unit 226 identifies videos H, K, and S that include divided videos that are highly related to the divided video A1, the thumbnail providing unit 230 provides thumbnails of videos H, K, and S to the information processing device 10. In this way, in the video sharing server 14, when the video providing unit 228 provides video A of player A selected by the user to the information processing device 10, the thumbnail providing unit 230 provides thumbnails of videos H, K, and S related to the divided video A1 to the information processing device 10.

[0085] In the information processing device 10, the acquisition unit 142 acquires video A from the video sharing server 14, and also acquires thumbnail images of video H, video K, and video S. The video display unit 116 plays video A, and the thumbnail display unit 118 displays thumbnails of video H, video K, and video S.

[0086] 10 shows an example of a game video playback screen. The video display unit 116 plays video A in the first display area 310. The video display unit 116 displays a seek bar that represents the overall time length of the game video as the length of a bar and indicates the playback position of the game video as the position of a slider on the bar. When the acquisition unit 142 acquires video A, the video display unit 116 plays divided video A1 located at the beginning of video A.

[0087] The thumbnail display unit 118 displays in the second display area 312 thumbnails of videos related to the image displayed in the first display area 310. In this example, the thumbnail display unit 118 displays thumbnails 314a, 314b, and 314c of videos H, K, and S related to the divided video A1 displayed in the first display area 310. When the user selects, for example, thumbnail 314a, the video display unit 116 may play video H in the first display area 310.

[0088] In the embodiment, the acquisition unit 142 notifies the video sharing server 14 of the playback status of video A. Specifically, the acquisition unit 142 notifies the video sharing server 14 of time information (playback time) indicating the playback time of video A. In the example shown in Fig. 10, the playback time of video A is 10 seconds, and the acquisition unit 142 notifies the video sharing server 14 of this playback time.

[0089] In the video sharing server 14, the related video identification unit 226 is notified of the playback time of video A and identifies the divided video being played in video A. Referring to Figure 7, since divided video A1 is in the time range of 0:00 to 0:20, if the playback time is 10 seconds, the related video identification unit 226 recognizes that divided video A1 is currently being played. If the playback time exceeds 20 seconds, the related video identification unit 226 recognizes that playback of divided video A1 has ended and playback of divided video A2 has started.

[0090] When a change is made to a divided video being played on the information processing device 10, the related video identification unit 226 identifies a game video including a divided video related to the newly changed divided video. When the related video identification unit 226 recognizes that divided video A2 is being played, it identifies the top three feature vectors with the highest cosine similarity to feature vector A2 of divided video A2.

[0091] Here, the cosine similarity between feature vector A2 and feature vector L4 of divided video L4 in video L is the highest, the cosine similarity between feature vector A2 and feature vector Y5 of divided video Y5 in video Y is the second highest, and the cosine similarity between feature vector A2 and feature vector G2 of divided video G2 in video G is the third highest. From the calculation results of the cosine similarities, the related video identification unit 226 recognizes that video L, video Y, and video G contain divided videos that are highly related to divided video A2. When the related video identification unit 226 identifies videos L, video Y, and video G that contain divided videos that are highly related to divided video A2, the thumbnail providing unit 230 provides thumbnails of videos L, video Y, and video G to the information processing device 10.

[0092] In the information processing device 10, the acquisition unit 142 acquires thumbnail images of video L, video Y, and video G from the video sharing server 14. Fig. 11 shows an example of a playback screen for a game video. The video display unit 116 is playing divided video A2 of video A in the first display area 310. The thumbnail display unit 118 displays thumbnails of videos related to the image displayed in the first display area 310 in the second display area 312. In this example, the thumbnail display unit 118 displays thumbnails 316a, 316b, and 316c of video L, video Y, and video G related to divided video A2 displayed in the first display area 310.

[0093] In this way, thumbnail display unit 118 changes the thumbnail of the related video to be displayed depending on the image displayed by video display unit 116. When video display unit 116 is playing video A, thumbnail display unit 118 displays thumbnails 316a, 316b, and 316c of videos related to the displayed image, so that when the user selects any of thumbnails 316a, 316b, and 316c, video display unit 116 can play a video that is highly related to the image of video A that was being displayed.

[0094] At least a playback start time may be set for a related video. For example, for video L, the start time of divided video L4 may be set as the playback start time, and when the user selects thumbnail 316a, the video display unit 116 may play video L from the start time of divided video L4. This allows the user to immediately watch divided video L4, which is related to divided video A2. Similarly, for video Y, the start time of divided video Y5 is set as the playback start time, and when the user selects thumbnail 316b, the video display unit 116 plays video Y from the start time of divided video Y5. Furthermore, for video G, the start time of divided video G2 is set as the playback start time, and when the user selects thumbnail 316c, the video display unit 116 plays video G from the start time of divided video G2.

[0095] Note that a playback end time may be set for each related video. For video L, the end time may be set to the end time of divided video L4, for video Y, the end time may be set to the end time of divided video Y5, and for video G, the end time may be set to the end time of divided video G2. When the video display unit 116 plays a related video up to the playback end time, it may resume playback of video A, or it may play another related video from its playback start time to its playback end time.

[0096] The above is an example in which the thumbnail display unit 118 changes the thumbnail image displayed in the second display area 312 each time the split video being played switches, but the thumbnail image may not be changed while video A is being played. Instead, the thumbnail display unit 118 may display a thumbnail of a video related to the image being paused and displayed when playback of video A is paused. Furthermore, after the user moves the slider on the seek bar, the thumbnail display unit 118 may display a thumbnail of a video related to the displayed image when playing a video.

[0097] 12 shows another example of a game video playback screen. The video display unit 116 plays video A in the first display area 310. The thumbnail display unit 118 displays thumbnails of multiple split videos of the video A being played in the third display area 318, and displays thumbnails of videos including split videos related to the split videos of video A in the second display area 312. By displaying thumbnails of videos related to the split videos of video A near the thumbnails of the split videos of video A, the user can efficiently watch videos related to the split videos of video A.

[0098] The above describes a playback screen of a game video displayed on a stationary output device 4. However, the output device 4 may also be a head-mounted display (HMD). Hereinafter, it is assumed that the HMD has a function for detecting the user's gaze. When the video display unit 116 is displaying video A on the HMD output device and the user looks away from video A for a predetermined period of time, the thumbnail display unit 118 displays thumbnails of multiple videos related to the image that was displayed when the user looked away. At this time, the video display unit 116 may display video A in a reduced size. In this state, when the user looks at a thumbnail of a related video, the video display unit 116 starts playing the related video, and the thumbnail display unit 118 hides the thumbnail of the related video. When the user wants to watch a video related to the currently displayed image, the user can display a thumbnail of the related video by looking away from the currently displayed image, making it easy to watch the related video.

[0099] The video sharing server 14 can analyze user preferences by monitoring the user's viewing status of related videos. The video sharing server 14 may more accurately understand user preferences by identifying the segmented videos that the user has viewed among the related videos. For example, the video sharing server 14 may refer to the cosine similarity of the related videos viewed by the user to investigate the trend in the cosine similarity of the related videos viewed by the user. For example, if the cosine similarity of the related videos viewed by the user tends to be equal to or greater than a predetermined value, related videos with cosine similarity below the predetermined value may not be presented to the user.

[0100] The present disclosure has been described above based on examples. These examples are merely illustrative, and those skilled in the art will understand that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present disclosure. In the examples, a case has been described in which the information processing device 10 plays game videos, but videos other than game videos may also be played.

[0101] In the embodiment, the video dividing unit 216 divides the game video in 20-second increments, but the video dividing unit 216 may divide the game video in other time increments, for example, in 5-second increments. Furthermore, the video dividing unit 216 may divide the game video in increments that take into account semantic continuity, rather than dividing the game video in predetermined time increments. For example, the video dividing unit 216 may divide the game video in scene increments, such as conversation scenes or battle scenes.

[0102] In a modified example, the information processing system 1 may include a cloud game server (not shown) that executes a game program in accordance with user operations. In this modified example, the information processing device 10 transmits information (game operation information) about user operations on the input device 6 to the game server. The game server generates game images and game sounds based on the game operation information and streams the game images and game sounds to the information processing device 10. In this modified example, the information processing device 10 does not need to have a function for executing a game program, and may be a terminal device that outputs game images and game sounds from the output device 4.

[0103] The present disclosure may include the following aspects. [Item 1] An information processing device having a function to play videos, comprising a circuit configured to: play a video; display a thumbnail of a video related to a displayed image; and, when the displayed image is changed, display a thumbnail of a video related to the changed image. [Item 2] The information processing device according to item 1, wherein, when playback of a video is paused, the circuit displays a thumbnail of a video related to the image displayed when playback is paused. [Item 3] The information processing device according to item 1, wherein, while playing a video, the circuit displays a thumbnail of a video related to the displayed image. [Item 4] The information processing device according to item 1, wherein the circuit displays multiple thumbnails of a video being played, and displays thumbnails of videos related to the thumbnails near the multiple thumbnails. [Item 5] The information processing device according to item 1, wherein, when a thumbnail of a related video is selected, the circuit plays the related video. [Item 6] The information processing device according to Item 5, wherein a playback start time is set for the related video, and the circuit plays the related video from the set playback start time. [Item 7] A method for displaying videos in an information processing device, comprising: playing a video; displaying a thumbnail of a video related to a displayed image; and, when the displayed image is changed, displaying a thumbnail of a video related to the changed image. [Item 8] A recording medium having recorded thereon a program executed on a computer of an information processing device, the program causing the computer to realize the following functions: playing a video; displaying a thumbnail of a video related to the displayed image; and, when the displayed image is changed, displaying a thumbnail of a video related to the changed image.

[0104] The present disclosure can be used in the technical field of allowing users to watch videos.

[0105] 1...information processing system, 4...output device, 5...server device, 6...input device, 10...information processing device, 12...management server, 14...video sharing server, 100...processing unit, 102...communication unit, 110...game execution unit, 112...game image generation unit, 114...display processing unit, 116...video display unit, 118...thumbnail display unit, 120...event information acquisition unit, 122...event data transmission unit, 130...game video acquisition unit, 132...game video transmission unit, 140...reception unit, 142 ...Acquisition unit, 154...Game video recording unit, 200...Processing unit, 202...Communication unit, 210...Game video acquisition unit, 212...Analysis unit, 214...Metadata setting unit, 216...Video division unit, 218...Vector setting unit, 220...Search request reception unit, 222...Search unit, 224...Selection reception unit, 226...Related video identification unit, 228...Video providing unit, 230...Thumbnail providing unit, 240...Recording device, 242...Video recording unit, 244...Metadata recording unit, 246...Vector recording unit.

Claims

1. An information processing device having a function to play videos, comprising: a video display unit that plays videos; and a thumbnail display unit that displays thumbnails of videos related to a displayed image, wherein the thumbnail display unit changes the thumbnail of the video to be displayed according to the image displayed by the video display unit.

2. The information processing device according to claim 1, characterized in that, when the video display unit pauses playback of a video, the thumbnail display unit displays a thumbnail of a video related to the image currently being displayed while playback is paused.

3. The information processing device according to claim 1, wherein, when the video display unit is playing a video, the thumbnail display unit displays a thumbnail of a video related to the image being displayed.

4. The information processing device according to claim 1, characterized in that the thumbnail display unit displays multiple thumbnails of the video being played, and also displays thumbnails of videos related to the multiple thumbnails in the vicinity of the multiple thumbnails.

5. The information processing device according to claim 1, wherein the video display unit plays the related video when a thumbnail of the related video is selected.

6. The information processing device according to claim 5, characterized in that a playback start time is set for the related video, and the video display unit plays the related video from the set playback start time.

7. A method for displaying moving images in an information processing device, comprising: a step of playing the moving image; a step of displaying a thumbnail of the moving image related to the displayed image; and a step of, when the displayed image is changed, displaying a thumbnail of the moving image related to the changed image.

8. A program for enabling a computer to perform the following functions: play videos; display thumbnails of videos related to the displayed image; and, when the displayed image is changed, display thumbnails of videos related to the changed image.

Images