Data processing apparatus, data processing method, and storage medium

Abstract

To provide a data processing apparatus, a data processing method, and a program that make it easier for a viewer to select a virtual viewpoint image intended by the viewer than in a case where only first data regarding a reproduction history and / or registration data of a virtual viewpoint image is used in a device.SOLUTION: In a data processing apparatus that performs communication with a plurality of user devices 12 through a communication network such as a LAN or a WAN, a video generating unit 26 performs control such that a virtual viewpoint image generated on the basis of a plurality of captured images is transmitted to the user device 12 in response to a request from each user device 12, first data regarding a reproduction history and / or registration data of the virtual viewpoint image is acquired, and second data regarding the virtual viewpoint image is transmitted to the user device 12 on the basis of the acquired first data.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a data processing device, a data processing method, and a program. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2020-009021 discloses a data processing device including a setting means and a generating means. The setting means sets a first virtual viewpoint position for generating a virtual viewpoint image based on multi-viewpoint images obtained from a plurality of cameras. The generating means generates viewpoint position information indicating a second virtual viewpoint position that differs from the first virtual viewpoint position set by the setting means in at least one of position and orientation and corresponds to a common time as the first virtual viewpoint position, based on the first virtual viewpoint position set by the setting means.

[0003] Japanese Patent Application Laid-Open Publication No. 2018-036956 discloses a display control device having a receiving means and a display control means. The receiving means receives a virtual viewpoint position in response to a user's operation. The display control means controls the display of a first virtual viewpoint image generated from a captured image and a second virtual viewpoint image having a wider virtual shooting range than the first virtual viewpoint image on a display screen based on the virtual viewpoint position.

[0004] Japanese Patent Application Laid-Open Publication No. 2015-187797 discloses an image data generating device that acquires viewpoint image data captured from at least one viewpoint position and generates free viewpoint image data using the acquired viewpoint image data, enabling playback of an image from any viewpoint position. The image data generating device is characterized by including an object placement data generating unit, a free viewpoint position metadata generating unit, an encoding unit, and a multiplexing unit. The object placement data generating unit generates object coordinate data indicating the placement position of an object in background model data created as image data by modeling a background. The free viewpoint position metadata generating unit generates recommended viewpoint position placement data including at least one of viewpoint position coordinate data indicating the position serving as the base point of the viewpoint position and rotation base point data indicating the initial position of the viewpoint position direction and indicating the rotation center when rotating the viewpoint position to display the free viewpoint position. The device further generates free viewpoint position metadata from the object coordinate data and the recommended viewpoint position placement data. The encoding unit encodes the viewpoint image data and area image data for identifying the area of ​​each object in the viewpoint image data. The multiplexing unit multiplexes the free viewpoint position metadata, the viewpoint image data, the area image data, and the background model data to generate free viewpoint image data. Summary of the Invention

[0005] One embodiment of the technology disclosed herein provides a data processing device, a data processing method, and a program that make it easier for a viewer to select the virtual viewpoint image they want, compared to when only first data related to the playback history and / or registration data of the virtual viewpoint image is used on the device. [Means for solving the problem]

[0006] A first aspect of the technology of the present disclosure is a data processing device that includes a processor and a memory built into or connected to the processor, and that transmits a virtual viewpoint image generated based on a captured image to a device, wherein the processor acquires first data related to the playback history and / or registration data of the virtual viewpoint image, and controls the transmission of second data related to the virtual viewpoint image to the device based on the acquired first data.

[0007] A second aspect of the technology of the present disclosure is a method for generating a virtual viewpoint image associated with first data by a processor. 1 is a data processing device according to a first aspect, which performs control to transmit second data relating to a second virtual viewpoint image that is a virtual viewpoint image different from the first virtual viewpoint image.

[0008] A third aspect of the technology of the present disclosure is the data processing device according to the second aspect, in which the first data includes viewpoint-related information related to a viewpoint of the virtual viewpoint image.

[0009] A fourth aspect of the technology disclosed herein is a data processing device according to the third aspect, in which the viewpoint-related information includes at least one of a viewpoint position, a line of sight direction, an angle of view, a gaze point, a viewpoint position path indicating a displacement of the viewpoint position over time, and a gaze point path indicating a displacement of the gaze point over time.

[0010] A fifth aspect of the technique of the present disclosure is the data processing device according to the fourth aspect, in which the first data includes first time information relating to the capture time of a captured image used to generate the virtual viewpoint image.

[0011] A sixth aspect of the technology of the present disclosure is the data processing device according to the fifth aspect, in which the second data includes at least one of a reduced image of a virtual viewpoint image and viewpoint-related information.

[0012] A seventh aspect of the technology of the present disclosure is a data processing device according to the sixth aspect, in which the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in order of increasing similarity with respect to the viewpoint-related information contained in the first data and decreasing similarity with respect to the first time information contained in the first data.

[0013] An eighth aspect of the technology of the present disclosure is a data processing device according to the sixth aspect, in which the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in order of low similarity with respect to viewpoint-related information contained in the first data and high similarity with respect to first time information contained in the first data.

[0014] A ninth aspect of the technology of the present disclosure is a data processing device according to the seventh or eighth aspect, in which a processor obtains a similarity regarding viewpoint-related information based on a difference in the position of the viewpoint position path or the gaze point path between the viewpoint-related information included in the first data and the viewpoint-related information included in the second data.

[0015] A tenth aspect of the technology of the present disclosure is a data processing device according to the seventh or eighth aspect, in which a processor obtains similarity regarding viewpoint-related information based on differences in tags assigned to the viewpoint position path or the gaze point path.

[0016] An eleventh aspect of the technology of the present disclosure is a data processing device according to any one of the sixth to tenth aspects, in which the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in an order determined based on the similarity between the attributes of a viewer who viewed the first virtual viewpoint image using the device and the attributes of the creator of the second virtual viewpoint image.

[0017] A twelfth aspect of the technology of the present disclosure is a data processing device according to any one of the sixth to eleventh aspects, in which the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in an order determined based on the difference between the playback history of the virtual viewpoint image by a viewer who viewed the first virtual viewpoint image using the device and the playback history of the virtual viewpoint image by another viewer.

[0018] In a thirteenth aspect of the technology of the present disclosure, the second data is displayed as a first virtual viewpoint image using a device. The data processing device is any one of the sixth to twelfth aspects, in which at least one of the reduced image and the viewpoint-related information is data displayed on the device in an order determined based on a playback tendency extracted from the playback history of virtual viewpoint images by a viewer who viewed the image.

[0019] A fourteenth aspect of the technology of the present disclosure is a data processing device according to any one of the sixth to thirteenth aspects, in which the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in an order determined based on comments added to each of a plurality of second virtual viewpoint images.

[0020] A fifteenth aspect of the technology disclosed herein is a data processing device according to the fourteenth aspect, in which the comment is a comment attached to a viewpoint position path included in the second data indicating a displacement of the viewpoint position over time, or a gaze point path included in the second data indicating a displacement of the gaze point over time, and the second data is data in which an image with the comment attached in the form of a viewpoint position path or a gaze point path is displayed on the device.

[0021] A 16th aspect of the technology of the present disclosure is a data processing device according to any one of the 6th to 15th aspects, in which the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in an order determined based on a first evaluation assigned to each of a plurality of second virtual viewpoint images.

[0022] A 17th aspect of the technology of the present disclosure is a data processing device according to any one of the 6th to 16th aspects, in which the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in an order determined according to the viewer's preferences estimated based on a second evaluation of the first virtual viewpoint image.

[0023] An 18th aspect of the technology of the present disclosure is a data processing device according to any one of the 6th to 17th aspects, in which when a new virtual viewpoint image is added, the second data is data that, based on the first data, displays on the device at least one of a reduced image of the added virtual viewpoint image and viewpoint-related information.

[0024] A 19th aspect of the technology of the present disclosure is a data processing device according to any one of the 6th to 18th aspects, in which the virtual viewpoint image is a video, and the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in an order determined based on the length of a viewpoint position path indicating the displacement of the viewpoint position included in the first data over time relative to the time from the start to the end of playback of the first virtual viewpoint image, or the length of a gaze point path indicating the displacement of the gaze point included in the first data over time.

[0025] A twentieth aspect of the technology of the present disclosure is a data processing device according to any one of the sixth to nineteenth aspects, in which the virtual viewpoint image is a video, and the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in an order determined based on the ratio between second time information relating to the capture time of the captured image used to generate the second virtual viewpoint image and the time from the start to the end of playback of the second virtual viewpoint image.

[0026] A 21st aspect of the technology of the present disclosure is a data processing device according to any one of the 6th to 20th aspects, in which the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in an order determined based on the data capacity of each of a plurality of second virtual viewpoint images.

[0027] In a 22nd aspect of the technique of the present disclosure, the second data is a plurality of second virtual viewpoint images. A data processing device according to any one of the sixth to twenty-first aspects, in which at least one of reduced images and viewpoint-related information is displayed on the device in an order determined based on parameters related to image quality.

[0028] A 23rd aspect of the technology of the present disclosure is a data processing device according to any one of the 6th to 22nd aspects, in which the second data is data in which at least one of a reduced image and viewpoint-related information is displayed on the device in an order determined based on audio data assigned to the second virtual viewpoint image.

[0029] A 24th aspect of the technology of the present disclosure is a data processing device according to any one of the 6th to 23rd aspects, in which the second data is data to be played on a device, in which a plurality of second virtual viewpoint images are played on the device sequentially on the condition that the processor receives a playback instruction instructing the playback of the second virtual viewpoint images.

[0030] A 25th aspect of the technology disclosed herein is a data processing device according to the 24th aspect, in which the second data is data in which a first image different from the first virtual viewpoint image and the second virtual viewpoint image is played on the device each time playback of each of the plurality of second virtual viewpoint images is completed.

[0031] A 26th aspect of the technology of the present disclosure is a data processing device according to any one of the 2nd to 25th aspects, in which the second data is data in which a second image that is a condition for playing a second virtual viewpoint image is played on a device.

[0032] A 27th aspect of the technology of the present disclosure is a data processing device according to the 26th aspect, in which the second data is data for a third image that is a condition for reproducing a second virtual viewpoint image based on image information regarding the content of the second image transmitted to the device and that is reproduced on the device.

[0033] A 28th aspect of the technology of the present disclosure is a data processing device according to any one of the 2nd to 27th aspects, in which the second data is data for playing on a device a composite image obtained by combining a second virtual viewpoint image selected from a plurality of second virtual viewpoint images in accordance with given instructions with a viewpoint position path indicating the displacement of the viewpoint position of other second virtual viewpoint images over time, or a gaze point path indicating the displacement of the gaze point over time.

[0034] A 29th aspect of the technology disclosed herein is a data processing device according to the 28th aspect, in which, when a path change instruction for a viewpoint position path or a gaze point path is received, the second data is data that stops playback of the synthetic image and plays on the device a second virtual viewpoint image corresponding to the viewpoint position path or the gaze point path included in the path change instruction.

[0035] A 30th aspect of the technology of the present disclosure is a data processing device according to any one of the second to 29th aspects, in which, when a change instruction is received for viewpoint-related information related to the viewpoint of a second virtual viewpoint image being played on a device, the second data is data that is played on the device as a virtual viewpoint image generated based on new viewpoint-related information derived based on the change instruction.

[0036] A 31st aspect of the technology of the present disclosure is a data processing device according to any one of the 2nd to 30th aspects, in which, when a replay image generation instruction is received for a second virtual viewpoint image being played on a device, the second data is data in which a virtual viewpoint image generated based on new viewpoint-related information generated based on viewpoint-related information related to the viewpoint of the second virtual viewpoint image is played on the device.

[0037] A thirty-second aspect of the technique of the present disclosure is to provide a method for refusing to view an object by displaying information on the object that is to be viewed. The data processing device is one of the second to thirty-first aspects, in which, when the information is received, the second data is data in which a virtual viewpoint image generated based on viewpoint-related information generated based on the object-to-be-viewed information is reproduced on the device.

[0038] A 33rd aspect of the technology disclosed herein is a data processing device according to any one of the second to 32nd aspects, in which the second virtual viewpoint image is a video including multiple frame images, and when a digest image display instruction is received for a reduced image of the second virtual viewpoint image included in the second data, the second data is data that is played on the device as a digest image generated by extracting multiple frame images having different viewpoint positions at predetermined intervals from the second virtual viewpoint image.

[0039] A 34th aspect of the technology of the present disclosure is a data processing device relating to any one of the 2nd to 33rd aspects, in which a processor controls the transmission to a device of a path image in which the shape of a viewpoint position path indicating the displacement of the viewpoint position of the first virtual viewpoint image over time, or the shape of a gaze point path indicating the displacement of the gaze point of the first virtual viewpoint image over time, is displayed in correspondence with second data, and the path image is an image in which the shape of the viewpoint position path or the shape of the gaze point path is displayed in a manner that is distinguishable from the second data.

[0040] A 35th aspect of the technology disclosed herein is a data processing device according to any one of the second to 34th aspects, in which the second data includes a reduced image of the second virtual viewpoint image and a shape of a viewpoint position path indicating the displacement of the viewpoint position of the second virtual viewpoint image over time, or a shape of a gaze point path indicating the displacement of the gaze point of the second virtual viewpoint image over time, and the second data includes first position information capable of identifying a position on the viewpoint position path or gaze point path of the reduced image.

[0041] A 36th aspect of the technology of the present disclosure is a data processing device according to any one of the 2nd to 35th aspects, wherein the second data includes second position information in which the second data related to the second virtual viewpoint image is arranged horizontally in chronological order and arranged vertically according to the similarity of the viewpoint-related information of the second virtual viewpoint image to the viewpoint-related information of the first virtual viewpoint image.

[0042] A 37th aspect of the technology of the present disclosure is a data processing device according to any one of the 2nd to 36th aspects, in which the registration data includes attribute data indicating the attributes of the viewer who viewed the first virtual viewpoint image, and preference data indicating the viewer's preferences.

[0043] A 38th aspect of the technology of the present disclosure is a data processing device according to the 37th aspect, in which the second data is data from which data relating to virtual viewpoint images that are subject to viewing restrictions among the virtual viewpoint images has been excluded based on attribute data.

[0044] A 39th aspect of the technology of the present disclosure is a data processing device relating to any one of the 2nd to 38th aspects, in which, when a time setting instruction is received for third time information relating to the capture time of an image used to generate a virtual viewpoint image, the second data is data relating to a second virtual viewpoint image, among a plurality of second virtual viewpoint images, whose third time information is within the range indicated by the time setting instruction.

[0045] A fortieth aspect of the technology disclosed herein is a data processing device according to the thirty-ninth aspect, in which, when a time setting instruction is accepted, the second data includes the number of time-limited virtual viewpoint images among the virtual viewpoint images whose third time information is within the range indicated by the time setting instruction, and the number of images included in the second virtual viewpoint image among the time-limited virtual viewpoint images.

[0046] A forty-first aspect of the technique of the present disclosure is a method for controlling a virtual viewpoint image by a processor. A data processing device according to any one of the first to fortieth aspects, which is capable of acquiring a playback history of a virtual viewpoint image for an associated account and outputting the playback history in association with viewpoint-related information related to the viewpoint of the virtual viewpoint image.

[0047] A 42nd aspect of the technology of the present disclosure is a data processing device according to any one of the 1st to 41st aspects, in which the processor is capable of tallying up how many times each virtual viewpoint image has been played and outputting the tallying results in association with viewpoint-related information related to the viewpoint of the virtual viewpoint image.

[0048] A 43rd aspect of the technology of the present disclosure is a data processing device relating to any one of the 1st to 42nd aspects, in which a processor controls transmission to a device of a path image in which the shape of a viewpoint position path indicating the displacement of the viewpoint position of a virtual viewpoint image over time, or the shape of a gaze point path indicating the displacement of the gaze point of the virtual viewpoint image over time, is displayed in correspondence with second data, and the path image is an image in which the viewpoint position path or gaze point path is displayed in a manner in which the played and unplayed parts of the virtual viewpoint image can be distinguished.

[0049] A 44th aspect of the technology of the present disclosure is a data processing device according to any one of the 1st to 43rd aspects, in which the first virtual viewpoint image includes a virtual viewpoint image, at least a portion of which has already been played on the device.

[0050] A 45th aspect of the technology of the present disclosure is a data processing device that includes a processor and a memory built into or connected to the processor, and that transmits to a device a plurality of virtual viewpoint images generated based on captured images, wherein the processor acquires first data regarding a first virtual viewpoint image, which is a virtual viewpoint image of the plurality of virtual viewpoint images, at least a portion of which has already been transmitted to the device, and controls the transmission to the device of second data regarding a second virtual viewpoint image, which is a virtual viewpoint image of the plurality of virtual viewpoint images that has not yet been transmitted to the device, based on the acquired first data.

[0051] A 46th aspect of the technology of the present disclosure is a data processing method for transmitting a virtual viewpoint image generated based on a captured image to a device, the data processing method including acquiring first data related to the playback history and / or registration data of the virtual viewpoint image, and controlling the transmission of second data related to the virtual viewpoint image to the device based on the acquired first data.

[0052] A 47th aspect of the technology of the present disclosure is a program for causing a computer to execute data processing for transmitting a virtual viewpoint image generated based on a captured image to a device, the data processing being a process including acquiring first data relating to the playback history and / or registration data of the virtual viewpoint image, and controlling the transmission of second data relating to the virtual viewpoint image to the device based on the acquired first data. [Brief explanation of the drawings]

[0053] [Figure 1] FIG. 1 is a conceptual diagram showing an example of a relationship between a data processing device and a user device. [Figure 2] 1 is a block diagram showing an example of an electrical hardware configuration of a data processing device and an example of a relationship with a user device. [Figure 3] FIG. 2 is a conceptual diagram illustrating an example of a menu screen displayed on the data processing device. [Figure 4] FIG. 2 is a block diagram showing an example of main functions of a moving image generating unit according to the first embodiment. [Figure 5] FIG. 3 is a conceptual diagram showing an example of a generated moving image list according to the first embodiment. [Figure 6] FIG. 2 is a conceptual diagram showing an example of a played moving image list according to the first embodiment. [Figure 7] FIG. 2 is a conceptual diagram showing an example of main functions of a video playback section according to the first embodiment. [Figure 8] FIG. 2 is a conceptual diagram showing an example of main functions of a video playback unit according to the first embodiment. [Figure 9]FIG. 2 is a conceptual diagram showing an example of a recommended video list according to the first embodiment. [Figure 10] FIG. 4 is a conceptual diagram showing an example of a recommendation screen according to the first embodiment displayed on a user device. [Figure 11] FIG. 2 is a conceptual diagram showing an example of main functions of a video playback unit according to the first embodiment. [Figure 12] FIG. 10 is a conceptual diagram showing an example of how a played virtual viewpoint video is added to a played video list according to the first embodiment. [Figure 13] 10 is a flowchart showing an example of the flow of a moving image generation process according to the first embodiment. [Figure 14] 10 is a flowchart showing an example of the flow of a video playback process according to the first embodiment. [Figure 15] 10 is a conceptual diagram showing an example of a viewpoint position path, a gaze point, and an angle of view set on a viewpoint-related information acceptance screen according to the second embodiment. FIG. [Figure 16] FIG. 11 is a conceptual diagram showing an example of a generated moving image list according to the second embodiment. [Figure 17] FIG. 11 is a conceptual diagram showing an example of a played moving image list according to the second embodiment. [Figure 18] FIG. 10 is a conceptual diagram showing an example of main functions of a video playback section according to a second embodiment. [Figure 19] FIG. 10 is a conceptual diagram showing an example of a recommendation screen according to the second embodiment displayed on a user device. [Figure 20] FIG. 10 is a conceptual diagram showing an example of main functions of a video playback section according to a second embodiment. [Figure 21] FIG. 10 is a conceptual diagram showing an example of how a digest image is displayed on a user device. [Figure 22] FIG. 10 is a conceptual diagram showing an example of a tag added to a viewpoint position path. [Figure 23] FIG. 10 is a conceptual diagram showing an example of an aspect in which the shape of a viewpoint position path related to a played virtual viewpoint video is displayed on a recommendation screen according to the second embodiment displayed on a user device. [Figure 24]FIG. 10 is a conceptual diagram showing another example of the recommendation screen according to the second embodiment displayed on the user device. [Figure 25] FIG. 10 is a conceptual diagram showing yet another example of the recommendation screen according to the second embodiment displayed on the user device. [Figure 26] FIG. 10 is a conceptual diagram showing an example of a mode in which recommendation data is generated using attribute data registered in association with an account. [Figure 27] FIG. 10 is a conceptual diagram showing an example of a mode in which recommended data is generated based on playback trends shown in a played video list. [Figure 28] FIG. 10 is a conceptual diagram showing an example of a mode in which recommended data is generated based on a playback history shown in a played video list. [Figure 29] FIG. 10 is a conceptual diagram showing an example of a mode in which recommended data is generated based on a comment added to a viewpoint position path. [Figure 30] FIG. 10 is a conceptual diagram illustrating an example of a recommendation screen showing recommended data generated based on a comment added to a viewpoint position path. [Figure 31] FIG. 10 is a conceptual diagram showing an example of a mode in which recommendation data is generated based on evaluations given to virtual viewpoint videos by users who have viewed the virtual viewpoint videos. [Figure 32] FIG. 10 is a conceptual diagram showing an example of a mode in which recommendation data is generated based on an evaluation given to a virtual viewpoint video that has been viewed by a user using a user device that is the sender of a video playback instruction. [Figure 33] FIG. 10 is a conceptual diagram showing an example of a mode in which a user is notified that a new virtual viewpoint video has been added, based on the user's preference data. [Figure 34] FIG. 10 is a conceptual diagram showing an example of a mode in which recommended data is generated based on the ratio of the viewpoint position path to the length of the video. [Figure 35] FIG. 10 is a conceptual diagram showing an example of a mode in which recommended data is generated based on the ratio of the length of the game elapsed time to the length of the video. [Figure 36] FIG. 10 is a conceptual diagram illustrating an example of a recommendation screen showing recommendation data generated based on the ratio of the length of the game elapsed time to the length of the video. [Figure 37] FIG. 10 is a conceptual diagram showing an example of a mode in which recommended data is generated based on the data capacity of a virtual viewpoint video. [Figure 38] FIG. 10 is a conceptual diagram showing an example of a mode in which recommended data is generated based on the resolution of a virtual viewpoint video. [Figure 39] FIG. 10 is a conceptual diagram showing an example of an aspect in which recommended data is generated based on the presence or absence of audio data added to a virtual viewpoint video. [Figure 40] FIG. 10 is a conceptual diagram showing an example of how virtual viewpoint videos are played back sequentially. [Figure 41] FIG. 10 is a conceptual diagram showing an example of a manner in which, when virtual viewpoint videos are played back sequentially, another video is inserted between the virtual viewpoint videos. [Figure 42] FIG. 10 is a conceptual diagram showing an example of an aspect in which an advertising video is inserted between virtual viewpoint videos when the virtual viewpoint videos are played back sequentially. [Figure 43] FIG. 10 is a conceptual diagram showing an example of an aspect in which an advertising video is displayed during playback of a virtual viewpoint video. [Figure 44] FIG. 10 is a conceptual diagram showing an example of how an advertising video is changed based on advertising information. [Figure 45] FIG. 10 is a conceptual diagram illustrating an example of a manner in which a composite video is displayed on a user device. [Figure 46] FIG. 10 is a conceptual diagram showing an example of how the virtual viewpoint video displayed on the user device is switched to a virtual viewpoint video having a different viewpoint position path based on a path change instruction. [Figure 47] 10 is a conceptual diagram showing an example of switching the viewpoint position of a virtual viewpoint video displayed on a user device based on an instruction to change the viewpoint position. FIG. [Figure 48] FIG. 10 is a conceptual diagram showing an example of how a replay video is played back based on a replay video generation instruction. [Figure 49] FIG. 10 is a conceptual diagram showing an example of a manner in which a virtual viewpoint video generated based on a new viewpoint position path corresponding to the viewing refusal subject information is played back. [Figure 50]FIG. 10 is a conceptual diagram showing an example of a mode in which playback of a virtual viewpoint video is restricted based on user attribute data. [Figure 51] FIG. 10 is a conceptual diagram showing an example of a mode in which recommendation data is generated based on a time setting instruction. [Figure 52] FIG. 10 is a conceptual diagram illustrating an example of a recommendation screen showing recommended data generated based on a time setting instruction. [Figure 53] FIG. 10 is a conceptual diagram showing an example of how a virtual viewpoint video that is viewed during playback is added to a list of played videos. [Figure 54] FIG. 10 is a conceptual diagram showing an example of a generated moving image list including viewing data. [Figure 55] FIG. 10 is a conceptual diagram illustrating an example of a management screen displayed on a tablet terminal. [Figure 56] FIG. 10 is a conceptual diagram showing an example of a management screen displayed on a user device of each user. [Figure 57] FIG. 1 is a conceptual diagram showing an example of how an operating program stored in a storage medium is installed in a computer of a data processing device. DETAILED DESCRIPTION OF THE INVENTION

[0054] An example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0055] First, the terms used in the following description will be explained.

[0056] CPU is an abbreviation for "Central Processing Unit". RAM is an abbreviation for "Random Access Memory". SSD is an abbreviation for "Solid State Drive". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". I / F is an abbreviation for "Interface". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". GUI is an abbreviation for "Graphical User Interface". SoC is an abbreviation for "System-on-a-chip". GPU is an abbreviation for "Graphics Processing Unit". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network". 3D is an abbreviation for "3 Dimensions." USB is an abbreviation for "Universal Serial Bus." ID is an abbreviation for "Identification." NVM is an abbreviation for "Non-volatile Memory."

[0057] For ease of explanation, a CPU is used below as an example of a "processor" according to the technology of the present disclosure; however, the "processor" according to the technology of the present disclosure may be a combination of multiple processing devices, such as a CPU and a GPU. When a combination of a CPU and a GPU is applied as an example of a "processor" according to the technology of the present disclosure, the GPU operates under the control of the CPU and is responsible for executing image processing. In the following explanation, "match" refers not only to a perfect match, but also to a match that includes an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure.

[0058] [First embodiment] As an example, as shown in FIG. 1, a data processing device 10 communicates with multiple user devices 12 via a communication network such as a LAN or WAN. The data processing device 10 performs video generation processing and video playback processing in response to requests from each user device 12. Note that in the example shown in FIG. 1, a smart device is illustrated as the user device 12, but the user device 12 is not limited to this and may be, for example, a personal computer, a tablet terminal, or a television with an Internet communication function. The data processing device 10 is an example of a "data processing device" according to the technology of the present disclosure. The user device 12 is an example of a "device" according to the technology of the present disclosure.

[0059] The video generation process is a process of generating a virtual viewpoint video 30 in response to a video generation request 50 (see FIG. 3) from each user device 12, and transmitting data of the generated virtual viewpoint video 30 to the user device 12 that issued the request. Unless otherwise specified below, data is transmitted and received within the data processing device 10, the user device 12, and between the data processing device 10 and the user device 12. In addition to being transmitted to the user device 12 that issued the request, the virtual viewpoint video 30 is also stored in a database 22 (see FIG. 2) within the data processing device 10. The virtual viewpoint video 30 is a video of the imaging area observed from a virtual viewpoint, and is a virtual video generated using 3D polygons. The virtual viewpoint video 30 is an example of a "virtual viewpoint image" according to the technology of the present disclosure.

[0060] The video playback process is a process of generating data related to a virtual viewpoint video 30 (hereinafter also referred to as an "untransmitted virtual viewpoint video 30") that has not been transmitted to a user device 12 that has received a video playback request 52 (see FIG. 3) from among the multiple virtual viewpoint videos 30 stored in the database 22, and transmitting the generated data to the user device 12 that has made the request. In the first embodiment, an example of the "data related to the untransmitted virtual viewpoint video 30" will be described, in which recommended data 32 that simplifies the user's selection of a virtual viewpoint video 30 is generated. The untransmitted virtual viewpoint video 30 is an example of a "second virtual viewpoint image" according to the technology of the present disclosure. The recommended data 32 is an example of "second data" according to the technology of the present disclosure.

[0061] The user device 12 plays the virtual viewpoint video 30 received from the data processing device 10 on a video playback screen 33 provided on the touch panel display 12A. The user device 12 also displays recommended data 32 below the video playback screen 33 on the touch panel display 12A. The data processing device 10 is a device equivalent to a server, and the user device The device 12 is a device equivalent to a client terminal for the data processing device 10.

[0062] The multiple user devices 12 are respectively used by multiple users A, B, ... Z. Each of the users A, B, ... Z uses the user device 12 to view the virtual viewpoint video 30 received from the data processing device 10. The users A, B, ... Z are examples of "viewers" according to the technology of the present disclosure. Hereinafter, when it is not necessary to distinguish between the users A, B, ... Z, they will be collectively referred to as "users" without being assigned reference numerals.

[0063] As an example, as shown in Fig. 2, a data processing device 10 includes a computer 14 and a communication I / F 16. The computer 14 includes a CPU 14A, an NVM 14B, and a RAM 14C, and the CPU 14A, the NVM 14B, and the RAM 14C are connected via a bus 20. In the example shown in Fig. 2, for convenience of illustration, a single bus is shown as the bus 20, but multiple buses may be used. Furthermore, the bus 20 may include a serial bus or a parallel bus configured of a data bus, an address bus, a control bus, etc.

[0064] The CPU 14A controls the entire data processing device 10. The NVM 14B is a non-volatile memory. Here, an EEPROM, an SSD, and an HDD are used as examples of the NVM 14B, but the NVM 14B is not limited to these and may be an HDD, an SSD, an EEPROM, or a combination of multiple non-volatile memories. The NVM 14B includes a database 22 that stores the virtual viewpoint video 30 generated by the data processing device 10. The NVM 14B is an example of a "memory" according to the technology of the present disclosure.

[0065] Various data are temporarily stored in the RAM 14C, which is used as a work memory by the CPU 14A.

[0066] An operating program 24 is stored in NVM 14B. CPU 14A reads out the operating program 24 from NVM 14B and executes the read operating program 24 on RAM 14C. CPU 14A performs a video generation process by operating as a video generation unit 26 and a video playback process by operating as a video playback unit 28 in accordance with the operating program 24 executed on RAM 14C. The operating program 24 is an example of a "program" according to the technology of the present disclosure. Computer 14 is an example of a "computer" according to the technology of the present disclosure. CPU 14A is an example of a "processor" according to the technology of the present disclosure.

[0067] In response to a request from each user device 12, the video generation unit 26 generates a virtual viewpoint video 30 based on a captured video 34 (see FIG. 4) obtained by capturing an image of an imaging area from multiple directions using multiple imaging devices (not shown). Note that instead of generating the virtual viewpoint video 30 itself, the video generation unit 26 may acquire a virtual viewpoint video 30 generated by an external server or the like. In the example shown in FIG. 2, the multiple imaging devices are installed in a soccer stadium and capture an image of an area within the soccer stadium as an imaging area. The virtual viewpoint video 30 is a video of the soccer stadium observed from any viewpoint within the soccer stadium. The video generation unit 26 stores the generated virtual viewpoint video 30 in the database 22. The database 22 stores multiple virtual viewpoint videos 30 generated by the video generation unit 26 in response to requests from multiple user devices 12.

[0068] In response to a request from each user device 12, the video playback unit 28 generates recommendation data 32 related to an untransmitted virtual viewpoint video 30 among the plurality of virtual viewpoint videos 30 stored in the database 22. The video playback unit 28 stores the generated recommendation data 32 in the database 22.

[0069] The communication I / F 16 is communicatively connected to the user device 12 via a communication network. The communication I / F 16 is realized, for example, by a device having an FPGA. The communication I / F 16 is connected to a bus 20. The communication I / F 16 controls the exchange of various data between the CPU 14A and the user device 12 by wireless communication. Note that the communication I / F 16 can also be configured as a fixed circuit instead of an FPGA. The communication I / F 16 may also be a circuit configured with an ASIC, FPGA, and / or PLD, etc.

[0070] The CPU 14A reads the virtual viewpoint video 30 and the recommended data 32 from the database 22 and transmits the read virtual viewpoint video 30 and the recommended data 32 to the user device 12 via the communication I / F 16. The user device 12 receives the virtual viewpoint video 30 and the recommended data 32, plays the received virtual viewpoint video 30 on a video playback screen 33 of the touch panel display 12A, and displays the received recommended data 32 on the touch panel display 12A. In the example shown in FIG. 2, the recommended data 32 is displayed below the video playback screen 33. Note that in the technology disclosed herein, "playback" refers to the process of displaying an image on the video playback screen 33 of the touch panel display 12A of the user device 12. The image played on the touch panel display 12A is not limited to the virtual viewpoint video 30, and may be another video or a still image.

[0071] A seek bar 31 indicating the transmission status and playback position of the virtual viewpoint video 30 is displayed below the video playback screen 33. The horizontal length of the seek bar 31 corresponds to the length of the virtual viewpoint video 30 being played. A playback position display portion 31A of the seek bar 31, which is shown in a dark color, indicates the playback position of the virtual viewpoint video 30 being played on the video playback screen 33. A transmission status display portion 31B of the seek bar 31, which is shown in a lighter color than the playback position display portion 31A, indicates the status of data transmission from the data processing device 10 to the user device 12 related to the virtual viewpoint video 30 being played on the video playback screen 33.

[0072] 3, when a user operates the user device 12, the user device 12 accesses the data processing device 10 using an account associated with each user, and communication between the user device 12 and the data processing device 10 is established, the data processing device 10 transmits a menu screen 40 to the user device 12. The user device 12 displays the menu screen 40 received from the data processing device 10 on the touch panel display 12A. The account associated with each user is an example of an "account" according to the technology of the present disclosure.

[0073] The menu screen 40 displays the titles 42 of events for which the virtual viewpoint videos 30 can be generated and viewed, and the dates 44 on which the events took place. The menu screen 40 also displays a generate button 46 and a view button 48.

[0074] When the user device 12 receives a GUI operation on the generate button 46 by the user, the user device 12 transmits a video generation request 50 requesting the generation of the virtual viewpoint video 30 to the video generation unit 26. When the video generation request 50 is received from the user device 12, the video generation unit 26 starts the video generation process. When the user device 12 receives a GUI operation on the watch button 48 by the user, the user device 12 transmits a video playback request 52 requesting the playback of the video to the video playback unit 28. When the video playback request 52 is received from the user device 12, the video playback unit 28 starts the video playback process.

[0075] An example of the video generation process performed by the video generation unit 26 will be specifically described below with reference to Fig. 4. When the video generation unit 26 receives a video generation request 50 from the user device 12, it generates a viewpoint-related information reception screen 54. The video generation unit 26 generates a viewpoint-related information reception screen 54 based on the generated viewpoint. The related information reception screen 54 is transmitted to the user device 12 that is the output source of the video generation request 50. The user device 12 receives the viewpoint related information reception screen 54 and displays the received viewpoint related information reception screen 54 on the touch panel display 12A.

[0076] The viewpoint-related information receiving screen 54 is a screen used to receive viewpoint-related information 58 related to the viewpoint of the virtual viewpoint video 30. In the first embodiment, the viewpoint-related information 58 is data including the viewpoint position, line-of-sight direction, and angle of view related to the virtual viewpoint. Note that the viewpoint-related information 58 is an example of "viewpoint-related information" according to the technology of the present disclosure.

[0077] A bird's-eye view image 56 of the soccer stadium is displayed on the viewpoint-related information reception screen 54. The bird's-eye view image 56 may be an image obtained by actually capturing an image of the soccer stadium from directly above or obliquely above, or may be an image generated artificially based on the aspect ratio of the soccer stadium. The bird's-eye view image 56 is an image that is similar to the soccer stadium, with a similarity ratio of, for example, several hundred:1.

[0078] The touch panel display 12A accepts user operations on the viewpoint-related information reception screen 54. The user specifies a virtual viewpoint position, a line of sight direction, and an angle of view on the viewpoint-related information reception screen 54. The user device 12 derives coordinates of a viewpoint position 58A (see FIG. 5) with respect to the soccer stadium based on the coordinates of the viewpoint position specified with respect to the overhead-view image 56. The user device 12 also derives a line of sight direction 58B (see FIG. 5) with respect to the soccer stadium based on the instruction of the line of sight direction with respect to the overhead-view image 56. The user device 12 also derives an angle of view 58C (see FIG. 5) with respect to the soccer stadium based on the instruction of the angle of view with respect to the overhead-view image 56. The user device 12 transmits the derived viewpoint position 58A, line of sight direction 58B, and angle of view 58C to the video generator 26 as viewpoint-related information 58.

[0079] The video generation unit 26 receives viewpoint-related information 58 and records the received viewpoint-related information 58 in a generated video list 60 of the database 22. The video generation unit 26 also generates a virtual viewpoint video 30 using the captured video 34 based on the received viewpoint-related information 58. Specifically, the video generation unit 26 generates a virtual viewpoint video 30 in which the captured area is observed from a viewpoint position 58A in a line of sight direction 58B at an angle of view 58C. Note that the viewpoint position 58A is an example of a "viewpoint position" according to the technology of the present disclosure. The line of sight direction 58B is an example of a "line of sight direction" according to the technology of the present disclosure. The angle of view 58C is an example of a "angle of view" according to the technology of the present disclosure.

[0080] The moving image generating unit 26 associates moving image information 62 related to the generated virtual viewpoint moving image 30 with viewpoint-related information 58 used to generate the virtual viewpoint moving image 30, and records the information in a generated moving image list 60 in the database 22. The moving image generating unit 26 also stores the generated virtual viewpoint moving image 30 in the database 22.

[0081] The video generation unit 26 transmits the generated virtual viewpoint video 30 to the user device 12 that output the video generation request 50. The user device 12 receives the virtual viewpoint video 30 and displays the received virtual viewpoint video 30 on the touch panel display 12A. The video generation unit 26 adds viewpoint-related information 58 and video information 62 related to the transmitted virtual viewpoint video 30 to a played video list 64 corresponding to the account of the user using the user device 12 that transmitted the virtual viewpoint video 30, among multiple played video lists 64 stored in the database 22. Hereinafter, the viewpoint-related information 58 and video information 62 will also be collectively referred to as "information."

[0082] 5, the generated moving image list 60 records, in list form, viewpoint-related information 58 and moving image information 62 relating to a plurality of virtual viewpoint moving images 30 generated by the moving image generating unit 26. The viewpoint-related information 58 includes a viewpoint position 58A of the virtual viewpoint moving image 30, a line-of-sight direction 58B, and the like. The video information 62 includes a video ID 62A assigned to each virtual viewpoint video 30, a creator 62B of the virtual viewpoint video 30, an elapsed time 62C of the soccer match corresponding to the virtual viewpoint video 30, a length 62D of the video, and a thumbnail image 62E of the virtual viewpoint video 30.

[0083] The elapsed match time 62C is a time calculated based on the capture time of the captured video 34 used to generate the virtual viewpoint video 30. Specifically, the value obtained by subtracting the game start time from the capture time of the first frame of the captured video 34 used to generate the virtual viewpoint video 30 is the start time of the elapsed match time 62C, and the value obtained by subtracting the game start time from the capture time of the last frame of the captured image is the end time of the elapsed match time 62C. The elapsed match time 62C is an example of "first time information" according to the technology of the present disclosure.

[0084] As an example, as shown in Fig. 6, the played video list 64 is a list generated for each user account. For example, Fig. 6 shows the played video list 64 of user A, and the account name of user A, "A," is recorded in the account 64A column. The played video list 64 shown in Fig. 6 records, in list form, data related to the virtual viewpoint videos 30 that have been played on the user device 12 that is accessing the data processing device 10 using user A's account, i.e., the virtual viewpoint videos 30 that user A has viewed.

[0085] Furthermore, the played video list 64 indicates the playback history of the virtual viewpoint video 30 for the user device 12 that is accessing the data processing device 10 using user A's account. Note that "playback" refers to the process of displaying an image on the video playback screen 33 of the touch panel display 12A of the user device 12, so the "playback history of the virtual viewpoint video 30 for the user device 12" is synonymous with the "display history of the virtual viewpoint video 30 on the video playback screen 33 of the touch panel display 12A." The playback history recorded in the played video list 64 is an example of a "playback history" according to the technology of the present disclosure. Furthermore, the content recorded in the played video list 64 is an example of "first data" according to the technology of the present disclosure.

[0086] An example of video playback processing performed by the video playback unit 28 will be specifically described below with reference to Fig. 7. For example, when the video playback unit 28 receives a video playback request 52 from the user device 12 of user A, the video playback unit 28 reads out a played video list 64 corresponding to user A's account from the database 22. The played video list 64 corresponding to user A's account contains information about a virtual viewpoint video 30 having an image ID 62A of "001" (hereinafter referred to as "virtual viewpoint video 30-001"). Note that the virtual viewpoint video 30-001 is an example of a "first virtual viewpoint image" according to the technology of the present disclosure.

[0087] The video playback unit 28 reads out the generated video list 60 from the database 22. The generated video list 60 records data related to the virtual viewpoint video 30 created by the video generation unit 26 (see FIG. 5 ). The video playback unit 28 deletes information related to the virtual viewpoint video 30-001 recorded in the played video list 64 from the generated video list 60. As a result, information related to a virtual viewpoint video 30 different from the virtual viewpoint video 30-001, i.e., information related to a virtual viewpoint video 30 that has not been transmitted to the user device 12 (an untransmitted virtual viewpoint video 30), remains in the generated video list 60. Note that the untransmitted virtual viewpoint video 30 is an example of a "second virtual viewpoint image" according to the technology of the present disclosure. Furthermore, the information related to the virtual viewpoint video 30-001 recorded in the played video list 64 does not necessarily need to be deleted from the generated video list 60; information indicating that the virtual viewpoint video 30-001 has been transmitted may be added to the information related to the virtual viewpoint video 30-001 to distinguish it from an untransmitted virtual viewpoint video 30.

[0088] The video playback unit 28 compares the viewpoint positions 58A of the untransmitted virtual viewpoint videos 30 remaining in the generated video list 60 with the viewpoint positions 58B of the virtual viewpoint video 30-001 recorded in the played video list 64. The video playback unit 28 calculates the difference in viewpoint position between the viewpoint position 58A of the untransmitted virtual viewpoint video 30-002 and the reference viewpoint position (hereinafter referred to as the "reference viewpoint position"). For example, the video playback unit 28 acquires coordinates (X2, Y2) indicating the viewpoint position 58A of the untransmitted virtual viewpoint video 30 having the video ID 62A of "002" (hereinafter referred to as the "untransmitted virtual viewpoint video 30-002") and coordinates (X1, Y1) of the reference viewpoint position. The video playback unit 28 calculates the difference in the X-axis direction between the viewpoint position 58A of the untransmitted virtual viewpoint video 30-002 and the reference viewpoint position (hereinafter referred to as the "X-axis direction viewpoint position difference") by taking the difference between X2 and X1. The video playback unit 28 also calculates the difference in the Y-axis direction between the viewpoint position 58A of the untransmitted virtual viewpoint video 30-002 and the reference viewpoint position (hereinafter referred to as the "Y-axis direction viewpoint position difference") by taking the difference between Y2 and Y1. The video playback unit 28 calculates the sum of the viewpoint position difference in the X-axis direction and the viewpoint position difference in the Y-axis direction as the viewpoint position difference of the untransmitted virtual viewpoint video 30-002.

[0089] The video playback unit 28 similarly calculates the viewpoint position difference between the viewpoint position 58A of the virtual viewpoint video 30 having the video ID 62A of "003" (hereinafter referred to as the "untransmitted virtual viewpoint video 30-003") and the reference viewpoint position. The video playback unit 28 sorts the untransmitted virtual viewpoint videos 30 in the generated video list 60 in order of the viewpoint position difference, that is, in order of the similarity to the reference viewpoint position. For example, if the viewpoint position difference of the untransmitted virtual viewpoint video 30-002 is 8 and the viewpoint position difference of the untransmitted virtual viewpoint video 30-003 is 6, the video playback unit 28 moves the untransmitted virtual viewpoint video 30-003 higher in the generated video list 60 than the untransmitted virtual viewpoint video 30-002.

[0090] 8, for example, the video playback unit 28 calculates the difference (hereinafter, "gaze direction difference") between the gaze direction 58B of each untransmitted virtual viewpoint video 30 and the gaze direction 58B of the virtual viewpoint video 30-001 (hereinafter, referred to as the "reference gaze direction") for the untransmitted virtual viewpoint videos 30 with the same viewpoint position difference. The video playback unit 28 sorts the untransmitted virtual viewpoint videos 30 with the same viewpoint position difference in the generated video list 60 in order of the smallest gaze direction difference, that is, in order of the highest similarity to the reference gaze direction.

[0091] The video playback unit 28 further compares the elapsed game time 62C of the untransmitted virtual viewpoint video 30 with the elapsed game time 62C of the virtual viewpoint video 30-001 recorded in the played video list 64 (hereinafter referred to as the "reference elapsed game time"). The video playback unit 28 moves the untransmitted virtual viewpoint video 30 whose elapsed game time 62C overlaps with the reference elapsed game time for more than a predetermined value (e.g., several tens of seconds) to the bottom of the generated video list 60. That is, the video playback unit 28 moves the untransmitted virtual viewpoint video 30 whose elapsed game time 62C is highly similar to the reference elapsed game time to the bottom of the generated video list 60. For example, in the first embodiment, the elapsed game time 62C of the virtual viewpoint video 30-003 overlaps with the reference elapsed game time for 25 seconds. Therefore, the video playback unit 28 moves the virtual viewpoint video 30-003 to the bottom of the generated video list 60.

[0092] The video playback unit 28 stores the rearranged generated video list 60 in the database 22 as a recommended video list 66. The video playback unit 28 generates recommended data 32 based on video information 62 related to the untransmitted virtual viewpoint videos 30 in the order of the untransmitted virtual viewpoint videos 30 recorded in the recommended video list 66.

[0093] As an example, as shown in Fig. 9, information about untransmitted virtual viewpoint videos 30 is recorded in the recommended video list 66 in the order of recommendation. The video playback unit 28 reads out from the database 22 five pieces of video information 62 about the untransmitted virtual viewpoint videos 30 recorded in the recommended video list 66 in the order of the untransmitted virtual viewpoint videos 30 recorded in the recommended video list 66. The video playback unit 28 performs the video information 62 about the five untransmitted virtual viewpoint videos 30 read out from the database 22, and the video ID 62A, creator 62B, and game progress time included in the video information 62. The video playback unit 28 generates recommendation data 32 including information on the interval 62C, the length 62D of the video, and the placement information of the thumbnail images 62E. The video playback unit 28 transmits the generated recommendation data 32 to the user device 12. The user device 12 receives the recommendation data 32 and displays the received recommendation data 32 as a recommendation screen 70 (see FIG. 10 ) on the touch panel display 12A.

[0094] As an example, as shown in FIG. 10 , the recommendation screen 70 displays recommended data 32. The recommendation screen 70 displays video information 62 related to five untransmitted virtual viewpoint videos 30. The number of untransmitted virtual viewpoint videos 30 displayed on the recommendation screen 70 is not limited to five and may be set arbitrarily by the user. On the recommendation screen 70, thumbnail images 62E are arranged on the left side of the recommendation screen 70. A length 62D of the corresponding untransmitted virtual viewpoint video 30 is arranged in the lower right corner of each thumbnail image 62E. A creator 62B, a video ID 62A, and a game elapsed time 62C are arranged on the right side of the corresponding thumbnail image 62E. The thumbnail image 62E is an example of a "reduced image" according to the technology of the present disclosure.

[0095] An update button 68 is located in the lower right corner of the recommendation screen 70. When the user touches the update button 68 on the touch panel display 12A, the user device 12 issues an instruction to the video playback unit 28 to update the recommended data 32. The video playback unit 28 receives the update instruction from the user device 12 and generates recommended data 32 based on the video information 62 related to the next five untransmitted virtual viewpoint videos 30, i.e., the video information 62 related to the sixth to tenth recorded untransmitted virtual viewpoint videos 30. The video playback unit 28 transmits the generated recommended data 32 to the user device 12.

[0096] When the user touches a thumbnail image 62E displayed on the recommendation screen 70, as shown in FIG. 11 as an example, the user device 12 transmits a playback instruction 72 for the untransmitted virtual viewpoint video 30 corresponding to the touched thumbnail image 62E to the video playback unit 28. The video playback unit 28 reads the target untransmitted virtual viewpoint video 30 from the database 22. The video playback unit 28 transmits the untransmitted virtual viewpoint video 30 read from the database 22 to the user device 12 that output the playback instruction 72. The user device 12 receives the untransmitted virtual viewpoint video 30 and plays the received virtual viewpoint video 30 on the touch panel display 12A. The playback instruction 72 is an example of a "playback instruction" according to the technology of the present disclosure.

[0097] The video playback unit 28 adds information about the virtual viewpoint video 30 sent to the user device 12 to the played video list 64 corresponding to the account of the user using the user device 12. As a result, as shown in FIG. 12 as an example, when a virtual viewpoint video 30-002 is sent to the user device 12 that is the sender of the playback instruction 72 in response to a playback instruction 72 given from the user device 12 that is accessing the data processing device 10 using the account of user A, data about the virtual viewpoint video 30-002 is added to the played video list 64 corresponding to the account of user A.

[0098] 12, when the played video list 64 includes information about multiple virtual viewpoint videos 30, the video playback unit 28 generates the recommended video list 66 based on information (hereinafter referred to as "reference information") about the virtual viewpoint video 30-002 that was added to the played video list 64 latest among the multiple virtual viewpoint videos 30 included in the played video list 64. Note that instead of using information about the virtual viewpoint video 30-002 that was added to the played video list 64 latest as the reference, the recommended video list 66 may be generated based on multiple virtual viewpoint videos 30.

[0099] Next, the operation of the moving image generating unit 26 according to the first embodiment will be described with reference to Fig. 13. The moving image generating process shown in Fig. 13 is carried out when the CPU 14A executes the operation program 24. 13 is started when the video generator 26 receives a video generation request 50 from at least one of the multiple user devices 12.

[0100] 13, first, in step ST101, the moving image generation unit 26 generates a viewpoint-related information acceptance screen 54 and transmits the generated viewpoint-related information acceptance screen 54 to the user device 12 that is the output source of the moving image generation request 50. The user device 12 receives the viewpoint-related information acceptance screen 54 and displays the received viewpoint-related information acceptance screen 54 on the touch panel display 12A. Thereafter, the moving image generation process proceeds to step ST102.

[0101] In step ST102, the moving image generation unit 26 determines whether or not the viewpoint-related information 58 has been received from the user device 12 that transmitted the viewpoint-related information reception screen 54. If the viewpoint-related information 58 has been received in step ST102, the determination is affirmative, and the moving image generation process proceeds to step ST103. If the viewpoint-related information 58 has not been received in step ST102, the determination is negative, and the moving image generation process proceeds to step ST102.

[0102] In step ST103, the moving image generating unit 26 stores the viewpoint-related information 58 received from the user device 12 in the generated moving image list 60 of the database 22. After that, the moving image generating process proceeds to step ST104.

[0103] In step ST104, the video generation unit 26 generates the virtual viewpoint video 30 based on the viewpoint-related information 58 received from the user device 12. After that, the video generation process proceeds to step ST105.

[0104] In step ST105, the moving image generating unit 26 associates the moving image information 62 related to the generated virtual viewpoint moving image 30 with the viewpoint-related information 58 and stores it in the generated moving image list 60 of the database 22. After that, the moving image generating process proceeds to step ST106.

[0105] In step ST106, the moving image generating unit 26 stores the generated virtual viewpoint moving image 30 in the database 22. After that, the moving image generating process proceeds to step ST107.

[0106] In step ST107, the video generation unit 26 transmits the generated virtual viewpoint video 30 to the user device 12 that is the output source of the video generation request 50. The user device 12 receives the virtual viewpoint video 30 and plays the received virtual viewpoint video 30 on the video playback screen 33. After that, the video generation process proceeds to step ST108.

[0107] In step ST108, the video generation unit 26 adds the generated virtual viewpoint video 30 to the played video list 64 corresponding to the account of the user who uses the user device 12 that is the output source of the video generation request 50. After that, the video generation process proceeds to step ST109.

[0108] In step ST109, the moving image generating unit 26 determines whether or not a termination condition is satisfied. An example of the termination condition is when a request to stop moving image generation is received from the user device 12. In step ST109, if the termination condition is satisfied, the determination is affirmative, and the moving image generation process is terminated. In step ST109, if the termination condition is not satisfied, the determination is negative, and the moving image generation process proceeds to step ST101.

[0109] Next, the operation of the video playback unit 28 according to the first embodiment will be described with reference to FIG. 14. The video playback process shown in FIG. 14 is realized by the CPU 14A executing the operation program 24. The video playback process shown in FIG. 14 is realized by the video playback unit 28 playing back a video of a plurality of users. The process is initiated when a video playback request 52 is received from at least one of the devices 12.

[0110] 14, first, in step ST201, the video playback unit 28 reads from the database 22 the generated video list 60 and the played video list 64 corresponding to the account of the user using the user device 12 that is the output source of the video playback request 52. Thereafter, the video playback process proceeds to step ST202.

[0111] In step ST202, the video playback unit 28 deletes the information about the virtual viewpoint video 30 stored in the played video list 64 from the generated video list 60. After this, the video playback process proceeds to step ST203.

[0112] In step ST203, the video playback unit 28 acquires reference information from the played video list 64, and rearranges the virtual viewpoint videos 30 in the generated video list 60 based on the acquired reference information. The video playback unit 28 stores the rearranged generated video list 60 in the database 22 as a recommended video list 66. After this, the video playback process proceeds to step ST204.

[0113] In step ST204, the moving image playback unit 28 sets N = 0. After that, the moving image playback process proceeds to step ST205.

[0114] In step ST205, the video playback unit 28 generates recommended data 32 based on the video information 62 of the 5N+1th to 5(N+1)th virtual viewpoint videos 30 in the recommended video list 66. That is, when N=0, the video playback unit 28 generates recommended data 32 based on the video information 62 of the 1st to 5th virtual viewpoint videos 30 in the recommended video list 66. The video playback unit 28 transmits the generated recommended data 32 to the user device 12. After that, the video playback process proceeds to step ST206.

[0115] In step ST206, the video playback unit 28 determines whether or not it has received, from the user device 12, a playback instruction 72 for any of the five virtual viewpoint videos 30 included in the recommended data 32. If the playback instruction 72 has been received, the video playback process proceeds to step ST209. If the playback instruction 72 has not been received, the video playback process proceeds to step ST207.

[0116] In step ST207, the video playback unit 28 determines whether or not an update instruction has been received from the user device 12. If an update instruction has been received in step ST207, the determination is affirmative, and the video playback process proceeds to step ST208. If an update instruction has not been received in step ST207, the determination is negative, and the video playback process proceeds to step ST206.

[0117] In step ST208, the moving image playback unit 28 increments N by 1. After that, the moving image playback process proceeds to step ST205.

[0118] In step ST209, the video playback unit 28 reads the virtual viewpoint video 30 specified by the playback instruction 72 from the database 22. The video playback unit 28 transmits the virtual viewpoint video 30 read from the database 22 to the user device 12. After that, the video playback process proceeds to step ST210.

[0119] In step ST210, the video playback unit 28 adds information about the virtual viewpoint video 30 transmitted to the user device 12 to the played video list 64. After this, the video playback process continues. Proceed to step ST211.

[0120] In step ST211, the video playback unit 28 determines whether or not a termination condition is satisfied. An example of the termination condition is when a request to stop watching a video is received from the user device 12. In step ST211, if the termination condition is satisfied, the determination is affirmative, and the video playback process is terminated. In step ST211, if the termination condition is not satisfied, the determination is negative, and the video playback process proceeds to step ST201.

[0121] As described above, in the first embodiment, the data processing device 10 transmits the virtual viewpoint video 30 generated based on the captured video 34 to the user device 12. When the video playback unit 28 receives a video playback request 52 from a user device 12 accessing the data processing device 10 with a certain user's account, the video playback unit 28 reads the played video list 64 corresponding to the requesting account from the database 22. The video playback unit 28 generates recommendation data 32 related to the untransmitted virtual viewpoint video 30 based on the viewpoint-related information 58 and the elapsed game time 62C included in the acquired played video list 64. The video playback unit 28 transmits the generated recommendation data 32 to the user device 12. Therefore, this configuration makes it easier for the user to select the virtual viewpoint video 30 that the user desires, compared to when only the played video list 64 is used in the user device 12.

[0122] Furthermore, in the first embodiment, the video playback unit 28 controls the transmission of recommended data 32 related to untransmitted virtual viewpoint videos 30 that are different from the virtual viewpoint videos 30 recorded in the played video list 64. Therefore, according to this configuration, it is possible to allow the user to more easily select the virtual viewpoint video 30 that the user desires, compared to when only the played video list 64 is used in the user device 12.

[0123] Furthermore, in the first embodiment, the played video list 64 includes viewpoint-related information 58 related to the viewpoint of the virtual viewpoint video 30. Therefore, according to this configuration, compared to when the played video list 64 includes information unrelated to the viewpoint-related information 58, it is possible to make it easier for the user to select a virtual viewpoint video 30 that is related to the viewpoint-related information 58 from among the untransmitted virtual viewpoint videos 30.

[0124] Furthermore, in the first embodiment, viewpoint-related information 58 includes viewpoint position 58A, line of sight direction 58B, and angle of view 58C. Therefore, according to this configuration, compared to when viewpoint-related information 58 includes information unrelated to viewpoint position 58A, line of sight direction 58B, and angle of view 58C, it is possible to make it easier for the user to select, from among untransmitted virtual viewpoint videos 30, a virtual viewpoint video 30 that is related to at least one of viewpoint position 58A, line of sight direction 58B, and angle of view 58C.

[0125] Furthermore, in the first embodiment, the played video list 64 includes an elapsed game time 62C related to the capture time of the captured video 34 used to generate the virtual viewpoint video 30. Therefore, according to this configuration, compared to when the played video list 64 includes information unrelated to the capture time of the captured video 34, it is possible to make it easier for the user to select a virtual viewpoint video 30 that is related to the capture time of the captured video 34 from among the untransmitted virtual viewpoint videos 30.

[0126] Furthermore, in the first embodiment, the recommended data 32 is data in which thumbnail images 62E of untransmitted virtual viewpoint videos 30 are displayed on the user device 12 in the order of high similarity between the viewpoint-related information 58 included in the played video list 64 and the viewpoint-related information 58, and low similarity between the game elapsed time 62C included in the played video list 64 and the game elapsed time 62C. Therefore, according to this configuration, the thumbnail images 62E of untransmitted virtual viewpoint videos 30 are displayed on the user device 12 in the order of low similarity between the viewpoint-related information 58 included in the played video list 64 and the game elapsed time 62C. This makes it easier for the user to select a virtual viewpoint video 30 that has a high similarity in terms of viewpoint-related information 58 contained in the list of played videos 64 with respect to viewpoint-related information 58, and a low similarity in terms of game elapsed time 62C contained in the list of played videos 64 with respect to game elapsed time 62C.

[0127] Furthermore, in the first embodiment, the data processing device 10 transmits to the user device 12 a plurality of virtual viewpoint videos 30 generated based on the captured video 34. When the video playback unit 28 receives a video playback request 52 from the user device 12, the video playback unit 28 acquires, from the played video list 64, information about virtual viewpoint videos 30, at least some of which have been transmitted to the user device 12 that transmitted the video playback request 52, among the plurality of virtual viewpoint videos 30. Based on the acquired information, the video playback unit 28 generates, as recommendation data 32, information about virtual viewpoint videos 30 (untransmitted virtual viewpoint videos 30) among the plurality of virtual viewpoint videos 30 that have not been transmitted to the user device 12 that transmitted the request. The video playback unit 28 transmits the generated recommendation data 32 to the user device 12 that transmitted the request. Therefore, according to this configuration, it is easier for the user to select a virtual viewpoint video 30 that the user intends, compared to when only the played video list 64 is used in the user device 12.

[0128] [Second embodiment] In the above first embodiment, an example was described in which the viewpoint-related information 58 includes a viewpoint position 58A, a line of sight direction 58B, and an angle of view 58C. In the present second embodiment, an example is described in which the viewpoint-related information 58 includes a viewpoint position path 58D and a point of interest 58E instead of the viewpoint position 58A and line of sight direction 58B. In the first embodiment, the video playback unit 28 generates the recommended data 32 including the video information 62 related to the untransmitted virtual viewpoint video 30 in descending order of similarity between the viewpoint-related information 58 included in the played video list 64 and the viewpoint-related information 58, and in descending order of similarity between the elapsed game time 62C included in the played video list 64 and the elapsed game time 62C. In the second embodiment, the video playback unit 28 generates the recommended data 32 including the video information 62 related to the untransmitted virtual viewpoint video 30 in descending order of similarity between the viewpoint-related information 58 included in the played video list 64 and the viewpoint-related information 58, and in descending order of similarity between the elapsed game time 62C included in the played video list 64 and the elapsed game time 62C. Differences from the first embodiment will be described below. In the following description, the same components as those in the first embodiment will be denoted by the same reference numerals as those in the first embodiment, and descriptions of the same components and functions as those in the first embodiment will be omitted. The recommended data 32 may also be generated in the first embodiment in a manner similar to that of the second embodiment. In the second embodiment, the recommendation data 32 may be generated in the same manner as in the first embodiment.

[0129] FIG. 15 shows an example of a viewpoint-related information reception screen 54 transmitted from the video generation unit 26 to the user device 12 in response to a video generation request 50 from the user device 12. A bird's-eye view image 56 is displayed on the viewpoint-related information reception screen 54. In the second embodiment, the user specifies a viewpoint position path, a gaze point, and a field of view on the viewpoint-related information reception screen 54. The viewpoint position path indicates the displacement of the viewpoint position over time. The gaze point is a gaze position within the imaging area. In the example shown in FIG. 15, it is set to the face of a featured player. However, the technology of the present disclosure is not limited to this. The gaze point may be the position of another object within the imaging area, such as the position of the ball, the position of a goal, or the position of a specific line. The gaze point may also be any position not including an object within the imaging area. The field of view is the field of view when observing the gaze point from each viewpoint position included in the viewpoint position path.

[0130] The user device 12 derives the coordinates of a viewpoint position path 58D (see FIG. 16) for the soccer stadium based on the coordinates of the viewpoint position path indicated for the overhead image 56. The user device 12 also derives the coordinates of a gaze point 58E (see FIG. 16) in the soccer stadium based on the coordinates of the gaze point indicated for the overhead image 56. The user device 12 also derives the coordinates of a gaze point 58E (see FIG. 16) in the soccer stadium based on the coordinates of the gaze point indicated for the overhead image 56. The user device 12 derives a field of view 58C (see FIG. 16 ) of the soccer stadium based on the field of view instruction for the overhead image 56. The user device 12 transmits the derived viewpoint position path 58D, point of gaze 58E, and field of view 58C to the video generator 26 as viewpoint-related information 58. Note that viewpoint position path 58D is an example of a “viewpoint position path” according to the technology of the present disclosure. Also, point of gaze 58E is an example of a “point of gaze” according to the technology of the present disclosure.

[0131] 16 as an example, the moving image generation unit 26 records viewpoint-related information 58 received from the user device 12 in a generated moving image list 60. Furthermore, the moving image generation unit 26 generates a virtual viewpoint video 30 using the captured video 34 based on the viewpoint-related information 58 received from the user device 12. The moving image generation unit 26 records moving image information 62 related to the generated virtual viewpoint video 30 in the generated moving image list 60 in association with the viewpoint-related information 58. Note that the generated moving image list 60 records viewpoint-related information 58 and moving image information 62 related to the multiple virtual viewpoint videos 30 generated by the moving image generation unit 26 based on the viewpoint-related information 58 received from the multiple user devices 12.

[0132] The video generation unit 26 transmits the generated virtual viewpoint video 30 to the user device 12 that issued the request, and records the transmitted virtual viewpoint video 30 in a played video list 64 corresponding to the account of the user who uses the user device 12 that issued the request. As an example, as shown in FIG. 17 , when the video generation unit 26 generates the virtual viewpoint video 30 based on viewpoint-related information 58 received from a user device 12 that is accessing the data processing device 10 using the account of user A, the video generation unit 26 transmits the generated virtual viewpoint video 30 to the user device 12 that issued the viewpoint-related information 58, and records the transmitted virtual viewpoint video 30 in a played video list 64 corresponding to the account of user A.

[0133] 18, the video playback process performed by the video playback unit 28 according to the second embodiment will be specifically described below. For example, when the video playback unit 28 receives a video playback request 52 from a user device 12 accessing the data processing device 10 using the account of user A, the video playback unit 28 reads out a played video list 64 corresponding to the account of user A from the database 22. The read played video list 64 contains information about a virtual viewpoint video 30 having a video ID 62A of "011" (hereinafter referred to as "virtual viewpoint video 30-011"). The virtual viewpoint video 30-011 is an example of a "first virtual viewpoint image" according to the technology of the present disclosure.

[0134] The video playback unit 28 reads out the generated video list 60 from the database 22. The generated video list 60 records data related to the virtual viewpoint video 30 generated by the video generation unit 26 (see FIG. 16 ). The video playback unit 28 deletes information related to the virtual viewpoint video 30-011 recorded in the played video list 64 from the generated video list 60. As a result, information related to the virtual viewpoint video 30 that has not yet been transmitted to the user device 12 that is the output source of the video playback request 52 remains in the generated video list 60.

[0135] The video playback unit 28 compares the elapsed game time 62C of each untransmitted virtual viewpoint video 30 with the elapsed game time 62C of the virtual viewpoint video 30-011 recorded in the played video list 64 (hereinafter referred to as the "reference elapsed game time"). The video playback unit 28 keeps the virtual viewpoint videos 30 whose elapsed game time 62C overlaps with the reference elapsed game time for more than a predetermined value (e.g., several tens of seconds), and deletes the other virtual viewpoint videos 30 from the generated video list 60. In other words, the video playback unit 28 keeps in the generated video list 60 the untransmitted virtual viewpoint videos 30 whose elapsed game time 62C is highly similar to the reference elapsed game time.

[0136] The video playback unit 28 receives the viewpoint position path 58D of each untransmitted virtual viewpoint video 30 and the viewpoint position path 58D of the virtual viewpoint video 30-011 recorded in the played video list 64 (hereinafter, “ The video playback unit 28 derives a similarity of the viewpoint position path 58D of each untransmitted virtual viewpoint video 30 with respect to the reference viewpoint position path by comparing the position of the viewpoint position path 58D of the virtual viewpoint video 30 having the video ID 62A of "012" (hereinafter referred to as "virtual viewpoint video 30-012"). S2 ,Y S2 ,Z S2 ) and end coordinate (X E2 ,Y E2 ,Z E2 ) and the starting coordinate of the reference viewpoint position path (X S1 ,Y S1 ,Z S1 ) and end coordinate (X E1 ,YE1 ,Z E1 The video playback unit 28 obtains the difference between the start coordinates and the difference between the end coordinates between the viewpoint position path 58D of the virtual viewpoint video 30-012 and the reference viewpoint position path.

[0137] Video playback section 28 is X S2 -X S1 By calculating Y, the video playback unit 28 calculates the difference in the X-axis direction between the start coordinates of the viewpoint position path 58D of the virtual viewpoint video 30-012 and the reference viewpoint position path (hereinafter referred to as the "X-axis direction start coordinate difference"). S2 -Y S1 By calculating Z, the video playback unit 28 calculates the difference in the Y-axis direction between the start coordinates of the viewpoint position path 58D of the virtual viewpoint video 30-012 and the reference viewpoint position path (hereinafter referred to as the "Y-axis direction start coordinate difference"). S2 -Z S1 The video playback unit 28 calculates the difference in the Z-axis direction between the start coordinates of the viewpoint position path 58D of the virtual viewpoint video 30-012 and the reference viewpoint position path (hereinafter referred to as the "Z-axis direction start coordinate difference") by calculating the difference in the Z-axis direction. The video playback unit 28 calculates the sum of the X-axis direction start coordinate difference, the Y-axis direction start coordinate difference, and the Z-axis direction start coordinate difference as the start coordinate difference.

[0138] In addition, the video playback unit 28 E2 -X E1 By calculating Y, the video playback unit 28 calculates the difference in the X-axis direction between the end coordinates of the viewpoint position path 58D of the virtual viewpoint video 30-012 and the reference viewpoint position path (hereinafter referred to as "X-axis direction end coordinate difference"). E2 -Y E1 By calculating Z, the video playback unit 28 calculates the difference in the Y-axis direction between the end coordinates of the viewpoint position path 58D of the virtual viewpoint video 30-012 and the reference viewpoint position path (hereinafter referred to as the "Y-axis direction end coordinate difference"). E2 -Z E1By calculating the difference in the Z-axis direction between the end coordinates of the viewpoint position path 58D of the virtual viewpoint video 30-012 and the reference viewpoint position path (hereinafter referred to as the "Z-axis direction end coordinate difference"). The video playback unit 28 calculates the sum of the X-axis direction end coordinate difference, the Y-axis direction end coordinate difference, and the Z-axis direction end coordinate difference as the end coordinate difference. The video playback unit 28 calculates the sum of the start coordinate difference and the end coordinate difference as the viewpoint position path difference. Note that the viewpoint position path difference is not limited to the sum of the start coordinate difference and the end coordinate difference. For example, in addition to the start coordinate difference and the end coordinate difference, the coordinate difference of an arbitrary point on the path may be obtained. More specifically, for example, the viewpoint coordinate difference may be calculated every second from the playback of the virtual viewpoint video 30 and added to the start coordinate difference and the end coordinate difference.

[0139] The video playback unit 28 similarly calculates the viewpoint position path difference between the viewpoint position path 58D of each untransmitted virtual viewpoint video 30 and the reference viewpoint position path. The video playback unit 28 sorts the untransmitted virtual viewpoint videos 30 in the generated video list 60 in descending order of viewpoint position path difference, that is, in descending order of similarity between the viewpoint position path 58D of the virtual viewpoint video 30 and the reference viewpoint position path.

[0140] The video playback unit 28 stores the rearranged generated video list 60 in the database 22 as a recommended video list 66. The video playback unit 28 reads information about the virtual viewpoint videos 30 from the database 22, five at a time, in the order in which the virtual viewpoint videos 30 are recorded in the recommended video list 66. The video playback unit 28 generates recommendation data 32 including a video ID 62A, a game elapsed time 62C, a thumbnail image 62E, and a viewpoint position path 58D related to the five untransmitted virtual viewpoint videos 30 read from the database 22, as well as their arrangement information. The video playback unit 28 transmits the generated recommendation data 32 to the user device 12.

[0141] As an example, as shown in FIG. 19, the user device 12 receives the recommendation data 32 and displays the received recommendation data 32 as a recommendation screen 70 on the touch panel display 12A. On the recommendation screen 70 according to the second embodiment, thumbnail images 62E relating to five untransmitted virtual viewpoint videos 30 are displayed on the left side of the recommendation screen 70. On the right side of each thumbnail image 62E, the video ID 62A and the shape of the viewpoint position path 58D of the untransmitted virtual viewpoint video 30 corresponding to the thumbnail image 62E are displayed. The shape of the viewpoint position path 58D is displayed superimposed on the overhead image 56 of the soccer stadium.

[0142] As a modified example of the display method for the shape of viewpoint position path 58D, the shape of viewpoint position path 58D may be displayed superimposed on a side view of the soccer stadium, as shown in Modification 1 of Fig. 19. Alternatively, the shape of viewpoint position path 58D may be displayed superimposed on a three-dimensional image showing the soccer stadium, as shown in Modification 2 of Fig. 19. Alternatively, an image in which the shape of viewpoint position path 58D is superimposed on overhead image 56 and an image in which the shape of viewpoint position path 58D is superimposed on the side view may be displayed side by side to the right of thumbnail image 62E.

[0143] 20 , when a user touches a thumbnail image 62E displayed on the recommendation screen 70 with a finger, a stylus pen, or the like, the user device 12 transmits a digest image display instruction 74 for the touched thumbnail image 62E to the video playback unit 28. When the video playback unit 28 receives the digest image display instruction 74 from the user device 12, it reads out the virtual viewpoint video 30 corresponding to the touched thumbnail image 62E from the database 22. The digest image display instruction 74 is an example of a “digest image display instruction” according to the technology of the present disclosure.

[0144] The video playback unit 28 extracts images (hereinafter referred to as "extracted images") 27 (see FIG. 21) of multiple frames whose viewpoint positions differ at predetermined intervals from the virtual viewpoint video 30 read from the database 22. The video playback unit 28 generates a digest image 76 (see FIG. 21) including the extracted image 27. The video playback unit 28 transmits digest data including the generated digest image 76 and layout information of the digest image 76 to the user device 12. The digest image 76 is an example of a "digest image" according to the technology of the present disclosure. The digest data is also an example of "second data" according to the technology of the present disclosure.

[0145] 21, the user device 12 receives the digest data and displays the received digest data on the touch panel display 12A, so that, for example, a digest image 76 including four extracted images 27 is displayed within the display frame of the thumbnail image 62E.

[0146] As described above, in the second embodiment, the recommended data 32 is data in which the thumbnail images 62E and viewpoint position paths 58D of untransmitted virtual viewpoint videos 30 are displayed on the user device 12 in order of decreasing similarity between the viewpoint-related information 58 and the viewpoint-related information 58 included in the played video list 64 and increasing similarity between the elapsed game time 62C and the elapsed game time 62C included in the played video list 64. Therefore, with this configuration, it is possible to make the user more likely to select a virtual viewpoint video 30 that has a low similarity between the viewpoint-related information 58 and the viewpoint-related information 58 included in the played video list 64 and a high similarity between the elapsed game time 62C and the elapsed game time 62C included in the played video list 64, compared to when the thumbnail images 62E and viewpoint position paths 58D of the untransmitted virtual viewpoint videos 30 are randomly displayed on the user device 12.

[0147] Furthermore, in the second embodiment, the video playback unit 28 derives the similarity regarding the viewpoint-related information 58 by comparing the position of the viewpoint position path 58D between the viewpoint-related information 58 included in the played video list 64 and the viewpoint-related information 58 included in the recommended data 32. Therefore, according to this configuration, the similarity can be derived with higher accuracy than when the similarity is derived without comparing any data.

[0148] Furthermore, in the second embodiment, the recommended data 32 includes the thumbnail image 62E of the untransmitted virtual viewpoint video 30 and the viewpoint-related information 58. Therefore, according to this configuration, it is possible to make it easier for the user to select the virtual viewpoint video 30 that the user intends, compared to when the recommended data 32 does not include either the thumbnail image 62E of the untransmitted virtual viewpoint video 30 or the viewpoint-related information 58.

[0149] Moreover, in the second embodiment, the viewpoint-related information 58 includes a viewpoint position path 58D and a point of interest 58E. Therefore, according to this configuration, compared to when the viewpoint-related information 58 does not include either the viewpoint position path 58D or the point of interest 58E, it is possible to make it easier for the user to select a virtual viewpoint video 30 that is related to at least one of the viewpoint position path 58D and the point of interest 58E from among the untransmitted virtual viewpoint videos 30.

[0150] Furthermore, in the second embodiment, when the video playback unit 28 receives a digest image display instruction 74 from the user device 12, it generates a digest image 76 by extracting extracted images 27 of multiple frames from the virtual viewpoint video 30, each of which has a different viewpoint position 58A at a predetermined interval. The video playback unit 28 transmits playback data for the generated digest image 76 to the user device 12, which is used to play the generated digest image 76 on the user device 12. Therefore, with this configuration, the user can select a virtual viewpoint video 30 with reference to the digest image 76. The selected virtual viewpoint video 30 is played on the touch panel display 12A, as in the first embodiment. Note that the operation for playing a digest image is not limited to the above. For example, a digest image may be played when the user presses and holds a thumbnail image 62E displayed on the recommendation screen 70 with a finger, a stylus pen, or the like, and the virtual viewpoint video 30 may be played when the user taps the thumbnail image 62E displayed on the recommendation screen 70 with a finger, a stylus pen, or the like.

[0151] In the second embodiment, when the user touches the thumbnail image 62E displayed on the recommendation screen 70, the user device 12 transmits the digest image display instruction 74 for the touched thumbnail image 62E to the video playback unit 28. However, the technology of the present disclosure is not limited to this. When the user's finger, stylus pen, or the like is placed over the thumbnail image 62E, or when the user touches a digest button (not shown) provided on the recommendation screen 70, the user device 12 may transmit the digest image display instruction 74 to the video playback unit 28.

[0152] Furthermore, in the second embodiment, the viewpoint-related information 58 includes the point of gaze 58E. However, the technology of the present disclosure is not limited to this. Instead of the point of gaze 58E, the viewpoint-related information 58 may include a point of gaze path indicating the displacement of the point of gaze 58E over time. In this case, the video playback unit 28 may derive the similarity of the viewpoint-related information 58 by comparing the positions of the point of gaze paths between the viewpoint-related information 58 included in the played video list 64 and the viewpoint-related information 58 included in the recommended data 32. The point of gaze path is an example of a "point of gaze path" according to the technology of the present disclosure. This configuration makes it easier for the user to select a virtual viewpoint video 30 related to the point of gaze path from among the untransmitted virtual viewpoint videos 30, compared to when the viewpoint-related information 58 does not include a point of gaze path.

[0153] In the second embodiment, the video playback unit 28 derives the similarity regarding the viewpoint-related information 58 by comparing the positions of the viewpoint position paths 58D between the viewpoint-related information 58 included in the played video list 64 and the viewpoint-related information 58 included in the recommended data 32. However, the technology of the present disclosure is not limited to this. As an example, as shown in FIG. 22, the video playback unit 28 derives the similarity regarding the object appearing in the image obtained by observing the fixation point 58E from the start viewpoint position, the intermediate viewpoint position, and the end viewpoint position among the multiple viewpoint positions 58A included in each viewpoint position path 58D. may be extracted using image recognition technology, and tags 78 indicating the extracted objects may be assigned to each viewpoint position path 58D. The video playback unit 28 may derive the similarity of the viewpoint-related information 58 by comparing the tags 78 assigned to the viewpoint position paths 58D. The tags 78 are an example of a "tag" according to the technology of the present disclosure.

[0154] In the example shown in FIG. 22, the viewpoint position path 58D-1 is assigned three tags: "goal," "spectator seats," and "player 11." The viewpoint position path 58D-2 is assigned three tags: "goal," "goal," and "player 11." The viewpoint position path 58D-3 is assigned three tags: "ball," "player 11," and "player 11." Player 11 refers to the player wearing the number 11 on his back. The players are identified using, for example, well-known face recognition technology.

[0155] When the tags 78 assigned to the viewpoint position path 58D-1 are compared with the tags 78 assigned to the viewpoint position path 58D-2, the tags "goal" and "player 11" are common. When the tags 78 assigned to the viewpoint position path 58D-1 are compared with the tags 78 assigned to the viewpoint position path 58D-3, the tag "player 11" is common. Therefore, the viewpoint position path 58D-2 has a higher similarity to the viewpoint position path 58D-1 than the viewpoint position path 58D-3.

[0156] 22, the similarity can be derived with higher accuracy than when the similarity is derived without comparing any data. Note that instead of the viewpoint position path 58D, a tag may be attached to the gaze point path, and the similarity regarding the viewpoint-related information 58 may be derived by comparing the tags attached to the gaze point paths.

[0157] Furthermore, in the second embodiment, the shape of the viewpoint position path 58D of the untransmitted virtual viewpoint video 30 is displayed superimposed on the overhead image 56 on the recommendation screen 70, but the technology of the present disclosure is not limited to this. As an example, as shown in FIG. 23 , in addition to the shape of the viewpoint position path 58D of the untransmitted virtual viewpoint video 30, the shape of the viewpoint position path 58D of the viewpoint-related information 58 included in the played video list 64 may be displayed superimposed on the overhead image 56.

[0158] In this case, the video playback unit 28 generates a path image in which the shape of the viewpoint position path 58D of the viewpoint-related information 58 included in the played video list 64 is displayed in association with the recommended data 32, and transmits the generated path image to the user device 12. In the path image, the shape of the viewpoint position path 58D of the viewpoint-related information 58 included in the played video list 64 is displayed in a manner that allows it to be distinguished from the shape of the viewpoint position path 58D of the untransmitted virtual viewpoint video 30 included in the recommended data 32. Note that the path image is an example of a "path image" according to the technology of the present disclosure.

[0159] 23, the user can select a virtual viewpoint video 30 by referring to the shape of viewpoint position path 58D of viewpoint-related information 58 included in played video list 64. When the shape of the point of interest path of an untransmitted virtual viewpoint video 30 is displayed superimposed on overhead image 56 on recommendation screen 70, the shape of the point of interest path of viewpoint-related information 58 included in played video list 64 may also be displayed superimposed on overhead image 56.

[0160] In the second embodiment, an example in which the recommendation screen 70 shown in Fig. 19 is displayed on the touch panel display 12A has been described, but the technology of the present disclosure is not limited to this. Hereinafter, another example of the recommendation screen 70 will be described with reference to Figs. 24 and 25.

[0161] 24 may be displayed on the touch panel display 12A. The recommended data 32 includes a seek bar 82, an overhead image 56, a thumbnail image 62E of the untransmitted virtual viewpoint video 30 included in the recommended data 32, and a viewpoint position path 58D. The shape includes the shape of the thumbnail image 62E of the virtual viewpoint video 30 included in the played video list 64 and the shape of the viewpoint position path 58D.

[0162] On the recommendation screen 70, the seek bar 82 indicates the elapsed match time 62C of the virtual viewpoint video 30 included in the recommended data 32. The shapes of the viewpoint position paths 58D of the virtual viewpoint videos 30 included in the played video list 64 and the shapes of the viewpoint position paths 58D of the untransmitted virtual viewpoint videos 30 included in the recommended data 32 are displayed on the recommendation screen 70 in a distinguishable manner. Each thumbnail image 62E is displayed at a corresponding position on each viewpoint position path 58D.

[0163] That is, the video playback unit 28 acquires information about the viewpoint from which each thumbnail image 62E was observed among the multiple viewpoints included in each viewpoint path 58D, and generates recommendation data 32 including first position information that can identify the position of each thumbnail image 62E on each viewpoint path 58D. The first position information is an example of "first position information" according to the technology of the present disclosure.

[0164] 24, compared to when the thumbnail image 62E and the viewpoint position path 58D are displayed separately on the recommendation screen 70, the user can select a virtual viewpoint video 30 by referring to the position of the thumbnail image 62E relative to the viewpoint position path 58D. Note that the recommendation screen 70 may be displayed including a gaze point path instead of the viewpoint position path 58D. In this case, the recommendation data 32 includes position information that can identify the position of the thumbnail image 62E on the gaze point path. Furthermore, in the example shown in FIG. 24, the viewpoint position path 58D and thumbnail image 62E related to one untransmitted virtual viewpoint video 30 are displayed on the recommendation screen 70, but the viewpoint position paths 58D and thumbnail images 62E related to multiple untransmitted virtual viewpoint videos 30 may be displayed on the recommendation screen 70.

[0165] 19 is displayed on the touch panel display 12A, but as an example, the recommendation screen 70 shown in Fig. 25 may be displayed on the touch panel display 12A. The recommendation screen 70 includes a seek bar 82 and video information 62 related to the multiple virtual viewpoint videos 30 included in the recommendation data 32, that is, a video ID 62A, a creator 62B, a game elapsed time 62C, a video length 62D, and a thumbnail image 62E.

[0166] On the recommendation screen 70, the seek bar 82 indicates the start time to the end time of the match. The video information 62 related to the virtual viewpoint video 30 is broadly categorized by the elapsed match time 62C included in the video information 62, and is arranged horizontally in chronological order. Specifically, video information 62 related to the virtual viewpoint video 30 whose start time of the elapsed match time 62C is between 0 and 10 minutes is arranged in the leftmost column of the recommendation screen 70, video information 62 related to the virtual viewpoint video 30 whose start time of the elapsed match time 62C is between 11 and 20 minutes is arranged in the second column from the left of the recommendation screen 70, video information 62 related to the virtual viewpoint video 30 whose start time of the elapsed match time 62C is between 21 and 30 minutes is arranged in the third column from the left of the recommendation screen 70, and video information 62 related to the virtual viewpoint video 30 whose start time of the elapsed match time 62C is between 31 and 45 minutes is arranged in the rightmost column of the recommendation screen 70.

[0167] In each column, the video information 62 relating to the virtual viewpoint video 30 is arranged from top to bottom in descending order of the viewpoint position pass difference with respect to the viewpoint position path 58D of the viewpoint-related information 58 included in the played video list 64, that is, in descending order of the similarity with respect to the viewpoint position path 58D. Note that the recommendation screen 70 shown in FIG. 25 is only a small part of the screen, and the recommendation screen 70 can be scrolled left and right and up and down on the touch panel display 12A. The division of the elapsed time of the match is not limited to the above division and any division may be used. Also, information on the vertical axis relating to the viewpoint position pass difference may be displayed on the touch panel display 12A. In this embodiment, In this embodiment, the information on the vertical axis relating to the viewpoint position path difference is, for example, information indicating that the viewpoint position path difference is larger the higher it is located, and smaller the viewpoint position path difference is located lower.

[0168] That is, the recommendation data 32 includes second position information that arranges video information 62 related to multiple virtual viewpoint videos 30 horizontally in chronological order of the elapsed match time 62C, and vertically in descending order of similarity regarding the viewpoint position path 58D with the viewpoint-related information 58 included in the played video list 64. The second position information is an example of the "second position information" related to the technology of the present disclosure.

[0169] 25 makes it easier for the user to select the virtual viewpoint video 30 that the user intends, compared to when video information 62 related to multiple virtual viewpoint videos 30 is randomly arranged. Note that in the example shown in Fig. 25, video information 62 related to virtual viewpoint videos 30 is arranged from top to bottom in descending order of viewpoint position path difference from the viewpoint position path 58D of viewpoint-related information 58 included in played video list 64, but the technology of the present disclosure is not limited to this, and video information 62 related to virtual viewpoint videos 30 may be arranged from top to bottom in descending order of viewpoint position path difference from the viewpoint position path 58D of viewpoint-related information 58 included in played video list 64.

[0170] In the first and second embodiments, an example has been described in which the video playback unit 28 generates recommended data 32 based on the similarity with the viewpoint-related information 58 included in the played video list 64 and the similarity with the elapsed game time 62C, but the technology of the present disclosure is not limited to this. Hereinafter, with reference to Figs. 26 to 39, an example will be described in which recommended data 32 is generated based on factors other than the similarity in the viewpoint-related information 58 and the elapsed game time 62C between the virtual viewpoint video 30 and the untransmitted virtual viewpoint video 30 included in the played video list 64.

[0171] 26, the video playback unit 28 may acquire registration data of the user and generate recommended data 32 based on the acquired registration data. One example of the registration data is attribute data indicating the attributes of the user.

[0172] In the example shown in FIG. 26, the played video list 64 corresponding to the account of user A records attribute data 64B of user A indicating "Japan." This attribute data indicates that user A is a fan of the "Japan" team. The attribute data may be registered in the data processing device 10 from the user device 12, or may be registered in the data processing device 10 from another device. The attribute data may be registered when the user registers an account, or may be included in a video playback request 52 received from the user device 12. Note that, in addition to the name of the user's favorite team, examples of the attribute data include the user's gender, age, and favorite player names. Note that the attribute data is an example of "attribute data" according to the technology of the present disclosure.

[0173] The video playback unit 28 may generate recommended data 32 that displays thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in descending order of similarity between the attribute data 64B of the user included in the played video list 64 and the attribute data 62F of the creator 62B of the untransmitted virtual viewpoint video 30. For example, in the example shown in FIG. 26 , the attribute data 64B of user A matches the attribute data 62F of creator C, so the video playback unit 28 weights the virtual viewpoint video 30 created by creator C. Note that the attribute data 64B is not limited to one attribute, and the video playback unit 28 may weight the virtual viewpoint video 30 according to the similarity of multiple attributes. The video playback unit 28 generates a recommended video list 66 by rearranging the virtual viewpoint videos 30 in the generated video list 60 in descending order of the weights assigned to the virtual viewpoint videos 30, and creates recommended data 32 according to the generated recommended video list 66.

[0174] 26, the video playback unit 28 generates recommended data 32 based on the user's registered data. Therefore, this configuration makes it easier for the user to select the virtual viewpoint video 30 that the user desires, compared to when the recommended data 32 is generated without taking the user's registered data into consideration.

[0175] 26, the user's registered data is attribute data 64B indicating the user's attributes. Therefore, this configuration makes it easier for the user to select the virtual viewpoint video 30 that the user desires, compared to when recommended data 32 is generated without taking the user's attribute data 64B into consideration.

[0176] 26, the recommended data 32 is data in which thumbnail images 62E and / or viewpoint-related information 58 are displayed on the user device 12 in an order determined based on the degree of similarity between attribute data 64B of the user using the user device 12 and attribute data 62F of the creator 62B of the virtual viewpoint video 30. Therefore, with this configuration, compared to when the recommended data 32 related to the virtual viewpoint video 30 is displayed randomly on the user device 12, it is possible to make it easier for the user to select the virtual viewpoint video 30 that the user intends based on the degree of similarity between the attribute data 64B of the user and the attribute data 62F of the creator 62B. Note that Figure 26 shows an example of an embodiment in which recommended data 32 is generated in order of decreasing similarity between attribute data 64B of user A included in played video list 64 and attribute data 62F of author 62B of virtual perspective video 30, but the technology disclosed herein is not limited to this, and recommended data 32 may also be generated in order of decreasing similarity between attribute data 64B of user A included in played video list 64 and attribute data 62F of author 62B of virtual perspective video 30.

[0177] As an example, as shown in FIG. 27, the video playback unit 28 may generate recommended data 32 that displays thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined based on playback trends extracted from the playback history of the virtual viewpoint video 30 shown in the played video list 64 corresponding to the account of the user using the user device 12.

[0178] In the example shown in FIG. 27, the played video list 64 corresponding to the account of user A records information about one virtual viewpoint video 30 created by user A and information about two virtual viewpoint videos 30 created by user E. The video playback unit 28 refers to the played video list 64 and determines that user A prefers to watch the virtual viewpoint videos 30 created by user E. The video playback unit 28 assigns a weight to the virtual viewpoint videos 30 whose creator 62B is "E" among the virtual viewpoint videos 30 recorded in the generated video list 60. The video playback unit 28 generates a recommended video list 66 by rearranging the virtual viewpoint videos 30 in the generated video list 60 in descending order of weight, and creates recommended data 32 in accordance with the generated recommended video list 66.

[0179] According to the configuration shown in FIG. 27, compared to when the recommended data 32 includes information on randomly selected virtual viewpoint videos 30, it is possible to make it easier for the user to select the virtual viewpoint video 30 that the user desires based on the user's tendency to play the virtual viewpoint video 30.

[0180] Furthermore, in the example shown in FIG. 27, the video playback unit 28 extracts playback trends from the playback history of the virtual viewpoint video 30 shown in the played video list 64, but the technology of the present disclosure is not limited to this. The user may pre-register their favorite creator "E" as preference data indicating the user's preferences, and the video playback unit 28 may create recommended data 32 based on the user's preference data. The preference data is an example of "preference data" according to the technology of the present disclosure. With this configuration, it is easier for the user to select the virtual viewpoint video 30 that the user desires, compared to when the recommended data 32 is generated without taking the user's preference data into consideration. Note that the preference data Examples of the data include a favorite author, a favorite viewpoint position, a favorite line of sight direction, a favorite angle of view, and the like.

[0181] As an example, as shown in FIG. 28, the video playback unit 28 may generate recommendation data 32 that displays thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined based on the results of comparing the playback history of virtual viewpoint videos 30 shown in the played video list 64 corresponding to the account of the user using the user device 12 with the playback history of virtual viewpoint videos 30 shown in the played video list 64 corresponding to the account of another user.

[0182] In the example shown in FIG. 28 , the played video list 64 corresponding to the account of user A records information about virtual viewpoint videos 30 having video IDs 62A of “001” and “003.” The virtual viewpoint videos 30 having video IDs 62A of “001” and “003” are the playback history of virtual viewpoint videos 30 by user A, i.e., user A's viewing history. The video playback unit 28 reads out the played video list 64 corresponding to the account of user C, which has a viewing history similar to that of user A. The video playback unit 28 compares the viewing history of user A with the viewing history of user C, and assigns weights to the virtual viewpoint videos 30 included in the viewing history of user C but not included in the viewing history of user A, among the virtual viewpoint videos 30 recorded in the generated video list 60. That is, in the example shown in FIG. 28 , the video playback unit 28 assigns weights to the virtual viewpoint video 30 having the video ID 62A of “004.” The video playback unit 28 rearranges the virtual viewpoint videos 30 in the generated video list 60 in descending order of weight, and stores the rearranged generated video list 60 as a recommended video list 66. The video playback unit 28 creates recommended data 32 in accordance with the generated recommended video list 66.

[0183] According to the configuration shown in FIG. 28, compared to when the recommended data 32 includes information on randomly selected virtual viewpoint videos 30, it is possible to make it easier for the user to select the virtual viewpoint video 30 that the user desires based on the comparison results between the playback history of the virtual viewpoint video 30 shown in the played video list 64 corresponding to the user's account and the playback history of the virtual viewpoint video 30 shown in the played video list 64 corresponding to another user's account.

[0184] As an example, as shown in FIG. 29, the video playback unit 28 may generate recommended data 32 that displays thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined based on the comments added to each virtual viewpoint video 30.

[0185] In the example shown in Fig. 29, a comment 86 is added to the viewpoint position path 58D of each virtual viewpoint video 30. The comment 86 may be added by the creator of each virtual viewpoint video 30, or may be added by a viewer who has viewed each virtual viewpoint video 30. The comment 86 is recorded in the generated video list 60 in association with a position on the viewpoint position path 58D of each virtual viewpoint video 30. Note that the comment 86 is an example of a "comment" according to the technology of the present disclosure.

[0186] The video playback unit 28 weights the virtual viewpoint videos 30 according to the number of comments 86 added. That is, in the example shown in FIG. 29 , a virtual viewpoint video 30 to which two comments 86 have been added is weighted more heavily than a virtual viewpoint video 30 to which one comment 86 has been added. The video playback unit 28 generates a recommended video list 66 by rearranging the virtual viewpoint videos 30 in the generated video list 60 in descending order of weight, and generates recommended data 32 according to the generated recommended video list 66. The video playback unit 28 transmits the generated recommended data 32 to the user device 12.

[0187] 30 , the user device 12 receives the recommended data 32 and displays the received recommended data 32 on the touch panel display 12A as a recommendation screen 70. On the recommendation screen 70, the comment 86 is associated with a position on the viewpoint position path 58D and is displayed superimposed on the overhead image 56. In other words, the recommended data 32 is data displayed on the user device 12 as an image with the comment 86 added in the form of the viewpoint position path 58D.

[0188] 29, compared to when the recommendation data 32 includes information on randomly selected virtual viewpoint videos 30, the configuration shown in Fig. 29 makes it easier for the user to select the virtual viewpoint video 30 that the user desires based on the comments 86 added to each virtual viewpoint video 30. Also, according to the configuration shown in Fig. 30, compared to when only the comments 86 are displayed on the recommendation screen 70, the configuration makes it easier for the user to select the virtual viewpoint video 30 that the user desires. Note that the comment 86 may be added to the gaze point path instead of the viewpoint position path 58D.

[0189] As an example, as shown in FIG. 31 , a rating 59 may be assigned to each virtual viewpoint video 30 in the generated video list 60. The rating 59 is a rating assigned by a user who has viewed each virtual viewpoint video 30. The rating 59 may be a rating assigned by a single user, or may be the average or most frequent value of ratings assigned by multiple users. The video playback unit 28 may generate recommendation data 32 that displays thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined based on the rating 59. The rating 59 is an example of a "first rating" according to the technology of the present disclosure.

[0190] In the example shown in FIG. 31, a rating of 3 stars 59 has been given to the virtual viewpoint video 30 having a video ID 62A of "011".

[0191] The virtual viewpoint videos 30 having the video IDs 62A of "012" and "013" have been given a rating 59 of two stars. The video playback unit 28 weights the virtual viewpoint videos 30 according to the number of stars. In other words, the video playback unit 28 generates a recommended video list 66 by rearranging the virtual viewpoint videos 30 in the generated video list 60 in descending order of the number of stars, and creates recommended data 32 according to the generated recommended video list 66.

[0192] According to the configuration shown in FIG. 31, compared to when the recommended data 32 includes information on randomly selected virtual viewpoint videos 30, it is possible to make it easier for the user to select the virtual viewpoint video 30 of their choice based on the rating 59 assigned to each virtual viewpoint video 30.

[0193] As an example, as shown in FIG. 32 , for example, in a played video list 64 corresponding to user A's account, each virtual viewpoint video 30 may be assigned a rating 90. The rating 90 is an evaluation assigned by user A. The video playback unit 28 estimates user A's preferences based on the ratings 90 assigned to each virtual viewpoint video 30. The video playback unit 28 may generate recommendation data 32 that displays thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined according to the estimated preferences of user A. The rating 90 is an example of a "second evaluation" according to the technology of the present disclosure.

[0194] In the example shown in FIG. 32, the virtual viewpoint video 30 created by creator A has been given a rating of one star 90. The virtual viewpoint video 30 created by creator B has been given a rating of two stars 90. The virtual viewpoint video 30 created by creator E has been given a rating of three stars 90. Based on the fact that the virtual viewpoint video 30 created by creator E has been given a high rating 90, the video playback unit 28 estimates that user A likes the virtual viewpoint video 30 created by creator E. The video playback unit 28 assigns a weight to the virtual viewpoint video 30 created by creator E in the generated video list 60. The video playback unit 28 rearranges the virtual viewpoint videos 30 in the generated video list 60 in descending order of weight, and stores the rearranged generated video list 60 as a recommended video list 66. The video playback unit 28 creates recommendation data 32 in accordance with the generated recommended video list 66.

[0195] According to the configuration shown in FIG. 32, compared to when the recommended data 32 includes information on randomly selected virtual viewpoint videos 30, it is easier for the user to select the virtual viewpoint video 30 that the user desires based on the user's preferences.

[0196] As an example, as shown in FIG. 33, when a new virtual viewpoint video 30 is added to the generated video list 60, the video playback unit 28 may generate recommendation data 32 that displays a thumbnail image 62E and / or viewpoint-related information 58 related to the added virtual viewpoint video 30 on the user device 12 based on the registration data of the user using the user device 12.

[0197] In the example shown in FIG. 33, a user using the user device 12 has registered creator E as a favorite creator. When a virtual viewpoint video 30 newly created by creator E is added to the generated video list 60, the video playback unit 28 assigns a weight to the added virtual viewpoint video 30. The video playback unit 28 sorts the virtual viewpoint videos 30 in the generated video list 60 in descending order of weight, and stores the sorted generated video list 60 as a recommended video list 66. The video playback unit 28 creates recommended data 32 in accordance with the generated recommended video list 66.

[0198] The recommended data 32 is displayed on the touch panel display 12A as a recommendation screen 70. Information about a virtual viewpoint video 30 newly created by creator E may be displayed at the top of the recommendation screen 70 in a manner that allows it to be distinguished from information about other virtual viewpoint videos 30.

[0199] The configuration shown in FIG. 33 makes it easier for the user to select the virtual viewpoint video 30 that the user desires based on the user's registration data, compared to when the recommended data 32 includes only information about existing virtual viewpoint videos 30. Note that in the example shown in FIG. 33, the video playback unit 28 creates the recommended data 32 including information about the virtual viewpoint video 30 that has been newly added to the generated video list 60 based on the user's registration data, but the technology of the present disclosure is not limited to this. The video playback unit 28 may create the recommended data 32 including information about the virtual viewpoint video 30 that has been newly added to the generated video list 60 based on the playback history of the virtual viewpoint video 30 shown in the played video list 64 corresponding to the user's account, instead of the user's registration data. Furthermore, the recommended data 32 may be created in a similar manner to the above based on the virtual viewpoint video 30 that has been registered as a favorite in the user's account, regardless of the playback history corresponding to the user's account.

[0200] As an example, as shown in FIG. 34, the video playback unit 28 may generate recommendation data 32 that displays thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined based on the length of a viewpoint position path 58D relative to the length 62D of the video. The length of the viewpoint position path 58D relative to the length 62D of the video indicates the degree of change in viewpoint position in the virtual viewpoint video 30. In other words, if the viewpoint position path 58D is long relative to the length 62D of the video, it can be estimated that the virtual viewpoint video 30 is a video with large changes in viewpoint position. On the other hand, if the viewpoint position path 58D is short relative to the length 62D of the video, it can be estimated that the virtual viewpoint video 30 is a video with small changes in viewpoint position. The degree of change in viewpoint position is expressed as the ratio of the length of the viewpoint position path 58D to the length 62D of the video (hereinafter referred to as the "viewpoint position change ratio").

[0201] In the example shown in FIG. 34, in the played video list 64 corresponding to the account of user A, Therefore, a high rating of 90 has been given to the virtual viewpoint video 30 having the video ID 62A of "015." The video playback unit 28 calculates the viewpoint position change ratio of the virtual viewpoint video 30 with a high rating of 90, and if the calculated viewpoint position change ratio is smaller than a predetermined value, it is estimated that user A prefers virtual viewpoint videos 30 with small changes in viewpoint position. The video playback unit 28 weights the virtual viewpoint videos 30 with viewpoint position change ratios smaller than the predetermined value in the generated video list 60. The video playback unit 28 sorts the virtual viewpoint videos 30 in the generated video list 60 in descending order of weight, and stores the sorted generated video list 60 as a recommended video list 66. The video playback unit 28 creates recommendation data 32 in accordance with the generated recommended video list 66. Note that if the calculated viewpoint position change ratio is larger than the predetermined value, the video playback unit 28 may weight the virtual viewpoint videos 30 with viewpoint position change ratios larger than the predetermined value in the generated video list 60.

[0202] The configuration shown in FIG. 34 makes it easier for the user to select the virtual viewpoint video 30 that the user desires based on the length of the viewpoint position path 58D relative to the video length 62D, compared to when the recommendation data 32 includes information on randomly selected virtual viewpoint videos 30. In the example shown in FIG. 34, the video playback unit 28 estimates the user's preferences by comparing the viewpoint position change ratio with a preset value, but the technology of the present disclosure is not limited to this. The video playback unit 28 may also estimate the user's preferences by comparing the calculated viewpoint position change ratio with the average or most frequent value of the viewpoint position change ratio. In the example shown in FIG. 34, the video playback unit 28 generates the recommendation data 32 based on the length of the viewpoint position path 58D relative to the video length 62D, but the technology of the present disclosure is not limited to this. The video playback unit 28 may also generate the recommendation data 32 based on the length of the gaze point path relative to the video length 62D. The example of referencing the viewpoint position path 58D is not limited to this. For example, the recommended data 32 may be created by assigning a high weight to the viewpoint position path 58D of the virtual viewpoint video 30 that has been given a high rating of 90, and arranging the virtual viewpoint videos 30 in order of similarity to the viewpoint position path 58D. Alternatively, the recommended data 32 may be created by assigning a high weight to both the creator of the virtual viewpoint video 30 that has been given a high rating of 90 and the viewpoint position path 58D.

[0203] 35, the video playback unit 28 may generate recommendation data 32 that displays thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined based on the length of the elapsed game time 62C relative to the video length 62D. The length of the elapsed game time 62C relative to the video length 62D indicates the average speed of the virtual viewpoint video 30. In other words, a virtual viewpoint video 30 in which the elapsed game time 62C is long relative to the video length 62D, i.e., a video with a fast average speed, is a fast-forwarded video that is a shortened version of a long captured video 34. The length of the elapsed game time 62C is an example of "second time information" according to the technology of the present disclosure.

[0204] 35, the video playback unit 28 calculates the average speed of each virtual viewpoint video 30 recorded in the generated video list 60, and rearranges the virtual viewpoint videos 30 in the generated video list 60 in descending order of the calculated average speed. The video playback unit 28 stores the rearranged generated video list 60 as a recommended video list 66. The video playback unit 28 creates recommended data 32 in accordance with the generated recommended video list 66.

[0205] As an example, as shown in FIG. 36, a recommendation screen 70 displays recommended data 32 generated based on the average speed of the virtual viewpoint video 30, and a seek bar 95 is displayed below the overhead image 56. In the seek bar 95, portions of the virtual viewpoint video 30 where the speed is faster than a default value (hereinafter referred to as "fast-forward portions") are indicated by hatching. Portions of the virtual viewpoint video 30 where the speed is slower than a default value (hereinafter referred to as "slow-forward portions") are presumed to correspond to noteworthy scenes such as good plays and / or goal scenes. A game elapsed time 62C is displayed corresponding to each slow-forward portion. Furthermore, in the viewpoint position path 58D, fast-forward portions and slow-forward portions are displayed in a manner that allows them to be distinguished.

[0206] According to the configurations shown in Figures 35 and 36, compared to when the recommended data 32 includes information on randomly selected virtual viewpoint videos 30, it is easier for the user to select the virtual viewpoint video 30 of their choice based on the length of the game elapsed time 62C relative to the video length 62D.

[0207] 37 as an example, the generated moving image list 60 stores the data capacity 96 of each virtual viewpoint moving image 30. The moving image playback unit 28 may generate recommended data 32 for displaying thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined based on the data capacity 96 of each virtual viewpoint moving image 30. Note that the data capacity 96 is an example of the "data capacity" according to the technology of the present disclosure.

[0208] The video playback unit 28 determines the communication speed of the user device 12 and rearranges the virtual viewpoint videos 30 in the generated video list 60 according to the determined communication speed of the user device 12. In other words, if the communication speed of the user device 12 is slower than a default value, the virtual viewpoint videos 30 in the generated video list 60 are rearranged in ascending order of data capacity 96. On the other hand, if the communication speed of the user device 12 is faster than the default value, the virtual viewpoint videos 30 in the generated video list 60 are rearranged in descending order of data capacity 96. The video playback unit 28 stores the rearranged generated video list 60 as a recommended video list 66. The video playback unit 28 creates recommended data 32 according to the generated recommended video list 66.

[0209] According to the configuration shown in FIG. 37, compared to when the recommended data 32 includes information on randomly selected virtual viewpoint videos 30, it is possible to make it easier for the user to select the virtual viewpoint video 30 of their choice based on the data capacity 96 of each virtual viewpoint video 30.

[0210] 37, the video playback unit 28 generates the recommended data 32 based on the data capacity 96 of the virtual viewpoint videos 30, but the technology of the present disclosure is not limited to this. The video playback unit 28 may generate the recommended data 32 to display the thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined based on the resolution 97 of each virtual viewpoint video 30 instead of the data capacity 96 of the virtual viewpoint video 30.

[0211] As an example, as shown in FIG. 38 , the generated video list 60 stores a resolution 97 of each virtual viewpoint video 30. The video playback unit 28 rearranges the virtual viewpoint videos 30 in the generated video list 60 according to the communication speed of the user device 12, and stores the rearranged generated video list 60 as a recommended video list 66. The video playback unit 28 creates recommended data 32 in accordance with the generated recommended video list 66. The resolution 97 is an example of an "image quality parameter" according to the technology of the present disclosure.

[0212] 38, compared to when the recommendation data 32 includes information on randomly selected virtual viewpoint videos 30, it is possible to make it easier for the user to select the virtual viewpoint video 30 that the user desires based on the resolution 97 of each virtual viewpoint video 30. Note that in the example shown in FIG. 38, the video playback unit 28 generates the recommendation data 32 based on the resolution 97 of each virtual viewpoint video 30, but the technology of the present disclosure is not limited to this. The video playback unit 28 may generate the recommendation data 32 based on the frame rate of each virtual viewpoint video 30 instead of the resolution 97 of each virtual viewpoint video 30.

[0213] 39, the generated moving image list 60 includes information on whether or not audio data 98 ​​has been added to each virtual viewpoint video 30. The audio data 98 ​​includes the voice of a commentator, the voice of a player, cheers, singing, and the like, which have been added to each virtual viewpoint video 30. The audio data 98 ​​includes audio included in the captured video 34 used to generate the virtual viewpoint video 30. may be added as is, or may be added to the virtual viewpoint video 30 after generation.

[0214] The video playback unit 28 may generate recommendation data 32 that displays thumbnail images 62E and / or viewpoint-related information 58 on the user device 12 in an order determined based on whether or not audio data 98 ​​is assigned to each virtual viewpoint video 30. In the example shown in FIG. 39, the video playback unit 28 rearranges the virtual viewpoint videos 30 in the generated video list 60 so that virtual viewpoint videos 30 that have audio data 98 ​​are positioned higher than virtual viewpoint videos 30 that do not have audio data 98, and stores the rearranged generated video list 60 as a recommended video list 66. The video playback unit 28 creates recommendation data 32 in accordance with the generated recommended video list 66. The audio data 98 ​​is an example of "audio data" according to the technology of the present disclosure.

[0215] According to the configuration shown in FIG. 39, compared to when the recommended data 32 includes information on randomly selected virtual viewpoint videos 30, it is possible to make it easier for the user to select the virtual viewpoint video 30 of their choice based on the presence or absence of audio data 98 ​​assigned to each virtual viewpoint video 30.

[0216] In the first and second embodiments described above, an example in which one virtual viewpoint video 30 is played on the user device 12 in response to a playback instruction 72 transmitted from the user device 12 by a user operation has been described, but the technology of the present disclosure is not limited to this. Hereinafter, with reference to Figs. 40 to 44, an example in which multiple virtual viewpoint videos 30 are played on the user device 12 in response to a playback instruction 72 transmitted from the user device 12 will be described.

[0217] 40 , the video playback unit 28 generates recommendation data 32 in any of the above-described embodiments and transmits the generated recommendation data 32 to the user device 12. The user device 12 receives the recommendation data 32 and displays the received recommendation data 32 as a recommendation screen 70 on the touch panel display 12A.

[0218] When the user touches the thumbnail image 62E displayed on the recommendation screen 70, the user device 12 transmits a playback instruction 72 to the video playback unit 28. When the video playback unit 28 receives the playback instruction 72 from the user device 12, it may read five virtual viewpoint videos 30 having the information displayed on the recommendation screen 70 from the database 22, and generate playback data 102 in which the five read virtual viewpoint videos 30 are sequentially played on the user device 12. The video playback unit 28 transmits the generated playback data 102 to the user device 12. The user device 12 receives the playback data 102 and plays the received playback data 102 on the touch panel display 12A. The playback data 102 is an example of "second data" according to the technology of the present disclosure.

[0219] The phrase "five virtual viewpoint videos 30 are played sequentially on the user device 12" means that the next virtual viewpoint video 30 may start playing immediately after the playback of one virtual viewpoint video 30 ends, or the next virtual viewpoint video 30 may start playing before the playback of one virtual viewpoint video 30 ends. Furthermore, the next virtual viewpoint video 30 may start playing after a predetermined time has elapsed since the playback of one virtual viewpoint video 30 ends. Furthermore, the five virtual viewpoint videos 30 may continue to be played repeatedly until a stop instruction is sent from the user device 12 by a user operation.

[0220] 40, it is possible to reduce the user's effort in operating the playback of the virtual viewpoint videos 30. Note that in the configuration shown in FIG. 40, all of the five virtual viewpoint videos 30 having information displayed on the recommendation screen 70 are sequentially played on the user device 12, but the technology of the present disclosure is not limited to this, and two or more of the five virtual viewpoint videos 30 may be sequentially played on the user device 12. It should play on Vice 12.

[0221] 40 illustrates an example in which the video playback unit 28 generates playback data 102 in which five virtual viewpoint videos 30 are sequentially played on the user device 12, but the technology of the present disclosure is not limited to this. As an example, as shown in FIG. 41, the video playback unit 28 may generate playback data 102 in which another video 106 is played on the user device 12 each time playback of each of the multiple virtual viewpoint videos 30 played sequentially on the user device 12 ends. Note that the video 106 is an example of a "first image" according to the technology of the present disclosure.

[0222] The video 106 may be the virtual viewpoint video 30 or the captured video 34. Alternatively, the video 106 may be a video separate from the virtual viewpoint video 30 or the captured video 34. In the example shown in FIG. 41 , the video 106 is played on the user device 12 each time the playback of each of the five virtual viewpoint videos 30 finishes, but the technology of the present disclosure is not limited to this, and the video 106 may be played between the playback of at least two virtual viewpoint videos 30. The video 106 may be, for example, a replay video or digest video of the virtual viewpoint video 30 that was played immediately before the playback of the video 106.

[0223] According to the configuration shown in FIG. 41, compared to when only the virtual viewpoint video 30 having the information displayed on the recommendation screen 70 is played back sequentially, the user can be made to watch another video 106 regardless of the user's intention.

[0224] 41 illustrates an example in which the video playback unit 28 generates playback data 102 for playing another video 106 on the user device 12 each time playback of each virtual viewpoint video 30 ends, but the technology of the present disclosure is not limited to this. As an example, as shown in FIG. 42, the video playback unit 28 may generate playback data 102 for playing an advertising video 108, which is a condition for playing the next virtual viewpoint video 30, on the user device 12 each time playback of each virtual viewpoint video 30 ends. The advertising video 108 is an example of a "second image" according to the technology of the present disclosure.

[0225] 42, the advertising effect can be more pronounced than when only the virtual viewpoint videos 30 having the information displayed on the recommendation screen 70 are played sequentially. In the example shown in FIG. 42, the advertising video 108 is played on the user device 12 each time playback of each virtual viewpoint video 30 is completed, but the technology of the present disclosure is not limited to this. A questionnaire image requesting the user using the user device 12 to answer a questionnaire may also be played. In the example shown in FIG. 42, the advertising video 108 is played on the user device 12 each time playback of each virtual viewpoint video 30 is completed, but the technology of the present disclosure is not limited to this. As an example, as shown in FIG. 43, the advertising video 108 may be played on the user device 12 while the virtual viewpoint video 30 is being played.

[0226] 42, the video playback unit 28 generates playback data 102 for playing the same advertising video 108 on the user device 12 each time playback of each virtual viewpoint video 30 ends, but the technology of the present disclosure is not limited to this. The video playback unit 28 may generate playback data 102 for playing the advertising video 108 that satisfies the conditions for playing the virtual viewpoint video 30 on the user device 12, based on advertising information 110 related to the content of the advertising video 108 transmitted to the user device 12. Note that the advertising information 110 is an example of "image information" related to the technology of the present disclosure.

[0227] As an example, as shown in FIG. 44, the played video list 64 corresponding to the account of user A stores advertisement information 110 indicating the content of advertisements displayed on the user device 12 after or during the playback of each virtual viewpoint video 30. The video playback unit 28 stores the played video list 64. The video playback unit 28 reads out the advertising information 110 from the virtual viewpoint video 30 and, if the same advertising video has been played consecutively for a predetermined number of times (e.g., several times) or more, generates playback data 102 in which a different advertising video is played on the user device 12. In the example shown in FIG. 44, the first advertising video is played three times in succession after or during the playback of virtual viewpoint videos 30 having video IDs 62A of "021," "022," and "023." Therefore, the video playback unit 28 switches the first advertising video to the second advertising video and generates playback data 102 in which the second advertising video is played on the user device 12. Note that the first advertising video is an example of a "second image" according to the technology of the present disclosure, and the second advertising video is an example of a "third image" according to the technology of the present disclosure.

[0228] According to the configuration shown in FIG. 44, the effectiveness of the advertisement can be improved more than when only the first advertisement video is repeatedly played on the user device 12.

[0229] 40 to 44, an example has been described in which the video playback unit 28 generates playback data 102 in which the entirety of a plurality of virtual viewpoint videos 30 is played back on the user device 12, but the technology of the present disclosure is not limited to this. Hereinafter, with reference to FIGS. 45 to 49, an example will be described in which the virtual viewpoint video 30 being played back on the user device 12 is switched to another virtual viewpoint video 30 in response to an instruction transmitted from the user device 12.

[0230] As an example, as shown in FIG. 45, when a user touches a thumbnail image 62E displayed on the recommendation screen (see FIG. 10), the user device 12 transmits a playback instruction 72 to the video playback unit 28. The video playback unit 28 generates a composite video 112 by combining a virtual viewpoint video 30-X selected from among the multiple virtual viewpoint videos 30 in accordance with the playback instruction 72 provided by the user device 12 with an arrow image 111 indicating a viewpoint position path 58D relating to the other virtual viewpoint videos 30-Y and 30-Z. The video playback unit 28 generates playback data 102 for the composite video 112 to be played back on the user device 12, and transmits the generated playback data 102 to the user device 12. The user device 12 reproduces the playback data 102 on the touch panel display 12A. The composite video 112 is an example of a "composite image" according to the technology disclosed herein.

[0231] As an example, as shown in FIG. 46 , when a user touches, for example, an arrow image 111 indicating a viewpoint position path 58D of a virtual viewpoint video 30-Y with a finger or a stylus pen, the user device 12 transmits a path change instruction 113 to the video playback unit 28. Upon receiving the path change instruction 113 from the user device 12, the video playback unit 28 stops playback of the composite video 112 and generates playback data 102 for playing back on the user device 12 the virtual viewpoint video 30-Y corresponding to the viewpoint position path 58D included in the path change instruction 113. The video playback unit 28 transmits the generated playback data 102 to the user device 12. The user device 12 receives the playback data 102 and plays back the received playback data 102 on the touch panel display 12A. Note that the path change instruction is an example of a “path change instruction” according to the technology of the present disclosure.

[0232] As a result, the composite video 112 based on the virtual viewpoint video 30-X being played on the user device 12 is switched to the virtual viewpoint video 30-Y. At the timing of the switching, an overhead image 56 is displayed in the lower right corner of the touch panel display 12A, on which the shapes of the viewpoint position paths 58D relating to the virtual viewpoint videos 30-X and 30-Y are superimposed. A viewpoint position path 58D of a switching video that smoothly switches from the viewpoint position path 58D of the virtual viewpoint video 30-X being played on the touch panel display 12A to the viewpoint position path 58D of the virtual viewpoint video 30-Y may be superimposed on the overhead image 56.

[0233] The switching video is a virtual viewpoint video 30 generated by the video generating unit 26 based on the viewpoint position path 58D of the virtual viewpoint video 30-X and the viewpoint position path 58D of the virtual viewpoint video 30-Y. In this case, the video playback unit 28 generates playback data 102 in which the composite video 112, the switching video, and the virtual viewpoint video 30-Y are played back in this order on the user device 12.

[0234] According to the configuration shown in FIG. 45, the video playback unit 28 generates playback data 102 for playing back a composite video 112, thereby enabling switching from a currently played virtual viewpoint video 30-X to another virtual viewpoint video 30-Y. Furthermore, according to the configuration shown in FIG. 46, compared to a case where switching from a currently played virtual viewpoint video 30 to another virtual viewpoint video 30 is not possible, it is possible to allow the user to more easily select the virtual viewpoint video 30 that the user intends. Note that, in the example shown in FIG. 45, an arrow image 111 indicating a viewpoint position path 58D is composited into the composite video 112, but the technology of the present disclosure is not limited to this, and an arrow image indicating a focus point path may be composited instead of the viewpoint position path 58D. In this case, the path change instruction is an instruction to change the focus point path, not the viewpoint position path 58D.

[0235] According to the configurations shown in Figures 45 and 46, the virtual viewpoint video 30 being played on the user device 12 is switched to another virtual viewpoint video 30 that has already been generated, but the technology of the present disclosure is not limited to this, and the virtual viewpoint video 30 being played on the user device 12 may be switched to a newly generated virtual viewpoint video 30 in accordance with an instruction for the virtual viewpoint video 30 being played.

[0236] 47 , when the video generation unit 26 receives, from the user device 12, a viewpoint position change instruction 114 for a viewpoint position 58A of a virtual viewpoint video 30 being played on the user device 12, the video generation unit 26 may generate a new virtual viewpoint video 30 based on the new viewpoint position 58A derived based on the received viewpoint position change instruction 114. The video playback unit 28 may generate playback data 102 for playing the new virtual viewpoint video 30 generated by the video generation unit 26 on the user device 12, and transmit the generated playback data 102 to the user device 12. Note that the viewpoint position change instruction 114 is an example of an “instruction to change viewpoint-related information” according to the technology of the present disclosure.

[0237] 47, when a user uses a finger, a stylus pen, or the like to designate an arbitrary position (hereinafter referred to as a "designated position") 115 on the touch panel display 12A while the virtual viewpoint video 30 is being played, the user device 12 transmits a viewpoint position change instruction 114 to the video generation unit 26. The viewpoint position change instruction 114 includes a match elapsed time 62C corresponding to the timing at which the viewpoint position change instruction 114 was received in the virtual viewpoint video 30 being played, and the coordinates of the designated position 115 relative to the image of the frame included in the virtual viewpoint video 30 being played and corresponding to the timing at which the viewpoint position change instruction 114 was received.

[0238] The video generation unit 26 receives a viewpoint position change instruction 114 from the user device 12. The video generation unit 26 derives a new viewpoint position 58A within the soccer stadium based on the coordinates of an indicated position 115 included in the viewpoint position change instruction 114. The video generation unit 26 generates a new virtual viewpoint video 30 from the game elapsed time 62C onwards based on the derived new viewpoint position 58A, using the captured video 34 corresponding to the game elapsed time 62C included in the viewpoint position change instruction 114. The video generation unit 26 stores the generated virtual viewpoint video 30 in the database 22.

[0239] The video playback unit 28 reads out the new virtual viewpoint video 30 from the database 22 and generates playback data 102 for playing the new virtual viewpoint video 30 on the user device 12. The video playback unit 28 transmits the generated playback data 102 to the user device 12. The user device 12 receives the playback data 102 and plays the received playback data 102 on the touch panel display 12A. Therefore, on the user device 12, the virtual viewpoint video 30 being played on the touch panel display 12A is switched to the newly generated virtual viewpoint video 30 based on the viewpoint position change instruction 114.

[0240] 47, the user can change viewpoint position 58A of the virtual viewpoint video 30 that is being viewed. Note that in the example shown in Fig. 47, viewpoint position 58A of virtual viewpoint video 30 is changed by the user pointing to an arbitrary position on touch panel display 12A on which virtual viewpoint video 30 is being played back, but the technology of the present disclosure is not limited to this, and line of sight direction 58B, angle of view 58C, point of gaze 58E, or the like may also be changed.

[0241] According to the configuration shown in FIG. 47, a new viewpoint position 58A is derived based on the instruction position 115 given by the user on the touch panel display 12A, but the technology of the present disclosure is not limited to this, and a new viewpoint position path 58D may also be determined based on the viewpoint position path 58D of the virtual viewpoint video 30 being played.

[0242] 48 , when the video generation unit 26 receives from the user device 12 a replay video generation instruction 116 for the virtual viewpoint video 30 being played on the user device 12, the video generation unit 26 may generate a replay video 120 based on viewpoint-related information 58 related to the virtual viewpoint video 30 being played on the user device 12. The video playback unit 28 may generate playback data 102 for playing the replay video 120 generated by the video generation unit 26 on the user device 12, and transmit the generated playback data 102 to the user device 12. The replay video generation instruction 116 is an example of a "replay image generation instruction" according to the technology of the present disclosure.

[0243] 48, a replay video generation button 118 is provided at the bottom right of the video playback screen 33 on which the virtual viewpoint video 30 is being played back on the touch panel display 12A. When the user touches the replay video generation button 118 provided on the video playback screen 33 while the virtual viewpoint video 30 is being played back with a finger, a stylus pen, or the like, the user device 12 transmits a replay video generation instruction 116 to the video generation unit 26. The replay video generation instruction 116 includes an elapsed game time 62C corresponding to the timing at which the replay video generation instruction 116 was received in the virtual viewpoint video 30 being played back, and a viewpoint position path 58D and a point of interest 58E related to the virtual viewpoint video 30 being played back.

[0244] The video generation unit 26 receives a replay video generation instruction 116 from the user device 12. Based on the viewpoint position path 58D included in the replay video generation instruction 116, the video generation unit 26 determines a new viewpoint position path 58D that is different from the viewpoint position path 58D related to the virtual viewpoint video 30 being played. Based on the determined new viewpoint position path 58D and the same point of interest 58E as the point of interest 58E included in the replay video generation instruction 116, the video generation unit 26 generates a replay video 120, which is a new virtual viewpoint video, using captured videos 34 captured during the period from several seconds to several tens of seconds before the game elapsed time 62C included in the viewpoint position change instruction 114 to the game elapsed time 62C. The video generation unit 26 stores the generated replay video 120 in the database 22.

[0245] The video playback unit 28 reads the replay video 120 from the database 22 and generates playback data 102 for playing the replay video 120 on the user device 12. The video playback unit 28 transmits the generated playback data 102 to the user device 12. The user device 12 receives the playback data 102 and plays the received playback data 102 on the touch panel display 12A. Therefore, on the user device 12, the virtual viewpoint video 30 being played on the touch panel display 12A is switched to the replay video 120.

[0246] According to the configuration shown in FIG. 48, the user can view a replay video 120 of the virtual viewpoint video 30 that he or she is currently viewing.

[0247] As an example, as shown in FIG. 49, the video generation unit 26 may receive from the user device 12 a viewing refusal request for a subject whose viewing is to be refused from the virtual viewpoint video 30 being played on the user device 12. When the objection subject information 122 is received, a new virtual viewpoint video 30 may be generated based on viewpoint-related information 58 based on the objection subject information 122. The video playback unit 28 may generate playback data 102 for playing the new virtual viewpoint video 30 generated by the video generation unit 26 on the user device 12, and transmit the generated playback data 102 to the user device 12. Note that the objection subject information 122 is an example of "objection subject information" according to the technology of the present disclosure.

[0248] 49, when the user touches, with a finger, a stylus pen, or the like, an object image 124 indicating a player that the user does not want to watch in the virtual perspective video 30 being played on the touch panel display 12A, the user device 12 transmits the refused-to-watch subject information 122 to the video generation unit 26. The refused-to-watch subject information 122 includes the object image 124, a game elapsed time 62C corresponding to the timing at which the refused-to-watch subject information 122 was received in the virtual perspective video 30 being played, and a point of gaze 58E for the virtual perspective video 30 being played.

[0249] The video generation unit 26 receives the refused viewing subject information 122 from the user device 12. Based on the object image 124 included in the refused viewing subject information 122, the video generation unit 26 identifies the players appearing in the object image 124 using well-known image recognition technology or the like. The video generation unit 26 determines viewpoint-related information 58 from which a virtual viewpoint video 30 is generated in which the identified players are not appearing. In this case, the determined viewpoint-related information 58 has the same point of gaze 58E as the point of gaze 58E included in the refused viewing subject information 122, and has a viewpoint position path 58D different from that of the virtual viewpoint video 30 being played back.

[0250] Based on the determined viewpoint-related information 58, the video generation unit 26 generates the virtual viewpoint video 30 using the captured video 34 captured after the match elapsed time 62C included in the refused viewing subject information 122. The video generation unit 26 stores the generated virtual viewpoint video 30 in the database 22.

[0251] The video playback unit 28 reads out the virtual viewpoint video 30 from the database 22, and generates playback data 102 that allows the virtual viewpoint video 30 to be played back on the user device 12. The video playback unit 28 transmits the generated playback data 102 to the user device 12. The user device 12 receives the playback data 102, and plays back the received playback data 102 on the touch panel display 12A.

[0252] 49, the user can view the virtual viewpoint video 30 that does not include any object that the user does not want to view. Note that, in the example shown in Fig. 49, the object image 124 is an image of a player that the user does not want to view, but the technology of the present disclosure is not limited to this, and the object image 124 may be an image of any object in a soccer stadium, including an advertising billboard, a goal, a line, etc.

[0253] 50, the recommended data 32 may be data that excludes virtual viewpoint videos 30 that are subject to viewing restrictions, based on attribute data 64B registered in association with a user's account 64A. In the example shown in FIG. 50, attribute data 64B of "under 12 years old" is registered in association with account 64A of "A." Furthermore, video information 62 of generated video list 60 includes target age 62G that limits users who can view the virtual viewpoint video 30 according to the user's age.

[0254] When receiving a video playback request 52 from the user device 12 used by user A, the video playback unit 28 generates information about the virtual viewpoint videos 30 that are restricted from being viewed by users under the age of 12, among the virtual viewpoint videos 30 recorded in the generated video list 60, based on the attribute data 64B of "under 12 years old" registered in association with the account 64A of user A. 50, in the generated video list 60, the virtual viewpoint video 30 having the video ID 62A of "003" has a target age 62G of "15 years or older." Therefore, the video playback unit 28 performs processing to delete the virtual viewpoint video 30 having the target age 62G of "15 years or older" from the generated video list 60, and stores the processed generated video list 60 in the database 22 as a recommended video list 66.

[0255] The configuration shown in FIG. 50 makes it easier for the user to select the virtual viewpoint video 30 that the user desires, compared to when the virtual viewpoint video 30 is not restricted by the user attribute data 64B.

[0256] As an example, as shown in FIG. 51 , a video playback request 52 transmitted from the user device 12 may include a time setting instruction 126 for an elapsed game time 62C. When the video playback unit 28 receives the video playback request 52 including the time setting instruction 126 from the user device 12, the video playback unit 28 may generate recommendation data 32 including information on virtual viewpoint videos 30 (hereinafter also referred to as “time-limited virtual viewpoint videos”), among the multiple virtual viewpoint videos 30, whose elapsed game time 62C falls within the range indicated by the time setting instruction 126. The recommendation data 32 may also include the number of time-limited virtual viewpoint videos and the number of virtual viewpoint videos that have been played on the user device 12 among the time-limited virtual viewpoint videos. The elapsed game time 62C is an example of “third time information” according to the technology of the present disclosure. The time-limited virtual viewpoint video is an example of a “time-limited virtual viewpoint image” according to the technology of the present disclosure.

[0257] When the video playback unit 28 receives a video playback request 52 including a time setting instruction 126 from the user device 12, it reads out a generated video list 60 from the database 22. Of the virtual viewpoint videos 30 recorded in the generated video list 60, the video playback unit 28 keeps only those time-limited virtual viewpoint videos whose elapsed game time 62C is within the range indicated by the time setting instruction 126, and deletes the other virtual viewpoint videos 30 from the generated video list 60. The video playback unit 28 acquires the number of time-limited virtual viewpoint videos remaining in the generated video list 60 as the total number of videos 130 (see FIG. 52 ).

[0258] The video playback unit 28 reads from the database 22 the played video list 64 for the user device 12 that is the output source of the video playback request 52. The video playback unit 28 deletes the played virtual viewpoint videos 30 included in the played video list 64 from the generated video list 60, and stores the generated video list 60 after the deletion as a recommended video list 66 in the database 22. The video playback unit 28 obtains the number of played videos 132 (see FIG. 52 ) as the value obtained by subtracting the number of virtual viewpoint videos 30 included in the recommended video list 66 from the total number of videos 130. The video playback unit 28 generates recommendation data 32 based on the recommended video list 66, the total number of videos 130, and the number of played videos 132, and transmits the generated recommendation data 32 to the user device 12.

[0259] 52, the user device 12 receives the recommendation data 32 and displays the received recommendation data 32 on the touch panel display 12A as a recommendation screen 70. The recommendation screen 70 includes time setting information 134 indicating the elapsed game time 62C set in the time setting instruction 126, a total video count 130 indicating the number of time-limited virtual viewpoint videos, and a number of played videos 132 among the time-limited virtual viewpoint videos.

[0260] 51, it is possible to allow the user to more easily select the virtual viewpoint video 30 that the user desires, compared to when the time setting instruction 126 is not accepted. Furthermore, the configuration shown in Fig. 52 allows the user to grasp the total number of videos 130 and the number of played videos 132.

[0261] 53, even if a user stops viewing a virtual viewpoint video 30 being played on the user device 12 midway, the video playback unit 28 may record at least a portion of the virtual viewpoint video 30 played on the user device 12 in the played video list 64. As a result, for example, if the user stops viewing the virtual viewpoint video 30 that they started playing on the user device 12 because it was different from what they intended, information about the virtual viewpoint video 30 that was stopped midway is recorded in the played video list 64. The video playback unit 28 references the played video list 64 and generates recommended data 32 based on information about the virtual viewpoint video 30 whose playback was stopped midway. In this case, for example, the recommended data 32 is generated using a method similar to that of the above embodiment based on information about the viewpoint position path 58D up to the point where the playback was stopped midway.

[0262] The configuration shown in FIG. 53 makes it easier for the user to select the virtual viewpoint video 30 that the user desires, compared to when the played video list 64 includes only the virtual viewpoint video 30 that has been played in its entirety.

[0263] 53, an example has been described in which the video playback unit 28 determines that the virtual viewpoint video 30, at least a portion of which has been played on the user device 12, has been played back, but the technology of the present disclosure is not limited to this. Hereinafter, with reference to FIGS. 54 to 56, an example will be described in which the virtual viewpoint video 30, the entirety of which has been played on the user device 12, is determined to have been played back.

[0264] 54 as an example, the generated moving image list 60 includes information on five virtual viewpoint videos 30. The five virtual viewpoint videos 30 are videos used for educational purposes, such as learning about how a fire spreads at a fire scene. The moving image generation unit 26 uses a captured video 34 obtained by capturing an image of a fire reproduction experiment to generate the virtual viewpoint videos 30 based on a viewpoint position path 58D created by the administrator of the data processing device 10, and records video information 62 and viewpoint-related information 58 related to the generated virtual viewpoint videos 30 in the generated moving image list 60.

[0265] In response to a request from a user device 12 accessing the data processing device 10 using the account of each of viewers A to C, the video playback unit 28 transmits the virtual viewpoint video 30 to the user device 12 that is the source of the request. The playback history of the virtual viewpoint video 30 by each user device 12, i.e., the viewing history of the virtual viewpoint video 30 by each viewer A, B, or C, is recorded in a generated video list 60. Note that viewer A, viewer B, and viewer C are examples of "viewers" according to the technology of the present disclosure. Hereinafter, when there is no need to distinguish between viewer A, viewer B, and viewer C, they will be collectively referred to as "viewers."

[0266] The generated moving image list 60 records viewing data 140 including viewer data 140A and viewing count 140B. The viewer data 140A indicates whether each virtual viewpoint video 30 has been viewed by each viewer. The viewing count 140B indicates the number of times each virtual viewpoint video 30 has been viewed by a viewer.

[0267] The video playback unit 28 can acquire, as viewer data 140A, a playback history of the virtual viewpoint video 30 for an account associated with a viewer of the virtual viewpoint video 30. The video playback unit 28 can also tally up, as a viewing count 140B, how many times each virtual viewpoint video 30 has been played on the user device 12. The video playback unit 28 can output the acquired viewer data 140A and viewing count 140B to the associated account of an administrator of the data processing device 10. The video playback unit 28 outputs the viewing data 140 in the form of a table shown in FIG. 54, for example.

[0268] According to the configuration shown in FIG. 54, the administrator of the data processing device 10 can associate the user device 12 to which the virtual viewpoint video 30 has been transmitted with the viewpoint-related information 58 of the virtual viewpoint video 30. This can be understood as follows.

[0269] In the example shown in FIG. 54, the video playback unit 28 outputs the viewing data 140 in a table format, but the technology of the present disclosure is not limited to this. The video playback unit 28 may output the viewing data 140 in the form of a path image shown in FIG. 55, for example. In the example shown in FIG. 55, the video playback unit 28 generates management data including a 3D image of a fire reproduction experiment, a video ID 62A for each virtual viewpoint video 30, a viewpoint position path 58D, and viewing data 140. The video playback unit 28 transmits the generated management data to a tablet terminal 142 that is accessing the data processing device 10 using an administrator's account. The tablet terminal 142 receives the management data and displays the received management data on a touch panel display 142A as a management screen 144. The management data is an example of "second data" according to the technology of the present disclosure.

[0270] On the management screen 144, the shape of the viewpoint position path 58D of each virtual viewpoint video 30 is displayed superimposed on a three-dimensional image of the fire reproduction experiment. On the management screen 144, the video ID 62A of the virtual viewpoint video 30 corresponding to each viewpoint position path 58D, the number of viewings 140B, and the names of viewers who have not yet viewed each virtual viewpoint video 30 are displayed in association with each viewpoint position path 58D. The shape of the viewpoint position path 58D for a virtual viewpoint video 30 that has been viewed by all of viewers A to C, i.e., the shape of the viewpoint position path 58D for a virtual viewpoint video 30 that has a viewing count 140B of "three times," is displayed on the management screen 144 in a manner that allows it to be distinguished from the shapes of the viewpoint position paths 58D for the other virtual viewpoint videos 30.

[0271] "Virtual viewpoint video 30 that has been viewed by all of viewers A to C" means that all of viewers A to C have viewed the entire virtual viewpoint video 30, that is, have viewed the viewpoint position path 58D related to the virtual viewpoint video 30 from start to finish. If a viewer stops viewing the virtual viewpoint video 30 midway through each viewpoint position path 58D, a tag 146 indicating the name of the viewer who stopped viewing midway and the stop position is displayed in association with the viewpoint position path 58D.

[0272] 55, the administrator of the data processing device 10 can grasp how many times each virtual viewpoint video 30 has been played by associating it with the viewpoint position path 58D of each virtual viewpoint video 30. Note that in the example shown in Fig. 55, the shape of the viewpoint position path 58D is superimposed on the three-dimensional image of the fire reproduction experiment, but the technology of the present disclosure is not limited to this, and the shape of the gaze point path may be superimposed on the three-dimensional image of the fire reproduction experiment.

[0273] The method of displaying the stop position when a viewer stops viewing the virtual viewpoint video 30 midway through each viewpoint position path 58D is not limited to the example shown in Fig. 55. As an example, as shown in Fig. 56, the video playback unit 28 may generate management data for each viewer, including a path image in which the shape of the viewpoint position path 58D is superimposed on a 3D image of the fire reproduction experiment, and transmit the generated management data to the user device 12 used by each viewer. The user device 12 receives the management data and displays the received management data on the touch panel display 12A as a management screen 148.

[0274] On the management screen 148, the shape of the viewpoint position path 58D is superimposed on the three-dimensional image of the fire reproduction experiment. The video ID 62A of the virtual viewpoint video 30 is displayed in association with the viewpoint position path 58D of the virtual viewpoint video 30. In the path image, the portion of the viewpoint position path 58D that has already been played back in the virtual viewpoint video 30 is displayed in a manner that allows it to be distinguished from the portion that has not yet been played back.

[0275] According to the configuration shown in FIG. 56, compared to when the played and unplayed portions of the virtual viewpoint video 30 in the viewpoint position path 58D are displayed in the same manner in the path image, the viewer can select the virtual viewpoint video 30 by referring to the viewpoint position path 58D in the path image. 56, the shape of the viewpoint position path 58D is displayed superimposed on the three-dimensional image of the fire reproduction experiment, but the technology of the present disclosure is not limited to this, and the shape of the gaze point path may be displayed superimposed on the three-dimensional image of the fire reproduction experiment.

[0276] In addition, in the above embodiment, an example was given of generating a virtual viewpoint video 30 using a captured video 34 of a soccer stadium or a fire reproduction experiment, but this is merely one example, and the virtual viewpoint video 30 may be generated using a captured video 34 of any location where multiple physical cameras can be installed, such as a baseball stadium, rugby field, curling court, athletics stadium, swimming pool, concert hall, outdoor music venue, and theater venue.

[0277] Furthermore, in the above embodiment, an example has been given in which the virtual viewpoint video 30 is generated and played back, but this is merely an example, and a virtual viewpoint image, which is a still image, may also be generated and played back.

[0278] Furthermore, in the above embodiment, the recommended data 32 is generated based on the untransmitted virtual viewpoint video 30, but the recommended data 32 does not necessarily have to be generated based on the untransmitted virtual viewpoint video 30. For example, even if a virtual viewpoint video 30 has been transmitted to the user device 12, it is conceivable to recommend the virtual viewpoint video 30 to the user as long as the virtual viewpoint video 30 has not been played by the user. In this case, the data processing device 10 acquires data regarding which virtual viewpoint video 30 has been played from the user device 12 or the user's account, and generates the recommended data 32 based on the unplayed virtual viewpoint video 30. Note that, since the user may wish to rewatch a virtual viewpoint video 30 that has already been played, the data processing device 10 may generate the recommended data 32 by including the played virtual viewpoint video 30.

[0279] Furthermore, in the above embodiment, the computer 14 is exemplified, but the technology of the present disclosure is not limited to this. For example, a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 14. Furthermore, a combination of a hardware configuration and a software configuration may be used instead of the computer 14.

[0280] In the above embodiment, the moving image generation process and the moving image playback process are executed by the CPU 14A of the data processing device 10. However, the technology of the present disclosure is not limited to this. Instead of the CPU 14A, a GPU or multiple CPUs may be employed. Furthermore, various processes may be executed by a single processor or multiple processors that are physically separated from each other.

[0281] Furthermore, in the above embodiment, the operating program 24 is stored in the NVM 14B, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 57, the operating program 24 may be stored in any portable storage medium 200. The storage medium 200 is a non-transitory storage medium. Examples of the storage medium 200 include an SSD and a USB memory. The operating program 24 stored in the storage medium 200 is installed in the computer 14, and the CPU 14A executes a video generation process and a video playback process in accordance with the operating program 24.

[0282] Alternatively, the operating program 24 may be stored in a program memory of another computer or server device connected to the computer 14 via a communication network (not shown), and the operating program 24 may be downloaded to the data processing device 10 in response to a request from the data processing device 10. In this case, the moving image generation process and moving image playback process based on the downloaded operating program 24 are executed by the CPU 14A of the computer 14.

[0283] The hardware resources for executing video generation and playback processes are as follows: A processor can be used, for example, as described above, a CPU, which is a general-purpose processor that functions as a hardware resource that executes data processing according to software, i.e., a program.

[0284] Other processors include dedicated electrical circuits, which are processors having circuit configurations specifically designed to perform specific processes, such as FPGAs, PLDs, or ASICs. All processors have built-in or connected memory, and all processors perform data processing by using the memory.

[0285] The hardware resource that executes data processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes data processing may be a single processor.

[0286] Examples of systems configured with a single processor include, first, a system in which one processor is configured with a combination of one or more CPUs and software, as typified by client and server computers, and this processor functions as a hardware resource that executes data processing. Second, a system in which a processor is used to implement the functions of an entire system, including multiple hardware resources that execute data processing, on a single IC chip, as typified by SoCs. In this way, data processing is implemented using one or more of the above-mentioned various processors as hardware resources.

[0287] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0288] Furthermore, the data processing described above is merely an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the invention.

[0289] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0290] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0291] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0292] The following additional notes are provided regarding the above-described embodiments.

[0293] [Additional note 1] 1. A data processing method for transmitting a plurality of virtual viewpoint images generated based on a captured image to a device, comprising: Obtaining first data related to a first virtual viewpoint image, which is a virtual viewpoint image at least a portion of which has been transmitted to the device, from among the plurality of virtual viewpoint images; and transmitting, to the device, second data relating to a second virtual viewpoint image, which is a virtual viewpoint image that has not yet been transmitted to the device among the plurality of virtual viewpoint images, based on the acquired first data; Data processing methods, including:

[0294] [Additional note 2] A program for causing a computer to execute data processing for transmitting a plurality of virtual viewpoint images generated based on a captured image to a device, The above data processing is Obtaining first data related to a first virtual viewpoint image, which is a virtual viewpoint image at least a portion of which has been transmitted to the device, from among the plurality of virtual viewpoint images; and and transmitting, to the device, second data relating to a second virtual viewpoint image, which is a virtual viewpoint image that has not yet been transmitted to the device, among the plurality of virtual viewpoint images, based on the acquired first data. program.

Claims

1. a processor; a memory embedded in or connected to the processor; A data processing device that transmits a virtual viewpoint image generated based on a captured image to a device, The processor: acquiring first data relating to a playback history and / or registration data of the virtual viewpoint image; Based on the acquired first data, control is performed to transmit second data related to the virtual viewpoint image to the device. Data processing device.

2. The data processing device according to claim 1 , wherein the processor controls transmission of the second data relating to a second virtual viewpoint image, which is different from a first virtual viewpoint image, which is the virtual viewpoint image associated with the first data.

3. The data processing device according to claim 2 , wherein the first data includes viewpoint-related information relating to a viewpoint of the virtual viewpoint image.

4. The data processing device according to claim 3, wherein the viewpoint-related information includes at least one of a viewpoint position, a line of sight direction, an angle of view, a gaze point, a viewpoint position path indicating the displacement of the viewpoint position over time, and a gaze point path indicating the displacement of the gaze point over time.

5. The data processing device according to claim 4 , wherein the first data includes first time information relating to the capture time of the captured image used to generate the virtual viewpoint image.

6. The data processing device according to claim 5 , wherein the second data includes at least one of a reduced image of the virtual viewpoint image and the viewpoint-related information.

7. 7. The data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in order of increasing similarity between the viewpoint-related information contained in the first data and the viewpoint-related information, and decreasing similarity between the first time information contained in the first data and the first time information.

8. 7. The data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in order of decreasing similarity between the viewpoint-related information contained in the first data and the viewpoint-related information and increasing similarity between the first time information contained in the first data and the viewpoint-related information.

9. 9. The data processing device according to claim 7, wherein the processor acquires the similarity regarding the viewpoint-related information based on a difference in the position of the viewpoint position path or the gaze point path between the viewpoint-related information included in the first data and the viewpoint-related information included in the second data.

10. 9. The data processing device according to claim 7, wherein the processor acquires the similarity regarding the viewpoint-related information based on a difference between tags assigned to the viewpoint position path or the gaze point path.

11. The second data is a list of the abridged images in an order determined based on the degree of similarity between the attributes of a viewer who viewed the first virtual viewpoint image using the device and the attributes of the creator of the second virtual viewpoint image. The data processing device according to claim 6 , wherein at least one of the small image and the viewpoint-related information is data to be displayed on the device.

12. 12. A data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined based on a difference between the playback history of the virtual viewpoint image by a viewer who viewed the first virtual viewpoint image using the device and the playback history of the virtual viewpoint image by another viewer.

13. 13. A data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined based on playback trends extracted from the playback history of the virtual viewpoint image by a viewer who viewed the first virtual viewpoint image using the device.

14. 14. The data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined based on comments added to each of the plurality of second virtual viewpoint images.

15. the comment is a comment added to a viewpoint position path included in the second data, which indicates a displacement of a viewpoint position over time, or a gaze point path included in the second data, which indicates a displacement of a gaze point over time, The second data is data for displaying an image, to which the comment is added in the form of the viewpoint position path or the form of the gaze point path, on the device.

15. A data processing apparatus according to claim 14.

16. 16. A data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined based on a first evaluation assigned to each of the plurality of second virtual viewpoint images.

17. 17. A data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined according to the viewer's preferences estimated based on a second evaluation of the first virtual viewpoint image.

18. 18. The data processing device according to claim 6, wherein the second data is data that, when a new virtual viewpoint image is added, displays on the device at least one of the reduced image and the viewpoint-related information related to the added virtual viewpoint image based on the first data.

19. the virtual viewpoint image is a video; 19. A data processing device according to any one of claims 6 to 18, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined based on the length of a viewpoint position path indicating the displacement of the viewpoint position included in the first data over time relative to the time from the start to the end of playback of the first virtual viewpoint image, or the length of a gaze point path indicating the displacement of the gaze point included in the first data over time.

20. the virtual viewpoint image is a video; 20. A data processing device according to any one of claims 6 to 19, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined based on a ratio between second time information relating to the capture time of the captured image used to generate the second virtual viewpoint image and the time from the start to the end of playback of the second virtual viewpoint image.

21. 21. A data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined based on the data capacity of each of the plurality of second virtual viewpoint images.

22. 22. A data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined based on parameters related to the image quality of each of the plurality of second virtual viewpoint images.

23. 23. The data processing device according to claim 6, wherein the second data is data in which at least one of the reduced image and the viewpoint-related information is displayed on the device in an order determined based on audio data assigned to the second virtual viewpoint image.

24. 24. The data processing device according to claim 2, wherein the second data is data for a plurality of the second virtual viewpoint images to be played back sequentially on the device on condition that the processor receives a playback instruction instructing playback of the second virtual viewpoint images.

25. 25. The data processing device according to claim 24, wherein the second data is data that causes a first image different from the first virtual viewpoint image and the second virtual viewpoint image to be reproduced on the device each time reproduction of each of the plurality of second virtual viewpoint images is completed.

26. 26. The data processing device according to claim 2, wherein the second data is data for reproducing a second image, which is a condition for reproducing the second virtual viewpoint image, on the device.

27. 27. The data processing device according to claim 26, wherein the second data is data for reproducing a third image on the device, the third image being a condition for reproducing the second virtual viewpoint image, based on image information relating to the content of the second image transmitted to the device.

28. A data processing device described in any one of claims 2 to 27, wherein the second data is data that is played on the device as a composite image obtained by combining a second virtual viewpoint image selected from a plurality of second virtual viewpoint images in accordance with given instructions with a viewpoint position path indicating the displacement of the viewpoint position of other second virtual viewpoint images over time, or a gaze point path indicating the displacement of the gaze point over time.

29. 29. The data processing device of claim 28, wherein when a path change instruction for the viewpoint position path or the gaze point path is received, the second data is data that stops playback of the composite image and plays the second virtual viewpoint image corresponding to the viewpoint position path or the gaze point path included in the path change instruction on the device.

30. Regarding viewpoint-related information related to the viewpoint of the second virtual viewpoint image being reproduced on the device 30. The data processing device according to claim 2, wherein, when a change instruction is received, the second data is data in which a virtual viewpoint image generated based on new viewpoint-related information derived based on the change instruction is reproduced on the device.

31. A data processing device described in any one of claims 2 to 30, wherein when a replay image generation instruction is accepted for the second virtual viewpoint image being played on the device, the second data is data in which a virtual viewpoint image generated based on new viewpoint-related information generated based on viewpoint-related information related to the viewpoint of the second virtual viewpoint image is played on the device.

32. A data processing device as described in any one of claims 2 to 31, wherein when information on a subject that is refused to be viewed is accepted, the second data is data that reproduces on the device a virtual viewpoint image generated based on viewpoint-related information generated based on the information on the subject that is refused to be viewed.

33. the second virtual viewpoint image is a moving image including a plurality of frames of images, 33. A data processing device according to claim 2, wherein when a digest image display instruction is received for a reduced image of the second virtual viewpoint image included in the second data, the second data is data that is played on the device as a digest image generated by extracting images of the plurality of frames from the second virtual viewpoint image, the frames having different viewpoint positions at a predetermined interval.

34. the processor controls to transmit to the device a path image in which a shape of a viewpoint position path indicating a displacement of a viewpoint position of the first virtual viewpoint image over time or a shape of a fixation point path indicating a displacement of a fixation point of the first virtual viewpoint image over time is associated with the second data, and the path image is displayed; The path image is an image in which the shape of the viewpoint position path or the shape of the gaze point path is shown in a manner that can be distinguished from the second data.

34. Data processing apparatus according to any one of claims 2 to 33.

35. the second data includes a reduced image of the second virtual viewpoint image, and a shape of a viewpoint position path indicating a displacement of a viewpoint position of the second virtual viewpoint image over time, or a shape of a gaze point path indicating a displacement of a gaze point of the second virtual viewpoint image over time, The data processing device according to claim 2 , wherein the second data includes first position information that can identify a position of the reduced image on the viewpoint position path or the gaze point path.

36. 36. A data processing device as claimed in any one of claims 2 to 35, wherein the second data includes second position information in which the second data relating to the second virtual viewpoint image is arranged horizontally in chronological order and arranged vertically according to the similarity of viewpoint-related information of the second virtual viewpoint image to viewpoint-related information of the first virtual viewpoint image.

37. 37. The data processing device according to claim 2, wherein the registration data includes attribute data indicating attributes of a viewer who has viewed the first virtual viewpoint image, and preference data indicating preferences of the viewer.

38. 38. The data processing device according to claim 37, wherein the second data is data obtained by excluding data relating to virtual viewpoint images that are subject to viewing restrictions among the virtual viewpoint images based on the attribute data.

39. 39. A data processing device as claimed in any one of claims 2 to 38, wherein when a time setting instruction for third time information relating to the capture time of the captured image used to generate the virtual viewpoint image is accepted, the second data is data relating to a second virtual viewpoint image among the plurality of second virtual viewpoint images, the third time information of which is within the range indicated by the time setting instruction.

40. 40. The data processing device of claim 39, wherein when the time setting instruction is accepted, the second data includes the number of time-limited virtual viewpoint images among the virtual viewpoint images whose third time information is within the range indicated by the time setting instruction, and the number of images included in the second virtual viewpoint image among the time-limited virtual viewpoint images.

41. The processor: A playback history of the virtual viewpoint image can be obtained for an account associated with a viewer of the virtual viewpoint image; The playback history can be output in association with viewpoint-related information related to the viewpoint of the virtual viewpoint image.

41. Data processing apparatus according to any one of claims 1 to 40.

42. The processor: It is possible to count how many times each of the virtual viewpoint images has been reproduced, The tabulation result can be output in association with viewpoint-related information relating to the viewpoint of the virtual viewpoint image.

42. Data processing apparatus according to any one of claims 1 to 41.

43. the processor controls to transmit to the device a path image in which a shape of a viewpoint position path indicating a displacement of a viewpoint position of the virtual viewpoint image over time or a shape of a gaze point path indicating a displacement of a gaze point of the virtual viewpoint image over time is displayed in association with the second data; The path image is an image in which the virtual viewpoint image of the viewpoint position path or the gaze point path is displayed in a manner that allows a distinction between a reproduced portion and an unreproduced portion.

43. Data processing apparatus according to any one of claims 1 to 42.

44. The data processing device according to claim 1 , wherein the first virtual viewpoint image includes a virtual viewpoint image, at least a part of which has already been reproduced on the device.

45. a processor; a memory embedded in or connected to the processor; A data processing device that transmits a plurality of virtual viewpoint images generated based on a captured image to a device, The processor: acquire first data relating to a first virtual viewpoint image, which is a virtual viewpoint image at least a part of which has been transmitted to the device, from among the plurality of virtual viewpoint images; Based on the acquired first data, control is performed to transmit to the device second data relating to a second virtual viewpoint image, which is a virtual viewpoint image not yet transmitted to the device among the plurality of virtual viewpoint images. Data processing device.

46. 1. A data processing method for transmitting a virtual viewpoint image generated based on a captured image to a device, comprising: acquiring first data relating to a playback history and / or registration data of the virtual viewpoint image; and and controlling transmission of second data relating to the virtual viewpoint image to the device based on the acquired first data. Data processing methods, including:

47. A program for causing a computer to execute data processing for transmitting a virtual viewpoint image generated based on a captured image to a device, The data processing comprises: Obtaining first data relating to a playback history and / or registration data of the virtual viewpoint image; and and a process including controlling transmission of second data relating to the virtual viewpoint image to the device based on the acquired first data. program.

Images