Information processing system, information processing method, and information processing program

Abstract

According to the present invention, a video generation system includes an editing terminal and a display terminal. The editing terminal acquires editing data created in a predetermined image format, executes editing rendering processing on the editing data, and generates editing video data for editing or preview. When the editing video is edited, the editing terminal transmits editing information related to editing of the editing data to the display terminal. The display terminal acquires display data the same as the editing data, receives the editing information from the editing terminal, executes display rendering processing different from the editing rendering processing on the display data reflecting the editing on the basis of the received editing information, and generates display video data for provision.

Description

Information processing system, information processing method, and information processing program

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program.

[0002] To generate arbitrary viewpoint images for three-dimensional object scenes, synthesis techniques using image formats such as MPI (Multi-Plane-Image), MCI (Multi-Cylinder-Image), MSI (Multi-Sphere-Image), or 3DGS (3D Gaussian Splatting) are used.

[0003] “NeX: Real-time View Synthesis with Neural Basis Expansion” Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai “3D Gaussian Splatting for Real-Time Radiance Field Rendering” Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuhler, George Drettakis

[0004] However, it is difficult to effectively display arbitrary viewpoint video. For example, it is difficult to generate and display arbitrary viewpoint video in a format that uses high resolution and real-time 3D rendering. On the other hand, if 3D rendering is not used, it is difficult for users to edit the video. Therefore, it is difficult for users to check the final display image of the arbitrary viewpoint video while editing the 3D model data in the editing video.

[0005] The present invention has been made in view of the above, and has as its object to effectively display images for different purposes.

[0006] An information processing system of one embodiment of the present disclosure is an information processing system including a first information processing device and a second information processing device, wherein the first information processing device acquires first data created in a predetermined image format, performs first rendering on the first data, generates first video data for editing or previewing, and, when the first video data is edited, transmits editing information regarding the editing of the first data to the second information processing device, and the second information processing device acquires second data identical to the first data, receives the editing information from the first information processing device, and performs second rendering different from the first rendering on the second data reflecting the editing based on the received editing information, thereby generating second video data for provision.

[0007] 1 is a diagram illustrating an example of a configuration and an example of processing of a video display system according to an embodiment; FIG. 2 is a diagram illustrating a basic principle of an arbitrary viewpoint video generation process according to an embodiment; FIG. 3 is a block diagram illustrating an example of a configuration of each device of a video display system according to an embodiment; FIG. 4 is a diagram illustrating an example of an editing video data storage unit of an editing terminal according to an embodiment; FIG. 5 is a diagram illustrating an example of a first editing information storage unit of an editing terminal according to an embodiment; FIG. 6 is a diagram illustrating an example of a second editing information storage unit of a display terminal according to an embodiment; FIG. 7 is a diagram illustrating an example of a display video data storage unit of a display terminal according to an embodiment; FIG. 8 is a diagram illustrating an example of a 3D model data database according to an embodiment; FIG. 9 is a diagram illustrating a specific example 1-1 of each process of a video display system according to an embodiment; FIG. 10 is a diagram illustrating a specific example 1-2 of each process of a video display system according to an embodiment; FIG. 11 is a diagram illustrating a specific example 2-1 of each process of a video display system according to an embodiment; FIG. 12 is a diagram illustrating a specific example 2-2 of each process of a video display system according to an embodiment; FIG. 13 is a diagram illustrating a specific example 3 of each process of a video display system according to an embodiment; FIG. 14 is a diagram illustrating a specific example 4 of each process of a video display system according to an embodiment; FIG. 15 is a diagram illustrating a specific example 5 of each process of a video display system according to an embodiment; FIG. 16 is a diagram illustrating a specific example 6 of each process of a video display system according to an embodiment; 1 is a flowchart showing an example of a flow of a final display control process of a display terminal according to an embodiment; FIG. 2 is a flowchart showing an example of a flow of a display rendering process of a display terminal according to an embodiment; FIG. 3 is a hardware configuration diagram showing an example of a computer that realizes the functions of an editing terminal and a display terminal according to an embodiment.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The present disclosure will be described in the following order: 1. Embodiment 1-1. Configuration and processing of the video display system 100 according to the embodiment 1-2. Configuration and processing of each device of the video display system 100 according to the embodiment 1-3. Specific examples of each process of the video display system 100 according to the embodiment 1-4. Processing flow of the video display system 100 according to the embodiment 2. Effects of the embodiment 3. Hardware configuration

[0010] 1. Embodiment 1-1. Configuration and processing of video display system 100 The configuration and processing of video display system 100, which is an information processing system according to an embodiment, will be described using Fig. 1. Below, an example of the overall configuration of video display system 100, an example of the overall processing of video display system 100, the basic principle of arbitrary viewpoint video generation processing related to video display system 100, and the effects of video display system 100 as a whole will be described.

[0011] (1-1-1. Example of the overall configuration of the video display system 100) An example of the overall configuration of the video display system 100 will be described using FIG. 1. FIG. 1 is a diagram showing an example of the configuration and processing of the video display system 100 according to an embodiment. The video display system 100 is made up of an editing terminal 10, a display terminal 20, a display screen 30, and a three-dimensional model data database 40. Here, the editing terminal 10, the display terminal 20, the display screen 30, and the three-dimensional model data database 40 are connected to each other via a predetermined communication network (not shown) so as to be able to communicate with each other by wire or wirelessly. Note that the predetermined communication network can be any of various communication networks such as the Internet or a dedicated line.

[0012] (Editing Terminal 10) The editing terminal 10 is a first information processing device used by the engineer E1, and is used to process editing video data D, which is three-dimensional video data for editing or previewing. Ｉ１and displays the editing video I1. Note that the video display system 100 shown in Fig. 1 may include a plurality of editing terminals 10. In addition, although the example in Fig. 1 shows a case where the editing terminal 10 is realized by a desktop PC (Personal Computer), it may also be realized by a notebook PC, a smartphone, a server device, a cloud system, etc.

[0013] (Display Terminal 20) The display terminal 20 (20A, 20B, ...) is a second information processing device, and is used to display the display video data D, which is the two-dimensional video data for final display, i.e., for provision. Ｉ２ 1 shows the case where the display terminal 20 is realized by a server device, it may also be realized by a desktop PC, a notebook PC, a smartphone, a cloud system, or the like.

[0014] (Display screen 30) The display screen 30 displays the display image I2 for final display, i.e., for provision, generated by the display terminal 20. Note that the image display system 100 shown in FIG. 1 may include a plurality of display screens 30. Also, while the example in FIG. 1 shows a case where the display screen 30 is realized by a display device, it may also be realized by a screen onto which an image is projected, etc. When actually producing video content such as a movie using the image display system 100, the display screen 30 is, for example, a large screen, and subjects such as actors are filmed using a filming device (not shown) such as a video camera, with the large screen as a background. At this time, the display terminal 20 acquires information regarding the filming viewpoint, such as the position and attitude of the filming device, and generates display image data D corresponding to that viewpoint information. Ｉ２ is generated and displayed on the display screen 30. That is, the display video data D Ｉ２is generated by the display terminal 20 through a rendering process performed in real time in response to changes in the viewpoint of the camera, and is displayed on the display screen 30 as a background image. In this way, the video display system 100 is equipped with an arbitrary viewpoint video generation processing function, so that a shooting environment in which the background image as well as the subject changes from moment to moment in response to the camera's shooting viewpoint can be provided, thereby supporting the production of realistic video content. Video content produced using this video display system 100 can be recorded on a non-transitory storage medium such as a hard disk or optical disk and distributed, or copied to an external non-transitory storage medium via an online line.

[0015] (Three-dimensional model data database 40) The three-dimensional model data database 40 stores the three-dimensional model data acquired by the editing terminal 10 and the display terminal 20. Note that the video display system 100 shown in Fig. 1 may include a plurality of three-dimensional model data databases 40. Also, although the example in Fig. 1 shows a case where the three-dimensional model data database 40 is realized by a database separate from the editing terminal 10 and the display terminal 20, it may be integrated with either the editing terminal 10 or the display terminal 20.

[0016] (1-1-2. Example of processing performed by the entire video display system 100) An example of processing performed by the entire video display system 100 according to the embodiment will be described. Note that the following processes may be executed in a different order. Furthermore, some of the following processes may be omitted.

[0017] First, the editing terminal 10 executes an editing data acquisition process (step S1). For example, the editing terminal 10 acquires the editing data D from the three-dimensional model data database 40. １ and acquires three-dimensional model data created in the MPI, MCI, MSI, or 3DGS image format.

[0018] Second, the display terminal 20 executes a display data acquisition process (step S2). For example, the display terminal 20 acquires the display data D from the three-dimensional model data database 40. ２The display terminals 20 (20A, 20B, ...) acquire the same three-dimensional model data as the three-dimensional model data acquired by the editing terminal 10, which is created in the image format of MPI, MCI, MSI, or 3DGS. ２ Get.

[0019] Third, the editing terminal 10 executes an editing rendering process (step S3). For example, the editing terminal 10 executes a 3D rendering process with processing reduction on the acquired 3D model data, and generates editing video data D, which is 3D video data for editing or preview. Ｉ１ At this time, the editing terminal 10 performs 3D rendering of the entire object scene in the 3D world. In addition, the editing terminal 10 reduces the display image quality by reducing the texture, applying LOD (Level Of Detail), etc., depending on the processing power of the editing terminal 10, thereby achieving a playback speed that allows editing operation processing, i.e., a rendering speed.

[0020] Fourth, the editing terminal 10 executes an editing and display process (step S4). For example, the editing terminal 10 displays the generated video data D Ｉ１ The editing video I1, which is a three-dimensional video, is displayed on the monitor screen using the above.

[0021] Fifth, the engineer E1 performs an editing operation process (step S5). For example, the engineer E1 performs various editing operations on the editing video I1 displayed on the monitor screen to generate the editing data D １ At this time, the engineer E1 performs editing operations such as moving, rotating, enlarging, and reducing the objects in the object scene in the three-dimensional world.

[0022] Sixth, the editing terminal 10 executes an editing information transmission process (step S4). For example, the editing terminal 10 transmits the editing data D edited by the engineer E1. １The editing terminal 10 transmits editing information including the content and location of changes to each of the display terminals 20 (20A, 20B, ...). At this time, the editing terminal 10 can scalably determine the number of display terminals 20 according to the resolution of the display video I2 that is ultimately output.

[0023] Seventh, the display terminal 20 executes a display rendering process (step S7). For example, the display terminal 20 reflects editing based on the editing information on the acquired three-dimensional model data, executes two-dimensional rendering by point rendering, and outputs the result as two-dimensional video data for provision as display video data D Ｉ２ Generate.

[0024] At this time, the display terminals 20 (20A, 20B, ...) execute the display rendering process in a distributed manner. For example, the display terminal 20A executes the display rendering process corresponding to the display area R1A out of the display areas R1 (R1A, R1B, ..., R1F) of the display screen 30 divided into six, and outputs the display video data D Ｉ２Ａ Furthermore, the display terminal 20B executes a display rendering process corresponding to R1B among the display areas R1 (R1A, R1B, ..., R1F) of the display screen 30 divided into six parts, and generates the display video data D Ｉ２Ｂ Generate.

[0025] Furthermore, the display terminals 20 (20A, 20B, . . .) can determine their own processing resolutions according to resource limitations, that is, the number of display terminals 20.

[0026] Eighth, the display terminal 20 executes a final display process (step S8). For example, the display terminal 20 performs a final display process on the generated display video data D Ｉ２ is used to display the display image I2, which is an arbitrary viewpoint image, on the display screen 30.

[0027] At this time, the display terminals 20 (20A, 20B, ...) execute the final display processing in a distributed manner. For example, the display terminal 20A displays a display image I2A corresponding to the display area R1A of the display area R1 (R1A, R1B, ..., R1F) of the display screen 30, which is divided into six, on the display area R1A of the display screen 30. Furthermore, the display terminal 20B displays a display image I2B corresponding to the display area R1B of the display area R1 (R1A, R1B, ..., R1F) of the display screen 30, on the display area R1B of the display screen 30.

[0028] The video display system 100 can be realized using not only 3D model data created in the image format of MPI, MCI, MSI, or 3DGS, but also data created in an image format to which point rendering can be applied. In addition, the video display system 100 can also be realized using the display video data D generated by the processing of steps S1 to S8 above. Ｉ２Ａ It is also possible to create a computer-readable non-transitory recording medium on which video content including the above is recorded.

[0029] (1-1-3. Basic Principle of Arbitrary Viewpoint Video Generation Processing in Video Display System 100) The basic principle of the arbitrary viewpoint video generation processing in the video display system 100 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the basic principle of the arbitrary viewpoint video generation processing according to the embodiment.

[0030] Fig. 2(1) shows an editing image I1, which is a 3D image for editing or previewing. In the example of Fig. 2(1), a rectangle, a triangle, a circle, and a parallelogram are displayed from the front to the back.

[0031] Fig. 2(2) shows a display image I2(a), which is a two-dimensional image for final display when viewed from viewpoint (a). In the example of Fig. 2(2), a two-dimensional image of the editing image I1 viewed from viewpoint (a) in the foreground center is displayed.

[0032] Fig. 2(3) shows the display image I2(b), which is a two-dimensional image for final display when viewed from viewpoint (b). In the example of Fig. 2(3), a two-dimensional image of the editing image I1 viewed from viewpoint (b) slightly to the right of the front is displayed.

[0033] As described above, the video display system 100 can generate an arbitrary viewpoint video using three-dimensional model data created in the MPI, MCI, MSI, or 3DGS image format.

[0034] (1-1-4. Effects of the Video Display System 100) Below, the problems with the arbitrary viewpoint video generation process according to the reference technology will be explained, and then the effects of the video display system 100 will be explained.

[0035] (Problems with the Arbitrary Viewpoint Video Generation Process of the Reference Technology) First, the arbitrary viewpoint video generation process of the reference technology has difficulty in generating and displaying an arbitrary viewpoint video in a high-resolution, real-time 3D rendering format. Therefore, the arbitrary viewpoint video generation process of the reference technology requires distributing the rendering process to multiple terminals by dividing the output video into regions to achieve parallel high-speed processing. Furthermore, the arbitrary viewpoint video generation process of the reference technology requires point rendering to be performed by directly calculating a 2D image to be projected onto a display at an arbitrary viewpoint using a high-speed rendering method, rather than 3D rendering, using an MPI, MCI, MSI, or 3DGS image format for a 3D object scene. If the above processing is not performed, the arbitrary viewpoint video generation process of the reference technology may not be able to achieve a final display with high image quality and resolution.

[0036] Second, in the arbitrary viewpoint video generation process according to the reference technology, it is difficult to edit 3D model data without 3D rendering. For example, in the arbitrary viewpoint video generation process according to the reference technology, editing operations are difficult using only the 2D image projected on the display, so engineer E1 must perform editing on the entire object scene in the 3D world. Furthermore, in the arbitrary viewpoint video generation process according to the reference technology, engineer E1 must simultaneously view and confirm the final display image from the intended viewpoint when editing the entire object scene in the 3D world. Furthermore, in the arbitrary viewpoint video generation process according to the reference technology, engineer E1 also desires to perform editing operations on a single display from a single terminal, from the perspectives of operability and system control.

[0037] (Overview of video display system 100) In order to solve the above problems, video display system 100 realizes a distributed processing viewer system with editability for MPI, MCI, MSI, or 3DGS image formats that can generate arbitrary viewpoint video, as shown below.

[0038] First, video display system 100 configures a viewer system for MPI, MCI, MSI, or 3DGS input using a single editing terminal 10, multiple display terminals 20 that perform distributed parallel processing, and a display screen 30 that serves as a final high-resolution display. Second, video display system 100 simultaneously performs rendering and display processing on the single editing terminal 10 and the multiple display terminals 20. Third, video display system 100 performs rendering using a reduced-processing MPI, MCI, MSI, or 3DGS 3D display method for the rendering and display processing on editing terminal 10. Fourth, video display system 100 generates and displays a projected 2D image that serves as a final viewpoint image in real time using a rendering method that is high-quality and capable of distributed processing, utilizing the characteristics of MPI, MCI, MSI, or 3DGS, for the rendering and display processing on the multiple display terminals 20. Fifth, when the video display system 100 reflects the editing results in MPI, MCI, MSI, or 3DGS, the editing results can be confirmed on the display screen 30 in a short time by simultaneously executing the editing operation control process of a single editing terminal 10 and the final display control process of multiple display terminals 20.

[0039] Furthermore, in the video display system 100, the MPI, MCI, MSI, or 3DGS image format can be extended to an "image format to which point rendering can be applied." This is because, if the image format is one to which point rendering can be applied, when a subject is photographed using the image displayed on the display screen 30 as the background image, it is easy to quickly render and display an arbitrary viewpoint image that is required at any time in response to movement of the viewpoint of the imaging device in real time.

[0040] (Advantages of the video display system 100) First, the video display system 100 can generate arbitrary viewpoint video in a high-resolution, real-time, 3D rendering format and display it to the engineer E 1. Second, the video display system 100 allows the engineer E 1 to edit the 3D model data.

[0041] As described above, the image display system 100 can effectively display images for different purposes.

[0042] 3 to 8, the configuration and processing of each device included in the video display system 100 shown in Fig. 1 will be described. Below, an example configuration of the entire video display system 100 according to the embodiment, an example configuration and processing of the editing terminal 10, an example configuration and processing of the display terminal 20, an example configuration and processing of the display screen 30, and an example configuration and processing of the three-dimensional model data database 40 will be described.

[0043] (1-2-1. Example of Overall Configuration of Video Display System 100) An example of the overall configuration of video display system 100 will be described using FIG. 3. FIG. 3 is a block diagram showing an example of the configuration of each device of video display system 100 according to the embodiment. As shown in FIG. 3, video display system 100 is composed of an editing terminal 10 which is a first information processing device, a plurality of display terminals 20 (20A, ...) which are second information processing devices, a display screen 30, and a three-dimensional model data database 40. Furthermore, editing terminal 10, display terminal 20, display screen 30, and three-dimensional model data database 40 are communicably connected via a communication network N1 which is realized by the Internet, a dedicated line, or the like. Furthermore, video display system 100 may have an imaging device (not shown) which images a subject.

[0044] (1-2-2. Example of Configuration and Processing of Editing Terminal 10) The following describes an example of the configuration and processing of the editing terminal 10. The editing terminal 10 has an input unit 11, an output unit 12, a first communication unit 13, a first storage unit 14, and a first control unit 15.

[0045] (Input Unit 11) The input unit 11 controls input of various information to the editing terminal 10. For example, the input unit 11 is realized by a keyboard, a mouse, etc., and accepts input of various information to the editing terminal 10.

[0046] (Output unit 12) The output unit 12 controls the output of various information from the editing terminal 10. For example, the output unit 12 is realized by a display or the like, and displays various information stored in the editing terminal 10. The output unit 12 also functions as a first display screen, and displays the editing video data D, which is the first video data generated by the first generation unit 15b of the first control unit 15 (described later). Ｉ１ is used to display the first image, ie, the editing image I1.

[0047] (First communication unit 13) The first communication unit 13 controls data communication with other devices. For example, the first communication unit 13 performs data communication with each communication device via a router or the like. The first communication unit 13 can also perform data communication with a terminal (not shown).

[0048] (First Storage Unit 14) The first storage unit 14 stores various information referenced by the first control unit 15 when it operates and various information acquired when the first control unit 15 operates. The first storage unit 14 is composed of an editing video data storage unit 14a and a first editing information storage unit 14b. Here, the first storage unit 14 can be realized, for example, by a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disk. Note that, in the example of FIG. 3, the first storage unit 14 is installed inside the editing terminal 10, but it may also be installed outside the editing terminal 10, or multiple storage units may be installed.

[0049] (Editing Video Data Storage Unit 14a) The editing video data storage unit 14a stores the editing video data D Ｉ１ For example, the editing video data storage unit 14a stores the editing video data D generated by the first generating unit 15b of the first control unit 15, which will be described later. Ｉ１ Here, an example of data stored in the editing video data storage unit 14a will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the editing video data storage unit 14a of the editing terminal 10 according to the embodiment. In the example of Fig. 4, the editing video data storage unit 14a has an item such as "editing video data."

[0050] "Video data for editing" is video data for editing or previewing, and is, for example, three-dimensional video data generated by performing three-dimensional rendering on three-dimensional model data created in the MPI, MCI, MSI, or 3DGS image format.

[0051] That is, Figure 4 shows an example in which data such as "editing video data #1," "editing video data #2," "editing video data #3," etc. are stored in the editing video data storage unit 14a.

[0052] (First editing information storage unit 14b) The first editing information storage unit 14b stores first editing information. For example, the first editing information storage unit 14b stores editing information output by the editing unit 15d of the first control unit 15, which will be described later. Here, an example of data stored in the first editing information storage unit 14b will be described using FIG. 5. FIG. 5 is a diagram showing an example of the first editing information storage unit 14b of the editing terminal 10 according to the embodiment. In the example of FIG. 5, the first editing information storage unit 14b has items such as "3D model data" and "editing information."

[0053] The "3D model data" indicates identification information for identifying data used in the free viewpoint video generation process, and is, for example, an identification number or identification symbol of the 3D model data created in the image format of MPI, MCI, MSI, or 3DGS. The "editing information" indicates the editing data D edited by the user via the editing video I1. １ This information (editing results) is information related to the editing of the video I1, such as the type of operation performed by engineer E1 on the video I1 to be edited, such as moving, rotating, or enlarging / reducing an object, identification information of the object, and position information of the object.

[0054] That is, FIG. 5 shows an example in which data such as {3D model data: "3D model data #1", editing information: "editing information #1"}, {3D model data: "3D model data #2", editing information: "editing information #2"}, {3D model data: "3D model data #3", editing information: "editing information #3"}, etc. are stored in the first editing information storage unit 14b.

[0055] (First control unit 15) The first control unit 15 is responsible for overall control of the editing terminal 10. The first control unit 15 is composed of a first acquisition unit 15a, a first generation unit 15b, a first display unit 15c, and an editing unit 15d. Here, the first control unit 15 can be realized by, for example, an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0056] (First Acquisition Unit 15a) The first acquisition unit 15a acquires various types of information. The first acquisition unit 15a may store the acquired various types of information in the first storage unit 14. The editing data acquisition process will be described below.

[0057] (Editing Data Acquisition Process) The first acquisition unit 15a executes the editing data acquisition process. For example, the first acquisition unit 15a acquires the editing data D, which is the first data created in a predetermined image format. １ At this time, the editing data D １ is three-dimensional model data created in, for example, an MPI, MCI, MSI, or 3DGS image format. １ is data created in an image format that allows point rendering, for example.

[0058] Specifically, the first acquiring unit 15a acquires the editing data D designated by the engineer E1 through operation of the editing terminal 10. １ As a result, "3D model data #1," "3D model data #2," "3D model data #3," . . . stored in the 3D model data database 40 are acquired.

[0059] (First Generator 15b) The first generator 15b generates various types of information. The first generator 15b stores the generated various types of information in the first storage unit 14. The editing video data generation process will be described below.

[0060] (Editing Video Data Generation Process) The first generation unit 15b executes editing video data generation process. For example, the first generation unit 15b generates editing data D １ Rendering for editing, which is the first rendering, is performed on the image data D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13, D14, D15, D16, D17, D18, D19, D20, D21, D22, D23, D24, D25, D26, D27, D28, D29, D3 Ｉ１ At this time, the first generating unit 15b performs three-dimensional rendering as rendering for editing, and generates the video data for editing D Ｉ１ The first generating unit 15b generates the three-dimensional video data as the editing data D １ Then, a model transformation is performed to convert from the local coordinate system to the world coordinate system, a projection transformation is performed to convert from the world coordinate system to the screen coordinate system, and a rasterizer is executed to render on the screen, thereby generating three-dimensional video data.

[0061] Specifically, first, the first generation unit 15b references the "3D model data #1" acquired from the 3D model data database 40 and reads data related to materials, color attributes, etc. Second, the first generation unit 15b performs model transformation to convert the "3D model data #1" from the local coordinate system to the world coordinate system. Third, the first generation unit 15b performs projection transformation to convert the "3D model data #1" from the world coordinate system to the screen coordinate system. Fourth, the first generation unit 15b uses a rasterizer, i.e., renders the "3D model data #1" on the screen according to materials, color attributes, etc., to generate "editing video data #1," which is 3D video data. Fifth, the first generation unit 15b stores the generated "editing video data #1" in the editing video data storage unit 14a.

[0062] (First display unit 15c) The first display unit 15c displays various types of information. Note that the first display unit 15c refers to and displays various types of information stored in the first storage unit 14. The editing video display process will be described below.

[0063] (Editing Video Display Process) The first display unit 15c executes editing video display process. For example, the first display unit 15c displays the generated editing video data DＩ１ The first display unit 15c displays the editing video I1, which is the first video, via the output unit 12 in a format that allows the user to edit it.

[0064] Specifically, first, the first display unit 15c refers to the "editing video data #1" stored in the editing video data storage unit 14a. Second, the first display unit 15c uses the "editing video data #1" to display the editing video I1 on the output unit 12 in a format that can be edited by the engineer E1.

[0065] (Editing unit 15d) The editing unit 15d edits various types of information. The editing unit 15d outputs the editing results. The editing unit 15d stores the output editing results in the first storage unit 14. The editing operation reception process and the editing information transmission process will be described below.

[0066] (Editing Operation Acceptance Process) The editing unit 15d executes the editing operation acceptance process. For example, the editing unit 15d accepts an editing operation on the editing video I1 from the user, executes the editing, and also creates the editing data D １ At this time, the editing unit 15d outputs editing information including the type of operation such as moving, rotating, or enlarging / reducing an object on an object scene of the editing video I1, identification information of the object, position information of the object, etc.

[0067] Specifically, first, the editing unit 15d receives an editing operation on the editing video I1 from the engineer E1, and generates the editing data D １ Secondly, the editing unit 15d executes editing of the edit data D １ Third, the editing unit 15d stores the output "editing information #1" in the first editing information storage unit 14b.

[0068] (Editing Information Transmission Process) The editing unit 15d executes the editing information transmission process. For example, when the edited video I1, which is the displayed first video, is edited, the editing unit 15d transmits the editing data D １The editing unit 15d also transmits editing information relating to the editing of the display video data D Ｉ２ The number of display terminals 20 to be used for the distributed processing can be determined according to the resolution of the display terminals 20.

[0069] Specifically, first, editing unit 15d references {3D model data: "3D model data #1", editing information: "editing information #1"} as the first editing information stored by first editing information storage unit 14b. Second, editing unit 15d identifies six display terminals 20 to which the first editing information is to be transmitted: "display terminal A," "display terminal B," "display terminal C," "display terminal D," "display terminal E," and "display terminal F." Third, editing unit 15d transmits the referenced first editing information to each of "display terminal A," "display terminal B," "display terminal C," "display terminal D," "display terminal E," and "display terminal F."

[0070] At this time, the editing section 15d edits the display video data D designated by the engineer E1. Ｉ２ If it is determined that it is difficult to achieve the resolution of the display terminal G in real time with six display terminals 20, the editing unit 15d can transmit the referenced first editing information to the display terminal G, which is the backup display terminal 20. Furthermore, the editing unit 15d can increase the number of areas in the display area R1 in accordance with the increased number of display terminals 20, and reallocate the areas to the multiple display terminals 20.

[0071] On the other hand, the editing section 15d edits the display video data D designated by the engineer E1. Ｉ２ If it is determined that the resolution can be realized in real time even with five display terminals 20, the editing unit 15d can transmit the referenced first editing information to the five display terminals 20 excluding "display terminal F." Furthermore, the editing unit 15d can reduce the number of areas in the display area R1 in accordance with the reduced number of display terminals 20, and reallocate the areas to the multiple display terminals 20.

[0072] (1-2-3. Example of configuration and processing of display terminal 20) An example of configuration and processing of display terminal 20 will be described using Fig. 3 again. Display terminal 20 has a second communication unit 21, a second storage unit 22, and a second control unit 23. Note that display terminal 20 may have an input unit (e.g., a mouse, a keyboard, etc.) that accepts various operations from the administrator of video display system 100, and an output unit (e.g., a display, etc.) that displays various information.

[0073] (Second communication unit 21) The second communication unit 21 controls data communication with other devices. For example, the second communication unit 21 performs data communication with each communication device via a router or the like. The second communication unit 21 can also perform data communication with a terminal (not shown).

[0074] (Second storage unit 22) The second storage unit 22 stores various pieces of information referenced by the second control unit 23 when it operates and various pieces of information acquired when the second control unit 23 operates. The second storage unit 22 is composed of a second editing information storage unit 22a and a display video data storage unit 22b. Here, the second storage unit 22 can be realized, for example, by a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disk. Note that, in the example of FIG. 3, the second storage unit 22 is installed inside the display terminal 20, but it may also be installed outside the display terminal 20, or multiple storage units may be installed.

[0075] (Second editing information storage unit 22a) The second editing information storage unit 22a stores second editing information. For example, the second editing information storage unit 22a stores editing information received by a receiving unit 23b of the second control unit 23, which will be described later. Here, an example of data stored in the second editing information storage unit 22a will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of the second editing information storage unit 22a of the display terminal 20 according to the embodiment. In the example of FIG. 6, the second editing information storage unit 22a has items such as "editing terminal," "3D model data," and "editing information."

[0076] "Editing terminal" indicates identification information for identifying the editing terminal 10 that transmitted the editing information, such as the identification number or identification symbol of the editing terminal 10. "3D model data" indicates identification information for identifying data used in the free viewpoint video generation process, such as the identification number or identification symbol of 3D model data created in the MPI, MCI, MSI, or 3DGS image format. "Editing information" indicates the editing data D edited by the user via the editing video I1. １ This information (editing results) is information related to the editing of the video I1, such as the type of operation performed by engineer E1 on the video I1 to be edited, such as moving, rotating, or enlarging / reducing an object, identification information of the object, and position information of the object.

[0077] That is, FIG. 6 shows an example in which the following data is stored in the second editing information storage unit 22a as editing information received from the editing terminal 10 identified as "editing terminal #1": {3D model data: "3D model data #1", editing information: "editing information #1"}, {3D model data: "3D model data #2", editing information: "editing information #2"}, {3D model data: "3D model data #3", editing information: "editing information #3"}, etc.

[0078] (Display Image Data Storage Unit 22b) The display image data storage unit 22b stores the display image data D Ｉ２ For example, the display video data storage unit 22b stores the display video data D generated by the second generation unit 23c of the second control unit 23, which will be described later. Ｉ２ Here, an example of data stored in the display video data storage unit 22b will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the display video data storage unit 22b of the display terminal 20 according to the embodiment. In the example of FIG. 7, the display video data storage unit 22b has items such as "display area" and "display video data."

[0079] The "display area" indicates identification information for identifying the multiple display areas R1 assigned to each of the multiple display terminals 20, and is, for example, an identification number or identification symbol of the display area R1 on the display screen 30. The "display video data" is video data for final display, i.e., for provision, and is, for example, two-dimensional video data generated by performing two-dimensional rendering on three-dimensional model data created in the MPI, MCI, MSI, or 3DGS image format.

[0080] That is, Figure 7 shows an example in which data such as {display area: "display area #1-A", display video data: "display video data #1-A"}, {display area: "display area #2-A", display video data: "display video data #2-A"}, {display area: "display area #3-A", display video data: "display video data #3-A"}, etc. are stored in the display video data storage unit 22b.

[0081] (Second control unit 23) The second control unit 23 is responsible for overall control of the display terminal 20. The second control unit 23 is composed of a second acquisition unit 23a, a reception unit 23b, a second generation unit 23c, and a second display unit 23d. Here, the second control unit 23 can be realized by, for example, an electronic circuit such as a CPU or an MPU, or an integrated circuit such as an ASIC or an FPGA.

[0082] (Second Acquisition Unit 23a) The second acquisition unit 23a acquires various types of information. The second acquisition unit 23a may store the acquired various types of information in the second storage unit 22. The display data acquisition process will be described below.

[0083] (Display Data Acquisition Process) The second acquisition unit 23a executes the display data acquisition process. For example, the second acquisition unit 23a acquires the edit data D １ The display data D is the same second data as ２ At this time, the display data D ２ is three-dimensional model data created in, for example, an MPI, MCI, MSI, or 3DGS image format. ２is data created in an image format that allows point rendering, for example.

[0084] Specifically, the second acquiring unit 23a acquires the editing data D designated by the engineer E1 through operation of the editing terminal 10. １ The same display data D ２ As a result, "3D model data #1," "3D model data #2," "3D model data #3," . . . stored in the 3D model data database 40 are acquired.

[0085] (Receiving Unit 23b) The receiving unit 23b receives various types of information. The receiving unit 23b stores the received various types of information in the second storage unit 22. The editing information receiving process will be described below.

[0086] (Editing Information Reception Process) The receiving unit 23b executes editing information reception process. For example, the receiving unit 23b receives editing information from the editing terminal 10. The receiving unit 23b also receives the display video data D Ｉ２ The resolution of each of the above can be determined.

[0087] Specifically, first, the receiving unit 23b receives {3D model data: "3D model data #1", editing information: "editing information #1"} as first editing information transmitted by the editing terminal 10 identified by "editing terminal #1." Second, the receiving unit 23b stores the received first editing information as second editing information corresponding to "editing terminal #1" in the second editing information storage unit 22a.

[0088] At this time, if the receiving unit 23b determines that it is difficult to achieve real-time operation with the current resolution for the number of editing terminals 10 specified by the engineer E1, it can make changes to lower the resolution for each display area R1 assigned to each of the multiple editing terminals 10.

[0089] On the other hand, if the receiving unit 23b determines that real-time realization is possible even if the resolution is increased for the number of editing terminals 10 specified by the engineer E1, it can make changes to increase the resolution for each display area R1 assigned to each of the multiple editing terminals 10.

[0090] (Second generation unit 23c) The second generation unit 23c generates various types of information. Note that the second generation unit 23c references the various types of information stored in the second storage unit 22. The second generation unit 23c also stores the generated various types of information in the second storage unit 22. The display video data generation process will be described below.

[0091] (Display Video Data Generation Process) The second generation unit 23c executes display video data generation process. For example, the second generation unit 23c generates display data D that reflects editing based on the received editing information. ２ 2. The second rendering, which is different from the first rendering, which is the rendering for editing, is performed to generate the second video data for display D Ｉ２ At this time, the second generating unit 23c performs two-dimensional rendering as the display rendering, and generates the display video data D Ｉ２ The second generator 23c generates two-dimensional video data as the display data D ２ The second generating unit 23c generates two-dimensional video data by executing a direct rasterizer that performs point rendering on the screen. ２ , and the display video data D corresponding to the allocated display areas R1 are generated. Ｉ２ The second generating unit 23c generates the display data D ２ may be stored in the three-dimensional model data database 40. As a result, the second generation unit 23c can generate the display image data D Ｉ２ When generating the edit data D １ The same display data D ２The display data D reflecting the edited data is obtained from the three-dimensional model data database 40 without repeating the process of acquiring the edited data and reflecting the edited information. ２ is directly acquired, and the display image data D Ｉ２ The second generating unit 23c can generate the display data D ２ If further editing is required for the display data D ２ After obtaining the data, it is used as the editing data D １ Further, the second generating unit 23c generates the display data D in accordance with the viewpoint from which the subject is photographed by the photographing device, with the display image I2 displayed on the display screen 30 as the background. ２ Rendering for display is performed on the image data D Ｉ２ can be generated.

[0092] Specifically, first, the second generation unit 23c references the "3D model data #1" acquired from the 3D model data database 40 and reads data related to materials, color attributes, and the like. Second, the second generation unit 23c identifies the display area R1A assigned to the display terminal 20A as the "display area #1-A." Third, the second generation unit 23c generates "display video data #1-A," which is two-dimensional video data, by directly rasterizing, i.e., by point rendering the "display area #1-A" of the "3D model data #1" onto the screen according to materials, color attributes, and the like. Fifth, the second generation unit 23c stores the generated "display video data #1-A" in the display video data storage unit 22b.

[0093] (Second display unit 23d) The second display unit 23d displays various types of information. Note that the second display unit 23d refers to and displays various types of information stored in the second storage unit 22. The display image display process will be described below.

[0094] (Display Image Display Process) The second display unit 23d executes the display image display process. For example, the second display unit 23d displays the generated display image data D Ｉ２The second display unit 23d displays the display image I2, which is the second image, using the generated display image data D Ｉ２ , and displays the display image I2 in each of the plurality of display areas R1 on the display screen 30. At this time, the second display unit 23d displays the display image I2 in a manner that allows the user to check the editing results via the display screen 30.

[0095] Specifically, first, second display unit 23d refers to "display video data #1-A" stored in display video data storage unit 22b. Second, second display unit 23d uses "display video data #1-A" to display display video I2A in "display area #1-A" on display screen 30 in a manner that allows engineer E1 to check the editing results.

[0096] (1-2-4. Example of Configuration and Processing of Display Screen 30) An example of the configuration and processing of the display screen 30 will be described again with reference to Figure 3. The display screen 30 is an output device for achieving high resolution, and can be realized, for example, by a display device or a screen onto which an image is projected. Note that the display screen 30 may be configured as an integrated unit with the display terminal 20 described above.

[0097] The display screen 30 controls the output of various information from the display terminal 20. For example, the display screen 30 is a second display screen, and displays the display video data D input by the display terminal 20. Ｉ２ At this time, the display screen 30 displays a display image I2, which is a second image, using the plurality of display image data D generated by and input from the plurality of display terminals 20 (20A, ...). Ｉ２ (D Ｉ２Ａ , ...) to display a plurality of display images I2 (I2A, ...) in a plurality of display areas R1 (R1A, ...). At this time, the display screen 30 displays the display images I2 in a manner that allows the user to check the editing results.

[0098] Specifically, first, display screen 30 accepts "display video data #1-A" input by display terminal 20A, accepts "display video data #1-B" input by display terminal 20B, accepts "display video data #1-C" input by display terminal 20C, accepts "display video data #1-D" input by display terminal 20D, accepts "display video data #1-E" input by display terminal 20E, and accepts "display video data #1-F" input by display terminal 20F. Second, the display screen 30 displays "display video data #1-A" in display area R1A, "display video data #1-B" in display area R1B, "display video data #1-C" in display area R1C, "display video data #1-D" in display area R1D, "display video data #1-E" in display area R1E, ​​and "display video data #1-F" in display area R1F.

[0099] (1-2-5. Example of Configuration and Processing of 3D Model Data Database 40) An example of the configuration and processing of the 3D model data database 40 will be described again with reference to Figure 3. The 3D model data database 40 is a storage device, and can be realized by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disk. The 3D model data database 40 may be configured to be integrated with either the editing terminal 10 or the display terminal 20 described above.

[0100] The 3D model data database 40 stores 3D model data. For example, the 3D model data database 40 stores 3D model data created by engineer E1 and transmitted by the editing terminal 10. Furthermore, the 3D model data database 40 may store modified 3D model data that reflects the results of editing performed by engineer E1 using the editing terminal 10. Furthermore, the 3D model data edited by the video display system 100 and stored in the 3D model data database 40 can be used as a data asset by copying it to a non-transitory recording medium such as a hard disk or optical disk and distributing it, or by copying it to an external non-transitory recording medium via an online connection. For example, if a video display system that generates free-viewpoint background images used in the production of video content is located at another site, the 3D model data can be copied to the non-transitory recording medium of the 3D model database of that video display system and utilized. The video display system may be a system similar to the video display system 100 of the present disclosure, or may have a different configuration as long as it has the capability of rendering 3D model data in the same format. An example of data stored in the three-dimensional model data database 40 will now be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of the three-dimensional model data database 40 according to the embodiment. In the example of Fig. 8, the three-dimensional model data database 40 has an item such as "three-dimensional model data."

[0101] The "3D model data" is data used in the arbitrary viewpoint video generation process, and is, for example, 3D model data created in an image format such as MPI, MCI, MSI, or 3DGS. The "3D model data" is also, for example, data created in an image format that allows point rendering.

[0102] That is, FIG. 8 shows an example in which data such as “3D model data #1,” “3D model data #2,” “3D model data #3,” etc. are stored in the 3D model data database 40.

[0103] 9 to 16, specific examples of the processes of the video display system 100 according to the embodiment will be described. Specific examples 1 to 6 of the processes of the video display system 100 according to the embodiment will be described below.

[0104] (1-3-1. Specific Example 1) Specific example 1 of each process of the video display system 100 according to the embodiment will be described using Figures 9 and 10. Figure 9 is a diagram showing specific example 1-1 of each process of the video display system 100 according to the embodiment. Figure 10 is a diagram showing specific example 1-2 of each process of the video display system 100 according to the embodiment. Below, the basic principles of MPI, MCI, MSI, and 3DGS for creating 3D model data according to the embodiment will be described.

[0105] (Basic Principles of MPI) The basic principles of MPI for creating 3D model data will be explained using Fig. 9(1). In MPI, information on a subject on a plane at screen coordinates SC1 is sampled from a field of view coordinate VC1, which is the camera viewpoint, and 3D model data is created by integrating the information sampled from multiple field of view coordinates VC1 and multiple screen coordinates SC1.

[0106] (Basic Principles of MCI) The basic principles of MCI for creating 3D model data will be explained using Fig. 9(2). In MCI, information on a subject on a curved screen coordinate SC2 is sampled from a field of view coordinate VC2, which is the camera viewpoint, and 3D model data is created by integrating the information sampled from multiple field of view coordinates VC2 and multiple screen coordinates SC2.

[0107] (Basic Principles of MSI) The basic principles of MSI for creating 3D model data will be explained using Figure 9 (3). In MSI, information on the subject on screen coordinates SC3 on the surface of a sphere is sampled from visual field coordinates VC3, which are the camera viewpoint, and 3D model data is created by integrating the information sampled from multiple visual field coordinates VC3 and multiple screen coordinates SC3.

[0108] (Basic Principles of 3DGS) The basic principles of 3DGS for creating 3D model data will be explained using Fig. 10. In 3DGS, information obtained from the field of view coordinate VC4, which is the camera viewpoint, is expressed as a 3D Gaussian (see Figs. 10(d) to (f)), and 3D model data is created by splatting data points onto screen coordinate SC4 on a plane (see Figs. 10(a) to (c)).

[0109] (1-3-2. Specific Example 2) Specific example 2 of each process of the video display system 100 according to the embodiment will be described using Fig. 11 and Fig. 12. Fig. 11 is a diagram showing a specific example 2-1 of each process of the video display system 100 according to the embodiment. Fig. 12 is a diagram showing a specific example 2-2 of each process of the video display system 100 according to the embodiment. Below, the basic principles of point rendering processing, the flow of three-dimensional rendering processing, and the flow of point rendering processing will be described as specific examples of rendering processing according to the embodiment.

[0110] (Basic Principles of Point Rendering) The basic principles of point rendering will be described using FIG. 11. The example in FIG. 11 shows point rendering, which is a high-speed rendering process that uses the image format of the MPI (see FIG. 9(1)). In point rendering, the pixel values ​​of each 2D image to be projected onto the display are calculated directly from the color and α value of each layer of the MPI through which the line of sight passes (see FIGS. 11(a) to 11(d)). Furthermore, point rendering makes it possible to quickly render arbitrary viewpoint images by omitting each step of the 3D rendering process.

[0111] (Flow of 3D rendering process) Using FIG. 12(1), the editing video data D Ｉ１In the example of Fig. 12(1), the 3D rendering process involves converting the local coordinates of the 3D model data into world coordinates through model transformation, converting the world coordinates of the 3D model data into screen coordinates through projection transformation, and then generating a 3D image through a rasterizer. During this model transformation, the 3D model data can be edited, such as by scaling or rotating.

[0112] (Flow of Point Rendering Process) The point rendering process for generating display video data DI2 according to the embodiment will be described using Fig. 12 (2). In the example of Fig. 12 (2), the point rendering process involves directly acquiring the screen coordinates of the 3D model data using a direct rasterizer, and generating a two-dimensional image. At this time, editing such as scaling or rotation of the 3D model data is not possible.

[0113] (1-3-3. Specific Example 3) Specific example 3 of each process of the video display system 100 according to the embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing specific example 3 of each process of the video display system 100 according to the embodiment. Below, an editing operation process using the editing terminal 10 according to the embodiment will be described.

[0114] As shown in the example of FIG. 13, an engineer E1 performs an editing operation on an edit video I1 displayed on the monitor screen of the editing terminal 10, thereby creating edit data D １ In the example of Fig. 13, engineer E1 performs editing operations on the object scene of video to be edited I1, such as moving the object (see Fig. 13(a)), rotating the object (see Fig. 13(b)), and scaling the object (see Fig. 13(c)).

[0115] (1-3-4. Specific Example 4) Specific example 4 of each process of the video display system 100 according to the embodiment will be described using FIG. 14. FIG. 14 is a diagram showing specific example 4 of each process of the video display system 100 according to the embodiment. Below, the distributed rendering process using the display terminal 20 according to the embodiment and the distributed display process using the display screen 30 will be described. In the embodiment, the display terminal 20 is made up of a plurality of display terminals 20A, 20B, ... for distributed processing that are capable of cooperating with each other to perform the distributed rendering process and the distributed display process.

[0116] (Distributed Rendering Process) As shown in the example of FIG. 14(1), the display terminal 20 receives the display video data D corresponding to the display area R1 of the display screen 30. Ｉ２ In the example of FIG. 14(1), the display terminal 20A generates the display video data D corresponding to the display area R1A (see FIG. 14(1)(a)). Ｉ２Ａ The display terminal 20B generates the display video data D corresponding to the display area R1B (see FIG. 14(1)(b)). Ｉ２Ｂ The display terminal 20C generates the display video data D corresponding to the display area R1C (see FIG. 14(1)(c)). Ｉ２Ｃ The display terminal 20D generates the display video data D corresponding to the display area R1D (see FIG. 14(1)(d)). Ｉ２Ｄ The display terminal 20E generates the display video data D corresponding to the display region R1E (see FIG. 14(1)(e)). Ｉ２Ｅ The display terminal 20F generates the display video data D corresponding to the display region R1F (see FIG. 14(1)(f)). Ｉ２Ｆ is generated.

[0117] (Distributed Display Processing) As shown in the example of Fig. 14(2), the display screen 30 displays the display images I2 corresponding to each display area R1 by distributed display processing. In the example of Fig. 14(2), the display screen 30 displays the display image I2A in the display area R1A, the display image I2B in the display area R1B, the display image I2C in the display area R1C, the display image I2D in the display area R1D, the display image I2E in the display area R1E, ​​and the display image I2F in the display area R1F.

[0118] (1-3-5. Specific Example 5) Specific example 5 of each process of the video display system 100 according to the embodiment will be described with reference to Fig. 15. Fig. 15 is a diagram showing specific example 5 of each process of the video display system 100 according to the embodiment. The final display video confirmation process by engineer E1 according to the embodiment will be described below.

[0119] As shown in the example of Figure 15, engineer E1 performs editing operations on edit video I1 displayed on the monitor screen of editing terminal 10 (see Figure 15 (1)). At the same time as performing the editing operations, engineer E1 can also perform a final display video confirmation process by viewing display video I2 displayed on display screen 30 (see Figure 15 (2)). As described above, engineer E1 can confirm display video I2, which is the final display video, particularly during real-time editing.

[0120] (1-3-6. Specific Example 6) Specific example 6 of each process of the video display system 100 according to the embodiment will be described using Fig. 16. Fig. 16 is a diagram showing specific example 6 of each process of the video display system 100 according to the embodiment. Below, an example of the software configuration of the editing terminal 10 and the display terminal 20 according to the embodiment, and software cooperation processing will be described.

[0121] 16, the editing terminal 10 has an editing server program SP-10, a launcher program LAP-10, and an editing operation processing program EP-10. Furthermore, the display terminals 20 (20A, 20B, ..., 20F) have a listener program LIP-20 (LIP-20A, LIP-20B, ..., LIP-20F) and a display processing program DP-20 (DP-20A, DP-20B, ..., DP-20F). The editing terminal 10 and the display terminals 20 (20A, 20B, ..., 20F) are communicatively connected via a communication network N1.

[0122] (Software collaboration processing) As shown in Figure 16, the launcher program LAP-10 launches the display processing program DP-20 (DP-20A, DP-20B, ..., DP-20F) via the editing operation processing program EP-10, the editing server program SP-10, and the listener program LIP-20 (LIP-20A, LIP-20B, ..., LIP-20F) (see solid arrows).

[0123] The editing operation processing program EP-10 and the display processing program DP-20 (DP-20A, DP-20B, . . . , DP-20F) are each connected to the editing server program SP-10 (see the dashed arrows).

[0124] 1-4. Process Flow of the Video Display System 100 The process flow of the video display system 100 according to the embodiment will be described using Figures 17 to 21. Below, the overall process flow of the video display system 100 will be described, and then each process, including the editing operation control process and editing rendering process of the editing terminal 10, and the final display control process and display rendering process of the display terminal 20, will be described.

[0125] (1-4-1. Overall Processing of Video Display System 100) The overall processing flow of the video display system 100 according to the embodiment will be described with reference to FIG. 17. FIG. 17 is a sequence diagram showing an example of the overall processing flow of the video display system 100 according to the embodiment. Note that the processing of steps S101 to S105 below may be executed in a different order. Also, some of the processing of steps S101 to S105 below may be omitted.

[0126] First, the editing terminal 10 executes a launcher program startup process (step S101). For example, when starting the editing operation control process of the editing terminal 10 and the final display control process of the display terminal 20, the editing terminal 10 starts the launcher program LAP.

[0127] Second, the display terminal 20 executes a listener program start-up process (step S102). For example, the display terminals 20 (20A, 20B, . . . , 20F) each start up a listener program LIP in response to the launcher program start-up process of the editing terminal 10.

[0128] Third, the editing terminal 10 executes an editing server program start-up process (step S103). For example, the editing terminal 10 starts up the editing server program EP when the editing terminal 10 starts up a launcher program.

[0129] Fourth, the editing terminal 10 executes an editing operation control process (step S104). For example, the editing terminal 10 executes the editing operation control process of steps S201 to S206, which will be described later, when the editing server program is started up in the editing terminal 10.

[0130] Fifth, the display terminal 20 executes a final display control process (step S105). For example, the display terminal 20 (20A, 20B, ..., 20F) executes the final display control process of steps S401 to S406, which will be described later, in response to the listener program startup process of each of the display terminals 20 (20A, 20B, ..., 20F).

[0131] (1-4-2. Editing Operation Control Processing of Editing Terminal 10) The flow of editing operation control processing of the editing terminal 10 according to the embodiment will be described using FIG. 18. FIG. 18 is a flowchart showing an example of the flow of editing operation control processing of the editing terminal 10 according to the embodiment. Note that the processing of steps S201 to S206 below can also be executed in a different order. Also, some of the processing of steps S201 to S206 below may be omitted.

[0132] First, the editing terminal 10 executes a three-dimensional model data acquisition process (step S201). For example, the editing terminal 10 acquires three-dimensional model data stored in the three-dimensional model data database 40 as editing data D in response to an operation by the engineer E1. １ Obtain as.

[0133] At this time, if the editing terminal 10 decides to end the display of the editing video I1 (step S202: Yes), it ends the editing operation control process. On the other hand, if the editing terminal 10 does not decide to end the display of the editing video I1 (step S202: No), it proceeds to the process of step S203.

[0134] If the editing operation is performed by the engineer E1 (step S203: Yes), the editing terminal 10 proceeds to the process of step S204. On the other hand, if the editing operation is not performed by the engineer E1 (step S203: No), the editing terminal 10 proceeds to the process of step S205.

[0135] Second, the editing terminal 10 executes an editing information transmission process (step S204). For example, the editing terminal 10 transmits the editing data D edited by the engineer E1. １ The editing information relating to the above is transmitted to each of the display terminals 20 (20A, 20B, . . . ).

[0136] Third, the editing terminal 10 executes rendering processing for editing (step S205). For example, the editing terminal 10 executes the processing of steps S301 to S304 described later to generate three-dimensional video data for editing D Ｉ１ Generate.

[0137] Fourth, the editing terminal 10 executes an editing and display process (step S206). For example, the editing terminal 10 displays the generated video data D Ｉ１ The editing video I1, which is a three-dimensional video, is displayed on the monitor screen using the above, and the process returns to step S202.

[0138] (1-4-3. Editing Rendering Process) The flow of editing rendering process of the editing terminal 10 according to the embodiment will be described using FIG. 19. FIG. 19 is a flowchart showing an example of the flow of editing rendering process of the editing terminal 10 according to the embodiment. Note that the processes of steps S301 to S304 below can also be executed in a different order. Furthermore, some of the processes of steps S301 to S304 below may be omitted.

[0139] First, the editing terminal 10 executes a data reading process (step S301). For example, the editing terminal 10 reads the editing data D １ , materials, color attributes, etc.

[0140] Second, the editing terminal 10 executes a model conversion process (step S302). For example, the editing terminal 10 converts the editing data D １ Transform from the local coordinate system to the world coordinate system.

[0141] Third, the editing terminal 10 executes a projection transformation process (step S303). For example, the editing terminal 10 converts the edit data D １ Transform from the world coordinate system to the screen coordinate system.

[0142] Fourth, the editing terminal 10 executes a rasterizer process (step S304). For example, the editing terminal 10 performs a rasterizer process on the editing data D １ By rendering the image data D on a screen according to the material, color attributes, etc., the image data D is rendered as 3D image data. Ｉ１ is generated, and the rendering process for editing is completed.

[0143] (1-4-4. Final display control processing of the display terminal 20) The flow of the final display control processing of the display terminal 20 according to the embodiment will be described using FIG. 20. FIG. 20 is a flowchart showing an example of the flow of the final display control processing of the display terminal 20 according to the embodiment. Note that the processing of steps S401 to S406 below can also be executed in a different order. Furthermore, some of the processing of steps S401 to S406 below may be omitted.

[0144] First, the display terminal 20 executes a three-dimensional model data acquisition process (step S401). For example, the display terminal 20 acquires three-dimensional model data stored in the three-dimensional model data database 40 as display data D ２ Obtain as.

[0145] At this time, if the display terminal 20 has finished displaying the display image I2 (step S402: Yes), it ends the final display control process. On the other hand, if the display terminal 20 has not finished displaying the display image I2 (step S402: No), it proceeds to the process of step S403.

[0146] Furthermore, if the display terminal 20 has received editing information from the editing terminal 10 (step S403: Yes), the display terminal 20 proceeds to the process of step S404. On the other hand, if the display terminal 20 has not received editing information from the editing terminal 10 (step S403: No), the display terminal 20 proceeds to the process of step S405.

[0147] Second, the display terminal 20 executes an editing information reflection process (step S404). For example, the display terminal 20 reflects the editing data D edited by the engineer E1. １ Editing information related to ２ Reflected in.

[0148] Third, the display terminal 20 executes a display rendering process (step S405). For example, the display terminal 20 executes the processes of steps S501 to S502 described later to generate display video data D, which is two-dimensional video data. Ｉ２ Generate.

[0149] Fourth, the display terminal 20 executes a final display process (step S406). For example, the display terminal 20 performs a final display process on the generated display video data D Ｉ２ is used to display the display image I2, which is an arbitrary viewpoint image, on the display screen 30, and the process returns to step S402.

[0150] (1-4-5. Editing Rendering Process) The flow of the display rendering process of the display terminal 20 according to the embodiment will be described using FIG. 21. FIG. 21 is a flowchart showing an example of the flow of the display rendering process of the display terminal 20 according to the embodiment. Note that the processes of steps S501 to S502 below can also be executed in a different order. Furthermore, some of the processes of steps S501 to S502 below may be omitted.

[0151] First, the display terminal 20 executes a data read process (step S501). For example, the display terminal 20 reads the display data D ２ , materials, color attributes, etc.

[0152] Second, the display terminal 20 directly executes rasterization processing (step S502). For example, the display terminal 20 directly executes rasterization processing on the display data D ２ By point-rendering the image data D to be displayed, which is two-dimensional image data, directly onto the screen according to the material, color attributes, etc. Ｉ２ is generated, and the rendering process for display is completed.

[0153] 2. Effects of the embodiment Finally, effects of the embodiment will be described below. Effects corresponding to the processing according to the embodiment will be described below.

[0154] In the process according to the embodiment described above, the video display system 100 includes an editing terminal 10 and a display terminal 20. The editing terminal 10 displays editing data D created in a predetermined image format. １ and obtain the editing data D １ Rendering processing for editing is performed on the image data D for editing or preview. Ｉ１ and generate the editing video data D Ｉ１ When the edited data D １ The display terminal 20 transmits editing information relating to the editing of the edit data D １ The same display data D ２ , receives editing information from the editing terminal 10, and generates display data D ２ Rendering processing for display, which is different from the rendering processing for editing, is performed on the image data D Ｉ２ Therefore, this process makes it possible to effectively display images for different purposes.

[0155] In the process according to the embodiment described above, the video display system 100 includes a plurality of display terminals 20 for distributed processing, and each of the plurality of display terminals 20 for distributed processing displays the display data D２ , and the display video data D corresponding to each allocated area among the plurality of display areas R1 is obtained. Ｉ２ Therefore, in this process, by performing distributed rendering, it is possible to effectively display images for different purposes.

[0156] In the process according to the embodiment described above, the editing terminal 10 Ｉ２ The number of display terminals 20 to be used for the distributed processing is determined according to the resolution of the display terminals 20. Therefore, in this process, by executing a distributed rendering process that allows the number of display terminals to be used to be changed, it is possible to effectively display images for different purposes.

[0157] In the process according to the embodiment described above, the display terminal 20 distributes the display video data D Ｉ２ Therefore, in this process, by performing distributed rendering that allows the final display resolution to be changed, it is possible to effectively display images for different purposes.

[0158] In the process according to the embodiment described above, the editing terminal 10 performs three-dimensional rendering as editing rendering, and renders the editing video data D Ｉ１ Therefore, in this process, the editing video data D Ｉ１ By generating three-dimensional image data as above, images for different purposes can be effectively displayed.

[0159] In the process according to the embodiment described above, the editing terminal 10 １ Then, model transformation is performed to convert from the local coordinate system to the world coordinate system, projection transformation is performed to convert from the world coordinate system to the screen coordinate system, and a rasterizer is executed to render on the screen, thereby generating three-dimensional video data. Therefore, in this process, by executing model transformation, projection transformation, and a rasterizer as three-dimensional rendering, it is possible to effectively display videos for different purposes.

[0160] In the process according to the embodiment described above, the display terminal 20 performs two-dimensional rendering as display rendering, and generates display video data D Ｉ２ Therefore, in this process, the display image data D Ｉ２ By generating two-dimensional image data as above, images for different purposes can be effectively displayed.

[0161] In the process according to the embodiment described above, the display terminal 20 displays the display data D ２ Then, a direct rasterizer is executed to perform point rendering on the screen to generate two-dimensional video data. Therefore, in this process, by executing the direct rasterizer as two-dimensional rendering, it is possible to effectively display videos for different purposes.

[0162] In the process according to the embodiment described above, the editing data D １ and display video data D Ｉ２ are created in the MPI, MCI, MSI, or 3DGS image format. Therefore, in this process, by using the MPI, MCI, MSI, or 3DGS image format, images for different purposes can be effectively displayed.

[0163] In the process according to the embodiment described above, the editing data D １ and display video data D Ｉ２ are created in an image format that allows point rendering to be performed. Therefore, in this process, by using an image format that allows point rendering to be performed, it is possible to effectively display images for different purposes.

[0164] In the process according to the embodiment described above, the video display system 100 further includes a first display screen such as a monitor screen of the editing terminal 10, and displays the editing video data D Ｉ１ Therefore, in this process, by using the display screen that displays the editing video I1, it is possible to effectively display videos for different purposes.

[0165] In the process according to the embodiment described above, the video display system 100 further includes a second display screen such as the display screen 30, and the display video data D Ｉ２ Therefore, in this process, by using the display screen that displays the display image I2, images for different purposes can be effectively displayed.

[0166] In the process according to the embodiment described above, the video display system 100 further includes a camera that captures an image of a subject against the background of a display image I2 displayed on a second display screen such as the display screen 30. The display terminal 20 generates display data D ２ Rendering for display is performed on the image data D Ｉ２ Therefore, in this process, it is possible to effectively display images for different purposes and to generate display image data D according to the viewpoint from which the image capturing device captures the subject. Ｉ２ can be generated.

[0167] 3. Hardware Configuration The information processing devices such as the editing terminal 10 and the display terminal 20 according to the above-described embodiments are realized by a computer 1000 having a configuration such as that shown in FIG. 22 . The editing terminal 10 and the display terminal 20 according to the embodiments will be described below as examples. FIG. 22 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the editing terminal 10 and the display terminal 20 according to the embodiments. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0168] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0169] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0170] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0171] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0172] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a touch panel, keyboard, mouse, microphone, and camera via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, and printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.

[0173] For example, when computer 1000 functions as editing terminal 10 and display terminal 20 according to the embodiment, CPU 1100 of computer 1000 executes an information processing program loaded onto RAM 1200 to realize the functions of first control unit 15, second control unit 23, etc. Also, HDD 1400 stores the information processing program according to the present disclosure and data in first storage unit 14, second storage unit 22, etc. Note that CPU 1100 reads and executes program data 1450 from HDD 1400, but as another example, these programs may be obtained from another device via external network 1550.

[0174] The present technology may also be configured as follows: (1) An information processing system including a first information processing device and a second information processing device, wherein the first information processing device comprises: a first control unit that acquires first data created in a predetermined image format, performs first rendering on the first data, and generates first video data for editing or previewing, and, when the first video data is edited, transmits editing information related to the editing of the first data to the second information processing device; and a second control unit that acquires second data identical to the first data, receives the editing information from the first information processing device, and performs second rendering different from the first rendering on the second data reflecting the editing based on the received editing information, and generates second video data for provision. (2) The second information processing device includes a plurality of information processing devices for distributed processing, and each of the plurality of information processing devices for distributed processing generates a portion of the second video data corresponding to an area allocated to it among a plurality of display areas using the second data reflecting the editing. (3) The information processing system according to (2), wherein the first information processing device determines the number of the plurality of information processing devices for distributed processing to be used by the second information processing device according to a resolution of the second video data. (4) The information processing system according to (2), wherein the second information processing device determines the resolution of the second video data according to the number of the plurality of information processing devices for distributed processing. (5) The information processing system according to any one of (1) to (4), wherein the first information processing device performs three-dimensional rendering as the first rendering and generates three-dimensional video data as the first video data.(6) The information processing system according to (5), wherein the first information processing device performs model transformation on the first data to convert it from a local coordinate system to a world coordinate system, performs projection transformation on the first data to convert it from the world coordinate system to a screen coordinate system, and executes a rasterizer that renders it on a screen, thereby generating the three-dimensional video data. (7) The information processing system according to any one of (1) to (6), wherein the second information processing device performs two-dimensional rendering as the second rendering and generates two-dimensional video data as the second video data. (8) The information processing system according to (7), wherein the second information processing device executes a direct rasterizer that performs point rendering on a screen on the second data that reflects editing, thereby generating the two-dimensional video data. (9) The information processing system according to any one of (1) to (8), wherein the first data and the second data are created in an image format of MPI (Multi-Plane-Image), MCI (Multi-Cylinder-Image), MSI (Multi-Sphere-Image), or 3DGS (3D Gaussian Splatting). (10) The information processing system according to any one of (1) to (9), wherein the first data and the second data are created in an image format in which point rendering is executable. (11) The information processing system according to any one of (1) to (10), further comprising a first display screen, wherein the first display screen displays a first image based on the first image data. (12) The information processing system according to any one of (1) to (11), further comprising a second display screen, wherein the second display screen displays a second image based on the second image data. (13) The information processing system described in (12) further includes a camera that captures an image of a subject with the second image displayed on the second display screen as a background, and the second information processing device performs the second rendering on the second data according to the viewpoint from which the camera captures the subject, thereby generating the second image data.(14) An information processing method executed by an information processing system including a first information processing device and a second information processing device, wherein the first information processing device acquires first data created in a predetermined image format, performs first rendering on the first data, and generates first video data for editing or previewing, and when the first video data is edited, transmits editing information regarding the editing of the first data to the second information processing device, and the second information processing device acquires second data identical to the first data, receives the editing information from the first information processing device, and performs second rendering different from the first rendering on the second data reflecting the editing based on the received editing information, and generates second video data for provision. (15) An information processing program that causes a first information processing device to execute the following process: acquire first data created in a predetermined image format, perform first rendering on the first data, generate first video data for editing or preview, and, if the first video data is edited, transmit editing information regarding the editing of the first data to a second information processing device; and cause the second information processing device to execute the following process: acquire second data identical to the first data, receive the editing information from the first information processing device, perform second rendering different from the first rendering on the second data reflecting the editing based on the received editing information, and generate second video data for provision.(16) A computer-readable non-transitory recording medium on which second video data is recorded, the second video data being generated by an information processing method in which a first information processing device acquires first data created in a predetermined image format, performs first rendering on the first data, and generates first video data for editing or previewing, and when the first video data is edited, transmits editing information regarding the editing of the first data to a second information processing device, and the second information processing device acquires second data identical to the first data, receives the editing information from the first information processing device, and performs second rendering different from the first rendering on the second data that reflects the editing based on the received editing information, thereby generating the second video data for provision. (17) A method for manufacturing a computer-readable non-transitory recording medium storing video content, comprising: performing first rendering on first data created in a predetermined image format by a first information processing device to generate first video data for editing or preview; displaying a first video using the generated first video data; when the displayed first video is edited, transmitting editing information regarding the editing of the first data to a second information processing device; performing second rendering, which is different from the first rendering, by the second information processing device on second data that reflects the editing of the first data based on the editing information to generate second video data for provision; displaying the second video using the generated second video data; and recording three-dimensional model data based on the second data on a computer-readable non-transitory recording medium.(18) A manufacturing method for a video display system that includes a three-dimensional model database having a computer-readable non-transitory recording medium and generates an arbitrary viewpoint video based on data recorded on the non-transitory recording medium, the manufacturing method comprising: performing first rendering on first data created in a predetermined image format by a first information processing device to generate first video data for editing or preview; displaying the first video using the generated first video data; when the displayed first video is edited, transmitting editing information regarding the editing of the first data to a second information processing device; performing second rendering, which is different from the first rendering, by the second information processing device on second data that reflects the editing of the first data based on the editing information to generate second video data for provision; displaying the second video using the generated second video data; and recording three-dimensional model data based on the second data on the non-transitory recording medium.

[0175] REFERENCE SIGNS LIST 10 Editing terminal 11 Input unit 12 Output unit 13 First communication unit 14 First storage unit 14a Editing video data storage unit 14b First editing information storage unit 15 First control unit 15a First acquisition unit 15b First generation unit 15c First display unit 15d Editing unit 20 Display terminal 21 Second communication unit 22 Second storage unit 22a Second editing information storage unit 22b Display video data storage unit 23 Second control unit 23a Second acquisition unit 23b Receiving unit 23c Second generation unit 23d Second display unit 30 Display screen 40 Three-dimensional model data database 100 Video display system

Claims

1. An information processing system comprising a first information processing device and a second information processing device, wherein the first information processing device: acquires first data created in a predetermined image format; performs first rendering on the first data to generate first video data for editing or preview; and, when the first video data is edited, comprises a first control unit: transmits editing information regarding the editing of the first data to the second information processing device; and the second information processing device: acquires second data identical to the first data; receives the editing information from the first information processing device; and performs second rendering different from the first rendering on the second data that reflects the editing based on the received editing information to generate second video data for provision.

2. The information processing system of claim 1, wherein the second information processing device includes a plurality of information processing devices for distributed processing, and each of the plurality of information processing devices for distributed processing uses the second data reflecting the editing to generate each portion of the second video data corresponding to an area assigned to it among a plurality of display areas.

3. The information processing system according to claim 2, wherein the first information processing device determines the number of the plurality of information processing devices for distributed processing to be used by the second information processing device according to the resolution of the second video data.

4. The information processing system according to claim 2, wherein the second information processing device determines the resolution of the second video data in accordance with the number of the plurality of information processing devices for distributed processing.

5. The information processing system according to claim 1, wherein the first information processing device performs three-dimensional rendering as the first rendering and generates three-dimensional video data as the first video data.

6. The information processing system according to claim 5, wherein the first information processing device generates the three-dimensional video data by executing a model transformation on the first data to convert it from a local coordinate system to a world coordinate system, a projection transformation to convert it from the world coordinate system to a screen coordinate system, and a rasterizer to render it on a screen.

7. The information processing system according to claim 1, wherein the second information processing device performs two-dimensional rendering as the second rendering and generates two-dimensional video data as the second video data.

8. The information processing system according to claim 7, wherein the second information processing device generates the two-dimensional video data by executing a direct rasterizer that performs point rendering on a screen on the second data that reflects the editing.

9. The information processing system according to claim 1, wherein the first data and the second data are created in an image format of MPI (Multi-Plane-Image), MCI (Multi-Cylinder-Image), MSI (Multi-Sphere-Image), or 3DGS (3D Gaussian Splatting).

10. The information processing system according to claim 1, wherein the first data and the second data are created in an image format in which point rendering is possible.

11. The information processing system according to claim 1, further comprising a first display screen, wherein the first display screen displays a first image based on the first image data.

12. The information processing system according to claim 1, further comprising a second display screen, wherein the second display screen displays a second image based on the second image data.

13. The information processing system of claim 12, further comprising a camera that captures an image of a subject with the second image displayed on the second display screen as a background, wherein the second information processing device performs the second rendering on the second data according to the viewpoint from which the camera captures the subject, and generates the second image data.

14. An information processing method executed by an information processing system including a first information processing device and a second information processing device, wherein the first information processing device: acquires first data created in a predetermined image format; performs first rendering on the first data to generate first video data for editing or preview; and, if the first video data is edited, transmits editing information regarding the editing of the first data to the second information processing device; and the second information processing device: acquires second data identical to the first data; receives the editing information from the first information processing device; and performs second rendering different from the first rendering on the second data that reflects the editing based on the received editing information to generate second video data for provision.

15. An information processing program that causes a first information processing device to execute the following processes: acquire first data created in a predetermined image format, perform first rendering on the first data, generate first video data for editing or preview, and, if the first video data is edited, send editing information regarding the editing of the first data to a second information processing device; and cause the second information processing device to execute the following processes: acquire second data identical to the first data, receive the editing information from the first information processing device, perform second rendering different from the first rendering on the second data that reflects the editing based on the received editing information, and generate second video data for provision.

Images