Information processing device, information processing method and program

Abstract

To solve the problem of being unclear whether a virtual viewpoint content that can be generated is high definition.SOLUTION: A clip list display part 244 acquires information showing a time code corresponding to a virtual viewpoint image generated on the basis of first three-dimensional shape data showing a three-dimensional shape of a subject, and a clip list display control part 247 notifies a user of a fact that generation processing of second three-dimensional shape data which shows the three-dimensional shape of the subject, is second three-dimensional shape data generated by taking a processing time longer than the first three-dimensional shape data, and corresponds to the time code is completed.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a technique for generating virtual viewpoint images.

Background Art

[0002] Recently, a technique has been attracting attention in which a plurality of cameras are installed at different positions to perform synchronous shooting from a plurality of viewpoints, and an image (virtual viewpoint image) from an arbitrary virtual camera (virtual viewpoint) is generated using the plurality of viewpoint images obtained by the shooting. Information on the position and orientation of the virtual camera is particularly referred to as camera work, and when the position and orientation change over time, it is a summary of them. According to such a technique, for example, it becomes possible to view highlight scenes of soccer or basketball from various angles, and it becomes possible to give the user a higher sense of presence compared to ordinary video content. In order to give a high sense of presence, immediacy of image reproduction and high definition of the reproduced image are important. However, being immediate and high definition are contradictory aspects, and they are used appropriately according to the application.

[0003] Patent Document 1 includes storage means for storing material data used for generating first virtual viewpoint content and material data used for generating second virtual viewpoint content of a type different from the first virtual viewpoint content. Further, a technique is disclosed in which information indicating at what time the first virtual viewpoint content can be generated and at what time the second virtual viewpoint content can be generated is output based on the stored information acquired by the acquisition means.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology described in Patent Document 1, it was not known whether the virtual viewpoint content that could be generated was high-definition or not.

[0006] Therefore, an object of the present disclosure is to enable easy determination of whether the virtual viewpoint content that can be generated is high-definition or not.

Means for Solving the Problem

[0007] To solve the above problems, the information processing apparatus of the present disclosure has the following configuration. That is, acquisition means for acquiring information indicating a time code corresponding to a virtual viewpoint image generated based on first three-dimensional shape data indicating the three-dimensional shape of a subject, and second three-dimensional shape data that indicates the three-dimensional shape of the subject and is generated with a processing time longer than that of the first three-dimensional shape data, and notification means for notifying that the generation process of the second three-dimensional shape data corresponding to the time code has been completed.

Effect of the Invention

[0008] According to the present disclosure, it is possible to easily determine whether the virtual viewpoint content that can be generated is high-definition or not.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

[0010] Hereinafter, embodiments of the present disclosure will be described in detail. Note that the components described in the following embodiments are examples of the forms of the present disclosure, and the scope of the present disclosure is not limited only to them.

[0011] <First Embodiment> FIG. 1 is a diagram showing an example of the configuration of an image processing system for creating a virtual viewpoint image according to the present disclosure.

[0012] The image processing system is a system that generates a virtual viewpoint image representing what is seen from a specified virtual viewpoint based on a plurality of captured images obtained by a plurality of imaging devices and a specified virtual viewpoint. The virtual viewpoint image in the present embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user. For example, an image corresponding to a viewpoint selected by the user from a plurality of candidates is also included in the virtual viewpoint image. In the present embodiment, the case where the virtual viewpoint is specified by a user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like. In the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

[0013] The viewpoint information used for generating a virtual viewpoint image is information indicating the position and orientation (line-of-sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including parameters representing the three-dimensional position of the virtual viewpoint and parameters representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter representing the size (angle of view) of the field of view of the virtual viewpoint. Also, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets respectively corresponding to a plurality of frames constituting a moving image of the virtual viewpoint image, and be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points.

[0014] The virtual viewpoint image is generated, for example, by the following method. First, a plurality of images (multi-viewpoint images) are acquired by imaging from different directions using a plurality of imaging devices. Next, a foreground image obtained by extracting a foreground region corresponding to a predetermined object such as a person or a ball and a background image obtained by extracting a background region other than the foreground region are acquired from the multi-viewpoint images. Also, a foreground model representing the three-dimensional shape of a predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring a background model representing the three-dimensional shape of a background such as a stadium is generated based on the background image. Then, the texture data is mapped to the foreground model and the background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information, thereby generating a virtual viewpoint image. However, the method for generating the virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating a virtual viewpoint image by projective transformation of an imaging image without using a three-dimensional model.

[0015] A foreground image is an image obtained by extracting an object area (foreground area) from a captured image captured by an imaging device. The object extracted as the foreground area refers to a dynamic object (moving body) that moves (its absolute position and shape can change) when imaging is performed from the same direction in time series. The object is, for example, in a competition, a person such as a player or a referee in the field where it is held, for example, a ball in a ball game, or a singer, a performer, a performer, a master of ceremonies, etc. in a concert or entertainment.

[0016] A background image is an image of an area (background area) that is at least different from the object that becomes the foreground. Specifically, the background image is an image in a state where the object that becomes the foreground is removed from the captured image. In addition, the background refers to an imaging object that is stationary or in a state close to stationary when imaging is performed from the same direction in time series. Such imaging objects are, for example, a stage for a concert or the like, a stadium for holding an event such as a competition, a structure such as a goal used in a ball game, a field, etc. However, the background is at least an area different from the object that becomes the foreground, and as an imaging object, other objects or the like may be included in addition to the object and the background.

[0017] A virtual camera is a virtual camera different from a plurality of imaging devices actually installed around an imaging area, and is a concept for conveniently explaining a virtual viewpoint related to the generation of a virtual viewpoint image. That is, the virtual viewpoint image can be regarded as an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. And the position and orientation of the viewpoint in the virtual imaging can be represented as the position and orientation of the virtual camera. In other words, it can be said that the virtual viewpoint image is an image that simulates the captured image obtained by the camera when it is assumed that a camera exists at the position of the virtual viewpoint set in the space. Also, in the present embodiment, the content of the transition of the virtual viewpoint over time is denoted as a virtual camera path. However, it is not essential to use the concept of a virtual camera to implement the configuration of the present embodiment. That is, at least information representing a specific position in the space and information representing the orientation are set, and a virtual viewpoint image may be generated according to the set information.

[0018] Reference numeral 100 denotes an arena, and a plurality of imaging processing units 110a to 110p for photographing from a plurality of directions are arranged so as to surround a field 101 where competitions are held. The imaging processing units 110a to 110p are connected to a three-dimensional data generation processing unit 130 by a network 120. The three-dimensional data generated by the three-dimensional data generation processing unit 130 is stored in a data storage unit 140. Also, since the imaging processing units 110a to 110p operate synchronously, the same time code is assigned to each imaging processing unit 110 for each photographing. As an acquisition means, an instruction of a user is acquired by a user operation processing unit 160 for providing a virtual viewpoint, data stored in the data storage unit 140 is read out by a rendering processing unit 150, and a virtual viewpoint image is generated. The generated virtual viewpoint image is displayed on a display device (not shown). Alternatively, the generated virtual viewpoint image is recorded on a recording medium (not shown). In the present embodiment, the arena 100 is described as an example of a photographing object, but the photographing object is not limited to this, and other places such as a gymnasium or a studio may be used. Note that in the present disclosure, it is assumed that the three-dimensional data generation processing unit 130, the data storage unit 140, the rendering processing unit 150, and the user operation processing unit 160 in the image processing system are each an independent information processing device, but it is not limited to this. One information processing device may include a plurality of processing units or storage units.

[0019] FIG. 2 is a functional block diagram showing an example of an image processing system according to the present disclosure.

[0020] A subject information extraction unit 211 extracts subject information from an image captured by the imaging processing unit 110. The subject information is composed of a time code, for example, a texture of the subject, and a mask image indicating a region of the subject. FIG. 4 shows an example of subject data extracted by the subject information extraction unit 211. FIG. 4(a) is an example of a subject texture, and FIG. 4(b) is an example of a subject mask image.

[0021] The three-dimensional data generation processing unit 130 includes a subject information extraction unit 211, a real-time three-dimensional data generation unit 212, and a high-definition three-dimensional data generation unit 213. Note that the three-dimensional data generated by the real-time three-dimensional data generation unit 212 and the high-definition three-dimensional data generation unit 213 are referred to as real-time three-dimensional data and high-definition three-dimensional data, respectively.

[0022] Based on the subject information extracted by the subject information extraction unit 211, the real-time three-dimensional data generation unit 212 generates, in real time, real-time three-dimensional data (first three-dimensional shape data) indicating the three-dimensional shape of the subject, for example, using a volume intersection method. As an application of the real-time three-dimensional data, for example, it is used for generating a virtual viewpoint image for considering camera work for creating a virtual viewpoint image clip described later, and it is important that the real-time three-dimensional data can be generated in real time. Here, the volume intersection method is an example, and other methods for generating real-time three-dimensional data may be used. Also, the real-time three-dimensional data includes information on the time code at the time of shooting.

[0023] The high-precision three-dimensional data generation unit 213 generates high-precision three-dimensional data (second three-dimensional shape data) indicating the three-dimensional shape of the subject based on the subject information stored in the subject information storage unit 221, for example, using the voxel coloring method in addition to the volume intersection method. As the use of the high-precision three-dimensional data, which is the second three-dimensional data, for example, in order to generate a virtual viewpoint image used for television broadcasting and generate a high-precision virtual viewpoint image, it is important to generate high-precision three-dimensional data. Comparing the generation process of high-precision three-dimensional data with the generation process of real-time three-dimensional data, since the voxel coloring method is additionally used, the shape accuracy becomes better than that of real-time three-dimensional data. Also, the processing time becomes longer than the generation of real-time three-dimensional data. Here, the volume intersection method and the voxel coloring method are just examples, and other methods for generating three-dimensional data may be used. For example, techniques such as machine learning that require high processing power and time may be used. Also, the high-precision three-dimensional data also includes the time code information at the time of shooting. In the present embodiment, the process of generating high-precision three-dimensional data compared with real-time three-dimensional data is called high-precision refinement, but it is not limited to this. For example, performing correction processing on real-time three-dimensional data may be regarded as high-precision refinement.

[0024] The subject information storage unit 221 stores the subject information extracted by the subject information extraction unit 211.

[0025] The three-dimensional data storage unit 222 stores the respective three-dimensional model data generated by the real-time three-dimensional data generation unit 212 and the high-definition three-dimensional data generation unit 213. Fig. 5 shows an image of the stored three-dimensional model data configuration. In Fig. 5, it shows that the three-dimensional model data is stored from 18:00:00.00 (18 hours 0 minutes 0 seconds 0 frames) to 20:00:00.00 (20 hours 0 minutes 0 seconds 0 frames). Two types of examples of storing the three-dimensional model data are shown below. Fig. 5(a) shows that the real-time three-dimensional model data 501 from 18:00:00.00 to 20:00:00.00 and the high-definition three-dimensional model data 502 from 18:00:00.00 to 18:30:00.00 are stored. Instead of storing the real-time three-dimensional model data and the high-definition three-dimensional model data separately, it is also possible to overwrite and store part or all of the real-time three-dimensional model data with the high-definition three-dimensional model data. In the example of Fig. 5(b), it shows that it is composed of the high-definition three-dimensional model data 503 from 18:00:00.00 to 18:30:00.00 and the real-time three-dimensional model data 504 from 18:30:00.00 to 20:00:00.00.

[0026] Fig. 12 shows an example of the data structure for realizing these. It is desirable to manage a plurality of three-dimensional model data by events, shooting locations, times, etc. In this embodiment, a group of these three-dimensional model data is managed by a three-dimensional model data group (3D_model_Data_Group). There is a header (3D_model_Data_Group_Header) at the beginning of the 3D_model_Data_Group, and a uniquely determined identification code (3D_model_Data_Group_ID) for each time period and event is entered therein. Subsequently, the number (number_of_3D_model_data_set) of the three-dimensional model data sets included in the three-dimensional model data group is described. Here, let its value be n.

[0027] Subsequently, n three-dimensional model data are stored. The three-dimensional model data set (3D_model_data_set) manages, for example, the three-dimensional model data represented in Fig. 5(a) as one three-dimensional model data set. The 3D_model_data_set includes a uniquely determined identification code (3D_model_data_set_ID) for identifying the three-dimensional model data set. Further, information regarding the first three-dimensional model data, which is real-time three-dimensional model data included in the three-dimensional model data set, follows. First, a flag (1 st _3D_model_data_flag) for determining whether the first three-dimensional model data exists is stored. For example, if the value of 1 st _3D_model_data_flag is 1, the first three-dimensional model data exists; if it is 0, it does not exist. When this value is 1, subsequently, the start time code (1 st _3D_model_data_start_time_code) of the real-time three-dimensional model data is stored. Further subsequently, the time code (1 st _3D_model_data_end_time_code) of the final frame is stored. In Fig. 5(a), 18:00:00.00 is entered in 1 st _3D_model_data_start_time_code, and 20:00:00.00 is entered in 1 st _3D_model_data_end_time_code. Subsequently, the size (data_size_of_1 st _3D_model_data) of this real-time three-dimensional model data is entered. This value can be used for reading the three-dimensional data, etc. Further, a pointer (Pointer_of_1 st _3D_model_data) indicating the start of the area (3D_model_data) where the real-time three-dimensional model data is actually stored is stored. The pointer may be an address of a recording medium or memory, a URL, etc. The recorded area 3D_model_data includes the number of frames of the three-dimensional model data. Thereafter, the start time code (1st From the frame of the (_3D_model_data_start_time_code), the three-dimensional model data (3D_mdel_data) is stored. First, a uniquely determined start code (frame_start_code) representing the frame boundary is stored. Subsequently, the time code (time_code) of that frame is stored, followed by the number of subjects (number_of_models) included in that frame, and IDs (model_ID) for identifying each subject are recorded. Next, a pointer (Pointer_of_model_data) indicating the area where the three-dimensional model data is actually stored is stored. The model_ID and Pointer_of_model_data are repeatedly stored in order for the number of models included in the frame. Further, the three-dimensional model data separated by the frame_start_code for the number of frames follows. If 1 st _3D_model_data_flag has a value of 0, then 1 st _3D_model_data_start_time_code to Pointer_of_1 st _3D_model_data does not exist.

[0028] Return to the three-dimensional model data set (3D_model_data_set). Pointer_of_1 st Following Pointer_of_1 nd _3D_model_data, information regarding the second three-dimensional model data, which is high-definition three-dimensional model data included in the three-dimensional model data set, follows. First, a flag (2 nd _3D_model_data_flag) for determining whether the second three-dimensional model data exists is stored. If this value is 1, then hereafter, similar to the first three-dimensional model data, the start time code (2 nd _3D_model_data_start_time_code) of the high-definition three-dimensional model data is stored. Subsequently, the time code (2 nd18:00:00.00 is entered into _3D_model_data_start_time_code, and 2 nd 18:30:00.00 is entered into _3D_model_data_end_time_code. Furthermore, the size of the high-definition three-dimensional model data (data_size_of_2 nd _3D_model_data) is entered. Subsequently, a pointer (Pointer_of_2 nd _3D_model_data) indicating the start of the area (3D_model_data) where the high-definition three-dimensional model data is stored is stored. The configuration of the three-dimensional model data (3D_mdel_data) is the same as that of the first three-dimensional model data.

[0029] In Fig. 5(b), 1 st 18:30:00.00 is entered into _3D_model_data_start_time_code, and 1 st 20:00:00.00 is entered into _3D_model_data_end_time_code. 2 nd 18:00:00.00 is entered into _3D_model_data_start_time_code, and 2 nd 18:30:00.00 is entered into _3D_model_data_end_time_code.

[0030] Based on the camera work information acquired from the user operation processing unit 160, the rendering processing unit 150 reads the subject information stored in the subject information storage unit 221 and the three-dimensional model data stored in the three-dimensional data storage unit 222. Subsequently, rendering processing is performed to generate a virtual viewpoint image.

[0031] The user operation unit 241 is an interface device composed of a keyboard, mouse, joystick, game controller, etc. It operates the time code and virtual camera position / attitude of the virtual viewpoint image, and acquires the time code and the position and attitude of the virtual camera.

[0032] The camera work generation unit 242 generates camera work information composed of time code and virtual camera position / attitude information based on the operations of the user operation unit 241. The camera work information is not limited to information indicating the trajectory of the positions of virtual cameras corresponding to a plurality of consecutive time codes. For example, it may be the time code at a certain time and information indicating the position and attitude of the virtual camera corresponding to that time code. The rendering processing unit 150 reads the data of the time code from the data storage unit 140 based on this camera work information, performs rendering processing on the image seen from the virtual camera position / attitude, and generates a virtual viewpoint image.

[0033] The camera work output unit 243 outputs the camera work information generated by the camera work generation unit 242 and the information of the clips registered in the clip list display unit 244 to the rendering processing unit 150.

[0034] The clip list display unit 244 is created by user operations and displays a list of a certain set of camera work (camera path) called a clip. The clip in the present disclosure refers to the virtual viewpoint video corresponding to the camera work information obtained from the camera work generation unit 242. The clips on the clip list can also be played by user operations. An example of the display of the clip list is shown in FIG. 6.

[0035] The time code acquisition unit 245 acquires the time code of the high-definition three-dimensional data among the three-dimensional data stored in the three-dimensional data storage unit 222.

[0036] The time code comparison unit 246 compares the time code of the clips registered in the clip list display unit 244 with the time code acquired by the time code acquisition unit 245.

[0037] If the three-dimensional data corresponding to the time code of the clip is high-definition, the clip list display control unit 247 displays information indicating that the high-definition process has been completed. Specifically, as a result of the comparison by the time code comparison unit 246, if the time code of the clip registered in the clip list display unit 244 is included in the time code acquired by the time code comparison unit 246, information indicating that the high-definition process has been completed is displayed. On the other hand, if the high-definition process has not been completed, the clip list display control unit 247 can also display information indicating that the high-definition process is incomplete.

[0038] The hardware configuration of the three-dimensional data generation processing unit 130 will be described with reference to FIG. 3. The three-dimensional data generation processing unit 130 includes a CPU 311, a ROM 312, a RAM 313, an auxiliary storage unit 314, a display interface 315, an input interface 316, a communication unit 317, and a bus 318. Note that the hardware configuration of the user operation processing unit 160 is the same as that of the three-dimensional data generation processing unit 130 described below.

[0039] The CPU 311 realizes each function of the three-dimensional data generation processing unit 130 using computer programs and data stored in the ROM 312 and the RAM 313. Note that the three-dimensional data generation processing unit 130 may have one or more dedicated hardware different from the CPU 311, and at least a part of the processing by the CPU 311 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor). The ROM 312 stores programs that do not require modification. The RAM 313 temporarily stores programs and data supplied from the auxiliary storage unit 314 and data supplied from the outside via the communication unit 317. The auxiliary storage unit 314 is configured by, for example, a hard disk drive or the like, and stores various types of data such as image data and audio data.

[0040] The display interface 315 is an interface for, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) for user operation, a virtual viewpoint image, and the like. The input interface 316 connects devices for inputting operations by the user, such as button switches, keyboards, mice, joysticks, touch panels, etc.

[0041] The communication unit 317 is used for communication with devices external to the three-dimensional data generation processing unit 130. For example, when connected to an external device by wire, a communication cable is connected to the communication unit 317. When having a function of wireless communication with an external device, the communication unit 317 includes an antenna. In the user operation processing unit 160 of the present embodiment, the user operation unit 241 is connected to the input interface 316, and the clip list display unit 244 is connected to the display interface 315. The bus 318 connects each part of the three-dimensional data generation processing unit 130 and transmits information.

[0042] FIG. 6 is a diagram showing an example of clip list display. The display unit 601 shows information for identifying a clip, such as the name of the clip or a unique number, and whether the clip is high definition. The "HQ" display in 601a indicates that the high definition process has been completed. There is no "HQ" display in 601b and 601c, indicating that the high definition process has not been completed. 602 is the title of the clip. 603 is the time and moment targeted by the clip. Also, a display of "RT" representing real time, for example, may be used to indicate that the high definition process has not been completed.

[0043] With reference to FIG. 7, the operation flow and sequence of the present image processing system will be described.

[0044] Operation 700 shows the processing flow by the user operation processing unit 160. Real-time image generation 710 shows the generation flow of a virtual viewpoint image using the real-time three-dimensional model data generated by the real-time three-dimensional data generation unit 212. High-definition image generation 720 shows the virtual viewpoint image generation flow using the high-definition three-dimensional data generated by the high-definition three-dimensional data generation unit 213.

[0045] In step S701, using the information acquired by the operation of the user operation unit 241, the camera work information generated by the camera work generation unit 242 is output to the rendering processing unit 150 via the camera work output unit 243.

[0046] In step S702, based on the camera work information generated by the camera work generation unit 242 by the operation of the user operation unit 241, a clip is created and registered and displayed on the clip list display unit 244. When generating a virtual viewpoint video based on the clip, the camera work information of the clip is output to the rendering processing unit 150 via the camera work output unit 243.

[0047] In step S703, the time code acquisition unit 245 acquires the time code of the high-definition three-dimensional model data from which the high-definition three-dimensional data is generated in the three-dimensional data storage unit 222.

[0048] In step S704, the time code comparison unit 246 compares the time code of the clip registered in the clip list display unit 244 with the time code obtained in step S703. Taking the cases shown in FIGS. 5 and 6 as an example, the time codes of the clips registered in the clip list display unit 244 are · 18:00:17.06 to 18:00:17.38 · 19:05:19.43 to 19:06:22.55 · 19:29:25.35 to 19:30:32.17 as follows. On the other hand, the time code obtained in step S703 is · 18:00:00.00 to 18:30:00.00 It is.

[0049] In step S705, the following processing is performed based on the comparison result in the time code comparison unit 246. When the time code of the clip registered in the clip list display unit 244 is included in the time code acquired by the time code comparison unit 246, the clip list display control unit 247 displays information indicating that the high-definition refinement is completed. In other words, if the three-dimensional data corresponding to the time code of the clip is high-definition, information indicating that the high-definition refinement is completed is displayed. Taking the cases shown in FIGS. 5 and 6 as an example, the time code · 18:00:17.06 to 18:00:17.38 is found to have completed high-definition refinement. Therefore, as shown in 601a, information indicating that the high-definition refinement is completed is displayed by displaying "HQ".

[0050] In step S706, the refined clips registered in the clip list display unit 244 are played back. When playing back a clip, the camera work information of the clip is output to the rendering processing unit 150 via the camera work output unit 243.

[0051] In step S711, the real-time three-dimensional data generation unit 212 generates real-time three-dimensional model data.

[0052] In step S712, based on the camera work information output from the user operation processing unit 160, the rendering processing unit 150 generates a virtual viewpoint image.

[0053] In step S721, the high-definition three-dimensional data generation unit 213 generates high-definition three-dimensional model data. Based on the instruction for time code acquisition by the time code acquisition unit 245, the time code of the three-dimensional data for which the high-definition three-dimensional data has been generated is output.

[0054] In step S722, based on the camera work information output from the user operation processing unit 160, the rendering processing unit 150 generates a high-definition virtual viewpoint image.

[0055] As described above, according to the image processing system according to the present disclosure, it is possible to know whether the generated virtual viewpoint image clip is high-definition. This information leads to the recognition of the image quality of the generated virtual viewpoint image and the reduction of the man-hour for re-checking the image quality.

[0056] In the present embodiment, the three-dimensional model data is managed in units of frames, but it is not limited thereto, and it may be managed in units of three-dimensional models. For example, the three-dimensional model data with the same model_ID may be managed by time code.

[0057] In the present embodiment, the high-definition refinement of the three-dimensional model has been mentioned, but it is not limited thereto, and processing with a large processing load such as high-definition refinement of the captured image used for rendering or color reproduction using transparency may be performed.

[0058] Also, in the present embodiment, the information indicating that the high-definition refinement has been completed is displayed, but it is not limited thereto, and the clip list display control unit 247 may display only the information indicating that the high-definition refinement is not completed. Further, when the high-definition refinement process is being executed, information indicating that it is in the process may be displayed.

[0059] Also, in the present embodiment, the user is notified that the high-definition refinement process has been completed by displaying the information indicating that the high-definition refinement has been completed in the clip list display control unit, but it is not limited thereto. For example, the information indicating that the high-definition refinement process has been completed may be output to an external device for notification.

[0060] <Second Embodiment> In the above-described first embodiment, the case where the high-definition three-dimensional data generation process is performed in parallel with the real-time three-dimensional data generation in the three-dimensional data generation processing unit 130 has been described. However, the present invention is not limited to this, and it is also possible to select a clip to be refined by a user operation and perform the high-definition three-dimensional data generation process.

[0061] FIG. 8 is a functional block diagram showing an example of an image processing system according to a second embodiment of the present disclosure. The same functional blocks as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

[0062] The high-definition instruction unit 801 enables the user to give a high-definition instruction to a clip unit on the clip list or to a plurality of clips.

[0063] The time code instruction unit 802 acquires the time code corresponding to the clip instructed by the high-definition instruction unit 801 and instructs the high-definition three-dimensional data generation unit 213 to generate high-definition three-dimensional data.

[0064] FIG. 9(a) shows an example of a user interface of the high-definition instruction unit 801. The same ones as the clip list shown in FIG. 6 are denoted by the same reference numerals, and the description thereof is omitted.

[0065] The display unit 901 is a button for instructing high-definition, which is a second three-dimensional data generation instruction means, and is also a display unit for indicating the state of high-definition. 901c is a button for instructing high-definition corresponding to a clip for which high-definition has not been completed. By pressing this button, the high-definition of the three-dimensional data of the time code portion corresponding to the clip is instructed to the high-definition three-dimensional data generation unit 213. 901b shows an example of a state in which the high-definition process is being performed. In this example, it represents that 50% of the entire clip has been refined. 901a shows an example of a state in which the button of the clip for which high-definition has already been completed is displayed in an inactive form.

[0066] As described above, according to this embodiment, the user can specify a clip to be refined and perform the refinement, and can also know the state thereof.

[0067] In the example of the user interface shown in FIG. 9(a), an example is shown in which a display unit 601 indicating that the entire clip has been refined and a display unit 901 for instructing the refinement are separated. In the present disclosure, it is not limited to this, and as shown in FIG. 9(b), the user interface for instructing the refinement and the user interface indicating that the refinement has been performed may be configured by the same part, and an example thereof is shown at 911. 911a indicates a state where the refinement has been completed, 911b indicates a refinement ratio which is the progress of the refinement, and 911c indicates a button for instructing the refinement.

[0068] As described above, according to the image processing system according to the present disclosure, a clip to be refined can be selected by a user operation, and a high-precision three-dimensional data generation process can be performed.

[0069] Note that the information indicating the state is not limited to the above. For example, when the refinement process is not performed, information indicating that it is unprocessed may be displayed. For example, when indicating a state where the refinement process is being performed, information indicating that it is being processed may be displayed without displaying the ratio of the process.

[0070] <Third Embodiment> In the above-described second embodiment, the instruction for high-definition refinement is given in units of clips. However, by specifying the time code, it is also possible to instruct the high-definition refinement of three-dimensional data. Fig. 10 shows another example of the user interface of the high-definition refinement instruction unit 801. 1001 is the time code input field, where the time code of the three-dimensional data to be refined is specified, and by pressing the execution button 1002, the high-definition refinement of the three-dimensional data corresponding to the time code is instructed. For example, a time code indicating a first time for defining a predetermined time and a time code indicating a second time may be determined based on a user operation, and the predetermined time may be treated as a clip. In that case, the user operation unit 241 determines a time code indicating a first time and a time code indicating a second time based on a user operation, and outputs them to the clip list display unit 244. When the clip list display unit 244 separately obtains an instruction to perform a high-definition processing on the acquired clip, the clip list display unit 244 outputs information indicating the time code of the clip to the time code comparison unit 246. The subsequent processing is the same as that of the first embodiment.

[0071] As described above, according to the image processing system according to the present disclosure, it is possible to specify a time code to be refined by a user operation and perform a high-definition three-dimensional data generation process.

[0072] <Other Embodiments> In the description of the above embodiment, the configuration in which the user operation processing unit 160, the three-dimensional data generation processing unit 130, the data storage unit 140, and the rendering processing unit 150 are directly connected has been described. However, the present disclosure is not limited to this. As shown in Fig. 11, the user operation processing unit 160 may be connected to the virtual viewpoint image generation control unit 1110, and the virtual viewpoint image generation control unit 1110, the three-dimensional data generation processing unit 130, the data storage unit 140, and the rendering processing unit 150 may be connected.

[0073] Furthermore, in the description of the above embodiment, only whether it is high-definition or not has been considered. However, the present disclosure is not limited to this, and further high-definition refinement may be performed. In this case, it is possible to cope by increasing the number of information on the stages of high-definition refinement and the 3D_model_data_flag.

[0074] Furthermore, the object of the present disclosure can also be achieved by supplying a storage medium storing the code of a computer program that realizes the above-described functions to a system, and the system reads and executes the code of the computer program. In this case, the code of the computer program read from the storage medium itself realizes the functions of the above-described embodiments, and the storage medium storing the code of the computer program constitutes the present disclosure. Also included is a case where, based on the instruction of the program code, an operating system (OS) or the like running on the computer performs part or all of the actual processing, and the above-described functions are realized by that processing.

[0075] Furthermore, it may be realized in the following form. That is, the computer program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Also included is a case where, based on the instruction of the code of the computer program, a CPU or the like provided in the function expansion card or the function expansion unit performs part or all of the actual processing, and the above-described functions are realized.

[0076] When the present disclosure is applied to the above storage medium, the storage medium stores the code of a computer program corresponding to the flowchart described above.

[0077] As described above, the present disclosure has been described based on a plurality of embodiments, but the present disclosure is not limited to the above embodiments, and various modifications are possible based on the spirit of the present disclosure, and they are not excluded from the scope of the present disclosure.

[0078] Note that part or all of the control in this embodiment may be supplied to an image processing system or the like via a network or various storage media as a computer program that realizes the functions of the above-described embodiments. Then, a computer (or a CPU, MPU, etc.) in the image processing system or the like may read and execute the program. In that case, the program and the storage medium storing the program will constitute the present disclosure.

[0079] Note that the disclosure of this embodiment includes the following configurations, methods, and programs.

[0080] (Configuration 1) Acquisition means for acquiring information indicating a time code corresponding to a virtual viewpoint image generated based on first three-dimensional shape data indicating the three-dimensional shape of a subject; Notification means for indicating the three-dimensional shape of the subject, which is second three-dimensional shape data generated with a processing time longer than that of the first three-dimensional shape data, and notifying that the generation process of the second three-dimensional shape data corresponding to the time code has been completed; An information processing apparatus comprising the same.

[0081] (Configuration 2) The information processing apparatus according to Configuration 1, wherein the notification means notifies that it is unprocessed when the generation process has not been executed, and notifies that it is being processed when the generation process is being executed.

[0082] (Configuration 3) The information processing apparatus according to Configuration 1 or 2, wherein the notification means outputs information indicating that the generation process has been completed to an external device.

[0083] (Configuration 4) The information processing apparatus according to at least any one of Configurations 1 to 3, wherein the notification means performs control to display an image indicating that the generation process has been completed on a display unit.

[0084] Configuration 5: The information processing apparatus according to at least any one of Configurations 1 to 4, wherein the generation process of the second three-dimensional shape data is different from the generation process of the first three-dimensional shape data.

[0085] Configuration 6: The information processing apparatus according to at least any one of Configurations 1 to 5, wherein the second three-dimensional shape data has higher shape accuracy than the first three-dimensional shape data.

[0086] Configuration 7: The information processing apparatus according to at least any one of Configurations 1 to 6, wherein the second three-dimensional shape data and the first three-dimensional shape data are generated based on a plurality of captured images.

[0087] Configuration 8: The information processing apparatus according to at least any one of Configurations 1 to 6, wherein the virtual viewpoint image is generated based on a plurality of captured images.

[0088] Configuration 9: The information processing apparatus according to at least any one of Configurations 1 to 8, further comprising output means for outputting information indicating the time code to an apparatus that generates the second three-dimensional shape data.

[0089] Configuration 10: The acquisition means acquires information indicating a plurality of time codes corresponding to a plurality of the virtual viewpoint images, The notification means notifies that the generation processes of the plurality of second three-dimensional shape data corresponding to the plurality of time codes are completed. The information processing apparatus according to at least any one of Configurations 1 to 9.

[0090] Configuration 11: The information processing apparatus according to Configuration 10, wherein the acquisition means acquires a camera path including information indicating the plurality of time codes.

[0091] Method: An acquisition step of acquiring information indicating a time code corresponding to a virtual viewpoint image generated based on first three-dimensional shape data indicating the three-dimensional shape of a subject, It shows the three-dimensional shape of the subject, and is second three-dimensional shape data generated by applying a processing time longer than that of the first three-dimensional shape data, and a notification step of notifying that the generation process of the second three-dimensional shape data corresponding to the time code has been completed. An information processing method characterized by having the following.

[0092] (Program) A program for causing a computer to function as the information processing apparatus according to any one of Configurations 1 to 11.

Explanation of Signs

[0093] 130 Three-dimensional data generation processing unit 212 Real-time three-dimensional data generation unit 213 High-definition three-dimensional data generation unit 222 Three-dimensional data storage unit 245 Time code acquisition unit 246 Time code comparison unit

Claims

1. Acquisition means for acquiring information indicating a time code corresponding to a virtual viewpoint image generated based on first three-dimensional shape data indicating a three-dimensional shape of a subject; Second three-dimensional shape data that indicates the three-dimensional shape of the subject and is generated with a processing time longer than that of the first three-dimensional shape data, and notifies that the generation process of the second three-dimensional shape data corresponding to the time code has been completed; Notification means; An information processing apparatus comprising:

2. The information processing apparatus according to claim 1, wherein the notification means notifies that the generation process is unprocessed when the generation process has not been executed, and notifies that the generation process is in progress when the generation process is being executed.

3. The information processing apparatus according to claim 1, wherein the notification means outputs information indicating that the generation process has been completed to an external device.

4. The information processing apparatus according to claim 1, wherein the notification means performs control to display an image indicating that the generation process has been completed on a display unit.

5. The information processing apparatus according to claim 1, wherein the generation process of the second three-dimensional shape data is different from the generation process of the first three-dimensional shape data.

6. The information processing apparatus according to claim 1, wherein the second three-dimensional shape data has higher shape accuracy than the first three-dimensional shape data.

7. The information processing apparatus according to claim 1, wherein the second three-dimensional shape data and the first three-dimensional shape data are generated based on a plurality of captured images.

8. The information processing apparatus according to claim 1, wherein the virtual viewpoint image is generated based on a plurality of captured images.

9. Furthermore, the information processing apparatus according to claim 1, further comprising output means for outputting information indicating the time code to an apparatus that generates the second three-dimensional shape data.

10. The acquisition means acquires information indicating a plurality of time codes corresponding to the plurality of virtual viewpoint images, The notification means notifies that generation processing of the plurality of second three-dimensional shape data corresponding to the plurality of time codes has been completed, the information processing apparatus according to claim 1.

11. The acquisition means acquires a camera path including information indicating the plurality of time codes, the information processing apparatus according to claim 10.

12. An acquisition step of acquiring information indicating a time code corresponding to a virtual viewpoint image generated based on first three-dimensional shape data indicating a three-dimensional shape of a subject; A notification step of notifying that generation processing of second three-dimensional shape data, which indicates the three-dimensional shape of the subject and is generated with a processing time longer than that of the first three-dimensional shape data, corresponding to the time code has been completed; An information processing method, characterized by comprising:

13. A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 12.

Images