Information processing system, information processing method, and program

Abstract

To reduce the amount of data of recording used to estimate whether a three-dimensional shape of an object to be inspected is duplicated on the basis of a created virtual viewpoint image.SOLUTION: An information processing system acquires three-dimensional models of one or more objects at a designated time. The information processing system creates a virtual viewpoint image corresponding to the designated time from a virtual viewpoint according to a designated virtual viewpoint parameter by using the three-dimensional models of the one or more objects at the designated time. The information processing system outputs the created virtual viewpoint image. The information processing system determines whether a predetermined condition is satisfied, which is a condition related to at least one of the time and the virtual viewpoint parameter. The information processing system records information used for specification of the three-dimensional model of the object corresponding to the virtual viewpoint image in accordance with satisfaction of the predetermined condition.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and a program, and particularly relates to virtual viewpoint image technology.

Background Art

[0002] Techniques for generating a virtual viewpoint image of a subject from a specified virtual viewpoint have attracted attention. Such a virtual viewpoint image can be generated using a plurality of images obtained by imaging with a plurality of imaging devices, as shown in, for example, Patent Document 1. Such a virtual viewpoint image can be generated based on a three-dimensional model representing the three-dimensional shape of the subject. The three-dimensional model can be generated based on a plurality of images. Regarding a method for generating a three-dimensional model, for example, the method described in Non-Patent Document 1 is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] As described above, virtual viewpoint images from various virtual viewpoints can be generated based on the three-dimensional shape of the subject estimated from a plurality of images. On the other hand, a user of such a virtual viewpoint image generation device can generate virtual viewpoint images from a plurality of virtual viewpoints. By using the virtual viewpoint images thus generated from different virtual viewpoints, the three-dimensional shape of the subject can be estimated. In other words, a third party can obtain a copy of the data indicating the three-dimensional shape. However, for a business operator providing various virtual viewpoint image contents, it is not desirable to allow free replication of the data indicating the three-dimensional shape used for generating the virtual viewpoint images.

[0006] The present disclosure aims to reduce the data amount of records used for estimating whether a three-dimensional shape of an inspection target is replicated based on a generated virtual viewpoint image.

Means for Solving the Problem

[0007] An information processing system according to an embodiment includes the following configuration. That is, model acquisition means for acquiring a three-dimensional model of one or more objects at a specified time from storage means for storing a three-dimensional model indicating the three-dimensional shape of an object at each time; generation means for generating a virtual viewpoint image corresponding to the specified time from a virtual viewpoint according to specified virtual viewpoint parameters using the three-dimensional model of the one or more objects at the specified time; output means for outputting the virtual viewpoint image generated by the generation means; determination means for determining whether a predetermined condition, which is a condition related to at least one of the time and the virtual viewpoint parameters, is satisfied; recording means for recording information used for specifying the three-dimensional model of the object corresponding to the virtual viewpoint image generated by the generation means in response to the satisfaction of the predetermined condition; and

Effect of the Invention

[0008] It is possible to reduce the amount of data of the record used to estimate whether it is a copy based on the virtual viewpoint image in which the three-dimensional shape of the inspection target is generated.

Brief Description of Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] (Generation of Virtual Viewpoint Image) FIG. 1 is a configuration diagram of an image generation system 10, which is an information processing system according to an embodiment. This image generation system can generate a virtual viewpoint image. The image generation system 10 includes a camera group 110, a model generation device 120, an image generation device 100, and an information processing device 101. Note that an information processing system according to an embodiment does not necessarily have all of these devices. For example, an information processing system according to an embodiment has the image generation device 100 and the information processing device 101. Also, an information processing system according to an embodiment does not necessarily have all of the processing units described below. For example, an information processing system according to an embodiment has a model acquisition unit 135, an image generation unit 136, an output unit 137, a determination unit 142, and an information recording unit 143.

[0012] The camera group 110 includes a plurality of cameras (for example, cameras 110a to 110f). The camera group 110 images the subject 111 from different directions. Then, the camera group 110 inputs each image to the model generation device 120.

[0013] The model generation device 120 generates information representing the three-dimensional shape of an object. For example, the model generation device 120 can generate a three-dimensional model representing the three-dimensional shape of the subject 111. The model generation device 120 can generate a three-dimensional model using a plurality of images input by the camera group 110. The method for generating the three-dimensional model is not particularly limited. For example, the model generation device 120 can generate a three-dimensional model using the method described in Non-Patent Document 1. An example of a specific method for generating a three-dimensional model includes a visual volume intersection method. Also, the camera group 110 may image a plurality of subjects 111. In this case, the model generation device 120 can generate a plurality of three-dimensional models. Here, each of the plurality of three-dimensional models may correspond to a different subject 111. On the other hand, the three-dimensional model generation unit 121 may generate a three-dimensional model of the subject without using the captured images. For example, the three-dimensional model generation unit 121 can generate a three-dimensional model according to user input. In this case, for example, CAD or a CG tool can be used.

[0014] The model generation device 120 may generate information representing the three-dimensional shape of a subject that changes over time. For example, the model generation device 120 can generate a sequence of three-dimensional models representing the three-dimensional shape of the subject at each time. By synchronizing between the cameras 110, the model generation device 120 can generate a time-series three-dimensional model from the video captured by the camera group 110. In this specification, a sequence is a collection of one or more three-dimensional models.

[0015] The image generation device 100 includes an information input unit 130, a sequence selection unit 131, a time specifying unit 132, a parameter specifying unit 133, a model recording unit 134, a model acquisition unit 135, an image generation unit 136, and an output unit 137.

[0016] The information input unit 130 acquires information necessary for virtual viewpoint image generation. The information input unit 130 is, for example, an input device such as a keyboard or a pointing device. The information input unit 130 may have a monitor for displaying information. The user can input information using the information input unit 130. For example, in order for the user to access the image generation device 100, the user can input user information for identifying the user using the information input unit 130. Also, the user can input information specifying a three-dimensional model (for example, a sequence) to be used for generating a virtual viewpoint image using the information input unit 130. Also, the user can specify the virtual viewpoint of the virtual viewpoint image using the information input unit 130. For example, the user can input information specifying the time, position, or orientation of the virtual viewpoint.

[0017] The information input unit 130 acquires information specifying the three-dimensional model in this way and outputs it to the sequence selection unit 131. The information input unit 130 also acquires information indicating the virtual viewpoint in this way and outputs it to the time specifying unit 132 and the parameter specifying unit 133. The information input unit 130 also acquires the user information in this way and outputs it to the user information acquisition unit 141.

[0018] The sequence selection unit 131 selects a three-dimensional model according to the information acquired from the information input unit 130. In this example, the sequence selection unit 131 selects a sequence of three-dimensional models. The sequence of three-dimensional models is stored in the model recording unit 134 described later. The image generation unit 136 described later generates a virtual viewpoint image using the sequence of three-dimensional models selected by the sequence selection unit 131.

[0019] The time specifying unit 132 specifies a frame in the sequence of three-dimensional models according to the information acquired from the information input unit 130. The image generation unit 136 described later generates a virtual viewpoint image using the three-dimensional model corresponding to the specified frame in the sequence of three-dimensional models selected by the sequence selection unit 131. In the present embodiment, this time may be referred to as the time of the virtual viewpoint. That is, the virtual viewpoint image from a specific virtual viewpoint at a specific time is generated using the three-dimensional model corresponding to this specific time.

[0020] The parameter specifying unit 133 sets virtual viewpoint parameters according to the information acquired from the information input unit 130. The types of virtual viewpoint parameters are not particularly limited. The virtual viewpoint parameters may include at least one of the position and orientation of the virtual viewpoint. Also, the virtual viewpoint parameters can be at least one of the external parameters and internal parameters of the virtual camera corresponding to the virtual viewpoint. For example, the virtual viewpoint parameters may be the position, orientation, and focal length of the virtual viewpoint. On the other hand, the virtual viewpoint parameters may be the position and focal length of the virtual viewpoint, and the position of the fixation point from the virtual viewpoint.

[0021] The model recording unit 134 stores the data of the three-dimensional model. The model recording unit 134 can store the data of the three-dimensional model generated by the model generation device 120. However, the model recording unit 134 may store the data of the three-dimensional model generated by another device. In the present embodiment, the model recording unit 134 stores a three-dimensional model indicating the three-dimensional shape of the object at each time. The model recording unit 134 can store sequence data indicating the three-dimensional models for each of a plurality of times for one or more objects. For example, the model recording unit 134 can store the data of the three-dimensional model input from the three-dimensional model generation unit 121 in units of sequences.

[0022] FIG. 2 shows an example of the three-dimensional model data stored in the model recording unit 134. The model recording unit 134 stores a list (Sequence List) of the sequences stored in the model recording unit 134. The number of sequences stored in the model recording unit 134 is recorded in this list. This number of sequences increases by one each time a new sequence is registered. Also, information regarding pointers (*Sequence Data) to the data of each sequence is recorded in this list. By referring to the pointer, the data (Sequence Data) of the sequence can be accessed.

[0023] Sequence Data records a Sequence ID that identifies the sequence, a Place such as the shooting location or the data generation location, and a Date Time of shooting or data generation. In this example, the data of the three-dimensional model is managed in units of frames. One sequence is composed of one or more frames. The Date Time represents the date and time of the first frame. Also, the data of the sequence records a Frame rate. According to the frame rate, the date and time for each frame can be calculated. The data of the sequence further records the Number of frames. Furthermore, the data of the sequence includes Material Data for each frame.

[0024] There may be three-dimensional models of a plurality of objects in each frame. Therefore, the Material Data of the frame records the number of models of the three-dimensional model. Also, the Material Data of the frame records a Time code indicating the time of the frame. Furthermore, the data of the frame includes Model Data for each three-dimensional model.

[0025] The Model Data records a Model ID for identifying the model. Also, the three-dimensional model data includes the location of the three-dimensional model and the data of the point cloud data representing the three-dimensional model.

[0026] The sequence selection unit 131 can select a sequence stored in the model recording unit 134 according to the information about the sequence input from the information input unit 130. For example, the sequence selection unit 131 can identify the sequence ID by performing a search based on the date and time, or location information. Then, the sequence selection unit 131 can select the sequence corresponding to the sequence ID.

[0027] Also, the time specifying unit 132 can receive the time of the virtual viewpoint or the frame number from the information input unit 130. The time specifying unit 132 can identify a desired frame from the sequence according to the date and time and the frame rate included in the sequence data.

[0028] The model acquisition unit 135 acquires the three-dimensional models of one or more objects at the specified time from the model recording unit 134. In the present embodiment, the model acquisition unit 135 reads out the data of the three-dimensional model corresponding to the frame specified by the time specifying unit 132 from among the sequences selected by the sequence selection unit 131 from the model recording unit 134. Then, the model acquisition unit 135 inputs the read data of the three-dimensional model to the image generation unit 136.

[0029] The image generation unit 136 generates a virtual viewpoint image corresponding to the specified time from the specified virtual viewpoint according to the specified virtual viewpoint parameters, using the three-dimensional models of one or more objects at the specified time. In the present embodiment, the image generation unit 136 generates a virtual viewpoint image according to the data of the three-dimensional model acquired from the model acquisition unit 135 and the virtual viewpoint parameters input from the parameter specifying unit 133. The method for generating the virtual viewpoint image is not particularly limited. For example, the image generation unit 136 can generate a virtual viewpoint image according to the ray tracing method.

[0030] The output unit 137 outputs the virtual viewpoint image generated by the image generation unit 136. The output unit 137 can transmit the virtual viewpoint image to a user device (not shown), such as a tablet, a computer, or a display device. The display device may be the display included in the information input unit 130.

[0031] Next, the information processing apparatus 101 according to an embodiment will be described. The information processing apparatus 101 includes a user information acquisition unit 141, a determination unit 142, and an information recording unit 143.

[0032] The user information acquisition unit 141 acquires information of the user who instructed the generation of the virtual viewpoint image. The user information acquisition unit 141 is connected to the information input unit 130 and can acquire information of the user accessing the image generation apparatus 100. The user information is information that can identify the user, for example, a name or a user ID.

[0033] The determination unit 142 determines whether or not a predetermined condition, which is a condition related to at least one of the time and the virtual viewpoint parameter, is satisfied. As described above, the image generation unit 136 generates a virtual viewpoint image corresponding to the specified time from the virtual viewpoint according to the specified virtual viewpoint parameter. The predetermined condition is a condition related to the specified time and the specified virtual viewpoint parameter. For this purpose, the determination unit 142 can acquire information indicating the selected sequence, time information, and virtual viewpoint parameters from the sequence selection unit 131, the time specification unit 132, and the parameter specification unit 133.

[0034] For example, the determination unit 142 can determine whether a predetermined condition is satisfied based on the time relationship of each of two or more virtual viewpoint images generated by the image generation unit 136. Specifically, the determination unit 142 can determine that the predetermined condition is satisfied when many virtual viewpoint images for times close to each other are generated. This is because it is relatively easy to create a three-dimensional model using many virtual viewpoint images for times close to each other. Note that the predetermined condition may be a condition regarding the time relationship of each of two or more virtual viewpoint images generated by the image generation unit 136 using the same sequence data. Further, the predetermined condition may be a condition regarding the time relationship of each of two or more virtual viewpoint images generated by the image generation unit 136 using a time-series three-dimensional model of the same object.

[0035] In addition, the determination unit 142 can determine whether a predetermined condition is satisfied based on the relationship of virtual viewpoint parameters of each of two or more virtual viewpoint images generated by the image generation unit 136. Specifically, the determination unit 142 can determine that the predetermined condition is satisfied when many virtual viewpoint images for adjacent portions in the virtual space are generated. This is because it is relatively easy to create a three-dimensional model using many virtual viewpoint images for adjacent portions. Note that the predetermined condition may be a condition regarding the relationship of virtual viewpoint parameters of each of two or more virtual viewpoint images generated by the image generation unit 136 using the same sequence data. Further, the predetermined condition may be a condition regarding the relationship of virtual viewpoint parameters of each of two or more virtual viewpoint images generated by the image generation unit 136 using a time-series three-dimensional model of the same object.

[0036] In the following embodiments, the determination unit 142 determines whether or not a predetermined condition, which is a condition related to both time and virtual viewpoint parameters, is satisfied. For example, the determination unit 142 can determine, based on time information and virtual viewpoint parameters, whether a plurality of virtual viewpoint images for times at the same time or within a short predetermined time period have been generated from a virtual viewpoint that satisfies a specific condition. The specific condition related to the virtual viewpoint may be, for example, that the virtual viewpoint is arranged around the same object.

[0037] The information recording unit 143 records information used for specifying the three-dimensional model of the object corresponding to the virtual viewpoint image generated by the image generation unit 136 in response to the satisfaction of a predetermined condition. The information recorded by the information recording unit 143 is not particularly limited, and any information that helps to specify the three-dimensional model used by the image generation unit 136 to generate the virtual viewpoint image can be recorded. Note that it is not necessary that the three-dimensional model can be specified based only on the information used for specifying the three-dimensional model. For example, the three-dimensional model may be specifiable based on the information used for specifying the three-dimensional model and other information (for example, user input).

[0038] For example, the information recording unit 143 can record at least one of time and virtual viewpoint parameters as information used for specifying the three-dimensional model of the object. Specifically, the information recording unit 143 can record information indicating the time of the virtual viewpoint corresponding to the generated virtual viewpoint image. As will be described later, the sequence data and the object corresponding to the three-dimensional model to be inspected can be specified manually or automatically. Then, among the time-series three-dimensional models of this object, the three-dimensional model corresponding to the time of the recorded virtual viewpoint can be specified as the three-dimensional model to be compared.

[0039] In addition, the information recording unit 143 can record information indicating virtual viewpoint parameters corresponding to the generated virtual viewpoint image. As will be described later, the sequence data corresponding to the three-dimensional model to be inspected can be specified manually or automatically. Then, among the plurality of objects indicated by this sequence data, an object within the viewing range according to the virtual viewpoint parameters can be specified as an object corresponding to the three-dimensional model to be inspected. Note that the selection of the three-dimensional model for comparison from the time-series three-dimensional model of the specified object can be specified manually or automatically as will be described later.

[0040] In addition, the information recording unit 143 can record information for specifying the sequence data used to generate the virtual viewpoint image as information used to specify the three-dimensional model of the object. For example, the information recording unit 143 can record the sequence ID of the sequence used to generate the virtual viewpoint image. Referring to such information, as will be described later, the sequence data corresponding to the three-dimensional model to be inspected can be specified.

[0041] Furthermore, the information recording unit 143 can record user information of the user who instructed the generation of the virtual viewpoint image. For example, the information recording unit 143 can record the user information regarding the user who instructed the generation of the virtual viewpoint image, which is acquired by the user information acquisition unit 141. Referring to such information, a user suspected of having replicated the three-dimensional model of the object can be specified.

[0042] Note that the information recording unit 143 can record history information indicating that the virtual viewpoint image has been generated in response to a predetermined condition being satisfied. The information recording unit 143 can store information used to specify the three-dimensional model of the object as described above as part of such history information. For example, the information recording unit 143 can record history information indicating that the virtual viewpoint image has been generated using the three-dimensional model of the object at the specified time.

[0043] Note that the information processing apparatus 101 starts up simultaneously with the startup of the image generation apparatus 100. Then, the processing of the information processing apparatus 101 ends simultaneously with the end of the processing of the image generation apparatus 100.

[0044] FIG. 3 is a flowchart showing an example of the processing performed by the determination unit 142. In S301, the determination unit 142 detects that the image generation apparatus 100 has started generating the virtual viewpoint image. The determination unit 142 can detect the generation of the virtual viewpoint image based on the input of user information. Note that before S302, or in parallel with S302 to S306, the image generation apparatus 100 generates the virtual viewpoint image. That is, before S302, or in parallel with S302 to S306, the acquisition of the three-dimensional model by the model acquisition unit 135, the generation of the virtual viewpoint image by the image generation unit 136, and the output of the virtual viewpoint image by the output unit 137 are performed. The generation of the virtual viewpoint image by the image generation unit 136 may be performed in response to the operation of the virtual viewpoint via the information input unit 130 by the user. For example, the image generation unit 136 can generate the virtual viewpoint image in response to the change in the time of the virtual viewpoint or the virtual viewpoint parameters.

[0045] In S302, the determination unit 142 acquires the user's information from the user information acquisition unit 141. In S303, the determination unit 142 acquires the information of the selected sequence used to generate the virtual viewpoint image from the sequence selection unit 131. In the present embodiment, the determination unit 142 acquires the sequence ID. However, the information of the sequence is not limited to the sequence ID. For example, the determination unit 142 may acquire the sequence name or the file name of the sequence data as the information of the sequence. In S304, the determination unit 142 acquires the information of the time of the virtual viewpoint used to generate the virtual viewpoint image from the time specification unit 132. In S305, the determination unit 142 acquires the virtual viewpoint parameters used to generate the virtual viewpoint image from the parameter specification unit 133. In this way, the determination unit 142 can acquire the information used to determine whether a predetermined condition is satisfied and the information to be recorded in the information recording unit 143. The order of S301 to S305 may be different. Also, it is not necessary for the determination unit 142 to acquire all of this information.

[0046] In S306, the determination unit 142 temporarily stores the information such as the sequence ID, time information, and virtual viewpoint parameters acquired in S302 to S305. In the present embodiment, the determination unit 142 stores this information in the memory inside the determination unit 142.

[0047] FIG. 5 shows an example of the information stored in the memory by the determination unit 142. In the example of FIG. 5, the position, orientation, and focal length of the virtual viewpoint are used as the virtual viewpoint parameters. The position of the virtual viewpoint is represented by three values for the x, y, and z axes. The orientation of the virtual viewpoint can be expressed, for example, by the quaternion shown in Equation (1). Q=(0:q x 、q y 、q x )…(1) In Equation (1), the left side of the colon represents the real part, and q x 、q y 、and q xrepresents the imaginary part. The focal length can be expressed by the zoom ratio. In the example of FIG. 5, the focal length is represented by α.

[0048] In S307, the determination unit 142 determines whether or not a series of virtual viewpoint operations has ended. When a series of inputs from the information input unit 130 has ended, the determination unit 142 can determine that the virtual viewpoint operation has ended. In this case, the process proceeds to S308. When there is a subsequent input, the process returns to S303. Then, the determination unit 142 continuously acquires information for generating the next virtual viewpoint image.

[0049] In S308, the determination unit 142 determines whether or not a predetermined condition, which is a condition related to at least one of the time and the virtual viewpoint parameters, is satisfied. The determination unit 142 performs this determination process for each sequence.

[0050] In this example, the predetermined condition includes a condition regarding the time interval between two virtual viewpoint images. Specifically, the predetermined condition is a condition related to the number D of pairs of specific virtual viewpoint images among the plurality of virtual viewpoint images generated by the image generation unit 136. Here, the time interval for each pair of the specific virtual viewpoint images is less than the threshold ThT. In the following example, the determination unit 142 determines whether or not such a condition regarding time is satisfied in S405.

[0051] Also, in this example, the predetermined condition includes a condition regarding the positional relationship of the virtual viewpoints for each of two or more virtual viewpoint images generated by the image generation unit 136. Specifically, the predetermined condition includes a condition regarding the similarity of the fixation points or the in - field regions of the virtual viewpoints for each of two or more virtual viewpoint images generated by the image generation unit 136. In the following example, the determination unit 142 determines whether or not such a condition regarding the positional relationship of the virtual viewpoints is satisfied in S407.

[0052] Figure 4 is a detailed flowchart of an example of the determination process in S308. In S401, the loop for each sequence starts. For all the sequences corresponding to the sequence ID saved in S306, the determination process is performed for each sequence. In S401, the determination unit 142 sequentially sets the sequences to be the objects of the determination process.

[0053] In S402, the determination unit 142 acquires, from the memory, all the sets of all the times and the virtual viewpoint parameters regarding the sequence set in S401. In the example of Figure 5, when the sequence with sequence ID = 1 is set as the object of the determination process, the determination unit 142 acquires four sets of time and virtual viewpoint parameters.

[0054] In S403, the determination unit 142 calculates the time intervals for all combinations of times. In the example of Figure 5, when the object of the determination process is the sequence with sequence ID = 1, the determination unit 142 can calculate the following six intervals. Δt12 = |t11 - t12| Δt13 = |t11 - t13| Δt14 = |t11 - t14| Δt23 = |t12 - t13| Δt24 = |t12 - t14| Δt34 = |t13 - t14|

[0055] In S404, the determination unit 142 determines the number D of intervals less than the threshold ThT among the calculated intervals Δt12 to Δt34. For example, the threshold ThT may be the interval between the first frame and the second frame, or the interval between the first frame and the third frame, according to the frame rate. Also, the threshold ThT may be dynamically changed according to the speed of the movement of the object. For example, when the speed of the movement of the object is small, the threshold ThT can be made larger. For example, when the speed of the movement of the object is large, the threshold ThT may be 1 / 30 second. Also, when the speed of the movement of the object is small, the threshold ThT may be 1 / 10 second.

[0056] In S405, the determination unit 142 determines whether the value D calculated in S404 is greater than or equal to the threshold value ThD. If the value D is greater than or equal to the threshold value ThD, the process proceeds to S406; otherwise, the process proceeds to S408. Note that the condition regarding the time interval is not limited to the value D being greater than or equal to the threshold value ThD in this way. For example, the predetermined condition may be a condition regarding the number of virtual viewpoint images corresponding to times included in a predetermined time range among the plurality of virtual viewpoint images generated by the image generation unit 136. For example, among the plurality of virtual viewpoint images generated using the sequence set in S401, when the times of three or more virtual viewpoint images are included within a predetermined time range (for example, within two frames), the determination unit 142 may determine that the predetermined condition is satisfied. Note that the predetermined condition may be set so as not to be satisfied when performing normal playback following the time series while moving the virtual viewpoint. For example, when the threshold value ThT is the interval between the first frame and the second frame according to the frame rate, the threshold value ThD can be set to 3 or more.

[0057] In S406, the determination unit 142 determines the fixation point or the in-field region of the virtual viewpoint for the set of virtual viewpoint images generated using the sequence set in S401. The fixation point or the in-field region of the virtual viewpoint can be determined according to the virtual viewpoint parameters of each virtual viewpoint image saved in S306. Note that when the three-dimensional model is arranged on the field, the fixation point can be represented using the intersection of the line of sight center of the virtual viewpoint and the field. Also, the in-field region can be represented using the region of the field within the field of view of the virtual viewpoint.

[0058] In S407, the determination unit 142 determines whether each virtual viewpoint in the set of virtual viewpoint images generated using the sequence set in S401 is targeting the same part. The determination unit 142 can determine the number P of virtual viewpoints targeting the same part in this way. For example, when the distance between the fixation points is equal to or less than the threshold value, or when the overlapping area of the regions within the visual field is equal to or greater than the threshold value, the determination unit 142 can determine that each virtual viewpoint is targeting the same part. As another example, when the virtual viewpoints are arranged on a general ellipse or a general sphere at equal intervals around the fixation point, or when the intervals between the virtual viewpoints are substantially equal, the determination unit 142 can determine that each virtual viewpoint is targeting the same part. If the number P of virtual viewpoints targeting the same part is equal to or greater than the threshold value ThP, the process proceeds to S409; otherwise, the process proceeds to S408.

[0059] In S408, the determination unit 142 determines that a predetermined condition is not satisfied for the sequence set in S401.

[0060] In S409, the determination unit 142 determines that a predetermined condition is satisfied for the sequence set in S401.

[0061] In S410, the determination unit 142 determines whether determination processing has been performed for all sequences. If determination processing for all sequences has not been performed, the process returns to S401, and the determination unit 142 sets the next sequence as the target of determination processing.

[0062] As described above, the determination processing in S308 is performed. In S309, the determination unit 142 determines whether there is a sequence for which it is determined that a predetermined condition is satisfied. If there is a sequence for which the predetermined condition is satisfied, the process proceeds to S310; otherwise, the process proceeds to S311.

[0063] In S310, the determination unit 142 outputs, for the virtual viewpoint image generated using the sequence determined to satisfy the predetermined condition, the information used to specify the three-dimensional model of the object corresponding to this virtual viewpoint image for recording in the information recording unit 143. When it is determined that the predetermined condition is satisfied for the sequence set in S401 (for example, sequence ID = 1), the determination unit 142 can record the information for each of the four virtual viewpoint images (times t11 to t14) in the information recording unit 143.

[0064] In the following example, the information used to specify the three-dimensional model of the object corresponding to the virtual viewpoint image is the information used to generate the virtual viewpoint image. For example, the determination unit 142 can record in the information recording unit 143 a combination of the sequence ID, the time of the virtual viewpoint, the virtual viewpoint parameters, and the user information. FIG. 6 shows an example of the data recorded in the information recording unit 143.

[0065] The information recording unit 143 stores a list of sequences (Sequence List) to be recorded. In this list, the number of sequences recorded is recorded. Also, in this list, pointers (*Sequence Data) to the data for each recorded sequence are recorded.

[0066] In the data regarding the sequence (Sequence Data), the sequence ID (Sequence ID) that specifies the sequence is recorded. Also, in the data regarding the sequence, although not essential, the shooting location of the object representing the sequence or the generation location of the sequence (Place), and the date and time or time code of shooting or generation (Date Time(Timecode)) can be recorded. In the data regarding the sequence, the number of record data to be further recorded is further recorded. Furthermore, in the data regarding the sequence, pointers (*Record data) to the respective record data are recorded.

[0067] Record Data stores information used to generate a virtual viewpoint image using a sequence that is determined to meet a predetermined condition at S308. In the example of FIG. 6, the Record Data records the user information acquired by the user information acquisition unit 141, in this example, the User ID. Further, the Record Data records the number of Virtual Camera information and pointers (*Camera Data) to the data of each virtual viewpoint.

[0068] The data (Camera Data) of each virtual viewpoint records the time of the virtual viewpoint, in this example, the Time code, and the Camera Parameter. The time of the virtual viewpoint and the virtual viewpoint parameter are the same as the information shown in FIG. 5. Note that in FIG. 6, instead of pointers to each data, the data itself may be recorded.

[0069] In S310, the determination unit 142 updates or adds the data recorded by the information recording unit 143. For example, when the sequence ID for a newly recorded sequence is not in the sequence list, the determination unit 142 adds 1 to the number of sequences. Further, the determination unit 142 secures a memory for recording the sequence data (Sequence Data) and adds a pointer (*Sequence Data) to this data.

[0070] The sequence ID is recorded in the sequence data indicated by this pointer. Further, the determination unit 142 may record information (for example, location or date and time) related to the sequence acquired from the model recording unit 134 in the information recording unit 143. Furthermore, the determination unit 142 sets 1 to the number of Record Data in which the sequence data is recorded. Then, the determination unit 142 secures a memory for recording the record data and adds a pointer (*Record Data No.1) to this data.

[0071] The record data indicated by this pointer records the user ID which is user information. Also, the determination unit 142 records the number of Virtual Camera information of the virtual viewpoints. Each virtual viewpoint information can record the time of the virtual viewpoint, for example, the time code, and the camera parameter of the virtual viewpoint.

[0072] Also, for example, when the sequence ID for a newly recorded sequence is in the sequence list, the determination unit 142 updates the sequence data for this sequence. For example, the determination unit 142 adds 1 to the number of record data where the sequence data is recorded. Also, the determination unit 142 secures a memory for recording the sequence data and adds a pointer (*Sequence Data) to this data. The information recorded in the record data indicated by this pointer is the same as when the sequence ID for the newly recorded sequence is not in the sequence list.

[0073] In S311, the determination unit 142 erases the information temporarily stored in the memory of the determination unit 142. For example, the determination unit 142 erases data such as the sequence ID, time, and virtual viewpoint parameter referred to in the determination process.

[0074] In S312, the determination unit 142 determines whether or not the generation of all virtual viewpoint images has been completed. If the generation of virtual viewpoint images continues, the process returns to S301. Otherwise, the process in FIG. 3 ends.

[0075] In the above example, the determination of whether or not the predetermined condition in S308 is satisfied is made for each sequence. However, the determination of whether or not the predetermined condition is satisfied may be made for each virtual viewpoint image. For example, in S308, the determination unit 142 can determine whether or not a condition regarding at least one of the time of the target virtual viewpoint image and the virtual viewpoint parameters is satisfied. Specifically, the determination unit 142 can calculate the intervals (Δt12 to Δt14) between the time of the target virtual viewpoint image (e.g., t11) and the times of other virtual viewpoint images (e.g., t12 to t14) generated using the same sequence. Then, the determination unit 142 can determine the number D of intervals less than the threshold ThT among the calculated intervals (Δt12 to Δt14). The determination unit 142 can determine that the time of the target virtual viewpoint image satisfies the predetermined condition when this value D is equal to or greater than the threshold ThD.

[0076] Similarly, the determination unit 142 can determine whether or not the virtual viewpoint of the target virtual viewpoint image is aimed at the same part as the virtual viewpoint of any of the other virtual viewpoint images generated using the same sequence. The determination unit 142 can determine that the virtual viewpoint parameters of the target virtual viewpoint image satisfy the predetermined condition when these virtual viewpoints are aimed at the same part. The determination unit 142 can determine that the predetermined condition regarding the target virtual viewpoint image is satisfied in response to determining that both the time and the virtual viewpoint parameters of the target virtual viewpoint image satisfy the predetermined condition. In this case, in S310, the information used to specify the three-dimensional model of the object corresponding to the target virtual viewpoint image can be recorded in the information recording unit 143.

[0077] With the above configuration, when a predetermined condition is satisfied, information used to identify the three-dimensional model of the object corresponding to the virtual viewpoint image can be stored. Generally, in order to record such information for all the generated virtual viewpoint images, a recording device with an extremely large capacity is required. By storing such information only when a predetermined condition is satisfied as in the present embodiment, the amount of information to be stored can be reduced. On the other hand, according to the present embodiment, the predetermined condition can be set so as to be satisfied when the virtual viewpoint image is generated for the purpose of replicating the three-dimensional model, or when the virtual viewpoint image that is easy to replicate the three-dimensional model is generated. Therefore, according to the present embodiment, for all the generated virtual viewpoint images, it is possible to effectively verify whether the three-dimensional model has been replicated based on the virtual viewpoint image without recording the information used to identify the three-dimensional model of the object corresponding to the virtual viewpoint image. In addition, by reducing the amount of information stored in this way, the load of the process of determining whether the three-dimensional model has been replicated based on the virtual viewpoint image by referring to the stored information can be reduced.

[0078] Note that a plurality of users may simultaneously generate virtual viewpoint images using the image generation device 100. In this case, for each user, it is possible to acquire and store information related to the generation of the virtual viewpoint image, determine a predetermined condition, and record information based on the determination result. Also, in this case, the access to the information recording unit 143 may be performed in a time-division manner or in the order of access by the users.

[0079] In addition, in order to copy three-dimensional data, it is also assumed that the virtual viewpoint images are generated not at one timing but at a plurality of timings. For this reason, instead of performing the determination process in S308 after a series of virtual viewpoint operations are completed as in S307, the processes of S308 to S311 may be performed after the generation of all virtual viewpoints is completed as in S312. Further, information regarding the generation of virtual viewpoint images using the image generation apparatus 100 may be stored for a certain period. In this case, the determination unit 142 may determine whether a predetermined condition is satisfied with reference to the stored information. Specifically, the determination unit 142 may determine whether a predetermined condition is satisfied based on the interval between the time of the virtual viewpoint when the virtual viewpoint image was generated in the past and the time of the virtual viewpoint when the virtual viewpoint image is generated this time. Further, the determination unit 142 may determine whether a predetermined condition is satisfied based on the commonality between the portion targeted by the virtual viewpoint when the virtual viewpoint image was generated in the past and the portion targeted by the virtual viewpoint when the virtual viewpoint image is generated this time. For example, the memory of the determination unit 142 can hold information stored for a certain period. When there is virtual viewpoint image generation by the same user, the determination unit 142 can read out the sequence ID, time, and virtual viewpoint parameters corresponding to the same user ID from the memory. Then, the determination unit 142 can perform a determination process based on the read information. The specific length of the certain period is not particularly limited. For example, this certain period may be an accessible period to the three-dimensional model data or a set predetermined period.

[0080] In the above embodiment, the user's own information (for example, user ID) is used as the user information. However, the type of user information is not particularly limited. For example, the user may have an information processing apparatus connected to the image generation apparatus 100 via a network or the like. In this case, the user can obtain the generated virtual viewpoint image by controlling the image generation apparatus 100 via the information processing apparatus owned by the user. In such an example, the user information may be the identification information (for example, serial number) of the information processing apparatus owned by the user. The information processing apparatus 101 may record such user information.

[0081] (Modified Example) In addition to the information used for generating the virtual viewpoint image, the determination unit 142 may further store at least a part of the virtual viewpoint image generated by the image generation unit 136. In such a modified example, in S310, the determination unit 142 reads out from the image generation unit 136 a virtual viewpoint image generated using a sequence determined to satisfy a predetermined condition. Then, the determination unit 142 outputs at least a part of the read virtual viewpoint image for storage in the image storage unit 144. The image storage unit 144 stores at least a part of the virtual viewpoint image output from the determination unit 142.

[0082] In such a modified example, as shown in FIG. 6, the virtual viewpoint data (Camera Data) recorded by the information recording unit 143 includes a virtual viewpoint image (Virtual View Point Image) in addition to the time code (Time code) and the virtual viewpoint parameter (Camera Parameter). This virtual viewpoint image is a virtual viewpoint image generated by the image generation unit 136 according to the time code and the virtual viewpoint parameter. Thus, in addition to the information regarding the generation of the virtual viewpoint image that satisfies a predetermined condition, the virtual viewpoint image generated according to this information can be stored. By storing the virtual viewpoint image, when estimating whether the three-dimensional model is replicated based on the virtual viewpoint image, it is possible to facilitate the search for the corresponding three-dimensional model or the corresponding time code recorded by the model recording unit 134.

[0083] Note that the image storage unit 144 may be included in the information recording unit 143. That is, the information recording unit 143 may store the virtual viewpoint image.

[0084] Also, the information recording unit 143 may record data in units of users. FIG. 8 shows an example of data recorded in units of users. The information recording unit 143 stores a user list (User List). The number of users is recorded in this list. Only the users who have performed the virtual viewpoint image generation operation determined by the determination unit 142 to satisfy a predetermined condition may be recorded in this list. Also, pointers to data for each user (*User Data) are recorded in this list.

[0085] Information specific to the user (User Data) records information for identifying the user, such as a user ID (User ID). The data for each user further includes a pointer to the data of the sequence (*Sequence Data) used by this user to create a virtual viewpoint image based on virtual viewpoint parameters that meet a predetermined condition, and the number of sequence data (number of sequence).

[0086] The data related to the sequence (Sequence Data) records a sequence ID (Sequence ID) for identifying the sequence. Also, the data related to the sequence records a pointer to a data set (*Data Set) including time information and virtual viewpoint parameters when a predetermined condition is met in the generation of a virtual viewpoint image using this sequence. The number of data sets (Number of Data Set) recorded in the data related to the sequence is further recorded.

[0087] Each data set (Data Set) records the time information (Date Time) when a predetermined condition is met and the number of pieces of information of the virtual viewpoints to be recorded (number of Virtual Camera). Each piece of information of the virtual viewpoints records time information, such as a time code (Time code), and virtual viewpoint parameters (Camera Parameter).

[0088] In such an example, when specific conditions are met, information such as the sequence, time, and virtual viewpoint parameters used for generating the virtual viewpoint image is recorded in the information recording unit 143 in units of users who operated using the information input unit 130. In this way, by managing the records for each user, when it is possible to estimate the user who replicated the three-dimensional model, the time required to verify whether the three-dimensional model has been replicated can be reduced.

[0089] (Verification of replication) As described above, the information processing apparatus 101 can store information related to the generation of the virtual viewpoint image. By referring to the information stored in this way, it is possible to estimate whether the three-dimensional shape produced by a third party is a replica based on the virtual viewpoint image by the image generation apparatus 100. Such determination may be made visually by the user, or may be made automatically or semi-automatically according to user input.

[0090] Hereinafter, an information processing system for performing such estimation will be described. An information processing system according to an embodiment includes an information processing apparatus 200. The information processing apparatus 200 compares a three-dimensional model to be inspected with a three-dimensional model used in the past for generating a virtual viewpoint image. The information processing apparatus 200 can access the data stored in the model recording unit 134 of the image generation apparatus 100. Then, the information processing apparatus 200 can estimate whether the inspection target 211 produced by a third party is replicated using the three-dimensional model stored in the model recording unit 134.

[0091] The information processing apparatus 200 includes a data acquisition unit 210, a data analysis unit 212, and a result output unit 213. The data acquisition unit 210 acquires a three-dimensional model of the inspection target. For example, the data acquisition unit 210 can acquire three-dimensional model data of the inspection target 211. The method for acquiring the three-dimensional model data is not particularly limited. For example, the model generation device 120 may generate a three-dimensional model of the inspection target 211 using the camera group 110. At this time, the data acquisition unit 210 can acquire the three-dimensional model of the inspection target 211 from the model generation device 120. Also, the three-dimensional model data may be generated using a measuring device such as a 3D scanner. For example, a contact type, laser beam type, or pattern light projection type 3D scanner can be used. Also, the format of the three-dimensional model data is not particularly limited. For example, the three-dimensional model data may be three-dimensional point cloud data representing a shape.

[0092] The data analysis unit 212 acquires, as a three-dimensional model for comparison, the three-dimensional model that was used to generate a virtual viewpoint image in the past from the model recording unit 134 that stores a three-dimensional model indicating the three-dimensional shape of the object. Specifically, the data analysis unit 212 can search for the three-dimensional model stored in the model recording unit 134 that corresponds to the three-dimensional model data of the inspection target 211. Here, the data analysis unit 212 can acquire the three-dimensional model for comparison by referring to the generation record of the virtual viewpoint image. For example, the data analysis unit 212 can search for the three-dimensional model stored in the model recording unit 134 based on the information recorded in the information recording unit 143. As described above, the information recording unit 143 can record at least one of the time corresponding to the virtual viewpoint image and the virtual viewpoint parameters as the generation record of the virtual viewpoint image. At this time, the data analysis unit 212 can acquire the three-dimensional model that was used to generate the virtual viewpoint image by referring to at least one of the time corresponding to the virtual viewpoint image and the virtual viewpoint parameters.

[0093] For example, the data analysis unit 212 identifies a sequence that is presumed to have been used to generate a virtual viewpoint image used to create the inspection target 211 from among the sequences recorded in the model recording unit 134. The data analysis unit 212 may automatically identify a sequence corresponding to the inspection target 211 based on the shape of the three-dimensional model of the inspection target 211 or textures such as a uniform or player name. On the other hand, the data analysis unit 212 may obtain a user input indicating a sequence corresponding to the inspection target 211 from the user.

[0094] Also, the data analysis unit 212 identifies the time code of the three-dimensional model that was used to generate the virtual viewpoint image that is presumed to have been used to create the inspection target 211 for the sequence thus identified. For example, the user may set the time code corresponding to the inspection target 211. In this case, the user can select the time code corresponding to the inspection target 211 from among the time codes for the identified sequence recorded in the information recording unit 143. For example, the user can select the time code such that the three-dimensional model for the selected time code resembles the inspection target 211. Then, the data analysis unit 212 can read out the three-dimensional model corresponding to the set time code from the model recording unit 134. The model recording unit 134 stores the data of the three-dimensional model that was input from the three-dimensional model generation unit 121 and stored in units of sequences.

[0095] Furthermore, the data analysis unit 212 identifies the three-dimensional model that was used to generate the virtual viewpoint image that is presumed to have been used to create the inspection target 211 for the sequence thus identified. For example, the data analysis unit 212 can identify a three-dimensional model of a comparison target corresponding to the three-dimensional model of the inspection target from among the three-dimensional models of a plurality of objects for the identified sequence. Here, the data analysis unit 212 can identify, as the three-dimensional model of the comparison target, the three-dimensional model that is within the visual field range according to the virtual viewpoint parameters at the identified time code. On the other hand, the user may identify the three-dimensional model of the comparison target corresponding to the inspection target 211.

[0096] Then, the data analysis unit 212 compares the three-dimensional model of the inspection target and the three-dimensional model of the comparison target. Specifically, the data analysis unit 212 can analyze and compare the three-dimensional model of the inspection target 211 and the three-dimensional model of the comparison target stored in the model recording unit 134. The result output unit 213 outputs the results of the analysis and comparison by the data analysis unit 212.

[0097] Furthermore, the data analysis unit 212 verifies the similarity between the three-dimensional model of the inspection target 211 acquired by the data acquisition unit 210 and the three-dimensional model read from the model recording unit 134. If it is determined that there is similarity, the result output unit 213 can output a presumption result that the inspection target 211 corresponds to a copy of the three-dimensional model stored in the model recording unit 134. At this time, the result output unit 213 can output user information (for example, user ID) regarding the user who generated the virtual viewpoint image using the three-dimensional model read from the model recording unit 134. Also, the result output unit 213 may output time information (time code) regarding this three-dimensional model. These pieces of information are recorded in the information recording unit 143. The method of outputting the information by the result output unit 213 is not particularly limited. For example, the result output unit 213 can notify, by display or voice, that it has been determined that there is similarity. Also, the result output unit 213 can output the result even when it is determined that there is no similarity.

[0098] In addition to or instead of outputting this information, the result output unit 213 can store it. For example, the result output unit 213 can store the user ID and time information. Further, the result output unit 213 may further store the three-dimensional model data of the inspection target 211 and the like. Furthermore, the information recording unit 143 may record the output from the result output unit 213.

[0099] FIG. 8 is a flowchart showing an example of the processing performed by the information processing apparatus 200. In the example shown in FIG. 8, the similarity of the three-dimensional model is evaluated in two stages. In the first stage, verification of the similarity based on the three-dimensional model bounding box is performed. The three-dimensional model bounding box used in the following example is the smallest cube surrounding the object region of the model. Such a three-dimensional model bounding box is used in object detection. In the second stage, verification of the similarity by another method (for example, comparison of three-dimensional point cloud data) is performed. By combining such a simple first-stage evaluation and a detailed second-stage evaluation in this way, the processing amount in the evaluation process can be reduced.

[0100] However, the method for verifying similarity is not limited to such a method. For example, it is not necessary to perform verification of similarity based on the three-dimensional model bounding box. As a specific method, the data analysis unit 212 can use the method for calculating the similarity between point clouds disclosed in Patent Document 2. Further, instead of performing the two-stage evaluation, a one-stage evaluation may be performed.

[0101] In S801, the data acquisition unit 210 acquires the three-dimensional model data of the inspection target 211. As described above, the data acquisition unit 210 can acquire the three-dimensional model data by using a measuring device such as a 3D scanner or a function for generating a three-dimensional model based on images from multiple viewpoints. In this example, the three-dimensional model data of the inspection target 211 has information on a three-dimensional point cloud representing the outer shape of the inspection target 211.

[0102] In S802, the data analysis unit 212 sets a three-dimensional model bounding box for the three-dimensional model of the inspection target 211 acquired in S801.

[0103] In S803, the data analysis unit 212 identifies a sequence that is presumed to correspond to the inspection target 211 among the sequences recorded in the information recording unit 143. As described above, the data analysis unit 212 identifies the sequence used by the image generation device 100 to generate a virtual viewpoint image that is presumed to have been used to create the inspection target 211.

[0104] In S804, the data analysis unit 212 selects one of the time codes recorded in the information recording unit 143 in association with the sequence identified in S803.

[0105] In S805, the data analysis unit 212 reads out the virtual viewpoint parameters associated with the time code selected in S804 from the information recording unit 143. Also, the data analysis unit 212 reads out the three-dimensional model corresponding to the time code selected in S804 from the model recording unit 134.

[0106] In S806, the data analysis unit 212 sets a three-dimensional model bounding box circumscribing the three-dimensional model read out from the model recording unit 134 in S805.

[0107] In S807, the data analysis unit 212 converts the three-dimensional model bounding box set in S802 so that it substantially coincides with the coordinates of the three-dimensional model bounding box set in S806. Then, the data analysis unit 212 compares the respective three-dimensional model bounding boxes. In S808, the data analysis unit 212 determines whether the vertex coordinates of the respective three-dimensional model bounding boxes substantially coincide. For example, the data analysis unit 212 can determine that the vertex coordinates substantially coincide when the sum of the distances between the pairs of the respective vertex coordinates is equal to or less than a threshold value. If it is determined that the vertex coordinates substantially coincide, the process proceeds to S809. Otherwise, the process proceeds to S814.

[0108] In S809, the data analysis unit 212 further compares the three-dimensional model of the inspection target acquired in S801 with the three-dimensional model acquired in S805. The data analysis unit 212 can compare the three-dimensional model data by an existing method. In S809, the data analysis unit 212 can use a comparison method different from the method of comparing the three-dimensional model bounding boxes in S807. For example, the data analysis unit 212 can compare the point cloud data representing the respective three-dimensional models. In S810, the data analysis unit 212 determines whether the three-dimensional models substantially coincide. If it is determined that the vertex coordinates substantially coincide, the process proceeds to S811. Otherwise, the process proceeds to S814.

[0109] In S811, the data analysis unit 212 acquires, from the user information acquisition unit 141, the user ID of the operator who instructed the generation of the virtual viewpoint image, which is associated with the time code selected in S804. In S812, the result output unit 213 notifies that there exists a three-dimensional model that is similar to the three-dimensional model of the inspection target and that was used for the generation of the virtual viewpoint image. In S813, the result output unit 213 outputs the user ID acquired in S811 and the time information (for example, the time code selected in S804) of the three-dimensional model used for the generation of the virtual viewpoint image.

[0110] In S814, the data analysis unit 212 determines whether a comparison has been made between the three-dimensional model corresponding to all time codes and the three-dimensional model of the object to be inspected. If the comparison of all three-dimensional models has not been completed, the process returns to S804. In S804, the data analysis unit 212 selects another one of the time codes recorded in the information recording unit 143 in association with the sequence specified in S803. If the comparison of all three-dimensional models has been completed, the process proceeds to S815. In S815, the result output unit 213 notifies that there is no three-dimensional model similar to the three-dimensional model of the object to be inspected and used for generating the virtual viewpoint image.

[0111] According to the above configuration, the information processing apparatus 200 can determine whether the three-dimensional shape produced by a third party is a copy based on the virtual viewpoint image generated by the image generation apparatus 100. Further, when it is determined that the three-dimensional shape produced by a third party is a copy, the information processing apparatus 200 can output or record the user ID of the operator who output the virtual viewpoint image and the time information about the three-dimensional model used for generating the virtual viewpoint image.

[0112] Note that it is not essential to compare the three-dimensional model corresponding to each of a plurality of time codes with the three-dimensional model of the object to be inspected. In S804, the data analysis unit 212 may select one time code according to the setting by the user or automatically as described above. Then, the data analysis unit 212 may perform only the comparison between the three-dimensional model corresponding to one time code and the three-dimensional model of the object to be inspected.

[0113] In addition, in order to evaluate the similarity of the three-dimensional model, the data analysis unit 212 may use an image-based comparison technique. For example, the data analysis unit 212 can determine the front of the three-dimensional model to be inspected. Then, the data analysis unit 212 can arrange the three-dimensional model to be inspected in the virtual space so as to have the same orientation as the three-dimensional model read from the model recording unit 134. The image generation device 100 can generate a virtual viewpoint image of the three-dimensional model to be inspected by using the three-dimensional model to be inspected arranged in this way and the virtual viewpoint parameters read in S805.

[0114] The data analysis unit 212 may compare the virtual viewpoint image based on the model data of the inspection target thus generated with the virtual viewpoint image previously generated by the image generation device 100. When these images are similar, it can be determined that the three-dimensional model used for generating the virtual viewpoint image is similar to the three-dimensional model to be inspected. As described above, the virtual viewpoint image generated by the image generation device 100 can be stored by the image storage unit 144 in association with the sequence and time code. The data analysis unit 212 may obtain the virtual viewpoint image generated by the image generation device 100 from the image storage unit 144. On the other hand, the data analysis unit 212 may restore the virtual viewpoint image generated in the past by using the three-dimensional model to be compared. For example, the data analysis unit 212 can request the image generation device 100 to generate a virtual viewpoint image according to the virtual viewpoint parameters read in S805 and the three-dimensional model. The virtual viewpoint image generated by the image generation device 100 at this time is the same as the virtual viewpoint image generated in the past.

[0115] (Hardware Configuration Example) The above-described image generation device 100, information processing device 101, model generation device 120, and information processing device 200 can be realized by a computer including a processor and a memory. FIG. 9 is a block diagram showing a hardware configuration example of a computer that can be used as the information processing device 101 as an example. Note that the image generation device 100, model generation device 120, and information processing device 200 may be realized by using the computer shown in FIG. 9.

[0116] The CPU 901 controls the entire computer using computer programs or data recorded in the RAM 902 or the ROM 903. Also, the CPU 901 executes each of the processes described above as those performed by the information processing apparatus 101. That is, the CPU 901 can function as each of the processing units shown in FIG. 1.

[0117] The RAM 902 has an area for temporarily storing a computer program or data read from the external storage device 906, or data acquired from the outside via the I / F (interface) 907, etc. Further, the RAM 902 has a work area used when the CPU 901 executes various processes. The RAM 902 can be allocated as, for example, a frame memory. Also, the RAM 902 can store various data such as the data stored in the model recording unit 134, the information recording unit 143, and the image storage unit 144. Further, the RAM 902 may record the virtual viewpoint image or the determination result output by each processing unit. The ROM 903 can record computer setting data or a boot program, etc.

[0118] The operation unit 904 is used for a user to input instructions. The operation unit 904 is, for example, a keyboard or a mouse. By a user of the computer operating the operation unit 904, various instructions can be input to the CPU 901. The output unit 905 is used for outputting information. The output unit 905 is, for example, a liquid crystal display. The output unit 905 can display the processing result obtained by the CPU 901. The operation unit 904 and the output unit 905 are not necessarily required. For example, input of instructions and output of information may be performed using a device connected via the I / F 907.

[0119] The external storage device 906 is a large-capacity information storage device. The external storage device 906 is, for example, a hard disk drive device. The external storage device 906 can store an OS (operating system) and a computer program for causing the CPU 901 to realize the functions of each part shown in FIG. 1. Further, the external storage device 906 may store various image data used for processing. Note that the data described as being recorded by the model recording unit 134 or the information recording unit 143 may be stored in the external storage device 906. The computer program or data stored in the external storage device 906 is loaded into the RAM 902 according to the control by the CPU 901 and used by the CPU 901.

[0120] The I / F 907 can connect a network such as a LAN or the Internet, or other devices such as a projection device or a display device. This computer can acquire or transmit various information via the I / F 907. The bus 908 connects the above-described respective parts.

[0121] As described above, by a processor such as the CPU 901 executing a program stored in a memory such as the RAM 902, the ROM 903, or the external storage device 906, the functions of each part shown in FIG. 1 can be realized. For example, the operations according to the above-described embodiment can be controlled centering around the CPU 901. Note that some or all of the functions of the image generation device 100, the information processing device 101, the model generation device 120, and the information processing device 200 shown in FIG. 2 may be realized by dedicated hardware. Also, an information processing system, an image generation device, an information processing device, or a model generation device according to an embodiment may be configured by a plurality of information processing devices connected via a network, for example. On the other hand, two or more of the functions of the image generation device 100, the information processing device 101, the model generation device 120, and the information processing device 200 may be realized by one device.

[0122] (Other Embodiments) The content of the present disclosure can also be realized by supplying a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that implements one or more functions.

[0123] The disclosure of this specification includes the following information processing system, information processing method, and program. (Item 1) Model acquisition means for acquiring a three-dimensional model of one or more objects at a specified time from storage means for storing a three-dimensional model showing the three-dimensional shape of an object at each time, Generation means for generating a virtual viewpoint image corresponding to the specified time from a virtual viewpoint according to specified virtual viewpoint parameters, using the three-dimensional model of the one or more objects at the specified time, Output means for outputting the virtual viewpoint image generated by the generation means, Determination means for determining whether or not a predetermined condition, which is a condition related to at least one of the time and the virtual viewpoint parameters, is satisfied, Recording means for recording information used for specifying the three-dimensional model of the object corresponding to the virtual viewpoint image generated by the generation means, in response to the satisfaction of the predetermined condition, An information processing system characterized by comprising the above. (Item 2) The information processing system according to Item 1, wherein the recording means records at least one of the time and the virtual viewpoint parameters as information used for specifying the three-dimensional model of the object. (Item 3) The storage means stores sequence data showing three-dimensional models for each of a plurality of times for one or more objects, The recording means records information for specifying the sequence data used to generate the virtual viewpoint image as information used to specify the three-dimensional model of the object, in the information processing system according to any one of items 1 to 2. (Item 4) The recording means further records at least a part of the virtual viewpoint image generated by the generation means, in the information processing system according to any one of items 1 to 3. (Item 5) The recording means further records user information of the user who instructed the generation of the virtual viewpoint image, in the information processing system according to any one of items 1 to 4. (Item 6) The virtual viewpoint parameter includes at least one of the position and orientation of the virtual viewpoint, in the information processing system according to any one of items 1 to 5. (Item 7) The determination means determines whether or not the predetermined condition is satisfied based on at least one of the relationship of the times for each of two or more virtual viewpoint images generated by the generation means and the relationship of the virtual viewpoint parameters for each of the two or more virtual viewpoint images, in the information processing system according to any one of items 1 to 6. (Item 8) The predetermined condition includes a condition regarding the time interval between the times for two of the virtual viewpoint images, in the information processing system according to any one of items 1 to 7. (Item 9) The predetermined condition includes a condition regarding the number of pairs of specific virtual viewpoint images among a plurality of virtual viewpoint images generated by the generation means, and the time interval for each of the pairs of the specific virtual viewpoint images is less than a threshold value, in the information processing system according to any one of items 1 to 8. (Item 10) The information processing system according to any one of items 1 to 8, wherein the predetermined condition includes a condition regarding the number of the virtual viewpoint images corresponding to times included in a predetermined time range among the plurality of virtual viewpoint images generated by the generation means. (Item 11) The information processing system according to any one of items 1 to 10, wherein the predetermined condition includes a condition regarding the positional relationship of the virtual viewpoints for each of two or more virtual viewpoint images generated by the generation means. (Item 12) The information processing system according to any one of items 1 to 11, wherein the predetermined condition includes a condition regarding the similarity of the fixation points or in-view areas of the virtual viewpoints for each of two or more virtual viewpoint images generated by the generation means. (Item 13) The information processing system according to any one of items 1 to 12, wherein the recording means records history information indicating that the virtual viewpoint image has been generated using the three-dimensional model of the object at the time in response to the satisfaction of the predetermined condition. (Item 14) An acquisition means for acquiring a three-dimensional model of an object to be inspected; A comparison means for acquiring, as a three-dimensional model to be compared, the three-dimensional model used in the past to generate a virtual viewpoint image from a storage means for storing a three-dimensional model indicating the three-dimensional shape of an object with reference to the generation record of the virtual viewpoint image, and comparing the three-dimensional model of the object to be inspected with the three-dimensional model to be compared; An information processing system characterized by comprising: (Item 15) The storage means stores three-dimensional models for each of a plurality of times for one or more objects, The generation record of the virtual viewpoint image includes at least one of the time corresponding to the virtual viewpoint image and the virtual viewpoint parameter. The information processing system according to item 14, wherein the comparison means acquires a three-dimensional model that was used to generate the virtual viewpoint image in the past, with reference to at least one of the time corresponding to the virtual viewpoint image and the virtual viewpoint parameters. (Item 16) The information processing system according to any one of items 14 to 15, wherein the comparison means restores the virtual viewpoint image generated in the past using the three-dimensional model to be compared. (Item 17) The information processing system according to any one of items 14 to 16, wherein the generation record of the virtual viewpoint image is information recorded by the recording means of the information processing system according to any one of items 1 to 13. (Item 18) An information processing method performed by an information processing system, a step of acquiring a three-dimensional model of one or more objects at a specified time from a storage means that stores a three-dimensional model showing the three-dimensional shape of the object at each time; a step of generating a virtual viewpoint image corresponding to the specified time from a specified virtual viewpoint according to the specified virtual viewpoint parameters, using the three-dimensional model of the one or more objects at the specified time; a step of outputting the generated virtual viewpoint image; a step of determining whether or not a predetermined condition, which is a condition related to at least one of the time and the virtual viewpoint parameters, is satisfied; a step of recording information used to identify the three-dimensional model of the object corresponding to the generated virtual viewpoint image, in response to the predetermined condition being satisfied; An information processing method, characterized by comprising the above steps. (Item 19) A program for causing a computer to function as the information processing system according to any one of items 1 to 17.

[0124] The invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the claims are appended to disclose the scope of the invention.

Explanation of Signs

[0125] 100: Image generation device, 101: Information processing device, 120: Model generation device, 130: Information input unit, 131: Sequence selection unit, 132: Time specification unit, 133: Parameter specification unit, 134: Model recording unit, 135: Model acquisition unit, 136: Image generation unit, 137: Output unit, 141: User information acquisition unit, 142: Determination unit, 143: Information recording unit, 144: Image storage unit, 200: Information processing device, 210: Data acquisition unit, 212: Data analysis unit, 213: Result output unit

Claims

1. Model acquisition means for acquiring a three-dimensional model of one or more objects at a specified time from storage means for storing a three-dimensional model showing the three-dimensional shape of an object at each time; Generation means for generating a virtual viewpoint image corresponding to the specified time from a virtual viewpoint according to specified virtual viewpoint parameters, using the three-dimensional model of the one or more objects at the specified time; Output means for outputting the virtual viewpoint image generated by the generation means; Determination means for determining whether a predetermined condition, which is a condition related to at least one of the time and the virtual viewpoint parameters, is satisfied; Recording means for recording information used to identify the three-dimensional model of the object corresponding to the virtual viewpoint image generated by the generation means, in response to the predetermined condition being satisfied; An information processing system, characterized by comprising the above.

2. The information processing system according to claim 1, wherein the recording means records at least one of the time and the virtual viewpoint parameters as information regarding the three-dimensional model of the object.

3. The storage means stores sequence data showing three-dimensional models for each of a plurality of times for one or more objects, The information processing system according to claim 1, wherein the recording means records information for specifying the sequence data used to generate the virtual viewpoint image as information used to identify the three-dimensional model of the object.

4. The information processing system according to claim 1, wherein the recording means further records at least a part of the virtual viewpoint image generated by the generation means.

5. The information processing system according to claim 1, wherein the recording means further records user information of a user who instructed generation of the virtual viewpoint image.

6. The information processing system according to claim 1, wherein the virtual viewpoint parameters include at least one of the position and orientation of the virtual viewpoint.

7. The determination means determines whether or not the predetermined condition is satisfied based on at least one of the relationship of the time for each of the two or more virtual viewpoint images generated by the generation means and the relationship of the virtual viewpoint parameters for each of the two or more virtual viewpoint images. The information processing system according to claim 1, characterized in that.

8. The information processing system according to claim 1, characterized in that the predetermined condition includes a condition regarding an interval of time between the two virtual viewpoint images.

9. The predetermined condition includes a condition regarding the number of pairs of specific virtual viewpoint images among the plurality of virtual viewpoint images generated by the generation means, and the interval of time for each of the pairs of the specific virtual viewpoint images is less than a threshold value. The information processing system according to claim 1, characterized in that.

10. The information processing system according to claim 1, characterized in that the predetermined condition includes a condition regarding the number of virtual viewpoint images corresponding to times included in a predetermined time range among the plurality of virtual viewpoint images generated by the generation means.

11. The information processing system according to claim 1, characterized in that the predetermined condition includes a condition regarding the positional relationship of the virtual viewpoints for each of the two or more virtual viewpoint images generated by the generation means.

12. The information processing system according to claim 1, characterized in that the predetermined condition includes a condition regarding the similarity of the fixation points or in-view regions of the virtual viewpoints for each of the two or more virtual viewpoint images generated by the generation means.

13. The recording means records history information indicating that the virtual viewpoint image has been generated using the three-dimensional model of the object at the time in response to the satisfaction of the predetermined condition. The information processing system according to claim 1, characterized in that.

14. An acquisition means for acquiring a three-dimensional model of an object to be inspected; Referring to the generation record of the virtual viewpoint image, a three-dimensional model used to generate the virtual viewpoint image in the past is acquired from a storage means for storing a three-dimensional model showing the three-dimensional shape of the object as a three-dimensional model for comparison, and the three-dimensional model of the inspection object and the three-dimensional model for comparison are compared. A comparison means; An information processing system, characterized by comprising.

15. The storage means stores three-dimensional models for each of a plurality of times for one or more objects, The generation and recording of the virtual viewpoint image includes at least one of the time corresponding to the virtual viewpoint image and the virtual viewpoint parameters, The comparison means refers to at least one of the time corresponding to the virtual viewpoint image and the virtual viewpoint parameters to obtain a three-dimensional model used to generate the virtual viewpoint image in the past. The information processing system according to claim 14, characterized in that.

16. The comparison means restores the virtual viewpoint image generated in the past using the three-dimensional model to be compared. The information processing system according to claim 14, characterized in that.

17. The generation and recording of the virtual viewpoint image is the information recorded by the recording means of the information processing system according to any one of claims 1 to 13. The information processing system according to claim 14, characterized in that.

18. An information processing method performed by an information processing system, Obtaining a three-dimensional model of one or more objects at a specified time from storage means for storing a three-dimensional model indicating the three-dimensional shape of the object at each time; Generating a virtual viewpoint image corresponding to the specified time from a virtual viewpoint according to the specified virtual viewpoint parameters using the three-dimensional model of the one or more objects at the specified time; Outputting the generated virtual viewpoint image; Determining whether a predetermined condition, which is a condition related to at least one of the time and the virtual viewpoint parameters, is satisfied; Recording information used to identify the three-dimensional model of the object corresponding to the generated virtual viewpoint image in response to the satisfaction of the predetermined condition; An information processing method characterized by comprising.

19. A program for causing a computer to function as the information processing system according to any one of claims 1 to 16.

Images