Image processing device, image processing method, and program

Abstract

The system allows for simultaneous viewing of the pose of the object at multiple points in time within a period that includes or is close to the timecode corresponding to the model figure. [Solution] The image processing apparatus 140 according to the present disclosure acquires shooting information including data of a captured image obtained by shooting a model figure corresponding to the shape of an object in an arbitrary time code, and camera parameters corresponding to the captured image, and generates virtual content including an image of the object corresponding to the model figure based on the shooting information.

Description

Technical Field

[0001] The present disclosure relates to an augmented reality (AR) display technology using figures generated based on images.

Background Art

[0002] There is a technology for shaping a 3D model figure by inputting 3D shape data corresponding to an object, which is generated by volumetric capture or three-dimensional (hereinafter referred to as "3D") computer graphics, into a 3D printer. By using a plurality of 3D shape data corresponding to a plurality of different time codes in a scene including a series of movements of the same object, 3D model figures (hereinafter referred to as "figures") corresponding to each time code can be obtained. A plurality of figures shaped based on a series of movements of the same object can be used for analyzing or appreciating the movement of the object in the scene including the series of movements.

[0003] Also, there is a technology for displaying a volumetric video or 3D digital content by superimposing it on a real-world video using a display device capable of expressing AR (augmented reality). Patent Document 1 discloses a technology for displaying a moving image corresponding to a scene during a period including a time code corresponding to a figure by photographing the figure. Specifically, in the technology disclosed in Patent Document 1, an image (hereinafter referred to as a "virtual viewpoint image") corresponding to the view from an arbitrary virtual viewpoint (hereinafter referred to as a "virtual viewpoint") is displayed as a moving image corresponding to the scene. According to the technology (hereinafter referred to as the "conventional technology") disclosed in Patent Document 1, a user can appreciate the movement of an object of interest in a scene related to the figure as a moving image by photographing the figure.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] In conventional technology, scenes from a period including the timecode corresponding to a figure are displayed as moving images. Therefore, conventional technology has the problem that it is difficult to compare and appreciate the posture of the object at multiple points in time within that period. The purpose of this disclosure is to provide a technology that allows simultaneous appreciation of the posture of the object at multiple points in time within a period including the timecode corresponding to a figure, or a period close to that timecode. [Means for solving the problem]

[0006] The image processing apparatus according to this disclosure includes acquisition means for acquiring shooting information including data of a captured image obtained by photographing a model figure corresponding to the shape of an object at an arbitrary time code and camera parameters corresponding to the captured image, and image generation means for generating virtual content including an image of the object corresponding to the model figure based on the shooting information. [Effects of the Invention]

[0007] According to the technology of this disclosure, it is possible to simultaneously view the posture of an object at multiple points in time during a period including a timecode corresponding to a figure, or a period close to such timecode. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the device configuration according to Embodiment 1. [Figure 2] This figure shows an example of an image captured according to Embodiment 1. [Figure 3] This is a diagram illustrating an example of data held in a storage device according to Embodiment 1. [Figure 4] This is a block diagram showing an example of the logical configuration of an image processing device according to Embodiment 1. [Figure 5] This figure shows an example of the arrangement area according to Embodiment 1. [Figure 6] This figure shows an example of the arrangement of virtual viewpoint images according to Embodiment 1. [Figure 7] This is a block diagram showing an example of the hardware configuration of an image processing device according to Embodiment 1. [Figure 8] This flowchart shows an example of the processing flow of the image processing apparatus according to Embodiment 1. [Figure 9] This figure shows an example of the arrangement of captured images and virtual viewpoint images according to Embodiment 2. [Figure 10] This figure shows another example of the arrangement of captured images and virtual viewpoint images according to Embodiment 2. [Modes for carrying out the invention]

[0009] Hereinafter, preferred embodiments of the technology of this disclosure will be described in detail with reference to the attached drawings. Note that the configurations shown in the following embodiments are merely examples, and the technology of this disclosure is not limited to these embodiments.

[0010] [Embodiment 1] This embodiment describes a method for viewing images generated based on captured images obtained by photographing multiple figures placed by a user, etc., using an AR-enabled device.

[0011] <Image processing system configuration> Figure 1 is a block diagram showing an example of the device configuration of the image processing system 100 according to Embodiment 1. The image processing system 100 includes a storage device 110, a molding device 120, a shooting device 130, an image processing device 140, and a display device 150.

[0012] The storage device 110 consists of a hard disk drive or the like and holds various data used for image generation processing in the image processing device 140 and figure manufacturing processing in the modeling device 120. Details of the various data held by the storage device 110 will be described later. The modeling device 120 consists of a 3D printer or the like and takes at least a portion of the various data held by the storage device 110 as input to manufacture a group of figures 170, including figures 171 and 172. Details of figures 171 and 172 manufactured by the modeling device 120 will be described later.

[0013] The imaging device 130 consists of a digital still camera or digital video camera, or a portable terminal such as a smartphone with an imaging function, and photographs the figures 171 and 172 placed by the user. The imaging device 130 outputs imaging information to the image processing device 140, including the data of the captured image obtained by photographing the figures 171 and 172, the camera parameters corresponding to the captured image, and information about the image sensor of the imaging device 130. Figure 2 is a diagram showing an example of captured images 201 to 204 obtained by imaging with the imaging device 130 according to Embodiment 1. In each of the captured images 201 to 204 shown in Figures 2(a) to (d), image 271 is an image of figure 171, and image 272 is an image of figure 172.

[0014] The image processing device 140 is configured by a computer or the like, and generates virtual content based on at least a part of the shooting information output from the shooting device 130 and various data held in the storage device 110. Here, the virtual content is an image including one or more images of objects corresponding to Figures 171 and 172. Hereinafter, the virtual content will be described as including one or more virtual viewpoint images generated based on information indicating the 3D shape of the object. Details and generation methods of the virtual content will be described later. The image processing device 140 outputs the data or signal of the virtual content and the data or signal of the captured image included in the shooting information to the display device 150. The display device 150 is configured by a liquid crystal display or the like, and superimposes and displays the virtual content and the captured image based on the data or signal of the virtual content output from the image processing device 140 and the data or signal of the captured image included in the shooting information.

[0015] In addition, in the present embodiment, each of the imaging device 130, the image processing device 140, and the display device 150 will be described as being mounted in different casings, but the configuration in the technology of the present disclosure is not limited to this. Specifically, at least two or more of the imaging device 130, the image processing device 140, and the display device 150 may be mounted in the same casing and operate as one device. For example, the imaging device 130 and the display device 150 may be mounted in the same casing, or the imaging device 130, the image processing device 140, and the display device 150 may be mounted in the same casing.

[0016] FIG. 3 is a diagram for explaining an example of data held in the memory device 110 according to Embodiment 1. FIG. 3(a) shows an example of object information 300 related to two objects A and B generated by volumetric capture or 3D computer graphics or the like. Hereinafter, an object is something that can be identified like an animal or a natural person and whose posture changes over time, and it will be described that due to the change in the posture of the object over time, the 3D shape corresponding to the object and the texture corresponding to the 3D shape change over time.

[0017] As shown in FIG. 3(a), the 3D shape data is held in association with an ID (identifier) 301 or 302 that can uniquely identify the corresponding object A or B and the data of the corresponding texture. Hereinafter, when a captured image obtained by volumetric capture includes images of a plurality of objects, it will be described that the object to be modeled in the figure or the object to be drawn in the virtual content generated by the image processing device 140 is uniquely identified. For example, an object can be uniquely identified based on the position (coordinates) of the image of the object in the captured image. When there is one object to be modeled in the figure or one object to be drawn in the virtual content, the ID of the object may be omitted. Hereinafter, a data set of the object ID, the 3D shape data, and the texture data associated with each other will be described as 3D model information. Also, since the postures of objects A and B change over time, the 3D model information at each time point is held in association with time codes 303 to 305.

[0018] In this embodiment, the 3D model information is described as being associated with a corresponding time code. However, the association with the 3D model information is not limited to a time code, as long as it is information that can uniquely identify the point in time corresponding to the 3D model information. For example, the 3D model information may be associated with the frame number of the captured image obtained by volumetric capture used to generate the 3D model information.

[0019] Figures 3(b) and 3(c) show examples of molding information 310 and 330 input to the molding device 120. For example, when molding information 310 is input to the molding device 120, the molding device 120 produces figure 171, and when molding information 330 is input to the molding device 120, the molding device 120 produces figure 172. Specifically, for example, molding information 310 and 330 have an object ID field 311, a link destination field 312, and a time code field 313. In addition, molding information 310 and 330 have a base size field 314, a two-dimensional barcode attachment position field 315, and a two-dimensional barcode size field 316.

[0020] The item values ​​317 and 337 of the Object ID field 311 store the ID of the object corresponding to the figure being printed. The item values ​​318 and 338 of the Link Destination field 312 store information such as a URL (Uniform Resource Locator) indicating the location where the object information 300 is stored. The item values ​​319 and 339 of the Time Code field 313 store the time code corresponding to the figure 171 or 172 being printed.

[0021] The item values ​​320 and 340 of the base size field 314 store information indicating the width, height, and depth of the base that supports the figure 171 or 172 to be printed. The item values ​​321 and 341 of the two-dimensional barcode placement field 315 store information indicating the position of the two-dimensional barcode formed on the surface of the figure 171 or 172 to be printed, or on the base that supports the figure 171 or 172. The item values ​​322 and 342 of the two-dimensional barcode size field 316 store information indicating the vertical and horizontal dimensions of the formed two-dimensional barcode.

[0022] When the molding device 120 molds the figure 171, it forms a two-dimensional barcode on the surface of the figure 171 or the base supporting the figure 171, containing the item values ​​317 to 322 corresponding to each field in the molding information 310. In other words, the figure 171 is a figure that shows the 3D shape of object A at the time corresponding to time code 1. Specifically, the molding device 120 forms the two-dimensional barcode at the size specified by the item value 322 of the two-dimensional barcode size field 316 and at the position specified by the item value 321 of the two-dimensional barcode placement position field 315.

[0023] Similarly, when the molding device 120 molds the figure 172, it forms a two-dimensional barcode on the surface of the figure 172 or the base supporting the figure 172, containing the item values ​​337 to 342 corresponding to each field in the molding information 330. That is, the figure 172 is a figure that shows the 3D shape of object A at the time corresponding to time code N. Specifically, the molding device 120 forms the two-dimensional barcode at the position specified by the item value 341 in the two-dimensional barcode placement position field 315, with a size specified by the item value 342 in the two-dimensional barcode size field 316. Hereafter, time code 1 will be described as a time code at a time earlier than the time of time code N.

[0024] Examples of two-dimensional barcodes include QR codes (registered trademark). In this embodiment, a two-dimensional barcode is described as being formed on the surface of the base that supports the figure, but it is not limited to two-dimensional barcodes as long as it can contain the values ​​of each item included in the molding information as information. Specifically, instead of a two-dimensional barcode, it may be implemented by a device such as a barcode, NFC (Near Field Communication), or RFID (radio frequency identification) tag that is formed or implemented when the figure is molded and can acquire the values ​​of each item included in the molding information when the figure is photographed.

[0025] <Logical configuration of the image processing device> Figure 4 is a block diagram showing an example of the logical configuration of the image processing device 140 according to Embodiment 1. The image processing device 140 includes an acquisition unit 400, a region determination unit 401, a generation determination unit 402, a condition determination unit 403, an image generation unit 404, and an output control unit 405. The acquisition unit 400 acquires the shooting information output from the shooting device 130. The acquisition unit 400 also acquires the item values ​​of each field included in the molding information based on the captured image included in the shooting information. Specifically, first, the acquisition unit 400 identifies the image of the two-dimensional barcode molded on the surface of the base supporting the figures 171 and 172, etc., which is included in the captured image. Subsequently, the acquisition unit 400 decodes the two-dimensional barcode identified as an image to acquire the item values ​​of each field of the molding information 310 and 330, which are included as information in the two-dimensional barcode. The generation determination unit 402 determines whether or not virtual content can be generated based on the item values ​​of each field of the molding information 310 and 330 acquired by the acquisition unit 400.

[0026] The region determination unit 401 determines the region (hereinafter referred to as the "placement region") in the virtual content where the virtual viewpoint image, generated based on the 3D model information contained in the object information 300, will be placed. Specifically, first, the region determination unit 401 estimates the placement and orientation of figures 171 and 172 using the captured image and camera parameters included in the shooting information acquired by the acquisition unit 400, and the item values ​​of the molding information 310 and 330. Subsequently, the region determination unit 401 determines the region (placement region) in the virtual content where the virtual viewpoint image will be placed, based on the estimated placement and orientation of figures 171 and 172. Specifically, the region determination unit 401 estimates the orientation and placement of the shooting device 130 and figures 171 and 172 based on the item values ​​of the molding information, the camera parameters included in the shooting information, and the shape, size, and position of the area of ​​the two-dimensional barcode image in the captured image.

[0027] For example, the region determination unit 401 uses information such as the size of the base supporting figures 171 and 172, the placement of the two-dimensional barcode, and the size of the two-dimensional barcode, which are included as item values ​​in the molding information, to estimate the orientation and arrangement of figures 171 and 172. Specifically, the region determination unit 401 estimates the orientation and arrangement of figures 171 and 172 based on the correspondence between the size of the base supporting the figures, the position and size of the two-dimensional barcode, and the shape of the figure image in the captured image and its position in the captured image.

[0028] The information used to estimate the orientation and arrangement of the imaging device 130 and the figures 171 and 172 is not limited to the information described above. For example, if the imaging device 130 has a depth sensor, a gyro sensor, or an acceleration sensor, the imaging device 130 outputs these sensor data along with the imaging information. In this case, the region determination unit 401 estimates the orientation and arrangement of the imaging device 130 and the figures 171 and 172 using the sensor data included in the imaging information in addition to the information described above. Alternatively, for example, the region determination unit 401 may determine the arrangement region based on the position of the region containing the images of figures 171 and 172 in the captured image. In this case, for example, the region determination unit 401 determines the arrangement region as the region sandwiched between the region containing the image of figure 171 in the captured image and the region containing the image of figure 172 in the captured image.

[0029] Figure 5 is a diagram showing an example of arrangement regions 501 to 504 determined by the region determination unit 401 according to Embodiment 1. Specifically, each of Figures 5(a) to (d) shows an example of arrangement regions 501 to 504 corresponding to each of the captured images 201 to 204 shown in Figures 2(a) to (d). In Figures 5(a) to (d), region 571 is the region that includes the image 271 of figure 171 included in the captured images 201 to 204, and region 572 is the region that includes the image 272 of figure 172 included in the captured images 201 to 204.

[0030] The condition determination unit 403 determines the generation conditions for the virtual viewpoint image to be placed in the placement area. Specifically, the condition determination unit 403 determines the time code corresponding to the 3D model information used when generating the virtual viewpoint image, the position of the virtual viewpoint used when generating the virtual viewpoint image using the 3D model information, and the direction of the line of sight at the virtual viewpoint. The condition determination unit 403 also determines the position in the placement area where the generated virtual viewpoint image will be placed. Details of the method for determining the generation conditions for the virtual viewpoint image to be placed in the placement area will be described later.

[0031] The image generation unit 404 generates virtual content. Specifically, first, the image generation unit 404 accesses the object information 300 using the item value of the linked field 312 included in the modeling information 310 or 330. Next, the image generation unit 404 acquires 3D model information from the 3D model information included in the object information 300 that corresponds to the item value of the object ID field 311 and the time code determined by the condition determination unit 403. Next, the image generation unit 404 uses the acquired object information 300 to generate a virtual viewpoint image corresponding to the view from the virtual viewpoint determined by the condition determination unit 403. Finally, the image generation unit 404 generates virtual content by placing the generated virtual viewpoint image at the position determined by the condition determination unit 403 within the placement area determined by the area determination unit 401.

[0032] The output control unit 405 outputs the virtual content data or signal generated by the image generation unit 404 and the captured image data or signal included in the captured information acquired by the acquisition unit 400 to the display device 150. The virtual content and captured image output from the output control unit 405 are superimposed and displayed on the display device 150.

[0033] <Example of placement of virtual viewpoint images in virtual content> Figure 6 is a diagram showing an example of the arrangement of virtual viewpoint images in virtual content generated by the image generation unit 404 according to Embodiment 1. Specifically, Figures 6(a) to (d) each show, in order, examples of virtual viewpoint images 611 to 613, 631, and 641 and 642 arranged in the arrangement areas 501 to 504 shown in Figures 5(a) to (d). Figure 6 represents the AR display state in which the virtual content generated by the image generation unit 404 and the captured image included in the shooting information acquired by the acquisition unit 400 are superimposed. As shown in Figures 6(a), (c), and (d), for example, when the virtual content is superimposed with the captured image on the display device 150, it is displayed between the images 271 and 272 of the two figures 171 and 172. Therefore, the placement areas 501 to 504 can be calculated as the areas sandwiched between the coordinates of area 571 containing the image 271 of figure 171 and area 572 containing the image 272 of figure 172, on a plane corresponding to the display surface of the captured image.

[0034] In the captured image 201 shown in Figure 2(a), the size of the image 271 of figure 171 and the size of the image 272 of figure 172 are approximately the same. In such a case, the image generation unit 404 generates virtual content in which virtual viewpoint images 611 to 614 of approximately the same size as the region 571 containing the image 271 of figure 171 and the region 572 containing the image 272 of figure 172, as shown in Figure 5(a), are placed in the placement region 501. Here, the determination of whether the size of the image 271 of figure 171 and the size of the image 272 of figure 172 are approximately the same can be made, for example, by whether the difference in size between region 571 and region 572 is within ±10%. The determination of the size of the image 271 of figure 171 and the image 272 of figure 172 is not limited to this. For example, the determination may be made based on the size of the images of the bases of figure 171 and figure 172, or the size of the image of the two-dimensional barcode, etc., included in the captured image.

[0035] In the captured image 201, region 571 containing the image 271 of figure 171 and region 572 containing the image 272 of figure 172 are located at separate locations from each other. Therefore, in the arrangement region 501 set between region 571 and region 572, multiple virtual viewpoint images 611 to 614 of roughly the same size as regions 571 and 572 are arranged, for example, at equal intervals. Specifically, the condition determination unit 403 determines the number of virtual viewpoint images that can be placed in the arrangement region 501, and the position in the arrangement region 501 where each virtual viewpoint image is placed, based on the widths of regions 571 and 572 and the width of the arrangement region 501. Here, the widths of regions 571 and 572 are, for example, the average value of the widths of region 571 and region 572.

[0036] Furthermore, the virtual viewpoint images 611 to 614 placed in the placement area 501 are images generated using 3D model information of the time codes included in the period between the time codes of figures 171 and 172, which are included as images in the captured image 201. The time codes of figures 171 and 172 are obtained by decoding the two-dimensional barcodes of figures 171 and 172, which are included as images in the captured image 201. The image generation unit 404 accesses the object information 300, which is designated as the link destination by the two-dimensional barcode, and obtains 3D model information of the time codes included in the period between the time codes of figures 171 and 172. Furthermore, the image generation unit 404 generates virtual viewpoint images based on the acquired 3D model information and the virtual viewpoint determined by the condition determination unit 403, and places each generated virtual viewpoint image at the position determined by the condition determination unit 403. This generates virtual content.

[0037] Here, if the number of 3D model information items that can be acquired is greater than the number of virtual viewpoint images that can be placed in the placement area 501, for example, the image generation unit 404 selects and acquires 3D model information so that the time codes of the 3D model information to be acquired are at approximately equal intervals. On the other hand, if the number of 3D model information items that can be acquired is less than the number of virtual viewpoint images that can be placed in the placement area 501, for example, the image generation unit 404 acquires all 3D model information that satisfies the conditions. In this case, the image generation unit 404 may re-determine the position for each virtual viewpoint image in the placement area 501 based on the number of 3D model information items that can be acquired. With the virtual content generated in this way, the shape or orientation of the object during the period between the time codes of figure 171 and figure 172 can be visualized in a comparable manner.

[0038] Figure 6(b) shows an example where the width of the placement area 502 is smaller than the width of area 571 or area 572. In such a case, it is not possible to place the virtual viewpoint image so that it fits within the placement area 502, and if it were to be placed, the virtual viewpoint image would overlap with the image 271 or 272 of figure 171 or 172. Therefore, the image generation unit 404 does not place the virtual viewpoint image in the placement area 502. Specifically, for example, in such a case, the condition determination unit 403 determines that the number of virtual viewpoint images to be placed in the placement area 502 is 0. Furthermore, the image generation unit 404 does not need to acquire 3D model information for generating the virtual viewpoint image, nor does it need to generate virtual content. However, if the virtual viewpoint image is not displayed on the display device 150 as virtual content, the user may feel uneasy. Therefore, for example, the image generation unit 404 may generate an image (hereinafter referred to as the "notification image") as virtual content in or near the placement area 502 to notify the user that the virtual viewpoint image cannot be displayed because the width of the placement area 502 is too narrow.

[0039] Figure 6(c) shows an example where the width of the placement area 503 is greater than 1 times and less than 2 times the width of area 571 or area 572. In such a case, the condition determination unit 403 determines that the number of virtual viewpoint images to be placed in the placement area 503 is 1, and determines that the placement position is approximately in the center of the placement area 503. For example, the image generation unit 404 acquires 3D model information of the time code at approximately the midpoint in the period between the time codes of figures 171 and 172 included as images in the captured image 203, and generates a virtual viewpoint image corresponding to the said 3D model information. Subsequently, the image generation unit 404 places the generated virtual viewpoint image approximately in the center of the placement area 503.

[0040] The virtual viewpoint images generated by the image generation unit 404 in such cases are not limited to those shown. For example, the image generation unit 404 may acquire 3D model information for multiple time codes included in the period from the time code of figure 171 to the time code of figure 172, and generate virtual viewpoint images corresponding to each piece of 3D model information. In this case, for example, the image generation unit 404 arranges the generated virtual viewpoint images alternately in the arrangement area 503 so that they are in chronological order of the time codes. With such alternating chronological arrangement, the virtual viewpoint images are displayed in the virtual content like a stop-motion video or a continuous animation, making changes in the shape or posture of objects during that period visible. Note that the alternating arrangement of virtual viewpoint images described above can also be performed when virtual viewpoint images are arranged at multiple positions in the arrangement area. When virtual viewpoint images are arranged at multiple positions in the arrangement area, it is desirable for the image generation unit 404 to arrange the virtual viewpoint images so that at each position where the virtual viewpoint images are arranged, virtual viewpoint images corresponding to 3D model information of different time codes are displayed.

[0041] Figure 6(d) shows an example where the size of the region 571 containing the image 271 of figure 171 and the size of the region 572 containing the image 272 of figure 172 are different in the captured image 204. In the captured image 204, the region 572 containing the image 272 of figure 172 is larger than the region 571 containing the image 271 of figure 171. Such a difference in size may be due to a difference in the size of the molded figures themselves, or it may be due to a difference in the distance between the imaging device 130 and figures 171 and 172. When the sizes of the regions containing the images of each figure in the captured image are different, as in the captured image 204, it is desirable that the virtual viewpoint image placed in the placement region 504 be sized according to the size of that region.

[0042] For example, in such a case, the image generation unit 404 determines the size of the virtual viewpoint image to be placed to a size that interpolates the difference in size between regions 571 and 572, which contain the images 271 and 272 of figures 171 and 172. Specifically, in the captured image 204, the width of region 572 containing the image 272 of figure 172 is larger than the width of region 571 containing the image 271 of figure 171. Assuming that the number of virtual viewpoint images to be placed in the placement region 504 is N, and that the 0th virtual viewpoint image is placed in region 571 and the (N+1)th virtual viewpoint image is placed in region 572, the width of each virtual viewpoint image can be determined to be such that it forms a geometric sequence with a predetermined common ratio. Here, the common ratio R of the geometric sequence can be calculated, for example, by the following formula (1).

[0043]

number

[0044] <Hardware configuration of the image processing device> Figure 7 is a block diagram showing an example of the hardware configuration of an image processing device 140 according to Embodiment 1. The image processing device 140 has a CPU 701, ROM 702, RAM 703, operation device 704, display device 705, storage device 706, and communication I / F (interface) 707 as its hardware configuration. Each part of the image processing device 140's hardware configuration is connected to each other via a bus 708 so as to be able to communicate with one another.

[0045] The CPU 701 controls the entire image processing device 140 by executing the computer program using the computer program and various data stored in the RAM 702, ROM 703, or storage device 706. In other words, each part of the image processing device 140 shown in Figure 4 is realized by the CPU 701 executing the computer program. The RAM 702 is a volatile memory that temporarily stores the computer program and various data loaded from the ROM 702 or storage device 706, as well as data acquired from the outside via the communication I / F 707. The RAM 702 operates as a work area used by the CPU 701 when executing various processes. The RAM 702 can also be allocated as a frame memory, or it can store various data such as shooting information and camera parameters. The RAM 702 also manages shooting information and molding information. The ROM 703 is a non-volatile memory that stores setting data related to the image processing device 140, as well as a boot program.

[0046] The operating device 704 consists of a keyboard or mouse, and receives operations from the user of the image processing device 140 and inputs various instructions corresponding to those operations to the CPU 701. The display device 705 consists of a liquid crystal display, and displays the results of processing by the CPU 701. The CPU 701 also operates as a control unit for the operating device 704 and the display device 705. In this embodiment, the image processing device 140 is described as having the operating device 704 and the display device 705, but the image processing device 140 does not have to have at least one of the operating device 704 and the display device 705. Specifically, for example, at least one of the operating device 704 and the display device 705 may be connected as an external device to the image processing device 140 via a communication I / F 1707.

[0047] The storage device 706 is a large-capacity information storage device, such as a hard disk drive. The storage device 706 may store, for example, an OS (operating system) and computer programs that enable the CPU 701 to implement the functions of each part of the image processing device 140 shown in Figure 4. The storage device 706 may also store various image data, shooting information, and molding information to be processed. The computer programs and various data stored in the ROM 702 or storage device 706 are loaded into the RAM 702 as appropriate, according to the control of the CPU 701, and become the target of processing by the CPU 701. The communication interface 707 is an interface to which a network such as a LAN (Local Area Network) or the Internet, as well as external devices such as the shooting device 130 and the display device 150, are connected. The image processing device 140 can acquire and output various information from external sources via the communication interface 707.

[0048] <Operation of the image processing device> Figure 8 is a flowchart showing an example of the processing flow of the image processing apparatus 140 according to Embodiment 1. The processing shown in the flowchart in Figure 8 is realized by the CPU 701 loading a computer program stored in the ROM 702 or storage device 706, etc., into the RAM 703 and executing it. In the following description, the symbol "S" preceding a reference numeral means a processing step.

[0049] First, in S801, the acquisition unit 400 acquires photographic information. The photographic image included in the photographic information includes, for example, images of the two-dimensional barcodes formed on the bases of figures 171 and 172, as shown in Figure 2(b). The acquisition unit 400 acquires the molding information 310 shown in Figure 3(b) by decoding the two-dimensional barcode of figure 171, which is included as an image in the photographic image. The acquisition unit 400 also acquires the molding information 330 shown in Figure 3(c) by decoding the two-dimensional barcode of figure 172, which is included as an image in the photographic image. The molding information 310 and 330 obtained by the acquisition unit 400 decoding the two-dimensional barcodes is transmitted to the generation determination unit 402. In addition, the photographic information acquired by the acquisition unit 400, as well as information indicating the size of the base, the placement position of the two-dimensional barcode, and the size of the two-dimensional barcode obtained by the acquisition unit 400 decoding the two-dimensional barcode, are transmitted to the area determination unit 401.

[0050] Next, in S802, the generation determination unit 402 determines whether or not virtual content can be generated using the modeling information 310 and 330 acquired in S801. Specifically, the generation determination unit 402 compares the item values ​​318 and 324 of the linked field 312 and the item values ​​317 and 323 of the object ID field 311 included in the modeling information 310 or 330. If these values ​​are the same, and the item values ​​319 and 339 of the timecode field 313 are different, then the captured image contains images of multiple types of figures corresponding to the same object. In this case, the generation determination unit 402 determines that virtual content can be generated; otherwise, the generation determination unit 402 determines that virtual content cannot be generated. If it is determined in S802 that virtual content can be generated, the image processing device 140 executes the process in S803; otherwise, the image processing device 140 terminates the process shown in the flowchart in Figure 8.

[0051] In S803, the region determination unit 401 determines the arrangement area by estimating the arrangement and orientation of the shooting device 130 and the figures 171 and 172. Specifically, the region determination unit 401 estimates the arrangement and orientation of the shooting device 130 and the figures 171 and 172 using the camera parameters, image sensor information, and captured image 201 included in the shooting information acquired in S801. For example, the region determination unit 401 uses the camera parameters and image sensor information to associate the position and shape of the figure base and the two-dimensional barcode detected from the captured image 201 with the actual size of the figures. This estimates the arrangement and orientation of the shooting device 130 and the figures 171 and 172. Here, the actual size of the figures can be determined using the shape information of the figure base and the size and attachment position information of the two-dimensional barcode included in the molding information 310 and 330. For example, if a figure is formed on a rectangular base, the orientation of the figure can be estimated from the orientation of the base and the position where the two-dimensional barcode is attached.

[0052] Furthermore, the estimation of the figure's orientation is not limited to estimation based on the orientation of the base or the placement of the two-dimensional barcode. For example, the region determination unit 401 may estimate the figure's orientation by identifying the texture or shape features of the figure in the captured image. Also, for example, if multiple types of two-dimensional barcodes are pre-formed on different surfaces of the base by the molding device 120, the region determination unit 401 identifies the correspondence with each two-dimensional barcode included as an image in the captured image. The figure's orientation can also be estimated by such a method. The information of the placement area determined by the region determination unit 401 is transmitted to the condition determination unit 403. In addition, information of regions 571 and 572, including images 271 and 272 of figures 171 and 172 in the captured image 201, obtained by image analysis by the region determination unit 401, is also transmitted to the condition determination unit 403.

[0053] Following S803, in S804, the condition determination unit 403 determines whether or not one or more virtual viewpoint images can be placed in the placement area, based on the information of the placement area determined in S803, as well as the information of areas 571 and 572. Specifically, the condition determination unit 403 determines the number of virtual viewpoint images to be placed in the placement area based on the sizes of areas 571 and 572 and the size of the placement area determined in S803. If the determined number is one or more, it is determined that one or more virtual viewpoint images can be placed in the placement area; if the determined number is zero, it is determined that one or more virtual viewpoint images cannot be placed in the placement area. If it is determined in S804 that one or more virtual viewpoint images can be placed in the placement area, the image processing device 140 executes the processes in S805 to S809; otherwise, the image processing device 140 executes the process in S811.

[0054] In S811, the image generation unit 404 generates a notification image as virtual content to notify the user that the virtual viewpoint image cannot be displayed due to reasons such as the width of the placement area 502 being too narrow. The notification image is not limited to notifying the user that the virtual viewpoint image cannot be displayed, and may also include a message prompting the user to change the placement of at least one of the figures 171 and 172. After S811, the image processing device 140 executes the process of S810.

[0055] In S805, the condition determination unit 403 determines the position in the placement area where the virtual viewpoint images will be placed. Specifically, the condition determination unit 403 uses the number of virtual viewpoint images to be placed in the placement area determined in S804, and the size information of the placement area, area 571, and area 572 obtained in S803 to determine the position in the placement area where the virtual viewpoint images will be placed. Next, in S806, the condition determination unit 403 uses the molding information 310 and 330 obtained by decoding the two-dimensional barcode in S801 to determine the time code of the 3D model information used to generate the virtual viewpoint images to be placed. Specifically, first, the condition determination unit 403 identifies a period starting with the item value 319 of the time code field 313 included in the molding information 310 and ending with the item value 325 of the time code field 313 included in the molding information 330. Next, the condition determination unit 403 divides the period into equal parts using the number of virtual viewpoint images to be placed in the arrangement area determined in S804, and calculates and determines the same number of time codes as that number.

[0056] Next, in S807, the condition determination unit 403 uses the arrangement and orientation of figures 171 and 172 obtained in S803, as well as information on camera parameters and image sensors, to determine the position of the virtual viewpoint and the direction of the line of sight used to generate the virtual viewpoint image. The position of the virtual viewpoint and the direction of the line of sight may be determined as fixed values ​​that do not change over time, or as a virtual camera path that changes over time. When the position of the virtual viewpoint and the direction of the line of sight are determined as fixed values, for example, the condition determination unit 403 determines the position of the virtual viewpoint and the direction of the line of sight as follows. For example, the condition determination unit 403 determines the position of the virtual viewpoint and the direction of the line of sight to be such that the orientation of the object included as an image in the generated virtual viewpoint image is perceived to substantially coincide with the orientation of figure 171 or 172 included as an image in the captured image. If the orientation of figure 171 and the orientation of figure 172 are significantly different, the condition determination unit 403 may determine the position of the virtual viewpoint and the direction of line of sight as the result of calculating the average value of the virtual viewpoint position and the direction of line of sight determined based on the orientation of each figure.

[0057] Furthermore, when the position and direction of the line of sight of a virtual viewpoint are determined as a virtual camera path, the region determination unit 401 determines the virtual camera path as follows, for example. For example, the region determination unit 401 determines the position and direction of the line of sight of the virtual viewpoint at the start of the virtual camera path to be a position and direction such that the orientation of the object included as an image in the virtual viewpoint image is perceived to be approximately the same as the orientation of the figure 171 included as an image in the captured image. The region determination unit 401 also determines the position and direction of the line of sight of the virtual viewpoint at the end of the virtual camera path to be a position and direction such that the orientation of the object included as an image in the virtual viewpoint image is perceived to be approximately the same as the orientation of the figure 172 included as an image in the captured image. When the position and direction of the line of sight of a virtual viewpoint are determined as a virtual camera path, the virtual content can be generated as a moving image for a period corresponding to the virtual camera path. Furthermore, during the period from the start to the end of the virtual camera path, the region determination unit 401 may determine a virtual camera path in which the position and direction of the line of sight of the virtual viewpoint are continuously connected linearly or by a predetermined curve.

[0058] Following S807, in S808, the image generation unit 404 generates a virtual viewpoint image. Specifically, first, the image generation unit 404 accesses the object information 300 and obtains the 3D model information of the time code determined in S806. Next, the image generation unit 404 generates a virtual viewpoint image by drawing the 3D model information based on the position and direction of the line of sight of the virtual viewpoint determined in S807. Then, in S809, the image generation unit 404 generates virtual content by placing the virtual viewpoint image generated in S808 at the position determined in S805 within the placement area determined in S803. After S809, the image processing device 140 executes the process in S810.

[0059] In S810, the output control unit 405 outputs the virtual content generated in S809 to the display device 150. The output control unit 405 also outputs the captured image acquired in S801 to the display device 150. The virtual content output in S810 is superimposed on the captured image and displayed on the display device 150. If the process in S811 was executed, in S810, the output control unit 405 outputs the notification image generated as virtual content in S811 to the display device 150. In this case, the notification image output in S810 is superimposed on the captured image and displayed on the display device 150. After S810, the image processing device 140 terminates the process shown in the flowchart in Figure 8.

[0060] As described above, Embodiment 1 describes a method for photographing a state in which multiple figures of the same object, each corresponding to a different time code, are arranged. According to the image processing device 140 of Embodiment 1, by adjusting the positions of the multiple figures, images of the object corresponding to the time codes of figures that have not been photographed can be displayed as virtual content. This allows the user to simultaneously view the shape or orientation of the target object at multiple points in time during a period including the time code corresponding to the figure, or a period close to that time code. Furthermore, the user can adjust the position or orientation of the figures through intuitive operation. This allows the user to easily view the display of images of the object corresponding to the time codes of figures that have not been photographed through intuitive operation.

[0061] [Embodiment 2] Embodiment 1 described a method of changing the form of generated virtual content by changing the number of virtual viewpoint images to be placed based on the positional relationship of multiple figures. Embodiment 2 describes the virtual content generated when one figure is placed and photographed. Note that the device configuration of the image processing system according to Embodiment 2 is the same as the device configuration of the image processing system 100 according to Embodiment 1 shown in Figure 1, so the explanation is omitted. Also, the hardware configuration of the image processing device according to Embodiment 2 is the same as the hardware configuration of the image processing device 140 according to Embodiment 1, so the explanation is omitted. Furthermore, the logical configuration of the image processing device according to Embodiment 2 is generally the same as the logical configuration of the image processing device 140 according to Embodiment 1, but some processing in the logical configuration differs from that of the image processing device 140 according to Embodiment 1. Hereinafter, the image processing device according to Embodiment 2 will be referred to as "image processing device 140," and the differences from the processing in the image processing device 140 according to Embodiment 1 will be explained in detail.

[0062] Figure 9 shows an example of the arrangement of a captured image included in the shooting information acquired by the acquisition unit 400 and a virtual viewpoint image in the virtual content generated by the image generation unit 404, according to Embodiment 2. Specifically, Figures 9(a) and (b) show examples of captured images 901 and 902 acquired by the acquisition unit 400, and captured images 901 and 902 include an image 271 of figure 171. Figures 9(c) and (d) show, respectively, an example of virtual content generated by the image generation unit 404 based on the captured image 901 and an example of virtual content generated by the image generation unit 404 based on the captured image 901. Figures 9(c) and (d) represent an AR display state in which the virtual content generated by the image generation unit 404 and the captured image 901 or 902 included in the shooting information acquired by the acquisition unit 400 are superimposed. As shown in Figure 9(c) or (d), virtual viewpoint images 911, 912, 921 to 923, and 931 to 936 are placed in the arrangement areas 910, 920, or 930.

[0063] The region determination unit 401 determines the region (placement region) in which the virtual viewpoint image, generated based on the 3D model information included in the object information 300, is placed in the virtual content. Specifically, first, the region determination unit 401 estimates the placement and orientation of the figure 171 using the captured image 901 or 902 included in the shooting information acquired by the acquisition unit 400, the camera parameters, and the item values ​​of the modeling information 310. The method for estimating the placement and orientation of the figure 171 is the same as the estimation method in the region determination unit 401 according to Embodiment 1, so a description is omitted. Next, the region determination unit 401 determines the placement regions 910 and 920 or the placement region 930 based on the estimated placement and orientation of the figure 171. Alternatively, the region determination unit 401 may determine the placement regions 910 and 920 or the placement region 930 based on the position of the region 571 including the image 271 of the figure 171 in the captured image 901 or 902. In this case, for example, the region determination unit 401 determines the regions adjacent to the region 571 containing the image 271 of figure 171 in the captured image 901 or 902 as placement regions 910 and 920 or placement region 930.

[0064] The condition determination unit 403 determines the generation conditions for virtual viewpoint images to be placed in the placement areas 910 and 920 or placement area 930. Specifically, the condition determination unit 403 determines the time code corresponding to the 3D model information used when generating the virtual viewpoint images, the position of the virtual viewpoint used when generating the virtual viewpoint images using the 3D model information, and the direction of the line of sight at the virtual viewpoint. The condition determination unit 403 also determines the positions in the placement areas where the generated virtual viewpoint images will be placed. The number of virtual viewpoint images and the method for determining the position of each virtual viewpoint image are the same as those of the condition determination unit 403 in Embodiment 1, so a description is omitted.

[0065] For example, in the case of placement areas 910 and 920, the condition determination unit 403 first selects and determines the number of time codes to be placed in placement area 910 from among the time codes before item value 319 of the time code field 313 included in the molding information 310. Next, the condition determination unit 403 selects and determines the number of time codes to be placed in placement area 920 from among the time codes after item value 319 of the time code field 313 included in the molding information 310. Similarly, in the case of placement area 930, for example, the condition determination unit 403 selects and determines the number of time codes to be placed in placement area 930 from among the time codes after item value 319 of the time code field 313 included in the molding information 310. The interval between the selected time codes may be any interval set in advance, or an interval specified by the user, etc. Furthermore, the condition determination unit 403 determines the position and direction of the virtual viewpoint such that the orientation of the object included as an image in the generated virtual viewpoint image substantially coincides with the orientation of the figure 171 included as an image in the captured image.

[0066] Figures 9(c) and 9(d) show, as an example, an arrangement area in which virtual viewpoint images are arranged side by side in the left-right direction of the figure, but the shape and arrangement of the arrangement area are not limited to this. Figure 10 is a diagram showing another example of the arrangement of virtual viewpoint images in the virtual content generated by the image generation unit 404 and the captured image included in the captured information acquired by the acquisition unit 400, according to Embodiment 2. Specifically, Figure 10(a) shows a captured image 1001 acquired by the acquisition unit 400, and the captured image 1001 includes an image 271 of figure 171.

[0067] Figure 10(b) shows an example of virtual content generated by the image generation unit 404 based on the captured image 1001. Figure 10(b) shows an AR display state in which the virtual content generated by the image generation unit 404 and the captured image 1001 included in the capture information acquired by the acquisition unit 400 are superimposed. As shown in Figure 10(b), virtual viewpoint images 1011 to 1013 and 1021 to 1023 are arranged in the arrangement area 1010 or 1020. The area determination unit 401 may determine an area as the arrangement area that extends in the vertical or diagonal direction of the figure, rather than in the horizontal direction of the figure, as shown as an example in Figure 10(b).

[0068] Furthermore, while Figures 9(c) and (d) and Figure 10(b) show, as an example, placement areas where the size of each virtual viewpoint image is approximately the same, the placement area determined by the area determination unit 401 is not limited to such areas. For example, the area determination unit 401 may determine a trapezoidal area as the placement area, as shown in the placement area 504 in Figure 6(d).

[0069] With the image processing device 140 configured as described above, it is possible to view the shape or posture of an object corresponding to a figure at multiple points in time simultaneously, even when multiple figures are not placed. Therefore, even users who have acquired only one figure through purchase or other means may be provided with an enhanced viewing experience.

[0070] [Other embodiments] In the embodiments described above, a form was described in which the generated virtual content is superimposed on the captured image and displayed; however, only the virtual content may be displayed. In this case, for example, the image generation unit 404 may generate a virtual viewpoint image corresponding to the time code of the figure included as an image in the captured image using 3D model information corresponding to the time code of the figure, and place the generated virtual viewpoint image at the position of the figure's image.

[0071] Furthermore, in the above-described embodiment, the image generation unit 404 generates a virtual viewpoint image corresponding to the time code determined by the condition determination unit 403 using 3D model information corresponding to the time code, and places the generated virtual viewpoint image in the placement area. However, the image to be placed in the placement area is not limited to a virtual viewpoint image. For example, the image generation unit 404 may cut out the area of ​​the image of the object corresponding to the figure from the captured image corresponding to the time code determined by the condition determination unit 403, and replace the virtual viewpoint image with the cut-out area and place it in the placement area.

[0072] Furthermore, in the above-described embodiment, a configuration was described in which the virtual content and the captured image are output to the display device 150, and the display device 150 displays an AR image in which the virtual content is superimposed on the captured image. However, the image processing device 140 may generate and output an AR image in which the virtual content is superimposed on the captured image, and the display device 150 may display the AR image output from the image processing device 140.

[0073] The technology of this disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by processing hardware such as an ASIC (Application-Specific Integrated Circuit) that implements one or more functions.

[0074] Furthermore, within the scope of this disclosure, the technologies described herein allow for free combination of each embodiment, modification of any component of each embodiment, or omission of any component in each embodiment.

[0075] [Technical Features of This Disclosure] This disclosure includes the following configurations, methods, and programs.

[0076] <Configuration 1> Acquisition means for acquiring shooting information including data of a captured image obtained by photographing a model figure corresponding to the shape of an object at an arbitrary time code, and camera parameters corresponding to said captured image, Image generation means for generating virtual content including an image of the object corresponding to the model figure based on the aforementioned shooting information, An image processing apparatus characterized by having

[0077] <Configuration 2> Area determination means for determining the placement area based on the aforementioned imaging information, It further possesses, The image generation means generates the virtual content by arranging the image of the object in the arrangement area. An image processing apparatus according to configuration 1, characterized by the above.

[0078] <Structure 3> The region determination means determines the arrangement region based on the position of the region in the captured image included in the captured information that contains the image of the model figure. The image processing apparatus according to configuration 2, characterized by the above.

[0079] <Structure 4> If the captured image included in the shooting information includes an image of a first model figure corresponding to the shape of the object in a first time code and an image of a second model figure corresponding to the shape of the object in a second time code different from the first time code, the region determination means determines the region between the region containing the image of the first model figure and the region containing the image of the second model figure in the captured image as the arrangement region. An image processing apparatus according to configuration 3, characterized by the above.

[0080] <Composition 5> A generation determination means that determines whether or not the virtual content can be generated based on the aforementioned shooting information. It further possesses, The image generation means generates the virtual content when it is determined that it is possible to generate the virtual content. An image processing apparatus according to any one of configurations 1 to 4 characterized by the above.

[0081] <Composition 6> The generation determination means determines that the virtual content cannot be generated if the captured image included in the captured information contains images of multiple model figures corresponding to each of multiple different objects. The image processing apparatus described in configuration 5, characterized by the above.

[0082] <Composition 7> A timecode determination means that determines the timecode of the image of the object to be included in the virtual content based on the timecode of the model figure, Furthermore, to have, An image processing apparatus according to any one of configurations 1 to 6 characterized by the above.

[0083] <Structure 8> If the captured image included in the captured information includes an image of a first model figure corresponding to the shape of the object in a first timecode, and an image of a second model figure corresponding to the shape of the object in a second timecode different from the first timecode, the timecode determination means determines the timecode between the first timecode and the second timecode as the timecode of the image of the object to be included in the virtual content. The image processing apparatus according to configuration 7, characterized by the above.

[0084] <Composition 9> A region determination means for determining the placement region based on the aforementioned imaging information, A number determination means for determining the number of images of the object to be placed in the placement area, based on the size of the area containing the image of the model figure in the captured image included in the capture information and the size of the placement area, Furthermore, to have, An image processing apparatus according to any one of configurations 1 to 8 characterized by the above.

[0085] <Composition 10> The image generation means generates a notification image to give a predetermined notification to the user when the number of images of the determined object satisfies predetermined conditions. An image processing apparatus according to configuration 9, characterized by the above.

[0086] <Composition 11> The image generation means, when the number of determined images of the object satisfies a predetermined condition, alternately places the images of the object corresponding to each of the multiple time codes at the same position in the virtual content. An image processing apparatus according to configuration 9 or 10, characterized by the above.

[0087] <Composition 12> The image of the object to be included in the virtual content is a virtual viewpoint image that corresponds to the view from a virtual viewpoint, generated based on three-dimensional shape data showing the shape of the object. An image processing apparatus according to any one of configurations 1 to 11 characterized by the above.

[0088] <Composition 13> The position and direction of the virtual viewpoint are determined based on the position and orientation of the model figure, which are estimated based on the captured image and camera parameters included in the shooting information. An image processing apparatus according to configuration 12, characterized by the above.

[0089] <Composition 14> The image of the object to be included in the virtual content is an image obtained by extracting a region containing the image of the object from an image obtained by photographing the object. An image processing apparatus according to any one of configurations 1 to 13 characterized by the above.

[0090] <Composition 15> The image generation means generates an image by superimposing the generated virtual content onto the captured image included in the capture information. An image processing apparatus according to any one of configurations 1 to 14 characterized by the above.

[0091] <Method> An acquisition step of acquiring shooting information including data of a captured image obtained by photographing a model figure corresponding to the shape of an object at an arbitrary time code, and camera parameters corresponding to the captured image, Image generation step of generating virtual content including an image of the object corresponding to the model figure based on the aforementioned shooting information, An image processing method characterized by including

[0092] <Program> A program for causing a computer to function as an image processing device described in any one of configurations 1 to 15. [Explanation of Symbols]

[0093] 140 Image Processing Devices 400 Acquisition Department 404 Image Generation Unit

Claims

1. Acquisition means for acquiring shooting information including data of a captured image obtained by photographing a model figure corresponding to the shape of an object at an arbitrary time code, and camera parameters corresponding to said captured image, Image generation means for generating virtual content including an image of the object corresponding to the model figure based on the aforementioned shooting information, An image processing apparatus characterized by having

2. Area determination means for determining the placement area based on the aforementioned imaging information, It further possesses, The image generation means generates the virtual content by arranging the image of the object in the arrangement area. The image processing apparatus according to claim 1, characterized in that

3. The region determination means determines the arrangement region based on the position of the region in the captured image included in the captured information that contains the image of the model figure. The image processing apparatus according to claim 2, characterized by the above.

4. If the captured image included in the shooting information includes an image of a first model figure corresponding to the shape of the object in a first time code and an image of a second model figure corresponding to the shape of the object in a second time code different from the first time code, the region determination means determines the region between the region containing the image of the first model figure and the region containing the image of the second model figure in the captured image as the arrangement region. The image processing apparatus according to claim 3, characterized by the following:

5. A generation determination means that determines whether or not the virtual content can be generated based on the aforementioned shooting information. It further possesses, The image generation means generates the virtual content when it is determined that it is possible to generate the virtual content. The image processing apparatus according to claim 1, characterized in that

6. The generation determination means determines that the virtual content cannot be generated if the captured image included in the captured information contains images of multiple model figures corresponding to each of multiple different objects. The image processing apparatus according to claim 5, characterized by the following:

7. A timecode determination means that determines the timecode of the image of the object to be included in the virtual content based on the timecode of the model figure, Furthermore, to have, The image processing apparatus according to claim 1, characterized in that

8. If the captured image included in the captured information includes an image of a first model figure corresponding to the shape of the object in a first timecode, and an image of a second model figure corresponding to the shape of the object in a second timecode different from the first timecode, the timecode determination means determines the timecode between the first timecode and the second timecode as the timecode of the image of the object to be included in the virtual content. The image processing apparatus according to claim 7, characterized by the following:

9. A region determination means for determining the placement region based on the aforementioned imaging information, A number determination means for determining the number of images of the object to be included in the virtual content based on the size of the area containing the image of the model figure in the captured image included in the capture information and the size of the arrangement area, Furthermore, to have, The image processing apparatus according to claim 1, characterized in that

10. The image generation means generates a notification image to give a predetermined notification to the user when the number of images of the determined object satisfies predetermined conditions. The image processing apparatus according to claim 9, characterized by the above.

11. The image generation means, when the number of determined images of the object satisfies a predetermined condition, alternately places the images of the object corresponding to each of the multiple time codes at the same position in the virtual content. The image processing apparatus according to claim 9, characterized by the above.

12. The image of the object to be included in the virtual content is a virtual viewpoint image that corresponds to the view from a virtual viewpoint, generated based on three-dimensional shape data showing the shape of the object. The image processing apparatus according to claim 1, characterized in that

13. The position and direction of the virtual viewpoint are determined based on the position and orientation of the model figure, which are estimated based on the captured image and camera parameters included in the shooting information. The image processing apparatus according to claim 12, characterized in that

14. The image of the object to be included in the virtual content is an image obtained by extracting a region containing the image of the object from an image obtained by photographing the object. The image processing apparatus according to claim 1, characterized in that

15. The image generation means generates an image by superimposing the generated virtual content onto the captured image included in the capture information. The image processing apparatus according to claim 1, characterized in that

16. An acquisition step of acquiring shooting information including data of a captured image obtained by photographing a model figure corresponding to the shape of an object at an arbitrary time code, and camera parameters corresponding to the captured image, Image generation step of generating virtual content including an image of the object corresponding to the model figure based on the aforementioned shooting information, An image processing method characterized by including

17. A program for causing a computer to function as an image processing device according to any one of claims 1 to 15.

Images