Image processing method, information processing apparatus, computer program, and storage medium

Abstract

The present application provides an image processing method, an information processing device, a computer program and a storage medium, which can efficiently perform reconstruction processing when an object is photographed and a dynamic image is reconstructed using an image of the photographed object. An image processing method for reconstructing a dynamic image includes the following steps: a processing unit generates first three-dimensional model data of a model object included in an image according to a plurality of images generated by a photographing unit photographing the appearance of the model object from multiple directions; the processing unit reconstructs the dynamic image by replacing second three-dimensional model data in template data of the dynamic image, which is previously configured to include the second three-dimensional model data, with the first three-dimensional model data; and an output unit outputs information indicating an acquisition destination of the reconstructed dynamic image.

Description

TECHNICAL FIELD

[0001] The present application relates to an image processing method, an information processing apparatus, and a computer program. BACKGROUND

[0002] Patent Literature 1 describes a technique of mapping a texture generated from a group of photographic images obtained by photographing an object from a plurality of photographing directions by a photographing section to a primitive in a virtual space, to generate a virtual space image viewed from a virtual camera in the virtual space.

[0003] PRIOR ART DOCUMENT

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2020-107251 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] In the above-described technique, it is desirable to efficiently perform reconstruction processing when an object is photographed and a dynamic image is reconstructed using an image of the photographed object.

[0008] Therefore, it is made possible to efficiently perform reconstruction processing when an object is photographed and a dynamic image is reconstructed using an image of the photographed object.

[0009] SOLUTION TO PROBLEM

[0010] One way to solve the above problem is an image processing method for reconstructing a dynamic image, including the following processes: a processing section generates first three-dimensional model data of a model object included in an image generated from a plurality of images obtained by photographing an appearance of the model object from a plurality of directions by a photographing section; the processing section reconstructs the dynamic image by replacing second three-dimensional model data included in template data of the dynamic image, which is configured in advance to include the second three-dimensional model data, with the first three-dimensional model data; and an output section outputs information indicating an acquisition destination of the reconstructed dynamic image.

[0011] EFFECT OF THE INVENTION

[0012] When an object is photographed and a dynamic image is reconstructed using an image of the photographed object, reconstruction processing can be efficiently performed. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a diagram showing a configuration example of an image processing system corresponding to the embodiments, and a diagram showing an example of a hardware configuration of the information processing apparatus 100.

[0014] Figure 2 Fig. 1 is an example of a diagram showing the appearance of a model corresponding to the embodiment.

[0015] Figure 3 Fig. 2 is an example of a diagram showing an example of installation of an image processing system and an AR marker corresponding to the embodiment.

[0016] Figure 4 Fig. 3 is an example of a diagram showing the data structure of a data table corresponding to the embodiment.

[0017] Figure 5 Fig. 4 is a flowchart corresponding to an example of a process for generating PV data corresponding to the embodiment.

[0018] Figure 6 Fig. 5 is a flowchart corresponding to an example of a photographing process corresponding to the embodiment.

[0019] Figure 7 Fig. 6 is a flowchart showing an example of a process of PV reconstruction corresponding to the embodiment.

[0020] Figure 8 Fig. 7 is a diagram for explaining a photographing environment when an image of a model corresponding to the embodiment is captured.

[0021] Figure 9 Fig. 8 is a diagram showing a comparative example of output results in a case where a PV template is applied to temporary data and a case where a PV template is applied to official data corresponding to the embodiment. DETAILED DESCRIPTION

[0022] Hereinafter, the embodiment will be described in detail with reference to the drawings. Note that the following embodiment does not limit the application covered by the claims, and all combinations of characteristics described in the embodiment are not necessarily essential to the application. Two or more of the features described in the embodiment can be arbitrarily combined. In addition, the same reference numerals are assigned to the same or similar structures, and repeated description is omitted. In addition, in each drawing, the up-down, left-right, front-rear directions with respect to the paper surface are set as the up-down, left-right, front-rear directions of the components (or accessories) in the present embodiment, and are used in the description herein.

[0023] First, the structure of an image processing system corresponding to the present embodiment will be described. Figure 1 Fig. 1(A) is an example of a diagram showing the structure of an image processing system 10 corresponding to the present embodiment. The image processing system 10 is configured by connecting a photographing section 110, a support arm 120, a model support device 130, a display device 140, a printing device 150, and the like to an information processing device 100. Note that the system structure is not limited to Figure 1The structure shown in (A) can also be that the information processing apparatus 100 further has an external server, a cloud server, or the like connected via a network. In this external server or the like, at least a part of the processing in the embodiments described below can be executed.

[0024] The information processing apparatus 100 controls the operation of at least any of the photographing section 110, the support arm 120, and the model support apparatus 130, generates a plurality of images by photographing an article as a photographing target from an arbitrary angle, and generates three-dimensional model data (official data) from the plurality of images. Then, a template (PV template) of a dynamic image (promotional video (PV data)) including temporary three-dimensional model data (temporary data) prepared in advance is used, the PV template is transformed by replacing the temporary data of the PV template with the official data, and thus the PV data is reconstructed. In addition, the information processing apparatus 100 can function as an image processing apparatus that generates an image displayed in a virtual space by using the generated official data as virtual space data. The image processing apparatus can be prepared separately from the information processing apparatus 100. In the present embodiment, the article as a photographing target is a plastic model that can be assembled, a posable figure (a figure having a joint part that can be posed), a toy, a doll, or the like, and these are collectively referred to as a "model article". In addition, in the present embodiment, a promotional video (PV) is not limited to a video for the purpose of advertisement promotion or promotion of a model article as a photographing target, but refers to a dynamic image (a short film, a movie, or a demonstration video, or the like, without limitation on the form or type of the dynamic image) that uses a model article as a photographing target and has appeal. In the following description, as a collective term including these dynamic images, "PV" or "promotional video" is also used.

[0025] Next, the photographing section 110 is a photographing apparatus that photographs (scans) a three-dimensional shape of a model article as a photographing target and outputs an image of the photographing target in accordance with the control of the information processing apparatus 100. As the photographing section 110, for example, a smartphone with a camera installed with an application for photographing a three-dimensional shape can be used. In addition, the photographing section 110 can be configured by a three-dimensional scanner apparatus that can output three-dimensional shape and color information. As the three-dimensional scanner apparatus, for example, a Space Spider manufactured by Artec (registered trademark) can be used. For example, a 3D dimensional model data of the entire model article can be acquired by acquiring a scan image at around 500 frames to 800 frames.

[0026] The support arm 120 is a position / attitude control device that moves the photographing section 110 to a prescribed photographing position and attitude in accordance with the control of the information processing device 100. The support arm 120 can be configured to be able to change the photographing position and attitude by a manual method and to be able to fixedly maintain the changed position and attitude, or can be configured to be able to be controlled by the information processing device 100. In the case where the support arm 120 is configured to be able to be controlled by the information processing device 100, for example, an xArm 7 manufactured by the company FAUCTORY (registered trademark) can be used. The xArm 7 is configured of seven joints and is able to perform the same motions as a human arm. In addition, instead of the support arm 120, positioning of the photographing section 110 can be performed manually.

[0027] The model support device 130 is a support table for supporting a model article whose attitude is fixed. The model support device 130 can also be configured to enable the model article to be rotated in a state where the model article is set on the support table or set on the front end of a support rod, for example. In the present embodiment, after the photographing section 110 is positioned at an arbitrary photographing position and photographing angle by the support arm 120, the model support device 130 is rotated one turn and photographing is performed. By performing this photographing at a plurality of photographing positions and photographing angles, an image of the entire model article can be acquired. In addition, here, by driving the support arm 120 and the model support device 130 synchronously, the photographing process can be performed more easily and with higher precision. In addition, instead of the model support device 130, the photographing section 110 can be manually moved around the model article to perform photographing from an arbitrary photographing position and photographing angle.

[0028] The display device 140 is a display device such as a liquid crystal display (LCD) that displays an image acquired by the photographing section 110, or displays official data of three-dimensional model data generated from the photographing image, or displays PV data reconstructed using the official data. In addition, the display device 140 can also display a two-dimensional bar code (QR code) showing a link of a download address in the screen. The printing device 150 is a printing device that prints a QR code of a link of a download address on a paper medium or the like for enabling a user to acquire the PV data after the PV data is reconstructed to view the PV data.

[0029] Figure 1(B) shows an example of a hardware structure of the information processing apparatus 100. The CPU 101 is an apparatus that performs overall control of the information processing apparatus 100, calculation, processing, and management of data. For example, it is possible to control the timing of image capturing and the number of captured images in the photographing section 110, and it is possible to control the joints of the arm of the support arm 120 to configure the photographing section 110 at an arbitrary capturing position and capturing angle. In addition, after the capturing position and capturing angle of the photographing section 110 are decided, it is possible to rotate the model support apparatus 130 and perform a capturing operation by the photographing section 110. In addition, the CPU 101 can also function as an image processing section that processes images output from the photographing section 110.

[0030] The RAM 102 is a volatile memory that serves as a main memory, a work area, and the like of the CPU 101. The ROM 103 is a non-volatile memory that stores image data, other data, various programs for the CPU 101 to perform operations, and the like in prescribed areas. The CPU 101 controls each section of the information processing apparatus 100 using the RAM 102 as a work memory, for example, in accordance with a program stored in the ROM 103. Note that the programs for the CPU 101 to perform operations are not limited to being stored in the ROM 103, and can be stored in the storage apparatus 104.

[0031] The storage apparatus 104 is constituted by a magnetic disk such as an HDD, a flash memory, or the like, for example. The storage apparatus 104 stores an application, an OS, a control program, a correlation program, a game program, and the like. In addition, data of a PV template that is an object of reconstruction is included. The PV template includes temporary data, and can also be different in the size and posture of the three-dimensional model of each PV template. The storage apparatus 104 can read or write data based on the control of the CPU 101. The storage apparatus 104 can also be used instead of the RAM 102 and the ROM 103.

[0032] The communication apparatus 105 is a communication interface that communicates with the photographing section 110, the support arm 120, and the model support apparatus 130 based on the control of the CPU 101. The communication apparatus 105 can also be configured to be able to communicate with an external server or the like. The communication apparatus 105 can include a wireless communication module that can include a publicly known circuit configuration including an antenna system, an RF transmitter-receiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chip set, a subscriber identification module card, a memory, and the like. Here, the communication of the information processing apparatus 100 with the photographing section 110, the support arm 120, and the model support apparatus 130 can also be performed by wireless communication.

[0033] In addition, the communication device 105 can also include a wired communication module for making a wired connection. The wired communication module enables communication with other devices including the display device 140 via one or more external ports. In addition, various software components for processing data can be included. The external ports are coupled to other devices directly or indirectly via a network via Ethernet, USB, IEEE 1394, or the like. Furthermore, the hardware devices can also be replaced by software for realizing functions equivalent to the above-described devices.

[0034] The operation section 106 is constituted by, for example, a button, a keyboard, a touch panel, or the like, and accepts operation input from a user. The display control section 107 functions as an interface for displaying information on the display device 140 connected to the information processing device 100, and controls the operation of the display device 140. Part of the functions of the operation section 106 can also be provided by the display device 140. That is, the display device 140 can also be constituted as a device provided with a touch panel such as a tablet terminal.

[0035] Next, an example of a model object that becomes a photographic target in the present embodiment will be described with reference to Figure 2 The model object 200 is a model object having an appearance of a humanoid (robot or human). The model object can be, for example, a model object assembled and painted as a plastic model. Alternatively, it can be a completed model object provided with a movable joint portion (movable model). Figure 2 The model object of the present embodiment is merely an example for explanation, and the shape of the model object is not limited to an appearance of a humanoid, and can be a model object of an arbitrary shape such as a general vehicle, a racing car, a military vehicle, an airplane, a ship, an animal, a virtual living body, or the like. Furthermore, the article that becomes a photographic target is not limited to a model object as long as it is an article of which a three-dimensional shape can be photographed by the photographing section 110.

[0036] The model object 200 has a head portion 201, a chest portion 202, a right arm portion 203, a left arm portion 204, a right torso portion 205, a left torso portion 206, a right leg portion 207, a left leg portion 208, a right foot portion 209, and a left foot portion 210, which are combined to constitute the model object 200. At least a part of each of the portions 201 to 210 is supported so as to be rotatable (or swingable) with respect to an adjacent portion. For example, the head portion 201 is supported so as to be rotatable with respect to the chest portion 202, and the right arm portion 203 and the left arm portion 204 are supported so as to be rotatable with respect to the chest portion 202. Since a joint structure is provided in each portion of the model object 200 as described above, the model object 200 can take an arbitrary posture.

[0037] Next, an example of installation of the image processing system 10 in the present embodiment will be described with reference to Figure 3 the drawing.Figure 3 (A) of FIG. 1 illustrates an example of installation in which PV data is reconstructed from formal data generated by photographing a model object and a download link for viewing the reconstructed PV data is issued by a printer. Figure 3 (B) and (C) of FIG. 1 are diagrams illustrating structural examples of AR (Augmented Reality) markers in the present embodiment.

[0038] In Figure 3 (A) of FIG. 1, an information processing apparatus 100, a drive system for driving a support arm 302, a drive system for driving a rotary table 306, and the like are included in a box 301. Further, the support arm 302 can also adjust a photographing direction and position by a manual method. A surface of the box 301 is configured to be flat so that a poster for advertisement can be posted.

[0039] The support arm 302 corresponds to the support arm 120, and can support a terminal 303 functioning as the photographing section 110 and fix a position thereof in accordance with a control of the information processing apparatus 100 or by a manual method. The support arm 302 can also act in a manner of controlling a tilt of the terminal 303.

[0040] The terminal 303 is a terminal of a touch panel type with a built-in camera, and can be, for example, a smartphone, a tablet terminal, a digital camera, or the like. The terminal 303 can photograph an image of a model object 200 and transmit the image to the information processing apparatus 100. A ring light 304 is an illumination device used when the model object 200 is photographed by the terminal 303, and can uniformly light the model object 200 so that a shadow is not easily generated. Further, as an additional light source, a ceiling light or an auxiliary light can be provided in addition to the ring light 304, on the left and right, and on the lower side.

[0041] A background cloth 305 is a background cloth for photography, and can be, for example, a white cloth. The rotary table 306 can mount the model object 200 and rotate the model object. A plurality of AR markers 310 as illustrated in (B) of FIG. 1 are arranged on the rotary table 306. The AR markers 310 are used to adjust an orientation and a position of a photographed model object. As illustrated in (B) of FIG. 1, Figure 3 Figure 3 (B) of FIG. 1, Figure 3 (C) of FIG. 1 illustrates a structure example of the AR marker 310. As illustrated in (B) of FIG. 1, Figure 3 (B) of FIG. 1, by making the three-dimensional coordinates determined in accordance with the AR marker 310 illustrated in (C) of FIG. 1 coincide with the reference, an orientation and a tilt of formal data can be correctly corrected. Figure 3

[0042] In Figure 3 ​​In (A), the model object is placed on a translucent (transparent) table, but in addition to this, a supporting device such as a so-called "action base" can also be used. The action base refers to a support provided on a base that is bent into a "く" shape, and the model object can be mounted on the front end of the support. In this case, an AR marker can also be arranged on the front end of the support. The method of indicating the model object can be changed according to the posture. For example, in the case of an upright posture, it can be placed on a transparent table for shooting. On the other hand, in the case of a flying posture where you want to shoot the soles of the feet, you can also use a movable bracket. In addition, the movable bracket can also be used to shoot an upright posture.

[0043] Display device 307 is a device corresponding to display device 140 and may also have a touch panel function. Display device 140 can display, for example, icon images as options for the PV data that the user wishes to reconstruct, and accept the user's selection. Printer 308, for example, a thermal printer, can print a QR code indicating a download URL after the PV data has been reconstructed. Alternatively, the QR code can be displayed on display device 307, allowing the user to read it using a smartphone or other device.

[0044] Next, refer to Figure 4 The data structures of various tables stored in the storage device 104 of the information processing device 100 are described below. Figure 4 In the PV management table 400 of (A), model type information 402 for identifying the model type of the model object, information 403 of temporary data included in the PV, and a storage destination 404 of the PV template data are registered in association with a PV name 401 .

[0045] First, the PV name 401 contains identification information uniquely identifying the template data of the promotional video (PV) serving as the prototype of the dynamic image being reconstructed in this embodiment. Model category information 402 is information for identifying the category of the model object to which the PV corresponds. In this embodiment, PVs may be prepared according to the type of model object. For example, in addition to preparing PVs for model objects having a normal first shape (e.g., an appearance obtained by directly reducing a life-size shape), PVs may also be prepared for model objects having a deformed second shape (e.g., an appearance obtained by deforming a life-size shape into a second or third body shape). In the model category information 402, M1 indicates that the corresponding PV corresponds to a model object having the first shape, and M2 indicates that the corresponding PV corresponds to a model object having the second shape.

[0046] The provisional data information 403 registers information of the provisional data included in the PV. In the present embodiment, the template data of the PV is configured with the provisional data, and the PV can be reconstructed by replacing the provisional data with the official data generated from the image obtained by the photographing. The save destination 404 indicates information of the save destination of the template data of the PV.

[0047] Other information related to the PV can be registered in the table 400. As the information, for example, there are an icon image for PV selection, a sample movie, information related to the recommended pose in the PV, and the like.

[0048] Next, Figure 4 (B) shows an example of the structure of a user information table 410 that registers information of the user who uses the image processing system 10. The table 410 registers a user name 411, a PV name 412, photographed image data 413, a three-dimensional model 414, and a URL 415. The user name 411 is information for uniquely identifying the user (or the player) who is the owner of the model object photographed by the photographing section 110. The PV name 412 registers the PV name selected by the user. The PV name 412 corresponds to one of the information registered as the PV name 401 in the table 400.

[0049] The photographed image data 413 registers information of the photographed image obtained by the photographing section 110 photographing the model object owned by the user registered in the user name 411. The three-dimensional model 414 registers the official data generated based on the photographed image data 413. The URL 415 is link information showing the save location of the PV data reconstructed using the official data of the three-dimensional model 414, and is the URL shown in the QR code printed by the printing device 150.

[0050] Next, an example of the process performed by the information processing apparatus 100 corresponding to the present embodiment will be described with reference to Figure 5 corresponding to the flowchart is realized by the CPU 101 of the information processing apparatus 100 executing the program stored in the ROM 103 or the storage device 104.

[0051] First, in S501, the CPU 101 accepts the user registration. The input of the name and the contact address of the user is accepted. The user identifier for uniquely identifying each user is given to each user. The user identifier corresponds to the user name 411 of (B). Figure 4 The CPU 101 stores the input user information in association with the time of accepting the input and the user identifier in the storage device 104.

[0052] In the following S502, the CPU 101 causes the display device 140 to display icons of selectable promotional videos (PVs) and accepts selection of a PV that the user wants to create, by the display control section 107. There are any number of icons of PVs, each of which contains a different scene. An example movie can also be displayed at this time. Further, the PVs can be prepared by the kind of model object, for example, PVs for a model object of a normal shape (for example, an appearance shape obtained by directly reducing a full-size shape) as a first shape can be prepared in addition to PVs for a model object of a transformed shape (for example, an appearance shape obtained by transforming a full-size shape into a two-thirds size or a one-third size) as a second shape. The user can determine whether the shape of the model object that the user holds is a normal shape or a transformed shape and select a PV that corresponds to the shape.

[0053] In the following S503, the CPU 101 accepts selection of a PV from the user. Each PV is previously given identification information, and the identification information of the selected PV is bound to the user information accepted in S501 and saved in the table 410 of the storage device 104.

[0054] In the following S504, the CPU 101 causes the display device 140 to display a recommended pose of a model object in the selected PV, by the display control section 107. In the present embodiment, the pose of the model object differs in each PV, and the appearance of the model object is photographed according to a pose that is appropriate for the selected PV, whereby a dynamic image that is more realistic can be reconstructed. The pose of the model object in the present embodiment is, for example, a standing pose, a flying pose, an attacking pose, and the like. Contents corresponding to the standing pose, contents corresponding to the flying pose, and contents corresponding to the attacking pose are prepared in the PVs, respectively, and therefore it is important to adopt a pose that corresponds to the contents of each PV in order to create a PV that does not have a sense of incongruity.

[0055] The user confirms the recommended pose of the model object displayed in the display device 140, decides the pose of the user's model object, and sets it to the model object support device 130. The CPU 101 can determine whether a model object is set to the model object support device 130 on the basis of a captured image obtained by the photographing section 110. Alternatively, a switch that is configured to be turned on when a model object is set to the model object support device 130 can be detected by the CPU 101 on the basis of a signal from the switch, whereby the determination can be made. Alternatively, a button for accepting an operation when the setting of the model object is completed can be displayed in the display device 140, and the CPU 101 can detect whether an operation with respect to the button is accepted. In S505, the CPU 101 detects that a model object is set to the model object support device 130 by any of the above-described methods. In response to this detection, the process proceeds to S506.

[0056] In S506, the photographing section 110 performs photographing processing to generate an image. The photographed image is transmitted to the information processing apparatus 100, and the CPU 101 saves the image in association with the user identifier or the like in the table 410 of the storage apparatus 104 as photographing image data 413. In the following S507, the CPU 101 performs reconstruction processing of the PV based on the photographing image data 413. The CPU 101 stores the reconstructed PV data in association with the user information registered in S501 in the storage apparatus 104.

[0057] In the following S508, a URL and a QR code for allowing the user to access the PV data are issued. The URL and the QR code can be transmitted to the user's electronic mail or can be displayed in the display apparatus 140. When the user acquires the URL and the QR code, the user can access the storage location of the PV data and download the reconstructed PV data in S509.

[0058] Next, details of the photographing processing in S506 will be described with reference to Figure 6 In the present embodiment, the driving information of the support arm 120 and the number of times of photographing of the photographing section 110 at the photographing position are registered for each photographing position. Further, when photographing is performed at each photographing position, photographing is performed by rotating the model support apparatus 130.

[0059] First, in S601, the CPU 101 controls the support arm 120 to move the photographing section 110 to any of the registered photographing positions. Alternatively, the support arm 120 can be manually controlled to move the photographing section 110 to any of the photographing positions. In addition, the model support apparatus 130 is controlled to move to a rotation start position. In the following S602, the CPU 101 causes the photographing section 110 to perform photographing while rotating the model support apparatus 130 at the photographing position, thereby acquiring images of the model 200. Thus, a plurality of images with different photographing angles can be acquired. For example, in a case where the photographing angle is changed every 15 degrees, 24 images can be photographed, and if the photographing angle is changed every 10 degrees, 36 images can be photographed. By making the change in the photographing angle smaller, the accuracy of the official data generated in the later stage is improved.

[0060] In addition, at this time, the model support apparatus 130 can be stopped at a specific rotation position, and the photographing section 110 can be moved in the vertical direction while changing the angle of the photographing section 110 by controlling the support arm 120 or manually by the photographer, thereby performing photographing of the model. Thus, the model can be photographed also from above or also from below. In particular, by performing photographing from below, the sole of the model can also be photographed. In this way, in the present embodiment, the photographing direction and the photographing angle are selected in a manner that includes the entire circumference of the model, and photographing is performed.

[0061] In the present embodiment, it is possible to suspend and take a photograph of the model in a form in which the model is set on a transparent support rod extending from the model support device 130. Alternatively, it is also possible to directly stand the model upright on the turntable of the model support device 130 and take a photograph. In the former photographing method, since it is also possible to take a photograph of the sole of the model, it is suitable for photographing of the flying posture. On the other hand, in the latter photographing method, it is not possible to take a photograph of the sole, but it is suitable for photographing of the standing posture. Furthermore, it is also possible to take a photograph of the standing posture in the former photographing method.

[0062] In addition, an AR marker is arranged on the support rod or the turntable of the model support device 130, and an image including the AR marker is generated at the time of photographing. The AR marker is used to correct the orientation and position of the photographed object.

[0063] In the next S603, the CPU 101 determines whether there is an unselected photographing position among the photographing positions set in advance, and in the case where there is an unselected photographing position, returns to S601 and continues the processing. On the other hand, in the case where there is no unselected photographing position, moves to S604. In S604, the photographing section 110 transmits the image obtained by the above steps to the information processing device 100, and the CPU 101 stores the received image as the photographing image data 413 in the table 410 of the storage device 104 in association with the user information and the like.

[0064] Next, details of the processing of S507 of reconstructing the PV based on the photographing image obtained in S506 will be described with reference to Figure 7 First, in S701, the CPU 101 acquires a plurality of images photographed by the photographing section 110 which are saved in the storage device 104 as the photographing image data 413. In the next S702, the CPU 101 generates three-dimensional model data of the model which becomes the official data, from the plurality of images obtained by photographing the model from different angles, which are acquired.

[0065] In the generation method of the three-dimensional model data, for example, it is possible to use the images of the model acquired from the plurality of images to determine the three-dimensional model corresponding to the shape of the model by using a known visual volume intersection method. In the visual volume intersection method, a cone (visual volume) having the camera position as the apex and the silhouette as the cross section is made using a plurality of model images, and the silhouette is back-projected to the three-dimensional space, and as the intersection part (common part) thereof, the three-dimensional shape is restored.

[0066] In S703, the CPU 101 performs a size adjustment process of the official data generated in S702 by applying a preset adjustment related to the position and size of the model. The size adjustment process refers to a process of adjusting the size of the model so as not to be out of view in the case where a part of the model is out of view in the display frame in the case where the model exceeds the angle of view of the preset PV. As for the size of the model, the vertical height, the vertical depth are decided for each PV in advance, and in the case where any of them exceeds the specified size, the size is adjusted in such a manner that the size of the model converges within the specified size. Further, it can also be that, for the PV for the first shape, the size adjustment process is performed for the model of the first shape, and the size adjustment process is not performed for the model of the second shape. Similarly, for the PV for the second shape, the size adjustment process can be performed for the model of the second shape, and the size adjustment process can not be performed for the model of the first shape.

[0067] In the following S704, the CPU 101 adjusts the direction and position of the model whose size has been adjusted by the adjustment. This process can also be performed before S703. The AR marker is included in the three-dimensional model, and the direction and position of the front surface of the three-dimensional model are decided in accordance with the orientation of the AR marker. From the AR marker, it is known to what extent the three-dimensional model is tilted or shifted in the XYZ three-dimensional coordinate system, and thus the tilt and the shift are reset to reset the three-dimensional coordinate system. Alternatively, a correction value in the original three-dimensional coordinate system can be calculated, and then the direction and position of the three-dimensional model are adjusted using the correction value.

[0068] Since the temporary data included in the PV template has also been corrected in the direction and position, by performing the correction with respect to the generated official data, even in the case of replacement with the temporary data, the shift in the direction and position of the model in the PV does not occur. After the process of S704, the AR marker can be removed from the official data.

[0069] In addition, in S704, the origin of the official data is decided. In the case of a model in an upright posture, display is performed with the model set on the ground or the like, and thus the origin is set at the lowermost portion of the official data, and adjustment of the direction and position of the official data is performed. In the case of a posture in an upright posture, since the sole becomes the origin, the model can be set on the ground regardless of which model. In addition, in the case of a posture in a flying posture, the position of the origin can also be set to the center of gravity position of the model, rather than the lowermost portion. Furthermore, in the case where the model has a weapon or other equipment, the center of gravity position can sometimes move due to the influence of the weapon or equipment, and thus, in the case where the center of gravity position is set, the center of gravity position of only the model main body after the weapon or equipment is removed can also be set. The origin set to the official data in the case of substitution with the temporary data of the PV template becomes a reference for deciding the arrangement position of the official data that is substituted.

[0070] In the following S705, the official data generated through the above steps is substituted with the temporary data included in the PV template, and is output as dynamic image data of the PV. In the template of the PV, the background, the action of the object, the camera angle are set, and the action, the camera angle (viewpoint position) of the temporary data are set, and it is possible to be transformed into dynamic image data that can view the template data and be output. Here, by substituting the temporary data with the official data generated in S702 and transforming it into dynamic image data, it is possible to reconstruct the PV data using arbitrary official data. The PV template can be generated, for example, using UNREAL ENGINE 5 (registered trademark).

[0071] More specifically, in the PV template, the position information of the object, the model in the three-dimensional space, the action start position of the object, the model, and the information of the action (change in position corresponding to time) in the three-dimensional space, the information of the camera angle, the effect information such as lighting, and the like are set corresponding to the time axis of the PV. In this template data, the temporary data is substituted with the official data generated in S702 and is caused to act, move, or the like in the three-dimensional space, is cut off as a two-dimensional image in units of frames, and is compressed in an arbitrary dynamic image encoding method such as H264, and thus it is possible to generate dynamic image data as PV data.

[0072] Alternatively, it can also be that, for each frame constituting the PV data, the camera angle of the three-dimensional model data is specified, the texture of the temporary data is substituted by generating the texture of the three-dimensional model corresponding to each camera angle and in units of frames, and the PV is reconstructed. In the following S706, the CPU 101 saves the PV data reconstructed in S705 to the storage device 104.

[0073] In the above description, the case where the plurality of images captured by the imaging section 110 are transmitted to the information processing apparatus 100 and the official data is generated is explained, but as for the processing of generating the official data from the plurality of images, an external server, a cloud server can be utilized. In this case, the information processing apparatus 100 can transmit the plurality of images to the external server and acquire the official data as a processing result. In addition, as for the reconstruction processing of the PV in S705, an external server, a cloud server can be similarly utilized.

[0074] Figure 8 is a diagram illustrating the lighting when the model object 200 is imaged by the imaging section 110. Figure 8 (A) of is a diagram showing a configuration example of the lighting observed from the upper side of the model object 200, and a ring-shaped lamp 304 is arranged on the front surface of the model object 200 as an imaging target. At this time, if the ring-shaped lamp 304 alone irradiates the model object 200 as an object to be imaged, only the irradiation light L0 from the ring-shaped lamp 304 can cause a shadow on the back side of the model object 200, and therefore an auxiliary lamp 801 and an auxiliary lamp 802 are arranged on the side of the ring-shaped lamp 304, and the irradiation light L1, L2 from each of the auxiliary lamps is used to avoid the imaging range from containing the shadow.

[0075] Next, Figure 8 (B) of is a diagram showing a configuration example of the lighting observed from the side of the model object 200, and a ceiling lamp 803 is arranged above the ring-shaped lamp 304 on the front surface of the imaging target 200, and an auxiliary lamp 804 is arranged below. This is because the head portion of the model object 200 is irradiated by the ceiling lamp 803 to clearly acquire an image of the head portion, and the shadow due to the irradiation light L3 of the ceiling lamp 803 is offset by the irradiation light L4 of the auxiliary lamp 804. In addition, the shadow due to the irradiation light L3 of the ceiling lamp 803 is also offset by the irradiation light L1, L2 of the above-described auxiliary lamps 801, 802.

[0076] As for the auxiliary lamps 801, 802, 804, the ceiling lamp 803 shown in (A) of Figure 8 (B) of, Figure 8 It is not necessary to utilize all of these lamps, and at least any one or more lamps can be used for the purpose of offsetting the shadow due to the ring-shaped lamp 304.

[0077] Figure 9 A comparative example showing a comparison between the output result in the case where the PV template is transformed for the provisional data and the output result in the case where the PV template is transformed for the official data. In Figure 9In (A) of FIG. 9, the output result in a case where the transformation is performed in the PV template using the provisional data 901 is displayed in the scene 900. In contrast, in the reconstructed scene 900, the output result in a case where the transformation is performed in the PV template using the official data 902 is displayed. The official data 902 is arranged in the same position as the provisional data 901, and it is understood that the replacement has been performed. In addition, Figure 9 (B) of FIG. 9 shows another scene 910 in the same PV, and here, the provisional data 911 is replaced with the official data 912 in the scene 910 as well.

[0078] Thus, in the present embodiment, the PV template generated with respect to the provisional data is used, and thus the same PV data can be efficiently reconstructed by only replacing the provisional data with the official data.

[0079] In the description of the above-described embodiment, the case where the moving image of the promotional video in which the three-dimensional model data generated from the image obtained by photographing the model object is displayed in the virtual space is explained. However, the embodiment is not limited to the generation of the PV data, and the three-dimensional model data can be used as game data for the virtual space. In addition, it can be widely applied to the processing of causing a character to act in the virtual space. For example, the character can appear in the activity, concert, sports, online meeting, and the like performed in the virtual space. Furthermore, in the image technology of the extended reality (XR) that fuses the real world and the virtual world so that an object that does not exist can be perceived in the real space, the technology of the present embodiment can be applied.

[0080] Thus, in the present embodiment, the three-dimensional model data is generated from the image obtained by photographing the appearance of the model object, and it can appear in the promotional video that the user likes. The model object is, for example, a model object such as a plastic model that is assembled by the user to be painted as his or her own work, and since it is possible to reflect the individuality of the user in the moving image and the character performance in the virtual space, it is possible to significantly improve the interest.

[0081] <Summary of Embodiment>

[0082] The above-described embodiment discloses at least the following image processing method, information processing apparatus, and computer program.

[0083] (1) An image processing method for reconstructing a moving image, the image processing method comprising the steps of:

[0084] generating first three-dimensional model data of a model object included in the image, by a processing unit, from a plurality of images generated by a photographing unit by photographing an appearance of the model object from a plurality of directions;

[0085] The reconstruction process, the processing section reconstructs the dynamic image by replacing the second three-dimensional model data with the first three-dimensional model data in template data of the dynamic image that is previously configured to include the second three-dimensional model data; and

[0086] The output process, the output section outputs information indicating an acquisition destination of the reconstructed dynamic image.

[0087] (2) The image processing method according to (1), wherein the plurality of images include a marker for determining an orientation and a position of the model object,

[0088] generates the first three-dimensional model data based on correction of the orientation and the position of the first three-dimensional model data based on the marker.

[0089] (3) The image processing method according to (1) or (2), wherein, with respect to the template data of the dynamic image, a plurality of template data is prepared according to a category of a model object,

[0090] In the reconstruction process, the reconstruction of the dynamic image is performed with respect to the template data selected from the user.

[0091] (4) The image processing method according to (3), wherein the category of the model object includes: a first category of a model object having an appearance shape obtained by reducing an equal-body-size shape; and a second category of a model object having an appearance shape obtained by deforming an equal-body-size shape.

[0092] (5) The image processing method according to (3) or (4), wherein, in the generation process, the size of the first three-dimensional model data is adjusted to a size corresponding to a content of the dynamic image, to generate the first three-dimensional model data.

[0093] (6) The image processing method according to (5), wherein the adjustment of the size is performed with respect to the first three-dimensional model data of a model object of the same category as the category of the model object of the template data of the dynamic image, and the adjustment of the size is not performed with respect to the first three-dimensional model data of a model object of a different category from the category of the model object of the template data of the dynamic image.

[0094] (7) The image processing method according to any one of (1) to (6), wherein the template data of the dynamic image is associated with a recommended posture with respect to the model object,

[0095] The plurality of images are generated by photographing an appearance when the model object is set to the recommended posture.

[0096] (8) The image processing method according to any one of (1) to (7), further comprising a process of accepting selection of template data of the moving image by the processing section.

[0097] (9) The image processing method according to (8), further comprising a process of causing a display device to display designation of a recommended pose for the moving image in response to acceptance of selection of the template data of the moving image by the processing section.

[0098] (10) The image processing method according to any one of (1) to (9), wherein the plurality of directions are set to encompass an entire circumference of the model object.

[0099] (11) The image processing method according to any one of (1) to (10), wherein the plurality of images are generated by the photographing section photographing the model object illuminated by a first light source and at least one second light source for canceling a shadow generated by illumination light from the first light source.

[0100] (12) The image processing method according to any one of (1) to (11), wherein

[0101] further comprising a process of receiving the plurality of images from the photographing section by the processing section.

[0102] (13) The image processing method according to any one of (1) to (12), wherein in the template data, information of at least any one of a position, a motion, a camera angle, and illumination of the second three-dimensional model data in a three-dimensional space is set in correspondence with a time axis of the moving image.

[0103] (14) An information processing apparatus comprising:

[0104] a generation unit that generates first three-dimensional model data of a model object included in images generated by photographing an appearance of the model object from a plurality of directions;

[0105] a reconstruction unit that reconstructs a moving image by replacing second three-dimensional model data in template data of the moving image, which is constructed in advance to include the second three-dimensional model data, with the first three-dimensional model data; and

[0106] a control unit that controls so that information indicating an acquisition destination of the reconstructed moving image is output from an output section.

[0107] (15) A computer program for causing a computer to execute the image processing method according to any one of (1) to (13).

[0108] The present application is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the present application.

Claims

1. An image processing method for reconstructing a dynamic image, the image processing method comprising the following steps: The generating step generates first three-dimensional model data of the model object included in the images, based on a plurality of images generated by the photographing step photographing the appearance of the model object having a joint structure from a plurality of directions. The joint structure can be used to change the posture of the model object; a reconstruction step in which the processing unit reconstructs the dynamic image by replacing the second three-dimensional model data with the first three-dimensional model data in template data of the dynamic image that is previously configured to include the second three-dimensional model data; as well as an output step, wherein the output unit outputs information indicating a destination for obtaining the reconstructed moving image; The image processing method further includes the following steps: the processing unit accepts selection of template data of the dynamic image, and In response to receiving the selection of the template data of the moving image, the processing unit causes a display device to display a designation of a posture recommended for the moving image, so that the user confirms the recommended posture displayed on the display device and decides on a posture of the model object. The template data of the dynamic image is associated with a posture recommended for the model object. The plurality of images are generated by capturing the appearance of the model object when the model object is in the recommended posture.

2. The image processing method according to claim 1, wherein: The plurality of images contain markings for determining the orientation and position of the model object, The orientation and position of the first three-dimensional model data are corrected based on the marker to generate the first three-dimensional model data.

3. The image processing method according to claim 1 or 2, wherein: Regarding the template data of the dynamic image, a plurality of template data are prepared according to the type of the model object. In the reconstruction step, the moving image is reconstructed based on the template data selected by the user.

4. The image processing method according to claim 3, wherein: The categories of the model objects include: a first category of model objects having an appearance shape obtained by reducing a life-size shape; and a second category of model objects having an appearance shape obtained by deforming a life-size shape.

5. The image processing method according to claim 3, wherein: In the generating step, the first three-dimensional model data is generated by adjusting the size of the first three-dimensional model data to a size corresponding to the content of the moving image. The image processing method according to claim 5 , wherein: The size adjustment is performed on the first three-dimensional model data of the model object of the same category as the model object of the template data of the dynamic image, while the size adjustment is not performed on the first three-dimensional model data of the model object of a category different from the category of the template data of the dynamic image.

7. The image processing method according to claim 1 or 2, wherein: The plurality of directions are set to encompass the entire circumference of the model object.

8. The image processing method according to claim 1 or 2, wherein: The plurality of images are generated by the imaging unit capturing the model object illuminated by a first light source and at least one second light source for canceling a shadow caused by the illumination light from the first light source.

9. The image processing method according to claim 1 or 2, wherein further comprising a process of receiving the plurality of images from the photographing section by the processing section.

10. The image processing method according to claim 1 or 2, wherein in the template data, information of at least any one of a position, a motion, a camera angle, and a lighting of the second three-dimensional model data in a three-dimensional space is set in correspondence with a time axis of the dynamic image.

11. An information processing apparatus, comprising: The generation unit generates first three-dimensional model data of the model object included in the images, from a plurality of images generated by photographing an appearance of the model object having a joint configuration from a plurality of directions, wherein a joint structure capable of changing a posture of the model object; a reconstruction unit that reconstructs a dynamic image by replacing the second three-dimensional model data with the first three-dimensional model data in template data of the dynamic image that is configured in advance to include the second three-dimensional model data; a control unit that controls so that information indicating an acquisition destination of the reconstructed dynamic image is output from an output section; the control unit further accepts a selection of the template data of the dynamic image, and in response to accepting the selection of the template data of the dynamic image, causes a display apparatus to display a designation of a posture recommended for the dynamic image, so that a user confirms the recommended posture displayed by the display apparatus and decides a posture of the model object, wherein the template data of the dynamic image is associated with the posture recommended for the model object, the plurality of images are generated by photographing an appearance of the model object when the model object is set to the recommended posture.

12. A computer program product including a computer program for causing a computer to execute the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium recording a computer program for causing a computer to execute the image processing method according to any one of claims 1 to 10.

Images