Information processing method, information processing device and computer program

Abstract

To efficiently execute reconstruction processing when photographing an object and reconstructing a moving image by using a video of the photographed object.SOLUTION: An image processing method for reconstructing a moving image, includes the steps of: causing a processing part to generate first three-dimensional model data of a model included in images out of a plurality of images formed by photographing an external appearance of the model from a plurality of directions by a photographing part; causing the processing part to reconstruct a moving image by replacing second three-dimensional model data by the first three-dimensional model data in template data of the moving image configured in advance to include the second three-dimensional model data; and causing an output part to output information representing an acquisition source of the reconstructed moving image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing method, an information processing device, and a computer program. [Background technology]

[0002] Patent document 1 describes a technology in which a texture generated from a group of captured images of an object taken from multiple angles by a photographing unit is mapped onto a primitive in a virtual space to generate a virtual space image as seen from a virtual camera in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-107251 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned technology, when an object is photographed and a moving image is reconstructed using the photographed image of the object, it is expected that the reconstruction process will be performed efficiently.

[0005] Therefore, when an object is photographed and a moving image is reconstructed using the photographed image of the object, the reconstruction process can be performed efficiently. [Means for solving the problem]

[0006] One form for solving the above problem is an image processing method for reconstructing a moving image, the method including the steps of: generating first three-dimensional model data of a model contained in a plurality of images generated by an imaging unit capturing images of the exterior of the model from a plurality of directions; reconstructing the moving image by replacing the second three-dimensional model data with the first three-dimensional model data in template data for the moving image that is configured in advance to include second three-dimensional model data; and outputting information indicating the source from which the reconstructed moving image was obtained by an output unit. Effect of the Invention

[0007] When an object is photographed and a moving image is reconstructed using the photographed image of the object, the reconstruction process can be performed efficiently. [Brief description of the drawings]

[0008] [Figure 1] 1A and 1B are a diagram showing an example of the configuration of an image processing system according to an embodiment, and a diagram showing an example of the hardware configuration of an information processing device 100. [Diagram 2] FIG. 1 is a diagram showing an example of the appearance of a model according to an embodiment. [Diagram 3] 1A to 1C are diagrams showing an implementation example of an image processing system according to an embodiment and an example of an AR marker. [Figure 4] FIG. 4 is a diagram showing an example of a data configuration of a data table according to the embodiment. [Diagram 5] 10 is a flowchart showing an example of a process for generating PV data according to the embodiment. [Figure 6] 10 is a flowchart showing an example of a photographing process according to the embodiment. [Figure 7] 1 is a flowchart showing an example of a PV reconstruction process according to the embodiment. [Figure 8] FIG. 13 is a diagram for explaining a shooting environment when an image of a model according to an embodiment is shot. [Figure 9]11A and 11B are diagrams showing comparative examples of output results when conversion of a PV template is performed on provisional data and when conversion is performed on actual data, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. Also, the same reference numbers are used for identical or similar configurations, and duplicated descriptions are omitted. Also, in each figure, the up, down, left, right, front and back directions relative to the paper surface are used in the description in the text as the up, down, left, right, front and back directions of the parts (or components) in this embodiment.

[0010] First, the configuration of an image processing system corresponding to this embodiment will be described. FIG. 1(A) is a diagram showing an example of the configuration of an image processing system 10 corresponding to this embodiment. The image processing system 10 is configured by connecting an imaging unit 110, a support arm 120, a model support device 130, a display device 140, a printing device 150, and the like to an information processing device 100. Note that the system configuration is not limited to that shown in FIG. 1(A), and the information processing device 100 may be further connected to an external server, a cloud server, and the like via a network. The external server, etc., can execute at least a part of the processing of the embodiment described below.

[0011] The information processing device 100 controls the operation of at least one of the photographing unit 110, the support arm 120, and the model support device 130, photographs an object to be photographed from an arbitrary angle to generate a plurality of images, and generates three-dimensional model data (main data) from the plurality of images. Then, using a template (PV template) of a moving image (promotional video: PV data) including provisional three-dimensional model data (provisional data) prepared in advance, the provisional data of the PV template is replaced with the main data and converted to reconstruct the PV data. In addition, the information processing device 100 can also function as an image processing device that generates an image to be displayed in a virtual space by using the generated main data as virtual space data. An image processing device may be prepared separately from the information processing device 100. In this embodiment, the object to be photographed is an assembleable plastic model, an action figure (a figure with movable joints), a toy body, a doll, etc., which are collectively referred to as a "model." In addition, in this embodiment, a promotional video (PV) is not limited to a video intended for advertising or promoting the model being filmed, but refers to an attractive moving image (such as a short movie, a movie, or a demo video, and the format and type of the moving image are not limited) using the model being filmed. In the following explanation, "PV" or "promotional video" will be used as a general term including these moving images.

[0012] Next, the photographing unit 110 is a photographing device that photographs (scans) the three-dimensional shape of a model to be photographed under the control of the information processing device 100 and outputs an image of the photographed object. For example, a smartphone with a camera on which an application for photographing three-dimensional shapes is installed can be used as the photographing unit 110. The photographing unit 110 may also be configured as a three-dimensional scanner device capable of outputting three-dimensional shape and color information. For example, Space Spider manufactured by Artec Corporation can be used as the three-dimensional scanner device. For example, 3D model data of the entire model can be obtained by acquiring about 500 to 800 frames of scanned images.

[0013] The support arm 120 is a position and orientation control device that moves the imaging unit 110 to a predetermined imaging position and orientation under the control of the information processing device 100. The support arm 120 may be configured to be capable of manually changing the imaging position and orientation and to be capable of fixing the changed position and orientation, or may be configured to be controllable by the information processing device 100. When the support arm 120 is configured to be controllable by the information processing device 100, for example, xArm 7 manufactured by UFACTORY can be used. The xArm 7 is made of seven joints and can move in the same manner as a human arm. Instead of using the support arm 120, the imaging unit 110 may be positioned manually.

[0014] The model support device 130 is a support stand that supports a model with a fixed pose. The model support device 130 may be configured to be rotatable when the model is placed on the support stand or at the end of a support rod. In this embodiment, the imaging unit 110 is positioned at an arbitrary imaging position and imaging angle by the support arm 120, and then the model support device 130 is rotated once to perform imaging. By performing this at a plurality of imaging positions and imaging angles, an image of the entire model can be obtained. Here, the imaging process can be performed more easily and with higher accuracy by synchronously driving the support arm 120 and the model support device 130. Alternatively, instead of the model support device 130, the imaging unit 110 may be manually moved around the model to perform imaging from an arbitrary imaging position and imaging angle.

[0015] The display device 140 is a display device such as a liquid crystal display (LCD), and displays images captured by the imaging unit 110, the original three-dimensional model data generated from the captured images, and PV data reconstructed using the original data. The display device 140 can also display a two-dimensional barcode (QR code) indicating a link to a download destination on the screen. The printing device 150 is a printing device that prints, on a paper medium or the like, a QR code indicating a link to a download destination so that a user can acquire and view the PV data after it has been reconstructed.

[0016] FIG. 1B shows an example of a hardware configuration of the information processing device 100. The CPU 101 is a device that performs overall control of the information processing device 100 and performs calculation, processing, and management of data. For example, the CPU 101 controls the timing of image capture and the number of images captured by the image capture unit 110, and controls the arm joint of the support arm 120 to place the image capture unit 110 at an arbitrary image capture position and image capture angle. After the image capture position and image capture angle of the image capture unit 110 are determined, the model support device 130 can be rotated to perform an image capture operation by the image capture unit 110. The CPU 101 can also function as an image processing unit that processes images output from the image capture unit 110.

[0017] The RAM 102 is a volatile memory and is used as a temporary storage area such as the main memory and work area of ​​the CPU 101. The ROM 103 is a non-volatile memory, and image data and other data, various programs for the operation of the CPU 101, and the like are stored in respective predetermined areas. The CPU 101 controls each part of the information processing device 100 using the RAM 102 as a work memory according to, for example, a program stored in the ROM 103. Note that the program for the operation of the CPU 101 is not limited to being stored in the ROM 103, and may be stored in the storage device 104.

[0018] The storage device 104 is composed of a magnetic disk such as a HDD or a flash memory. The storage device 104 stores application programs, an OS, control programs, related programs, game programs, etc. It also includes data of a PV template to be reconstructed. The PV template includes provisional data, and the size and pose of the three-dimensional model may differ for each PV template. The storage device 104 can read and write data under the control of the CPU 101. The storage device 104 may be used in place of the RAM 102 or the ROM 103.

[0019] The communication device 105 is a communication interface for communicating with the photographing unit 110, the support arm 120, and the model support device 130 under the control of the CPU 101. The communication device 105 may further be configured to be able to communicate with an external server or the like. The communication device 105 may include a wireless communication module, which may include known circuitry including an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module card, a memory, and the like. Here, communication between the information processing device 100 and the photographing unit 110, the support arm 120, and the model support device 130 may be performed by wireless communication.

[0020] The communication device 105 may also include a wired communication module for wired connection. The wired communication module enables communication with other devices including the display device 140 via one or more external ports. The communication device 105 may also include various software components for processing data. The external ports are connected to other devices directly via Ethernet, USB, IEEE1394, etc., or indirectly via a network. Note that software that realizes the same functions as the above devices may be used to replace the hardware devices.

[0021] The operation unit 106 is composed of, for example, a button, a keyboard, a touch panel, etc., and accepts operation input from a user. The display control unit 107 functions as an interface for displaying information on a display device 140 connected to the information processing device 100, and controls the operation of the display device 140. A part of the functions of the operation unit 106 may be provided in the display device 140. In other words, the display device 140 may be configured as a device equipped with a touch panel such as a tablet terminal.

[0022] Next, referring to FIG. 2, an example of a model to be photographed in this embodiment will be described. The model 200 is a model having a doll-like (robot or human) appearance. The model may be, for example, a plastic model that has been assembled and painted. Alternatively, the model may be a completed model such as a figure (action figure) with movable joints. The model in FIG. 2 is merely an example for the purpose of explanation, and the shape of the model is not limited to a doll-like appearance, but may be any shape such as a general vehicle, a racing vehicle, a military vehicle, an aircraft, a ship, an animal, a virtual life form, etc. It is to be noted that the object to be photographed is not limited to a model as long as the object can be photographed three-dimensionally by the photographing unit 110.

[0023] The model 200 has a head 201, a chest 202, a right arm 203, a left arm 204, a right torso 205, a left torso 206, a right leg 207, a left leg 208, a right foot 209, and a left foot 210, which are joined together to form the model. At least a part of each of the parts 201 to 210 is supported so as to be rotatable (or swingable) relative to the adjacent parts. For example, the head 201 is supported so as to be rotatable relative to the chest 202, and the right arm 203 and the left arm 204 are supported so as to be rotatable relative to the chest 202. In this way, each part of the model 200 is provided with a joint structure, so that the model 200 can take any posture.

[0024] Next, an implementation example of the image processing system 10 in this embodiment will be described with reference to Fig. 3. Fig. 3(A) shows an implementation example in which PV data is reconstructed from main data generated by photographing a model, and a download link for viewing the reconstructed PV data can be issued by a printer. Figs. 3(B) and (C) are diagrams showing configuration examples of an AR (Augmented Reality) marker in this embodiment.

[0025] 3A, the case 301 contains the information processing device 100, a drive system for driving the support arm 302, a drive system for driving the turntable 306, etc. The support arm 302 can also be manually adjusted in shooting direction and position. The surface of the case 301 has a flat structure so that a promotional poster can be attached.

[0026] The support arm 302 corresponds to the support arm 120, and can support and fix the position of the terminal 303 functioning as the imaging unit 110 under the control of the information processing device 100 or manually. The support arm 302 can also operate to control the inclination of the terminal 303.

[0027] The terminal 303 is a touch panel terminal with a built-in camera, and may be, for example, a smartphone, a tablet terminal, or a digital camera. The terminal 303 can take an image of the model 200 and transmit it to the information processing device 100. The ring light 304 is a lighting device used when the terminal 303 takes an image of the model 200, and can evenly shine light on the model 200 to reduce the appearance of shadows. In addition to the ring light 304, a top light or auxiliary lights on the left, right, and bottom may be installed as additional light sources.

[0028] The background sheet 305 is a background sheet for shooting, and for example, a white sheet can be used. The turntable 306 can mount the model 200 and rotate the model. A plurality of AR markers 310 as shown in FIG. 3(B) are arranged on the turntable 306. The AR markers 310 are used to adjust the orientation and position of the photographed model. As shown in FIG. 3(B) and FIG. 3(C), three-dimensional coordinates can be identified from the AR marker 310, and the inclinations of the X-axis, Y-axis, and Z-axis can be identified. For example, when the three-dimensional coordinates identified from the AR marker 310 shown in FIG. 3(B) are used as a reference, the orientation and inclination of the data can be correctly corrected by matching the three-dimensional coordinates identified from the AR marker 310 shown in FIG. 3(C) with the reference.

[0029] In FIG. 3(A), the model is placed on a semi-transparent (transparent) stand, but other support devices such as an "action base" may be used. The action base has a support pole that is configured to be bent into an "U" shape on top of a stand, and the model can be attached to the tip of the pole. In this case, an AR marker may be placed on the tip of the pole. The method of indicating the model can be changed depending on the pose. For example, in the case of an upright posture, the model can be placed on a transparent stand and photographed. On the other hand, in the case of a flying posture, when it is desired to photograph the soles of the feet, an action base may be used. Note that an action base may be used to photograph an upright posture.

[0030] The display device 307 is a device corresponding to the display device 140, and may have a touch panel function. For example, the display device 140 can display icon images as options for PVs that the user wishes to reconstruct, and accept a selection from the user. The printer 308 is, for example, a thermal printer, and can print a QR code indicating a URL for a download destination after reconstruction of PV data is completed. Also, the QR code may be displayed on the display device 307 so that the user can read it with his or her own smartphone or the like.

[0031] Next, the data structures of various tables stored in the storage device 104 of the information processing device 100 will be described with reference to Fig. 4. In a PV management table 400 in Fig. 4(A), model type information 402 for identifying the model type of the model, information 403 of provisional data included in the PV, and a storage destination 404 of the PV template data are registered in association with a PV name 401.

[0032] First, in the PV name 401, identification information for uniquely identifying template data of a promotional video (PV) that is the source of a moving image to be reconstructed in this embodiment is registered. The model type information 402 is information for identifying the type of model to which the PV corresponds. In this embodiment, a PV may be prepared for each type of model, and may be prepared for a model of a normal shape (e.g., an external shape obtained by simply reducing a life-size shape) as a first shape, and a model of a deformed shape (e.g., an external shape obtained by deforming a life-size shape into a two- or three-life size) as a second shape. The model type information 402 indicates that the corresponding PV corresponds to a model of the first shape with M1, and indicates that the corresponding PV corresponds to a model of the second shape with M2.

[0033] The provisional data information 403 registers information on provisional data included in the PV. In this embodiment, the provisional data is used to construct template data for the PV, and the PV can be reconstructed by replacing the provisional data with the real data generated from a captured image. The storage destination 404 indicates information on the storage destination of the template data for the PV.

[0034] Other information related to the PV can be registered in the table 400. This information includes, for example, icon images for selecting a PV, sample movies, information about recommended poses in the PV, and the like.

[0035] 4(B) shows an example of the configuration of a user information table 410 for registering information on users who use the image processing system 10. The table 410 registers a user name 411, a PV name 412, photographed image data 413, a three-dimensional model 414, and a URL 415. The user name 411 is information for uniquely identifying a user (or a player) who is the owner of a model photographed by the photographing unit 110. The PV name 412 is registered as a PV name selected by the user. The PV name 412 corresponds to any one of the pieces of information registered as the PV name 401 in the table 400.

[0036] The captured image data 413 is registered with information on a captured image obtained by the imaging unit 110 capturing an image of a model owned by a user registered in the user name 411. The three-dimensional model 414 is registered with the main data generated based on the captured image data 413. The URL 415 is link information indicating a storage location of PV data reconstructed using the main data of the three-dimensional model 414, and is a URL indicated by a QR code printed by the printing device 150.

[0037] Next, an example of processing executed by the information processing device 100 according to the present embodiment will be described with reference to Fig. 5. At least a part of the processing corresponding to the flowchart is realized by the CPU 101 of the information processing device 100 executing a program stored in the ROM 103 or the storage device 104.

[0038] First, in S501, CPU 101 accepts user registration. Input of the user's name and contact information is accepted. A user identifier for uniquely identifying each individual user is assigned to each user. The user identifier corresponds to user name 411 in FIG. 4(B). CPU 101 stores the input user information in storage device 104 in association with the time when the input was accepted and the user identifier.

[0039] In the next S502, the CPU 101 displays icons of selectable promotional videos (PVs) on the display device 140 via the display control unit 107, and accepts the selection of a PV that the user wishes to create. There are any number of PV icons, and each PV includes a different scene. At this time, a sample movie may be displayed. Also, a PV may be prepared for each type of model, and may be prepared for a model having a normal shape (e.g., an external shape obtained by simply reducing a life-size shape) as a first shape, and a deformed shape (e.g., an external shape obtained by deforming a life-size shape into a two- or three-life size shape) as a second shape. The user can determine whether the shape of the model he or she owns is a normal shape or a deformed shape, and select a PV that corresponds to the shape.

[0040] In the next S503, the CPU 101 accepts a PV selection from the user. Each PV is assigned with identification information in advance, and the identification information of the selected PV is linked to the user information accepted in S501 and stored in the table 410 of the storage device 104.

[0041] In the next step S504, the CPU 101 displays the recommended pose of the model in the selected PV on the display device 140 via the display control unit 107. In this embodiment, the pose of the model is different for each PV, and by photographing the exterior of the model in a pose that matches the selected PV, it is possible to reconstruct a more realistic moving image. Poses of the model in this embodiment include, for example, an upright posture, a flying posture, an attacking posture, and the like. As PVs corresponding to the upright posture, the flying posture, and the attacking posture are respectively prepared, it is important to take a pose that matches the content of each PV in order to create a natural PV.

[0042] The user checks the recommended poses for the model displayed on the display device 140, decides on a pose for the model, and sets it on the model support device 130. The CPU 101 can determine whether or not the model has been set on the model support device 130 based on an image captured by the photographing unit 110. Alternatively, a switch that is turned on when the model is set on the model support device 130 may be provided, and the CPU 101 may detect a signal from the switch to make the determination. Alternatively, a button for accepting an operation when the setting of the model is completed may be displayed on the display device 140, and the CPU 101 may detect whether an operation in response to the button operation has been accepted. In S505, the CPU 101 detects that the model has been set on the model support device 130 by any of the methods described above. In response to this detection, the process proceeds to S506.

[0043] In S506, the photographing unit 110 executes photographing processing to generate an image. The photographed image is transmitted to the information processing device 100, and the CPU 101 stores the photographed image data 413 in the table 410 of the storage device 104 in association with a user identifier, etc. In the following S507, the CPU 101 performs a PV reconstruction process based on the photographed image data 413. The CPU 101 stores the reconstructed PV data in the storage device 104 in association with the user information registered in S501.

[0044] In the next S508, a URL and a QR code are issued for the user to access the PV data. The URL and the QR code may be sent to the user's email address or may be displayed on the display device 140. Upon acquiring the URL or the QR code, the user can access the storage location of the PV data and download the PV data reconstructed in S509.

[0045] Next, the details of the imaging process in S506 will be described with reference to Fig. 6. In this embodiment, the driving information of the support arm 120 and the number of imaging times of the imaging unit 110 at the imaging position are registered for each imaging position. Then, when imaging is performed at each imaging position, imaging is performed by rotating the model support device 130.

[0046] First, in S601, the CPU 101 controls the support arm 120 to move the photographing unit 110 to any of the registered photographing positions. Alternatively, the support arm 120 may be manually controlled to move to any of the photographing positions. In addition, the CPU 101 controls the model support device 130 to move to a rotation start position. In the following S602, the CPU 101 rotates the model support device 130 at the photographing position while the photographing unit 110 photographs, thereby acquiring an image of the model 200. This makes it possible to acquire multiple images with different photographing angles. For example, when the photographing angle is changed by 15 degrees, 24 images can be captured, and when the photographing angle is changed by 10 degrees, 36 images can be captured. By making the change width of the photographing angle smaller, the accuracy of the main data generated in the later stage is improved.

[0047] Also, at this time, the model support device 130 may be stopped at a specific rotation position, and the support arm 120 may be controlled, or the photographer may manually change the angle of the imaging unit 110, so that the model is photographed while moving the imaging unit 110 vertically. This allows the model to be photographed from above or below. In particular, photographing from below makes it possible to photograph the soles of the model's feet. In this way, in this embodiment, the photographing direction and angle are selected so as to cover the entire circumference of the model, and photographing is performed.

[0048] In this embodiment, the model can be set on a transparent support rod extending from the model support device 130, allowing it to be photographed floating. Alternatively, it can be photographed by standing it directly upright on the turntable of the model support device 130. In the former shooting mode, the soles of the model's feet can also be photographed, making it suitable for photographing the flying posture. On the other hand, in the latter shooting mode, the soles of the feet cannot be photographed, but it is suitable for photographing the upright posture. Note that the upright posture may also be photographed in the former shooting mode.

[0049] In addition, AR markers are placed on the support rods and turntable of the model support device 130, and when taking a photograph, an image is generated including the AR markers. The AR markers are used to correct the orientation and position of the object to be photographed.

[0050] In the next S603, the CPU 101 determines whether or not there is an unselected shooting position among the shooting positions set in advance, and if there is an unselected shooting position, the process returns to S601 and continues. On the other hand, if there is no unselected shooting position, the process proceeds to S604. In S604, the shooting unit 110 transmits the image obtained as described above to the information processing device 100, and the CPU 101 associates the received image with user information and the like and stores it as shot image data 413 in the table 410 of the storage device 104.

[0051] Next, the details of the process of S507 for reconstructing a PV based on the captured images obtained in S506 will be described with reference to Fig. 7. First, in S701, CPU 101 acquires a plurality of images captured by imaging unit 110 and stored as captured image data 413 in storage device 104. In the following S702, CPU 101 generates 3D model data of the model, which will become the main data, from a plurality of images of the acquired model captured from different angles.

[0052] In the method for generating 3D model data, for example, a 3D model corresponding to the shape of the model can be determined by a known volume intersection method using images of the model acquired from multiple images. In this volume intersection method, a cone (volume) is created using multiple model images, with the camera position as the apex and the silhouette as the cross section, the silhouette is back-projected into 3D space, and the 3D shape is restored as the intersection (common part) of the cone.

[0053] In S703, CPU 101 applies regulations regarding the position and size of the model set in advance, and performs resizing of the data generated in S702. This resizing is a process for adjusting the size so that the model is not cut off, since if the model exceeds the angle of view of the planned PV, a part of the model will protrude from the display screen frame and will be cut off. The size of the model is predetermined for each PV in terms of length, width, height, and depth, and if any of them exceeds the designated size, the model is resized to fit within the designated size. Note that the resizing is performed on the model of the first shape for the PV for the first shape, but does not have to be performed on the model of the second shape. Similarly, the resizing is performed on the model of the second shape for the PV for the second shape, but does not have to be performed on the model of the first shape.

[0054] In the next step S704, the CPU 101 adjusts the direction and position of the model resized by the regulation. This process may be performed before step S703. The three-dimensional model includes an AR marker, and the direction and position of the front of the three-dimensional model are determined according to the orientation of the AR marker. The AR marker indicates how much the three-dimensional model is tilted or displaced in the XYZ three-dimensional coordinate system, so the tilt and displacement are reset and the three-dimensional coordinate system is reset. Alternatively, a correction value in the original three-dimensional coordinate system may be calculated, and then the correction value may be used to adjust the direction and position of the three-dimensional model.

[0055] Since the direction and position of the provisional data included in the PV template have already been corrected, by correcting the actual data to be generated, even if the provisional data is replaced, there will be no deviation in the direction or position of the model in the PV. After the process of S704, the AR marker can be removed from the actual data.

[0056] In addition, in S704, the origin of this data is determined. In the case of a model in an upright posture, since it is placed on the ground or the like and displayed, the origin is set at the bottom of this data, and the direction and position of this data are adjusted. In the case of a pose in an upright posture, the sole of the foot becomes the origin, so any model can be placed on the ground. In addition, in the case of a flying posture rather than an upright posture, the position of the origin may be set at the center of gravity of the model, not at the bottom. Note that if the model has weapons or other equipment, the center of gravity may move due to their influence, so when setting the center of gravity, the center of gravity may be set to the center of gravity of only the model body excluding weapons and equipment. The origin set in this data is used as a reference for determining the placement position of this data to be replaced when replacing it with the provisional data of the PV template.

[0057] In the next step S705, the real data generated in the above steps is replaced with the provisional data included in the PV template, and output as PV video data. In the PV template, the movement of the provisional data and the camera angle (viewpoint position) are set along with the movement of the background and objects and the camera angle, and the template data can be converted into viewable video data and output. Here, by replacing the provisional data with the real data generated in S702 and converting it into video data, it becomes possible to reconstruct PV data using any real data. The PV template can be generated, for example, using UNREAL ENGINE 5 (registered trademark).

[0058] More specifically, the PV template includes, in accordance with the time axis of the PV, information on the position of objects and models in a three-dimensional space, information on the action start positions of objects and models and their movements in a three-dimensional space (positional changes over time), information on camera angles, information on effects such as lighting, etc. In this template data, the provisional data is replaced with the real data generated in S702, and is operated and moved in a three-dimensional space, which is then cut out as two-dimensional images in frame units and compressed using any video encoding method such as H264, thereby generating video data as PV data.

[0059] Alternatively, the camera angle of the three-dimensional model data is specified for each frame constituting the PV data, and the PV may be reconstructed by generating a texture of the three-dimensional model corresponding to each camera angle and replacing the texture of the provisional data on a frame-by-frame basis. In the following S706, the CPU 101 stores the PV data reconstructed in S705 in the storage device 104.

[0060] In the above description, a case has been described in which the image capturing unit 110 transmits a plurality of images to the information processing device 100 to generate the present data, but an external server or a cloud server may be used for the process of generating the present data from a plurality of images. In this case, the information processing device 100 transmits a plurality of images to the external server and can obtain the present data as a processing result. Similarly, an external server or a cloud server may be used for the PV reconstruction process in S705.

[0061] Fig. 8 is a diagram for explaining lighting when photographing model 200 by photographing unit 110. Fig. 8(A) is a diagram showing an example of lighting arrangement seen from above model 200, in which ring light 304 is arranged in front of model 200, which is the subject of photographing. In this case, if ring light 304 alone illuminates model 200, which is the subject of photographing, a shadow will be cast on the back of model 200 with only light L0 emitted from ring light 304, so auxiliary lights 801 and 802 are arranged on the sides of ring light 304, and light L1 and L2 emitted from each are used to prevent the shadow from being included in the photographing range.

[0062] 8(B) is a diagram showing an example of lighting arrangement when model 200 is viewed from the side, with a top light 803 arranged above ring light 304 in front of subject 200, and an auxiliary light 804 arranged below. This is to allow the top light 803 to illuminate the head of model 200 so that an image of the head can be obtained mainly as a line, and to offset the shadow caused by light L3 irradiated by top light 803 with light L4 irradiated by auxiliary light 804. Note that light L3 irradiated by top light 803 is also offset by light L1 and L2 irradiated by auxiliary lights 801 and 802 described above.

[0063] It is not necessary to use all of the auxiliary lights 801, 802, 804 and top light 803 shown in Figures 8(A) and 8(B), and at least one of them can be used to offset the shadow cast by the ring light 304.

[0064] 9 shows a comparative example of the output results when PV template conversion is performed on provisional data and when it is performed on actual data. In FIG. 9(A), the output result when conversion is performed with a PV template using provisional data 901 is displayed in scene 900. In contrast, the output result when conversion is performed with a PV template using actual data 902 is displayed in the reconstructed scene 900. The actual data 902 is placed in the same position as the provisional data 901, and it can be seen that replacement has been performed. FIG. 9(B) shows another scene 910 in the same PV, where provisional data 911 has also been replaced with actual data 912 in scene 910.

[0065] In this manner, in this embodiment, by using a PV template generated for provisional data, it becomes possible to efficiently reconstruct similar PV data simply by replacing the provisional data with real data.

[0066] In the above-described embodiment, a case has been described in which a promotional video is generated in which three-dimensional model data generated from an image obtained by photographing a model is displayed in a virtual space. However, the embodiment is not limited to the generation of PV data, and the three-dimensional model data can also be used as game data for the virtual space. The present invention can also be widely applied to a process of moving a character in a virtual space. For example, the character can be made to appear in an event, concert, sports, online conference, etc., held in a virtual space. Furthermore, the technology of the present embodiment can be applied to a video technology such as cross reality (XR) that fuses the real world with the virtual world and makes it possible to perceive things that do not exist in the real space.

[0067] In this way, in this embodiment, three-dimensional model data is generated from images of the exterior of the model, and this can be made to appear in a promotional video of the user's choice. Models, such as prefabricated plastic models, can be finished as unique works by the user through painting, etc., and these individual characteristics can be reflected in the character representation in the video or virtual space, greatly improving the user's preferences.

[0068] <Summary of the embodiment> The above-described embodiments disclose at least the following image processing method, information processing device, and computer program.

[0069] (1) An image processing method for reconstructing a moving image, comprising the steps of: A step of generating, by a processing unit, first three-dimensional model data of the model included in a plurality of images generated by capturing images of the appearance of the model from a plurality of directions using a capturing unit; a processing unit replacing the first three-dimensional model data with the second three-dimensional model data in template data of the moving image, the template data including the second three-dimensional model data, thereby reconstructing the moving image; an output unit outputting information indicating a source of the reconstructed moving image; An image processing method comprising:

[0070] (2) the plurality of images includes markers for identifying an orientation and a position of the model; The image processing method according to (1), wherein the first three-dimensional model data is generated by correcting an orientation and a position based on the marker.

[0071] (3) A plurality of template data for the moving image are prepared according to the type of the model; The image processing method according to (1) or (2), wherein in the reconstructing step, the video is reconstructed for template data selected by a user.

[0072] (4) The image processing method described in (3), wherein the types of models include a first type of model having an exterior shape that is a reduced version of a life-size shape, and a second type of model having an exterior shape that is a deformed version of a life-size shape.

[0073] (5) The image processing method according to (3) or (4), wherein in the generating step, the first three-dimensional model data is generated with its size adjusted according to the content of the moving image.

[0074] (6) The image processing method described in (5), wherein the size adjustment is performed on the first three-dimensional model data of a model of the same type as the type of the model in the template data of the video, and is not performed on the first three-dimensional model data of a type different from the type of the model in the template data of the video.

[0075] (7) A pose recommended for the model is associated with the template data of the moving image; The image processing method according to any one of (1) to (6), wherein the plurality of images are generated by photographing the appearance of the model in the recommended pose.

[0076] (8) The image processing method according to any one of (1) to (7), further comprising the step of the processing unit accepting a selection of template data for the moving image.

[0077] (9) The image processing method according to (8), further comprising a step in which the processing unit, in response to receiving a selection of template data for the video, causes a display device to display a specification of a recommended pose for the video.

[0078] (10) The image processing method according to any one of (1) to (9), wherein the multiple directions are defined so as to cover the entire circumference of the model.

[0079] (11) An image processing method according to any one of (1) to (10), wherein the plurality of images are generated by the photographing unit photographing the model illuminated by a first light source and at least one second light source that cancels out shadows caused by light emitted from the first light source.

[0080] (12) The image processing method according to any one of (1) to (11), further comprising the step of the processing unit receiving the plurality of images from the photographing unit.

[0081] (13) The image processing method according to any one of (1) to (12), wherein the template data includes at least one of information regarding the position, movement, camera angle, and lighting of the second three-dimensional model data in three-dimensional space, which is set in accordance with the time axis of the video.

[0082] (14) A generating means for generating first three-dimensional model data of the model included in a plurality of images generated by photographing images of the exterior of the model from a plurality of directions; a reconstruction means for reconstructing the moving image by replacing the first three-dimensional model data with the second three-dimensional model data in template data of the moving image that is previously configured to include the second three-dimensional model data; a control means for controlling the output unit to output information indicating the source of the reconstructed video; An information processing device comprising:

[0083] (15) A computer program for causing a computer to execute the image processing method according to any one of (1) to (13).

[0084] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. 1. An image processing method for reconstructing a moving image, comprising: A step of generating, by a processing unit, first three-dimensional model data of the model included in a plurality of images generated by capturing images of the appearance of the model from a plurality of directions by a capturing unit; a processing unit replacing the first three-dimensional model data with the second three-dimensional model data in template data of the moving image, the template data including the second three-dimensional model data, thereby reconstructing the moving image; a step of receiving a selection of template data of the video by the processing unit; In response to receiving the selection of template data for the video, the processing unit causes a display device to display a recommended pose designation for the video; An image processing method comprising:

2. the plurality of images includes markers for identifying an orientation and a position of the model; The image processing method according to claim 1 , wherein the first three-dimensional model data is generated with its orientation and position corrected based on the marker.

3. A plurality of template data for the moving image are prepared according to the type of the model, The image processing method according to claim 1 , wherein in the reconstructing step, the moving image is reconstructed for template data selected by a user.

4. 4. The image processing method according to claim 3, wherein the types of models include a first type of model having an external shape that is a reduced version of a life-size shape, and a second type of model having an external shape that is a deformed version of a life-size shape.

5. 4. The image processing method according to claim 3, wherein in said generating step, said first three-dimensional model data is generated with its size adjusted according to the content of said moving image.

6. 6. The image processing method according to claim 5, wherein the size adjustment is performed on the first three-dimensional model data of a model of the same type as that of the template data of the moving image, and is not performed on the first three-dimensional model data of a type different from that of the model of the template data of the moving image.

7. A recommended pose for the model is associated with the template data of the motion image; The image processing method according to claim 1 , wherein the plurality of images are generated by photographing the appearance of the model in the recommended pose.

8. The image processing method according to claim 1 , further comprising the step of: the processing unit accepting a selection of template data for the moving image.

9. The image processing method according to claim 8 , further comprising the step of: in response to receiving a selection of template data for the moving image, the processing unit causing a display device to display a designation of a recommended pose for the moving image.

10. The image processing method according to claim 1 , wherein the multiple directions are determined so as to cover an entire circumference of the model.

11. 2. The image processing method according to claim 1, wherein the plurality of images are generated by the photographing unit photographing the model illuminated by a first light source and at least one or more second light sources that cancel out shadows caused by light irradiated from the first light source.

12. The image processing method according to claim 1 , further comprising the step of: the processing unit receiving the plurality of images from the imaging unit.

13. 2. The image processing method according to claim 1, wherein the template data includes at least one of information regarding the position, movement, camera angle, and lighting of the second three-dimensional model data in a three-dimensional space, which is set in accordance with a time axis of the moving image.

14. a generating means for generating, from a plurality of images generated by photographing images of the exterior of the model from a plurality of directions, first three-dimensional model data of the model contained in the images; a reconstruction means for reconstructing the moving image by replacing the first three-dimensional model data with the second three-dimensional model data in template data of the moving image that is previously configured to include the second three-dimensional model data; a control means for controlling the output unit to output information indicating the source of the reconstructed video; An information processing device comprising:

15. A computer program for causing a computer to execute the image processing method according to any one of claims 1 to 13.

Images