Image creation method, image creation program, image creation device, and image creation system

Abstract

To provide an image forming method capable of further mitigating a load of work for setting a common feature point.SOLUTION: An image forming method includes: a pre-photographing step of photographing a reference object with a plurality of three-dimensional sensors, and acquiring images of respective three-dimensional sensors as a one frame reference image; a feature point setting step of setting a common feature point on the basis of an image of a marker arranged on a reference object in an area where reference images of adjacent frames overlap each other; a conversion matrix calculation step of calculating a conversion matrix so that feature points of reference images in adjacent frames match each other; a main photographing step of photographing a target object with the plurality of three-dimensional sensors, and acquiring the images of respective three-dimensional sensors as one frame target image; and an image formation step of forming a three-dimensional model image by performing coordinate conversion to the acquired target images of adjacent frames on the basis of the conversion matrix between the same frames calculated by the conversion matrix calculation step.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image creation method, an image creation program, an image creation device, and an image creation system.

Background Art

[0002] Conventionally, as a method for creating a three-dimensional image from 360 degrees around a target object, for example, a method of measuring in advance the positional relationship between a three-dimensional sensor and the target object and performing alignment between images based on the measured positional relationship, or a method of rotating the target object with respect to a fixed three-dimensional sensor and measuring the positional relationship between the three-dimensional sensor and the target object from various directions is known. Further, a method of photographing a target object with a three-dimensional sensor installed around a person and creating a three-dimensional image from 360 degrees around the person by aligning the respective images is also known (see, for example, Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the method of photographing a target object with a three-dimensional sensor installed around a person and creating a three-dimensional image by coordinate-transforming each image as in the above patent documents, for example, the work is performed in the following procedure. (a) Install a plurality of three-dimensional sensors around the person. (b) Photograph the person with the plurality of three-dimensional sensors and acquire the images of each frame (the area of the image photographed by one three-dimensional sensor). (c) In areas where images from adjacent frames overlap, the operator manually sets common feature points. (d) Coordinate transformation is performed on images from adjacent frames so that their feature points coincide, and a 3D image is created from 360 degrees around the target object.

[0005] In step (c) above, the operator displays the captured images of adjacent frames on the monitor screen and compares the overlapping areas of the images to set common feature points. This process is performed for all adjacent frames. When the target object is a person, the eyes, nose, lips, ears, etc., are often set as feature points, but in images of the head, hair makes up the majority. Therefore, it is difficult to find feature points, the process of setting feature points is time-consuming and laborious, and the possibility of errors is high.

[0006] The object of the present invention is to provide an image creation method, an image creation program, an image creation apparatus, and an image creation system that can further reduce the burden of setting common feature points. [Means for solving the problem]

[0007] The present invention solves the above problem by the following means. The first invention relates to an image creation method comprising: a pre-shooting step of photographing a reference object with a plurality of three-dimensional sensors installed around the reference object and acquiring an image from each of the three-dimensional sensors as a reference image for one frame; a feature point setting step of setting common feature points based on images of markers provided on the reference object in the region where the reference images of adjacent frames overlap; a transformation matrix calculation step of performing a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and calculating the transformation matrix used for the coordinate transformation, for all the reference images of adjacent frames; a main shooting step of photographing a target object with a plurality of the three-dimensional sensors and acquiring an image from each of the three-dimensional sensors as a target image for one frame; and an image creation step of performing a coordinate transformation on the target images of adjacent frames acquired in the main shooting step based on the transformation matrix between the same frames calculated in the transformation matrix calculation step, to create a three-dimensional model image of the target object. The second invention relates to an image creation method according to the first invention, wherein the reference object is a model body created to resemble the target object. The third invention relates to an image creation method according to the first or second invention, wherein the marker is an identification information marker to which identification information has been added, and in the feature point setting step, a common feature point is set based on the identification information added to the identification information marker. The fourth invention relates to an image creation method comprising: a pre-shooting step of photographing a reference object with a plurality of 3D sensors installed around the reference object and acquiring an image from each of the 3D sensors as a reference image for one frame; a feature point setting step of setting common feature points by image recognition in the region where the reference images of adjacent frames overlap; a transformation matrix calculation step of performing a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and calculating the transformation matrix used for that coordinate transformation, for all the reference images of adjacent frames; a main shooting step of photographing a target object with a plurality of the 3D sensors and acquiring an image from each of the 3D sensors as a target image for one frame; and an image creation step of performing a coordinate transformation on the target images of adjacent frames acquired in the main shooting step based on the transformation matrix between the same frames calculated in the transformation matrix calculation step, to create a 3D model image of the target object. The fifth invention relates to an image creation program executed on a computer, wherein the computer functions as: a feature point setting unit that acquires images of a reference object captured by a plurality of three-dimensional sensors from each of the three-dimensional sensors as a reference image for one frame, and sets common feature points based on images of markers provided on the reference object in the region where the reference images of adjacent frames overlap; a transformation matrix calculation unit that performs a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and calculates the transformation matrix used for the coordinate transformation, for all the reference images of adjacent frames; and an image creation unit that acquires images of a target object captured by a plurality of three-dimensional sensors from each of the three-dimensional sensors as a target image for one frame, and performs a coordinate transformation on the target images of adjacent frames based on the transformation matrix between the same frames acquired by the transformation matrix calculation unit, to create a three-dimensional model image of the target object. The sixth invention relates to an image creation program according to the fifth invention, which sets common feature points based on identification information attached to the markers. The seventh invention relates to an image creation program executed on a computer, wherein the computer functions as: a feature point setting unit that acquires images of a reference object captured by a plurality of three-dimensional sensors from each of the three-dimensional sensors as a reference image for one frame, and sets common feature points by image recognition in the region where the reference images of adjacent frames overlap; a transformation matrix calculation unit that performs a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and calculates the transformation matrix used for that coordinate transformation, for all the reference images of adjacent frames; and an image creation unit that acquires images of a target object captured by a plurality of three-dimensional sensors from each of the three-dimensional sensors as a target image for one frame, and performs a coordinate transformation on the target images of adjacent frames based on the transformation matrix between the same frames acquired by the transformation matrix calculation unit, and creates a three-dimensional model image of the target object. The eighth invention relates to an image creation device comprising: a feature point setting unit that acquires images of a reference object captured by a plurality of three-dimensional sensors arranged around the reference object as a reference image for one frame, and sets common feature points based on images of markers provided on the reference object in the region where the reference images of adjacent frames overlap; a transformation matrix calculation unit that performs a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and calculates the transformation matrix used for the coordinate transformation, for all the reference images of adjacent frames; and an image creation unit that acquires images of a target object captured by a plurality of three-dimensional sensors as a target image for one frame, and performs a coordinate transformation on the target images of adjacent frames based on the transformation matrix between the same frames acquired by the transformation matrix calculation unit, to create a three-dimensional model image of the target object. The ninth invention relates to an image creation apparatus according to the eighth invention, wherein the marker is an identification information marker to which identification information has been added, and the feature point setting unit sets common feature points based on the identification information added to the identification information marker. The tenth invention relates to an image creation device comprising: a feature point setting unit that acquires images of a reference object captured by a plurality of three-dimensional sensors arranged around the reference object as a reference image for one frame, and sets common feature points by image recognition in the region where the reference images of adjacent frames overlap; a transformation matrix calculation unit that performs a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and calculates the transformation matrix used for the coordinate transformation, for all the reference images of adjacent frames; and an image creation unit that acquires images of a target object captured by a plurality of three-dimensional sensors as a target image for one frame, and performs a coordinate transformation on the target images of adjacent frames based on the transformation matrix between the same frames acquired by the transformation matrix calculation unit, to create a three-dimensional model image of the target object. The eleventh invention relates to an image creation device according to any of the eighth to tenth inventions, and a plurality of three-dimensional sensors that photograph the reference object from all sides and output one frame of the reference image to the feature point setting unit, and photograph the target object from all sides and output one frame of the target image to the feature point setting unit. [Effects of the Invention]

[0008] According to the image creation method, image creation program, image creation apparatus, and image creation system of the present invention, the burden of setting common feature points can be further reduced. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the overall configuration of the image creation system 100 according to the first embodiment. [Figure 2] This is a functional block diagram of the image creation system 100. [Figure 3] This diagram illustrates the placement of the subject P1 and the marker with identification information. [Figure 4] (A) to (C) are diagrams illustrating the different types of markers. [Figure 5] (A) and (B) are diagrams for explaining the difference due to the shooting position of the marker M2 with information. [Figure 6] (A) to (C) are conceptual diagrams for explaining an example in which the target images of adjacent frames are subjected to coordinate transformation based on a transformation matrix. [Figure 7] It is a flowchart showing the procedure of the process for calculating the transformation matrix from the captured image (reference image) of the subject P1. [Figure 8] It is a flowchart showing the procedure of the process for creating a 3D model image from the captured image (target image) of the subject P2. [Figure 9] In the second embodiment, it is a flowchart showing the procedure of the process for calculating the transformation matrix from the captured image (reference image) of the subject P1.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of an image creation method, an image creation program, an image creation device, and an image creation system according to the present invention will be described. Note that all the drawings attached to this specification are schematic diagrams, and in consideration of ease of understanding and the like, the shapes, scales, vertical and horizontal dimension ratios, etc. of each part are changed or exaggerated from the actual object.

[0011] (First Embodiment) FIG. 1 is a diagram showing the overall configuration of the image creation system 100 according to the first embodiment. FIG. 2 is a functional block diagram of the image creation system 100. FIG. 3 is a diagram for explaining the arrangement of the subject P1 and the marker with identification information. FIGS. 4(A) to (C) are diagrams for explaining the types of markers. FIGS. 5(A) and (B) are diagrams for explaining the difference due to the shooting position of the marker M2 with information. FIGS. 6(A) to (C) are conceptual diagrams for explaining an example in which the target images of adjacent frames are subjected to coordinate transformation based on a transformation matrix.

[0012] The image creation system 100 shown in FIG. 1 is a system that creates a three-dimensional image of a person or the like. The three-dimensional image created by the image creation system 100 can be used, for example, in games or events held in commercial facilities, or for personal authentication in office buildings.

[0013] As shown in FIG. 1, the image creation system 100 of the first embodiment includes 3D sensors (three-dimensional sensors) 1a to 1f and an image creation device 2. In the following description, when the 3D sensors 1a to 1f are not distinguished, they are simply referred to as "3D sensors". In this embodiment, as shown in FIG. 1, the 3D sensor 1a located at the left end in the figure is defined as the 3D sensor located in front of the subject P.

[0014] The 3D sensors 1a to 1f are devices that capture a 3D image including RGB image information and depth information of the subject P. As the 3D sensor, for example, a TOF camera that irradiates the subject with infrared rays to calculate the depth can be used. As shown in FIG. 1, the six 3D sensors 1a to 1f are installed at equal intervals (60 degrees) so as to surround the subject P 360 degrees. Each 3D sensor is supported by a camera stand (not shown), but may be installed on a frame-shaped pedestal installed so as to surround the subject P. The 3D sensors 1a to 1f are installed such that the respective captured images captured by adjacent 3D sensors include overlapping regions. As a result, the respective captured images captured by adjacent 3D sensors include common feature points (images of the marker M described later).

[0015] In this embodiment, the subject P is a general term for a model body (reference object) or a person (target object) created by imitating a person. The model body is, for example, the head of a mannequin. In the following description, when the subject P is a model body, it is referred to as "subject P1", and when it is a person, it is referred to as "subject P2". When the model body and the person are not distinguished, they are simply referred to as "subject P".

[0016] Each 3D sensor captures the subject P under the control of a control unit 10 (described later) located in the image creation device 2. Each 3D sensor captures the subject P simultaneously, and the captured 3D images are transmitted (output) to the image creation device 2 via cables (not shown) and stored in the storage unit 20 (described later). In this specification, the area (image range) in which the subject P is captured by one 3D sensor is referred to as "one frame" or "frame". One 3D sensor captures a reference image and a target image, described later, as 3D images for one frame. In the following description, the 3D images for one frame captured by one 3D sensor are collectively referred to as "captured images".

[0017] Image creation device 2 is a device that performs coordinate transformations so that multiple 3D images captured by a 3D sensor are continuous, and creates a 3D image (hereinafter also called "3D model image") from 360 degrees around the subject P. Image creation device 2 is an information processing device such as a personal computer (PC) or a tablet terminal. As shown in Figure 2, image creation device 2 comprises a control unit 10, a storage unit 20, an input unit 30, and a display unit 40.

[0018] The control unit 10 is a central processing unit (CPU) that controls the entire image creation device 2. The control unit 10 works in cooperation with each piece of hardware to perform various functions by appropriately reading and executing the operating system (OS) and various application programs stored in the memory unit 20. The control unit 10 includes a feature point setting unit 11, a transformation matrix calculation unit 12, and an image creation unit 13 as functional blocks executed by the image creation program, transformation matrix calculation program, etc.

[0019] The feature point setting unit 11 acquires images of the subject P1 (model body) captured by each 3D sensor as reference images for one frame. Then, it performs image recognition on the acquired reference images and sets common feature points in the region where the reference images of adjacent frames overlap, based on the images of markers M (described later) placed on the subject P1. The reference images of adjacent frames are, for example, a combination of a reference image of one frame captured by 3D sensor 1a and a reference image of one frame captured by 3D sensor 1f in Figure 1.

[0020] As shown in Figure 3, the subject P1 is provided with multiple markers M. Markers M are markers placed at positions corresponding to the eyes, nose, lips, ears, etc., of a person. The markers M on the subject P1 become common feature points in the region where the reference images of adjacent frames overlap. The markers M are arranged so that in the region where the reference images of adjacent frames overlap, each frame's captured image contains three or more images of the same marker M. For example, in Figure 3, the marker M shown by the dashed circle is, for example, a marker that appears in the respective reference images captured by 3D sensors 1a and 1f.

[0021] Markers M must be present in at least three of the captured image in each frame. The more markers M that serve as feature points there are, the better the accuracy of the coordinate transformation described later. Note that if the markers M are placed close together, the accuracy of the coordinate transformation will decrease, so it is desirable to place them far apart on the subject P1 captured in a single frame. Also, in order not to degrade the amount of data on the surface of the subject P1, it is desirable to make the markers M as small as possible while still being identifiable by image recognition.

[0022] The marker M1(M) shown in Figure 4(A) is a marker without identification information. A marker without identification information is, for example, a plain, round sticker as shown in Figure 4(A). The markers M2(M) shown in Figures 4(B) and (C) are markers with identification information (markers with identification information). Identification information is, for example, information that makes it possible to identify the corresponding marker by performing image recognition on the captured image. The marker M2 shown in Figure 4(B) is composed of a one-dimensional barcode. The marker M2 shown in Figure 4(C) is composed of a two-dimensional barcode. In either case, the identification information recorded on the marker can be read by performing image recognition on the captured image. In the following explanation, a marker without identification information will also be called "marker M1," and a marker with identification information will be called "marker M2." Also, when markers M1 and M2 are not distinguished, they will simply be called "marker."

[0023] Figures 5(A) and (B) show the same markers M2a to M2c (three locations) in images captured by 3D sensors at different positions. Each marker M2a to M2c has different identification information attached to it. Figure 5(A) is an image captured by 3D sensor 1a located in front of the subject P, and Figure 5(B) is an image captured by 3D sensor 1f, which is positioned 60 degrees to the left of 3D sensor 1a (0 degrees). Note that "rotated position" means the position rotated around the subject P as the central axis in Figure 1. By performing image recognition on each captured image, it can be determined that the markers M2 at each position shown in Figures 5(A) and (B) are the same marker.

[0024] As shown in Figures 5(A) and (B), the feature point setting unit 11 can extract markers M2a to M2c, which are common feature points, based on the marker images in region A where the reference images of adjacent frames overlap, using image recognition. Note that in Figures 5(A) and (B), region A where the reference images overlap is conceptually represented. The feature point setting unit 11 stores the position information of the extracted markers M2a to M2c in the storage unit 20, associating it with the identification information of each marker M2a to M2c. For all markers M2a to M2c extracted from the reference image, the feature point setting unit 11 performs a process to store the position information and identification information of each marker M2a to M2c (feature point) as an association.

[0025] The transformation matrix calculation unit 12 performs a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and calculates the transformation matrix used for that coordinate transformation, performing this process for all adjacent reference images. For the coordinate transformation between adjacent reference images, for example, a transformation matrix calculation program (transformation matrix calculation unit 12) that corresponds to the ICP (Iterative Closest Point) algorithm can be used. Specifically, the transformation matrix calculation program can perform a coordinate transformation between adjacent reference images by inputting the data of each reference image of adjacent frames and the position information of the feature points associated with those reference images.

[0026] The transformation matrix calculation program calculates a transformation matrix, which is a parameter that minimizes the distance between common feature points in the region where the reference images of adjacent frames overlap, based on the input reference image data and the position information of common feature points. The transformation matrix is ​​a determinant that represents the amount of movement and rotation in the XYZ directions of the image to be transformed relative to the reference image (see Figure 6(B)). The transformation matrix calculation unit 12 uses the transformation matrix calculation program to calculate the transformation matrix for all adjacent reference images. The same number of transformation matrices are calculated as the number of 3D sensors.

[0027] The image creation unit 13 acquires the 3D sensor images of the subject P2 (person) captured by each 3D sensor as target images for one frame, and performs coordinate transformations on the target images of adjacent frames based on the transformation matrix between the same frames calculated by the transformation matrix calculation unit 12 to create a 3D model image of the subject P2.

[0028] Here, we will explain an example in which the image creation unit 13 performs a coordinate transformation on adjacent target images based on a transformation matrix. Figure 6(A) is a diagram illustrating the target image captured by the 3D sensor 1a located in front of the subject P2 in Figure 1. Figure 6(B) is a diagram illustrating the target image captured by the 3D sensor 1f, which is installed in Figure 1 at a position rotated 60 degrees to the left with 3D sensor 1a as the reference (0 degrees).

[0029] When performing a coordinate transformation on an image captured by 3D sensor 1a using an image captured by 3D sensor 1f, the data of the image captured by 3D sensor 1f is multiplied by a transformation matrix, as shown in Figure 6(B). Then, the data of the image captured by 3D sensor 1f, multiplied by the transformation matrix, is combined with the image captured by 3D sensor 1a. This allows for the coordinate transformation of the image captured by 3D sensor 1f to be performed on the image captured by 3D sensor 1a.

[0030] Figure 6(C) shows an example of an image obtained by performing a coordinate transformation on the target image captured by 3D sensor 1a and the target image captured by 3D sensor 1f. Specifically, Figure 6(C) shows a portion of the continuous 3D model images around subject P2, from the position of 3D sensor 1a to 3D sensor 1f, which is rotated 60 degrees to the left. By performing the same process as described above—performing a coordinate transformation on the target image captured by 3D sensor 1a and the target image captured by 3D sensor 1f—for all adjacent frames of target images, a 3D model image of subject P2 viewed from 360 degrees around can be created. The transformation matrix shown in Figure 6(B) is an example of an actually calculated transformation matrix.

[0031] Returning to Figure 2, the storage unit 20 is a storage medium such as a hard disk or semiconductor memory for storing programs, data, etc., necessary for the control unit 10 to perform various processes. The storage unit 20 includes a program storage unit 21, a data storage unit 22, and an image storage unit 23. The program storage unit 21 is a storage area for storing programs. The program storage unit 21 stores control programs for executing various functions of the control unit 10, image creation programs for the control unit 10 to execute image creation processing, transformation matrix calculation programs for the control unit 10 to calculate transformation matrices, image recognition programs for performing image recognition in image creation processing, and the like. The data storage unit 22 is a storage area for storing various types of data. The data storage unit 22 stores, for example, marker position information, transformation matrix data, and the like. The image storage unit 23 is a storage area for storing data of various images. The image storage unit 23 stores, for example, captured image data, 3D model image data, and the like.

[0032] The input unit 30 is a device for the operator to input various data, instructions, etc., to the image creation device 2. The input unit 30 is composed of, for example, a keyboard, mouse, etc. The display unit 40 is a device that displays captured images, etc., on a screen. The display unit 40 is composed of, for example, a liquid crystal display device.

[0033] Next, the process of creating a 3D model image of subject P2 (person) using the image creation system 100 of the first embodiment will be explained with reference to the flowcharts in Figures 7 and 8. First, the process of calculating a transformation matrix from a captured image of subject P1 (model body) will be explained. Figure 7 is a flowchart showing the procedure for calculating a transformation matrix from a captured image (reference image) of subject P1. In the pre-shooting step (described later) of the first embodiment, the subject P1 to be photographed is a model body equipped with a marker M2 (marker with identification information). The flowcharts shown in Figures 7 and 8 (described later) are executed by the control unit 10 reading and executing the image creation program, transformation matrix calculation program, etc., stored in the storage unit 20.

[0034] In step S1 shown in Figure 7, the control unit 10 (feature point setting unit 11) controls each 3D sensor installed around the subject P1 (model body) to photograph the subject P1, and acquires the image of the subject P1 captured by each 3D sensor as a reference image for one frame (pre-shooting step). The reference image data acquired from each 3D sensor is stored in the storage unit 20 (image storage unit 23).

[0035] In step S2, the control unit 10 (feature point setting unit 11) performs image recognition on the reference images acquired from each 3D sensor and sets common feature points based on the marker images in the region where the reference images of adjacent frames overlap (feature point setting step). Specifically, the control unit 10 extracts the image of the marker M2 provided on the subject P1 by image recognition and sets it as a common feature point. The control unit 10 stores the extracted feature point's position information and identification information in the storage unit 20 (data storage unit 22). This information is used together with the reference image data in the processing of step S3, which will be described later.

[0036] In step S3, the control unit 10 (transformation matrix calculation unit 12) performs a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and calculates the transformation matrix used for that coordinate transformation, for all the reference images of adjacent frames (transformation matrix calculation step). In step S4, the control unit 10 (transformation matrix calculation unit 12) stores the transformation matrix calculated between each frame in the storage unit 20 (data storage unit 22).

[0037] Next, we will explain the process of creating a 3D model image from an image of subject P2 using a transformation matrix calculated from the image of subject P1. Figure 8 is a flowchart showing the procedure for creating a 3D model image from an image of subject P2 (target image). In the image creation process, the subject P2, a person, is the subject of the image capture.

[0038] In step S11 shown in Figure 8, the control unit 10 (image creation unit 13) controls each 3D sensor installed around the subject P2 (person) to photograph the subject P2, and acquires the 3D sensor images of the subject P2 captured by each 3D sensor as a target image for one frame (this shooting step).

[0039] In step S12, the control unit 10 (image creation unit 13) performs a coordinate transformation on the target images of adjacent frames based on the transformation matrix between the same frames calculated by the transformation matrix calculation unit 12, and creates a 3D model image of the subject P2 (image creation step). In step S13, the control unit 10 (image creation unit 13) stores the created 3D model image of the subject P2 in the storage unit 20 (image storage unit 23). Furthermore, even if subject P2 is replaced with another person, a 3D model image can still be created by following the steps in the flowchart shown in Figure 8.

[0040] The image creation apparatus 2 of the first embodiment described above provides, for example, the following effects. The control unit 10 (feature point setting unit 11) of the image creation device 2 sets common feature points in the overlapping region of reference images of adjacent frames captured by each 3D sensor, based on the image of a marker (identification information of the marker with identification information) provided on the subject P1. Therefore, compared to the conventional method in which the operator displays the captured images of adjacent frames on a monitor screen and sets common feature points while comparing the overlapping regions of the images, the burden of setting common feature points can be greatly reduced. In addition, it is possible to create a 3D model image with higher accuracy compared to when the operator sets common feature points manually.

[0041] The image creation system 100 equipped with the image creation device 2 of the first embodiment is useful, for example, when capturing 3D model images of visitors arriving one after another at games or events held in commercial facilities. In particular, when the shooting equipment is temporarily set up at events, the position where the 3D camera is installed changes each time, so by using the image creation system 100 of this embodiment, the workload of the operator can be reduced.

[0042] In the first embodiment, even when a marker without identification information is provided on the subject P1, the image creation device 2 allows the operator to set common feature points based on the image of the marker provided on the subject P1 in the overlapping region of the reference images of adjacent frames captured by each 3D sensor. Specifically, the operator can easily set common feature points by displaying the captured images of adjacent frames on a monitor screen and associating the markers at corresponding positions in the overlapping region of the images. Therefore, compared to the conventional method where the operator displays the captured images of adjacent frames on a monitor screen, compares the overlapping regions of the images, searches for common feature points such as a person's eyes, nose, lips, and ears in the captured images, and associates these feature points, the burden of setting common feature points can be reduced.

[0043] In the first embodiment, a model body created to resemble a person is used as the subject P1 (see Figure 3). Therefore, compared to the case where a marker without identification information is placed on the person's head, the operator can easily identify the position of the marker without identification information. Furthermore, compared to the case where a marker with identification information is placed on the person's head, the reading of identification information by image recognition can be performed more quickly and accurately.

[0044] (Second Embodiment) The image creation apparatus 2 of the second embodiment differs from the first embodiment in that the feature point setting unit 11 calculates a transformation matrix from the captured image of the subject P1. Therefore, in the second embodiment, only a flowchart showing the process of calculating the transformation matrix from the captured image of the subject P1 is shown, and the illustration of the entire apparatus is omitted. In addition, in the description and drawings of the second embodiment, components equivalent to those in the first embodiment are denoted by the same reference numerals as in the first embodiment, and redundant explanations are omitted.

[0045] Figure 9 is a flowchart showing the procedure for calculating a transformation matrix from a captured image (reference image) of the subject P1 in the feature point setting unit 11 of the second embodiment. The feature point setting unit 11 of the second embodiment differs from the first embodiment in that it sets common feature points by image recognition in the region where the reference images of adjacent frames overlap. In the pre-shooting step of the second embodiment, the subject P1 to be photographed is a model body without markers M. The processing of the flowchart shown in Figure 9 proceeds by the control unit 10 reading and executing the image creation program, transformation matrix calculation program, image recognition program, etc., stored in the storage unit 20.

[0046] In step S21 shown in Figure 9, the control unit 10 (feature point setting unit 11) controls each 3D sensor installed around the subject P1 to photograph the subject P1, and acquires the image of the subject P1 captured by each 3D sensor as a reference image for one frame (pre-shooting step). The reference images acquired from each 3D sensor are stored in the storage unit 20 (image storage unit 23).

[0047] In step S22, the control unit 10 (feature point setting unit 11) performs image recognition on the reference images acquired from each 3D sensor and sets common feature points in the region where the reference images of adjacent frames overlap (feature point setting step). Specifically, the control unit 10 extracts feature points of the model body captured as reference images (for example, parts corresponding to the eyes, nose, lips, ears, etc. of a person in the model body) by image recognition and sets them as common feature points. The control unit 10 stores the extracted feature point location information and identification information in the storage unit 20 (data storage unit 22).

[0048] In step S23, the control unit 10 (transformation matrix calculation unit 12) performs a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and calculates the transformation matrix used for that coordinate transformation, for all the reference images of adjacent frames (transformation matrix calculation step).

[0049] In step S24, the control unit 10 (transformation matrix calculation unit 12) stores the transformation matrix calculated between each frame in the storage unit 20 (data storage unit 22). The subsequent process of creating a 3D model image from the captured image of subject P2 (person) using the transformation matrix calculated from the captured image of subject P1 is the same as in the first embodiment (Figure 8), so the explanation will be omitted.

[0050] In the image creation apparatus 2 of the second embodiment, the control unit 10 (feature point setting unit 11) sets common feature points by image recognition in the region where the reference images of adjacent frames captured by each 3D sensor overlap. This eliminates the need to pre-set markers or the like on the subject P1 to be photographed, thus reducing the burden of setting common feature points and also reducing the burden of preparation work for creating 3D model images.

[0051] In the second embodiment, subject P2 (person) may be used instead of subject P1 (model body). In that case, a 3D model image can be created based on the captured image of the person obtained in the pre-shooting step, thus simplifying the process of creating the 3D model image.

[0052] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible, as shown in the modified forms described later, and these are also included within the technical scope of the present invention. Furthermore, the effects described in the embodiments are merely a list of the most preferred effects resulting from the present invention and are not limited to those described in the embodiments. The embodiments described above and the modified forms described later can be used in combination as appropriate, but a detailed explanation is omitted.

[0053] (Transformed form) In the first embodiment, the marker with identification information is not limited to the one-dimensional barcode or two-dimensional barcode shown in Figures 4(B) and (C). For example, a plain marker may have numbers, letters, symbols, etc., attached to it, or pictures or patterns attached to it, and each may have a different shape. In other words, the identification information attached to the marker can be anything as long as it can be identified by image recognition.

[0054] In the first and second embodiments, an example was described in which six 3D sensors are installed around a subject. However, the number of 3D sensors to be installed can be appropriately selected depending on the specifications of the 3D sensors, the shape and size of the subject, the shooting range, etc.

[0055] In the first and second embodiments, instead of using a model body such as a mannequin's head as the subject P1, a cube that does not mimic a human body may be used. If, for example, a sphere is used as the subject P1, the coordinate transformation between reference images can be performed with greater precision. [Explanation of Symbols]

[0056] 1a~1f 3D Sensor 2. Image creation device 10 Control Unit 11 Feature Point Setting Section 12 Transformation Matrix Calculation Unit 13 Image Creation Section 20 Memory section 21 Program Storage Unit 22 Data Storage Unit 23 Image storage unit 100 Image Creation Systems

Claims

1. A pre-shooting step in which a plurality of markers with different identification information attached are provided on a reference object, the reference object is photographed by a plurality of three-dimensional sensors installed around the reference object, and the image from each of the three-dimensional sensors is acquired as a reference image for one frame, A feature point setting step involves reading the identification information recorded on the marker provided on the reference object in the region where the reference images of adjacent frames overlap by image recognition, setting common feature points based on the identification information, and storing the reference image and the positional information of the feature points in association with each other. A transformation matrix calculation step is performed for all adjacent frames, using the respective reference images and feature point position information of adjacent frames, to perform a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and to calculate the transformation matrix used for that coordinate transformation. This shooting step involves capturing the target object with multiple 3D sensors and acquiring an image from each 3D sensor as a single frame of the target image. Image creation step: Performs coordinate transformation on adjacent target images obtained in the above shooting step based on the transformation matrix between the same frames calculated in the transformation matrix calculation step to create a three-dimensional model image of the target object. Image creation method including [specific details].

2. The image creation method according to claim 1, wherein the reference object is a model body made to resemble the target object.

3. The image creation method according to claim 1 or claim 2, wherein the reference object is provided with markers such that three or more images of the markers are included in the region where the reference images of adjacent frames overlap.

4. A pre-shooting step comprising: capturing images of a reference object, which is a human model body made to resemble a target object, using a plurality of three-dimensional sensors placed around the reference object, and acquiring an image from each of the three-dimensional sensors as a reference image for one frame; A feature point setting step involves identifying facial features by image recognition in the region where reference images of adjacent frames overlap, setting identical identified facial features as common feature points, and storing the reference images and the positional information of the feature points in association with each other. A transformation matrix calculation step is performed for all adjacent frames, using the respective reference images and feature point position information of adjacent frames, to perform a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and to calculate the transformation matrix used for that coordinate transformation. This shooting step involves capturing the target object with multiple 3D sensors and acquiring an image from each 3D sensor as a single frame of the target image. Image creation step: Performs coordinate transformation on adjacent target images obtained in the above shooting step based on the transformation matrix between the same frames calculated in the transformation matrix calculation step to create a three-dimensional model image of the target object. Image creation method including [specific details].

5. An image creation program executed on a computer, The reference object is equipped with multiple markers, each with different identification information. The aforementioned computer, A feature point setting unit acquires images of the reference object captured by multiple 3D sensors from each of the 3D sensors as a reference image for one frame, reads the identification information recorded on the markers provided on the reference object by image recognition in the region where the reference images of adjacent frames overlap, sets common feature points based on the identification information, and stores the reference image and the position information of the feature points in association. A transformation matrix calculation unit performs a process for all adjacent frames' reference images, using the respective reference images and feature point position information of adjacent frames, to perform a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and to calculate the transformation matrix used for that coordinate transformation. An image creation program that functions as an image creation unit, which acquires an image of a target object captured by multiple three-dimensional sensors, one image from each of the three-dimensional sensors, as a target image for one frame, and performs a coordinate transformation on adjacent target images based on the transformation matrix between the same frames acquired by the transformation matrix calculation unit, thereby creating a three-dimensional model image of the target object.

6. An image creation program executed on a computer, The aforementioned computer, A feature point setting unit acquires images from multiple 3D sensors of a reference object, which is a human model created to resemble the target object, as a reference image for one frame. In the region where the reference images of adjacent frames overlap, it identifies facial features through image recognition, sets identical identified facial features as common feature points, and stores the reference image and the positional information of the feature points in association. A transformation matrix calculation unit performs a process for all adjacent frames' reference images, using the respective reference images and feature point position information of adjacent frames, to perform a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and to calculate the transformation matrix used for that coordinate transformation. An image creation program that functions as an image creation unit, which acquires an image of a target object captured by multiple three-dimensional sensors, one image from each of the three-dimensional sensors, as a target image for one frame, and performs a coordinate transformation on adjacent target images based on the transformation matrix between the same frames acquired by the transformation matrix calculation unit, thereby creating a three-dimensional model image of the target object.

7. A feature point setting unit that provides a reference object with multiple markers to which different identification information is attached, acquires an image of the reference object from each of the multiple 3D sensors positioned around the reference object as a reference image for one frame, reads the identification information recorded on the markers provided on the reference object by image recognition in the region where the reference images of adjacent frames overlap, sets common feature points based on the identification information, and stores the reference image and the position information of the feature points in association with each other. A transformation matrix calculation unit performs a process for all adjacent frames' reference images, using the respective reference images and feature point position information of adjacent frames, to perform a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and to calculate the transformation matrix used for that coordinate transformation. An image creation unit acquires an image of a target object captured by multiple three-dimensional sensors, one image from each of the three-dimensional sensors, as a target image for one frame, and performs a coordinate transformation on adjacent target images based on the transformation matrix between the same frames acquired by the transformation matrix calculation unit to create a three-dimensional model image of the target object. An image creation device equipped with the following features.

8. A feature point setting unit that acquires an image of a reference object, which is a human model made to resemble a target object, taken by a plurality of three-dimensional sensors placed around the reference object, and uses each of the three-dimensional sensors to obtain a reference image for one frame, identifies facial features by image recognition in the region where the reference images of adjacent frames overlap, sets identical identified facial features as common feature points, and stores the reference image and the position information of the feature points in association, A transformation matrix calculation unit performs a process for all adjacent frames' reference images, using the respective reference images and feature point position information of adjacent frames, to perform a coordinate transformation on the reference images of adjacent frames so that their feature points coincide, and to calculate the transformation matrix used for that coordinate transformation. An image creation unit acquires an image of a target object captured by multiple three-dimensional sensors, one image from each of the three-dimensional sensors, as a target image for one frame, and performs a coordinate transformation on adjacent target images based on the transformation matrix between the same frames acquired by the transformation matrix calculation unit to create a three-dimensional model image of the target object. An image creation device equipped with the following features.

9. An image creation apparatus according to claim 7 or claim 8, Multiple 3D sensors capture images of the reference object from all sides and output one frame of the reference image to the feature point setting unit, and capture images of the target object from all sides and output one frame of the target image to the feature point setting unit, An image creation system equipped with the following features.

Images