3D head model generation system, program, and 3D head model generation method

Abstract

To provide a system, program, and method for generating a three-dimensional head model, which can generate an accurate three-dimensional model even when generating a three-dimensional model of the head of a subject who is difficult to keep still. [Solution] The head 3D model generation system 1 comprises an electronic terminal 2 having an imaging unit 21 that photographs a head 41 and a processing unit 20 that processes the image of the head 41, and a server 3. The processing unit 20 of the electronic terminal 2 has a terminal-side image acquisition unit 201 that acquires multiple images of the head 41 photographed from different directions by the imaging unit 21, and a filtering processing unit 205 that determines whether the images acquired by the terminal-side image acquisition unit 201 are suitable images that satisfy predetermined conditions. The server 3 has a 3D model generation unit 303 that generates a 3D model showing the three-dimensional external shape of the head 41 based on the multiple suitable images.

Description

[Technical field]

[0001] The present invention relates to a three-dimensional head model generating system, a program, and a three-dimensional head model generating method. [Background technology]

[0002] Conventionally, a cranial shape correction helmet that corrects the shape of the head of a subject such as an infant who has a cranial deformity that requires treatment, such as plagiocephaly, has been known. For example, Patent Document 1 describes this type of cranial shape correction helmet.

[0003] Patent Document 1 describes a skull correction helmet for infants that is designed based on the external shape of an infant's head and includes an exterior member for pressing against the head to correct the shape of the skull, and an interior member that is attached to the inner surface of the exterior member and cushions the irritation to the head caused by the exterior member. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-183330 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, the cranial shape correction helmet is manufactured based on a three-dimensional model that shows the three-dimensional external shape of the head of the subject, which is generated from a plurality of images of the subject's head taken from different directions. In this case, it is important to read accurate information about the external shape of the head from the image in order to generate an accurate three-dimensional model. However, when the subject is an infant or the like, the infant may move when the image is taken, making it difficult to read accurate information about the external shape of the head from the taken image.

[0006] The present invention aims to provide a 3D head model generation system, program, and method that can generate an accurate 3D model even when generating a 3D model of the head of a subject who is difficult to keep still. [Means for solving the problem]

[0007] The head 3D model generation system comprises an electronic terminal having an imaging unit that images the head of a subject and a processing unit that processes the images captured by the imaging unit, and a server capable of communicating with the electronic terminal, wherein the processing unit of the electronic terminal has an image acquisition unit that acquires multiple images of the head captured from different directions by the imaging unit, and a filtering processing unit that determines whether the images acquired by the image acquisition unit are suitable images that satisfy predetermined conditions, and the server has a 3D model generation unit that generates a 3D model showing the three-dimensional external shape of the head based on the multiple suitable images determined by the filtering processing unit to satisfy the predetermined conditions, wherein the predetermined conditions include conditions related to the degree of blurring of the image, and the filtering processing unit performs a Fourier transform on the image acquired by the image acquisition unit to acquire transformed data, and identifies the degree of blurring of the image based on the average value of spatial frequencies in the converted transformed data excluding low spatial frequency components. Effect of the Invention

[0008] According to the present invention, even when generating a 3D model of a subject's head that is difficult to keep still, an accurate 3D model can be generated. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing a system for generating a three-dimensional head model according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the hardware configuration of a server in the head 3D model generation system according to one embodiment of the present invention. [Diagram 3] 1 is a block diagram showing a hardware configuration of an electronic terminal in a three-dimensional head model generation system according to an embodiment of the present invention. [Figure 4] FIG. 4 is a schematic diagram showing the display unit of the electronic terminal when photographing a head. [Diagram 5] 5 is a schematic diagram showing an example of an image captured by a capturing unit of the electronic terminal in the state shown in FIG. 4. [Figure 6] 2 is a block diagram showing a functional block configuration of an electronic terminal in the head 3D model generation system according to one embodiment of the present invention. FIG. [Figure 7A] FIG. 2 is a diagram showing an image in which the subject is stationary and in focus. [Figure 7B] FIG. 7B is a diagram showing transformed data obtained by subjecting the image shown in FIG. 7A to Fourier transform. [Figure 8A] FIG. 2 shows an image in which the subject is stationary but out of focus. [Figure 8B] FIG. 8B is a diagram showing transformed data obtained by subjecting the image shown in FIG. 8A to Fourier transform. [Figure 9A] FIG. 13 is a diagram showing an image captured while a subject is moving. [Figure 9B] FIG. 9B is a diagram showing transformed data obtained by subjecting the image shown in FIG. 9A to Fourier transform. [Figure 10] 10 is an explanatory diagram of conditions regarding the distance and angle between the head and the imaging unit. FIG. [Figure 11] FIG. 11 is an explanatory diagram of conditions regarding the distance between the head and the imaging unit. [Figure 12] 2 is a block diagram showing a configuration of functional blocks of a server in the head 3D model generation system according to an embodiment of the present invention. FIG. [Figure 13] 11 is a flowchart showing an example of a three-dimensional model generation process executed by an electronic terminal according to an embodiment of the present invention. [Figure 14] 13 is a flowchart showing an example of a three-dimensional model generation process executed by a server according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a schematic diagram showing a head 3D model generating system 1 and a baby 4 according to the present embodiment.

[0011] The head 3D model generation system 1 according to this embodiment is a system for generating a 3D model showing the three-dimensional external shape of the head 41 of a subject such as an infant or small child 4. In this embodiment, the generation of a 3D model used in the case of creating a skull shape correction helmet will be described as an example.

[0012] The cranial shape correction helmet will be described. The cranial shape correction helmet is a helmet that is put on the head 41 of an infant 4 as a subject, and promotes deformation so that the shape of the skull is corrected as the skull grows. The subject infant 4 includes, for example, infants and young children who have cranial deformations requiring treatment, such as plagiocephaly, brachycephaly, and dolichocephaly. Plagiocephaly is a deformed shape in which the skull is not symmetrical but is tilted to one side. Brachycephaly is a deformed shape in which the front-to-back dimension of the skull is significantly short. Dolichocephaly is a deformed shape in which the front-to-back dimension of the skull is significantly long. The subject is not particularly limited, and may be other than an infant, a child, or an adult.

[0013] The cranial shape correction helmet is designed based on a three-dimensional model of the ideal head 41 (hereinafter referred to as the ideal three-dimensional model) that is the treatment target for each subject. The ideal three-dimensional model of the head 41 is created based on a three-dimensional model of the actual head 41 (hereinafter simply referred to as the three-dimensional model) that is generated using data on the external shape of the head 41 of each infant 4, such as an image, obtained from the acquired data. For this reason, when creating the cranial shape correction helmet, it is important to obtain accurate data on the external shape of the head 41 of the infant 4 and generate an accurate three-dimensional model.

[0014] The following describes the three-dimensional head model generation system 1. The three-dimensional head model generation system 1 includes an electronic terminal 2 and a server 3 capable of communicating with the electronic terminal 2, as shown in FIG.

[0015] The electronic terminal 2 is a device that acquires multiple images of the head 41 taken from different directions and performs pre-processing to generate a three-dimensional model. The electronic terminal 2 is not particularly limited as long as it has a function of taking an image and a communication function. Examples of the electronic terminal 2 include a smartphone and a tablet.

[0016] The server 3 is a device that generates a three-dimensional model based on a plurality of images received from the electronic terminal 2.

[0017] An example of the hardware configuration of the server 3 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the hardware configuration of the server 3 in the head 3D model generating system 1.

[0018] 2, the server 3 includes a computer 33, a storage unit 31, and a communication unit 32. A bus 334 and the like connect these units together.

[0019] The computer 33 includes a processor 331, and a read-only memory (ROM) 332 and a random-access memory (RAM) 333 as main storage devices. The processor 331 is a central processing unit (CPU), a micro processing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). Alternatively, the processor 331 is a combination of a plurality of these. The processor 331 may also be a combination of these with a hardware accelerator or the like. The processor 331 controls each unit to realize various functions of the server 3 based on programs such as firmware, system software, and application software stored in the ROM 332, the RAM 333, or an auxiliary storage device that is a part of the storage unit 31. Note that a part or all of the programs may be incorporated in the circuit of the processor 331.

[0020] The storage unit 31 is a storage area for various programs for causing the hardware group to function as the server 3, various data, etc., and can be configured with a ROM, a RAM, a flash memory, a solid-state drive (SSD), a hard disk drive (HDD), etc. Specifically, the storage unit 31 stores programs for causing the computer 33 to execute each function of this embodiment, images acquired from the electronic terminal 2, a generated 3D model of the head 41, etc.

[0021] The communication unit 32 executes a process for the server 3 to communicate with the electronic terminal 2 and other devices via the network. The communication unit 32 in this embodiment is configured to be capable of wireless communication between the electronic terminal 2 and the server 3.

[0022] An example of the hardware configuration of the electronic terminal 2 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the hardware configuration of the electronic terminal 2 in the head 3D model generation system 1.

[0023] 3, the electronic terminal 2 includes a computer 26, an image capturing unit 21, a storage unit 22, a communication unit 23, an input unit 24, and a display unit 25. A bus 264 and the like connect these units together.

[0024] The computer 26 includes a processor 261, and a read-only memory (ROM) 262 and a random-access memory (RAM) 263 as main storage devices. The processor 261 is a central processing unit (CPU), a micro processing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). Alternatively, the processor 261 is a combination of a plurality of these. The processor 261 may also be a combination of these with a hardware accelerator or the like. The processor 261 controls each unit to realize various functions of the electronic terminal 2 based on programs such as firmware, system software, and application software stored in the ROM 262, the RAM 263, or an auxiliary storage device that is a part of the storage unit 22. Note that a part or all of the programs may be incorporated in the circuitry of the processor.

[0025] The photographing unit 21 is a device that photographs a subject, and mainly includes a lens 211, an adjustment mechanism 212, an image sensor 213, and an A / D conversion unit 214.

[0026] The lens 211 is composed of a lens that collects light in order to capture an image of a subject. Examples of the lens 211 include a lens that forms an image of a subject on the light receiving surface of the image sensor 213 and a lens that can freely change the focal length within a certain range.

[0027] The adjustment mechanism 212 is a mechanism for adjusting the exposure time, aperture, ISO sensitivity, focal length, and the like.

[0028] The image sensor 213 forms an image of a subject on a light receiving surface where pixels are arranged two-dimensionally in a matrix, and converts the image into an electrical signal. The image sensor 213 may be, for example, a CMOS image sensor or a CCD image sensor.

[0029] The A / D conversion unit 214 converts the electrical signal read out from the image sensor 213 into a digital signal. An image is formed by the digital signal converted by the A / D conversion unit 214. The image captured by the imaging unit 21 is transmitted to the computer 26, the storage unit 22, the display unit 25, etc.

[0030] The storage unit 22 is a storage area for various programs and various data for causing the hardware group to function as the electronic terminal 2, and can be configured with a ROM, a RAM, a flash memory, a solid-state drive (SSD), a hard disk drive (HDD), etc. Specifically, the storage unit 22 stores programs for causing the computer 26 to execute each function of this embodiment, the size of the image sensor 213 of the photographing unit 21, the size and shape of an identification marker 52 described later, quality standards for images described later, etc.

[0031] The communication unit 23 executes a process for the electronic terminal 2 to communicate with the server 3 and other devices. The communication unit 23 of this embodiment is configured to be capable of wireless communication between the electronic terminal 2 and the server 3.

[0032] The input unit 24 is a user interface that is electrically connected to the computer 26. The input unit 24 is composed of buttons and a display 27. The display 27 is composed of, for example, a liquid crystal display (LCD) or an organic EL (Electro-Luminescent) display, and a touch panel that detects a position touched by a user is provided on the image display surface of the display 27. The user can input information by touching the image display surface of the display 27.

[0033] The display unit 25 is a user interface electrically connected to the computer 26. The display unit 25 is configured with a display 27. The display unit 25 displays images and various information transmitted from the photographing unit 21, the storage unit 22, the computer 26, etc.

[0034] The imaging unit 21 of the present embodiment may have a function of capturing an image of a subject such as the head 41 as a still image, a function of capturing an image as a moving image, or a function of capturing both a still image and a moving image. Note that in this specification, an image means a still image.

[0035] Here, a manner in which a user captures an image of the head 41 of an infant 4 using the electronic terminal 2 will be described with reference to Figs. 4 and 5. Fig. 4 is a schematic diagram showing an example of a display screen of the display unit 25 when the head 41 of the infant 4 is captured by the electronic terminal 2. Fig. 5 is a schematic diagram showing an example of an image captured by the imaging unit 21 in the state shown in Fig. 4. Note that users of the electronic terminal 2 include, for example, doctors and guardians of the infant 4.

[0036] When photographing the head 41, the shape including the hair etc. is captured as the external shape of the head 41, so a cap 5 is placed on the head 41 to hold the hair down against the scalp and reduce the effect of the hair on the image of the head 41.

[0037] The cap 5 will now be described. The cap 5 is made of a sheet 50 that has elasticity and can cover the head 41. In this embodiment, an image of the head 41 covered with the sheet 50 is taken by the imaging unit 21.

[0038] The material of the sheet 50 may be, for example, recycled fiber, semi-synthetic fiber, synthetic fiber such as nylon, polyurethane, polyester, or a mixture of two or more of these fibers. The sheet 50 of this embodiment is a cloth-like member made of any of these materials.

[0039] A surface 51 of the portion of the sheet 50 covering the head 41 is formed with a plurality of identification markers 52 and a plurality of color patterns 53 that can be identified by the photographing unit 21.

[0040] The color pattern 53 is a combination of colors and patterns, and is formed over the entire surface 51 of the sheet 50. The multiple color patterns 53 formed on the surface 51 of the sheet 50 may have the same combination of colors and patterns, or may not have the same combination of colors and patterns. In terms of generating an accurate three-dimensional model, it is preferable that all the color patterns 53 formed on the surface 51 of the sheet 50 have different combinations of colors and patterns. In this embodiment, all the color patterns 53 formed on the surface 51 of the sheet 50 are different from each other. This makes it easier to accurately distinguish each color pattern.

[0041] The colors of the color pattern 53 and background formed on the cap 5 of this embodiment are mainly white, flesh color, pink, red, light blue, orange, yellow-green, green, etc. In Fig. 4, white is shown by white, flesh color by small and sparse dots, pink by small and dense dots, red by large and dense dots, light blue by solid hatching extending diagonally to the upper left, orange by dashed hatching extending diagonally to the upper right, green by solid hatching extending diagonally to the upper right, and yellow-green by diagonal lattice hatching.

[0042] The multiple identification markers 52 are configured to be distinguishable from one another, and are formed at intervals across the entire surface 51 of the portion of the sheet 50 that covers the head 41. Examples of the identification markers 52 include a QR code (registered trademark) and an ArUco marker. The identification marker 52 of this embodiment is an ArUco marker that has corners and has a unique pattern formed on its surface that can be read by the electronic terminal 2. The identification marker 52 of this embodiment is rectangular in plan view. That is, the identification marker 52 of this embodiment has four corners. The multiple identification markers 52 each have a different pattern formed on their surface in black and white designs.

[0043] The memory unit 22 stores information regarding the positional relationship between each of the identification markers 52 and the position where each of the identification markers 52 is formed on the sheet 50. Examples of the positional relationship between each of the identification markers 52 include information regarding each of the identification markers 52 and adjacent identification markers 52. For example, information such as that the identification marker 52 with identification number 1 is adjacent to and surrounded by the identification markers 52 with identification numbers 2 to 5. When the baby 4 is wearing the cap 5 and the head 41 is covered by the sheet 50, each identification marker 52 indicates the position information of the head 41.

[0044] In this embodiment, 50 identification markers 52 are formed on the surface 51 of the sheet 50. The number of identification markers 52 formed on the surface 51 of the sheet 50 is not particularly limited, and may be less than 50 or may be 50 or more.

[0045] Next, various types of information that are displayed on the display unit 25 when the user images the head 41 of the baby 4 using the electronic terminal 2 will be described.

[0046] In this embodiment, as shown in Fig. 5, a three-dimensional model of the head 41, which is the subject, is generated using a plurality of images in which the entire head 41 is located within a predetermined range (hereinafter, first detection area) A from the center C of the image and each identification marker 52 is located within a predetermined range (hereinafter, second detection area) B from the center of the image. Note that in Fig. 5, the first detection area A is the area inside the dashed line, and the second detection area B is the area inside the broken line.

[0047] 4, the display unit 25 displays a live view image, which is a real-time image captured by the image sensor 213 of the imaging unit 21, when the imaging unit 21 captures an image. The live view image displayed on the display 27 of the display unit 25 displays only an image within the first detection area A in the captured image. This allows the user to easily capture an image in which the entire head 41 of the subject, the baby 4, fits within the first detection area A.

[0048] The display unit 25 also displays on the display 27 a storage display 270 indicating the storage status of the identification marker image in the storage unit 22, which will be described later. The display unit 25 displays, as the storage display 270, a front storage display 271 indicating the storage status of the identification marker image in the front region of the head 41, a rear storage display 272 indicating the storage status of the identification marker image in the rear region of the head 41, a right storage display 273 indicating the storage status of the identification marker image in the right region of the head 41, a left storage display 274 indicating the storage status of the identification marker image in the left region of the head 41, and an entire storage display 275 indicating the storage status of the identification marker image in the entire region of the head 41. In the entire storage display 275, a 3D model of the head 41 is displayed, and the location of the head 41 corresponding to the identification marker that has been photographed and stored is indicated in this 3D model. These storage displays 270 allow the user to easily grasp the images that need to be photographed in order to generate a 3D model of the head 41.

[0049] Next, a description will be given of the functional configuration of the electronic terminal 2 for performing the 3D model generation process by the electronic terminal 2 of the head 3D model generation system 1. FIG.

[0050] The electronic terminal 2 is mainly configured by a processor 261, and includes a processing unit 20 that executes a 3D model generation process. As shown in FIG. 6, the processing unit 20 includes a terminal-side image acquisition unit 201, a filtering processing unit 205, a display control unit 206, and a transmission processing unit 207.

[0051] The terminal side image acquisition unit 201 includes an acquisition unit 202 , an image position determination unit 203 , and an identification marker detection unit 204 .

[0052] The acquisition unit 202 executes a process of acquiring an image of the head 41 photographed by the imaging unit 21. The acquisition unit 202 acquires, together with the image of the head 41, various parameters of the imaging unit 21, such as the exposure time, aperture, ISO sensitivity, focal length, etc., set by the user or the adjustment mechanism 212 when photographing the head 41. The acquisition unit 202 may extract a still image from a moving image of the head 41 photographed by the imaging unit 21, and acquire the extracted still image as the image.

[0053] The image position determination unit 203 executes a process of determining whether or not the entire head 41 of the infant 4 on the image acquired by the acquisition unit 202 is located within the first detection area A. The image position determination unit 203 may determine whether or not the entire head 41 on the image is located within the first detection area A using a learning model constructed by performing supervised learning, for example. The supervised learning may be, for example, a case in which an image of the head 41 of the infant 4 wearing the cap 5 is taken as input data, an evaluation result of whether or not the entire head 41 is located within the first detection area A in the image is used as a label, and many pairs of the input data and the label are used as teacher data. When the image position determination unit 203 determines that the entire head 41 on the image is not located within the first detection area A, it discards the image.

[0054] The identification marker detection unit 204 executes a process of determining whether or not at least one identification marker 52 on an image in which it has been determined that the entire head 41 is located within the first detection area A is located within the second detection area B. For example, the identification marker detection unit 204 detects the identification marker 52 and its identification information based on information on the identification marker 52 read by the imaging unit 21. Then, the identification marker detection unit 204 determines whether or not the detected identification marker 52 is located within the second detection area B on the image based on a comparison result between the detected identification marker 52 and information on the second detection area B stored in the storage unit 22. When the identification marker detection unit 204 determines that the detected identification marker 52 is located within the second detection area B on the image, it acquires the image as an image corresponding to the detected identification marker 52 (hereinafter referred to as an identification marker image) and transfers it to the filtering processing unit 205. When multiple identification markers 52 exist within the second detection area B of the image, the identification marker detection unit 204 identifies the identification marker 52 closest to the center C, acquires the image as an identification marker image of the identified identification marker 52, and transfers it to the filtering processing unit 205. On the other hand, when the identification marker detection unit 204 determines that all of the detected identification markers 52 are not located within the second detection area B on the image, it discards the image.

[0055] The filtering processing unit 205 executes a filtering process to determine whether or not the identification marker image acquired by the terminal-side image acquisition unit 201 is a matching image that satisfies a predetermined condition. In this embodiment, the filtering processing unit 205 executes a filtering process when an identification marker image of the same identification marker 52 as the identification marker image transferred from the terminal-side image acquisition unit 201 is not stored in the storage unit 22. On the other hand, when an identification marker image of the same identification marker 52 as the identification marker image transferred from the terminal-side image acquisition unit 201 is stored in the storage unit 22, the filtering processing unit 205 discards the transferred image.

[0056] The filtering process will be described. The predetermined conditions in the filtering process are image quality standards from the viewpoint of generating an accurate three-dimensional model. In this embodiment, the image quality standards include three conditions: a condition on the degree of blurring of the image, a condition on the distance between the imaging unit 21 and the head 41 when the imaging unit 21 captures the head 41, and a condition on the orientation of the imaging unit 21 with respect to the head 41 when the imaging unit 21 captures the head 41. The filtering processing unit 205 of this embodiment determines that the identification marker image is a suitable image when all of the above three image quality standards are satisfied. In the filtering process, the process of determining the quality standard on the degree of blurring of the image is called blur determination process, the process of determining the quality standard on the angle of the imaging unit 21 with respect to the head 41 is called angle determination process, and the process of determining the quality standard on the distance between the imaging unit 21 and the head 41 is called distance determination process.

[0057] First, the blur determination process will be described with reference to Figs. 7A to 9B. If the baby 4 moves the head 41 at the moment the image of the head 41 is captured, the focus of the subject, head 41, often shifts, resulting in an out-of-focus image being captured. It becomes difficult to obtain accurate data on the outer shape of the head 41 from such an out-of-focus image. The blur determination process is executed to select only images in which the degree of blur is within an allowable range for generating a three-dimensional model.

[0058] In the blur determination process, the degree of blur in an image is determined based on the average value of the spatial frequency of the transformed data obtained by Fourier transforming the image. Specifically, in the blur determination process, the following processes (a) to (d) are executed in this order. (a) Transformation data is obtained by Fourier transforming the identification marker image acquired by the terminal-side image acquisition unit 201. The transformation data is obtained, for example, by performing a two-dimensional Fourier transform on the identification marker image, which is a Fourier transform in the horizontal direction and then a Fourier transform in the vertical direction. (b) The converted data acquired in (a) is subjected to a process of removing low spatial frequency components, i.e., spatial frequency components lower than a predetermined spatial frequency. For example, the process of (b) may be a process of rearranging the converted data acquired in (a) so that spatial frequency components with lower spatial frequencies are located closer to the center of the image, and removing spatial frequency components in a region (hereinafter referred to as a masking region) E within a predetermined range from the center of the image of the rearranged converted data. (c) Calculate the average spatial frequency of the transformed data from which the low spatial frequency components have been removed in (b). (d) It is determined whether or not the average value of the spatial frequency calculated in (c) is equal to or greater than a predetermined threshold value (hereinafter, blur determination value).

[0059] If the filtering processing unit 205 determines in (d) above that the average value of the spatial frequency is equal to or greater than the blur judgment value, it proceeds to angle judgment processing, and if it determines that the average value of the spatial frequency is less than the blur judgment value, it stops the filtering processing and discards the identification marker image that was the subject of judgment.

[0060] The relationship between spatial frequency and image blur will be described with reference to Figs. 7A to 9B. Fig. 7A is a diagram showing an image in focus when the subject is stationary, Fig. 8A is a diagram showing an image in focus when the subject is stationary, and Fig. 9A is a diagram showing an image taken when the subject is moving. Fig. 7B is a diagram showing transformed data obtained by Fourier transforming the image shown in Fig. 7A, Fig. 8B is a diagram showing transformed data obtained by Fourier transforming the image shown in Fig. 8A, and Fig. 9B is a diagram showing transformed data obtained by Fourier transforming the image shown in Fig. 9A. Specifically, Figs. 7B, 8B, and 9B are images of transformed data obtained by performing the two-dimensional Fourier transform described above in (a) on the images shown in Figs. 7A, 8A, and 9A, and rearranging the transformed data obtained so that the spatial frequency components with lower spatial frequencies are located closer to the center of the image. Note that in Figs. 7B, 8B, and 9B, the darker the white, the lower the spatial frequency, and the darker the black, the higher the spatial frequency. The inside of the dashed circle shown in FIG. 7B, FIG. 8B and FIG. 9B indicates a masking region E that is removed from the converted data in (b).

[0061] Comparing FIG. 7B, FIG. 8B, and FIG. 9B, in the masking region E in which the spatial frequency in the Fourier transformed transformation data is low, there is a tendency for the difference between the three transformation data to be small. On the other hand, it can be confirmed that in the transformation data, in the region other than the masking region E, FIG. 7B, which shows the transformation data of the image in focus in a stationary state, has many regions with high spatial frequency compared with FIG. 8B and FIG. 9B. Therefore, in the blur determination process, in order to more clearly grasp the difference in spatial frequency between an image in focus and an image that is out of focus due to the movement of the head 41 during shooting, the data of the masking region E with low spatial frequency is removed, and then the spatial frequency of the transformation data is compared. This makes it possible to more accurately determine the degree of blur in the image.

[0062] Next, the angle determination process and the distance determination process will be described with reference to Fig. 10 and Fig. 11. Fig. 10 is an explanatory diagram of conditions related to the distance and angle between the head 41 and the imaging unit 21. Fig. 11 is an explanatory diagram of conditions related to the distance between the head 41 and the imaging unit 21.

[0063] In the electronic terminal 2 shown by the dashed line in Fig. 10, the angle of the lens 211 with respect to the identification marker 52 is significantly shifted compared to the electronic terminal 2 shown by the solid line in which the lens 211 of the photographing unit 21 faces the surface on which the pattern of the identification marker 52 is formed. With an image of the head 41 photographed in such a state, it is difficult to obtain accurate data on the external shape of the head 41. Also, if the distance L between the head 41 and the lens 211 of the photographing unit 21 is too close or too far, it is difficult to obtain accurate data on the external shape of the head 41 from the image. In order to select only images in which the angle and distance of the photographing unit 21 with respect to the head 41 are within the ranges allowable for generating a three-dimensional model, an angle determination process and a distance determination process are executed.

[0064] In the angle determination process, the following processes (e) to (h) are executed in this order. (e) In order to grasp the orientation of the identification marker 52, position information u indicating the position of the identification marker 52 on the image is identified from the position information of the corners of the identification marker 52. Using the position information of the identification marker 52 identified in (f) and (e), parameter information K including the focal length set in the imaging unit 21 and the size of the image sensor 213, and a preset three-dimensional space coordinate X, a perspective transformation matrix Rt shown in the following equation (2) is calculated according to the following equation (1). u = K × Rt × X Equation (1)

[0065]

number

[0066] From the perspective transformation matrix Rt calculated in (g) and (f), the orientation of the identification marker 52 is estimated as the orientation of the imaging unit 21 relative to the head 41. Specifically, a rotation matrix R, which is a portion related to the rotation of the object and is shown in the following equation (3), is extracted from the perspective transformation matrix Rt, and the rotation matrix R is converted into Euler angles.

[0067]

number

[0068] If the filtering processing unit 205 determines that all of the Euler angles ψ, θ, and φ obtained in (h) above are within the range of the angle judgment value, it proceeds to distance judgment processing, and if it determines that at least any of the Euler angles ψ, θ, and φ are outside the range of the angle judgment value, it stops the filtering processing and discards the identification marker image that was the subject of judgment.

[0069] In the distance determination process, the above processes (e) and (f) and the following processes (i) to (k) are executed in this order. (i) The components of the translation vector t of the perspective transformation matrix Rt calculated in (f), i.e., the norm of the components shown in the following equation (4), are calculated using the following equation (5). This calculation result indicates the actual distance L between the lens 211 of the imaging unit 21 and the identification marker 52.

[0070]

number

number

[0071] (j) Determine whether the distance L calculated in (i) is within a range of a predetermined threshold (hereinafter, distance determination value). Note that FIG. 11 shows an example in which the distance L2 is the lower limit value of the distance determination value and the distance L1 is the upper limit value.

[0072] When the filtering processing unit 205 determines that the distance L calculated in (k) above is within the range of the distance judgment value, it determines that the identified marker image that was the subject of the judgment is a suitable image that satisfies the quality standard, and stores the identified marker image as a suitable image in the storage unit 22. On the other hand, when the filtering processing unit 205 determines that the distance L is outside the range of the distance judgment value, it discards the identified marker image that was the subject of the judgment.

[0073] The display control unit 206 executes a process of controlling the display on the display unit 25 according to the storage status of the matching image in the storage unit 22. For example, when an identification marker image of a new identification marker 52 is stored in the storage unit 22 as a matching image, the display control unit 206 changes the storage display 270 to one that reflects the storage status of the matching image. Furthermore, when identification marker images of all predetermined identification markers 52 are stored in the storage unit 22 as matching images, the display control unit 206 may display an indication on the display unit 25 notifying the user of this fact.

[0074] The transmission processing unit 207 executes a process of transmitting a plurality of matching images of the head 41 stored in the storage unit 22 to the server 3. The transmission processing unit 207 transmits the identification marker images of all the predetermined identification markers 52 stored in the storage unit 22 as matching images to the server 3. For example, when the identification marker images of all the identification markers 52 are stored in the storage unit 22, the transmission processing unit 207 may automatically transmit the identification marker images of all the identification markers 52 to the server 3. For example, when a user performs an operation to transmit the matching image in the storage unit 22 to the server 3 via the input unit 24, the transmission processing unit 207 may transmit images corresponding to all the identification markers 52 to the server 3 based on the user operation.

[0075] Note that all of the predetermined identification markers 52 may be, for example, all of the identification markers 52 formed over the entire surface 51 of the portion of the seat 50 that covers the head 41 .

[0076] Since a series of processes such as image shooting, acquisition, filtering, and storage are performed on a single electronic terminal 2, for example, the time lag between shooting an image and obtaining the result of whether or not it is a suitable image is reduced, and a large number of images required for creating a 3D model can be smoothly acquired.

[0077] Next, a description will be given of the functional configuration of the server 3 for performing the 3D model generation process by the server 3 of the head 3D model generation system 1. FIG.

[0078] The server 3 is mainly composed of a processor 331, and includes a processing unit 30 that executes a 3D model generation process. As shown in FIG. 12, the processing unit 30 includes a server-side image acquisition unit 301, a background processing unit (background removal unit) 302, a 3D model generation unit 303, and a transmission processing unit 304.

[0079] The server-side image acquisition unit 301 executes a process of acquiring, via the communication unit 32, images matching all of the predetermined identification markers 52 from the electronic terminal 2.

[0080] The background processing unit 302 executes background removal processing for each of the multiple matching images acquired by the server-side image acquisition unit 301. The background removal processing is processing for removing information other than the detection target image including the head 41 of the infant 4 on the matching image. The detection target image may be, for example, an image including only the head 41 of the infant 4, an image including the part above the face, an image including the part above the neck of the infant 4, or an image including the entire infant 4. For example, the background processing unit 302 may execute background removal processing by constructing a learning model for detecting the detection target image on the image by machine learning or the like, detecting the detection target image on the matching image using the constructed learning model, and removing information other than the detected detection target image. For example, the background removal processing may be performed using a program such as "Rembg" or "Detectron 2", which are Python libraries. As a result, information other than the detection target image required for generating a three-dimensional model is removed from the image, so that the feature amount used during generation can be narrowed down, and overlapping feature points of the head 41 between each image can be detected more accurately.

[0081] The three-dimensional model generating unit 303 executes a process of generating a three-dimensional model of the head 41 based on a plurality of matching images. In this embodiment, the three-dimensional model generating unit 303 generates a three-dimensional model of the head 41 by using a photogrammetry technique based on a plurality of matching images from which information other than the detection target image has been removed by the background processing unit 302. Specifically, the three-dimensional model generating unit 303 detects overlapping feature points between images corresponding to each identification marker 52, and generates a three-dimensional model by three-dimensionally superimposing the images. At this time, the three-dimensional model generating unit 303 generates a three-dimensional model of the head 41 based on position information on the head 41 indicated by the identification marker 52 and the color pattern 53 of the surface 51 of the sheet 50. By using the position information indicated by the identification marker 52, it is easy to grasp which part of the head 41 each image is an image of. In addition, since there are no parts with the same color pattern 53, the difference in feature amount becomes clearer depending on the part of the head 41, and it is possible to more accurately detect overlapping feature points between each image. Therefore, a more accurate three-dimensional model of the head 41 can be generated.

[0082] The transmission processing unit 304 executes a process of transmitting the 3D model generated by the 3D model generation unit 303 to the electronic terminal 2 or other devices. A user of the device such as the electronic terminal 2 that has received the 3D model can confirm the generated 3D model of the head 41 by displaying it on the display 27 or the like.

[0083] Next, an example of the flow of the 3D model generation process executed by the processing unit 20 of the electronic terminal 2 of the head 3D model generation system 1 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the 3D model generation process executed by the electronic terminal 2.

[0084] 13, in step S11, the acquisition unit 202 of the processing unit 20 acquires an image captured by the imaging unit 21. At this time, the acquisition unit 202 acquires parameter information including the focal length and the size of the image sensor set in the imaging unit 21 when the image was captured together with the image.

[0085] In step S12, the image position determination unit 203 determines whether or not the entire head 41 on the image acquired in step S11 is located within the first detection area A. If the image position determination unit 203 determines that the entire head 41 is not located within the first detection area A (NO in step S12), the process returns to step S11. On the other hand, if the image position determination unit 203 determines that the entire head 41 is located within the first detection area A (YES in step S12), the process proceeds to step S13.

[0086] In step S13, the identification marker detection unit 204 detects the identification marker 52 on the image in which it has been determined in step S12 that the entire head 41 is located within the first detection area A. At this time, the identification marker detection unit 204 also acquires the identification information of the detected identification marker 52.

[0087] In step S14, the identification marker detection unit 204 determines whether or not at least one of the identification markers 52 detected in step S13 is located within the second detection region B. When the identification marker detection unit 204 determines that the detected identification marker 52 is not located within the second detection region B (NO in step S14), the process returns to step S11. On the other hand, when the identification marker detection unit 204 determines that the detected identification marker 52 is located within the second detection region B (YES in step S14), the process proceeds to step S15. At this time, the identification marker detection unit 204 identifies the identification marker 52 that is closest to the center C of the image among the identification markers 52 in the second detection region B.

[0088] In step S15, the filtering processing unit 205 determines whether or not a matching image in which the identification marker 52 identified in step S14 is located at the very center of the image is stored in the storage unit 22. If the filtering processing unit 205 determines that a matching image in which the identification marker 52 is located at the very center of the image is already stored in the storage unit 22 (YES in step S15), the processing returns to step S11. On the other hand, if the filtering processing unit 205 determines that a matching image in which the identification marker 52 is located at the very center of the image is not stored in the storage unit 22 (NO in step S15), the processing proceeds to step S16.

[0089] In step S16, the filtering processing unit 205 executes a blur determination process on the image corresponding to the identification marker 52 identified in step S14. Specifically, the filtering processing unit 205 performs a Fourier transform on the image to obtain transformed data, and calculates an average value of spatial frequencies of the transformed data by excluding low spatial frequency components from the obtained transformed data.

[0090] In step S17, if the filtering processing unit 205 determines in step S16 that the average value is less than the blur determination value (NO in step S17), the process returns to step S11. On the other hand, if the filtering processing unit 205 determines in step S16 that the average value is equal to or greater than the blur determination value (YES in step S17), the filtering processing unit 205 determines that the degree of blur in the image is within the allowable range, and the process proceeds to step S18.

[0091] In step S18, the filtering processing unit 205 executes angle determination processing for the image in which the degree of blur is determined to be within the allowable range (greater than or equal to the blur determination value) in step S17. Specifically, the filtering processing unit 205 detects the position of a corner of the identification marker 52, specifies position information of the identification marker 52 on the image based on the detected corner position, and calculates a perspective transformation matrix using parameter information such as the focal length set in the imaging unit 21 and the size of the image sensor 213 acquired in step S11. Then, a rotation matrix is ​​extracted from the calculated perspective transformation matrix and converted into Euler angles, and it is determined whether or not each of the obtained Euler angles ψ, θ, and φ is within the range of the angle determination value.

[0092] In step S19, if the filtering processing unit 205 determines in step S18 that at least one of the Euler angles ψ, θ, and φ is outside the range of the angle judgment value (NO in step S19), the processing returns to step S11. On the other hand, if the filtering processing unit 205 determines in step S18 that all of the Euler angles ψ, θ, and φ are within the range of the angle judgment value (YES in step S19), the filtering processing unit 205 determines that the angle of the imaging unit 21 with respect to the identification marker 52 during imaging is within the allowable range, and the processing proceeds to step S20.

[0093] In step S20, the filtering processing unit 205 executes distance determination processing for the image for which the angle has been determined to be within the allowable range in step S19. Specifically, the filtering processing unit 205 calculates the actual distance L between the imaging unit 21 and the identification marker 52 at the time of imaging using the norm of the components of the translation vector from the perspective transformation matrix calculated in step S18, using the above-mentioned equation (5). Then, it is determined whether the calculated distance L is within the range of the distance determination value.

[0094] In step S21, if the filtering processing unit 205 determines in step S20 that the distance L is outside the range of the distance judgment value (NO in step S21), the processing returns to step S11. On the other hand, if the filtering processing unit 205 determines in step S20 that the distance L is within the range of the distance judgment value (YES in step S21), the filtering processing unit 205 determines that the distance L between the identification marker 52 and the imaging unit 21 at the time of shooting is within the allowable range, and the processing proceeds to step S22.

[0095] In step S22, the filtering processing unit 205 determines that the image for which the distance L is determined to be within the allowable range in step S21 is a suitable image, and stores the determined image in the storage unit 22.

[0096] In step S23, the display control unit 206 controls the display unit 25 to reflect, in the display showing the storage status of the identification marker images, that the image for the new identification marker 52 has been stored as a matching image.

[0097] In step S24, the transmission processing unit 207 determines whether or not the identification marker images of all the predetermined identification markers 52 are stored as matching images in the storage unit 22. If the transmission processing unit 207 determines that the identification marker images of all the identification markers 52 are not stored as matching images (NO in step S24), the processing returns to step S11. On the other hand, if the transmission processing unit 207 determines that the identification marker images of all the identification markers 52 are stored as matching images (YES in step S24), the processing proceeds to step S25.

[0098] In step S25, the transmission processing unit 207 transmits all of the matching images of the head 41, which is the subject of the current generation of a 3D model, stored in the storage unit 22, to the server 3. Thereafter, the 3D model generation process in the processing unit 20 ends.

[0099] Next, an example of the flow of the 3D model generation process executed by the server 3 of the head 3D model generation system 1 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the 3D model generation process executed by the server 3.

[0100] As shown in FIG. 14, in step S31, the server-side image acquisition unit 301 of the processing unit 30 acquires all the matching images transmitted in step S25.

[0101] In step S32, the background processor 302 executes background removal processing for each of all the matching images acquired in step S25. As a result, information other than the detection target image is removed from the matching images.

[0102] In step S33, the three-dimensional model generating unit 303 executes a process of generating a three-dimensional model of the head 41 using photogrammetry technology, based on the matching image that has been subjected to the background removal process in step S32.

[0103] In step S34, the three-dimensional model generation unit 303 stores the three-dimensional model generated in step S33. After that, the processing unit 30 ends the three-dimensional model generation process.

[0104] Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment and can be modified as appropriate.

[0105] In the above embodiment, the cap 5 having the identification marker 52 and the color pattern 53 formed on the surface 51 of the sheet 50 was used when photographing the head 41, but a cap 5 having the identification marker 52 and the color pattern 53 not formed on the surface 51 of the sheet 50 may be used, a cap 5 having the identification marker 52 and no color pattern 53 may be used, or a cap 5 having the color pattern 53 and no identification marker 52 may be used. Also, instead of the identification marker 52, the cap 5 may have no identification function and may simply have a marker that is square in plan view formed thereon.

[0106] In the above embodiment, the filtering processing unit 205 executes the blur determination processing, the angle determination processing, and the distance determination processing as filtering processing, but it is also possible to execute one or two of the three processing instead of executing all three processing. That is, it is also possible to execute multiple types of filtering processing to extract suitable images, and generate a 3D model based on the extracted multiple suitable images. This allows a more accurate 3D model of the head 41 to be generated.

[0107] In the above embodiment, the filtering processing unit 205 performs the blur determination process, the angle determination process, and the distance determination process in this order, but the order in which the processes are performed is not particularly limited. For example, the filtering processing may be performed in the order of the blur determination process, the distance determination process, and the angle determination process, or the filtering processing may be performed in the order of the distance determination process, the blur determination process, and the angle determination process, or the filtering processing may be performed in the order of the distance determination process, the angle determination process, and the blur determination process, or the filtering processing may be performed in the order of the angle determination process, the distance determination process, and the blur determination process, or the filtering processing may be performed in the order of the angle determination process, the blur determination process, and the distance determination process.

[0108] In the above embodiment, the server 3 has the background processing unit 302, but the configuration may not include the background processing unit 302, and background removal processing may not be performed in the 3D model generation processing. Alternatively, the electronic terminal 2 may have the background processing unit 302 instead of the server 3, and the electronic terminal 2 may transmit to the server 3 a matching image that has been subjected to background removal processing.

[0109] In the above embodiment, the electronic terminal 2 has the filtering processing unit 205, but the server 3 may have the filtering processing unit 205.

[0110] In the above embodiment, the terminal side image acquisition unit 201 had an acquisition unit 202, an image position determination unit 203, and an identification marker detection unit 204, but it may be configured not to include at least one of the image position determination unit 203 and the identification marker detection unit 204.

[0111] Furthermore, for example, the identification marker detection unit 204 may use a learning model constructed by performing supervised learning to determine whether or not at least one identification marker 52 on an image is entirely located within the second detection region B. In this case, the supervised learning may be performed in such a manner that, for example, an image including, as a subject, an identification marker 52 formed on the surface 51 of the sheet 50 covering the head 41, is used as input data, an evaluation result of whether or not at least one identification marker 52 in the image is entirely located within the second detection region B is used as a label, and many pairs of the input data and the label are used as teacher data.

[0112] In the above embodiment, when an identification marker image of the same identification marker 52 as the identification marker image transferred from the terminal side image acquisition unit 201 is stored in the memory unit 22, the filtering processing unit 205 discards the transferred image. However, the transferred image may not be discarded, but may be stored as a backup in a backup memory area of ​​the memory unit 22.

[0113] According to the embodiment and the modified examples described above, the following effects are achieved.

[0114] (1) The head 3D model generation system 1 comprises an electronic terminal 2 having an imaging unit 21 that images the head 41 of an infant 4 and a processing unit 20 that processes images of the head 41 imaged by the imaging unit 21, and a server 3 that can communicate with the electronic terminal 2. The processing unit 20 of the electronic terminal 2 has a terminal-side image acquisition unit 201 that acquires multiple images of the head 41 imaged from different directions by the imaging unit 21, and a filtering processing unit 205 that determines whether the images acquired by the terminal-side image acquisition unit 201 are suitable images that satisfy specified conditions. The server 3 has a 3D model generation unit 303 that generates a 3D model showing the three-dimensional external shape of the head 41 based on the multiple suitable images determined by the filtering processing unit 205 to satisfy the specified conditions.

[0115] As a result, even when generating a 3D model of the head 41 of an infant or young child 4, which is difficult to keep still, an accurate 3D model of the head can be generated since the 3D model of the head is generated using only images that satisfy predetermined conditions.

[0116] (2) In the head 3D model generation system 1 described in (1), the predetermined conditions include a condition regarding the degree of blurring of the image.

[0117] As a result, even if, for example, the baby 4 moves at the moment of photographing and the image of the head 41 is likely to become blurred, the images to be used for generating the 3D model are selected according to the degree of blurring, so that a more accurate 3D model of the head 41 can be generated.

[0118] (3) In the head 3D model generation system 1 described in (2), the filtering processing unit 205 performs a Fourier transform on the image acquired by the terminal side image acquisition unit 201 to acquire transformed data, and determines the degree of blurring of the image based on the average value of spatial frequencies in the transformed data excluding low spatial frequency components.

[0119] This allows a Fourier transform to be performed, and uses only the spatial frequency components excluding the low-frequency spatial frequency components in which it is difficult to confirm differences between images in the obtained transformed data, so that the degree of blur can be specified more accurately.

[0120] (4) In the head 3D model generation system 1 described in (1), the predetermined conditions include a condition regarding the distance L between the imaging unit 21 and the head 41 when the imaging unit 21 images the head 41.

[0121] As a result, even in situations where it is difficult to photograph an infant 4, who has difficulty in remaining still during photography, from various angles under fixed conditions, images to be used for generating a 3D model are selected taking into consideration the distance between the photography unit 21 and the head 41, so that a more accurate 3D model of the head 41 can be generated.

[0122] (5) In the head 3D model generation system 1 described in (4), the photographing unit 21 photographs the head 41 covered by a sheet, and a plurality of markers are formed at intervals on the surface of the portion of the sheet 50 covering the head 41, and the terminal side image acquisition unit 201 acquires an image in which the marker is located within a second detection area B, which is a predetermined range from the center C, as an image corresponding to the marker, and the filtering processing unit 205 determines the distance between the photographing unit 21 and the marker based on position information indicating the position of the marker on the image corresponding to the marker and parameter information including the focal length set in the photographing unit 21 and the size of the image sensor.

[0123] As a result, the distance is calculated based on the marker located on the center C side of the captured image, so that the distance between the imaging unit 21 and the head 41 can be specified more accurately.

[0124] (6) In the head 3D model generation system 1 described in (1), the predetermined conditions include a condition regarding the orientation of the imaging unit 21 with respect to the head 41 when the imaging unit 21 images the head 41.

[0125] As a result, even in a situation where it is difficult to photograph an infant 4, who has difficulty in remaining still during photography, from various angles and under fixed conditions, images to be used for generating a 3D model are selected taking into consideration the angle between the photography unit 21 and the head 41, so that a more accurate 3D model of the head 41 can be generated.

[0126] (7) In the head 3D model generation system 1 described in (1), the photographing unit 21 photographs the head 41 covered by the sheet 50, and a plurality of markers are formed at intervals on the surface of the portion of the sheet 50 covering the head 41, and the terminal side image acquisition unit 201 acquires an image in which the marker is located within a second detection area B that is a predetermined range from the center as an image corresponding to the marker, and the filtering processing unit 205 identifies the orientation of the photographing unit 21 relative to the head 41 based on positional information indicating the position of the marker on the image corresponding to the marker and parameter information including the focal length set in the photographing unit 21 and the size of the image sensor.

[0127] As a result, since the angle is estimated from the information of the marker positioned on the side of the center C of the captured image, the angle of the imaging unit 21 with respect to the head 41 can be specified more accurately.

[0128] (8) In the head 3D model generation system 1 described in any one of (1) to (7), the photographing unit 21 photographs the head 41 covered with a sheet 50, and a plurality of identification markers 52 associated with position information of the head 41 are formed at intervals on the surface of the portion of the sheet 50 covering the head 41, and a plurality of colored patterns 53 are formed, the terminal side image acquisition unit 201 acquires an image in which the identification marker 52 is located within a second detection area B that is a predetermined range from the center C as an image corresponding to the identification marker 52, the filtering processing unit 205 determines whether the image corresponding to the identification marker 52 acquired by the terminal side image acquisition unit 201 is a suitable image, and the 3D model generation unit 303 generates a 3D model based on the position information and colored patterns 53 associated with the identification marker 52 of the suitable image of the image corresponding to all of the predetermined identification markers 52.

[0129] As a result, position information of the head 41 associated with the identification marker 52 and information on the color and pattern 53 are used to generate a 3D model, making it easier to more accurately determine the positional relationship between each of the multiple matching images, and enabling a more accurate 3D model of the head 41 to be generated.

[0130] In the head 3D model generating system 1 described in (9) and (8), all of the color patterns 53 formed on the front surface 51 of the sheet 50 are different from each other.

[0131] As a result, there are no parts with the same color and pattern 53 on the surface 51 of the sheet 50 covering the head 41, so the differences in features according to the parts of the head 41 become clearer, making it easier to more accurately determine which part of the head 41 the acquired image is from, and a more accurate three-dimensional model of the head 41 can be generated.

[0132] (10) In the head 3D model generation system 1 described in any one of (1) to (9), the specified conditions include a condition regarding the distance between the imaging unit 21 and the head 41 when the imaging unit 21 images the head 41, and a condition regarding the orientation of the imaging unit 21 relative to the head 41 when the imaging unit 21 images the head 41.

[0133] As a result, even in situations where it is difficult to photograph an infant 4, which has difficulty in remaining still during photography, from various angles under fixed conditions, images to be used for generating a 3D model are selected taking into consideration the distance and angle between the photography unit 21 and the head 41, so that a more accurate 3D model of the head 41 can be generated.

[0134] (11) In the head 3D model generation system 1 according to any one of (1) to (10), the predetermined conditions further include a condition regarding the degree of blurring of the image.

[0135] This allows the images used to generate the 3D model to be selected based on the degree of blur, resulting in a more accurate 3D model of the head.

[0136] (12) In the head 3D model generation system 1 described in any one of (1) to (11), the filtering processing unit 205 determines whether or not the image acquired by the terminal side image acquisition unit 201 satisfies a condition regarding the degree of blur, and then determines whether or not the image determined to satisfy the condition regarding the degree of blur satisfies a condition regarding the distance between the imaging unit 21 and the head 41 when the imaging unit 21 images the head 41, and a condition regarding the orientation of the imaging unit 21 with respect to the head 41 when the imaging unit 21 images the head 41.

[0137] As a result, since the image is selected according to the degree of blur and then the distance and angle are judged using a clear image, the distance and angle can be specified more accurately. Therefore, the distance and angle can be judged more accurately, and a more accurate 3D model of the head 41 can be generated.

[0138] (13) In the head 3D model generation system 1 described in any one of (1) to (12), the server 3 further has a background processing unit 302 that detects a detection target image including the head 41 of the infant 4 on the matching image and removes information other than the detection target image on the matching image, and the 3D model generation unit 303 generates a 3D model showing the three-dimensional external shape of the head 41 based on the multiple matching images from which information other than the detection target image has been removed by the background processing unit 302.

[0139] As a result, a matching image is used from which only information about the detection target image including the head 41 of the infant 4 is extracted, so that the feature amounts required for generating a 3D model of the head 41 can be narrowed down, and a more accurate 3D model of the head 41 can be generated.

[0140] (14) The program causes a computer to execute an image acquisition step of acquiring multiple images of the head 41 of an infant 4 taken from different angles, a filtering processing step of determining whether the images acquired in the image acquisition step are suitable images that satisfy predetermined conditions, and a three-dimensional model generation step of generating a three-dimensional model showing the three-dimensional external shape of the head based on the suitable images determined in the filtering processing step to satisfy the predetermined conditions.

[0141] (15) The program causes a computer to execute a three-dimensional model generation process for generating a three-dimensional model together with a server 3 that generates a three-dimensional model showing the three-dimensional external shape of the head 41 of an infant 4 based on multiple images of the head 41. The three-dimensional model generation process includes an image acquisition process for acquiring multiple images of the infant's head taken from different directions, a filtering process process for determining whether the images acquired in the image acquisition process are suitable images that satisfy specified conditions, and a transmission process for transmitting the suitable images determined in the filtering process to the server 3 as satisfying the specified conditions.

[0142] (16) The three-dimensional model generation method is a method executed by a computer and includes an image acquisition step of acquiring multiple images of the head 41 of an infant 4 taken from different directions, a filtering processing step of determining whether the images acquired in the image acquisition step are suitable images that satisfy predetermined conditions, and a three-dimensional model generation step of generating a three-dimensional model showing the three-dimensional external shape of the head 41 based on the suitable images determined in the filtering processing step to satisfy the predetermined conditions. [Explanation of symbols]

[0143] 1. 3D head model generation system 2. Electronic Devices 3 Server 4. Infants and toddlers (target group) 20 Processing section 21 Photography Department 41 Head 201 Image acquisition unit (terminal side image acquisition unit) 205 Filtering processing section 303 3D model generation unit

Claims

1. A head 3D model generation system including an electronic terminal having an imaging unit that images a subject's head and a processing unit that processes an image of the head captured by the imaging unit, and a server that can communicate with the electronic terminal, The processing unit of the electronic terminal an image acquisition unit that acquires a plurality of images of the head captured from different directions by the imaging unit; a filtering processing unit that determines whether or not the image acquired by the image acquisition unit is a suitable image that satisfies a predetermined condition, The server, a three-dimensional model generating unit that generates a three-dimensional model indicating a three-dimensional external shape of the head based on a plurality of matching images that are determined to satisfy the predetermined condition by the filtering processing unit, the predetermined condition includes a condition regarding a degree of blur of an image, The filtering processing unit includes: performing a Fourier transform on the image acquired by the image acquisition unit to acquire transformed data; A system for generating a three-dimensional head model, which identifies the degree of blurring of the image based on an average value of spatial frequencies in the converted data, excluding low spatial frequency components.

2. A head 3D model generation system including an electronic terminal having an imaging unit that images a subject's head and a processing unit that processes an image of the head captured by the imaging unit, and a server that can communicate with the electronic terminal, The processing unit of the electronic terminal an image acquisition unit that acquires a plurality of images of the head captured from different directions by the imaging unit; a filtering processing unit that determines whether or not the image acquired by the image acquisition unit is a suitable image that satisfies a predetermined condition, The server, a three-dimensional model generating unit that generates a three-dimensional model indicating a three-dimensional external shape of the head based on a plurality of matching images that are determined to satisfy the predetermined condition by the filtering processing unit, The imaging unit images the head while it is covered with a sheet, A plurality of identification markers associated with position information of the head are formed at intervals on a surface of the portion of the sheet covering the head, and a plurality of color patterns are formed; the image acquisition unit acquires the image in which the identification marker is located within a detection area that is a predetermined range from a center as an image corresponding to the identification marker; The filtering processing unit determines whether or not an image corresponding to the identification marker acquired by the image acquisition unit is the matching image; The three-dimensional model generation unit is a head three-dimensional model generation system that generates the three-dimensional model based on the position information and the color pattern associated with the identification markers of the compatible images of images corresponding to all of the predetermined identification markers.

3. A head 3D model generation system including an electronic terminal having an imaging unit that images a subject's head and a processing unit that processes an image of the head captured by the imaging unit, and a server that can communicate with the electronic terminal, The processing unit of the electronic terminal an image acquisition unit that acquires a plurality of images of the head captured from different directions by the imaging unit; a filtering processing unit that determines whether or not the image acquired by the image acquisition unit is a suitable image that satisfies a predetermined condition, The server, a three-dimensional model generating unit that generates a three-dimensional model indicating a three-dimensional external shape of the head based on a plurality of matching images that are determined to satisfy the predetermined condition by the filtering processing unit, the predetermined conditions include a condition regarding a degree of blurring of the image, a condition regarding a distance between the imaging unit and the head when the imaging unit images the head, and a condition regarding an orientation of the imaging unit with respect to the head when the imaging unit images the head, The filtering processing unit determines whether or not the image acquired by the image acquisition unit satisfies a condition related to the degree of blur, and then determines whether or not the image determined to satisfy the condition related to the degree of blur satisfies a condition related to the distance between the imaging unit and the head when the head is photographed by the imaging unit, and a condition related to the orientation of the imaging unit relative to the head when the head is photographed by the imaging unit.

4. 2. The system for generating a three-dimensional head model according to claim 1, wherein the predetermined conditions include a condition regarding a distance between the imaging unit and the head when the imaging unit captures an image of the head.

5. The imaging unit images the head while it is covered with a sheet, A plurality of markers are formed at intervals on a surface of the portion of the seat covering the head, the image acquisition unit acquires the image in which the marker is located within a detection area that is a predetermined range from a center as an image corresponding to the marker; The filtering processing unit includes:

5. The head three-dimensional model generation system according to claim 3, wherein the distance between the imaging unit and the marker is specified based on position information indicating the position of the marker on the image corresponding to the marker and parameter information including a focal length set in the imaging unit and a size of an image sensor.

6. The system for generating a three-dimensional head model according to claim 1 , wherein the predetermined conditions include a condition regarding an orientation of the imaging unit with respect to the head when the imaging unit captures an image of the head.

7. The imaging unit images the head while it is covered with a sheet, A plurality of markers are formed at intervals on a surface of the portion of the seat covering the head, the image acquisition unit acquires an image in which the marker is located within a detection area that is a predetermined range from a center as an image corresponding to the marker; The filtering processing unit includes:

7. The head three-dimensional model generation system according to claim 3 or 6, wherein an orientation of the imaging unit with respect to the head is specified based on position information indicating a position of the marker on an image corresponding to the marker and parameter information including a focal length set in the imaging unit and a size of an image sensor.

8. 3. The system for generating a three-dimensional head model according to claim 2, wherein all of the color patterns formed on the surface of the sheet are different from each other.

9. 3. The system for generating a three-dimensional head model according to claim 2, wherein the predetermined conditions include a condition regarding a degree of blurring of the image.

10. an image acquisition step of acquiring a plurality of images of the subject's head, each imaged from a different direction; a filtering process step for determining whether or not the image acquired in the image acquisition step is a suitable image that satisfies a predetermined condition; a three-dimensional model generating step of generating a three-dimensional model showing a three-dimensional external shape of the head based on a plurality of matching images determined to satisfy the predetermined condition in the filtering processing step, the predetermined condition includes a condition regarding a degree of blur of an image, The filtering process includes: Obtaining transformed data by performing a Fourier transform on the image acquired in the image acquisition step; A program for specifying the degree of blur of the image based on an average value of spatial frequencies excluding low spatial frequency components in the converted data.

11. an image acquisition step of acquiring a plurality of images of the subject's head, each imaged from a different direction; a filtering process step for determining whether or not the image acquired in the image acquisition step is a suitable image that satisfies a predetermined condition; a three-dimensional model generating step of generating a three-dimensional model showing a three-dimensional external shape of the head based on a plurality of matching images determined to satisfy the predetermined condition in the filtering processing step, each of the plurality of images of the head is an image of the head covered by a sheet; A plurality of identification markers associated with position information of the head are formed at intervals on a surface of the portion of the sheet covering the head, and a plurality of color patterns are formed; The image acquisition step acquires the image in which the identification marker is located within a detection area that is a predetermined range from a center as an image corresponding to the identification marker, The filtering process step determines whether or not the image corresponding to the identification marker acquired in the image acquisition step is the matching image; The three-dimensional model generating step is a program for generating the three-dimensional model based on the position information and the color and pattern associated with the identification markers of the matching images of images corresponding to all of the predetermined identification markers.

12. an image acquisition step of acquiring a plurality of images of the subject's head taken from different directions by an imaging unit that images the subject's head; a filtering process step for determining whether or not the image acquired in the image acquisition step is a suitable image that satisfies a predetermined condition; a three-dimensional model generating step of generating a three-dimensional model showing a three-dimensional external shape of the head based on a plurality of matching images determined to satisfy the predetermined condition in the filtering processing step, the predetermined conditions include a condition regarding a degree of blurring of the image, a condition regarding a distance between the imaging unit and the head when the imaging unit images the head, and a condition regarding an orientation of the imaging unit with respect to the head when the imaging unit images the head, The filtering processing process is a program that determines whether or not the image acquired in the image acquisition process satisfies a condition regarding the degree of blur, and then determines whether or not the image that is determined to satisfy the condition regarding the degree of blur satisfies a condition regarding the distance between the imaging unit and the head when the head is photographed by the imaging unit, and a condition regarding the orientation of the imaging unit relative to the head when the head is photographed by the imaging unit.

13. A server that generates a three-dimensional model showing a three-dimensional external shape of a subject's head based on a plurality of images of the head of the subject, and a program that causes a computer to execute a three-dimensional model generation process for generating the three-dimensional model, The three-dimensional model generation process includes: an image acquisition step of acquiring a plurality of images of the subject's head, each imaged from a different direction; a filtering process step for determining whether or not the image acquired in the image acquisition step is a suitable image that satisfies a predetermined condition; a transmission step of transmitting a plurality of matching images determined to satisfy a predetermined condition in the filtering processing step to the server, the predetermined condition includes a condition regarding a degree of blur of an image, The filtering process includes: Obtaining transformed data by performing a Fourier transform on the image acquired in the image acquisition step; A program for specifying the degree of blur of the image based on an average value of spatial frequencies excluding low spatial frequency components in the converted data.

14. A server that generates a three-dimensional model showing a three-dimensional external shape of a subject's head based on a plurality of images of the head of the subject, and a program that causes a computer to execute a three-dimensional model generation process for generating the three-dimensional model, The three-dimensional model generation process includes: an image acquisition step of acquiring a plurality of images of the subject's head taken from different directions by an imaging unit that images the subject's head; a filtering process step for determining whether or not the image acquired in the image acquisition step is a suitable image that satisfies a predetermined condition; a transmission step of transmitting a plurality of matching images determined to satisfy a predetermined condition in the filtering processing step to the server, the predetermined conditions include a condition regarding a degree of blurring of the image, a condition regarding a distance between the imaging unit and the head when the imaging unit images the head, and a condition regarding an orientation of the imaging unit with respect to the head when the imaging unit images the head, The filtering processing process is a program that determines whether or not the image acquired in the image acquisition process satisfies a condition regarding the degree of blur, and then determines whether or not the image that is determined to satisfy the condition regarding the degree of blur satisfies a condition regarding the distance between the imaging unit and the head when the head is photographed by the imaging unit, and a condition regarding the orientation of the imaging unit relative to the head when the head is photographed by the imaging unit.

15. 1. A computer-implemented method for generating a three-dimensional model, comprising the steps of: an image acquisition step of acquiring a plurality of images of the subject's head, each imaged from a different direction; a filtering process step for determining whether or not the image acquired in the image acquisition step is a suitable image that satisfies a predetermined condition; a three-dimensional model generating step of generating a three-dimensional model showing a three-dimensional external shape of the head based on a plurality of matching images determined to satisfy the predetermined condition in the filtering processing step, the predetermined condition includes a condition regarding a degree of blur of an image, In the filtering process, Obtaining transformed data by performing a Fourier transform on the image acquired in the image acquisition step; A three-dimensional model generating method that specifies the degree of blurring of the image based on an average value of spatial frequencies excluding low spatial frequency components in the converted data.

16. 1. A computer-implemented method for generating a three-dimensional model, comprising the steps of: an image acquisition step of acquiring a plurality of images of the subject's head, each imaged from a different direction; a filtering process step for determining whether or not the image acquired in the image acquisition step is a suitable image that satisfies a predetermined condition; a three-dimensional model generating step of generating a three-dimensional model showing a three-dimensional external shape of the head based on a plurality of matching images determined to satisfy the predetermined condition in the filtering processing step, each of the plurality of images of the head is an image of the head covered by a sheet; A plurality of identification markers associated with position information of the head are formed at intervals on a surface of the portion of the sheet covering the head, and a plurality of color patterns are formed; In the image acquisition step, the image in which the identification marker is located within a detection area having a predetermined range from a center is acquired as an image corresponding to the identification marker, In the filtering process, it is determined whether or not the image corresponding to the identification marker acquired in the image acquisition process is the matching image; A three-dimensional model generation method in which the three-dimensional model generation process generates the three-dimensional model based on the position information and the color pattern associated with the identification markers of the compatible images of images corresponding to all of the predetermined identification markers.

17. 1. A computer-implemented method for generating a three-dimensional model, comprising the steps of: an image acquisition step of acquiring a plurality of images of the subject's head taken from different directions by an imaging unit that images the subject's head; a filtering process step for determining whether or not the image acquired in the image acquisition step is a suitable image that satisfies a predetermined condition; a three-dimensional model generating step of generating a three-dimensional model showing a three-dimensional external shape of the head based on a plurality of matching images determined to satisfy the predetermined condition in the filtering processing step, the predetermined conditions include a condition regarding a degree of blurring of the image, a condition regarding a distance between the imaging unit and the head when the imaging unit images the head, and a condition regarding an orientation of the imaging unit with respect to the head when the imaging unit images the head, In the filtering processing step, after determining whether or not the image acquired in the image acquisition step satisfies a condition regarding the degree of blur, the method determines whether or not the image determined to satisfy the condition regarding the degree of blur satisfies a condition regarding the distance between the imaging unit and the head when the head is photographed by the imaging unit, and a condition regarding the orientation of the imaging unit relative to the head when the head is photographed by the imaging unit.

Images