ITEM IMAGE SYNTHESIS DEVICE, MODEL INFORMATION GENERATION DEVICE, ITEM IMAGE SYNTHESIS METHOD, AND PROGRAM

Abstract

To output a natural composite image which does not cause the sense of discomfort.SOLUTION: An item image composition device 100 comprises: a model information acquisition unit 141; a user image acquisition unit 142; a user lattice point setting unit 143; and an item image composition unit 144. The model information acquisition unit 141 acquires a model image obtained by imaging a model wearing an item, an item image corresponding to the item extracted from the model image, and model lattice points at a prescribed interval set at least in a portion of the model shape showing the shape of the model wearing the item in association with one another. The item image composition unit 144 deforms the item image by making the model lattice point correspond to a user lattice point and combines the deformed item image and a user image.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present disclosure relates to an item image synthesis device, a model information generation device, an item image synthesis method, and a program. [Background technology]

[0002] In recent years, technologies have been developed that display what a person will look like when trying on clothing. For example, Patent Document 1 (JP 2019-128923 A) discloses an information providing device that allows a person to select clothing that fits their body type by referring to images and descriptions of multiple pieces of clothing that can be viewed on an e-commerce site. Summary of the Invention [Problem to be solved by the invention]

[0003] However, conventional techniques may output unnatural composite images that cause discomfort. [Means for solving the problem]

[0004] The item image synthesis device according to a first aspect transforms an item image showing an item by matching model lattice points in a model image showing a model wearing the item with user lattice points in a user image, and synthesizes the transformed item image with the user image. In this way, the item image synthesis device according to the first aspect transforms the item image by matching model lattice points with user lattice points, and can output a natural synthesized image that does not create an awkward feeling.

[0005] An item image compositing device according to a second aspect is the item image compositing device according to the first aspect, and when the grid point interval in the model image is equal to or greater than a predetermined value, performs a predetermined enlargement process on the grid point interval in the user image, thereby preventing the item image from appearing unnatural when combined with the user image.

[0006] An item image synthesis device according to a third aspect is the item image synthesis device according to the first or second aspect, in which model lattice points are set for at least a part of a model shape extracted from a model image based on model skeleton information acquired from the model image. Since the item image is transformed using such model lattice points, it is possible to prevent an unnatural feeling from occurring when the transformed item image is synthesized with a user image.

[0007] An item image compositing device according to a fourth aspect is the item image compositing device according to the third aspect, in which the model lattice points are set to correspond to the shape of the model when not wearing the item. Since the item image is transformed using these model lattice points, even if the item image does not reflect the model's body shape, it is possible to avoid creating an awkward feeling when the transformed item image is composited with the user image.

[0008] An item image synthesis device according to a fifth aspect is an item image synthesis device according to any one of the first to fourth aspects, and includes a model information acquisition unit, a user image acquisition unit, a user lattice point setting unit, and an item image synthesis unit. The model information acquisition unit acquires a model image showing a model wearing an item, an item image corresponding to an item extracted from the model image, and model lattice points set at predetermined intervals on at least a portion of the model shape showing the shape of the model wearing the item, in association with each other. The user image acquisition unit acquires a user image showing a user. The user lattice point setting unit sets user lattice points at predetermined intervals on at least a portion of the user shape showing the shape of the user shown in the user image. The item image synthesis unit transforms the item image by associating the model lattice points with the user lattice points, and synthesizes the transformed item image with the user image. In this way, the item image synthesis device according to the fifth aspect transforms the item image by associating the model lattice points with the user lattice points, and synthesizes the transformed item image with the user image, thereby outputting a natural-looking synthesized image that does not create an awkward feeling.

[0009] An item image synthesis device according to a sixth aspect is the item image synthesis device according to the fifth aspect, wherein the user lattice point setting unit includes a user shape extraction unit, a user skeletal information acquisition unit, and a user lattice point processing unit. The user shape extraction unit extracts a user shape indicating the shape of the user from the user image. The user skeletal information acquisition unit acquires user skeletal information for the user from the user image. The user lattice point processing unit sets user lattice points for at least a portion of the user shape based on the user skeletal information. Since the item image is transformed using these user lattice points, it is possible to prevent an unnatural feeling from occurring when the transformed item image is synthesized with the user image.

[0010] An item image compositing device according to a seventh aspect is the item image compositing device according to the first to sixth aspects, further comprising a user image generating unit that generates a user image based on at least one of attribute data of height, weight, and sex. With this configuration, any user image can be prepared.

[0011] In an item image compositing method according to an eighth aspect, a model image showing a model wearing an item, an item image corresponding to the item extracted from the model image, and model grid points set at predetermined intervals on at least a portion of the model shape showing the shape of the model wearing the item are acquired in association with each other. Next, this method acquires a user image showing a user. Next, this method sets user grid points at predetermined intervals on at least a portion of the user shape showing the shape of the user shown in the user image. Next, this method transforms the item image by correlating the model grid points with the user grid points, and combines the transformed item image with the user image. This makes it possible to output a natural-looking composite image that does not create an awkward feeling when the item image is combined with the user image.

[0012] A program according to a ninth aspect causes a computer to execute the item image synthesis method according to the eighth aspect. By executing such a program, the computer can output a natural synthesized image that does not create an awkward feeling when the item image is synthesized with the user image.

[0013] A model information generation device according to a tenth aspect includes a model image acquisition unit, an item image extraction unit, a model shape extraction unit, a model skeleton information acquisition unit, a model lattice point setting unit, and a model information generation unit. The model image acquisition unit acquires a model image showing a model wearing an item. The item image extraction unit extracts an item image corresponding to the item from the model image. The model shape extraction unit extracts a model shape indicating the shape of the model wearing the item from the model image. The model skeleton information acquisition unit acquires model skeleton information for the model from the model image. The model lattice point setting unit sets model lattice points for at least a portion of the model shape based on the model skeleton information. The model information generation unit generates model information by associating the model image with the model lattice points. In this way, the model information generation device according to the eighth aspect extracts item images from the model image and sets model lattice points based on the model skeleton information, thereby providing information for synthesizing item images without creating an awkward look with a user image of a user with a body type other than that of the model.

[0014] A model information generating device according to an eleventh aspect is the model information generating device according to the tenth aspect, in which the model lattice point setting unit sets model lattice points for the shape of the model when not wearing the item. Since the item image is transformed using these model lattice points, even if the item image does not reflect the body shape of the model, it is possible to prevent an unnatural appearance when the transformed item image is combined with the user image.

[0015] A model information generation method according to a twelfth aspect acquires a model image showing a model wearing an item. Next, the method extracts an item image corresponding to the item from the model image. Next, the method extracts a model shape showing the shape of the model wearing the item from the model image. Next, the method acquires model skeleton information for the model from the model image. Next, the method sets model lattice points for at least a portion of the model shape based on the model skeletal information. Next, the method generates model information by associating the model image with the model lattice points. This method can provide information for synthesizing an item image without creating an awkward look with a user image of a user with a body type other than that of the model.

[0016] An item image synthesis device according to a thirteenth aspect includes a model information acquisition unit, a user image acquisition unit, a user skeletal information acquisition unit, and an item image synthesis unit. The model information acquisition unit acquires a model image of a model wearing an item, an item image corresponding to an item extracted from the model image, and model skeletal information for the model acquired from the model image, in association with each other. The user image acquisition unit acquires a user image of a user. The user skeletal information acquisition unit acquires user skeletal information for the user from the user image. The item image synthesis unit transforms the user image by matching the model skeletal information with the user skeletal information, and synthesizes the transformed user image with the item image. In this way, the item image synthesis device according to the thirteenth aspect transforms the user image by matching the model skeletal information with the user skeletal information, and synthesizes the transformed user image with the item image, thereby outputting a natural-looking synthesized image that does not create an awkward feeling. An item image compositing method according to a fourteenth aspect acquires a model image of a model wearing an item, an item image corresponding to an item extracted from the model image, and model skeleton information for the model acquired from the model image, in association with each other. Next, the method acquires a user image of a user. Subsequently, the method acquires user skeleton information for the user from the user image. Next, the method transforms the user image by correlating the model skeleton information with the user skeleton information, and combines the transformed user image with the item image. This method makes it possible to output a natural composite image that does not create an awkward feeling. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a schematic diagram illustrating a configuration of an item image synthesis system 1 according to an embodiment of the present disclosure. [Figure 2] 2 is a schematic diagram showing a configuration of an item image composition device 100 according to the embodiment. FIG. [Figure 3] FIG. 10 is a schematic diagram showing the concept of a "model image M1" according to the embodiment. [Figure 4] 10 is a schematic diagram showing the concept of an "item image M2" according to the embodiment. FIG. [Figure 5] FIG. 10 is a schematic diagram showing the concept of a "model lattice point M5" according to the embodiment. [Figure 6] FIG. 10 is a schematic diagram showing the concept of a "model shape M3" according to the embodiment. [Figure 7] FIG. 10 is a schematic diagram showing the concept of “model skeleton information M4” according to the embodiment. [Figure 8] FIG. 10 is a schematic diagram showing the configuration of a user grid point setting unit 143 according to the embodiment. [Figure 9] FIG. 2 is a schematic diagram showing the configuration of a model information generating device 200 according to the embodiment. [Figure 10] 10 is a flowchart illustrating a procedure for storing model information according to the embodiment. [Figure 11] FIG. 10 is a schematic diagram for explaining a procedure for storing model information according to the embodiment. [Figure 12] 10 is a flowchart for explaining the operation of the item image composition device 100 according to the embodiment. [Figure 13] FIG. 10 is a schematic diagram for explaining the setting of user lattice points according to the embodiment. [Figure 14] FIG. 10 is a schematic diagram for explaining a composite image U6 according to the same embodiment. [Figure 15] FIG. 10 is a schematic diagram for explaining a composite image U8 according to the same embodiment. [Figure 16] FIG. 10 is a schematic diagram for explaining an inappropriate composite image U9 according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] (1) Configuration of the item image synthesis system Hereinafter, a configuration of an item image synthesis system according to an embodiment of the present disclosure will be described with reference to the drawings.

[0019] FIG. 1 is a schematic diagram showing the configuration of an item image synthesis system 1 according to this embodiment. In the item image synthesis system 1, a user image is transmitted from a user terminal 5 to an item image synthesis device 100. A plurality of model images are output to the user terminal 5 by the item image synthesis device 100, and one model image is selected by the user terminal 5. The item image synthesis device 100 uses the model image selected by the user terminal 5 and the user image transmitted from the user terminal 5 to synthesize the item image included in the model image with the user image, and provides the synthesized image to the user terminal 5. Note that model information required for synthesizing images is generated by a model information generation device 200 and provided to the item image synthesis device 100. Details of each device are described below.

[0020] (1-1) Item image synthesis device FIG. 2 is a schematic diagram showing the configuration of the item image synthesis device 100. The item image synthesizing device 100 is a device that can synthesize an item shown in an arbitrary model image onto a user image, thereby enabling a user to virtually try on an item worn by an arbitrary model.

[0021] The item image synthesis device 100 can be realized by any computer, and includes a storage unit 110, an input / output unit 120, a communication unit 130, and a processing unit 140. The item image synthesis device 100 may also be realized as hardware using an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0022] The storage unit 110 stores various types of information and is realized by any storage device such as a memory or a hard disk. Here, the storage unit 110 stores at least a model image and an item image in association with each other as "model information." Items shown in the item images are classified into first type items and second type items.

[0023] A "first type item" is an item that fits at least partly to the model's body shape. For example, it is an item like a sweatshirt that fits the model's body shape around the wrists but is loose around the armpits. The model information corresponding to the first type item includes model grid points.

[0024] "Type 2 items" are items that are considered to have no part that fits the model's body shape. For example, they are items that loosely cover the model's entire body shape, such as a coat. Model information corresponding to type 2 items includes model skeleton information.

[0025] A "model image" is an image showing a model wearing an item such as clothing, as shown in M1 in Figure 3. Here, one model image corresponds to one model. Furthermore, the model image is associated with the model's height as metadata.

[0026] An "item image" is an image showing the appearance of an item when worn by a model, as shown in M2 in Figure 4. For example, an item image can be an image corresponding to an item such as a shirt, suit, pants, or shoes worn by a model. Note that Figure 4 shows a first type of item. Also, for convenience, the shirt and pants are extracted together in Figure 4, but they are treated as separate item images.

[0027] As shown in M5 in Figure 5, "model grid points" are grid point data set at a predetermined interval on at least a portion of the model shape representing the shape of a model wearing a first type of item. Specifically, the model shape is extracted from the model image using a machine learning model such as DeepLab v3 and saved as a silhouette image (see M3 in Figure 6). Therefore, the model shape deviates from the model's actual body shape by the amount of the item being worn. Therefore, the model grid points are set for the shape of the model when not wearing the item. In other words, the model grid points are not necessarily set over the entire interior area of ​​the model shape, but are set to fit the model's body shape. Such model grid points are set based on model skeletal information (see M4 in Figure 7). The model skeletal information is extracted using a machine learning model such as LightWeight OpenPose. In the example shown in Figure 7, the model skeletal information is generated to connect 18 feature points. Once the model skeletal information is generated, body parts and their central axes are identified based on the model skeletal information. The body part is then divided into a predetermined number of sections along the central axis, and a predetermined number of grid points are assigned in the direction perpendicular to the central axis at the divided positions to set model grid points. The silhouette image is normalized by the height of the model.

[0028] The input / output unit 120 is realized by any input device such as a keyboard, mouse, or touch panel, and / or any output device such as a display, touch panel, or speaker, and inputs and outputs various types of information to and from the computer. The communication unit 130 is realized by any network card or the like, and enables communication with communication devices on a network via wired or wireless communication.

[0029] The processing unit 140 executes various information processes and is realized by a processor such as a CPU and / or a GPU and a memory. Here, one or more programs stored in the storage unit 110 are loaded into the CPU, GPU, etc. of the computer, causing the processing unit 140 to function as a model information acquisition unit 141, a user image acquisition unit 142, a user grid point setting unit 143, and an item image synthesis unit 144.

[0030] The model information acquisition unit 141 acquires model information including a model image, an item image, and model grid points. Here, the model information acquisition unit 141 acquires the model information from the model information generation device 200. The acquired model information is stored in the storage unit 110.

[0031] The user image acquisition unit 142 acquires a user image of the user. Here, the user image is captured by the user terminal 5 operated by the user, and is transmitted from the user terminal 5 to the item image synthesis device 100.

[0032] The user grid point setting unit 143 sets "user grid points" at predetermined intervals in at least a part of a "user shape" that indicates the shape of the user reflected in the user image. Specifically, as shown in Fig. 8, the user grid point setting unit 143 includes a user shape extraction unit 143A, a user skeleton information acquisition unit 143B, and a user grid point processing unit 143C.

[0033] The user shape extraction unit 143A extracts a user shape from a user image. Specifically, the user shape is extracted using a machine learning model such as DeepLab v3 and saved as a silhouette image. The user image is normalized based on the user's height. In this case, the user image used is one in which the user is wearing loose clothing (such as underwear) or is naked, so that the user's body shape is reflected.

[0034] The user skeleton information acquisition unit 143B acquires user skeleton information for the user from the user image. For example, the user skeleton information is extracted using a machine learning model such as LightWeight OpenPose.

[0035] The user grid point processing unit 143C sets user grid points for at least a part of the user shape based on the user skeleton information. The user grid points are set to correspond to the model grid points, and the number of user grid points and the number of model grid points are the same.

[0036] If the item shown in the item image is a first type item, the item image synthesis unit 144 transforms the item image by associating the model lattice points with the user lattice points, and synthesizes the transformed item image with the user image. Here, if the lattice point spacing in the model image is equal to or greater than a predetermined value, the item image synthesis unit 144 performs a predetermined enlargement process on the lattice point spacing in the user image. This allows the item image to be synthesized without creating a sense of incongruity when synthesized with the user image. On the other hand, if the item shown in the item image is a second type item, the item image synthesis unit 144 modifies the user skeleton information to correspond to the angles of the central axes of each part obtained from the model skeleton information, and deforms the user image based on the modified user skeleton information. Then, the item image synthesis unit 144 synthesizes the item image with the transformed user image.

[0037] (1-2) Model information generation device FIG. 9 is a schematic diagram showing the configuration of the model information generating device 200. The model information generation device 200 provides model information to the item image synthesis device 100, and can be realized by any computer. When a program is loaded into the computer of the model information generation device 200, a processor such as a CPU and / or GPU functions as a model image acquisition unit 241, an item image extraction unit 242, a model shape extraction unit 243, a model skeleton information acquisition unit 244, a model lattice point setting unit 245, and a model information generation unit 246.

[0038] The model image acquisition unit 241 acquires an arbitrary model image (see M1 in FIG. 3). When acquiring the model image, data on the height of the model shown in the model image is acquired in association with the model image.

[0039] The item image extraction unit 242 extracts item images (see M2 in FIG. 4) corresponding to items from the model image. For example, the item image extraction unit 242 extracts the upper body part (shirt, jacket, etc.) and the lower body part (pants, skirt, etc.) of the clothes worn by the model shown in the model image as separate item images. However, without being limited to this, the item image extraction unit 242 may extract the entire clothes worn by the model shown in the model image as an item image.

[0040] The model shape extraction unit 243 extracts the model shape (see M3 in FIG. 6) from the model image. The model shape is saved as a silhouette image. At this time, the silhouette image is normalized by the height of the model.

[0041] The model skeleton information acquisition unit 244 acquires model skeleton information (see M4 in FIG. 7) for the model from the model image. For example, the model skeleton information is extracted using a machine learning model such as LightWeight OpenPose.

[0042] When a first type item is shown as the item image, the model grid point setting unit 245 sets model grid points (see M5 in FIG. 5) for at least a part of the model shape based on the model skeleton information.

[0043] When a first type item is displayed as an item image, the model information generation unit 246 generates model information by associating at least a model image M1, an item image M2, and a model grid point M5. When a second type item is displayed as an item image, the model information generation unit 246 generates model information by associating at least a model image M1, an item image M2, and model skeleton information M4. The model information generation unit 246 also sends the model information to the item image synthesis device 100 in response to a request.

[0044] (1-3) User terminal The user terminal 5 is a terminal used by a user and is realized by any computer. For example, a smart device functions as the user terminal 5 when a program is loaded into the smart device. The user terminal 5 has input / output functions, communication functions, photography functions, and the like. These functions enable the user terminal 5 to take a picture of the user, generate a user image, and transmit the user image to the item image synthesis device 100. The user terminal 5 can also receive a composite image from the item image synthesis device 100 and display it on a display. When transmitting a user image, the user terminal 5 accepts input of information such as the user's attributes (height, weight, gender), and transmits the information to the item image synthesis device 100 along with the user image.

[0045] (2) Operation of the item image synthesis system Next, the operation of the item image synthesis system 1 will be described.

[0046] (2-1) Preparing model information In the item image synthesis system 1, the item image synthesis device 100 synthesizes an item image onto a user image in response to a request from the user terminal 5. In order for the item image synthesis device 100 to synthesize an image, model information must be stored in advance in the storage unit 110. Therefore, the procedure for storing model information in the storage unit 110 will be described using the flowchart in FIG. 10. Images generated in each step are shown in FIG. 11. In FIG. 11, solid lines correspond to the generation of model information for a first type of item, and dotted lines correspond to the generation of model information for a second type of item. However, the second type of item image is not shown in FIG. 11.

[0047] First, the model information generating device 200 (model image acquiring unit 241) acquires a model image M1 (R1). Here, the model image M1 is an image of a model person photographed from the front while wearing the clothing of the product. When photographing, the model is asked to pose with their feet shoulder-width apart and their arms open at an angle of about 30 degrees. However, the model's pose is not limited to this.

[0048] Next, the model information generating device 200 (item image extracting unit 242) extracts an item image M2 from the model image M1 (R2). Specifically, an image obtained by cutting out only the clothes worn by the model from the model image M1 is extracted as the item image M2.

[0049] Next, the model information generation device 200 (model shape extraction unit 243) extracts a model shape M3 from the model image M1 (R3). For example, the model shape is extracted using a machine learning model such as DeepLab v3 and saved as a silhouette image. The silhouette image is normalized by the height of the model.

[0050] Next, the model information generating device 200 (model skeleton information acquiring unit 244) acquires model skeleton information M4 for the model from the model image M1 (R4). For example, the model skeleton information is extracted using a machine learning model such as LightWeight OpenPose.

[0051] Next, if a first type item is shown as the item image, the model information generating device 200 (model lattice point setting unit 245) sets model lattice points M5 for at least a portion of the model shape M3 based on the model skeleton information M4 (R5, R6). Here, the model information generating device 200 (model lattice point setting unit 245) identifies each body part and the central axis of each part based on the model skeleton information M4. Then, the model information generating device 200 (model lattice point setting unit 245) divides each part into a predetermined number of sections along the central axis and sets model lattice points so that a predetermined number of lattice points are assigned at each divided position in a direction perpendicular to the central axis. At this time, the model information generating device 200 (model lattice point setting unit 245) virtually calculates the thickness of the arm based on the model skeleton information and compares the calculated arm thickness with the arm thickness obtained from the silhouette image of the model shape. If the arm thickness obtained from the silhouette image is greater than the arm thickness calculated based on the model skeleton information, the model information generating device 200 (model lattice point setting unit 245) sets model lattice points in an area corresponding to the arm thickness calculated based on the model skeleton information. In this way, model lattice points are set for the shape of the model when not wearing the item. Note that if the shape of the item fits the model's body type, the shape of the model when not wearing the item will approximately match the shape of the model when wearing the item.

[0052] Next, if a first type item is shown as the item image, the model information generation device 200 (model information generation unit 246) generates model information by associating at least the model image M1, the item image M2, and the model grid point M5 (R7).

[0053] On the other hand, when a second type item is shown as the item image, the model information generating device (model information generating unit 246) generates model information by associating at least the model image M1, the item image M2, and the model skeleton information M4 (R5, R7).

[0054] Then, the model information generation device 200 sends the model information to the item image synthesis device 100. The item image synthesis device 100 stores the acquired model information in the storage unit 110 (R8).

[0055] (2-2) Operation of the item image synthesizer Next, a procedure for synthesizing an item image with a user image using the item image synthesizing device 100 will be described with reference to the flowchart in Fig. 12. Images generated in each step are shown in Figs.

[0056] First, a user image U1 of the user is captured via the user terminal 5 or the like, and the user image U1 is transmitted to the item image synthesis device 100. At this time, the user's attribute data (height, weight, gender, etc.) is also transmitted to the item image synthesis device 100. As a result, the item image synthesis device 100 acquires the user image U1 associated with the attribute data (S1). Note that the user image U1 used here is one in which the user is wearing loose clothing or is naked, so that the user's body shape is reflected.

[0057] Next, a model image M1 is selected by the user operating the user terminal 5 or the like (S1). Here, a plurality of model images showing a model wearing clothes etc. are displayed in a selectable manner on the user terminal 5. The user selects a model image M1 wearing clothes of their choice from the plurality of model images.

[0058] When the model image M1 is selected, the item image synthesis device 100 determines the type of item shown in the model image M1 (S3). Here, it determines whether the type of clothing is a first type or a second type.

[0059] In step S3, if the item image synthesis device 100 determines that the item is a first type based on the model image M1, it extracts a user shape U3 (see FIG. 13) from the user image U1 (S4). The user shape U3 is saved as a silhouette image. The user image U1 is normalized based on the user's height.

[0060] Next, the item image synthesis device 100 acquires user skeleton information U4 (see FIG. 13) from the user image U1 (S5).

[0061] Next, the item image synthesis device 100 sets user grid points U5 for at least a part of the user shape U3 (see FIG. 13) based on the user skeleton information U4 (S6).

[0062] Next, the item image compositing device 100 compares the model lattice points M5 corresponding to the model image M1 with the user lattice points U5 corresponding to the user image U1 (S7, see FIG. 14). At this time, enlargement processing may be performed depending on the difference between the spacing between the model lattice points M5 and the spacing between the user lattice points U5. For example, if the spacing between the model lattice points M5 corresponding to the arm region is greater than the spacing between the user lattice points U5, the spacing between the user lattice points U5 is enlarged (S7-YES, S8). This prevents the item image M2 from shrinking and creating an unnatural appearance when the item image M2 included in the model image M1 is composited with the user image U1. On the other hand, if the spacing between the model lattice points M5 corresponding to the arm region is smaller than the spacing between the user lattice points U5, the process proceeds to the next step without enlarging the spacing between the lattice points of the user image U1 (S7-NO).

[0063] Next, the item image synthesis device 100 transforms the item image M2 included in the model image M1 so that it corresponds from the model grid point M5 to the user grid point U5 (S9). Specifically, each model grid point and each user grid point correspond one-to-one, and the item image M2 in the area where the model grid point is set is transformed into the item image in the area where the user grid point is set so that the position of each model grid point transitions to the position of each user grid point. This generates an item image U2 (see FIG. 14) that has been transformed to fit the user's body shape.

[0064] Next, the item image combining device 100 combines the transformed item image U2 with the user image U1 (S10). Then, the combined image U6 (see FIG. 14) is displayed on the display of the user terminal 5 or the like.

[0065] On the other hand, if the item image synthesis device 100 determines in step S3 that the item is a second type based on the model image M1, it deforms the user image U1 so that the user's posture is the same as that of the model (S11). For example, with respect to the arm region, the user image U1 is deformed so that the arm angle obtained from the user skeleton information U4 is transformed to the arm angle obtained from the model skeleton information M4. This results in a deformed user image U7 (see FIG. 15).

[0066] Next, the item image combining device 100 combines the item image with the transformed user image U7 (S12). Then, a combined image U8 (see FIG. 15) is sent to the user terminal 5 and displayed on a display or the like.

[0067] (3) Features of the item image synthesis system As described above, the item image synthesis device 100 according to this embodiment transforms the item image for a first type of item by matching the model grid points with the user grid points, and synthesizes the transformed item image with the user image. This allows the item image synthesis device 100 to transform the item image so as to change it from the model's body shape to the user's body shape, and synthesizes the transformed item image with the user image, thereby outputting a natural-looking synthesized image that does not create a sense of incongruity. This allows the user to appear as if they are trying on the item when the item image is synthesized with the user image.

[0068] Specifically, by using the item image synthesizing device 100, an image can be synthesized that shows the user (trying on the clothing) trying on the clothing by overlaying an item image in which only the clothing is cut out from a model image. At this time, the item image is transformed according to the differences in body shape, height, and pose between the user (trying on the clothing) and the model, thereby enhancing the realism of the trying-on experience. In other words, by simply taking a photo of the model wearing the item (wearing the clothing) and cutting out the item (clothing), a synthesized image can be generated that shows the item as it is being tried on for any user.

[0069] Furthermore, by using the item image compositing device 100 according to this embodiment, if the item worn by the model is loose (such as a thick outer layer or pants or skirts with flared hems), the item image is composited so that it covers the user image, allowing the user to see the item as if they were trying it on. Additionally, if the user image is composited so that it protrudes from the item image, an unnatural image that would not occur in reality will result. In contrast, for the second type of item, the item image compositing device 100 according to this embodiment can composite the user image so that it does not protrude from the item image by transforming the user image based on the user's skeletal information and then compositing the item image. The item image compositing device 100 may also change the size of the item image based on the user's skeletal information. This allows the generation of a composite image with adjusted overall balance.

[0070] Note that if a user image and a model image in which the user and the model are photographed in similar poses are used, the amount of deformation of the item image can be reduced, thereby reducing the sense of incongruity in the composite image.

[0071] Furthermore, the item image synthesis device 100 generates a synthetic image based on a model image showing a model wearing an item, allowing the user to select an item while imagining how the item will be worn.

[0072] Furthermore, when the grid point spacing in the model image is equal to or greater than a predetermined value, the item image synthesis device 100 performs a predetermined enlargement process on the grid point spacing in the user image, thereby generating a synthesized image that does not create an unnatural appearance when the item image is synthesized with the user image. Furthermore, if the model's arm is thicker than the user's arm, and the above-described enlargement process is not performed, the item image will be shrunk to match the thickness of the user's arm. In this regard, the inventors have found that such shrinkage of the item image can create an unnatural appearance in the synthesized image, and that such shrinkage has a significant impact on the overall impression of the item, particularly in the hands and around the arms. Therefore, it is preferable to perform the above-described enlargement process when transforming the item image of the arm.

[0073] Furthermore, for the first type of item, the item image synthesis device 100 stores the model image and the model grid points in association with each other, and can therefore deform the item image to fit the user's shape based on the model grid points and the user grid points.

[0074] Furthermore, depending on the shape of the item, the item image synthesis device 100 stores model grid points set for the shape of the model when the model is not wearing the item. This allows for the generation of a natural-looking synthetic image even if the item worn by the model does not fit the model's body shape. Specifically, if the model grid points are set to cover the entire interior area of ​​the model shape rather than corresponding to the model's actual body shape, the item image will deform to fit the user's body shape. For example, when synthesizing an item image M2 with loose armpits as shown in FIG. 4 with a user image, an inappropriate synthetic image U9 is generated in which the item is not present in the user's armpits a1 and a2, which are actually part of the item (see FIG. 16(a)). In contrast, the item image synthesis device 100 sets the model grid points using model skeleton information. Therefore, if the model is wearing clothing that fits the model's body shape relatively loosely, the item image can be synthesized with the user image so that the item fits the user's body shape loosely (see FIG. 16(b)).

[0075] Furthermore, the item image synthesizing device 100 sets user grid points for at least a part of the user shape based on the user skeletal information, and can therefore deform the item image to fit the user shape based on the model grid points and the user grid points.

[0076] Furthermore, the item image synthesis device 100 stores the model information generated by the model information generation device 200. Specifically, for a first type item, the item image synthesis device 10 associates the model image with the model lattice points and stores them in the storage unit 11. The model information generation device 200 generates model information in which the model lattice points are set for the shape of the model when the model is not wearing the item. By using such model information, the item image synthesis device 100 can generate a natural-looking synthesized image even if the item worn by the model does not necessarily fit the model's body shape.

[0077] (4) Variations In the above description, the user image is a photograph of the user, but the user image may also be virtually generated. For example, a user image may be generated three-dimensionally based on the user's attribute data (height, weight, gender, etc.) and image data projected from a specific viewpoint may be used. In this case, the item image generation device further includes a user image generation unit that generates a user image based on at least one of the attribute data of height, weight, and gender.

[0078] Furthermore, in the above description, if a second type of item is shown as the item image, the annotated portion in the item image M2 may be included in the model information. For example, if a coat is shown as the item, the arm portion of the coat may be annotated and the annotated portion may be included in the model information. This allows, for example, the annotated portion to be stretched so that the cuffs of the coat shown in the item image M2 match the wrists of the user shown in the user image, and when the item image is superimposed on the user image, it appears as if the user is trying on the item.

[0079] Furthermore, in the above description, the items shown in the model image are divided into first-type items and second-type items, but these definitions are for convenience's sake. For example, even if an item is defined as the second-type item described above, it may be treated as a first-type item and the image synthesis process may be performed on it. Also, even if an item is defined as the first-type item described above, it may be treated as a second-type item and the image synthesis process may be performed on it. In short, the image synthesis processes for the first-type items and the second-type items may be combined as appropriate, or only one of them may be performed.

[0080] <Other embodiments> The present disclosure is not limited to the above-described embodiments as they are. The present disclosure can be embodied by modifying the components within the scope of the gist of the disclosure in the implementation stage. Furthermore, the present disclosure can be formed into various disclosures by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, the components may be appropriately combined in different embodiments. [Explanation of symbols]

[0081] 1. Item image synthesis system 5. User terminal 100 Item Image Synthesizer 110 Storage section 120 Input / output section 130 Communications Department 140 Processing section 141 Model information acquisition unit 142 User image acquisition unit 143 User grid point setting section 143A User shape extraction unit 143B User skeleton information acquisition unit 143C User grid point processing unit 144 Item Image Synthesis Unit 200 Model information generation device 241 Model Image Acquisition Unit 242 Item Image Extraction Unit 243 Model Shape Extraction Unit 244 Model skeleton information acquisition unit 245 Model Grid Point Setting Section 246 Model Information Generation Unit [Prior art documents] [Patent documents]

[0082] [Patent Document 1] Patent Publication No. 2019-128923

Claims

1. a model grid point in a model image showing a model wearing an item and a user grid point in a user image, and the item image showing the item is transformed by associating the model grid point in the model image showing the model wearing the item with the user image; and the transformed item image is combined with the user image; If the grid point interval in the model image is equal to or greater than a predetermined value, a predetermined enlargement process is performed on the grid point interval in the user image. Item image synthesizer.

2. the model grid points are set for at least a part of a model shape extracted from the model image based on model skeleton information acquired from the model image; The item image synthesizing device according to claim 1 .

3. The model grid points are set to a shape when the model is not wearing the item. The item image synthesizing device according to claim 2 .

4. a model information acquisition unit that acquires, in association with one another, a model image showing a model wearing an item, an item image corresponding to the item extracted from the model image, and model grid points at predetermined intervals set on at least a part of a model shape showing the shape of the model wearing the item; a user image acquisition unit that acquires a user image showing a user; a user grid point setting unit that sets user grid points at predetermined intervals on at least a part of a user shape that indicates the shape of the user shown in the user image; an item image synthesis unit that transforms the item image by making the model lattice points correspond to the user lattice points and synthesizes the transformed item image with the user image; The item image synthesizing device according to claim 1 , further comprising:

5. The user grid point setting unit a user shape extraction unit that extracts a user shape indicating a shape of the user from the user image; a user skeleton information acquisition unit that acquires user skeleton information for the user from the user image; a user grid point processing unit that sets user grid points for at least a part of the user shape based on the user skeleton information; The item image synthesizing device according to claim 4 , comprising:

6. further comprising a user image generating unit that generates the user image based on at least one of attribute data of height, weight, and sex; The item image synthesizing device according to any one of claims 1 to 5.

7. Acquire a model image showing a model wearing an item, an item image corresponding to the item extracted from the model image, and model grid points at predetermined intervals set on at least a part of a model shape showing the shape of the model wearing the item, in association with each other; Acquire a user image showing the user, setting user grid points at predetermined intervals on at least a part of a user shape that indicates the shape of the user shown in the user image; The item image is transformed by making the model grid points correspond to the user grid points, and the transformed item image is synthesized with the user image; If the grid point interval in the model image is equal to or greater than a predetermined value, a predetermined enlargement process is performed on the grid point interval in the user image. Item image synthesis method.

8. A program for causing a computer to execute the item image synthesis method according to claim 7.

9. a model image acquisition unit that acquires a model image showing a model wearing the item; an item image extraction unit that extracts an item image corresponding to the item from the model image; a model shape extraction unit that extracts a model shape representing a shape of a model wearing the item from the model image; a model skeleton information acquisition unit that acquires model skeleton information for the model from the model image; a model grid point setting unit that sets model grid points for at least a part of the model shape based on the model skeleton information; a model information generation unit that generates model information by associating the model image with model grid points; A model information generating device comprising:

10. the model grid point setting unit sets model grid points for a shape of the model when the model is not wearing the item; The model information generating device according to claim 9.

11. Obtain a model image showing a model wearing the item, extracting an item image corresponding to the item from the model image; extracting a model shape representing the shape of a model wearing the item from the model image; acquiring model skeleton information for the model from the model image; setting model grid points for at least a part of the model shape based on the model skeleton information; generating model information by associating the model image with the model grid points; Model information generation method.

12. a model information acquisition unit that acquires, in association with one another, a model image showing a model wearing an item, an item image corresponding to the item extracted from the model image, and model skeleton information for the model acquired from the model image; a user image acquisition unit that acquires a user image showing a user; a user skeleton information acquisition unit that acquires user skeleton information for the user from the user image; an item image synthesis unit that transforms the user image by correlating the model skeleton information with the user skeleton information and synthesizes the transformed user image with the item image; An item image synthesis device comprising:

13. Acquire a model image of a model wearing an item, an item image corresponding to the item extracted from the model image, and model skeleton information for the model acquired from the model image, in association with each other; Acquire a user image showing the user, acquiring user skeletal structure information for the user from the user image; transforming the user image by making the model skeleton information correspond to the user skeleton information, and synthesizing the transformed user image with the item image; Item image synthesis method.

Images