Virtual try-on system, virtual try-on method, information processing device, and storage medium

Abstract

The virtual fitting system (1) comprises a learning unit (12C), an acquisition unit (12A), an export unit (12D), and a generation unit (26H). The learning unit (12C) learns a learning model by machine learning using three-dimensional data of a fitting person, the learning model taking a teaching fitting person image as input data and taking body shape parameters representing the body shape of the fitting person presented by the teaching fitting person image and synthesis position information of a clothing image in the teaching fitting person image as output data. The acquisition unit (26G) acquires the fitting person image. The export unit (12D) uses the fitting person image and the learning model to export output data of the fitting person presented by the fitting person image. The generation unit (26H) uses the exported output data to generate a virtual fitting image by synthesizing the fitting person image and the clothing image.

Description

Technical Field

[0001] Embodiments of the present invention relate to a virtual try-on system, a virtual try-on method, a virtual try-on program, an information processing device, and learning data. Background Art

[0002] There is a technology for displaying a virtual image showing a state where a user is wearing clothes to be tried on. For example, there is a technology for displaying a synthetic image showing a state where a user is trying on clothes. Summary of the invention

[0003] However, in the past, when displaying a virtual try-on image, information related to the three-dimensional shape of the person trying on the clothes is obtained, and deformation and alignment are performed based on the obtained information related to the three-dimensional shape, thereby synthesizing the clothing image and the person trying on the clothes. Therefore, it has been difficult to provide a highly accurately synthesized virtual try-on image with a simple structure.

[0004] The problem to be solved by the present invention is to provide a virtual try-on system, a virtual try-on method, a virtual try-on program, an information processing device, and learning data that can easily provide a virtual try-on image synthesized with high accuracy.

[0005] Means for solving problems

[0006] The virtual try-on system of the embodiment comprises a learning unit, an acquisition unit, a derivation unit, and a generation unit. The learning unit learns a learning model using a teaching try-on person image as input data, body shape parameters representing the body shape of the try-on person presented by the teaching try-on person image, and synthesis position information of a clothing image in the teaching try-on person image as output data by machine learning using the three-dimensional data of the try-on person. The acquisition unit acquires the try-on person image. The derivation unit uses the try-on person image and the learning model to derive the output data of the try-on person presented by the try-on person image. The generation unit uses the derived output data to generate a virtual try-on image synthesized by the try-on person image and the clothing image.

[0007] According to the virtual try-on system having the above configuration, it is possible to easily provide a virtual try-on image that is aligned with high accuracy and corrected to fit the body shape with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the virtual try-on system.

[0009] Figure 2 It is a schematic diagram showing the positional relationship between the main body and the person trying on the wearer.

[0010] Figure 3 This is a functional block diagram of an information processing device and a terminal device.

[0011] Figure 4 It is a schematic diagram showing an example of the data structure of learning data.

[0012] Figure 5 This is an explanatory diagram of an example of synthesized position information.

[0013] Figure 6 : is a schematic diagram showing an example of a virtual try-on image.

[0014] Figure 7 This is a flowchart showing an example of the procedure of information processing.

[0015] Figure 8 This is a flowchart showing an example of the procedure of information processing.

[0016] Fig. 9 This is a block diagram showing a hardware configuration example of an information processing device, a terminal device, and an external server. DETAILED DESCRIPTION

[0017] Hereinafter, one embodiment of a virtual try-on system, a virtual try-on method, a virtual try-on program, an information processing device, and learning data will be described in detail with reference to the accompanying drawings.

[0018] Figure 1 Schematic diagram of the virtual try-on system 1 according to the present embodiment.

[0019] The virtual try-on system 1 includes an information processing device 10, a terminal device 26, and an external server 28. The information processing device 10, the terminal device 26, and the external server 28 are connected via a communication line 34. The communication line 34 is a well-known network. The information processing device 10, the terminal device 26, and the external server 28 are connected to the communication line 34 by wire or wirelessly.

[0020] In addition, in the present embodiment, the virtual try-on system 1 is described as an example in which it includes one information processing device 10, one external server 28, and one terminal device 26. However, the virtual try-on system 1 may be configured to include one or more information processing devices 10, one or more external servers 28, and one or more terminal devices 26.

[0021] The terminal device 26 is a device that displays a virtual try-on image. The virtual try-on image is a composite image in which a clothing image is composited onto a try-on person image 50 of a try-on person. In other words, the virtual try-on image is an image showing a state in which a try-on person virtually tries on clothing. In this embodiment, the terminal device 26 generates and displays a virtual try-on image using a learning model generated in the information processing device 10. Details of the learning model will be described later.

[0022] The user who performs the virtual fitting operation operates the terminal device 26. The terminal device 26 is, for example, a known personal computer, a portable mobile terminal, a smart phone, or the like.

[0023] The tryer is the object of trying on the clothes. The tryer can be a living being or a non-living being as long as it is the object of trying on the clothes. An example of a living being is a person. In addition, the living being is not limited to a person, and can also be an animal other than a person, such as a dog or a cat. Furthermore, an inanimate object can include a model or other object that imitates the shape of a human body or an animal, but is not limited thereto. Furthermore, the tryer can also be a living being or a non-living being in a state of wearing clothes. In this embodiment, the case where the tryer is a person is described.

[0024] In addition, in the present embodiment, when describing the try-on person and persons other than the try-on person as a general term, they will be described simply as users.

[0025] Clothing refers to items that a person who is trying on the clothing can wear. Examples of clothing include tops, skirts, pants, shoes, hats, accessories, swimsuits, wigs, etc. In addition, clothing is not limited to tops, skirts, pants, shoes, hats, accessories, swimsuits, wigs, etc.

[0026] The terminal device 26 includes a photographing unit 26A, an input unit 26B, and a display unit 26C. The photographing unit 26A photographs the person trying on the clothes and obtains a person trying on image 51 of the person trying on the clothes. The person trying on image 51 is an example of the person trying on image 50. The person trying on image 51 shows the person trying on image 50 photographed by the terminal device 26.

[0027] The imaging unit 26A is a well-known digital camera. Specifically, the try-on user image 51 (try-on user image 50 ) is try-on user image data. In the present embodiment, for simplicity of description, the try-on user image 51 (try-on user image 50 ) will be referred to for description.

[0028] The data format of the try-on person image 50 is not limited. For example, the try-on person image 50 is an image in which a pixel value indicating the color or brightness of the try-on person is defined for each pixel. The try-on person image 50 is not limited to a bitmap image.

[0029] The display unit 26C displays various images. In the present embodiment, the display unit 26C displays a virtual try-on image. The display unit 26C is a well-known LCD (Liquid Crystal Display) or organic EL (Electro-Luminescence) or the like.

[0030] The input unit 26B receives operation instructions from the user. The input unit 26B is a keyboard, a mouse, an input button, etc. In addition, the display unit 26C and the input unit 26B may be integrally formed. In this case, the display unit 26C and the input unit 26B may be formed as a touch panel.

[0031] The external server 28 is a server device capable of processing clothing images and big data. In the present embodiment, the external server 28 pre-associates and stores clothing images used for virtual try-on images and clothing IDs used for identifying clothing images. In the external server 28, a plurality of clothing images are pre-associated with their respective clothing IDs. Furthermore, the external server 28 analyzes the purchase information of users and the like accumulated in various server devices on the Internet by processing big data.

[0032] The information processing device 10 is a device for generating a learning model. The learning model is an application for deriving body parameters and synthetic position information of a try-on person represented by the try-on person image from the try-on person image. The details of the learning model will be described later. The information processing device 10 can be accessed from one or more terminal devices 26, and provides the learned learning model or output data derived from the learning model.

[0033] The information processing device 10 includes an information processing unit 11 and a main body 16. The information processing unit 11 includes a control unit 12 and a storage unit 14. The control unit 12 controls each unit provided in the information processing device 10. The storage unit 14 is a known hard disk device that stores various data.

[0034] The main body 16 includes a display unit 18 , an imaging unit 20 , a three-dimensional data acquisition unit 21 , and an input unit 22 .

[0035] The input unit 22 receives various operation instructions from the user. The input unit 22 is, for example, a keyboard, a touch panel, a pointing device, a mouse, or the like.

[0036] The imaging unit 20 captures the image of the person trying on the clothing and obtains a person trying on the clothing image 50 of the person trying on the clothing. In addition, the person trying on the clothing image 50 captured by the imaging unit 20 provided in the information processing device 10 will be referred to as a teaching person trying on the clothing image 40 for explanation.

[0037] The teaching try-on person image 40 is a try-on person image 50 used for generating a learning model. The imaging unit 20 is a known imaging device capable of acquiring the teaching try-on person image 40. The imaging unit 20 is, for example, a digital camera.

[0038] In addition, Figure 1As an example, one imaging unit 20 is shown. However, the information processing device 10 may also be configured to include a plurality of imaging units 20. In this case, the plurality of imaging units 20 may be disposed at positions where the try-on person can be photographed from different photographing directions. That is, each of the plurality of imaging units 20 may be disposed at a position where the teaching try-on person images 40 can be photographed at different photographing angles.

[0039] Furthermore, the information processing device 10 may be configured to include a plurality of different types of imaging units 20. Different types of imaging units 20 means that at least one of the angle of view, lens (telephoto lens, wide-angle lens, short-focus lens, zoom lens, fisheye lens, etc.), imaging magnification, resolution, and manufacturer of the imaging unit 20 is different.

[0040] The three-dimensional data acquisition unit 21 is a device for acquiring three-dimensional data of the wearer. The three-dimensional data includes, for example, 3D (three-dimensional) feature data, surface data, entity data, 3D CAD (computer-aided design) data, depth map, and the like.

[0041] The three-dimensional data acquisition unit 21 may be a known device capable of acquiring the three-dimensional data. For example, when the three-dimensional data is a depth map, the three-dimensional data acquisition unit 21 is a photographing device that acquires the depth map by photographing. The depth map is sometimes referred to as a distance image. The depth map is an image that specifies the distance from the three-dimensional data acquisition unit 21 for each pixel.

[0042] In this embodiment, the photographing unit 20 and the three-dimensional data acquisition unit 21 photograph the try-on person at the same time. The photographing unit 20 and the three-dimensional data acquisition unit 21 are controlled by the control unit 12 to synchronously and sequentially photograph at the same time. Furthermore, the photographing unit 20 and the three-dimensional data acquisition unit 21 sequentially output the teaching try-on person image 40 and the three-dimensional data of the try-on person obtained by photographing to the control unit 12.

[0043] In this embodiment, the imaging unit 20 and the three-dimensional data acquisition unit 21 capture the wearer from multiple imaging angles. Therefore, the imaging unit 20 obtains the teaching wearer image 40 corresponding to each of the multiple imaging angles. Furthermore, the three-dimensional data acquisition unit 21 obtains the three-dimensional data corresponding to each of the multiple imaging angles.

[0044] The multiple shooting angles can be any shooting angles as long as they are different from each other. For example, the multiple shooting angles are shooting angles at each predetermined angle when the torso of the try-on wearer is rotated 360° with the torso of the try-on wearer as the rotation axis. Specifically, it can be shooting angles that are obtained by adding a predetermined angle (for example, 5°) each time from 0° until 360° is reached. For example, a shooting angle of 0° represents a shooting angle in a state where the try-on wearer is facing the display unit 18 with his front face. And, for example, a shooting angle of 180° represents a shooting angle in a state where the try-on wearer is facing the display unit 18 with his back facing the display unit 18.

[0045] Specifically, the imaging unit 20 and the three-dimensional data acquisition unit 21 are arranged at various positions of the try-on person that can be photographed from various different multiple photographing directions and face the display surface of the display unit 18. Furthermore, the imaging unit 20 and the three-dimensional data acquisition unit 21 can obtain the teaching try-on person image 40 and the three-dimensional data corresponding to each of the multiple photographing angles by obtaining the photographed images.

[0046] Furthermore, the control unit 12 may control the output of a message urging the try-on user to strike a predetermined pose when the try-on user faces the display surface of the display unit 18. For example, the control unit 12 may control the output of a message urging the try-on user to strike a pose so that an image is captured by the try-on user holding a camera or other imaging unit (so-called Selfie).

[0047] For example, the control unit 12 displays the message on the display unit 18. Furthermore, for example, the main body 16 may further include a speaker, and the control unit 12 may output the message from the speaker.

[0048] In this case, the imaging unit 20 can capture the image of the tryer in the same posture as that in the self-photographing, and obtain the teaching tryer image 40 of the tryer in the posture.

[0049] Furthermore, the imaging unit 20 and the three-dimensional data acquisition unit 21 can acquire a plurality of teaching try-on images 40 and three-dimensional data having different imaging times by continuously performing imaging.

[0050] The display unit 18 is a device that displays various images. The display unit 18 is, for example, a known display device such as a liquid crystal display device or an organic EL (Electro Luminescence). In the present embodiment, the display unit 18 displays the teaching try-on image 40 captured by the imaging unit 20. Therefore, the try-on person facing the display unit 18 can visually confirm the display unit 18 in the same way as visually confirming through a mirror. In addition, the display unit 18 can also display the overlapping image (described later in detail) generated by the control unit 12 and described later.

[0051] The display unit 18 is assembled on one surface of a rectangular housing, for example. In this embodiment, the display unit 18 is configured to be larger than the size of a person's body. In addition, the size of the display unit 18 is not limited to this size.

[0052] Figure 2 1 is a schematic diagram showing the positional relationship between the main body 16 and the person P trying on the clothing in this embodiment.

[0053] Control unit 12 (in Figure 2 (not shown in the figure) the teaching try-on image 40 of the try-on person P, the superimposed image W showing the state of the try-on person P trying on various clothes, etc. are displayed on the display unit 18. Figure 2 , as an example, an overlapping image W of the teaching try-on image 40 and the clothing image 42 is shown. The try-on person P, such as a person, visually checks the overlapping image W2 presented on the display unit 18, for example, from a position facing the display surface of the display unit 18. In addition, the shooting position and shooting direction of the shooting unit 20 and the three-dimensional data acquisition unit 21 are adjusted in advance so that the try-on person P facing the display surface of the display unit 18 can be shot.

[0054] return Figure 1 Next, the functions of the information processing device 10 and the terminal device 26 included in the virtual try-on system 1 will be described. Figure 3 This is an example of a functional block diagram of the information processing device 10 and the terminal device 26 included in the virtual try-on system 1 .

[0055] First, the information processing device 10 is described. The information processing device 10 includes a control unit 12, an imaging unit 20, a three-dimensional data acquisition unit 21, a storage unit 14, a display unit 18, and an input unit 22. The imaging unit 20, the three-dimensional data acquisition unit 21, the storage unit 14, the display unit 18, and the input unit 22 are connected to the control unit 12 in a manner capable of transmitting and receiving data and signals.

[0056] The storage unit 14 stores various data. In the present embodiment, the storage unit 14 stores learning data 14A, clothing DB 14B, and three-dimensional data DB 14C. The details of the learning data 14A will be described later.

[0057] The clothing DB 14B establishes a correspondence between the clothing ID and the clothing image. The clothing ID is identification information for identifying the corresponding clothing image. In addition, the clothing DB 14B may also be stored in the external server 28. In the present embodiment, a case where a plurality of clothing IDs and clothing images corresponding to each of the plurality of clothing IDs are associated with each other is pre-stored in the storage unit 14 as an example for explanation. In addition, the clothing DB 14B stored in the information processing device 10 may also be appropriately updated with the help of the clothing DB 14B stored in the external server 28.

[0058] The three-dimensional data DB 14C establishes a correspondence between the three-dimensional data of the try-on person and the teaching try-on person image ID. The teaching try-on person image ID is identification information for identifying the teaching try-on person image 40. That is, in the present embodiment, the storage unit 14 stores the three-dimensional data of the try-on person presented in the teaching try-on person image 40 in correspondence with each teaching try-on person image 40. The three-dimensional data DB 14C is updated by the control unit 12.

[0059] The control unit 12 includes an acquisition unit 12A, a communication unit 12B, a learning unit 12C, and a derivation unit 12D. The learning unit 12C includes a model learning unit 12E, a correction unit 12F, an extended image generation unit 12G, a display control unit 12H, a reception unit 12I, and a change unit 12J.

[0060] Part or all of the acquisition unit 12A, the communication unit 12B, the learning unit 12C, the export unit 12D, the model learning unit 12E, the correction unit 12F, the extended image generation unit 12G, the display control unit 12H, the acceptance unit 12I, and the change unit 12J may be implemented by software, for example, by making a processing device such as a CPU (Central Processing Unit) execute a program, or by hardware such as an IC (Integrated Circuit), or by using both software and hardware.

[0061] The communication unit 12B communicates with the external server 28 and the terminal device 26 via the communication line 34 .

[0062] The acquisition unit 12A acquires the teaching try-on person image 40. In the present embodiment, the acquisition unit 12A acquires the teaching try-on person image 40 from the shooting unit 20. In addition, as described above, the shooting unit 20 sometimes acquires a plurality of teaching try-on person images 40 with at least one different shooting time, shooting angle, type of shooting unit 20 used in shooting, and posture of the try-on person during shooting for one try-on person. In this case, the acquisition unit 12A acquires the above-mentioned plurality of types of teaching try-on person images 40 for one try-on person. Furthermore, the acquisition unit 12A may also acquire the teaching try-on person images 40 for each of the plurality of try-ons. Furthermore, the acquisition unit 12A may also acquire a single teaching try-on person image 40 including the plurality of try-ons.

[0063] In addition, the acquisition unit 12A may acquire the teaching try-on person image 40 from the storage unit 14 or the external server 28. In the present embodiment, a case where the acquisition unit 12A acquires the teaching try-on person image 40 from the imaging unit 20 will be described as an example.

[0064] Then, the acquisition unit 12A acquires the three-dimensional data of the person trying on the clothing. In the present embodiment, the acquisition unit 12A acquires the three-dimensional data of the person trying on the clothing represented by the teaching person trying on the clothing image 40 from the three-dimensional data acquisition unit 21 .

[0065] Then, the acquisition unit 12A acquires the characteristic information of the person trying on the clothing input via the input unit 22 .

[0066] The feature information includes, for example, the appearance features and intrinsic features of the wearer. The appearance features are features estimated or analyzed from the appearance of the wearer. The intrinsic features are features that cannot be estimated or analyzed from the appearance of the wearer, or that are difficult to estimate or analyze.

[0067] Specifically, the characteristic information indicates at least one of the skin color, exposure from clothing, hairstyle, nationality, dominant hand, personality, recommended shooting direction, eye color, hair color, favorite color, gender, age, and recommended posture of the person trying on the clothes. The recommended shooting direction indicates the shooting direction preferred by the person trying on the clothes. The exposure from clothing indicates the position and ratio of the area exposed from clothing (i.e., the skin area exposed to the outside) in the captured subject image. The recommended posture indicates the preferred posture (favorite posture or action) of the person trying on the clothes when shooting.

[0068] For example, the tryer P inputs the feature information of the tryer P by operating the input unit 22 when the teaching tryer image 40 is captured by the imaging unit 20 . Thus, the acquisition unit 12A acquires the feature information of the tryer P represented by the teaching tryer image 40 .

[0069] Furthermore, the acquisition unit 12A acquires the try-on person image 51 captured by the terminal device 26 from the terminal device 26 via the communication unit 12B and the communication line 34. Furthermore, the acquisition unit 12A acquires display result information indicating the display result of the virtual try-on image displayed on the terminal device 26 from the terminal device 26 via the communication unit 12B and the communication line 34. The display result information will be described in detail later.

[0070] The learning unit 12C learns the learning model. As described above, the learning model is an application for deriving the body shape parameters and synthetic position information of the try-on user presented by the try-on user image 50 from the try-on user image 50. In the present embodiment, the learning model is a learning model that takes the teaching try-on user image 40 as input data, and takes the body shape parameters representing the body shape of the try-on user presented by the teaching try-on user image 40 and the synthetic position information of the clothing image in the teaching try-on user image 40 as output data. In other words, the learning model is an application for modeling and calculating the input-output relationship between the teaching try-on user image 40 and the body shape parameters and the synthetic position information. In addition, the learning model can also be represented by a mathematical expression such as a function.

[0071] In the present embodiment, the learning unit 12C learns the learning model by machine learning using the three-dimensional data of the try-on person. Specifically, the learning unit 12C learns the learning model by machine learning using the learning data.

[0072] Figure 4 1 is a schematic diagram showing an example of the data structure of the learning data 14A.

[0073] A plurality of teaching data 60 are registered in the learning data 14A. In addition, in the learning data 14A, the learning unit 12C processes the registration of new teaching data 60, the change or correction of the registered teaching data 60, and the like. Therefore, a plurality of types of teaching data 60 (for example, teaching data 60A to teaching data 60E) are registered in the learning data 14A.

[0074] The teaching data 60 includes input data 80 and output data 82 corresponding to the input data 80. The input data 80 is data that is a factor in predicting or determining the corresponding output data 82. The output data 82 is data indicating a response derived from the corresponding input data 80.

[0075] Regarding the teaching data 60 , the type of data included in the teaching data 60 or the output data 82 differs depending on the type of the teaching data 60 .

[0076] First, the teaching data 60A will be described. The input data 80 of the teaching data 60A includes the teaching try-on user image 40. In addition, one input data 80, that is, one teaching try-on user image 40 is included in one teaching data 60A.

[0077] Furthermore, the input data 80 may include feature information 70. The feature information 70 is information indicating the features of the person trying on the clothing. The feature information 70 included in the input data 80 is information indicating the features of the person trying on the clothing presented in the teaching person trying on image 40 included in the input data 80.

[0078] The output data 82 includes the body shape parameters 72 and the synthesized position information 56 .

[0079] The body shape parameter 72 is a parameter indicating the body shape of the try-on user presented in the teaching try-on user image 40 included in the corresponding input data 80 .

[0080] Body shape parameters include one or more parameters. Parameters are measured values ​​of one or more parts of the human body. In addition, the measured values ​​are not limited to the actual measured values, but also include values ​​estimated from the measured values ​​or values ​​equivalent to the measured values.

[0081] Specifically, the body shape parameters include at least one parameter of chest circumference, abdominal circumference, waist circumference, height, and shoulder width. In addition, the parameters included in the body shape parameters are not limited to the above parameters. For example, the body shape parameters may also include parameters such as sleeve length and inseam.

[0082] Furthermore, the body shape parameter may be information indicating the body shape of the person trying on the garment, or may be information indicating the size of the garment when the person trying on the garment is assumed to be wearing the garment of a size that matches the body shape. In this case, the body shape parameter may also include parameters such as garment length, shoulder width, garment width, sleeve length, etc. In this embodiment, the case of using garment length, shoulder width, garment width, and sleeve length as body shape parameters is described. However, the body shape parameter is not limited to this.

[0083] The synthesis position information 56 is information indicating the synthesis position of the clothing image in the teaching try-on person image 40 included in the corresponding input data 80. The synthesis position is used for alignment when synthesizing the clothing image with respect to the region representing the try-on person included in the teaching try-on person image 40 (hereinafter referred to as the try-on person region 40A).

[0084] The synthesis position information 56 may be, for example, information indicating the position of a specific part (e.g., shoulder) of the try-on person, or information indicating a synthesis target region of the clothing image in the teaching try-on person image 40. In the present embodiment, the synthesis position information is described as information indicating a synthesis target region of the clothing image in the teaching try-on person image 40.

[0085] Figure 5 : is an explanatory diagram of an example of the synthesis position information 56. For example, the synthesis position information 56 is information indicating a rectangular synthesis target area surrounding the try-on person area 40A representing the try-on person in the teaching try-on person image 40. In addition, the synthesis position information 56 may also be information indicating the positions of the vertices (56A, 56B, 56C, 56D) of the rectangle surrounding the try-on person area 40A. Furthermore, the synthesis position information 56 may also be information indicating the position of at least one vertex among the above four vertices.

[0086] The synthesis position information 56 is not limited to information indicating a rectangular synthesis target region. For example, the synthesis position information 56 may be information indicating at least a part of the outer shape of the try-on user region 40A included in the teaching try-on user image 40 .

[0087] Furthermore, the composite position information 56 may be information indicating a specific part (eg, shoulder) of the try-on person in the teaching try-on person image 40 .

[0088] return Figure 4 The description will be continued. In addition, the output data 82 may further include the characteristic information 70 . That is, at least one of the input data 80 and the output data 82 may further include the characteristic information 70 .

[0089] The characteristic information 70 included in the output data 82 is characteristic information of the person trying on the clothing represented by the teaching person trying on image 40 included in the input data 80 corresponding to the output data 82 .

[0090] The teaching data 60A to 60E will be described later. The plurality of types of teaching data 60 (teaching data 60A to 60E) are appropriately registered and updated by the processing of the learning unit 12C (details will be described later).

[0091] return Figure 3 The description will be continued. The learning unit 12C includes a model learning unit 12E, a correction unit 12F, an extended image generation unit 12G, a display control unit 12H, a reception unit 12I, and a change unit 12J.

[0092] The model learning unit 12E generates and updates the learning data 14A and generates a learning model.

[0093] First, the generation of the learning data 14A by the model learning unit 12E is described. The model learning unit 12E acquires the teaching try-on person image 40, the three-dimensional data of the try-on person represented by the teaching try-on person image 40, and the characteristic information 70 of the try-on person from the acquisition unit 12A. Furthermore, the model learning unit 12E uses the acquired three-dimensional data to derive output data 82 for the input data 80 including the teaching try-on person image 40 and the characteristic information 70.

[0094] For example, the model learning unit 12E prepares in advance a function or a model for deriving body shape parameters represented by the three-dimensional data from the three-dimensional data. A known method may be used to prepare the function or the model for deriving body shape parameters from the three-dimensional data.

[0095] Furthermore, the model learning unit 12E derives the body shape parameters 72 by introducing the three-dimensional data acquired by the acquisition unit 12A into the function or the model.

[0096] For example, the model learning unit 12E derives a rectangular area surrounding a try-on user area 40A representing a try-on user in the teaching try-on user image 40 as the synthesis position information 56 (see also FIG. 5 ). Figure 4 ). Then, the model learning unit 12E obtains the feature information 70 of the person trying on the clothing obtained by the obtaining unit 12A.

[0097] Furthermore, the model learning unit 12E generates the derived body shape parameters 72 , the synthesized position information 56 , and the feature information 70 as output data 82 .

[0098] In addition, the model learning unit 12E registers the teaching data 60A in the learning data 14A, and the teaching data 60A includes input data 80 including the teaching try-on image 40 and feature information 70 obtained from the acquisition unit 12A, and output data 82 (body shape parameters 72, synthetic position information 56, feature information 70) generated using the input data 80 and three-dimensional data.

[0099] In the model learning unit 12E, the acquisition unit 12A generates the teaching data 60A using the above method each time the acquisition unit 12A acquires the teaching try-on person image 40, the three-dimensional data, and the characteristic information 70. In other words, the model learning unit 12E generates the teaching data 60A each time the imaging unit 20 and the three-dimensional data acquisition unit 21 acquire the teaching try-on person image 40 and the three-dimensional data by imaging. Furthermore, the model learning unit 12E registers the generated teaching data 60A in the learning data 14A.

[0100] Therefore, the teaching data 60A is generated for each teaching try-on person image 40 and is registered in the learning data 14A of the storage unit 14 .

[0101] Next, the generation of the learning model is described. The model learning unit 12E learns the learning model by machine learning using the learning data 14A including the plurality of teaching data 60. Machine learning can use a known method. For example, the model learning unit 12E learns the learning model by deep learning using an algorithm such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0102] In addition, the learning unit 12C may also modify or change the teaching data 60 registered in the learning data 14A. In this case, the model learning unit 12E may learn the learning model using the learning data 14A including the teaching data 60 before modification (for example, the teaching data 60A) and the modified or changed teaching data 60. Furthermore, the learning unit 12C may also learn the learning model using the modified or changed teaching data 60.

[0103] In the present embodiment, the correction unit 12F of the learning unit 12C corrects the output data 82 corresponding to the input data 80 based on the feature information 70 included in the input data 80 of the teaching data 60A.

[0104] For example, regarding the correction unit 12F, the higher the exposure from clothing indicated by the characteristic information 70, the correction unit 12F corrects the body shape parameter 72 in the output data 82 corresponding to the input data 80 including the characteristic information 70 to indicate a body shape of a smaller size. At this time, the correction unit 12F may be configured so that the higher the exposure from clothing of a part of the human body indicated by the characteristic information 70 (for example, the back), the body shape parameter 72 is corrected to indicate a body shape of a smaller size.

[0105] And, for example, the correction unit 12F identifies the amount of hair based on the hairstyle indicated by the feature information 70. Furthermore, the correction unit 12F corrects the body shape parameter 72 in the output data 82 corresponding to the input data 80 including the feature information 70 to indicate a smaller body shape as the amount of hair increases. At this time, the correction unit 12F may correct the body shape parameter 72 so that the head of the human body indicated by the feature information 70 appears smaller.

[0106] Furthermore, for example, the correction unit 12F may correct the output data 82 according to the nationality indicated by the characteristic information 70. In this case, for example, the correction unit 12F corrects the body shape parameter 72 in the output data 82 corresponding to the input data 80 including the characteristic information 70 to indicate a body shape of a smaller size, as the nationality indicated by the characteristic information 70 indicates the nationality of a citizen who wears a larger amount of clothing or a larger number of layers of clothing.

[0107] Furthermore, for example, the correction unit 12F may correct the output data 82 according to the gender indicated by the feature information 70. In this case, for example, when the gender indicated by the feature information 70 is female, the correction unit 12F corrects the body shape parameter 72 in the output data 82 corresponding to the input data 80 including the feature information 70 to indicate a smaller body shape. Furthermore, for example, when the gender indicated by the feature information 70 is male, the correction unit 12F corrects the body shape parameter 72 in the output data 82 corresponding to the input data 80 including the feature information 70 to indicate a larger body shape.

[0108] In this way, the correction unit 12F of the learning unit 12C can correct the output data 82 of the teaching wearer image 40 acquired from the imaging unit 20 and the three-dimensional data acquisition unit 21 or the teaching data 60A created using the three-dimensional data.

[0109] Furthermore, the learning unit 12C may generate an expanded image by expanding the teaching try-on person image 40 included in the teaching data 60 registered in the learning data 14A.

[0110] In this case, the extended image generating unit 12G of the learning unit 12C generates an extended image. The extended image is an image generated by changing at least a part of the teaching try-on person image 40 registered in the learning data 14A. Through the processing performed by the extended image generating unit 12G, a plurality of extended images similar to the teaching try-on person image 40 are generated from one teaching try-on person image 40.

[0111] The extended image is, for example, at least one of a clothing overlapping image, a masked image, a cut-out image, a processed image, and a changed image.

[0112] The clothing overlapped image is an image in which the clothing image 42 is overlapped on the try-on person region 40A included in the teaching try-on person image 40. For example, the extended image generating unit 12G generates an extended image of the teaching try-on person image 40 by generating a clothing overlapped image in which a predetermined clothing image 42 is overlapped on the teaching try-on person image 40 registered in the learning data 14A. The clothing overlapped image may be generated using a known image synthesis method.

[0113] The mask image is an image in which a part of the teaching try-on person image 40 is masked. For example, the extended image generating unit 12G generates an extended image of the teaching try-on person image 40 by generating a mask image in which a part of the teaching try-on person image 40 registered in the learning data 14A is changed to a predetermined color such as black or white. The mask image may be generated using a known image processing method.

[0114] The cut-out image is an image obtained by cutting out a part of the teaching try-on person image 40. For example, the extended image generating unit 12G generates an extended image of the teaching try-on person image 40 by generating a cut-out image obtained by cutting out a part of the teaching try-on person image 40 registered in the learning data 14A. The cut-out image may be generated using a known image processing method.

[0115] The processed image is an image processed from at least a part of the teaching try-on image 40. The processing may be performed using a known processing technique for processing into a painting style such as a sketching style, a crayon style, a painting style such as an oil painting or a Japanese painting, a line drawing or a sketching style, etc. The processed image may be generated using a known image processing method.

[0116] The changed image is an image obtained by changing at least a part of the teaching try-on person image 40 to another image. For example, the changed image is an image obtained by changing the background area, which is an area other than the try-on person area 40A included in the teaching try-on person image 40, to another image. The generation of the changed image may be performed using a known image processing method.

[0117] Furthermore, the extended image generating unit 12G registers the generated extended image as new teaching data 60 in the learning data 14A. Specifically, the extended image generating unit 12G determines the teaching data 60A including the teaching try-on person image 40 used in the generation of the extended image. Furthermore, the extended image generating unit 12G uses the characteristic information 70 included in the determined teaching data 60A and the generated extended image as new input data 80. Furthermore, the extended image generating unit 12G uses the output data 82 included in the determined teaching data 60A as output data 82 corresponding to the new input data 80. Furthermore, the extended image generating unit 12G updates the learning data 14A by registering the teaching data 60 including the new input data 80 and the output data 82 in the learning data 14A.

[0118] Therefore, if Figure 4 As shown, in the learning data 14A, teaching data 60C including the expanded image 52 obtained by expanding the teaching try-on person image 40 as the input data 80 is registered as new teaching data 60 .

[0119] In this case, the model learning unit 12E can learn the learning model using a plurality of types of teaching data 60 (including the teaching data 60A and the teaching data 60C) registered in the learning data 14A.

[0120] return Figure 3 The description will be continued. Furthermore, the learning unit 12C can change the output data 82 registered in the learning data 14A according to a change instruction of the output data 82 inputted by an operation instruction of the input unit 22 by the user.

[0121] In this case, the display control unit 12H of the learning unit 12C causes the display unit 18 to display an overlapping image W, in which the clothing image 42 deformed according to the body shape parameters 72 included in the output data 82 is overlapped at a composite position on the teaching try-on person image 40 included in the teaching data 60A, and is represented by the composite position information 56 included in the output data 82 corresponding to the input data 80 including the teaching try-on person image 40.

[0122] The clothing image 42 deformed according to the body shape parameter 72 is a process of enlarging, reducing, or deforming a pre-stored clothing image 42 so as to present the size of the body shape indicated by the body shape parameter 72. The deformation process of the clothing image 42 may be performed using a known technique.

[0123] Therefore, for example, the display unit 18 displays Figure 2 The user refers to the superimposed image W displayed on the display unit 18 and inputs a change instruction for at least one of the composite position information 56 and the body shape parameter 72. The change instruction includes, for example, information after the change of at least one of the position, size, and range of the composite position information 56, or information after the change of the body shape parameter 72. Then, the accepting unit 12I of the learning unit 12C accepts the change instruction from the input unit 22 via the acquiring unit 12A.

[0124] return Figure 3 The changing unit 12J of the learning unit 12C changes the output data 82 corresponding to the input data 80 including the teaching try-on image 40 used in the superimposed image W displayed on the display unit 18 most recently in the teaching data 60A to the changed information indicated by the change instruction accepted by the accepting unit 12I.

[0125] In this case, the model learning unit 12E can learn the learning model using a plurality of types of teaching data 60 (including the teaching data 60A, the modified teaching data 60A, and the teaching data 60C) registered in the learning data 14A.

[0126] The user who operates the input unit 22 can input a change instruction for changing the output data 82 to a negative example. In this case, the accepting unit 12I of the learning unit 12C accepts the change instruction indicating the negative example information from the input unit 22 via the acquiring unit 12A.

[0127] Then, the changing unit 12J of the learning unit 12C changes the output data 82 corresponding to the input data 80 including the teaching try-on person image 40 in the learning data 14A to the output data 82 indicating a negative example (see Figure 4 In the teaching data 60B in the example). When the output data 82 represents a negative example, the answer derived from the corresponding input data 80 becomes a negative example. In this way, the change unit 12J can change the output data 82 to a negative example according to the operation instruction of the input unit 22 performed by the user.

[0128] In this case, the model learning unit 12E can learn the learning model using the teaching data 60 (including the teaching data 60A, the modified teaching data 60A, the teaching data 60B, and the teaching data 60C) registered in the learning data 14A.

[0129] As described above, the terminal device 26 generates and displays a virtual try-on image using a learning model generated by the information processing device 10. Furthermore, as described above, the acquisition unit 12A of the information processing device 10 sometimes acquires display result information indicating a display result of a virtual try-on image displayed on the terminal device 26 from the terminal device 26.

[0130] In this case, the learning unit 12C of the information processing device 10 may register the new teaching data 60 in the learning data 14A using the display result information acquired from the terminal device 26 .

[0131] The display result information is information indicating the display result of the virtual try-on image on the terminal device 26. In the present embodiment, the display result information indicates the first display result information or the second display result information.

[0132] The first display result information includes positive example information indicating a positive example, a try-on person image 51, and output data 82. The positive example information included in the first display result information is information indicating that the output data 82 included in the first display result information is a positive example. Furthermore, the try-on person image 51 included in the first display result information is a try-on person image 50 used for a virtual try-on image synthesized and displayed in the terminal device 26 using the output data 82 included in the first display result information.

[0133] That is, the first display result information is information including the try-on person image 51 used in displaying the virtual try-on image by the terminal device 26 , the output data 82 , and the positive example information indicating that the output data 82 is a positive example.

[0134] The second display result information includes negative example information indicating a negative example, a try-on person image 51, and output data 82. The negative example information included in the second display result information is information indicating that the output data 82 included in the second display result information is a negative example. Furthermore, the try-on person image 51 included in the second display result information is a try-on person image 50 used for a virtual try-on image synthesized and displayed in the terminal device 26 using the output data 82 included in the second display result information.

[0135] That is, the second display result information is information including the try-on person image 51 used in displaying the virtual try-on image by the terminal device 26 , the output data 82 , and negative example information indicating that the output data 82 is a negative example.

[0136] When the model learning unit 12E of the learning unit 12C obtains the first display result information using the acquisition unit 12A, it registers new teaching data 60 that establishes correspondence between the input data 80 of the try-on person image 51 included in the first display result information and the output data 82 included in the first display result information in the learning data 14A.

[0137] Therefore, if Figure 4 As shown, in the learning data 14A, new teaching data 60D is registered in the learning data 14A, in which the input data 80 including the try-on user image 51 included in the first display result information and the output data 82 included in the first display result information are associated with each other. The teaching data 60D is an example of the teaching data 60 .

[0138] On the other hand, when the second display result information is obtained using the acquisition unit 12A, the model learning unit 12E of the learning unit 12C will register new teaching data 60 that establishes correspondence between the input data 80 including the try-on person image 51 included in the second display result information and the output data 82 including negative example information representing a negative example in the learning data 14A.

[0139] Therefore, if Figure 4 As shown, in the learning data 14A, new teaching data 60E in which input data 80 including the try-on person image 51 included in the second display result information and output data 82 including negative example information are associated with each other is registered in the learning data 14A. The teaching data 60E is an example of the teaching data 60 .

[0140] In this case, the model learning unit 12E can learn the learning model using the teaching data 60 (including the teaching data 60A, the changed teaching data 60A, the teaching data 60B, the teaching data 60C, the teaching data 60D, and the teaching data 60E) registered in the learning data 14A.

[0141] return Figure 3 Next, the derivation unit 12D will be described. When the acquisition unit 12A acquires the try-on person image 51 from the terminal device 26, the derivation unit 12D uses the try-on person image 51 and the learning model generated by the learning unit 12C to derive the output data 82 of the try-on person represented by the try-on person image 51.

[0142] The deriving unit 12D inputs the try-on person image 51 acquired from the terminal device 26 as input data 80 to the learning model generated by the learning unit 12C, thereby deriving output data 82 corresponding to the try-on person image 51 .

[0143] Furthermore, the derivation unit 12D transmits the derived output data 82 to the terminal device 26 via the communication unit 12B. Therefore, the information processing device 10 can transmit the output data 82 (body shape parameters 72, synthesis position information 56, and output data 82) for the try-on person image 51 of the virtual try-on object received from the terminal device 26 to the terminal device 26.

[0144] Next, the function of the terminal device 26 will be described.

[0145] The terminal device 26 includes an imaging unit 26A, an input unit 26B, a display unit 26C, a storage unit 26D, and a control unit 26E. The imaging unit 26A, the input unit 26B, the display unit 26C, and the storage unit 26D are connected to the control unit 26E so as to be able to exchange data or signals. The storage unit 26D stores various data.

[0146] The control unit 26E includes a communication unit 26F, an acquisition unit 26G, a generation unit 26H, a display control unit 26I, and an acceptance unit 26J. Part or all of the control unit 26E, the communication unit 26F, the acquisition unit 26G, the generation unit 26H, the display control unit 26I, and the acceptance unit 26J may be implemented by, for example, causing a processing device such as a CPU to execute a program, that is, by software, or by hardware such as an IC, or by a combination of software and hardware.

[0147] The communication unit 26F communicates with the information processing device 10 or the external server 28 via the communication line 34 .

[0148] The display control unit 26I generates various images on the display unit 26C. The receiving unit 26J receives various information inputted by the user's operation instruction of the input unit 26B from the input unit 26B. The acquiring unit 26G acquires the output data 82 from the information processing device 10 via the communication unit 26F and the communication line 34 .

[0149] The generating unit 26H generates a virtual try-on image by combining the try-on person image 51 and the clothing image 42 .

[0150] For example, it is assumed that the clothing ID of the clothing image 42 of the virtual try-on object is input through the operation instruction of the input unit 26B by the user. For example, the generation unit 26H sends a browsing request of the clothing DB 14B to the information processing device 10 via the communication unit 26F. Then, the generation unit 26H receives the clothing DB 14B from the information processing device 10 via the acquisition unit 26G and the communication unit 26F. Furthermore, the display control unit 26I displays the list of clothing images displayed by the received clothing DB 14B on the display unit 26C.

[0151] The user selects a clothing image to be tried on by operating the input unit 26B while referring to the list of clothing images displayed on the display unit 26C. Then, the receiving unit 26J receives the clothing ID of the selected clothing image from the input unit 26B. Furthermore, the generating unit 26H transmits the received clothing ID to the information processing device 10 via the communication unit 26F.

[0152] The communication unit 12B of the information processing device 10 reads the clothing image 42 corresponding to the clothing ID received from the terminal device 26 from the clothing DB 14B and transmits it to the terminal device 26. Through this process, the generation unit 26H of the terminal device 26 acquires the clothing image 42 selected by the user from the information processing device 10 via the acquisition unit 26G and the communication unit 26F.

[0153] Then, the generating unit 26H obtains the try-on person image 51 from the imaging unit 26A via the obtaining unit 26G. For example, the user inputs an imaging instruction to the imaging unit 26A by operating the input unit 26B. Then, the imaging unit 26A obtains the try-on person image 51 by imaging and outputs it to the control unit 26E.

[0154] The generation unit 26H of the control unit 26E acquires the try-on person image 51 of the virtual try-on object from the imaging unit 26A via the acquisition unit 26G. Alternatively, the generation unit 26H may acquire the try-on person image 51 from the storage unit 26D.

[0155] In addition, the try-on person image 51 acquired from the imaging unit 26A or the storage unit 26D may not include the clothing wearing area.

[0156] The clothing wearing area indicates the area of ​​the person who is to wear the clothing image 42 as the virtual try-on object. For example, when the clothing image 42 is a hat, the clothing wearing area is the head of the person who is to try on. And when the clothing image 42 is a top, the clothing wearing area is the upper body of the person who is to try on.

[0157] In this case, the generation unit 26H complements the acquired try-on person image 51 so that it includes the clothing wearing area. Alternatively, the generation unit 26H may complement the acquired try-on person image 51 so that it includes the clothing wearing area by changing the shooting angle of the try-on person represented by the try-on person image 51. Furthermore, the generation unit 26H may use the complemented try-on person image 51 as the try-on person image 51 used in the virtual try-on.

[0158] In addition, the complementation may be performed by a known method. For example, the generating unit 26H may prepare a human body image of a standard human body shape in advance. Furthermore, the generating unit 26H may use a composite image obtained by superimposing an image excluding the clothing wearing area, that is, the try-on person image 51, on the human body image as the complemented try-on person image 51.

[0159] Next, the generation unit 26H transmits the try-on person image 51 used in the virtual try-on to the information processing device 10 via the communication unit 26F. Then, as described above, the deriving unit 12D of the information processing device 10 derives the output data 82 of the try-on person represented by the try-on person image 51, using the try-on person image 51 obtained from the terminal device 26 and the learning model generated by the learning unit 12C.

[0160] The communication unit 12B of the information processing device 10 transmits the output data 82 derived by the deriving unit 12D to the terminal device 26 .

[0161] Therefore, the generator 26H of the terminal device 26 obtains the output data 82 from the information processing device 10 via the communication unit 26F. In other words, the generator 26H obtains the output data 82 of the try-on person represented by the try-on person image 51 derived using the try-on person image 51 of the virtual try-on object and the learning model.

[0162] Then, the generation unit 26H uses the try-on person image 51 of the virtual try-on object and the output data 82 of the try-on person represented by the try-on person image 51 obtained from the information processing device 10 to generate a virtual try-on image that is a composite of the try-on person image 51 and the clothing image 42 .

[0163] Specifically, the generation unit 26H generates a virtual fitting image in which the clothing image 42 deformed according to the body shape parameters 72 included in the output data 82 derived using the learning model is synthesized at the synthesis position in the try-on person image 51 .

[0164] The generation unit 26H may generate the virtual try-on image in the same manner as the generation of the superimposed image W by the learning unit 12C of the information processing device 10. That is, the generation unit 26H may enlarge, reduce, or deform the clothing image 42 of the virtual try-on object indicated by the user operating the terminal device 26 so as to present the size of the body shape represented by the body shape parameter 72 included in the derived output data 82. This process may use a known technique.

[0165] Furthermore, the generator 26H generates a virtual try-on image by synthesizing the deformed clothing image 42 at the synthesized position indicated by the synthesized position information 56 included in the derived output data 82. The synthesizing at the synthesized position indicated by the synthesized position information 56 may be performed using a known method.

[0166] The display control unit 26I displays the virtual try-on image generated by the generating unit 26H on the display unit 26C.

[0167] Figure 6 2 is a schematic diagram showing an example of a virtual try-on image 54 displayed on the display unit 26C of the terminal device 26. Figure 6 As shown, a virtual try-on image 54 is displayed on the display unit 26C.

[0168] As described above, the virtual try-on image 54 is a virtual try-on image 54 generated by the generating unit 26H. Therefore, the display unit 26C displays the virtual try-on image 54 in which the clothing image 42 deformed to the body shape represented by the body shape parameter 72 included in the output data 82 derived using the learning model is synthesized at the synthesized position represented by the synthesized position information 56 included in the output data 82 derived using the learning model. That is, the clothing image 42 enlarged, reduced, and deformed so as to fit the body shape of the try-on user with high accuracy is displayed in a superimposed manner on the try-on user region 50A of the virtual try-on image 54.

[0169] Therefore, the virtual try-on system 1 can easily provide the user with a highly precisely synthesized virtual try-on image 54. Specifically, the virtual try-on system 1 can easily provide a virtual try-on image 54 that has been corrected to be precisely aligned and to fit the body shape.

[0170] Next, the procedure of the virtual try-on process executed in the virtual try-on system 1 will be described.

[0171] Figure 7 This is a flowchart showing an example of the procedure of information processing executed by the information processing device 10 .

[0172] First, the acquisition unit 12A determines whether a shooting instruction is received from the input unit 22 (step S100). For example, when registering the teaching data 60A in the information processing device 10, the user facing the front of the display unit 18 inputs a shooting instruction by operating the input unit 22. Then, the acquisition unit 12A receives the shooting instruction from the input unit 22.

[0173] If a positive judgment is made in step S100 (step S100: Yes), the process proceeds to step S102. In step S102, the photographing unit 20 and the three-dimensional data acquisition unit 21 photograph the person trying on the clothes and obtain the teaching person trying on the clothes image 40 and the three-dimensional data of the person trying on the clothes. Then, the acquisition unit 12A acquires the teaching person trying on the clothes image 40 from the photographing unit 20 (step S104). Furthermore, the acquisition unit 12A acquires the three-dimensional data from the three-dimensional data acquisition unit 21 (step S106).

[0174] Next, the acquisition unit 12A acquires the characteristic information 70 from the input unit 22 (step S108). For example, the try-on user facing the front of the display unit 18 inputs the characteristic information 70 by operating the input unit 22. Then, the input unit 22 acquires the characteristic information 70 of the try-on user represented by the teaching try-on user image 40 acquired in step S104.

[0175] Next, the model learning unit 12E of the learning unit 12C generates output data 82 for the teaching data 60A using the teaching try-on image 40 , the three-dimensional data, and the feature information 70 acquired in steps S104 to S108 (step S110 ).

[0176] Specifically, the model learning unit 12E imports the three-dimensional data obtained in step S106 into a pre-generated function or model to derive the body shape parameter 72. Furthermore, for example, the model learning unit 12E derives the rectangular area surrounding the try-on area 40A representing the try-on person in the teaching try-on person image 40 obtained in step S104 as the synthesized position information 56 (see also Figure 4 ).

[0177] Furthermore, the model learning unit 12E generates the derived body shape parameter 72 , the synthesized position information 56 , and the feature information 70 acquired in step S108 as output data 82 .

[0178] Next, the model learning unit 12E registers the teaching data 60A including the input data 80 and the output data 82 (body shape parameters 72, synthetic position information 56, feature information 70) generated using the input data 80 and the three-dimensional data in the learning data 14A, and the input data 80 includes the teaching try-on image 40 and the feature information 70 obtained in steps S104 and S108 (step S112).

[0179] Next, the display control unit 12H generates a superimposed image W using the teaching data 60A registered in step S112 (step S114). In step S114, the display control unit 12H generates a superimposed image W in which the clothing image 42 deformed according to the body shape parameter 72 included in the output data 82 is superimposed on the teaching try-on person image 40 included in the teaching data 60A at the composite position indicated by the composite position information 56 included in the output data 82 included in the teaching data 60A. Furthermore, the display control unit 12H displays the superimposed image W generated in step S114 on the display unit 18 (step S116).

[0180] Through the processing of step S114 and step S116 , it is possible to provide a superimposed image W synthesized in accordance with the movement or posture of the person trying on the clothing when the teaching data 60 is generated.

[0181] Next, the change unit 12J determines whether a change instruction has been received from the input unit 22 via the acquisition unit 12A (step S118). If a negative determination is made in step S118 (step S118: No), this routine ends. On the other hand, if an affirmative determination is made in step S118 (step S118: Yes), the routine proceeds to step S120.

[0182] In step S120, the change unit 12J changes the output data 82 included in the teaching data 60A registered in step S112 to the changed information indicated by the change instruction received in step S118 (step S120). Through the processing of step S120, at least a part of the information of the output data 82 included in the teaching data 60A is changed to the change content instructed by the user. Then, this routine ends.

[0183] On the other hand, if a negative judgment is made in the above step S100 (step S100: No), the process proceeds to step S122. In step S122, the model learning unit 12E determines whether to generate a learning model (step S122). For example, the model learning unit 12E determines to generate a learning model when the learning data 14A has been updated. This indicates that the learning data 14A has been updated, at least a part of the teaching data 60 registered in the learning data 14A has been changed, or new teaching data 60 has been registered in the learning data 14A.

[0184] If an affirmative determination is made in step S122 (step S122: Yes), the process proceeds to step S124. In step S124, the model learning unit 12E reads the learning data 14A stored in the storage unit 14 (step S124).

[0185] Next, the model learning unit 12E learns the learning model by machine learning using the learning data 14A read in step S124 (step S126). Furthermore, the model learning unit 12E stores the learning model learned in step S126 in the storage unit 14 (step S128). In addition, there is a case where a learning model is already stored in the storage unit 14. In this case, the model learning unit 12E deletes the already stored learning model from the storage unit 14 and stores the latest learned learning model in the storage unit 14. Then, this routine ends.

[0186] On the other hand, if a negative determination is made in the above-mentioned step S122 (step S122: No), the process proceeds to step S130.

[0187] In step S130, the communication unit 12B determines whether the clothing ID has been obtained from the terminal device 26 (step S130). If a negative determination is made in step S130 (step S130: No), the process proceeds to step S136 described later. If an affirmative determination is made in step S130 (step S130: Yes), the process proceeds to step S132. In step S132, the communication unit 12B reads the clothing image 42 corresponding to the clothing ID obtained in step S130 from the clothing DB 14B (step S132). Furthermore, the communication unit 12B transmits the clothing image 42 read in step S132 to the terminal device 26 as the acquisition source of the clothing ID (step S134).

[0188] Next, the deriving unit 12D determines whether the try-on person image 51 has been acquired from the terminal device 26 by the communication unit 12B (step S136 ). If an affirmative determination is made in step S136 (step S136 : Yes), the process proceeds to step S138 .

[0189] In step S138 , the try-on person image 51 acquired in step S136 is introduced into the learning model stored in the storage unit 14 , thereby deriving the output data 82 (step S138 ).

[0190] Next, the deriving unit 12D transmits the output data 82 derived in step S138 to the terminal device 26 which is the transmission source of the try-on person image 51 acquired in step S136 via the communication unit 12B (step S140 ). Then, this routine ends.

[0191] If a negative judgment is made in the above step S136 (step S136: No), the process proceeds to step S142. In step S142, the model learning unit 12E determines whether the first display result information has been acquired from the terminal device 26 by the acquisition unit 12A (step S142). If a negative judgment is made in step S142 (step S142: No), the process proceeds to step S146 described later. If an affirmative judgment is made in step S142 (step S142: Yes), the process proceeds to step S144.

[0192] In step S144 , the model learning unit 12E registers the input data 80 including the try-on user image 51 included in the first display result information acquired in step S142 , the output data 82 included in the first display result information, and the teaching data 60D in the learning data 14A (step S144 ).

[0193] Next, the model learning unit 12E determines whether the second display result information has been obtained from the terminal device 26 by the acquisition unit 12A (step S146). If a negative judgment is made in step S146 (step S146: No), this routine ends. If a positive judgment is made in step S146 (step S146: Yes), the process proceeds to step S148. In step S148, the model learning unit 12E registers the new teaching data 60E that establishes a correspondence between the input data 80 of the try-on person image 51 included in the second display result information obtained in step S146 and the output data 82 including the negative example information indicating the negative example in the learning data 14A (step S148). Then, this routine ends.

[0194] Next, an example of the procedure of information processing executed by the terminal device 26 is described. Figure 8 This is a flowchart showing an example of the procedure of information processing executed by the terminal device 26.

[0195] First, the receiving unit 26J receives the clothing ID of the clothing image 42 of the virtual try-on object from the input unit 26B (step S200). Then, the generating unit 26H transmits the clothing ID received in step S200 to the information processing device 10 via the communication unit 26F (step 202). Furthermore, the generating unit 26H obtains the clothing image 42 corresponding to the clothing ID transmitted in step S200 from the information processing device 10 (step S204).

[0196] Next, the receiving unit 26J repeatedly makes negative determinations (step S206: No) until it determines that a photographing instruction has been received from the input unit 26B (step S206: Yes). If it determines that a photographing instruction has been received (step S206: Yes), the process proceeds to step S208.

[0197] In step S208, the imaging unit 26A captures the image of the person trying on the clothing and acquires the image of the person trying on the clothing 51 (step S208). Then, the acquisition unit 26G acquires the image of the person trying on the clothing 51 from the imaging unit 26A (step S210).

[0198] Next, the generating unit 26H determines whether the try-on person image 51 acquired in step S210 includes the clothing wearing region of the clothing image 42 acquired in step S204 (step S212 ).

[0199] If it is determined that the clothing wearing region is not included (step S212: No), the process proceeds to step S214. In step S214, the generation unit 26H complements the try-on person image 51 acquired in step S210 so that the try-on person image 51 becomes an image of the clothing wearing region of the clothing image 42 acquired in step S204 (step S214). In this case, the generation unit 26H adopts the complemented try-on person image 51 as the try-on person image 51 used in the virtual try-on.

[0200] Next, the generator 26H transmits the try-on person image 51 obtained in step S210 or the try-on person image 51 completed in step S214 to the information processing device 10 via the communication unit 26F (step S216). Then, as described above, the deriving unit 12D of the information processing device 10 derives the output data 82 of the try-on person represented by the try-on person image 51 using the try-on person image 51 obtained from the terminal device 26 and the learning model generated by the learning unit 12C, and transmits the output data 82 to the terminal device 26.

[0201] Therefore, the generation unit 26H of the terminal device 26 obtains the output data 82 from the information processing device 10 via the communication unit 26F (step S218). In other words, the generation unit 26H obtains the output data 82 of the try-on person represented by the try-on person image 51 derived using the try-on person image 51 of the virtual try-on object and the learning model.

[0202] Therefore, the generation unit 26H uses the try-on person image 51 of the virtual try-on object sent to the information processing device 10 in step S216 and the output data 82 of the try-on person represented by the try-on person image 51 obtained from the information processing device 10 to generate a virtual try-on image 54 (step S220) by synthesizing the try-on person image 51 and the clothing image 42.

[0203] Next, the display control unit 26I displays the virtual try-on image 54 generated in step S220 on the display unit 26C (step S222 ). Therefore, a highly accurate virtual try-on image 54 is displayed on the display unit 26C of the terminal device 26 .

[0204] Next, the receiving unit 26J determines whether error information has been received from the input unit 26B (step S224).

[0205] There is a case where the user who operates the terminal device 26 feels a sense of discomfort with the virtual try-on image 54 displayed on the display unit 26C. For example, there is a case where the positional relationship between the try-on person image 51 and the clothing image 42 in the virtual try-on image 54, or the size of the displayed clothing image 42, etc. are different from the user's imagination. In such a case, the user inputs error information for making the output data 82 used in the generation of the virtual try-on image 54 a negative example by operating the input unit 26B. Then, the receiving unit 26J receives the error information from the input unit 22 (step S224: Yes).

[0206] If a positive judgment is made in step S224 (step S224: Yes), the process proceeds to step S226. In step S226, the receiving unit 26J includes negative example information indicating a negative example, the try-on person image 51 obtained in step S210, and the output data 82 obtained in step S218. The second display result information is sent to the information processing device 10 via the communication unit 26F (step S226). Then, the process proceeds to step S228. In addition, if a negative judgment is made in step S224 (step S224: No), the process proceeds to step S228.

[0207] In step S228, the receiving unit 26J determines whether a correction instruction has been received from the input unit 22 (step S228).

[0208] For example, there is a case where the user operating the terminal device 26 wishes to correct the synthesis position of the clothing image 42, the size of the clothing image 42, etc., with respect to the virtual try-on image 54 displayed on the display unit 26C. In such a case, the user operates the input unit 22 to input a correction instruction for at least one of the body shape parameter 72 and the synthesis position information 56. In addition, there is a case where the user operating the terminal device 26 does not wish to make changes such as corrections with respect to the virtual try-on image 54 displayed on the display unit 26C. In such a case, the user operates the input unit 22 to input an instruction indicating no correction.

[0209] Then, in step S228, the accepting unit 26J determines whether or not a correction instruction has been accepted from the input unit 22. If the accepting unit 26J accepts a correction instruction from the input unit 22 (step S228: Yes), the process proceeds to step S230.

[0210] In step S230, the accepting unit 26J transmits the first display result information including the positive example information indicating the positive example, the try-on person image 51, and the output data 82 corrected to the correction content indicated by the correction instruction accepted in step S228 to the information processing device 10 via the communication unit 26F (step S230). In addition, the try-on person image 51 is the try-on person image 51 used in the virtual try-on image 54 displayed on the display unit 26C in step S222. Then, this routine ends.

[0211] On the other hand, if the receiving unit 26J receives an instruction indicating no correction from the input unit 26B (step S228: No), the process proceeds to step S232. In step S232, the receiving unit 26J transmits the first display result information including the positive example information indicating the positive example, the try-on person image 51, and the output data 82 obtained in step S218 to the information processing device 10 via the communication unit 26F (step S232). In addition, the try-on person image 51 is the try-on person image 51 used in the virtual try-on image 54 displayed on the display unit 26C in step S222. Then, this routine ends.

[0212] As described above, the virtual try-on system 1 of the present embodiment includes a learning unit 12C, an acquisition unit 12A, a derivation unit 12D, and a generation unit 26H. The learning unit 12C learns a learning model that uses the teaching try-on person image 40 as input data 80, the body shape parameters 72 representing the body shape of the try-on person represented by the teaching try-on person image 40, and the synthesis position information 56 of the clothing image 42 in the teaching try-on person image 40 as output data 82 by machine learning using the three-dimensional data of the try-on person. The acquisition unit 26G acquires the try-on person image 51. The derivation unit 12D uses the try-on person image 51 and the learning model to derive the output data 82 of the try-on person represented by the try-on person image 51. The generation unit 26H uses the derived output data 82 to generate a virtual try-on image 54 that is a synthesis of the try-on person image 51 and the clothing image 42.

[0213] In this way, in the virtual try-on system 1 of the present embodiment, the output data 82 (body shape parameters 72 and synthesis position information 56) of the try-on person represented by the try-on person image 51 is derived using the learning model learned by machine learning using the three-dimensional data of the try-on person. Furthermore, using the derived output data 82, a virtual try-on image 54 is generated by synthesizing the try-on person image 51 and the clothing image 42.

[0214] Therefore, in the virtual try-on system 1 of the present embodiment, when displaying the virtual try-on image 54, it is not necessary to obtain information related to the three-dimensional shape of the person trying on the clothing. It is possible to use the body shape parameters 72 and the synthesis position information 56 derived using the learning model to generate a virtual try-on image 54 that is a synthesis of the try-on person image 51 and the clothing image 42.

[0215] Therefore, in the virtual try-on system 1 of the present embodiment, it is possible to easily provide the virtual try-on image 54 synthesized with high accuracy.

[0216] <Modification>

[0217] In the above embodiment, the information processing device 10 is provided with the learning unit 12C and the deriving unit 12D, and the terminal device 26 is provided with the generating unit 26H. However, the information processing device 10 may be provided with at least one of the acquiring unit 26G, the generating unit 26H, the accepting unit 26J, and the display control unit 26I. Furthermore, the terminal device 26 may be provided with at least one of the acquiring unit 12A, the learning unit 12C, and the deriving unit 12D.

[0218] In this case, one information processing device 10 or one terminal device 26 can be used to generate learning data 14A, learn the learning model, export the output data 82 of the try-on person represented by the try-on person image 51, and generate and display the virtual try-on image 54.

[0219] Next, the hardware configuration of the information processing device 10, the terminal device 26, and the external server 28 according to the present embodiment will be described. Fig. 9 1 is a block diagram showing a hardware configuration example of the information processing device 10 , the terminal device 26 , and the external server 28 according to the present embodiment.

[0220] The information processing device 10, terminal device 26, and external server 28 of this embodiment are formed as a display unit 91, a communication I / F unit 93, an input unit 94, a CPU 86, a ROM (Read Only Memory) 88, a RAM (Random Access Memory) 90, and a HDD 92, which are interconnected via a bus 96 and utilize the hardware structure of a conventional computer.

[0221] The CPU 86 is a computing device that controls the processing of the information processing device 10, the terminal device 26, and the external server 28 of the present embodiment. The RAM 90 stores data required for various processing performed by the CPU 86. The ROM 88 stores programs and the like that implement various processing performed by the CPU 86. The HDD 92 stores data stored in the above-mentioned storage unit 14 and the storage unit 26D. The communication I / F unit 93 is an interface for connecting to an external device or an external terminal via a communication line, etc., and for sending and receiving data between the connected external device or external terminal. The display unit 91 is equivalent to each of the above-mentioned display unit 18 and the display unit 26C. The input unit 94 receives operation instructions from the user. The input unit 94 is equivalent to the above-mentioned input unit 22 and the input unit 26B.

[0222] Programs for executing the above-described various processes executed by the information processing device 10 , the terminal device 26 , and the external server 28 according to the present embodiment are provided by being preliminarily incorporated in the ROM 88 or the like.

[0223] In addition, the program executed by the information processing device 10, terminal device 26, and external server 28 of the present embodiment can also be configured to be provided by recording a file in a form that can be installed in the above-mentioned devices or in a form that can be executed by the above-mentioned devices on a recording medium that can be read by a computer, such as a CD-ROM, floppy disk (FD), CD-R, DVD (Digital Versatile Disk), etc.

[0224] Furthermore, the program executed by the information processing device 10, the terminal device 26, and the external server 28 of the present embodiment may be stored in a computer connected to a network such as the Internet, and provided by downloading via the network. Furthermore, the program for executing the above-mentioned processing in the information processing device 10, the terminal device 26, and the external server 28 of the present embodiment may be provided or distributed via a network such as the Internet.

[0225] The program for executing the above-described various processes executed by the information processing device 10, the terminal device 26, and the external server 28 of the present embodiment is formed so that the above-described respective components are generated on the main storage device.

[0226] In addition, various information stored in the HDD 92 may be stored in an external device. In this case, the external device and the CPU 86 may be connected via a network or the like.

[0227] According to the virtual try-on system having the structure of the embodiment described above, it is possible to easily provide a virtual try-on image that is aligned with high accuracy and corrected to fit the body shape with high accuracy.

[0228] In addition, the embodiments of the present invention are described above, but the above embodiments are only provided as examples and are not intended to limit the scope of the invention. The above new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the invention. The embodiment and its variations are included in the scope or the main idea of ​​the invention, and are included in the invention described in the technical solution and its equivalent scope.

Claims

1. A virtual try-on system, comprising: a learning unit for learning a learning model by machine learning using the three-dimensional data of the try-on person, the learning model taking the teaching try-on person image as input data and taking body shape parameters representing the body shape of the try-on person presented by the teaching try-on person image and synthetic position information of the clothing image in the teaching try-on person image as output data; A first acquisition unit acquires an image of the person trying on the clothing captured by the capturing unit; A second acquisition unit acquires the three-dimensional data of the person trying on the clothing; an exporting unit that uses the try-on person image and the learning model to export the output data of the try-on person represented by the try-on person image; and a generating unit, using the derived output data, to generate a virtual try-on image by synthesizing the try-on person image and the clothing image; The photographing unit and the second acquisition unit are controlled to photograph synchronously. The learning unit includes an extended image generating unit for generating an extended image by extending the teaching try-on image. The learning model is trained using the input data including the teaching wearer image and the input data including the extended image. The extended image generating unit generates at least one of the following images as the extended image, namely: a clothing overlapping image in which a clothing image is overlapped on a try-on person area included in the teaching try-on person image, a masking image in which a portion of the teaching try-on person image is masked, a cut-out image in which a portion of the teaching try-on person image is cut out, and a processed image in which at least a portion of the teaching try-on person image is processed.

2. The virtual try-on system according to claim 1, wherein: The generating unit generates the virtual try-on image by combining the clothing image deformed according to the body shape parameter included in the output data at a combining position indicated by the combining position information included in the derived output data in the try-on person image.

3. The virtual try-on system according to claim 1, wherein: The learning unit learns the learning model including feature information of the try-on person in at least one of the input data and the output data.

4. The virtual try-on system according to claim 3, wherein: The feature information indicates at least one of the appearance feature and the inner feature of the wearer.

5. The virtual try-on system according to claim 3, wherein: The characteristic information indicates at least one of the skin color, exposure degree from clothing, hairstyle, nationality, dominant hand, personality, and recommended shooting direction of the person trying on the clothing.

6. The virtual try-on system according to claim 1, wherein: The imaging unit and the second acquiring unit capture images of the person trying on the clothing from a plurality of imaging angles.

7. The virtual try-on system according to claim 6, wherein: The input data includes a plurality of the teaching try-on images that are different in at least one of the shooting time, the shooting angle, the type of the shooting unit used for shooting, and the posture of the try-on during shooting.

8. The virtual try-on system according to claim 1, wherein: The learning unit learns the learning model by machine learning using learning data, wherein the learning data includes a plurality of teaching data corresponding to the input data including the teaching try-on image and the output data corresponding to the teaching try-on image derived based on the three-dimensional data of the try-on presented by the teaching try-on image.

9. The virtual try-on system according to claim 8, wherein: The input data includes the characteristic information of the wearer. The learning unit includes a correction unit that corrects the output data derived from the three-dimensional data in the learning data based on the feature information included in the input data.

10. The virtual try-on system according to claim 8, wherein: The above-mentioned learning department has: The display control unit displays the following superimposed image on the display unit, namely, a superimposed image of the clothing image deformed according to the body shape parameters included in the output data is superimposed on the teaching try-on person image at a composite position indicated by the composite position information included in the output data corresponding to the input data including the teaching try-on person image in the learning data; a receiving unit that receives a change instruction for the output data corresponding to the teaching try-on person image included in the displayed superimposed image in the learning data; and The changing unit changes the output data corresponding to the input data including the teaching try-on person image in the learning data to the output data indicated by the change instruction.

11. The virtual try-on system according to claim 1, wherein: The above-mentioned synthetic position information is 1) information representing a rectangular synthetic object area in the teaching try-on image that surrounds the try-on person area, 2) information representing the positions of each vertex of the rectangle that surrounds the try-on person area, 3) information representing at least a part of the outer shape of the try-on person area contained in the teaching try-on image, or 4) information representing a specific part of the try-on person in the teaching try-on image.

12. The virtual try-on system according to claim 8, wherein: The acquisition unit acquires first display result information including positive example information indicating a positive example, the image of the person trying on the wearer, and the output data, or second display result information including negative example information indicating a negative example and the image of the person trying on the wearer, The above-mentioned Learning Department, When the first display result information is obtained, the new teaching data corresponding to the input data including the try-on person image included in the first display result information and the output data included in the first display result information is registered in the learning data. When the second display result information is acquired, the input data including the try-on person image included in the second display result information and the output data indicating a negative example are registered in the learning data as new teaching data.

13. A virtual try-on method, comprising: The step of learning a learning model by machine learning using three-dimensional data of a try-on person, the learning model taking a teaching try-on person image as input data and outputting body shape parameters representing the body shape of the try-on person presented by the teaching try-on person image and synthetic position information of a clothing image in the teaching try-on person image; A first acquisition step of acquiring an image of a person trying on the clothing captured by a capturing unit; A second step of acquiring the three-dimensional data of the wearer; Using the image of the try-on user and the learning model, a step of deriving the output data of the try-on user represented by the image of the try-on user; and The step of generating a virtual try-on image by synthesizing the try-on person image and the clothing image using the output data derived above, The photographing unit and the second acquisition step are controlled to photograph synchronously. The step of learning the learning model includes an extended image generation step of generating an extended image by expanding the teaching try-on image. The learning model is trained using the input data including the teaching wearer image and the input data including the extended image. In the above-mentioned extended image generating step, at least one of the following images is generated as the above-mentioned extended image, namely: a clothing overlapping image in which a clothing image is overlapped on the tryer area included in the above-mentioned teaching tryer image, a masking image in which a part of the teaching tryer image is masked, a cut-out image in which a part of the teaching tryer image is cut out, and a processed image in which at least a part of the teaching tryer image is processed.

14. A storage medium storing a virtual try-on program, wherein the virtual try-on program enables a computer to execute the following steps: The step of learning a learning model by machine learning using three-dimensional data of a try-on person, the learning model taking a teaching try-on person image as input data and outputting body shape parameters representing the body shape of the try-on person presented by the teaching try-on person image and synthetic position information of a clothing image in the teaching try-on person image; A first acquisition step of acquiring an image of a person trying on the clothing captured by a capturing unit; A second step of acquiring the three-dimensional data of the wearer; Using the image of the try-on user and the learning model, a step of deriving the output data of the try-on user represented by the image of the try-on user; and The step of generating a virtual try-on image by synthesizing the try-on person image and the clothing image using the output data derived above, The virtual fitting program is controlled so that the photographing unit and the second acquisition step are synchronously photographed. The step of learning the learning model includes an extended image generation step of generating an extended image by expanding the teaching try-on image. The learning model is trained using the input data including the teaching wearer image and the input data including the extended image. In the above-mentioned extended image generating step, at least one of the following images is generated as the above-mentioned extended image, namely: a clothing overlapping image in which a clothing image is overlapped on the tryer area included in the above-mentioned teaching tryer image, a masking image in which a part of the teaching tryer image is masked, a cut-out image in which a part of the teaching tryer image is cut out, and a processed image in which at least a part of the teaching tryer image is processed.

15. An information processing device comprising: a learning unit for learning a learning model by machine learning using the three-dimensional data of the try-on person, the learning model taking the teaching try-on person image as input data and taking body shape parameters representing the body shape of the try-on person presented by the teaching try-on person image and synthetic position information of the clothing image in the teaching try-on person image as output data; A first acquisition unit acquires an image of the person trying on the clothing captured by the capturing unit; A second acquisition unit acquires the three-dimensional data of the person trying on the clothing; an exporting unit that uses the try-on person image and the learning model to export the output data of the try-on person represented by the try-on person image; and a generating unit, using the derived output data, to generate a virtual try-on image by synthesizing the try-on person image and the clothing image; The photographing unit and the second acquisition unit are controlled to photograph synchronously. The learning unit includes an extended image generating unit for generating an extended image by extending the teaching try-on image. The learning model is trained using the input data including the teaching wearer image and the input data including the extended image. The extended image generating unit generates at least one of the following images as the extended image, namely: a clothing overlapping image in which a clothing image is overlapped on a try-on person area included in the teaching try-on person image, a masking image in which a portion of the teaching try-on person image is masked, a cut-out image in which a portion of the teaching try-on person image is cut out, and a processed image in which at least a portion of the teaching try-on person image is processed.

Images