Information processing device, information processing method, and information processing program

Abstract

To generate appropriate synthesized information.SOLUTION: An information processing device according to the present application includes an acquisition unit and a generation unit. The acquisition unit acquires first object information being a first object as a synthesizing object, second object information being a second object as a synthesizing object, and range specification information indicating a specified range specified as a range for suppressing variations in the first object information. The generation unit suppresses variations of the specified range in the first object information on the basis of the range specification information, and generates synthesized information obtained by synthesizing the first object information and the second object information. For example, the generation unit generates the synthesized information by using a learning model which has learned a feature of relevancy between the first object, the second object, and the specified range, and the first object in which variations of the specified range in the first object is suppressed and the second object.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, various techniques have been proposed for generating composite information by combining a plurality of pieces of information, such as a technique for generating a composite image by combining a plurality of images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-004203 [Non-patent literature]

[0004] [Non-Patent Document 1] “U-Net: Convolutional Networks for Biomedical Image Segmentation”, Olaf Ronneberger, Philipp Fischer, and Thomas Brox <Internet> https: / / arxiv.org / pdf / 1505.04597.pdf (Retrieved May 31, 2022) Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is room for improvement in the above-described conventional technology. For example, the above-described conventional technology merely generates composite information in which the composite ratio of each image has been changed by changing the composite ratio of each of a plurality of images to generate a composite image, and does not necessarily generate appropriate composite information. Therefore, it is desirable to generate appropriate composite information.

[0006] The present application has been made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that generate appropriate composite information. [Means for solving the problem]

[0007] The information processing device according to the present application is characterized by comprising: an acquisition unit that acquires first object information, which is a first object to be synthesized, second object information, which is a second object to be synthesized, and range designation information indicating a designated range designated as a range in which fluctuations are to be suppressed in the first object information; and a generation unit that suppresses fluctuations in the designated range in the first object information based on the range designation information, and generates synthesized information that synthesizes the first object information and the second object information. [Effects of the Invention]

[0008] According to one aspect of the embodiment, it is possible to produce an effect that appropriate synthesis information can be generated. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a model used for information processing. [Figure 3] FIG. 3 is a diagram illustrating an example of a model used for information processing. [Figure 4] FIG. 4 is a diagram illustrating an example of a model used for information processing. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of an information processing device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a learning data storage unit according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a model information storage unit according to the embodiment. [Figure 9]FIG. 9 is a diagram illustrating an example of a synthesis result information storage unit according to the embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of the flow of information processing executed by the information processing device. [Figure 11] FIG. 11 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a detailed description will be given of an information processing device, an information processing method, and an information processing program (hereinafter referred to as an "embodiment") according to the present application, with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiment. Furthermore, the same components in the following embodiments are denoted by the same reference numerals, and redundant description will be omitted.

[0011] (Embodiment) [1. Information Processing] First, an example of information processing executed by the information processing device 100 will be described with reference to FIG. 1. FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. Below, a case will be described as an example in which the synthesis target is an image, and the information processing device 100 generates an image (also referred to as a "synthetic image") as synthesis information. In FIG. 1, the first object information is an image (also referred to as a "first image") including a person's head, the second object information is an image (also referred to as a "second image") including hair (head hair, hair), and the information processing device 100 generates a synthetic image by synthesizing the first image and the second image. Specifically, in FIG. 1, the information processing device 100 specifies a part of a person's face as a range (also referred to as a "designated range") in which fluctuations are to be suppressed in the first image, which is the first object information, and generates a synthetic image by synthesizing the first image and the second image by suppressing fluctuations in the person's face in the first image.

[0012] Note that the images to be synthesized are not limited to images of a person's head, and may be various images. For example, the first image may be an image including the outline of a person, and the second image may be an image including items worn by the person, such as clothing and accessories. In this case, the information processing device 100 generates a synthesized image in which the items included in the second image are reflected on the outline of the person included in the first image. For example, if the items are clothing, the information processing device 100 generates a synthesized image in which the person included in the first image is wearing the clothing included in the second image.

[0013] Furthermore, the images to be synthesized are not limited to images of the human body, and may be various other images. For example, the first image may be an image including the exterior wall of a property, and the second image may be an image including a pattern. In this case, the information processing device 100 generates a synthesized image in which the pattern included in the second image is reflected on the exterior wall of the property included in the first image. For example, the information processing device 100 generates a synthesized image in which the pattern of the exterior wall of the property included in the first image is changed to the pattern included in the second image.

[0014] Furthermore, while an image is an example of a synthesis target, the synthesis target is not limited to an image and may be various information. For example, the synthesis target may be audio. In this case, the information processing device 100 generates synthetic audio by synthesizing a first audio signal with a second audio signal. For example, the information processing device 100 specifies pitch, vowels, consonants, etc. as a range (specified range) in which fluctuations are to be suppressed in the first audio signal, which is the first target information, and suppresses fluctuations in the specified range in the first audio signal to generate synthetic audio by synthesizing the first audio signal with the second audio signal. Note that the synthesis target described above is merely an example, and may be various information, not limited to images and audio, as long as it is an object that can be processed by the information processing device 100.

[0015] [1-1. Information Processing System Configuration] Prior to describing the processes shown in FIGS. 1 to 4, the configuration of an information processing system 1 will be described with reference to FIGS. 1 and 5. FIG. 5 is a diagram showing an example of the configuration of an information processing system according to an embodiment. As shown in FIG. 5, the information processing system 1 includes a terminal device 10, an information providing device 50, and an information processing device 100. The terminal device 10, the information providing device 50, and the information processing device 100 are connected to each other via a predetermined network N so as to be able to communicate with each other via wired or wireless communication. Note that the information processing system 1 shown in FIG. 5 may include a plurality of terminal devices 10, a plurality of information providing devices 50, and a plurality of information processing devices 100.

[0016] The information processing device 100 is a computer that executes information processing (also referred to as a "generation process") to generate composite information by combining first object information and second object information to be combined. The information processing device 100 suppresses fluctuations in a specified range in the first object information based on information indicating a specified range specified in the first object information as a range in which fluctuations are to be suppressed (also referred to as "range designation information"), and generates composite information by combining the first object information and the second object information. For example, the information processing device 100 executes processing to generate a composite image by combining a first image and a second image to be combined. The information processing device 100 suppresses fluctuations in a specified range in the first image based on range designation information indicating a specified range specified in the first image as a range in which fluctuations are to be suppressed, and generates composite information by combining the first image and the second image.

[0017] The terminal device 10 is a computer used by a user. The terminal device 10 is a device (terminal device) that can be carried by a user. The terminal device 10 is realized, for example, by a smartphone, a tablet terminal, a notebook PC (Personal Computer), a mobile phone, a PDA (Personal Digital Assistant), or the like. FIG. 1 shows a case where the terminal device 10 is a smartphone. Note that, hereinafter, the terminal device 10 may be referred to as a user. That is, hereinafter, the user may be read as the terminal device 10.

[0018] For example, the terminal device 10 is a computer installed with an application (also referred to as a "display app") that displays composite information provided by the information processing device 100. For example, the terminal device 10 displays various types of information using the display app. For example, the terminal device 10 displays composite information received from the information processing device 100 using the display app. The terminal device 10 transmits information indicating second target information specified by a user to the information processing device 100. The terminal device 10 receives composite information reflecting the second target information specified by a user from the information processing device 100. The terminal device 10 receives a composite image from the information processing device 100. The terminal device 10 displays the composite image received from the information processing device 100. For example, the terminal device 10 displays the composite image using the display app.

[0019] The terminal device 10 transmits to the information processing device 100 information used by the information processing device 100 to generate a composite image. The terminal device 10 transmits to the information processing device 100 information used for input to a model (e.g., model M1) that outputs the composite image. For example, the terminal device 10 transmits to the information processing device 100 a first image capturing an image of the user's head. For example, the terminal device 10 transmits to the information processing device 100 a second image including hair specified by the user. For example, the terminal device 10 transmits to the information processing device 100 range designation information indicating a designated range that is the range of the user's face in the first image.

[0020] The terminal device 10 receives a composite image in which the hair included in the second image is reflected on the head of the user included in the first image from the information processing device 100. The terminal device 10 copies the face of the user included in the first image and receives a composite image in which the hairstyle of the user included in the first image is reflected on a hairstyle corresponding to the hair included in the second image from the information processing device 100.

[0021] The terminal device 10 displays a composite image in which the hair included in the second image is reflected on the head of the user included in the first image. The terminal device 10 copies the face of the user included in the first image and displays a composite image in which the hairstyle of the user included in the first image is reflected on the hairstyle corresponding to the hair included in the second image.

[0022] The information providing device 50 is a computer that stores information for providing various information related to a service. For example, the information providing device 50 collects information from various external devices such as a web server, and provides the collected information to the information processing device 100. For example, the information providing device 50 is realized by, for example, a server device or a cloud system. For example, the information providing device 50 provides, as various types of information, information related to the service (service information) and information related to the user who uses the service (user information).

[0023] For example, the information providing device 50 may provide a service of providing data used for learning (also referred to as "learning data") to the information processing device 100. In this case, the information providing device 50 provides the information processing device 100 with data (learning data) used by the information processing device 100 to learn the model M1. Furthermore, the information providing device 50 may provide a service of providing the information processing device 100 with a trained learning model (also simply referred to as "model"). In this case, the information providing device 50 provides the information processing device 100 with a model (for example, model M1) used by the information processing device 100 to generate composite information.

[0024] [1-2. Overall overview of processing in information processing systems] Next, an example of information processing performed by the information processing system 1 will be described with reference to Figures 1 to 4. First, an overview of the processing in the information processing system 1, including transmission and reception of information between devices in the information processing system 1, will be described with reference to Figure 1, and then an example of the configuration of model M1, which is an example of a model used to generate a composite image, will be described with reference to Figures 2 to 4. Note that detailed description of the information processing performed in the information processing system 1 that is the same as conventional processing will be omitted as appropriate.

[0025] 1, the information processing device 100 acquires information used to execute a generation process for generating a composite image from the information providing device 50 (step S1). The information processing device 100 receives information used to execute the generation process from the information providing device 50. For example, the information processing device 100 acquires a model M1, which is a model that outputs a composite image, from the information providing device 50. Details of the model M1 will be described later. Furthermore, when learning the model M1, the information processing device 100 may acquire data (learning data) used in the learning process of the model M1 from the information providing device 50.

[0026] The information processing device 100 stores the information acquired from the information providing device 50 in the storage unit 120 (see FIG. 6) (step S2). The information processing device 100 stores the information received from the information providing device 50 in the storage unit 120. For example, when the information processing device 100 acquires a model M1 from the information providing device 50, the information processing device 100 stores the model M1 in the model information storage unit 122 (see FIG. 6). Furthermore, when the information processing device 100 acquires learning data to be used in the learning process from the information providing device 50, the information processing device 100 may store the learning data in the learning data storage unit 121 (see FIG. 6). Note that when the information processing device 100 has already acquired information to be used for executing the generation process to generate a composite image, it does not need to perform the processes of steps S1 and S2.

[0027] Furthermore, the information processing device 100 accepts a request to generate a composite image from the terminal device 10 (step S3). For example, the information processing device 100 accepts a request to generate a composite image from the terminal device 10 by receiving various types of information used to generate the composite image from the terminal device 10.

[0028] For example, the information processing device 100 receives an image to be used as a first image from the terminal device 10. In FIG. 1, the information processing device 100 receives an image IM11 to be used as the first image from the terminal device 10. For example, the image IM11 is an image including a user's head. Furthermore, the information processing device 100 receives an image to be used as a second image from the terminal device 10. In FIG. 1, the information processing device 100 receives an image IM12 to be used as the second image from the terminal device 10. For example, the image IM12 is an image including hair in a hairstyle desired by the user. In this manner, the information processing device 100 receives the image IM12, which is the second image specified by the user, from the terminal device 10. Furthermore, the information processing device 100 receives information (range designation information) indicating a designated range in which variation is suppressed in the first image from the terminal device 10. In FIG. 1, the information processing device 100 receives range designation information RD1 indicating the range of the user's face in the image IM11 from the terminal device 10. For example, the range designation information RD1 is information indicating the position (area) of the user's face in the image IM11.

[0029] Note that the above is merely an example, and when the information processing device 100 has already acquired various types of information to be used in generating a composite image, it may accept designation of the information to be used in generating a composite image from the terminal device 10. In this case, the information processing device 100 receives information for identifying the first image, information for identifying the second image, information for identifying the designated range, etc. from the terminal device 10. For example, the information processing device 100 receives information indicating the image selected by the user as the first image, information indicating the image selected by the user as the second image, information indicating the range selected by the user as the designated range, etc. from the terminal device 10.

[0030] Furthermore, the information processing device 100 may receive at least one of the first image, the second image, and the specified range from the terminal device 10. For example, when the information processing device 100 receives the second image from the terminal device 10, the information processing device 100 receives information for identifying the first image and the specified range from the terminal device 10. In this way, the information processing device 100 may acquire any information from the terminal device 10 used by the user, as long as the information processing device 100 can generate a composite image desired by the user.

[0031] Then, the information processing device 100 executes a generation process to generate a composite image (step S4). The information processing device 100 executes a generation process to suppress fluctuations in the specified range in the image IM11, which is the first image, based on the range designation information RD1, and generate a composite image IS1 by combining the image IM11 and the image IM12, which is the second image. For example, the information processing device 100 generates the composite image IS1 using a model M1. First, the information processing device 100 inputs input information including information related to the images IM11, IM12, and the range designation information RD1 to the model M1 (step S4-1). In FIG. 1, the information processing device 100 inputs the images IM11, IM12, the range designation information RD1, and the like as input information to the model M1.

[0032] The model M1 to which the input information has been input outputs the synthesis result (step S4-2). In FIG. 1, the model M1 to which the input information including the image IM11, the image IM12, the range designation information RD1, etc. has been input outputs image information as shown in the synthetic image IS1. In this way, by using the model M1, the information processing device 100 copies the user's face included in the image IM11 and generates the synthetic image IS1 in which the hairstyle of the user included in the image IM11 reflects the hairstyle corresponding to the hair included in the image IM12.

[0033] Then, the information processing device 100 provides the information generated by the generation process to the user (step S5). For example, the information processing device 100 transmits the composite image IS1 generated by the generation process to the terminal device 10 used by the user.

[0034] In this way, the information processing device 100 can generate an appropriate composite image by suppressing fluctuations in the user's face, which is the specified range in the first image, and generating a composite image by combining the first image and the second image. That is, the information processing device 100 can generate appropriate composite information by suppressing fluctuations in the specified range in the first object information and generating composite information by combining the first object information and the second object information. Furthermore, the information processing device 100 can provide the user with the information generated by the generation process, thereby appropriately providing the user with the composite information.

[0035] [1-2-1. Examples of models used in information processing] Assuming that the information processing system 1 performs the information processing described above, an example of a model used by the information processing device 100 when generating composite information will be described with reference to FIGS. 2 to 4. FIGS. 2 to 4 are diagrams showing an example of a model used in information processing. First, an overview of the overall configuration of a model that outputs composite information will be described with reference to FIG. 2. For example, FIG. 2 is a diagram showing an overview of the overall configuration of model M1, which is an example of a model used in information processing. Note that explanations of points similar to those described above and points similar to models used in generating conventional composite images will be omitted as appropriate.

[0036] Note that the configuration of model M1 shown in Figure 2 is just one example, and any configuration can be adopted for model M1 as long as it can suppress fluctuations in the specified range in the first target information based on the range specification information and output composite information that combines the first target information and the second target information.

[0037] For example, the model M1 may be configured using various techniques such as an autoencoder or a VAE (variational autoencoder), as appropriate. For example, the model M1 may be configured to transmit information via feature information (feature quantities) such as latent variables. For example, the model M1 may be configured to transmit information to a decoder via feature information extracted by an encoder. For example, the model M1 may be configured to transmit information to a decoder via feature information (feature quantities) such as latent variables based on random variables such as probability distribution parameters. For example, the model M1 may be configured to estimate random variables, generate latent variables from the random variables, and transmit information via the generated latent variables.

[0038] For example, model M1 may be configured to include a fully convolutional network (FCN). For example, model M1 may be configured to include a network for estimating the location of a recognition target in an image, such as the method (U-net) disclosed in Non-Patent Document 1. For example, model M1 may be configured to transmit information regarding the position of an image by transmitting information via latent variables and transmitting features of each layer. For example, model M1 may be configured to transmit features from each of multiple layers of the encoder to the corresponding layer of the decoder.

[0039] For example, the model M1 may be configured to use information obtained by downsampling an image. For example, the model M1 may be configured to use information obtained by downsampling an image including a human face. For example, the model M1 may be configured to transmit information obtained by downsampling an image including a human face to a decoder. For example, the model M1 may be configured to transmit information obtained by downsampling an image including a human face to each layer of the decoder.

[0040] Here, an overview of model M1 shown in Fig. 2 will be explained. Model M1 is a model that has learned the characteristics of the relationship between a first image including a person's head, a second image including hair, a specified range indicating the range (area) of the person's face in the first image, and synthesis of the first image and the second image that suppresses fluctuations in the person's face in the first image. For example, model M1 is a model that includes an encoder M11 that performs dimensional compression using the first image and the second image as input, and a decoder M12 that increases the dimension of the information dimensionally compressed by encoder M11.

[0041] First, we will explain the encoder M11, which is an example of a partial model of the model M1. For example, the encoder M11 is an encoder that extracts feature information corresponding to an input image from the input image. The encoder M11 also extracts feature information corresponding to a first image from a first image. The encoder M11 also extracts feature information corresponding to a second image from a second image.

[0042] The encoder M11, denoted as "encoder" in FIG. 2, has multiple layers, such as layers EL1, EL2, EL3, and EL4. Hereinafter, when describing the layers of the encoder M11, such as layers EL1, EL2, EL3, and EL4, without making any particular distinction, the layers may be referred to as "layers EL." Each layer EL of the encoder M11 may have any configuration. For example, each layer EL of the encoder M11 may use any activation function, such as a ReLU function such as LeakyReLU or a sigmoid function. Furthermore, for example, each layer EL of the encoder M11 may be configured to perform convolution processing or downsampling.

[0043] 2 illustrates four layers, EL1, EL2, EL3, and EL4, in order from shallowest to deepest, but the number of layers in the encoder M11 is not limited to four and may be any number of layers. In FIG. 2, the shallowest layer of the multiple layers in the encoder M11 is layer EL1, which is the part that learns the features of the most superficial part of the image input to the encoder M11. As the layers become deeper, such as layers EL2, EL3, and EL4, they become the parts that learn the features of parts that are not superficial parts of the image input to the encoder M11.

[0044] For example, the encoder M11 may have a configuration as shown in FIG. 3. FIG. 3 is a diagram showing an example of the configuration of the encoder M11. In FIG. 3, the top layer labeled "block 512x512x64" corresponds to layer EL1 in FIG. 2. The encoder M11 generates information indicating image features using the configuration shown in FIG. 3. Note that the configuration of the encoder M11 shown in FIG. 3 is merely an example, and the encoder M11 may have any configuration as long as it can generate desired information.

[0045] 3 corresponds to data (e.g., feature information) generated by encoder IM11 when image IM11 is input. Data ED12 represented as "e2" in Fig. 3 corresponds to data generated by the top layer (corresponding to layer EL1 in Fig. 2) represented as "block 512x512x64" of layers EL of encoder IM11 when image IM11 is input.

[0046] Returning to FIG. 2, the decoder M12, which is an example of a partial model of the model M1, will be described. For example, the decoder M12 is a decoder that outputs a composite image based on feature information generated by the encoder M11 and range designation information. The decoder M12 is a decoder that outputs a composite image based on feature information of the first image and feature information of the second image generated by the encoder M11 and range designation information. The decoder M12 is a decoder that outputs a composite image that reflects the feature information and information corresponding to the designated range in the first image.

[0047] The decoder M12, denoted as "decoder" in FIG. 2, has multiple layers, such as layers DL1, DL2, DL3, and DL4. Hereinafter, when describing the layers of the decoder M12, such as layers DL1, DL2, DL3, and DL4, without making any particular distinction, the layers may be referred to as "layers DL." Each layer DL of the decoder M12 may have any configuration. For example, each layer DL of the decoder M12 may use any activation function, such as a ReLU function such as LeakyReLU or a sigmoid function. Furthermore, for example, each layer DL of the decoder M12 may be configured to perform convolution processing, upsampling, or downsampling.

[0048] 2 shows four layers, DL1, DL2, DL3, and DL4, in order from shallowest to deepest, but the number of layers in decoder M12 is not limited to four and may be any number of layers. In FIG. 2, the shallowest layer of the multiple layers in decoder M12 is layer DL1, and layer DL1 is the part of decoder M12 that has generated information up to the surface layer.

[0049] Although not shown in FIG. 2, for example, the model M1 has a configuration similar to that of a U-net, which transmits features from each layer EL of the encoder M11 to a layer DL of the decoder M12 corresponding to that layer EL. The model M1 has a configuration for transmitting features in layer EL1 from layer EL1 of the encoder M11 to layer DL1 of the decoder M12 corresponding to that layer EL1. The model M1 also has a configuration for transmitting features in layer EL2 from layer EL2 of the encoder M11 to layer DL2 of the decoder M12 corresponding to layer EL2. The model M1 also has a configuration for transmitting features in layer EL3 from layer EL3 of the encoder M11 to layer DL3 of the decoder M12 corresponding to layer EL3. The model M1 also has a configuration for transmitting features in layer EL4 from layer EL4 of the encoder M11 to layer DL4 of the decoder M12 corresponding to layer EL4.

[0050] In the configuration in which features are transmitted from each layer EL of the encoder M11 to the layer DL of the decoder M12 corresponding to that layer EL, the model M1 also increases the weight of information transmitted from the layer EL1 of the encoder M11 to the layer DL1 of the decoder M12 corresponding to that layer EL. This allows the model M1 to transmit the superficial features (appearance) of a human face extracted by the encoder M11 to the decoder M12 as is. In other words, the model M1 can transmit the appearance of a human face contained in an image input to the encoder M11 as is to the decoder M12, and can generate a synthetic image that copies the human face contained in the input image.

[0051] Among the feature information FI in FIG. 2, the first feature F1 labeled "face" corresponds to feature information extracted from an image including a human face. For example, the first feature F1 corresponds to feature information extracted from an image IM11 including a user's face. Furthermore, among the feature information FI in FIG. 2, the second feature F2 labeled "hair" corresponds to feature information extracted from an image including hair. For example, the second feature F2 corresponds to feature information extracted from an image IM12 including hair. That is, in FIG. 2, the face and hair (hair) are encoded separately and features are extracted. This allows the information processing device 100 to freely change the combination of face and hair type (hairstyle, etc.).

[0052] Furthermore, the processing block MD of the model M1, labeled "Merge & Expand" in FIG. 2, merges and expands the first feature F1 and the second feature F2. For example, the processing block MD combines two pieces of feature information, the first feature F1 and the second feature F2, and expands them into multiple pieces of information. For example, the processing block MD expands latent variables to generate information to be transmitted as style variables. For example, the processing block MD separates information for each resolution to generate information to be transmitted as style variables. This enables the model M1 to generate high-resolution images. For example, the processing block MD expands feature information (feature quantities), such as latent variables, generated by the encoder M11 into a number corresponding to the number of layers of the decoder M12. In FIG. 2, the processing block MD expands the feature information FI into "4," which corresponds to the number of layers of the decoder M12, and generates a feature information group IG.

[0053] The model M1 then transmits each piece of information in the feature information group IG to the corresponding layer DL of the decoder M12 as a style variable. For example, the model M1 transmits information from the feature information group IG that corresponds to layer DL1 of the decoder M12 to layer DL1 as a style variable for layer DL1. For example, the model M1 transmits information from the feature information group IG that corresponds to layer DL2 of the decoder M12 to layer DL2 as a style variable for layer DL2. For example, the model M1 transmits information from the feature information group IG that corresponds to layer DL3 of the decoder M12 to layer DL3 as a style variable for layer DL3. For example, the model M1 transmits information from the feature information group IG that corresponds to layer DL4 of the decoder M12 to layer DL4 as a style variable for layer DL4.

[0054] Furthermore, a processing block FL of the model M1, denoted as a "filter" in FIG. 2, generates facial feature information FF as information for transmitting information about a human face included in an image to the decoder M12. For example, the model M1 generates facial feature information FF by inputting information into the processing block FL. The model M1 generates facial feature information FF using information from layer EL1 of the encoder M11, which has learned the features of the most superficial part of the image, and range designation information indicating the area to be copied (designated range). For example, the processing block FL increases the weight of values ​​corresponding to areas containing a human face in the information from layer EL1 and decreases the weight of values ​​corresponding to other areas, thereby generating facial feature information FF for transmitting the features of a human face to the decoder M12. For example, the processing block FL multiplies values ​​corresponding to areas containing a human face in the information from layer EL1 by 1 and multiplies other areas by less than 1 (for example, 0) to generate facial feature information FF.

[0055] For example, the model M1 generates facial feature information FF as information for transmitting information about a human facial structure to the decoder M12. For example, the model M1 generates facial feature information FF as information for transmitting information indicating the boundaries of hair to the decoder M12. In Figure 2, the model M1 generates facial feature information FF, which is information obtained by downsampling the image IM11 including the user's face, using information on the layer EL1 of the encoder M11 to which the image IM11 is input and range designation information RD1 indicating the range in the image IM11 that includes the user's face.

[0056] For example, the model M1 generates facial feature information FF through processing as shown in Fig. 4. Fig. 4 is a diagram showing an example of processing in the model M1. Data ED11 in Fig. 4 corresponds to data ED11 denoted as "e1" in Fig. 3. Data ED12 in Fig. 4 corresponds to data ED12 denoted as "e2" in Fig. 3.

[0057] 4 indicates data (e.g., feature information) generated by encoder IM11 when image IM12 is input. Data ED22 indicated as "e2" in Fig. 3 indicates data generated by the top layer (corresponding to layer EL1 in Fig. 2) indicated as "block 512x512x64" of layers EL of encoder IM11 when image IM12 is input.

[0058] 4, data ID12 represented as "m3" corresponds to feature information generated by encoder IM11. For example, model M1 generates data ID12 indicating the features of the input image using data ED11 generated by encoder IM11 when image IM11 is input, and data ED21 generated by encoder IM11 when image IM12 is input.

[0059] Data ID11, denoted as "m1" in FIG. 4, corresponds to feature information related to a person's face in the image. Model M1 generates data ID11 related to a user's face using data ED12 and range designation information RD1. For example, model M1 generates data ID11 in FIG. 4, which corresponds to facial feature information FF, using data ED12 in FIG. 4, which corresponds to information on layer EL1 of encoder M11 to which image IM11 is input, and range designation information RD1. For example, model M1 generates facial feature information FF including information indicating the sculptural features of the user's face in image IM11. For example, model M1 generates facial feature information FF including information indicating the boundary between the user's hair and face in image IM11. Note that the information processing device 100 may generate range designation information RD1 from an image including a person's face by process SP1, denoted as "split" in FIG. 4. Note that the above is merely an example, and range designation information RD1 may be any information indicating the range of a face in an image including a person's face, and may be generated by any method.

[0060] The model M1 transmits facial feature information FF to the decoder M12. For example, the model M1 transmits facial feature information FF indicating the sculptural features of a person's face and the boundary between the hair and the face to each layer DL of the decoder M12. In FIG. 2, the model M1 transmits facial feature information FF indicating the sculptural features of the user's face in the image IM11 and the boundary between the user's hair and the face in the image IM11 to each layer DL of the decoder M12. For example, the model M1 increases the weight of the facial feature information FF transmitted to the surface layer DL1 of the decoder M12 and transmits it to each layer DL of the decoder M12. This allows the information processing device 100 to improve the reproducibility of the face. The information processing device 100 also allows the user to edit the hairstyle.

[0061] Conventional synthetic image generation has had problems such as the face being slightly different even though the overall shape can be reproduced. On the other hand, the information processing device 100 can generate an appropriate synthetic image by using the model M1 described above. Specifically, the information processing device 100 copies the face part in a form that is independent of learning data (learning data) and synthesizes the quality and shape of the hair, thereby generating an image in which the hairstyle has been appropriately converted.

[0062] In this way, the information processing device 100 receives a first merging target, a second merging target, and a designation of a range in which variation tolerance is low for the first merging target. Then, the information processing device 100 generates merging information by merging the first merging target and the second merging target while suppressing variation in the designated range of the first merging target.

[0063] For example, a conventional model is trained to output an image with a different hairstyle when a face image and a hair image are input. This causes the face to be influenced by the hair image and change, which can negatively impact the user's impression. In other words, parts of the first composite target other than the part to be composited with the second composite target may change depending on the characteristics of the second composite target.

[0064] Meanwhile, the information processing device 100 acquires the output of a layer (within a predetermined range from the input layer) of the encoder M11 of the model M1 that is relatively close to the input device and that learns the features of the surface portion, and inputs the features within the specified range of the output to a predetermined layer of the decoder M12. For example, the information processing device 100 weights the output of the predetermined layer of the decoder M12. Specifically, the information processing device 100 transmits the transmission information of the surface portion of the model M1 to a layer close to the decoder M12 output layer with a stronger weight for the specified range. This allows the information processing device 100 to suppress changes in parts of the first synthesis target other than the parts to be synthesized with the second synthesis target, depending on the features of the second synthesis target. In this way, the information processing device 100 learns the features of the first image and the second image in multiple layers, and generates a synthesized image by synthesizing the first image and the second image, with emphasis on the surface expression of the first image, in areas with low tolerance for variation.

[0065] [1-2-2. Other examples] Note that the processing using the above-mentioned model M1 is merely an example, and the information processing device 100 may generate the composite information by any processing as long as it can suppress fluctuations in the specified range in the first target information and generate composite information that combines the first target information and the second target information.

[0066] For example, the information processing device 100 may use a model (model M2) that inputs a first image and a second image and outputs a first composite image by combining the first image and the second image. In this case, the information processing device 100 may edit the first composite image output by the model M2 based on range designation information that indicates a designated range that is the range of the user's face in the first image, thereby generating a second composite image in which fluctuations in the designated range in the first image are suppressed.

[0067] For example, the information processing device 100 may replace an area in the first composite image output by the model M2 that corresponds to an area where a human face is located in the first image with an image of the area where the human face is located in the first image. In this way, the information processing device 100 may generate a second composite image in which the area where the human face is located in the first image is replaced with a specified area in the first image, suppressing fluctuations in the area where the face is located in the composite image. For example, the information processing device 100 may generate a second composite image in which the image of the area where the human face is located in the first image is copied into an area in the first composite image that corresponds to an area where the human face is located in the first image.

[0068] Furthermore, the information processing device 100 may accept a user's evaluation of the composite information. For example, the information processing device 100 accepts an evaluation of the composite information from a user who has provided the composite information (also referred to as a "recipient user"). The information processing device 100 transmits a composite image (also referred to as a "provided composite image") obtained by combining a first image including the face of the recipient user with a second image including hair in a hairstyle specified by the recipient user to the terminal device 10 used by the recipient user. The information processing device 100 also transmits content for accepting a user's evaluation of the provided composite image (also referred to as "content for receiving evaluation") to the terminal device 10 used by the recipient user.

[0069] The terminal device 10 used by the destination user displays the received provided composite image together with content for accepting an evaluation of the provided composite image (content for receiving an evaluation). The terminal device 10 displays the content for receiving an evaluation, which includes a numeric input box for inputting a score from 0 to 100 and a button for transmitting the score (also referred to as an "evaluation transmission button"). When the destination user inputs a numeric value into the numeric input box of the content for receiving an evaluation and selects the evaluation transmission button, the terminal device 10 transmits the numeric value input by the destination user to the information processing device 100 as information indicating the user's evaluation of the composite information (also referred to as "evaluation information").

[0070] The information processing device 100 receives evaluation information indicating an evaluation by a destination user of the provided composite image. The information processing device 100, which has received the evaluation information from the terminal device 10, registers the received evaluation information in the storage unit 120. The information processing device 100 may generate learning data based on the evaluation by the destination user.

[0071] For example, when the evaluation by the destination user satisfies a predetermined standard, information processing device 100 may add the provided composite image provided to the destination user to the training data as a positive example. For example, when input information at the time of generating a provided composite image determined as a positive example is input, information processing device 100 may update model M1 by performing a learning process to output the provided composite image. For example, when input information at the time of generating a provided composite image determined as a positive example is input, information processing device 100 may update model M1 by performing a learning process to output the provided composite image.

[0072] For example, when the score of the destination user is equal to or greater than a predetermined threshold, information processing device 100 may add the provided composite image provided to the destination user as a positive example to the learning data. For example, when the score of the destination user is equal to or greater than a predetermined threshold, information processing device 100 may add a combination of the provided composite image provided to the destination user and input information used to generate the provided composite image to the learning data.

[0073] For example, if the evaluation by the destination user does not satisfy a predetermined standard, the information processing device 100 may request the destination user to edit the provided composite image. In this case, the information processing device 100 may use the image edited by the destination user (also referred to as an "edited composite image") as learning data. In this case, when input information at the time of generating a provided composite image corresponding to the edited composite image is input, the information processing device 100 may update the model M1 by performing a learning process so as to output the edited composite image. For example, when input information at the time of generating a provided composite image corresponding to the edited composite image is input, the information processing device 100 may update the model M1 by performing a learning process so as to output the edited composite image. Note that the above-described process is merely an example, and the information processing device 100 may perform a learning process to learn a model such as the model M1 through various processes.

[0074] 2. Configuration of Information Processing Device Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 6, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may also have an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0075] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network via a wired or wireless connection, and transmits and receives information between the terminal device 10 and the information providing device 50.

[0076] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in Fig. 6, the storage unit 120 according to the embodiment includes a learning data storage unit 121, a model information storage unit 122, and a synthesis result information storage unit 123. The storage unit 120 may store various types of information, not limited to the above.

[0077] (Learning data storage unit 121) The learning data storage unit 121 according to the embodiment stores various information related to data used for learning. Note that, when acquiring a learning model such as the model M1 from the information providing device 50, the information processing device 100 does not need to include the learning data storage unit 121.

[0078] The learning data storage unit 121 stores learning data (dataset) used for learning. FIG. 7 is a diagram illustrating an example of the learning data storage unit according to the embodiment. For example, the learning data storage unit 121 stores various information related to various data such as learning data used for learning and evaluation data used for accuracy evaluation (measurement). FIG. 7 illustrates an example of the learning data storage unit 121 according to the embodiment. In the example of FIG. 7, the learning data storage unit 121 includes items such as "dataset ID," "data ID," "input information," "correct answer information," and "date and time."

[0079] "Dataset ID" indicates identification information for identifying a dataset. "Data ID" indicates identification information for identifying data. "Input information" indicates input information for data (teaching data) identified by the data ID.

[0080] The "correct answer information" indicates correct answer information (synthetic information) of the corresponding data (teaching data). For example, the "correct answer information" may be synthetic information (synthetic image) generated when the corresponding input information is used.

[0081] Furthermore, "date and time" indicates the time (date and time) related to the corresponding data. Note that, although "DA1" and the like are illustrated in the example of Fig. 7, "date and time" may be a specific date and time such as "May 27, 2022, 17:48:37", or information indicating from which model learning the data began to be used, such as "Use started from model learning of version XX", may be stored.

[0082] The example in Figure 7 shows that a dataset (dataset DS1) identified by a dataset ID "DS1" includes multiple data identified by data IDs "DID1," "DID2," "DID3," etc. For example, each piece of data (learning data) identified by a data ID "DID1," "DID2," "DID3," etc. is training data used for model training.

[0083] For example, data identified by the data ID "DID1" indicates that input information is data DT1, and that the synthesized information to be generated when the data DT1 is input to the model M1 is answer information LB1.

[0084] The training data storage unit 121 may store various information depending on the purpose, without being limited to the above. For example, the training data storage unit 121 may store data in a manner that allows identification of whether each piece of data is training data or evaluation data. For example, the training data storage unit 121 stores training data and evaluation data in a manner that allows identification. The training data storage unit 121 may store information that identifies whether each piece of data is training data or evaluation data. The information processing device 100 trains a model based on each piece of data used as training data and correct answer information. The information processing device 100 calculates the accuracy of the model based on each piece of data used as evaluation data and correct answer information. The information processing device 100 calculates the accuracy of the model by collecting a result of comparing the output result output by the model when evaluation data is input with the correct answer information.

[0085] (Model information storage unit 122) The model information storage unit 122 according to the embodiment stores information about models. For example, the model information storage unit 122 stores information (model data) about a trained model (model) trained (generated) through a training process. FIG. 8 is a diagram illustrating an example of a model information storage unit according to the embodiment. In the example shown in FIG. 8, the model information storage unit 122 includes items such as "model ID," "use," "partial model ID," "partial use," and "model data." Note that while FIG. 8 illustrates only model M1, multiple pieces of model information such as models M2 and M3 may be stored.

[0086] "Model ID" indicates identification information for identifying a model. For example, a model identified by model ID "M1" corresponds to model M1 shown in the example of Figure 1. "Use" indicates the use of the corresponding model. "Partial model ID" indicates identification information for identifying a partial model of the corresponding model. For example, a partial model identified by partial model ID "M11" corresponds to encoder M11 shown in the example of Figure 1. "Partial use" indicates the use of the corresponding partial model. Furthermore, "model data" indicates data of the corresponding partial model. For example, "model data" includes information including nodes in each layer, functions employed by each node, connection relationships between nodes, and connection coefficients set for connections between nodes.

[0087] 8, a model M1 is used for image synthesis. The model M1 includes an encoder M11, a decoder M12, etc. For example, the encoder M11 is a partial model (encoder) that extracts features from an image, and the decoder M12 is a partial model (decoder) that generates an image using the features extracted by the encoder M11.

[0088] For example, the model (encoder M11) identified by the partial model ID "M11" has a use of "feature extraction," indicating that it is used to extract features from input image information. Also, in FIG. 8, the encoder M11 indicates that the model data is model data MDT11. Also, the model (decoder M12) identified by the partial model ID "M12" has a use of "image generation," indicating that it is used to extract features from input image information. Also, in FIG. 8, the decoder M12 indicates that the model data is model data MDT12.

[0089] The model information storage unit 122 may store various types of model information according to the purpose, not limited to the above. The model information storage unit 122 may store data used for learning (learning data) in association with a trained model (model).

[0090] (Synthesis result information storage unit 123) The synthesis result information storage unit 123 according to the embodiment stores various information related to the processing results. Fig. 9 is a diagram illustrating an example of the synthesis result information storage unit according to the embodiment. The synthesis result information storage unit 123 shown in Fig. 9 includes items such as "processing result ID," "pre-synthesis information," and "synthesis result information."

[0091] "Processing result ID" indicates identification information that identifies the processing result. "Pre-composition information" indicates input information, etc., used to perform the composition. "Pre-composition information" includes items such as "first image," "second image," and "range specification information." "Composition result information" indicates the result (information) of the generation process, such as a composite image obtained by the generation process.

[0092] For example, in the process identified by the processing result ID "RS1", the first image IM11, the second image IM12, and the range designation information RD1 were used, and a composite image IS1 was obtained as the processing result.

[0093] The synthesis result information storage unit 123 is not limited to the above, and may store various types of information depending on the purpose.

[0094] (control unit 130) 6, the control unit 130 is a controller, and is realized by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs (corresponding to examples of information processing programs) stored in a storage device inside the information processing device 100 using a RAM as a work area. The control unit 130 is also a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0095] 6, control unit 130 has an acquisition unit 131, a learning unit 132, a determination unit 133, a generation unit 134, and a provision unit 135, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of control unit 130 is not limited to the configuration shown in FIG. 6, and may be any other configuration that performs the information processing described below.

[0096] (Acquisition part 131) The acquisition unit 131 acquires various pieces of information. The acquisition unit 131 acquires various pieces of information from an external device such as the terminal device 10. The acquisition unit 131 also acquires various pieces of information from the storage unit 120. The acquisition unit 131 also acquires various pieces of information from the learning data storage unit 121, the model information storage unit 122, the synthesis result information storage unit 123, etc.

[0097] The acquisition unit 131 receives various pieces of information from an external information processing device via the communication unit 110. The acquisition unit 131 stores the received information in the storage unit 120. The acquisition unit 131 receives various pieces of information from the terminal device 10 or the information providing device 50. The acquisition unit 131 acquires information from the terminal device 10 used by the user. For example, the acquisition unit 131 acquires various pieces of information from the information providing device 50. For example, the acquisition unit 131 receives learning data used for learning a model from the information providing device 50. For example, the acquisition unit 131 receives a model from the information providing device 50.

[0098] The acquisition unit 131 acquires first object information which is a first object to be combined. The acquisition unit 131 acquires second object information which is a second object to be combined. The acquisition unit 131 acquires range designation information which indicates a designated range which is designated in the first object information as a range in which fluctuations are to be suppressed. The acquisition unit 131 acquires range designation information which indicates a designated range which is designated in the first object information as a range to be copied to the combined information.

[0099] The acquisition unit 131 acquires a first image that is an image serving as first object information. The acquisition unit 131 acquires a second image that is an image serving as second object information. The acquisition unit 131 acquires range designation information that indicates a designated range in the first image.

[0100] The acquisition unit 131 acquires a first image including a person's head. The acquisition unit 131 acquires range designation information indicating the range of the person's face in the first image. The acquisition unit 131 acquires a second image including hair. The acquisition unit 131 acquires second target information designated by the user.

[0101] The acquisition unit 131 acquires a first image, which is first object information obtained by capturing an image of the user's head. The acquisition unit 131 acquires a second image, which is second object information including hair designated by the user. The acquisition unit 131 acquires range designation information indicating a designated range, which is the range of the user's face in the first image.

[0102] (Learning Section 132) The learning unit 132 executes a learning process to learn a learning model (model). Note that the information processing device 100 does not need to include the learning unit 132 when acquiring a learning model such as the model M1 from the information providing device 50.

[0103] For example, the learning unit 132 executes the learning process based on various information acquired by the acquisition unit 131. The learning unit 132 executes the learning process based on information from an external information processing device and information stored in the storage unit 120. The learning unit 132 executes the learning process based on information stored in the model information storage unit 122. The learning unit 132 stores the model generated by learning in the model information storage unit 122.

[0104] The learning unit 132 performs a learning process. The learning unit 132 performs various types of learning. The learning unit 132 learns various types of information based on the information acquired by the acquisition unit 131. The learning unit 132 learns (generates) a model. The learning unit 132 learns various types of information such as a model. The learning unit 132 generates a model through learning. The learning unit 132 learns the model using various machine learning techniques. For example, the learning unit 132 learns parameters of the model (network). The learning unit 132 learns the model using various machine learning techniques.

[0105] The learning unit 132 generates various learning models such as the model M1. The learning unit 132 learns network parameters. For example, the learning unit 132 learns network parameters of various learning models such as the model M1. The learning unit 132 performs a learning process using learning data stored in the model information storage unit 122, thereby generating various learning models such as the model M1. For example, the learning unit 132 generates a model used to generate synthetic information. The learning unit 132 generates various learning models such as the model M1 by learning network parameters of various learning models such as the model M1.

[0106] The learning unit 132 performs a learning process based on the learning data (teacher data) stored in the model information storage unit 122. The learning unit 132 performs a learning process using the learning data stored in the model information storage unit 122, thereby generating various learning models such as model M1.

[0107] For example, the learning unit 132 performs a learning process using a method such as backpropagation (backpropagation method) so that the synthesis information output by the model M1 approaches correct answer information (information indicating a desirable synthesis result) associated with the input information (first target information, second target information, and range designation information) input to the model M1. For example, the learning unit 132 performs a learning process so that the synthesis image output by the model M1 to which input information including the first image, the second image, and the range designation information has been input approaches correct answer information associated with the input information.

[0108] For example, the learning unit 132 adjusts the values ​​of weights (i.e., connection coefficients) that are taken into account when values ​​are transmitted between nodes through the learning process. In this way, the learning unit 132 learns the model M1 through processing such as backpropagation, which corrects parameters (connection coefficients) so as to reduce the error between the output of the model M1 and the correct information corresponding to the input. For example, the learning unit 132 generates the model M1 by performing processing such as backpropagation so as to minimize a predetermined loss function. This allows the learning unit 132 to perform a learning process to learn the parameters of the model M1.

[0109] The model learning method is not limited to the above-described method, and any known technique can be applied. Each model may be generated using various conventional machine learning techniques, as appropriate. For example, the model may be generated using a supervised machine learning technique such as SVM (Support Vector Machine). Furthermore, the model may be generated using an unsupervised machine learning technique. For example, the model may be generated using a deep learning technique. For example, the model may be generated using various deep learning techniques, such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), or CNN (Convolutional Neural Network). The above description of the model generation is merely an example, and the model may be generated using a learning technique appropriately selected depending on the obtainable information, etc. In other words, the learning unit 132 may generate the model M1 using any technique as long as the model M1 can be trained to output a score corresponding to the correct answer information when input information included in the training data is input.

[0110] As described above, the learning method used by the learning unit 132 is not particularly limited, but for example, learning data in which data (input information) and its correct answer information (output information) are linked may be prepared, and the learning data may be input to a computational model based on a multilayer neural network for learning. Alternatively, a method based on DNNs such as CNN and 3D-CNN may be used. When targeting time-series data such as speech, the learning unit 132 may use a method based on a recurrent neural network (RNN) or LSTM (Long Short-Term Memory units), which is an extension of an RNN.

[0111] (Decision unit 133) The determination unit 133 executes a determination process to determine various pieces of information. The determination unit 133 stores the information determined by the determination process in the storage unit 120. For example, the determination unit 133 executes the determination process based on various pieces of information acquired by the acquisition unit 131. The determination unit 133 executes the determination process based on various pieces of information stored in the storage unit 120. For example, the determination unit 133 executes the determination process based on various pieces of information received from an external information processing device.

[0112] (Generation unit 134) The generation unit 134 executes a generation process to generate various types of information. For example, the generation unit 134 executes the generation process based on various types of information acquired by the acquisition unit 131. For example, the generation unit 134 executes the generation process based on information stored in the storage unit 120.

[0113] The generation unit 134 executes the generation process using the model learned by the learning unit 132. For example, the generation unit 134 executes the generation process based on information determined by the determination unit 133. The generation unit 134 generates synthesis information using the model stored in the model information storage unit 122.

[0114] The generation unit 134 suppresses fluctuations in the designated range in the first object information based on the range designation information, and generates combined information by combining the first object information and the second object information. The generation unit 134 generates combined information by copying the designated range in the first object information. The generation unit 134 suppresses fluctuations in the designated range in the first image, and generates a combined image that is combined information by combining the first image and the second image.

[0115] The generation unit 134 generates a composite image in which fluctuations in a range corresponding to a specified range of the head of a person included in the first image are suppressed. The generation unit 134 generates a composite image in which fluctuations in the face of the person in the first image are suppressed. The generation unit 134 generates a composite image in which hair included in the second image is reflected in the head of the person included in the first image.

[0116] The generation unit 134 generates the composite information using a learning model that has learned characteristics of the association between the first object, the second object, and the specified range, and the composition of the first object and the second object with the fluctuation of the specified range of the first object suppressed. The generation unit 134 generates the composite information using a learning model that includes an encoder that performs dimensional compression using the first object information and the second object information as input, and a decoder that increases the dimension of the information dimensionally compressed by the encoder.

[0117] The generation unit 134 generates the combined information using a learning model including an encoder that extracts feature information from the first object information and the second object information, and a decoder that outputs combined information based on the feature information and range designation information. The generation unit 134 generates the combined information using a learning model including an encoder and a decoder that outputs combined information that reflects the feature information and information corresponding to the designated range in the first object information.

[0118] The generation unit 134 receives the first image and the second image as input and generates a composite image using a learning model including an encoder that extracts feature information and a decoder that outputs a composite image that is composite information obtained by combining the first image and the second image based on the feature information and range designation information. The generation unit 134 uses the first image including a human head, the second image, the range designation information, and the learning model to generate a composite image in which fluctuations in the range corresponding to the designated range of the human head included in the first image are suppressed.

[0119] The generation unit 134 uses the first image, the second image, range designation information indicating the range of the person's face in the first image, and a learning model to generate a composite image in which fluctuations in the person's face in the first image are suppressed. The generation unit 134 uses the first image, the second image including hair, range designation information, and a learning model to generate a composite image in which the hair included in the second image is reflected on the head of the person included in the first image.

[0120] The generation unit 134 generates composite information using the second target information acquired from the user. The generation unit 134 generates a composite image, which is composite information that reflects the hair included in the second image on the head of the user included in the first image. The generation unit 134 copies the user's face included in the first image and generates a composite image that reflects a hairstyle corresponding to the hair included in the second image on the hairstyle of the user included in the first image.

[0121] The generation unit 134 generates content. For example, the generation unit 134 generates a screen (content) to be provided to the terminal device 10 by appropriately using various technologies such as Java (registered trademark). Note that the generation unit 134 may generate the screen (content) to be provided to the terminal device 10 based on the format of CSS, JavaScript (registered trademark), or HTML. Furthermore, for example, the generation unit 134 may generate the screen (content) in various formats such as JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), or PNG (Portable Network Graphics). The generation unit 134 generates content for receiving evaluations.

[0122] (Provider 135) The providing unit 135 provides various types of information. The providing unit 135 transmits the various types of information to an external information processing device via the communication unit 110. The providing unit 135 transmits the various types of information to the terminal device 10 and the information providing device 50. The providing unit 135 transmits the results of the generation process to the terminal device 10.

[0123] The providing unit 135 provides the information determined by the determining unit 133. The providing unit 135 provides the information generated by the generating unit 134. The providing unit 135 provides the composite information to the terminal device 10 used by the user. The providing unit 135 transmits the composite image IS1 to the terminal device 10 used by the user. The providing unit 135 transmits the evaluation receiving content generated by the generating unit 134 to the terminal device 10 used by the user.

[0124] [3. Processing Procedure] Next, the procedure of information processing executed by the information processing device 100 according to the embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the flow of information processing executed by the information processing device.

[0125] 10, the information processing device 100 acquires first object information that is a first object to be synthesized (step S101). For example, the information processing device 100 acquires a first image that is an image as the first object information.

[0126] Furthermore, the information processing device 100 acquires second object information, which is a second object to be synthesized (step S102). For example, the information processing device 100 acquires a second image, which is an image, as the second object information.

[0127] Furthermore, the information processing device 100 acquires range designation information indicating a designated range designated as a range in which fluctuations are to be suppressed in the first object information (step S103). For example, the information processing device 100 acquires range designation information indicating a designated range in the first image.

[0128] Then, the information processing device 100 suppresses fluctuations in the specified range in the first object information based on the range specification information, and generates composite information by combining the first object information and the second object information (step S104). For example, the information processing device 100 suppresses fluctuations in the specified range in the first image, and generates a composite image that is composite information by combining the first image and the second image.

[0129] [4. Effects] As described above, the information processing device 100 according to the embodiment includes an acquisition unit 131 and a generation unit 134. The acquisition unit 131 acquires first object information, which is a first object to be combined, second object information, which is a second object to be combined, and range designation information, which indicates a designated range designated as a range in which variation is to be suppressed in the first object information. The generation unit 134 suppresses variation in the designated range in the first object information based on the range designation information, and generates combined information by combining the first object information and the second object information.

[0130] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by suppressing fluctuations in the specified range in the first target information based on the range specification information and generating composite information by combining the first target information and the second target information.

[0131] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires first target information, second target information, and range designation information indicating a designated range designated in the first target information as a range to be copied to the combined information. The generation unit 134 generates combined information by copying the designated range in the first target information.

[0132] In this way, the information processing apparatus 100 according to the embodiment can generate appropriate combined information by generating combined information by copying the designated range in the first object information.

[0133] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a first image as the first object information, a second image as the second object information, and range designation information indicating a designated range in the first image. The generation unit 134 suppresses fluctuations in the designated range in the first image and generates a composite image as composite information obtained by combining the first image and the second image.

[0134] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by suppressing fluctuations in the specified range in the first image and generating a composite image that is composite information obtained by combining the first image and the second image.

[0135] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a first image including a person's head. The generation unit 134 generates a composite image in which fluctuations in a range corresponding to a specified range of the person's head included in the first image are suppressed.

[0136] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by generating a composite image that suppresses fluctuations in the range of the person's head included in the first image that corresponds to the specified range.

[0137] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires range designation information indicating the range of a human face in the first image. The generation unit 134 generates a composite image in which fluctuations in the human face in the first image are suppressed.

[0138] In this way, the information processing apparatus 100 according to the embodiment can generate appropriate composite information by generating a composite image in which fluctuations in the human face in the first image are suppressed.

[0139] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a second image including hair. The generation unit 134 generates a composite image in which the hair included in the second image is reflected on the head of the person included in the first image.

[0140] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by generating a composite image that reflects the hair contained in the second image on the head of the person contained in the first image.

[0141] Furthermore, in the information processing device 100 according to the embodiment, the generation unit 134 generates synthesis information using a learning model that has learned the characteristics of the relationship between the first object, the second object, and the specified range, and the synthesis of the first object and the second object with the fluctuation of the specified range in the first object suppressed.

[0142] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by using a learning model that has learned the characteristics of the relationship between the first object, the second object, and the specified range, and the synthesis of the first object and the second object while suppressing fluctuations in the specified range of the first object.

[0143] Furthermore, in the information processing device 100 according to the embodiment, the generation unit 134 generates synthetic information using a learning model including an encoder that performs dimensional compression using the first target information and the second target information as input, and a decoder that increases the dimension of the information dimensionally compressed by the encoder.

[0144] In this way, the information processing device 100 according to the embodiment can generate appropriate synthetic information by generating synthetic information using a learning model including an encoder that performs dimensional compression using the first target information and the second target information as input, and a decoder that increases the dimension of the information that has been dimensionally compressed by the encoder.

[0145] Furthermore, in the information processing device 100 according to the embodiment, the generation unit 134 generates synthetic information using a learning model including an encoder that extracts feature information from the first target information and the second target information, and a decoder that outputs synthetic information based on the feature information and range designation information.

[0146] In this way, the information processing device 100 according to the embodiment can generate appropriate synthetic information by generating synthetic information using a learning model including an encoder that extracts feature information from the first target information and the second target information, and a decoder that outputs synthetic information based on the feature information and range designation information.

[0147] In addition, in the information processing device 100 according to the embodiment, the generation unit 134 generates synthetic information using a learning model including an encoder and a decoder that outputs synthetic information that reflects information corresponding to a specified range in the first target information and feature information.

[0148] In this way, the information processing device 100 according to the embodiment can generate appropriate synthetic information by generating synthetic information using a learning model including an encoder and a decoder that outputs synthetic information that reflects information corresponding to a specified range in the first target information and feature information.

[0149] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a first image that is an image serving as first object information, a second image that is an image serving as second object information, and range designation information that indicates a designated range in the first image. The generation unit 134 generates a composite image using a learning model that includes an encoder that receives the first image and the second image as input and extracts feature information, and a decoder that outputs a composite image that is composite information obtained by combining the first image and the second image based on the feature information and the range designation information.

[0150] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by generating a composite image using a learning model including an encoder that extracts feature information from the first image and the second image as input, and a decoder that outputs a composite image, which is composite information obtained by combining the first image and the second image, based on the feature information and range designation information.

[0151] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a first image including a human head. The generation unit 134 uses the first image including the human head, the second image, range designation information, and a learning model to generate a composite image in which fluctuations in a range corresponding to the designated range of the human head included in the first image are suppressed.

[0152] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by using a first image including a person's head, a second image, range designation information, and a learning model to generate a composite image that suppresses fluctuations in the range of the person's head included in the first image that corresponds to the designated range.

[0153] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires range designation information indicating the range of a human face in the first image. The generation unit 134 generates a composite image in which fluctuations in the human face in the first image are suppressed, using the first image, the second image, the range designation information indicating the range of the human face in the first image, and a learning model.

[0154] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by using the first image, the second image, range designation information indicating the range of the human face in the first image, and a learning model to generate a composite image that suppresses fluctuations in the human face in the first image.

[0155] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a second image including hair. The generation unit 134 uses the first image, the second image including hair, the range designation information, and a learning model to generate a composite image in which the hair included in the second image is reflected on the head of the person included in the first image.

[0156] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by using the first image, the second image including hair, range designation information, and a learning model to generate a composite image that reflects the hair included in the second image on the head of the person included in the first image.

[0157] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires second object information designated by the user. The generation unit 134 generates composite information using the second object information acquired from the user.

[0158] In this way, the information processing apparatus 100 according to the embodiment can generate appropriate composite information by using the second target information acquired from the user to generate composite information.

[0159] Moreover, the information processing device 100 according to the embodiment includes a providing unit 135. The providing unit 135 provides the combined information to the terminal device 10 used by the user.

[0160] In this way, the information processing device 100 according to the embodiment can provide appropriately generated composite information to the user by providing the composite information to the terminal device 10 used by the user.

[0161] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a first image, which is first object information obtained by capturing an image of the user's head, a second image, which is second object information including hair specified by the user, and range designation information indicating a designated range, which is the range of the user's face in the first image. The generation unit 134 generates a composite image, which is composite information that reflects the hair included in the second image on the user's head included in the first image.

[0162] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by generating a composite image that is composite information that reflects the hair contained in the second image on the user's head contained in the first image.

[0163] In addition, in the information processing device 100 according to the embodiment, the generation unit 134 copies the face of the user included in the first image and generates a composite image in which the hairstyle of the user included in the first image is reflected in a hairstyle corresponding to the hair included in the second image.

[0164] In this way, the information processing device 100 according to the embodiment can generate appropriate composite information by copying the user's face included in the first image and generating a composite image in which the hairstyle of the user included in the first image reflects the hairstyle corresponding to the hair included in the second image.

[0165] [5. Hardware Configuration] The information processing device 100 according to the embodiment described above is realized by, for example, a computer 1000 configured as shown in Fig. 11. Fig. 11 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a HDD (Hard Disk Drive) 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0166] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0167] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via the network N and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the network N.

[0168] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0169] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0170] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via the network N.

[0171] Although some of the embodiments and variants of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that have undergone various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the Disclosure of the Invention.

[0172] [6. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0173] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0174] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0175] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit," etc. For example, an acquisition unit can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0176] 1. Information Processing Systems 100 Information processing device 121 Learning data storage unit 122 Model information storage unit 123 Synthesis result information storage unit 130 Control Unit 131 Acquisition Department 132 Learning Department 133 Decision Section 134 Generation part 135 Provision Department 10 Terminal Equipment 50 Information provision device N Network

Claims

1. an acquisition unit that acquires first target information, which is a first target to be combined, second target information, which is a second target to be combined designated by a user, and range designation information, which indicates a designated range designated as a range in which fluctuations are to be suppressed in the first target information; a generating unit that suppresses fluctuations in the specified range in the first target information based on the range specifying information and generates composite information by combining the first target information and the second target information acquired from the user; a providing unit that provides the combined information and evaluation receiving content that receives the user's evaluation of the combined information to a terminal device used by the user; Equipped with The acquisition unit receiving, from the terminal device used by the user, information indicating the user's evaluation of the composite information received from the user through the evaluation receiving content; 1. An information processing device comprising:

2. The acquisition unit acquiring the first target information, the second target information, and the range designation information indicating the designated range designated in the first target information as a range to be copied to the combined information; The generation unit generating the composite information by copying the specified range in the first target information; 2. The information processing apparatus according to claim 1, wherein:

3. The acquisition unit acquiring a first image as the first target information, a second image as the second target information, and the range designation information indicating the designated range in the first image; The generation unit suppressing fluctuations in the specified range in the first image and generating a composite image, which is the composite information obtained by combining the first image and the second image; 2. The information processing apparatus according to claim 1, wherein:

4. The acquisition unit acquiring the first image including a head of a person; The generation unit generating the composite image in which fluctuations in a range corresponding to the specified range of the person's head included in the first image are suppressed; 4. The information processing apparatus according to claim 3,

5. The acquisition unit acquiring the range designation information indicating the range of the person's face in the first image; The generation unit generating the composite image in which fluctuations in the person's face in the first image are suppressed; 5. The information processing apparatus according to claim 4,

6. The acquisition unit acquiring the second image including hair; The generation unit generating the composite image in which the hair included in the second image is reflected on the head of the person included in the first image; 5. The information processing apparatus according to claim 4,

7. The generation unit The synthesis information is generated using a learning model that has learned characteristics of the association between the first object, the second object, and the specified range, and a synthesis of the first object and the second object in which fluctuations in the specified range of the first object are suppressed.

2. The information processing apparatus according to claim 1, wherein:

8. The generation unit The synthetic information is generated using the learning model including an encoder that performs dimensional compression using the first target information and the second target information as input, and a decoder that increases the dimension of the information dimensionally compressed by the encoder.

8. The information processing apparatus according to claim 7,

9. The generation unit generating the synthesized information using the learning model including the encoder that extracts feature information from the first target information and the second target information, and the decoder that outputs the synthesized information based on the feature information and the range designation information; 9. The information processing apparatus according to claim 8,

10. The generation unit The synthesized information is generated using the learning model including the encoder and the decoder that outputs synthesized information that reflects the information corresponding to the specified range in the first target information and the feature information.

10. The information processing apparatus according to claim 9,

11. The acquisition unit acquiring a first image as the first target information, a second image as the second target information, and the range designation information indicating the designated range in the first image; The generation unit The synthesized image is generated using the learning model including the encoder that extracts feature information from the first image and the second image as input, and the decoder that outputs a synthesized image that is synthesis information obtained by synthesizing the first image and the second image based on the feature information and the range designation information.

9. The information processing apparatus according to claim 8,

12. The acquisition unit acquiring the first image including a head of a person; The generation unit Using the first image including the person's head, the second image, the range designation information, and the learning model, the synthetic image is generated in which fluctuations in a range corresponding to the designated range of the person's head included in the first image are suppressed.

12. The information processing apparatus according to claim 11,

13. The acquisition unit acquiring the range designation information indicating the range of the person's face in the first image; The generation unit generating the synthetic image in which fluctuations in the person's face in the first image are suppressed using the first image, the second image, the range designation information indicating the range of the person's face in the first image, and the learning model; 13. The information processing apparatus according to claim 12.

14. The acquisition unit acquiring the second image including hair; The generation unit Using the first image, the second image including the hair, the range designation information, and the learning model, the synthetic image is generated in which the hair included in the second image is reflected on the head of the person included in the first image.

13. The information processing apparatus according to claim 12.

15. The acquisition unit acquire a first image as the first target information obtained by capturing an image of the user's head, a second image as the second target information including hair designated by the user, and the range designation information indicating the designated range, which is a range of the user's face in the first image; The generation unit generating a composite image that is the composite information in which the hair included in the second image is reflected on the head of the user included in the first image; 2. The information processing apparatus according to claim 1, wherein:

16. The generation unit A face of the user included in the first image is copied, and the composite image is generated by reflecting a hairstyle corresponding to the hair included in the second image in the hairstyle of the user included in the first image.

16. The information processing apparatus according to claim 15,

17. 1. A computer-implemented information processing method, comprising: an acquisition step of acquiring first target information, which is a first target to be combined, second target information, which is a second target to be combined designated by a user, and range designation information, which indicates a designated range designated as a range in which fluctuations are to be suppressed in the first target information; a generating step of suppressing fluctuations in the specified range in the first target information based on the range specifying information and generating composite information by combining the first target information and the second target information acquired from the user; a providing step of providing the composite information and evaluation receiving content for receiving the user's evaluation of the composite information to a terminal device used by the user; Including, The obtaining step includes: receiving, from the terminal device used by the user, information indicating the user's evaluation of the composite information received from the user through the evaluation receiving content; 1. An information processing method comprising:

18. an acquisition step of acquiring first target information, which is a first target to be combined, second target information, which is a second target to be combined designated by a user, and range designation information, which indicates a designated range designated as a range in which fluctuations are to be suppressed in the first target information; a generating step of suppressing fluctuations in the specified range in the first target information based on the range specifying information, and generating composite information by combining the first target information and the second target information acquired from the user; a provision step of providing the composite information and evaluation receiving content for receiving the user's evaluation of the composite information to a terminal device used by the user; on the computer, The acquisition procedure includes: receiving, from the terminal device used by the user, information indicating the user's evaluation of the composite information received from the user through the evaluation receiving content; An information processing program characterized by:

Images