Composite data generation device, composite data generation method, and program

Abstract

Provided is a composite data generation device comprising: an extension target data acquisition unit that acquires extension target data; a change location candidate display unit that displays one or more change location candidates associated with the extension target data; a change location designation unit that designates one or more change locations from the one or more change location candidates; a composite data generation unit that generates composite data in which extension data is applied to the one or more change locations; and an output unit that outputs the composite data.

Description

Synthetic data generation device, synthetic data generation method, and program

[0001] The present invention relates to a synthetic data generation device, a synthetic data generation method, and a program.

[0002] It is known to generate image data by applying extended data obtained by extending a characteristic portion of an image to another image (see, for example, Patent Document 1). General disclosure

[0003] In a first aspect of the present invention, a synthetic data generation device is provided, comprising: an extension target data acquisition unit that acquires extension target data; a change location candidate display unit that displays one or more change location candidate associated with the extension target data; a change location designation unit that designates one or more change locations from the one or more change location candidate; a synthetic data generation unit that generates synthetic data in which extension data is applied to the one or more change locations; and an output unit that outputs the synthetic data.

[0004] The synthetic data generation device may include an extension data specification unit that specifies properties of the extension data to be applied to the extension target data.

[0005] In any of the above synthetic data generation devices, the extended data designation unit may designate the property using at least one of an image, a video, text, metadata, audio, music, an image, and three-dimensional data.

[0006] Any of the synthetic data generation devices described above may include a change candidate designation unit that designates properties of the one or more change candidate, and the change candidate display unit may display the one or more change candidate designated by the change candidate designation unit.

[0007] In any of the synthetic data generation devices described above, the change candidate designation unit may designate a category of the one or more change candidate to designate the one or more change candidate.

[0008] Any of the above synthetic data generation devices may include an augmented data candidate display unit that displays, as the augmented data candidates, at least one of one or more augmented data candidates and properties of the one or more augmented data candidates.

[0009] Any of the synthetic data generation devices described above may include an augmented data candidate evaluation unit that evaluates the one or more augmented data candidates.

[0010] Any of the above synthetic data generation devices may include a synthetic data candidate display unit that displays at least one of the evaluation results of the synthetic data candidates when one or more extension data candidates are applied to the extension target data or the properties of the synthetic data candidates.

[0011] Any of the above-described composite data generation devices may include an augmented data candidate display unit that displays one or more augmented data candidates in association with the one or more change portion candidates.

[0012] Any of the above synthetic data generation devices may include an augmented data candidate acquisition unit that acquires one or more augmented data candidates to be applied to the one or more changed portions.

[0013] Any of the above-described composite data generation devices may include an augmented data candidate display unit that displays the one or more augmented data candidates in association with the one or more changed portions.

[0014] In any of the above-mentioned synthetic data generation devices, the extension data candidate display unit may select and display the one or more extension data candidates depending on the degree of relevance with the content of the one or more changed parts before synthesis.

[0015] Any of the above synthetic data generation devices may include an augmented data candidate designation unit that designates related information of one or more augmented data candidates to be applied to the one or more change locations.

[0016] In any of the synthetic data generation devices described above, the data to be augmented may include at least one of text, images, videos, audio, music, presentation slides, design files, program code, and three-dimensional data.

[0017] The synthetic data generation device may further include a part identification data acquisition unit configured to acquire part identification data for identifying predetermined parts from the extension target data, and the change part designation unit may designate the one or more change parts based on the part identification data.

[0018] Any of the synthetic data generation devices described above may include a classification unit that classifies the one or more change candidate candidates into one or more categories, and the change candidate display unit may display the one or more change candidate candidates by classifying them into the one or more categories.

[0019] In any of the above synthetic data generation devices, the synthetic data generation unit may use a predetermined specified generation model to specify the extension data and generate the synthetic data to which the specified extension data has been applied.

[0020] In any of the above-described composite data generation devices, the extension target data acquisition unit may acquire first composite data generated by the composite data generation unit as the extension target data. The change candidate display unit may display the one or more change candidate associated with the first composite data as the extension target data. The change candidate designation unit may designate the one or more change candidate from the one or more change candidate associated with the first composite data. The composite data generation unit may generate second composite data in which extension data different from the extension data applied to the one or more change candidate is applied to the one or more change candidate. The output unit may output the second composite data.

[0021] In any of the above synthetic data generation devices, the output unit may output at least one of the data to be extended, the one or more changes, the extended data, the synthetic data, or related information regarding the generation of the synthetic data.

[0022] In a second aspect of the present invention, a synthetic data generation method is provided, comprising the steps of acquiring data to be extended, displaying one or more candidate changes associated with the data to be extended, specifying one or more change locations from the one or more candidate changes, generating synthetic data in which extension data is applied to the one or more change locations, and outputting the synthetic data.

[0023] In a third aspect of the present invention, a program is provided which, when executed by a computer, causes the computer to acquire data to be extended, display one or more candidate changes associated with the data to be extended, specify one or more changes from the one or more candidate changes, generate composite data in which extension data has been applied to the one or more changes, and output the composite data.

[0024] The above summary of the invention does not list all of the features of the present invention, and subcombinations of these features may also be inventions.

[0025] 22 shows an overview of the configuration of the synthetic data generating device 100. FIG. 22 shows an example of synthetic data generation ... displaying candidate change parts. FIG. 22 shows a modified example of the synthetic data generating device 100. FIG. 22 shows an example of displaying candidate change parts. FIG. 22 shows a modified example of the synthetic data generating device 100. FIG. 22 shows an example of displaying candidate augmented data, evaluation of candidate augmented data, evaluation of candidate synthetic data, and properties of candidate synthetic data. FIG. 22 shows an example of displaying candidate augmented data. FIG. 22 shows an example of displaying candidate augmented data.

[0026] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0027] FIG. 1 shows an overview of the configuration of a synthetic data generation device 100. The synthetic data generation device 100 generates synthetic data in which extension data is applied to change portions of extension target data. The synthetic data generation device 100 of this example includes an extension target data acquisition unit 10, a change portion candidate display unit 15, a change portion designation unit 20, a synthetic data generation unit 25, and an output unit 30. Note that the blocks shown are functionally separated functional blocks and may not necessarily correspond to the actual device configuration. In other words, blocks shown as a single block in this diagram may not necessarily be configured by a single device. Furthermore, blocks shown as separate blocks in this diagram may not necessarily be configured by separate devices. The same applies to blocks in other diagrams.

[0028] The augmentation target data acquisition unit 10 acquires augmentation target data. The augmentation target data may include at least one of text, images, videos, audio, music, presentation slides, design files, program code, and three-dimensional data. However, the types of augmentation target data are not limited to these. The augmentation target data acquisition unit 10 may supply information about the acquired augmentation target data to the change part candidate display unit 15 and the composite data generation unit 25.

[0029] The change candidate display unit 15 displays one or more change candidate associated with the data to be expanded. The change candidate display unit 15 may display the change candidate in any manner. For example, the change candidate display unit 15 may display the change candidate using words or sentences, using a segmentation mask, or using an image or video. The change candidate display unit 15 may supply information regarding the change candidate to the change part designation unit 20.

[0030] When the data to be expanded includes a sentence, the change candidate display unit 15 may display the change candidate by highlighting specific words, phrases, and / or sentences included in the sentence. The change candidate display unit 15 may display the change candidate by displaying the page number, paragraph number, and / or line number of the sentence.

[0031] When the data to be extended includes an image, the change candidate display unit 15 may display the change candidate by displaying an image obtained by cutting out a portion of the image. The change candidate display unit 15 may display the change candidate by displaying a segmentation mask of the image. The change candidate display unit 15 may display the change candidate by displaying a word or sentence indicating the target portion.

[0032] For example, if the data to be augmented is an image including a product, the change candidate display unit 15 may display the product parts that are the change candidate parts as cutout images, segmentation masks, or words or sentences. For example, if the data to be augmented is an image including a person, the change candidate display unit 15 may display areas that include the person's face and / or clothes that are the change candidate parts as cutout images, segmentation masks, and / or words or sentences. As an example, if the data to be augmented is an image including a person, the change candidate display unit 15 displays the change candidate part as the word "face."

[0033] When the extension target data includes a video, the change part candidate display unit 15 may display the change part candidates by displaying a partial scene of the video. The change part candidate display unit 15 may display the change part candidates by displaying an image of an object shown in the video. The change part candidate display unit 15 may display the change part candidates by displaying a word or sentence indicating the target portion. For example, the change part candidate display unit 15 may display a partial scene of the video, the start and end times of the partial scene of the video, or an image or sentence of a person or building shown in the video, as the change part candidates.

[0034] When the data to be extended includes audio, the change candidate display unit 15 may display the change candidate by playing a portion of the audio. The change candidate display unit 15 may display the change candidate by displaying a sentence corresponding to the audio. The change candidate display unit 15 may display the change candidate by displaying the start time and end time of the portion that is the change candidate.

[0035] When the data to be expanded includes music, the change part candidate display unit 15 may display the change part candidates by playing a portion of the music. The change part candidate display unit 15 may display the change part candidates by displaying a sentence representing a portion of the music. The change part candidate display unit 15 may display the change part candidates by displaying the start and end times of the portion that is the change part candidate. For example, when the change part candidate display unit 15 displays the chorus of a piece of music as a change part candidate, the chorus may be played, the word "chorus" may be displayed, and the start and end times of the chorus may be displayed.

[0036] When the data to be extended includes presentation slides, the change candidate display unit 15 may display the change candidate by highlighting the title or message line of each slide included in the presentation slides.The change candidate display unit 15 may display the change candidate by highlighting specific words, phrases, and / or sentences included in the presentation slides.

[0037] When the extension target data includes a design file, the change part candidate display unit 15 may display the change part candidates by displaying a part of the target designed by the design file. The change part candidate display unit 15 may display the change part candidates by displaying data indicating settings such as color correction or special effects used in the design.

[0038] When the data to be extended includes program code, the change candidate display unit 15 may display the change candidate by highlighting specific words, variables, and / or functions included in the program code. The change candidate display unit 15 may display the change candidate by displaying the line numbers of the program code.

[0039] When the data to be extended includes three-dimensional data, the change part candidate display unit 15 may display the change part candidates by displaying a portion of the three-dimensional data.The change part candidate display unit 15 may display the change part candidates by displaying a coordinate range of the three-dimensional data that is the change part candidate.The change part candidate display unit 15 may display the change part candidates by displaying a word or sentence that indicates the target part.

[0040] In the above description, a method in which the change candidate display unit 15 displays the change candidate has been exemplified, but there is no limitation on the method in which the change candidate display unit 15 displays the change candidate. The change candidate display unit 15 may display the change candidate in any method.

[0041] The change part designation unit 20 designates one or more change parts from one or more change part candidates. The change part designation unit 20 may designate one or more change parts by having the user select one or more change parts from one or more change part candidates displayed by the change part candidate display unit 15. The change part designation unit 20 may designate one or more change parts from one or more change part candidates displayed by the change part candidate display unit 15 based on predetermined designation criteria. The change part designation unit 20 may supply information about the designated change parts to the composite data generation unit 25.

[0042] The composite data generator 25 generates composite data in which the augmented data is applied to one or more of the changes. The augmented data may include at least one of text, images, videos, audio, music, presentation slides, design files, program code, and three-dimensional data. The composite data generator 25 may supply the generated composite data to the output unit 30.

[0043] The type of data included in the augmented data and the type of data included in the augmented data may be the same or different. For example, if the augmented data includes a sentence, the synthetic data generation unit 25 generates synthetic data by replacing words, phrases, and / or sentences specified as the change portions with other words, phrases, and / or sentences that are the augmented data. As another example, if the augmented data includes presentation slides, the synthetic data generation unit 25 may generate synthetic data by replacing the title or message line of each slide specified as the change portion with words, phrases, and / or sentences that are the augmented data. The same applies when the augmented data and / or the augmented data include other types of data.

[0044] When the data to be augmented is an image including a product, the composite data generation unit 25 may apply the augmented data to the change portion specified by the change portion specification unit 20 to generate composite data in which a defect has occurred in the product. In this way, the composite data generation unit 25 may generate defective data to be used in product inspection by AI or by an operator.

[0045] When the data to be augmented is an image including a person, the composite data generation unit 25 may generate composite data in which other faces or clothes, etc., which are augmented data, are applied to the faces or clothes, etc., of the change portion specified by the change portion specification unit 20. In this way, the composite data generation unit 25 may generate composite data that takes into consideration the privacy of people included in the image, the copyright of the image, etc.

[0046] When the data to be extended includes music, the composite data generation unit 25 may apply the extension data to the change portion specified by the change portion specification unit 20, thereby generating composite data in which at least a part of the music is converted into other music. For example, the composite data generation unit 25 generates composite data in which the chorus of the music has been changed.

[0047] When the data to be extended includes a presentation slide, the composite data generating unit 25 may generate composite data in which the text or the like at the change location specified by the change location specifying unit 20 has been changed. For example, the composite data generating unit 25 generates composite data including a slide whose content matches the title of the presentation slide.

[0048] When the data to be extended includes a moving image, the composite data generation unit 25 may generate composite data in which the people, etc. at the change location specified by the change location specification unit 20 have been changed. The composite data generation unit 25 may generate composite data in which the scenes, etc. at the change location specified by the change location specification unit 20 have been changed.

[0049] The above description has provided examples of the generation and use of synthetic data by the synthetic data generation unit 25, but the invention is not limited to these. The synthetic data generation unit 25 may appropriately generate synthetic data in accordance with the data to be extended and / or the extended data.

[0050] The output unit 30 outputs the composite data. The output unit 30 may transmit the composite data to an external device, may display the composite data on a display device or the like, or may store the composite data in a storage device or the like. In other words, outputting the composite data may include transmitting, displaying, and / or storing the composite data. However, the method of outputting the composite data is not limited to these.

[0051] The output unit 30 may output at least one of the augmented data or the changed portion used to generate the composite data in association with the composite data. For example, the augmented data, the changed portion, and the composite data may be transmitted, displayed, and stored as a set.

[0052] In this example, the composite data generation device 100 displays candidate changes, specifies the changes, and generates the composite data. This makes it easier to generate the composite data than when the user specifies the changes when generating the composite data. For example, a user generating the composite data can specify the changes from the candidate changes displayed by the candidate change display unit 15 and generate the composite data, so that the user can easily generate the composite data without having to manually specify the areas, portions, ranges, etc. that will be changed.

[0053] The extension target data acquisition unit 10 may acquire, as extension target data, first extension data generated by the extension data generation unit 25. The change location candidate display unit 15 may display one or more change location candidates associated with the first extension target data. The change location designation unit 20 may designate the one or more change locations from one or more change location candidates associated with the first extension data. The extension data generation unit 25 may generate second extension data in which extension data different from the extension data applied to the first extension data is applied to the one or more change locations. The output unit 30 may output the second extension data.

[0054] That is, the synthetic data generation device 100 may repeatedly generate synthetic data by repeatedly applying extension data to the generated synthetic data. For example, the synthetic data generation device 100 may apply first extension data to extension target data to generate first synthetic data. The synthetic data generation device 100 may use the first synthetic data as extension target data and apply second extension data to generate second synthetic data. The synthetic data generation device 100 may use the second synthetic data as extension target data and apply third extension data to generate third synthetic data. The synthetic data generation device 100 may repeat this process, and use the (N-1)th synthetic data as extension target data and apply the Nth extension data to generate Nth synthetic data. Here, N may be any natural number.

[0055] The output unit 30 may output at least one of the data to be augmented, one or more changes, the augmented data, the combined data, and related information related to the generation of the combined data. The related information related to the generation of the combined data may include the generator who instructed the generation of the combined data, the date and time the combined data was generated, and other information related to the generation of the combined data. In this way, the output unit 30 may output history information related to the combined data.

[0056] Here, the extension target data may be extension target data to which extension data is first applied, or may be synthesized data as extension target data. Similarly, the synthesized data may be first synthesized data, second synthesized data, or synthesized data to which extension data has been applied an arbitrary number of times. In other words, when the synthetic data generation device 100 repeatedly applies extension data to generate synthetic data, the output unit 30 may output history information related to the synthesized data at the arbitrary number of repetitions.

[0057] As described above, outputting the composite data by the output unit 30 may include transmitting, displaying, and / or storing the composite data. For example, when the composite data generation device 100 repeatedly applies augmented data to repeatedly generate composite data, the output unit 30 may store at least one of the augmented data, one or more changes, the augmented data, the composite data, or related information related to the generation of the composite data for all the repetitions in a storage device or the like. Then, in response to a user's designation of a specific number of repetitions, the output unit 30 may display at least one of the augmented data, one or more changes, the augmented data, the composite data, or related information related to the generation of the composite data for the specific number of repetitions on a display device or the like.

[0058] 2A shows an example of generating synthetic data, in which the data to be augmented is an image containing multiple people.

[0059] The extension target data acquisition unit 10 acquires an image including multiple people as extension target data. The change part candidate display unit 15 displays areas including the people's faces and clothes as cutout images as four change part candidates associated with the extension target data. The change part candidate display unit 15 may display the areas including the people's faces and clothes superimposed on the extension target data.

[0060] The change portion designation unit 20 designates a male face as the change portion. The composite data generation unit 25 generates composite data in which the face of another male, which is extension data, is applied to the designated change portion. The output unit 30 outputs the generated composite data.

[0061] In this example, one change location is specified and one extension data is applied. When multiple change locations are specified, the extension data applied to the multiple change locations may be the same or different. For example, when the faces of both people are specified as change locations, the same extension data may be applied to generate composite data in which both people have faces based on the same extension data, or different extension data may be applied to generate composite data in which both people have faces based on different extension data.

[0062] 2B shows an example of synthetic data generation. In this example, the change candidate display unit 15 displays areas including a person's face and clothing as segmentation masks, as four change candidate areas associated with the data to be augmented. Segmentation mask M1 corresponds to a man's face, segmentation mask M2 corresponds to the man's clothing, segmentation mask M3 corresponds to a woman's face, and segmentation mask M4 corresponds to the woman's clothing.

[0063] The change portion designation unit 20 designates the segmentation mask M1 as the change portion. The composite data generation unit 25 generates composite data in which the face of another man, which is the augmented data, is applied to the designated change portion. The output unit 30 outputs the generated composite data.

[0064] 2C shows an example of generating composite data. In this example, the candidate change display unit 15 displays an area including a person's face and clothes as two candidate changes associated with the data to be expanded, using the words "face" and "clothes."

[0065] The change portion designation unit 20 designates the word "face" as the change portion. The composite data generation unit 25 generates composite data in which the face of another man, which is the augmented data, is applied to the designated change portion. The output unit 30 outputs the generated composite data.

[0066] 3A shows an example of generating synthetic data, where the data to be augmented is an image containing bins.

[0067] The extension target data acquisition unit 10 acquires an image including bins as extension target data. The change candidate display unit 15 displays, as segmentation masks, regions including the upper and lower parts of the bins as two change candidate regions associated with the extension target data. Segmentation mask M1 corresponds to the upper part of the bin, and segmentation mask M2 corresponds to the lower part of the bin.

[0068] The change portion designation unit 20 designates the segmentation mask M1 as the change portion. The composite data generation unit 25 generates composite data in which the missing portion, which is the extended data, is applied to the designated change portion. The output unit 30 outputs the generated composite data.

[0069] 3B shows an example of generating composite data. In this example, the candidate change display unit 15 displays the regions including the top and bottom of the bin as the two candidate change regions associated with the data to be expanded, using the character strings “top of the bin” and “bottom of the bin.”

[0070] The change portion designation unit 20 designates the character string "top of the bin" as the change portion. The composite data generation unit 25 generates composite data in which the missing portion, which is extension data, is applied to the designated change portion. The output unit 30 outputs the generated composite data.

[0071] 4 shows an example of generating synthesized data. In this example, the data to be extended is music. For example, the music is composed of an intro, an A melody, a B melody, and a C melody.

[0072] The extension target data acquisition unit 10 acquires music as extension target data. The change part candidate display unit 15 displays the music structure as character strings, "intro," "verse verse," "verse verse," and "verse verse," as four change part candidates associated with the extension target data.

[0073] The change portion designation unit 20 designates the character string "B melody" as the change portion. The synthesis data generation unit 25 generates synthesis data in which another melody line, which is extension data, is applied to the designated change portion. The output unit 30 outputs the generated synthesis data.

[0074] 5 shows an example of a display of candidate changes. In this example, the data to be extended is a presentation slide.

[0075] The extension target data acquisition unit 10 acquires a presentation slide as the extension target data. Note that in this example, one page of a presentation slide is shown as the extension target data, but the extension target data acquisition unit 10 may acquire multiple pages of a presentation slide as the extension target data.

[0076] The change candidate display unit 15 highlights a part of the slide title or the message line as the change part. The change part designation unit 20 may designate at least one of the highlighted part of the slide title or the message line as the change part. The synthetic data generation unit 25 may generate synthetic data in which other words, phrases, and / or sentences, which are extension data, are applied to the designated change part. The output unit 30 may output the generated synthetic data.

[0077] 6 shows a modified example of the synthetic data generation device 100. The synthetic data generation device 100 of this example differs from the embodiment of FIG. 1 in that it includes a change location candidate designation unit 35, a classification unit 40, a location identification data acquisition unit 45, and a model storage unit 50. In this example, differences from the embodiment of FIG. 1 will be particularly described, and the rest may be the same as the embodiment of FIG. 1.

[0078] The change candidate designation unit 35 may designate one or more change candidate properties to designate one or more change candidate properties. The change candidate designation unit 35 may designate the properties of the change candidate properties using at least one of an image, a video, a text, metadata, audio, music, an image, or three-dimensional data. The metadata may be data in a format not expressed by properties. For example, if the augmentation target data includes an image, the metadata may be a collection of tags representing the change candidate properties, such as "woman," "Japanese clothing," "nature," or "realistic." However, the metadata is not limited to this. The change candidate designation unit 35 may display a screen on the display device for selecting properties of the change candidate properties, allow the user to select a property of the change candidate properties, and designate the property of the selected change candidate properties. The change candidate display unit 15 may display one or more change candidate properties designated by the change candidate designation unit 35.

[0079] The change candidate designation unit 35 may designate one or more categories of change candidate to designate one or more change candidate. For example, if the data to be extended is an image including a person, the change candidate designation unit 35 designates at least one of "face," "clothes," or "whole body" as the category of the change candidate. However, the properties or categories of the change candidate designated by the change candidate designation unit 35 are not limited to these.

[0080] The composite data generation device 100 of this example includes a change candidate designation unit 35. This makes it easier to generate composite data compared to when the user sets the change locations when generating the composite data. For example, a user generating composite data can use the change candidate designation unit 35 to designate the change candidate locations displayed by the change candidate display unit 15. This allows the user to easily generate composite data by designating the desired change locations without having to manually designate the areas, portions, ranges, etc. that will be changed.

[0081] The classification unit 40 may classify one or more change candidate parts into one or more categories. For example, if the augmentation target data is an image including a person, the classification unit 40 may classify the change candidate parts into the category of "face" or "clothes." The classification unit 40 may classify the change candidate parts into one or more categories based on the properties of the change candidate parts. The classification unit 40 may display a screen on the display device for selecting the properties of the change candidate parts, allow the user to select the properties of the change candidate parts, and classify the change candidate parts into one or more categories based on the properties of the selected change candidate parts. The change candidate display unit 15 may classify the change candidate parts into one or more categories and display one or more change candidate parts.

[0082] The composite data generation device 100 of this example includes a classification unit 40. This makes it easier to generate composite data compared to when a user sets the changes when generating the composite data. For example, a user generating composite data can check the change candidate displayed by the change candidate display unit 15 in a state categorized by category. This allows the user to easily generate composite data by specifying the desired changes without having to set the area, portion, range, or the like to be changed.

[0083] The location identification data acquisition unit 45 may acquire location identification data for identifying a predetermined location from the extension target data. The location identification data acquisition unit 45 may acquire sentences for identifying a location in the extension target data. If the extension target data includes an image, the location identification data acquisition unit 45 may acquire mask data for the image. If the extension target data is an image including a product, the location identification data acquisition unit 45 may acquire mask data using a design drawing of the product. The change location designation unit 20 may designate one or more change locations based on the location identification data.

[0084] The composite data generation device 100 of this example includes a location identification data acquisition unit 45. This makes it easier to generate composite data compared to when the user sets the change locations when generating the composite data. For example, the user generating the composite data can specify the change locations based on the location identification data in addition to the change location candidates displayed by the change location candidate display unit 15. This allows the user to easily generate composite data by specifying appropriate change locations without having to manually set the areas, portions, ranges, etc. that will be changed.

[0085] The model storage unit 50 may store a designated generative model 52 for generating synthetic data. The model storage unit 50 may store a plurality of designated generative models 52. For example, the model storage unit 50 stores a plurality of designated generative models 52 corresponding to the type of data to be augmented. The designated generative model 52 may be a model that has been trained in advance to generate synthetic data, such as a generative adversarial network (GAN) or a diffusion model.

[0086] The synthetic data generation unit 25 may use a predetermined designated generative model 52 to specify extension data and generate synthetic data to which the designated extension data has been applied. The synthetic data generation unit 25 may select a designated generative model 52 corresponding to the extension target data, the change portion, and / or the extension data from a plurality of designated generative models 52 stored in the model storage unit 50, and generate synthetic data using the selected designated generative model 52. The synthetic data generation unit 25 may input the extension target data, the change portion, and the extension data to the designated generative model 52. The designated generative model 52 may output synthetic data in response to the input of the extension target data, the change portion, and the extension data from the synthetic data generation unit 25.

[0087] In this example, the synthetic data generation device 100 is described as including the model storage unit 50, but is not limited to this. For example, the synthetic data generation unit 25 may generate synthetic data using a designated generation model 52 stored on a server or the like external to the synthetic data generation device 100.

[0088] 7 shows an example of a display of the change candidate. The change candidate display unit 15 in this example displays one or more change candidate specified by the change candidate specifying unit 35.

[0089] For example, the change candidate designation unit 35 designates a change candidate by designating "face" as a property of the change candidate. The change candidate display unit 15 may display change candidates whose property corresponds to "face." In this example, other change candidates whose property is "clothes" are not displayed.

[0090] However, the properties of the change part candidates designated by the change part candidate designation unit 35 are not limited to this. For example, if the data to be extended is an image including a person, the change part candidate designation unit 35 may designate the person's gender, age, or other properties as the properties of the change part candidates.

[0091] 8 shows an example of a display of the change candidate. The change candidate display unit 15 of this example classifies the change candidate into one or more categories classified by the classification unit 40 and displays one or more change candidate.

[0092] For example, the classification unit 40 classifies the change part candidates into the categories of “face” and “clothes.” The change part candidate display unit 15 may display the change part candidates by classifying them into the categories of “face” and “clothes.”

[0093] However, the categories classified by the classification unit 40 are not limited to this. For example, if the data to be augmented is an image including a person, the classification unit 40 may classify the candidate change parts based on the gender, age, and other properties of the person.

[0094] 9 shows a modified example of the synthetic data generation device 100. The synthetic data generation device 100 of this example differs from the embodiment of FIG. 1 in that it includes an extension data designation unit 55, an extension data candidate display unit 60, an extension data candidate acquisition unit 65, an extension data candidate evaluation unit 70, a synthetic data candidate display unit 75, and an extension data candidate designation unit 80. In this example, differences from the embodiment of FIG. 1 will be particularly described, and the rest may be the same as the embodiment of FIG. 1.

[0095] The augmented data specification unit 55 may specify the properties of the augmented data to be applied to the augmented target data. The augmented data specification unit 55 may specify the properties using at least one of an image, a video, a sentence, metadata, audio, music, an image, or three-dimensional data. For example, if the augmented target data includes music, the augmented data specification unit 55 may specify the properties of the augmented data using the sentence "relaxing music." The augmented data specification unit 55 may supply the specified properties of the augmented data to the composite data generation unit 25. The composite data generation unit 25 may acquire augmented data according to the properties of the augmented data and generate composite data to which the acquired augmented data is applied.

[0096] The extension data designation unit 55 may designate a property of the extension data according to the property of the change portion candidate and / or the change portion. The property of the change portion candidate and / or the change portion and the property of the extension data may be the same or different. For example, if the extension target data is an image including a person and the property of the change portion is "face," the extension data designation unit 55 may designate "face," which is the same as the property of the change portion, as the property of the extension data. As another example, if the extension target data includes music and the property of the change portion is "pop," the extension data designation unit 55 may designate "rock," which is different from the property of the change portion, as the property of the extension data.

[0097] The extension data designation unit 55 may designate multiple properties of the extension data. For example, if the data to be extended is an image including a person and the property of the part to be changed is "face," the extension data designation unit 55 may designate "face" as a property of the extension data, and may also designate the person's gender, age, facial expression, and other properties.

[0098] The composite data generating device 100 of this example includes an extension data specifying unit 55. This allows the user to specify extension data according to the data to be extended and / or the portion to be changed, and easily generate composite data.

[0099] The extension data candidate display unit 60 may display at least one of one or more extension data candidates and properties of one or more extension data candidates as extension data candidates. The extension data candidate display unit 60 may display at least one of extension data candidates and properties of extension data candidates corresponding to the change location candidates and / or properties of the change locations.

[0100] For example, if the data to be extended is an image including a person and the property of the change portion is a "face," the extended data candidate display unit 60 displays one or more "face" images as extended data candidates. The extended data candidate display unit 60 may display one or more properties, such as a "happy face," an "angry face," a "sad face," or a "happy face," as the properties of the extended data candidates. The extended data candidate display unit 60 may display one or more corresponding properties in addition to one or more "face" images.

[0101] The extension data candidate display unit 60 may allow the user to select from the displayed extension data candidates or properties of the extension data candidates, and supply the selected extension data candidate or properties of the extension data candidate to the extension data designation unit 55. The extension data designation unit 55 may designate the properties corresponding to the selected extension data candidate or the properties of the selected extension data candidate as properties of the extension data to be applied to the data to be extended. The extension data candidate display unit 60 may supply the extension data candidate or properties of the extension data candidate to the composite data generation unit 25 without going through the extension data designation unit 55.

[0102] The extension data candidate acquisition unit 65 may acquire one or more extension data candidates to be applied to one or more change locations. The extension data candidate acquisition unit 65 may acquire extension data candidates according to the change location candidates and / or properties of the change locations. The extension data candidate acquisition unit 65 may supply the acquired extension data candidates to the extension data candidate display unit 60.

[0103] The extension data candidate display unit 60 may display one or more extension data candidates in association with one or more change locations. The extension data candidate display unit 60 may select and display one or more extension data candidates according to the degree of relevance to the content of one or more change locations before combination. The extension data candidate display unit 60 may display one or more extension data candidates in association with one or more change location candidates.

[0104] For example, if the data to be augmented is an image including a person and the property of the change portion is "face," the augmented data candidate display unit 60 may select and display augmented data candidates that are highly related to the face that is the change portion from among the faces included in the augmented data candidates. The augmented data candidate display unit 60 may select and display augmented data candidate that has a higher degree of relatedness with the content of the change portion before synthesis than a predetermined standard. For example, if the property of the change portion is "male face," the augmented data candidate display unit 60 may select and display a male face from among the faces included in the augmented data candidates.

[0105] The composite data generation device 100 of this example includes an extension data candidate display unit 60. This allows a user generating composite data to check extension data candidates before applying them to the extension target data, and thus allows the user to easily generate composite data by specifying extension data corresponding to the extension target data and / or the portion to be changed.

[0106] The extension data candidate evaluation unit 70 may evaluate one or more extension data candidates. As the evaluation of the extension data candidate, the extension data candidate evaluation unit 70 may evaluate the variation value of the evaluation when the extension data candidate is applied to the extension target data. The extension data candidate evaluation unit 70 may display the evaluation result of the extension data candidate. The extension data candidate evaluation unit 70 may display the evaluation result of the extension data candidate using numerical values ​​or text. However, the method of displaying the evaluation result by the extension data candidate evaluation unit 70 is not limited to this.

[0107] The synthetic data generation device 100 of this example includes an extension data candidate evaluation unit 70. This allows a user generating synthetic data to check extension data candidates before applying them to the extension target data and to check the evaluation of the extension data candidates, thereby enabling easy generation of synthetic data by specifying extension data corresponding to the extension target data and / or the change location.

[0108] The composite data candidate display unit 75 may display at least one of the evaluation results of the composite data candidates or the properties of the composite data candidates when one or more extension data candidates are applied to the extension target data. The composite data candidate display unit 75 may display the evaluation results of the composite data candidates using numerical values ​​or text. However, the method of displaying the evaluation results by the composite data candidate display unit 75 is not limited to this. The composite data candidate display unit 75 may allow the user to select the evaluation results or properties of the displayed extension data candidates, and supply the evaluation results or properties of the selected extension data candidate to the extension data designation unit 55. The extension data designation unit 55 may designate the properties of the extension data corresponding to the evaluation results or properties of the selected extension data candidate as the properties of the extension data to be applied to the extension target data. The composite data candidate display unit 75 may supply the evaluation results or properties of the composite data candidates to the composite data generation unit 25 without going through the extension data designation unit 55.

[0109] The composite data generation device 100 of this example includes a composite data candidate display unit 75. This allows a user generating composite data to check at least one of the evaluation results of the composite data candidate or the properties of the composite data candidate before applying the augmentation data candidate to the augmentation target data, thereby enabling the user to easily generate composite data by specifying augmentation data corresponding to the augmentation target data and / or the change portion.

[0110] The extension data candidate designation unit 80 may designate related information of one or more extension data candidates to be applied to one or more change locations. The related information of the extension data candidates may be information for narrowing down the extension data candidates. The extension data candidate designation unit 80 may display a screen on the display device for selecting related information of the extension data candidates, allow the user to select related information of the extension data candidates, and designate the related information of the selected extension data candidate. The extension data candidate designation unit 80 may display a screen on the display device for inputting related information of the extension data candidates, allow the user to input related information of the extension data candidates, and designate the related information of the input extension data candidate. The extension data candidate display unit 60 may display one or more extension data candidates according to the related information designated by the extension data candidate designation unit 80.

[0111] For example, if the data to be augmented is an image including a person and the property of the change portion is "face," multiple pieces of data including the person's face may be candidates for data to be augmented. The augmented data candidate designation unit 80 may designate related information for the augmented data candidate using, for example, a character string such as "cute." The augmented data candidate display unit 60 may display augmented data candidates according to the related information, "cute," designated by the augmented data candidate designation unit 80.

[0112] The composite data generating device 100 of this example includes an extended data candidate specifying unit 80. This allows desired extended data to be specified and composite data to be easily generated.

[0113] Each functional block illustrated in each of Figures 1, 6, and 9 may be provided together with functional blocks illustrated in other figures, i.e., the synthetic data generating device 100 according to the present invention may include all or any combination of the functional blocks illustrated in Figures 1, 6, and 9.

[0114] 10 shows an example of displaying candidate augmented data, candidate augmented data evaluation, candidate composite data evaluation, and candidate composite data properties. In this example, the changes are made to a male face and a female face in an image containing a person, which is the data to be augmented.

[0115] The extension data candidate display unit 60 may display extension data candidates. In this example, the extension data candidate display unit 60 displays extension data candidates in association with the change portion. In this example, the extension data candidate display unit 60 selects and displays one or more extension data candidates according to the degree of association with the content of the change portion. For example, the extension data candidate display unit 60 links a change portion that is a male face to display extension data candidates representing other male faces that are highly associated, and links a change portion that is a female face to display extension data candidates representing other female faces that are highly associated.

[0116] The extension data candidate evaluation unit 70 may evaluate the extension data candidates. In this example, the extension data candidate evaluation unit 70 evaluates the value of the change in evaluation when the extension data candidate is applied to the extension target data, as the evaluation of the extension data candidate. For example, the extension data candidate evaluation unit 70 indicates that when the left extension data candidate of a male face is applied to the extension target data for a change portion that is a male face, the evaluation changes by +12, and when the right extension data candidate of a male face is applied to the extension target data, the evaluation changes by -5.

[0117] The composite data candidate display unit 75 may display at least one of the evaluation results of the composite data candidates and the properties of the composite data candidates. In this example, the composite data candidate display unit 75 displays both the evaluation results of the composite data candidates and the properties of the composite data candidates.

[0118] In this example, an example in which both information about the augmented data candidates and information about the composite data candidates are displayed has been shown, but the present invention is not limited to this. The composite data generation device 100 may display either the information about the augmented data candidates or the information about the composite data candidates.

[0119] FIG. 11 shows an example of displaying extension data candidates. In this example, the change location candidates are a face and clothes in an image including a person, which is data to be extended. The extension data candidate display unit 60 may display one or more extension data candidates in association with one or more change location candidates. That is, the extension data candidate display unit 60 may display the extension data candidates in association with the change location candidates before a change location is specified from the change location candidates. In this example, the extension data candidate display unit 60 displays an image including a person's face, which is an extension data candidate, in association with the change location candidate "face," and displays an image including the person's clothes, which is an extension data candidate, in association with the change location candidate "clothes."

[0120] 12 shows an example of a display of candidate augmented data. In this example, the changes are made to the face and clothes of an image containing a person, which is the data to be augmented.

[0121] The extension data candidate designation unit 80 may designate related information for the extension data candidate. In this example, the extension data candidate designation unit 80 designates a character string "cute" as related information for the extension data candidate to be applied to the changed portion of "face." The extension data candidate display unit 60 may display extension data candidates according to the related information "cute" designated by the extension data candidate designation unit 80 as extension data candidates to be applied to the changed portion of "face." In this example, the extension data candidate designation unit 80 designates a character string "smart" as related information for the extension data candidate to be applied to the changed portion of "clothes." The extension data candidate display unit 60 may display extension data candidates according to the related information "smart" designated by the extension data candidate designation unit 80 as extension data candidates to be applied to the changed portion of "clothes."

[0122] 13 illustrates an example of a computer 2200 in which aspects of the present invention may be embodied, in whole or in part. Programs installed on the computer 2200 may cause the computer 2200 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to embodiments of the present invention, and / or to perform a process or steps of the process according to embodiments of the present invention. Such programs may be executed by the CPU 2212 to cause the computer 2200 to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0123] A computer 2200 according to this embodiment includes a CPU 2212, a RAM 2214, a graphics controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0124] The CPU 2212 operates according to programs stored in the ROM 2230 and RAM 2214, thereby controlling each unit. The graphics controller 2216 acquires image data generated by the CPU 2212 into a frame buffer or the like provided in the RAM 2214 or into the graphics controller 2216 itself, and causes the image data to be displayed on the display device 2218.

[0125] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides the programs or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0126] ROM 2230 stores therein a boot program or the like that is executed by computer 2200 upon activation, and / or programs that depend on the hardware of computer 2200. I / O chip 2240 may also connect various I / O units to I / O controller 2220 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0127] The programs are provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The programs are read from the computer-readable medium, installed in the hard disk drive 2224, RAM 2214, or ROM 2230, which are also examples of computer-readable media, and executed by the CPU 2212. Information processing described in these programs is read by the computer 2200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by realizing information manipulation or processing in accordance with the use of the computer 2200.

[0128] For example, when communication is performed between computer 2200 and an external device, CPU 2212 may execute a communication program loaded into RAM 2214 and instruct communication interface 2222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 2212, communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in RAM 2214, hard disk drive 2224, DVD-ROM 2201, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0129] Furthermore, the CPU 2212 may cause all or a necessary portion of a file or database stored on an external recording medium such as the hard disk drive 2224, the DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc. to be read into the RAM 2214, and may perform various types of processing on the data on the RAM 2214. The CPU 2212 then writes back the processed data to the external recording medium.

[0130] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 2212 may perform various types of processing on data read from the RAM 2214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 2214. The CPU 2212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored on the recording medium, the CPU 2212 may search for an entry that matches a condition specified by the attribute value of the first attribute from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0131] The above-described programs or software modules may be stored in a computer-readable medium on or near the computer 2200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable medium, thereby providing the programs to the computer 2200 via the network.

[0132] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0133] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order.

[0134] 10... Extension target data acquisition unit, 15... Change location candidate display unit, 20... Change location designation unit, 25... Synthetic data generation unit, 30... Output unit, 35... Change location candidate designation unit, 40... Classification unit, 45... Location identification data acquisition unit, 50... Model storage unit, 52... Designated generation model, 55... Extension data designation unit, 60... Extension data candidate display unit, 65... Extension data candidate acquisition unit, 70... Extension data candidate evaluation unit, 75... Synthetic data candidate display unit, 80... Extension data candidate designation unit, 100 ...Synthetic data generating device, 2200...Computer, 2201...DVD-ROM, 2210...Host controller, 2212...CPU, 2214...RAM, 2216...Graphics controller, 2218...Display device, 2220...Input / output controller, 2222...Communication interface, 2224...Hard disk drive, 2226...DVD-ROM drive, 2230...ROM, 2240...Input / output chip, 2242...Keyboard

Claims

1. A synthetic data generation device comprising: an extension data acquisition unit that acquires data to be extended; a change candidate display unit that displays one or more change candidate locations associated with the data to be extended; a change candidate designation unit that designates one or more change candidate locations from the one or more change candidate locations; a synthetic data generation unit that generates synthetic data in which extension data has been applied to the one or more change candidate locations; and an output unit that outputs the synthetic data.

2. The synthetic data generating device according to claim 1, further comprising an extension data specifying unit that specifies properties of the extension data to be applied to the extension target data.

3. The synthetic data generating device according to claim 2, wherein the extension data designation unit designates the property using at least one of an image, a video, a text, metadata, audio, music, an image, and three-dimensional data.

4. A synthetic data generation device as described in claim 1, further comprising a change candidate designation unit that designates the one or more change candidate properties and designates the one or more change candidate properties, and the change candidate display unit displays the one or more change candidate properties designated by the change candidate designation unit.

5. The synthetic data generating device according to claim 4, wherein the change candidate designation unit designates a category of the one or more change candidate, and designates the one or more change candidate.

6. The synthetic data generating device according to claim 1, further comprising an augmented data candidate display unit that displays at least one of one or more augmented data candidates or properties of the one or more augmented data candidates as the augmented data candidates.

7. The synthetic data generating device according to claim 5, further comprising an augmented data candidate evaluation unit that evaluates the one or more augmented data candidates.

8. The synthetic data generation device according to claim 1, further comprising a synthetic data candidate display unit that displays at least one of the evaluation results of the synthetic data candidate when one or more augmentation data candidates are applied to the augmentation target data or the properties of the synthetic data candidate.

9. A synthetic data generation device according to any one of claims 1 to 8, further comprising an augmented data candidate display unit that displays one or more augmented data candidates in association with the one or more change candidate parts.

10. A synthetic data generation device according to any one of claims 1 to 8, further comprising an augmented data candidate acquisition unit that acquires one or more augmented data candidates to be applied to the one or more changed locations.

11. The synthetic data generating device according to claim 10, further comprising an augmented data candidate display unit that displays the one or more augmented data candidates in association with the one or more changed portions.

12. The synthetic data generating device according to claim 11, wherein the extension data candidate display unit selects and displays the one or more extension data candidates according to the degree of relevance with the content of the one or more changed portions before synthesis.

13. A synthetic data generating device according to any one of claims 1 to 8, further comprising an extension data candidate designation unit that designates related information of one or more extension data candidates to be applied to the one or more change locations.

14. A synthetic data generation device according to any one of claims 1 to 8, wherein the data to be augmented includes at least one of text, images, video, audio, music, presentation slides, design files, program code, and three-dimensional data.

15. A synthetic data generation device as described in any one of claims 1 to 8, further comprising a location identification data acquisition unit that acquires location identification data for identifying predetermined locations from the data to be extended, and the change location designation unit designates the one or more change locations based on the location identification data.

16. A synthetic data generation device as described in any one of claims 1 to 8, further comprising a classification unit that classifies the one or more candidate change parts into one or more categories, and the candidate change part display unit classifies the one or more candidate change parts into the one or more categories and displays the one or more candidate change parts.

17. A synthetic data generation device according to any one of claims 1 to 8, wherein the synthetic data generation unit uses a predetermined specified generation model to specify the extension data and generate the synthetic data to which the specified extension data has been applied.

18. A synthetic data generation device as described in any one of claims 1 to 8, wherein the extension target data acquisition unit acquires first synthetic data generated by the synthetic data generation unit as the extension target data, the change candidate display unit displays the one or more change candidate associated with the first synthetic data as the extension target data, the change candidate designation unit designates the one or more change candidate from the one or more change candidate associated with the first synthetic data, the synthetic data generation unit generates second synthetic data in which extension data different from the extension data applied in the first synthetic data is applied to the one or more change candidate, and the output unit outputs the second synthetic data.

19. A synthetic data generation device according to any one of claims 1 to 8, wherein the output unit outputs at least one of the data to be extended, the one or more changes, the extended data, the synthetic data, or related information relating to the generation of the synthetic data.

20. A synthetic data generation method comprising: a step of acquiring data to be extended; a step of displaying one or more candidate changes associated with the data to be extended; a step of specifying one or more changes from the one or more candidate changes; a step of generating synthetic data in which extension data is applied to the one or more changes; and a step of outputting the synthetic data.

21. A program that, when executed by a computer, causes the computer to: acquire data to be extended; display one or more candidate changes associated with the data to be extended; specify one or more changes from the one or more candidate changes; generate composite data in which extension data has been applied to the one or more changes; and output the composite data.

Images