Method for supporting determination of creation prompt input to learning model, as well as image creation method, system, and its production method executing this method, and program, as well as classification system using learning model trained according to this method

Abstract

To provide means that can accurately create an image that cannot be easily acquired as learning data.SOLUTION: There is provided a method for supporting determination of a creation prompt according to the present invention, the creation prompt to be input to a creation model capable of image creation according to a creation parameter, and the method causes one or more computers to execute: a classification information receiving step of receiving, from an interface, classification information of images corresponding to one or two or more creation parameters; and a recommendation creation prompt output step of, after receiving in the classification information receiving step an answer where the classification information is a third classification that is not a first classification or a second classification, for at least any one of the images corresponding to the one or two or more creation parameters, outputting a recommendation creation prompt.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a decision support method for generating prompts that are input to a learning model, an image generation method and system for carrying out this method, a production method and program for this method, and a classification system that uses a learning model trained by this method. [Background technology]

[0002] Video analysis technologies and services using sensing technology have been widely used in many fields. In particular, with the development of deep learning and AI technologies, automation technologies that replace tasks that have traditionally been done manually are becoming widespread and commonplace.

[0003] For example, in manufacturing sites such as factories, video analytics technology is being actively adopted to reduce labor costs and potential hazards such as accidents in product inspection and production line status monitoring. One example of its adoption is a method for inspecting and determining whether a product passes or fails by comparing images of the product being inspected with images of normal products that have passed inspection and examining differences in image features. This method is applied to picking products that do not meet standards for acceptable shipping. In this case, for example, a deep learning model that has trained a large number of images of normal and abnormal products can be used to calculate the image features of the image of the product being inspected. The pass / fail judgment can then be automatically made by calculating whether the image features are closer to the image features of the normal product or the abnormal product in the latent space of the deep learning model.

[0004] Large-scale language models, a field of deep learning models, began to spread around 2020, and research into fusing large-scale language models with image models has been progressing since around 2022. Image generation models, such as Stable Diffusion, a representative example of diffusion models, have emerged, which take natural language (text) as input and generate unknown images that correspond to the input, and image generation is now being used for a variety of purposes, from video analysis solutions to the arts.

[0005] Patent Document 1 describes an anomaly detection system that includes a restored image generation unit that inputs an image of the test subject to a learning model that has been trained using normal images of the test subject as training data, and generates a restored image, and a detection unit that determines whether the test subject in the test subject image is abnormal using the restored image and a normal restored image that is a restored image generated by inputting a normal image of the test subject to the learning model. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2023-8416 Summary of the Invention [Problem to be solved by the invention]

[0007] It is desirable that the anomaly detection system described in Patent Document 1 has sufficiently trained the learning model in advance. However, in factories and other places where inspections are conducted, operations are carried out with the aim of achieving and maintaining a normal state, so while normal images that indicate a normal state are easy to obtain, abnormal images that indicate an abnormal state are relatively difficult to obtain. Therefore, from the perspective of collecting a large amount of training data, a means is desired that can accurately generate images, such as abnormal images, that are difficult to obtain as training data. [Means for solving the problem]

[0008] The method for supporting the determination of a generation prompt according to the present invention is a method for supporting the determination of a generation prompt to be input into a generation model capable of generating images according to generation parameters, and comprises causing one or more computers to execute a classification information receiving step for receiving from an interface classification information of images corresponding to each of one or more generation parameters, and a recommended generation prompt output step for outputting a recommended generation prompt after receiving in the classification information receiving step an answer that the classification information for at least one of the images corresponding to the one or more generation parameters is a third classification that is neither the first classification nor the second classification. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating a functional configuration of an image generation system according to a first embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing a space (latent space) represented by generation parameters before an image generation model by an image generation unit is decoded into an image. [Figure 3] FIG. 10 is a diagram showing a method for determining a recommended norm (recommended generation prompt) from a first image, which is one end of a variation vector, to the other end. [Figure 4] 1 is a diagram illustrating an example of a hardware configuration of an image generating system according to a first embodiment. [Figure 5] 10 is an example of a flowchart illustrating the procedure of a method for determining recommended generated prompts according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing a designation screen for accepting designation of whether an application including an image generation system is to be used as an image generation system or an image inspection system (classification system). [Figure 7A] FIG. 10 is a diagram showing a selection screen for classification information used in image generation of the image generation system according to the first embodiment. [Figure 7B] 10 is an image presentation screen display used in an image presentation step and a classification information reception step. [Figure 8] FIG. 2 is a diagram showing an image generation screen of the image generation system according to the first embodiment. [Figure 9A] FIG. 10 is a diagram showing a case where a base image editing button on the image generation screen is pressed. [Figure 9B] FIG. 10 is a diagram showing a case where a mask is designated on the image generation screen. [Figure 10] FIG. 10 is a diagram showing an image generation screen of the image generation system according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] In this embodiment, we consider generating training data for a classification device (classification system) that receives an input of an image of an object, such as an inspection target, and outputs classification information about the object based on a learning model. To generate the training data, we use an image generation system that generates images.

[0011] The image generation system uses a learning model that has previously learned information related to images that can be prepared by the operator of the device. The first image is an image showing an inspection object belonging to a first class, and the second image is an image showing an inspection object belonging to a second class. The learning model has previously learned at least one image belonging to the first class and one image belonging to the second class.

[0012] The image generation system can generate a third image, which is an image showing intermediate features between the first image and the second image, or an image showing features that are more distant from the first image than the second image. The third image is an image showing an object to be inspected that belongs to a third category that is neither the first category nor the second category. In this embodiment, the image generation system can assist in determining generation parameters for the image generation model to generate the third image desired by the operator of the image generation system. The learning model preferably has learned one or more images that belong to the third category.

[0013] According to the embodiment, appropriate generation parameters can be automatically determined or suggested, for example, so that even if the operator is an unskilled person with no knowledge of image generation or AI, the third image of the quality expected by the operator can be easily generated.

[0014] In the following explanation, we will assume that the jigs and tools used in the production process at a manufacturing plant are the objects of inspection. Molds and molds, which are examples of jigs and tools, can be considered to be in a normal state if the parts and products produced using them are free of problems and pass inspection at the manufacturing plant without any problems. In other words, even if the jigs and tools have dirt, discoloration, or scratches due to aging, they can be considered to be in a normal state as long as the produced parts and products do not suffer from any abnormalities. Since this normal state is a state that is aimed for through the production process, it is expected that the features of the appearance of the inspection object will be similar to one another. In other words, it is expected that the normal state can be easily classified into a single type. In the embodiment, the first classification will be explained as the normal state. An image of the inspection object in a normal state (normal image) will be referred to as the first image.

[0015] On the other hand, if a clear abnormality occurs in a part or product produced using the inspection object, for example, due to a deterioration in appearance, strength, or processability, and the part or product fails to pass inspection at the manufacturing plant, the inspection object is considered to be in a clearly abnormal state (clearly abnormal state). For example, this is the case when a part or product produced using the inspection object requires repair or disposal. More specifically, a clearly abnormal state occurs when the vent holes of a mold or tool are substantially blocked by resin or metal residue. Clearly abnormal states can be further classified into a wide variety of categories, including the clear abnormality of the vent holes described above and clear abnormalities in various other parts. In other words, it is estimated that the features of different types of clearly abnormal states are often dissimilar to each other. In the examples, the second category is described as a clearly abnormal state. An image of an inspection object in a clearly abnormal state (clearly abnormal image) is referred to as the second image.

[0016] Now, there may be a situation where the parts or products produced using the inspection object do not exhibit any abnormalities, but at least the appearance of the inspection object shows signs of approaching one of the clearly abnormal states. For example, this may be the case when a small portion of the vent holes in a mold or tool is blocked with resin, metal, or other residue. Here, the state of the inspection object in this manner will be referred to as an intermediate state or an intermediate abnormal state. In the embodiment, the third category will be described as an intermediate state. An image of an inspection object in an intermediate state (an intermediate image or an intermediate abnormal image) will be referred to as the third image.

[0017] The object shown in the image is not limited to an inspection object as long as it is an object that can be expected to be classified into three or more categories and there is a category that the operator finds difficult to acquire. Furthermore, the inspection object is not particularly limited. It is not limited to jigs and tools used to produce parts and products as described above, but can also be manufactured parts and products, such as home appliances and industrial products, as well as their parts and manufacturing equipment. For example, it includes conveyors, heavy machinery, etc.

[0018] It is expected that operators will want to obtain images showing inspection objects that belong to the third category, which are not easy to obtain as learning data for inspection equipment. For example, images showing an inspection object in an intermediate state where a clear abnormality is about to occur may not be easy to obtain, even in factories that manage the inspection objects. In this way, by using an image generation system to generate images that are relatively difficult for operators to obtain, it is possible to increase the amount of learning data that can be used to train the inspection equipment.

[0019] In this specification, the same or corresponding components are denoted by the same reference numerals, and repeated explanations of these components may be omitted.

[0020] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. [Example]

[0021] <Overview of Image Generation System 100> 1 is a diagram showing the functional configuration of an image generation system 100 according to this embodiment. The image generation system 100 according to this embodiment can be configured, for example, with one or more computers, and includes an image generation unit 102, a conversion unit 103, a generation parameter reception unit 104, an image / feature database 105, an image generation assistant unit 106, and an interface 201.

[0022] The interface 201 enables an operator to input data to the image generation system 100 and check output data from the image generation system 100. For example, the interface 201 is a PC (personal computer) equipped with input devices such as a keyboard and a mouse, a display device such as a liquid crystal display, and a storage device such as a hard disk drive. The interface 201 can refer to information related to an image captured using, for example, a camera (e.g., raw image data or image feature data) from a storage device, an image / feature database 105, or other devices.

[0023] The image generation unit 102 causes the image generation model 132 to generate a new image in response to input of generation parameters for specifying the features of the image to be generated. The image generation model 132 can be a large-scale learning model that has previously learned images, natural language information, and the like.

[0024] The conversion unit 103 converts, as necessary, the images generated by the image generation unit 102 or the images input via the interface 201. The conversion unit 103 can convert images into a data format that can be processed by the interface 201 or into a data format that is suitable for storage in the image / feature database 105.

[0025] The image / feature database 105 stores information related to images previously input through the interface 201 and images generated by the image generator 102, etc., and may include a storage device such as a hard disk drive. Specifically, the database stores, for example, visual information of an image in association with classification information. The visual information related to an image is not limited to raw image data, but may also include numerical data such as matrix data or vectors (feature vectors) obtained by converting an image using a specific procedure, or text data describing the image. The classification information for an image may be, for example, information indicating whether the image belongs to the first, second, or third classification. In this embodiment, this information indicates whether the image is in a normal state, an intermediate state (intermediate abnormal state), or a clearly abnormal state. Each state is not necessarily limited to one type. For example, depending on the characteristics of the object to be inspected and the operator's preferences, a clearly abnormal state in the second classification may be further classified into multiple subclasses. For example, a first clearly abnormal state may be classified as a scratch on a certain portion of the object to be inspected, and a second clearly abnormal state may be classified as a blockage of a hole in a certain portion. Similarly, intermediate abnormal conditions can also be classified into subcategories according to the type of clear abnormal condition, such as a first intermediate abnormal condition corresponding to a first clear abnormal condition, a second intermediate abnormal condition corresponding to a second clear abnormal condition, and so on.

[0026] The generation parameter receiving unit 104 acquires information relating to the image stored in the image / feature database 105 or input via the interface 201 .

[0027] The image generation assistant unit 106 determines mutation vectors 123, each with a point belonging to the first classification on the latent space 120 of the image generation model 132 at one end and a point belonging to the second classification on the other end, according to the information related to the loaded image (mutation vector creation step). It also determines a recommended value or recommended range (recommended generation prompt) indicating a point or range on the latent space 120 where the generation of the third image is likely to be expected. Details will be described later.

[0028] The interface 201 further outputs information related to the recommended generation prompt presented by the image generation assistant unit 106 in a format that can be perceived by the operator. Upon receiving a generation prompt via the interface 201, the image generation unit 102 causes the image generation model 132 to generate a generated image. The interface 201 may display the generated image, or may accept an instruction to store the generated image in the image and feature database 105. When issuing an instruction to store the generated image in the image and feature database 105, it is preferable to also accept input of classification information for the generated image.

[0029] <Image Generation Method by Image Generation Unit 102> The image generation unit 102 determines the generation parameters and causes the image generation model 132 to generate an image. The generation parameters are information specifying a point in a space (latent space 120) where image features learned by the image generation model 132 are collected. For example, a feature vector expressing the feature amount of an image can be used as the generation parameter, or text data explaining the content depicted in the image can be converted into the generation parameter using a known encoder (e.g., a stable defusion model). In other words, the generation parameters are an example of visual information related to an image.

[0030] In this embodiment, a mutation vector 123 is considered, which is a vector with the first image at one end and the second image at the other end in the latent space. The third image (i.e., an image of an intermediate state) desired by the operator in this embodiment is considered to exist in a region at least a certain distance from the first image, which indicates a normal state, toward a clearly abnormal state, i.e., toward the other end of this mutation vector. The image generation assistance unit 106 calculates the distance (norm) from the first image to the second image, at which the operator's desired third image is likely to be obtained, as a point or range, and presents this as a recommended distance (recommended generation parameter) via the interface 201. When the operator specifies a distance (inputs a generation prompt) via the interface 201 with reference to the recommended distance, the image generation unit 102 causes the image generation model 132 to generate an image using a point at the specified distance from one end of the mutation vector 123 toward the other end in the latent space 120 as a generation parameter. This allows an unknown image to be restored (i.e., generated).

[0031] Even when generating an image using the same generation parameters through the input of the same generation prompt, the image generation model 132 may generate a different image each time by adding random initial values ​​sampled from random noise, for example, to the generation parameters. This allows the generation of an image corresponding to generation parameters close to the obtained generation parameters.

[0032] <Creation of mutation vector 123> The step of creating a mutation vector will now be described.

[0033] 2 is a diagram showing a space (latent space 120) represented by the generation parameters 121 before the image generation model 132 by the image generation unit 102 decodes the image. For ease of explanation, the latent space 120 of the image generation model 132 is conveniently shown as a two-dimensional space in FIG. 2. The latent space 120 of the image generation model 132 may be a multidimensional space of three or more dimensions.

[0034] 2 shows a first image 122a of the mold to be inspected in a normal state and a second image 122d of a subcategory (first clear abnormal state) that is one type of clear abnormal state. The image 122 is expressed as generative parameters 121 in the latent space 120. The first image 122a and the second image 122d can refer to, for example, images stored in the image and feature database 105 or images input from the interface 201.

[0035] In the latent space 120 of this image generation model 132, a vector with the position of generation parameter 121a at one end and the position of generation parameter 121d at the other end is the mutation vector 123. There are no particular limitations on the direction of the mutation vector 123, indicating its positive or negative sign, but in this embodiment, it is a vector pointing from one end to the other. The number of mutation vectors 123 can be created equal to the product of the number of known classifications in the first classification and the number of known classifications in the second classification. In this embodiment, the first classification is assumed to include only normal states, i.e., the number of classifications is 1. Therefore, the number of mutation vectors 123 can be created equal to the number of classifications in the second classification, i.e., the number of subclassifications of clearly abnormal states.

[0036] As the first image 122a, any image belonging to the first class may be used as a representative, or, for example, a representative point of the generation parameters of multiple images belonging to the first class may be found by a known method such as clustering, and an image generated using the generation parameters of the found representative point may be treated as the first image 122a. The same applies to the second image 122d. Therefore, information regarding the two images that are the start and end points of the mutation vector 123 can be obtained by the trained learning model itself.

[0037] The second category of clearly abnormal conditions has multiple lower-level categories (subcategories), so for each subcategory, any image belonging to that subcategory may be used as a representative. The following describes one of the subcategories, the first clearly abnormal condition.

[0038] As a characteristic of the generated image 122, the closer the generation parameters 121 are to the generation parameter 121d in the latent space 120, the higher the degree of the first clear abnormality the generated image will be. For example, the generated image 122c generated from the generation parameters 121c that are closer to the generation parameter 121d than the generation parameter 121b has a more severe first clear abnormality than the generated image 122b generated from the generation parameters 121b.

[0039] <Determination of Recommended Norm (Recommended Generation Prompt) by Image Generation Assistant Unit 106> FIG. 3 is a diagram showing a method for determining a recommended norm (recommended generation prompt) from the first image 122a, which is one end of the variation vector 123, to the other end.

[0040] In a first method for determining the recommended norm by the image generation assistant unit 106, the image generation model 132 is first made to generate images, for example, for each fixed norm, between both ends of the mutation vector 123. In FIG. 3, images are generated using a total of four generation parameters 121e, 121f, 121g, and 121h, from point 121e, which is about 40% away from the position of generation parameter 121a toward generation parameter 121d, to point 121h, which is about 60% away, and are presented to the operator via interface 201. Of course, there are no limitations on how far away from generation parameter 121a the image is presented, or how far away from generation parameter 121d the image is presented. One or both of these two generation parameters 121 may be specified by the operator.

[0041] In Figure 3, only two images 122f and 122g are shown out of the four images corresponding to the four generation parameters, but in reality all four images are presented. Based on their own knowledge and experience, the operator selects an image that is closest to the normal state, which is the classification to which the starting point of the mutation vector 123 belongs, but is not in the normal state (i.e., an image belonging to the third classification). Here, it is assumed that image 122f is selected.

[0042] Then, the image generation assistant unit 106 can determine that the norm from generation parameter 121a to generation parameter 121f is, for example, the lower limit of the recommended norm. Similarly, the operator may select an image that is closest to the first definite abnormal state, which is the classification to which the end point of mutation vector 123 belongs, but is not the first definite abnormal state, based on his or her own knowledge, experience, etc. In this case, the image generation assistant unit 106 can determine that the norm from generation parameter 121a to the generation parameter of the selected image is, for example, the upper limit of the recommended norm.

[0043] The image generation assistant unit 106 may repeat this procedure. That is, when the image 122f is selected, a new image may be generated at a point slightly closer to the starting point of the mutation vector 123 than the image 122f, and the operator may be asked whether this is a normal state (in this embodiment, whether it is a normal state or an intermediate state). The same applies to the end point of the mutation vector 123. In this way, the range of the recommended norm can be narrowed down more accurately. The recommended norm can be narrowed down either for the lower limit value only, the upper limit value only, or for the lower limit value and the upper limit value.

[0044] It is also possible to present images one by one in sequence and ask the operator to indicate which category they think the images belong to, rather than presenting multiple images at once. In either case, if the operator answers that at least one image belongs to the third category, the range of recommended norms can be narrowed down.

[0045] As a second method for determining the recommended norm by the image generation assistant unit 106, an image that is close to, preferably closest to, the normal state but not the normal state and / or an image that is close to, preferably closest to, the first clearly abnormal state but not the clearly abnormal state, as determined by the image generation model 132, may be automatically presented. In this case, the operator can answer by determining to which category the presented image belongs, such as the normal state, the intermediate state, or the first clearly abnormal state. The image generation assistant unit 106 may learn the operator's answer and then repeat this procedure, for example, by presenting the image again. In this way, the range of the recommended norm can be narrowed down in the same way as in the first determination method described above.

[0046] Note that the generation parameters 121 for generating the intermediate abnormal image do not necessarily exist only on the deviation vector 123. For this reason, for example, points near the deviation vector 123 may be presented by adding noise as described above.

[0047] As a third method for determining the recommended norm by the image generation assistance unit 106, the operator can input, for example, one image that the operator considers to be, for example, the most likely intermediate state from among the images on hand. In this case, the norm indicating the generation parameters of this image can be presented as the recommended norm value via the interface 201. In this case, the generation parameters of the on-hand image are not necessarily points on the mutation vector 123. In this case, the norm from this generation parameter to the intersection of a perpendicular line dropped to the mutation vector 123 can be set as the recommended norm value. When two or more images are input, the recommended norm can also be presented as a range.

[0048] By applying this third determination method, the image generation system 100 can also function as a classification system (inspection system) that receives an input image of an object and classifies the object in the image. Since the position of the generation parameters of the input image in the latent space 120 of the image generation model 132 can be determined, it is possible to determine which classification region of the image generation model 132 this position belongs to. For example, if the corresponding position in the latent space 120 of the input image matches the generation parameters 121 of a known clear anomaly or intermediate anomaly image stored in the image feature database 105, or if the distance between the corresponding position in the latent space 120 of the input image and the known generation parameters 121 is shorter than a predetermined distance, the image generation system 100 can determine that the input image contains the clear anomaly or intermediate anomaly. In this way, the image generation system 100 and the inspection system may be implemented in the same system.

[0049] Any method can be used to calculate the generation parameters 121, as long as it converts the input image into generation parameters 121 corresponding to the latent space 120 of the image generation model 132 to be used. For example, a method can be used that uses CLIP features calculated by encoding from the image using a pre-trained learning model, or a method that directly uses the output of an image caption generation model such as BLIP or ClipCap. Alternatively, CLIP features obtained by inputting the output of such an image caption generation model into a CLIP encoder may be used.

[0050] <Hardware configuration of image generation system 100> 4 is a diagram illustrating an example of the hardware configuration of the image generation system 100 according to this embodiment. The image generation system 100 includes the interface 201 shown in FIG. 1, as well as a CPU (Central Processing Unit) 114, a main storage device 115, and an auxiliary storage device 116.

[0051] The CPU 114 controls the image generation system 100, performs data calculations, etc. The CPU 114 executes a program deployed in the main storage device 115, for example, to realize the functions of the image generation system 100.

[0052] The main memory device 115 can be configured, for example, with a RAM (Random Access Memory). The main memory device 115 stores programs including software codes for executing various functions and methods executed by the image generation system 100, such as the image generation unit 102, the generation parameter reception unit 104, and the image generation auxiliary unit 106.

[0053] The auxiliary storage device 116 can be configured, for example, with a hard disk drive or the like, and realizes the image and feature database 105. The auxiliary storage device 116 stores various data, such as image data 131 as stored images, an image generation model 132, and stored generation parameters 133. The image data 131 includes images captured by an operator using a camera or the like, and generated images 122 generated by the image generation system 100. The stored generation parameters 133 store, for some, or preferably all, of the image data 131, a set of the generation parameters 121 and classification information, such as a normal state, an intermediate state, and a clearly abnormal state (and further, a first clearly abnormal state, a second clearly abnormal state, etc.). In this embodiment, there is considered to be only one type of normal state, and multiple types of each of the intermediate state and the clearly abnormal state, but this is not necessarily limited to this. The classification information can be appropriately defined by the operator as needed.

[0054] The auxiliary storage device 116 may also store mutation vectors 123 connecting each representative point on the latent space 120 of a subclassification (first subclassification) belonging to the first classification with each representative point on the latent space 120 of a subclassification (second subclassification) belonging to the second classification, and recommended norm values ​​or recommended norm ranges (recommended generation prompts) corresponding to each mutation vector 123.

[0055] There is no particular limit to the number of devices (computers) that make up image generation system 100, but one possible mode is to provide an interface 201 in a computer (terminal such as a PC or tablet) managed by an operator, and a CPU 114, a main memory device 115, and an auxiliary memory device 116 in a computer (server) managed by an image generation service provider. The computer managed by the operator and the computer managed by the image generation service provider can send and receive information and programs via a known communication network such as the Internet.

[0056] The CPU 114, the main storage device 115, and the auxiliary storage device 116 may each be implemented by multiple devices. For example, they may be provided in a device managed by the operator, or in other devices. That is, the CPU 114, the main storage device 115, the auxiliary storage device 116, and the program that implement the image generation system 100 may be integrated into a computer managed by the operator, a computer managed by the image generation service provider, or a computer managed by any other party. The CPU 114, the main storage device 115, and the auxiliary storage device 116 may all be installed in a single computer managed by one party, and the program may be installed therein. Alternatively, the CPU 114, the main storage device 115, and the auxiliary storage device 116 may be distributed among two or more computers managed by different parties, and the program may also be distributed among two or more computers managed by different parties. In this way, the implementation of the image generation system 100 is not particularly limited.

[0057] When a portion of the program is installed on a computer managed by an operator, the image generation service provider can provide the necessary program to the computer managed by the operator via a communications network. Alternatively, the entire program may be installed on the computer managed by the image generation service provider, and the computer managed by the operator may only input and output data to and from that computer. These programs may be stored on a hard disk drive, solid-state drive, optical disk, or other storage medium or recording medium.

[0058] <Flowchart showing the steps for how the recommended generated prompts are determined> Fig. 5 is an example of a flowchart showing the procedure of a method for determining a recommended generation prompt according to this embodiment. That is, Fig. 5 shows an example of a flowchart of a process in which the image generation system 100 according to this embodiment determines a recommended generation prompt.

[0059] In step S101, classification information is assigned to prepared images (preparation step). For example, images stored in the image / feature database 105 or images stored by an operator in the interface 201 are read, and one classification information is assigned to each image. In this embodiment, the images are classified into one group selected from a group including at least the first classification, normal state, and the second classification, clearly abnormal state. This group may also include the third classification, intermediate abnormal state. For classifications that are further divided into subclasses, classification information related to the subclass is also assigned. In this embodiment, the clearly abnormal state and intermediate abnormal state are further classified into subclasses. This selection may be performed by the operator by visually determining the image, or may be performed automatically using known image recognition technology. In this step, it is sufficient to have two images that belong to at least two different classification information. In this embodiment, for example, it is sufficient to have one or more images classified into a normal state and one or more images classified into one of the subclasses of clearly abnormal states.

[0060] If the number of images that can be prepared is small, it is also possible to have the image generation model 132 generate additional images and assign classification information to those images.

[0061] Automatic classification is also possible using the operator's work history information. For example, consider a system that automatically captures images of inspection objects during a manufacturing process. If a product is manufactured properly without the involvement of a manufacturing worker, the images of the inspection object involved in the manufacturing process can be classified as showing a normal state. Furthermore, if some abnormality is found in the product, the manufacturing worker can input classification information for the abnormality, and the images of the inspection object involved in the manufacturing process can be classified as showing a clear abnormality corresponding to the type of abnormality. Furthermore, if a manufacturing worker performs some minor repair work on a product, the inspection object involved in the manufacturing process can be determined to be in an intermediate abnormal state, so an image of the inspection object immediately before the repair work can be classified as an intermediate abnormal image. Furthermore, an image of the inspection object after the repair work can be classified as an intermediate abnormal image or a normal image.

[0062] The presence or absence of repair work on the inspection target may be determined, for example, by referring to the work history of the operator, or may be determined automatically by using an image recognition model for recognizing the movements and positions of workers, such as general human detection or skeletal detection. The person who performs the repair work is not particularly limited as long as they are able to perform the repair work expected at the manufacturing site, and may be, for example, a human such as a manufacturing worker, or a machine such as a robot introduced in the manufacturing process.

[0063] In step S102, the image generation assistant unit 106 converts the image to which classification information has been assigned in step S101 into generation parameters corresponding to points on the latent space 120 of the image generation model 132 (generation parameter preparation step). At this time, for each generation parameter to which certain classification information has been assigned, generation parameters of the surrounding space may also be treated as generation parameters belonging to the same classification information.

[0064] Furthermore, if there is a generation parameter to which no classification information has been assigned, a classification is assigned with a provisional name to that generation parameter and its surroundings. For example, an area within a certain distance from this generation parameter in the latent space is classified with the same provisional name. Note that an image to which no classification information has been assigned is assumed to be classified with a wide variety of subclassifications. In this embodiment, since it is assumed that there is a high possibility that it belongs to the second classification, a provisional name is assigned as a subclassification belonging to the second classification.

[0065] In step S103, the image generation assistant unit 106 determines representative points for each of the first class (normal state) and the second class (clearly abnormal state) in the latent space 120 (representative point determination step). If subclassifications are defined for the first and second classes, it is advisable to also determine representative points for each subclassification. Then, mutation vectors 123 connecting the generation parameters corresponding to the representative points are created (mutation vector preparation step). The method for creating representative points is as described above. One image belonging to each class may be appropriately selected, and this generation parameter may be used as the representative point. The steps up to this point may be performed in advance using prepared images.

[0066] Next, images are presented to the operator and information about the third category is acquired, but before that, the selection of the minor category to be used this time may be received from the operator (minor category selection step).

[0067] In step S104, the image generation assistant unit 106, for example, sequentially selects one or more generation parameters 121 on the mutation vector 123 for each of the mutation vectors 123 determined in step S103, generates an image, and presents it to the operator (image presentation step). Then, it accepts a response from the operator regarding the classification information of each image (classification information acceptance step). In addition, it is possible to acquire generation parameters for the third classification (intermediate abnormal state) as described in the method for determining the recommended norm above.

[0068] In step S105, the image generation assistant unit 106 determines the value or range of the recommended norm as a recommended generation prompt for each mutation vector 123 using the information on the generation parameters of the third classification obtained in step S104, and outputs the recommended generation prompt to the interface 201 (recommended generation prompt output step). The recommended generation prompt may be displayed directly as a generation parameter rather than as a norm from one end of the mutation vector 123. In this way, the image generation system 100 also operates as a support system for determining recommended generation prompts.

[0069] <Selection screen for classification information to be used for image generation> 6 is a diagram showing a specification screen 17 that accepts specification of whether an application including the image generation system 100 is to be used as the image generation system 100 or an image inspection system (classification system). The specification screen includes a message section 171 that displays a message prompting the operator to select a function to be used on the interface 201 of the computer managed by the operator, an image generation function button 172 that starts the image generation system 100, and an image inspection function button 173 that starts the image inspection system.

[0070] <Selection screen for classification information to be used for image generation> FIG. 7A is a diagram showing a selection screen 250 for classification information to be used for image generation by the image generation system 100 according to this embodiment. When the operator presses the image generation function button 172, a selection screen for classification information to be used for image generation is displayed. The image generation system 100 displays registered (known) classification information. For example, the image generation system 100 references classification information linked to images stored in the image / feature database 105 and displays each discovered classification information on the selection screen 250. In this embodiment, it is assumed that only one subclassification, the first normal state classification, is registered as the normal state, and three subclassifications, the first to third clear abnormal state classifications, are registered as the clear abnormal state. If there is insufficient linked classification information, the operator can add more as appropriate.

[0071] The selection screen 250 includes a message section 251 requesting the selection of the desired subcategory information, a classification display section 252 displaying classification information extracted by referencing the image and feature database 105, and a usage selection section 253 accepting the operator's selection of whether to use each subcategory. In FIG. 6, the first subcategory (normal state) is automatically selected because it has only one subcategory. Furthermore, only the first and second clearly abnormal states are selected from the subcategories of the second subcategory (clearly abnormal state). As will be described in detail later, from the viewpoint of uniquely determining the generation parameters corresponding to the generation prompt, it is preferable that the number of subcategories in at least either the first or second subcategory is one. After completing these subcategory selection steps, proceed to the next step.

[0072] 7B shows an image presentation screen display 450 used in the image presentation step and classification information reception step described above. The image presentation screen display 450 displays images 122e-122h generated using the generation parameters 121e-121h described above, and a message section 451 requesting an answer as to the classification of these images. The image presentation screen display 450 also displays a first image 122a generated using the generation parameters at one end (representative point of the first classification) of a certain mutation vector 123, and a second image 122d generated using the generation parameters at the other end (representative point of the second classification), and further displays answer sections 453, 454, and 455 used to answer the classification of the presented images. For each of the images 122e-122h presented by the image generation system 100, the operator drags and drops the images 122e-122h to the first classification answering unit 453 if the image is determined to be the first classification, to the second classification answering unit 454 if the image is determined to be the second classification, or to the third classification answering unit 455 if the image is determined to be the third classification. This allows the image generation system 100 to receive the operator's response regarding the images 122e-122h presented in the image presentation step (classification information receiving step). While this method is preferable in terms of accuracy because it provides an answer as to which of the first, second, or third classifications each presented image belongs to, it is not limited to this method. For example, the operator may request an answer as to which of the presented images is determined to be the third classification. Furthermore, the first image 122a and the second image 122d used do not need to be strictly images at one end and the other end of a certain variation vector 123. The operator may prepare and use images that they deem appropriate. In this case, the displacement vector 123 may be calculated again using the designated image.

[0073] <Recommended generation prompt output and generation prompt input screen> 8 is a diagram showing an image generation screen 150 of the image generation system 100 according to this embodiment. The interface 201 displays the image generation screen 150 on a screen such as a liquid crystal display.

[0074] The image generation screen 150 includes a display area 151 for the first image, a display area 152 for the generated image, and a generation prompt setting area 153.

[0075] The display area 151 displays a base image set as the initial value when generating a generated image. The base image is an image expressed by the generation parameters at one end of the mutation vector 123. In this embodiment, an image belonging to the normal state is set as the base image. For example, the same image may be set each time from among the images belonging to the normal state. The display area 151 may also be left blank, with nothing displayed.

[0076] The display area 152 displays the generated image 122 that is the output of the image generator 102 .

[0077] The generation prompt setting area 153 includes a recommended generation prompt display section 154 that displays a recommended generation prompt, and a generation prompt instruction section 159 that specifies a generation prompt to be input to the image generation assistance section 106. The contents of the generation prompt instruction section 159 can be changed by the operator using the interface 201 (for example, a keyboard or a mouse).

[0078] The generation prompt setting area 153 of this embodiment is in the form of a scroll bar, with the start point (on the first image side) on the mutation vector 123 corresponding to each clear abnormal state being 0 and the end point (on the second image side) being 100. The recommended generation prompt display unit 154 is in the form of a dot indicating a recommended norm value for the first clear abnormality, and in the form of a rectangular area indicating a recommended norm range for the second clear abnormality. The generation prompt designation unit 159 is in the form of a single value being designated on a scroll bar.

[0079] However, the form of the generation prompt setting area 153 is not limited to this. The generation prompt may be defined in a continuous or discrete format, for example, in a numerical range other than 0 to 100. Alternatively, the latent space 120 may be expressed in a two-dimensional or three-dimensional format as shown in FIG. 3 , i.e., the latent space 120 may be expressed in a two-dimensional or three-dimensional format, and the start and end generation parameters 121a and 121d, the mutation vector 123 connecting them, and the recommended norm position or range 121f on the mutation vector 123 may be displayed on the interface 201. In this case, the operator may be able to specify a position on the displayed latent space 120 using the generation prompt specifying unit 159. In this case, the position that can be specified may be on the mutation vector 123 or elsewhere. As a method for expressing the latent space 120 in a low-dimensional two-dimensional or three-dimensional format, for example, it is possible to calculate the most effective direction for explaining the characteristics of the data using a known method such as principal component analysis and express these directions.

[0080] The operator specifies a generation prompt using the generation prompt instruction unit 159 (generation prompt reception step). Then, when the image generation button 157 is pressed, a generated image is generated using generation parameters uniquely determined by the generation prompt, i.e., generation parameters that are separated from the starting point of the mutation vector 123 by the norm specified in the generation prompt (image generation step). The generated image is displayed in the display area 152. There are as many generation prompts as the product of the number of subcategories of the selected first classification and the number of subcategories of the second classification. There are also the same number of mutation vectors 123 to be used. The uniquely determined generation parameters are determined, for example, as follows.

[0081] In this embodiment, since the number of subclasses in the first class is 1, a common generation parameter belonging to the first class can be selected at one end of each mutation vector 123. From this generation parameter toward the other end of each mutation vector 123, a norm is specified by the generation prompt specifying unit 159 as a generation prompt for each mutation vector 123. In other words, a vector is obtained that is oriented in the same direction as each mutation vector 123 and has the specified norm as its quantity. The point moved from the common generation parameter by the sum of these vectors is determined as the generation parameter.

[0082] Image generation button 157 is not essential, and a generated image may be displayed in display area 152 in response to an instruction from generation prompt instruction unit 159. Furthermore, when database registration button 158 is pressed, a generated image generated in response to the generation prompt instructed by generation prompt instruction unit 159 at the time of pressing can be stored in image / feature database 105 (generated image storage step). At this time, classification information of the generated image may also be stored.

[0083] In the generation prompt setting area 153, the operable range of the generation prompt setting section may be limited or an error message may be displayed so that the operator cannot set an inappropriate generation prompt that is lower than the recommended generation prompt. This makes it possible to prevent the generation of unintended images due to erroneous operation by the operator.

[0084] Furthermore, by pressing the base image change button 155, the base image can be changed to, for example, another image stored in the image and feature database 105 or an image based on other randomly sampled generation parameters. If it is difficult to generate a desirable third image as the generated image, the base image that is the starting point of the mutation vector 123 can be changed.

[0085] FIG. 9A is a diagram showing a case where the base image editing button 156 on the image generation screen 150 is pressed.

[0086] The image generation system 100 according to this embodiment can partially change the base image in the display area 151 in response to an input from the operator.

[0087] When the operator presses the base image editing button 156, for example, rectangular masks 1561, 1562 can be set on the base image in the display area 151 (mask setting step). The shape of the mask 1561 is not particularly limited, and it may be freely deformed, enlarged, or reduced through operation of the interface 201. The image generation assistance unit 106 instructs the image generation unit 102 to mutate the base image only within the areas of the set masks 1561, 1562. The image generation unit 102 may mutate the base image, for example, by prioritizing the range specified by the masks 1561, 1562 over the unspecified range. The mutation method is the same as described above, where the base image is mutated in the direction of the mutation vector 123 in response to a generation prompt from the generation prompt instruction unit 159, but the area where the mutation is ultimately performed is limited, for example, to the range of the masks 1561, 1562. Image generation models such as inpainting are known as techniques for performing image conversion or image generation within a specific range.

[0088] By providing such functionality, the image generation system 100 can efficiently generate an image that compensates for the missing image data when there is a lack of image data for an abnormality in a specific area as learning data for the inspection system.

[0089] FIG. 9B is a diagram showing a case where a mask 1561 on the image generation screen 150 is designated.

[0090] The operator can specify a specific mask by performing a predetermined operation on the interface 201 (for example, by moving the pointer onto the mask 1561 and then double-clicking the mouse). The specified mask 1561 can be confirmed to have been specified by, for example, a change in color or pattern within the mask. When the mask 1561 is specified, the value specified by the generation prompt specifying unit 159 is also switched to the value specified for the area within the mask 1561 being specified. With the mask 1561 specified, the value of the generation prompt specifying unit 159 can be changed. Similarly, the mask 1562 can be specified and the value specified for the area within the mask 1562 can be changed.

[0091] Furthermore, areas of the base image other than the masks 1561 and 1562 may be made specifiable, or may not be made specifiable. When the area is made specifiable, the value of the generation prompt indicator 159 may be treated as an indicator value for the entire base image including the masks 1561 and 1562, or may be treated as an indicator value for only the base image portion other than the masks 1561 and 1562. In the former case, the final indicator value for the masks 1561 and 1562 may be, for example, the average value of the values ​​of the generation prompt indicator 159 indicated for each of the masks 1561 and 1562 and the value of the generation prompt indicator 159 indicated for the portion other than the masks 1561 and 1562. When the area is made specifiable, the value of the generation prompt indicator 159 may be automatically set by the image generation assistance unit 106 with reference to the indicator values ​​for the masks 1561 and 1562, or may be set to an indicator value of 0 (i.e., no mutation).

[0092] By providing such a function, the image generation system 100 can efficiently generate a third image when, for example, the operator knows that the location of a certain type of clear abnormality is limited to a specific region or does not occur in a specific region. For example, in the former case, a mask is set in the specific region, and the instruction value for this mask is increased, and the instruction value for other regions is decreased or set to 0. In the latter case, a mask is set in the specific region, and the instruction value for this mask is decreased or set to 0, and the instruction value for other regions is increased.

[0093] The image generation system 100 according to this embodiment has the above configuration, and even if the operator is an unskilled person with no knowledge of image generation or AI, the operator can easily generate an image (intermediate image) that matches the degree of variation expected by the operator through intuitive operations. The generation parameter determination method according to this embodiment can determine parameters (recommended generation parameters) for generating such a third image. [Example]

[0094] The second embodiment of the present invention can be configured in the same manner as the first embodiment, except for the following points.

[0095] FIG. 10 is a diagram showing an image generation screen 350 of the image generation system 100 of this embodiment.

[0096] In this embodiment, the value of the generation prompt instruction unit 159 can specify an area farther from the first image than the set second image. That is, when the position of the first image, which is one end of the mutation vector 123 in the latent space 120, is set to 0 and the position of the second image, which is the other end, is set to, for example, 100, the image generation screen 350 can specify a value greater than 100. The image generation screen 350 includes a generation prompt limit change unit 190, and the operator can set an upper limit value greater than 100 via the interface 201.

[0097] The generated image generated from the generation parameters 121 selected in this manner is an image containing a severe abnormality that has never been observed in a real environment, or an image containing an unknown abnormality, when a clearly abnormal image is set as the second image. Furthermore, when an intermediate image is set as the second image, the image is closer to a clearly abnormal state. When an operator references an image generated in this manner and determines that the abnormality present in the image is an abnormality that could actually occur, the operator can save the generated image as an abnormal image and use it for training the inspection device, or for assisting the operator in becoming proficient or for risk prediction.

[0098] In the first embodiment, it was explained that the mutation vector 123 can be set for each type of clear abnormal condition. However, it is assumed that the difference between the types of clear abnormal conditions is so small that the operator determines that they should be treated as clear abnormal conditions of the same subcategory. Therefore, the image generation assistant unit 106 may be able to integrate multiple types of subcategories. That is, if the operator wants to integrate a first clear abnormal condition and, for example, a fifth clear abnormal condition from among the defined types of subcategories and define it as a first clear abnormal condition, the operator can instruct the image generation system 100 to integrate these two clear abnormal conditions into the first clear abnormal condition. Upon receiving the instruction, the image generation system 100 extracts data related to these two clear abnormal conditions from various data stored in the image / feature database 105, reclassifies all data as data related to the first clear abnormal condition, and recalculates the mutation vector, etc., related to the first clear abnormal condition.

[0099] In this embodiment, this procedure can prevent a large number of mutation vectors 123 from being generated.

[0100] Furthermore, when the generation parameters are input to the image caption generation model, they can be converted into a caption for the generated image generated by the image generation system 100 according to this embodiment. By displaying this caption on the interface 201 or storing it in a storage device such as the image / feature database 105, the operator can easily understand the abnormality that has occurred.

[0101] Furthermore, in the image generation system 100 according to this embodiment, the mutation vector 123 can be perturbed within a preset range. When the mutation vector 123 is perturbed, the start point and / or end point of the mutation vector in the latent space 120 fluctuates slightly. The mutation vector 123 is uniquely determined by the start point and end point, but by using the perturbed mutation vector 123 as a new mutation vector 123 and using this new mutation vector 123, it is expected that an image similar to the image generated by the mutation vector 123 before the perturbation is generated can be generated.

[0102] The present invention is not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to embodiments including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to delete part of the configuration of each embodiment, or to add or replace other configurations.

[0103] In addition, although the embodiments have described decision support for recommended generation prompts related to image generation and classification and inspection of images, the present invention can also be applied to decision support, classification, and inspection of recommended generation prompts related to the generation of text, music, and other objects. [Explanation of symbols]

[0104] 17...designation screen, 100...image generation system, 102...image generation unit, 103...conversion unit, 104...generation parameter reception unit, 105...image feature database, 106...image generation auxiliary unit, 114...CPU, 115...main memory device, 116...auxiliary memory device, 120...latent space, 121...generation parameters, 121a to 121h, 121j, 121k...generation parameters, 122...generated image, 122a...first image, 122b...generated image, 122c...generated image, 122d...second image, 122e to 122h...image, 123...mutation vector, 124...recommended anomaly generation prompt, 131...image data, 132...image generation model, 133...stored generation parameters, 150...image generation screen, 151...base image display area , 152...display area for generated image, 153...generation prompt setting area, 154...recommended generation prompt display section, 155...base image change button, 156...base image edit button, 157...image generation button, 158...database registration button, 159...generation prompt instruction section, 171...message section, 172...image generation function button, 173...image inspection function button, 190...generation prompt restriction change section, 201...interface, 250...selection screen, 251...message section, 252...classification display section, 253...usage selection section, 350...image generation screen, 450...image presentation screen display, 451...message section, 453...first classification answer section, 454...second classification answer section, 455...third classification answer section, 1561, 1562...mask.

Claims

1. A method for supporting the determination of generation prompts to be input to a generative model capable of generating images according to generation parameters, comprising: On one or more computers, a classification information receiving step of receiving, from an interface, classification information of images corresponding to one or more generation parameters; A generation prompt determination support method that executes a recommended generation prompt output step that outputs a recommended generation prompt after receiving in the classification information receiving step an answer that the classification information for at least one of the images corresponding to the one or more generation parameters is a third classification that is neither the first classification nor the second classification.

2. performing an image presentation step of outputting one or more images showing characteristics between the images belonging to the first class and the images belonging to the second class; 2. The method for supporting determination of a prompt to be generated according to claim 1, wherein the classification information receiving step executes a process of receiving, from the interface, classification information about the image output in the image presenting step.

3. The method for supporting the determination of a generation prompt according to claim 1, wherein the classification information receiving step executes a process of receiving, from the interface, information relating to an image to which the third classification has been assigned as the classification information, which has been stored in advance.

4. 2. The method for supporting determination of a generation prompt according to claim 1, wherein the recommended generation prompt output step executes a process of outputting the recommended generation prompt as a numerical value or a numerical range.

5. 2. The method for supporting determination of a generation prompt according to claim 1, wherein the recommended generation prompt output step executes a process of outputting the recommended generation prompt as a point or a range in a two-dimensional or three-dimensional space.

6. a subcategory selection step for accepting selection of a part or all of two or more registered subcategories for the second category; 2. The method for supporting determination of a generation prompt according to claim 1, wherein the recommended generation prompt output step executes a process of outputting the recommended generation prompt for each of the minor categories selected in the minor category selection step.

7. a preparation step of assigning classification information of one of a group including at least the first classification and the second classification to a plurality of prepared images; a generation parameter preparation step of creating generation parameters corresponding to the image to which classification information has been added; a representative point determination step of determining a representative point for each of the first and second classes; The method for supporting the determination of a generation prompt according to claim 1 , further comprising: a step of creating a mutation vector having a representative point of the first classification at one end and a representative point of the second classification at the other end.

8. 10. An image generation method for implementing the prompt decision support method of claim 6, comprising: a generation prompt receiving step of receiving a designation of the generation prompt for each of the minor categories; an image generating step of generating information relating to an image of generation parameters in accordance with the generation prompt received in the generation prompt receiving step;

9. 10. An image generation method for implementing the prompt decision support method of claim 1, comprising: a mask setting step of displaying, on the interface, a base image belonging to the first category and a mask that specifies a partial range of the base image; a generation prompt receiving step of receiving a designation of the generation prompt for each of the masks; and an image generation step of generating an image that is mutated from the base image by limiting the range specified by the mask or by prioritizing the specified range over the unspecified range.

10. 10. An image generation method for implementing the prompt decision support method of claim 1, comprising: The image generating method includes a generated image storing step of storing information related to the image generated by the generation prompt together with classification information of the image.

11. 10. A system for implementing the method of generating prompts for decision support according to claim 1, comprising: By running the program, the classification information receiving step; A system that performs the recommendation generation prompt output step.

12. 10. A program including software code for use in implementing the method for generating prompts and decision support according to claim 1, comprising: In the one or more computers, the classification information receiving step; and a recommendation generation prompt output step, Optionally, a program transmitted to said one or more computers via a communication line.

13. By providing the program according to claim 12 to the one or more computers, the classification information receiving step; A method for producing a system that executes the recommendation generation prompt output step.

14. Using a learning model trained by the decision support method for generating prompts according to claim 1, A classification system that assigns a classification to an image of an object, the classification being one of the first classification, the second classification, and the third classification.

Images