Image creation device

Abstract

To provide an image creation device that allows users to easily create images and animations even without being familiar with the format of text information to be input to an image generation device or motion data generation device. [Solution] An image creation device that performs the process of creating an image using a processor, wherein the processor accepts a task requested by the user, uses a large-scale language model (LLM) to extract image-related information necessary to perform the task from the task, and acquires an image from an image generation device based on the said information.

Description

Technical Field

[0001] This disclosure relates to an image creation device.

Background Art

[0002] In recent years, due to the progress of deep learning technology, attention has been paid to using CG (Computer Graphics) instead of real images as learning images. Patent Document 1 discloses a technique for creating various images by a program only by changing CG parameters and generating a large number of images suitable for learning images.

[0003] Also, as image generation technologies utilizing AI (Artificial Intelligence), devices for generating images from natural language (Text-to-Image) and devices for generating motion data (Text-to-Motion) have attracted attention. By using these technologies, it has become possible to easily generate images and animations based on user requests.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the conventional technology, it was necessary to create text information input to an image generation device or a motion data generation device in a content and format suitable for these devices. Therefore, the user needed to be familiar with the creation of the text input to each device. Also, when it was desired to modify the generated data, it was necessary to appropriately instruct how to modify the data.

[0006] Therefore, the purpose of this disclosure is to provide a device that allows users to easily create the images and animations they need, even if they are not familiar with creating text information to be input into an image generation device or motion data generation device. [Means for solving the problem]

[0007] The image creation apparatus disclosed herein is an image creation apparatus that performs the process of creating an image using a processor, wherein the processor accepts a task requested by a user, uses a large-scale language model to extract from the task information necessary to perform the task, and acquires the image from an image generation apparatus based on the information. [Effects of the Invention]

[0008] According to the image creation device disclosed herein, a large-scale language model is used to extract information about images from tasks requested by the user, and images are obtained from the image generation device based on the extracted information. Therefore, even if the user is not familiar with creating text information to input into the image generation device, they can obtain images from the image generation device simply by inputting the requested tasks. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the configuration of the image creation system for learning purposes. [Figure 2] This is a functional configuration diagram of the image creation device. [Figure 3] This is a functional configuration diagram of an image correction device. [Figure 4] This is a functional configuration diagram of a learning image generation device. [Figure 5] This is a functional configuration diagram of the learning device. [Figure 6] This is a diagram showing the hardware configuration. [Figure 7] This is a prompt for extracting information. [Figure 8] This is extracted information. [Figure 9] This is extracted information. [Figure 10]A prompt for generating character data. [Figure 11] A prompt for generating motion data. [Figure 12] A prompt for generating background data. [Figure 13] Character data parameters. [Figure 14] Motion data parameters. [Figure 15] Background data parameters. [Figure 16] Character data. [Figure 17] Motion data. [Figure 18] Background data. [Figure 19] An integrated image. [Figure 20] A flowchart (overall). [Figure 21] A flowchart (image creation device). [Figure 22] A flowchart (image correction device). [Figure 23] A flowchart (learning image creation device). [Figure 24] A flowchart (learning device). [Figure 25] An input screen. [Figure 26] An input screen.

Modes for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. Therefore, each component, the arrangement position and connection form of each component, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, the components not described in the independent claims are described as optional components.

[0011] Furthermore, each figure is a schematic diagram and not necessarily a strictly accurate representation. Note that in each figure, substantially identical components are denoted by the same reference numerals, and redundant explanations may be omitted or simplified.

[0012] (System Overview) First, an overview of the learning image creation system 100 of this disclosure will be described with reference to Figure 1. The learning image creation system 100 includes, for example, an image creation device 1 that creates a representative image based on a task input by a user, an image correction device 2 that corrects the representative image based on instructions from the user, a learning image creation device 3 that creates a learning image based on the representative image created by the image creation device 1 or the representative image corrected by the image correction device 2, and a learning device 4 that trains a machine learning model based on the learning image and acquires a trained model. It may also have a storage device 5 that stores data created by the image creation device 1, the image correction device 2, the learning image creation device 3, and the learning device 4. Furthermore, it may have a user terminal 6 that is connected to the image creation device 1, the image correction device 2, the learning image creation device 3, and the learning device 4 by a network, for the user to input information such as tasks and display the created images. The image creation device 1, the image correction device 2, the learning image creation device 3, the learning device 4, the storage device 5, and the user terminal 6 may each be configured as separate devices and connected by a network, or some or all of the devices may be configured as an integrated unit.

[0013] According to the training image creation system 100 of this disclosure, a representative image is created by the image creation device 1 based on a task input by the user. The created representative image is modified by the image modification device 2 as needed based on instructions from the user. Based on the representative image or the modified representative image, a training image similar to the representative image or the modified representative image is created by the training image creation device 3. The created training image is used by the training device 4 to train a machine learning model, and a trained model for performing the task can be obtained.

[0014] (Image creation device) Next, the image creation device 1 of this disclosure will be described. Figure 2 is a functional configuration diagram of the image creation device 1 of this disclosure. Each functional unit named "○○ unit" corresponds to each function performed by the processor 50 (see Figure 6) of the image creation device 1 when it executes an image creation program. The image creation device 1 uses a large-scale language model 7 and an image generation device 8 to create images necessary for executing a task input by the user. In addition, the image creation device 1 can activate a GUI (Graphical User Interface) that enables interactive processing in conjunction with the execution of the image creation program, thereby realizing user operations related to image creation.

[0015] The prompt generation unit 21 generates prompts to be input to the large-scale language model 7 in order to extract the information necessary to create an image from the task entered by the user. Here, the task can be text information that represents the behavioral event that the user wants to recognize using the created image, such as "I want to detect people looking around nervously in the city," and it does not necessarily have to describe what kind of image the user wants to create. In other words, a description of the purpose in which the user wants to detect or recognize a specific event related to moving objects included in surveillance footage can be used as the task.

[0016] A prompt may be written by inserting a task into a predetermined guidance. Guidance is a pre-prepared format for creating prompts, and a prompt is generated by combining guidance and a task. Figure 7 shows an example of the procedure for generating a prompt based on a task. (A) is a task entered by the user. (B) is a predetermined guidance. The guidance may include, for example, an "instruction prompt" where the position where the user-entered task is inserted is defined as "**", an "output specification" which defines the description format of the data output by the large-scale language model, and "extracted information" which defines the information to be extracted from the task.

[0017] The "instruction prompt" is a task entered by the user, and the text information entered by the user is pasted directly into the "**" field.

[0018] The "output specification" is the description format for the data output by the large-scale language model 7. For example, the output format is JSON (JavaScript Object Notation), and it is defined that 0 should be entered for information that cannot be extracted from the instruction prompt.

[0019] "Extracted information" is information extracted from the instruction prompt, and may be divided into categories such as "action task," "person attributes," "movement type," and "background type." "Action task" refers to actions performed using the image to be created. "Person attributes" refers to information about the attributes of the person appearing in the created image, and may be defined to extract information such as clothing, gender, height, hairstyle, and age. "Motion type" refers to information about the movement of the person appearing in the created image, and may be defined to extract information such as movement and speed. "Background type" refers to information about the background of the created image, and may be defined to extract information such as location, time of day, and level of crowding.

[0020] The prompt generation unit 21 generates a prompt (C) based on the task (A) entered by the user and the pre-set guidance (B). Specifically, it generates a prompt (C) by replacing the "**" in the guidance (B) with the task.

[0021] The information extraction unit 22 inputs the prompt generated by the prompt generation unit 21 to the large-scale language model 7 and obtains extracted information from the large-scale language model 7. The large-scale language model 7 can use a general-purpose AI, such as ChatGPT (registered trademark: OpenAI OpCo, LLC). Figure 8 shows an example of extracted information obtained by inputting the prompt from Figure 7(C) to the large-scale language model 7. Information corresponding to each item, "action task," "person attribute," "movement type," and "background type," is extracted from the instruction prompt. The extracted information may be stored in the storage device 5.

[0022] As an "action task," information about the task, "recognize a person looking around," is extracted from the instruction prompt.

[0023] Since the instruction prompt did not contain any information regarding "person attributes," no information was extracted for the fields for clothing, gender, height, hairstyle, and age, and all items were entered as 0.

[0024] For "Movement Type," information about the movement, "Person looking around," is extracted from the instruction prompt. On the other hand, since no information about speed was included in the instruction prompt, no information is extracted for the speed field, and 0 is entered.

[0025] The background information "urban area" is extracted from the instruction prompt as the "background type." On the other hand, since the instruction prompt did not contain any information regarding the time of day or congestion level, no information is extracted for the time of day and congestion level fields, and 0 is entered.

[0026] Furthermore, if there is any missing information in the extracted information obtained from the large-scale language model 7 (for example, if there are items where 0 is entered), the information extraction unit 22 may complete the extracted information by entering a pre-set initial value instead of 0. Figure 9 shows an example in which initial values ​​have been entered for items where 0 was entered in the information extracted by the large-scale language model 7 (Figure 8) to complete the extracted information. For the person attribute, "clothing" is entered as "black," "gender" as "male," "height" as "170," "hairstyle" as "short haircut," and "age" as "20." Also, for the movement type, "speed" is entered as "normal," and for the background type, "time of day" as "daytime" and "crowding level" as "small." Note that the initial values ​​may be set by the user using a settings screen (not shown) to set fixed values ​​for each attribute and type. Alternatively, if the user specifies automatic setting for each attribute and type from the settings screen, the initial values ​​for those attributes and types may be set randomly.

[0027] The prompt generation unit 23 generates prompts to obtain parameters to be input to the image generation device 8 from the large-scale language model 7 based on the extracted information in Figure 9. The image generation device 8 can be a general-purpose device such as a device that generates images from natural language (Text-to-Image) or a device that generates motion data (Text-to-Motion). In this embodiment, a person data generation device 10 that generates a 3D model of a person, a motion data generation device 11 that generates motion data of a person, and a background data generation device 12 that generates background data may be used.

[0028] The prompt generation unit 23 generates prompts for obtaining parameters to be input to the person data generation device 10 based on the extracted information, prompts for obtaining parameters to be input to the motion data generation device 11, and prompts for obtaining parameters to be input to the background data generation device 12. In this way, the prompt generation unit 23 generates prompts for each of the three categories: person, motion, and background. Note that the categories are not limited to these three; person may be replaced with moving objects such as vehicles, motion may be replaced with facial expressions, or additional categories such as foreground, season, and weather may be added.

[0029] Figure 10 shows an example of generating a person data parameter acquisition prompt based on extracted information. (A) is the information extracted as "person attributes" from the extracted information. (B) is a predetermined guidance. The prompt generation unit 23 generates a person data parameter acquisition prompt (C) from the person attributes (A) of the extracted information and the guidance for generating the person data parameter acquisition prompt (B). Specifically, the person data parameter acquisition prompt (C) is generated by replacing "**" in the guidance for generating the person data parameter acquisition prompt (B) with the person attributes (A).

[0030] Figure 11 shows an example of generating a motion data parameter acquisition prompt based on extracted information. (A) is the information extracted as "motion type" from the extracted information. (B) is a predetermined guidance. The prompt generation unit 23 generates a motion data parameter acquisition prompt (C) from the motion type (A) of the extracted information and the guidance for generating the motion data parameter acquisition prompt (B). Specifically, the motion data parameter acquisition prompt (C) is generated by replacing "**" in the guidance for generating the motion data parameter acquisition prompt (B) with the motion type (A).

[0031] Figure 12 shows an example of generating a background data parameter acquisition prompt based on extracted information. (A) is the information extracted as "background type" from the extracted information. (B) is a predetermined guidance. The prompt generation unit 23 generates a background data parameter acquisition prompt (C) from the background type (A) of the extracted information and the guidance for generating the background data parameter acquisition prompt (B). Specifically, the background data parameter acquisition prompt (C) is generated by replacing "**" in the guidance for generating the background data parameter acquisition prompt (B) with the background type (A).

[0032] The parameter acquisition unit 24 inputs a prompt for acquiring a person data parameter, a prompt for acquiring a motion data parameter, and a prompt for acquiring a background data parameter to the large-scale language model 7, and acquires person data parameters, motion data parameters, and background data parameters from the large-scale language model 7. The acquired parameters may be stored in the storage device 5.

[0033] Figure 13 shows an example of person data parameters obtained by inputting a person data parameter acquisition prompt into the large-scale language model 7. The person data parameters are prompts that include parameters to be input into the person data generation device 10, generated based on the extracted information.

[0034] Figure 14 shows an example of motion data parameters obtained by inputting a motion data parameter acquisition prompt into the large-scale language model 7. The motion data parameters are prompts that include parameters to be input into the motion data generation device 11, generated based on the extracted information.

[0035] Figure 15 shows an example of background data parameters obtained by inputting a background data parameter acquisition prompt into the large-scale language model 7. The background data parameters are prompts that include parameters to be input into the background data generation device 12, generated based on the extracted information.

[0036] The material data acquisition unit 25, consisting of a person data acquisition unit 26, a motion data acquisition unit 27, and a background data acquisition unit 28, inputs the person data parameters, motion data parameters, and background data parameters acquired by the parameter acquisition unit 24 from the large-scale language model 7 into the material data generation device 9 to acquire material data.

[0037] The person data acquisition unit 26 inputs person data parameters to the person data generation device 10 and acquires person data from the person data generation device 10. Figure 16 shows an example of person data. The person data is, for example, a 3D model 61 of a person generated by the person data generation device 10 based on the person data parameters. The person data is defined, for example, as a 3D model 61 that has information about appearance.

[0038] The motion data acquisition unit 27 inputs motion data parameters to the motion data generation device 11 and acquires motion data from the motion data generation device 11. Figure 17 shows an example of motion data. The motion data is, for example, motion data of a person 63 generated by the motion data generation device 11 based on motion data parameters. The motion data is defined, for example, as the movement of a person 63 (skeleton, joint positions, etc.) along a predetermined path 62 and the trajectory 64 of the person's movement.

[0039] The background data acquisition unit 28 inputs background data parameters to the background data generation device 12 and acquires background data from the background data generation device 12. Figure 18 shows an example of background data. The background data is, for example, background data generated by the background data generation device 12 based on background data parameters. The background data is defined as, for example, a space 65 in which objects such as buildings are arranged in three dimensions.

[0040] The image acquisition unit 29 inputs person data, motion data, and background data to the data integration device 13 to acquire an image in which a person with the appearance of the person data moves according to the motion data on the background data. The data integration device 13 can use a general device that integrates a 3D model of a person, motion data, and a 3D model of space to create a video, such as Unity (registered trademark: Unity Technologies). Figure 19 is an example of an image acquired by the image acquisition unit 29, which is created as an image (video) in which a person model 67 moves according to the motion data on the background data 66.

[0041] The display parameter determination unit 30 determines the parameters for rendering and displaying the image acquired by the image acquisition unit 29. The parameters may have initial values ​​set in advance, or they may be determined based on values ​​entered by the user. The image acquired by the image acquisition unit 29 is rendered based on the display parameters and displayed as a representative image, for example, on the user terminal 6.

[0042] (Image correction device) Next, the image correction device 2 of this disclosure will be described. Figure 3 is a functional configuration diagram of the image correction device 2 of this disclosure. Each functional unit named "○○ unit" corresponds to each function performed by the processor 50 of the image correction device 2 when it executes an image correction program. The image correction device 2 uses a large-scale language model 7 and an image generation device 8 to create a corrected image by correcting a representative image according to the user's correction instructions. In addition, the image correction device 2 can activate a GUI (Graphical User Interface) that enables interactive processing when the image correction program is executed, and can realize user operations related to image correction.

[0043] The prompt acquisition unit 33 of the image correction device 2 acquires user correction instructions for a representative image. The correction instructions are input by the user as text information, such as "Make the head turn a little more."

[0044] The parameter acquisition unit 34 acquires the person data parameters, motion data parameters, and background data parameters used when creating the representative image from the storage device 5.

[0045] The modified parameter acquisition unit 35 acquires the modified parameters from the large-scale language model 7 based on the modification instructions and parameters. The modified parameters are the person data parameters, motion data parameters and background data parameters that have been modified (changed) in part or in whole from the person data parameters, motion data parameters and background data parameters used when creating the representative image. The modified parameter acquisition unit 35 may also create a prompt to be input to the large-scale language model 7 based on the modification instructions and guidance, and acquire the modified parameters as output by inputting the prompt to the large-scale language model 7. The guidance can create a prompt that instructs the large-scale language model 7 to generate and output the modified parameters according to the modification instructions by inserting the modification instructions at a predetermined position.

[0046] The material data acquisition unit 25 inputs the modified parameters to the material data generation device 9 to acquire material data, specifically person data, motion data, and background data. The function of the material data acquisition unit 25 is the same as that of the material data acquisition unit 25 of the image creation device 1, so a detailed explanation is omitted.

[0047] The image acquisition unit 29 inputs the material data acquired by the material data acquisition unit 25, specifically the person data, motion data, and background data, into the data integration device 13 to acquire an image in which a person with the appearance of the person data moves according to the motion data on top of the background data. The function of the image acquisition unit 29 is the same as that of the image acquisition unit 29 of the image creation device 1, so a detailed explanation is omitted.

[0048] The display parameter determination unit 30 determines the parameters for rendering and displaying the image acquired by the image acquisition unit 29. The parameters may have initial values ​​set in advance, or they may be determined based on values ​​entered by the user. The image acquired by the image acquisition unit 29 is rendered based on the display parameters and displayed as a representative image, for example, on the user terminal 6.

[0049] (Image creation device for learning) Next, the learning image creation device 3 of this disclosure will be described. Figure 4 is a functional configuration diagram of the learning image creation device 3 of this disclosure. Each functional unit named "○○ unit" corresponds to the functions performed by the processor 50 of the learning image creation device 3 when it executes a learning image creation program. The learning image creation device 3 uses a large-scale language model 7 and an image generation device 8 to create learning images necessary for executing a task input by the user.

[0050] The prompt acquisition unit 36 ​​of the learning image creation device 3 acquires the user's instructions for creating a learning image. The instructions are input by the user as text information, for example, "Create a learning image."

[0051] The parameter acquisition unit 37 acquires the person data parameters, motion data parameters, and background data parameters used when creating the representative image from the storage device 5.

[0052] The learning parameter acquisition unit 38 acquires parameters generated according to the creation instructions from the large-scale language model 7 based on the creation instructions and parameters. The generated parameters are person data parameters, motion data parameters, and background data parameters, some or all of which have been modified (changed) for learning purposes when creating the representative image. The learning parameter acquisition unit 38 may also create prompts to be input to the large-scale language model 7 based on the creation instructions and guidance, and acquire the modified parameters as output by inputting the prompts to the large-scale language model 7. The guidance can create prompts that instruct the large-scale language model 7 to generate and output parameters according to the creation instructions by inserting the creation instructions at predetermined positions.

[0053] The material data acquisition unit 25 inputs the modified parameters to the material data generation device 9 to acquire material data, specifically person data, motion data, and background data. The function of the material data acquisition unit 25 is the same as that of the material data acquisition unit 25 of the image creation device 1, so a detailed explanation is omitted.

[0054] The image acquisition unit 29 inputs the material data acquired by the material data acquisition unit 25, specifically the person data, motion data, and background data, into the data integration device 13 to acquire an image in which a person with the appearance of the person data moves according to the motion data on top of the background data. The function of the image acquisition unit 29 is the same as that of the image acquisition unit 29 of the image creation device 1, so a detailed explanation is omitted.

[0055] The display parameter determination unit 30 determines the parameters for rendering and displaying the image acquired by the image acquisition unit 29. The parameters may have initial values ​​set in advance, or they may be determined based on values ​​entered by the user. The image acquired by the image acquisition unit 29 is rendered based on the display parameters and displayed as a representative image, for example, on the user terminal 6.

[0056] (Learning device) Next, the learning device 4 of this disclosure will be described. Figure 5 is a functional configuration diagram of the learning device 4 of this disclosure. Each functional unit named "○○ unit" corresponds to each function performed by the processor 50 of the learning device 4 when it executes a learning program. The learning device 4 learns an initial model for learning using learning images, and further evaluates the accuracy of the learned model to determine its accuracy.

[0057] The task acquisition unit 41 of the learning device 4 acquires the task entered by the user from the storage device 5. Then, the initial model selection unit 42 selects an initial model suitable for executing the task from among the existing models. Once the initial model is selected, the model learning unit 43 learns the initial model based on the learning images created by the learning image creation device 3. Existing learning methods can be used for the learning method.

[0058] Once a trained model is obtained, the accuracy evaluation unit 44 evaluates the accuracy of the trained model. For example, pre-prepared evaluation images may be used to evaluate accuracy, and the probability that the output obtained by inputting the evaluation image into the trained model is the desired output may be calculated. For example, if the task is "to detect people looking around in the city," then images of people walking in the city may be input into the trained model as evaluation images, and the probability that the person is looking around in the image will be recognized, and the probability that the person is not looking around in the image will be calculated. The performance of the trained model can be evaluated using generally known evaluation methods for trained models.

[0059] The accuracy determination unit 45 determines whether the trained model is successful or unsuccessful based on the evaluation results output by the accuracy evaluation unit 44, and outputs the result to, for example, the user terminal 6. The pass / fail determination may be made, for example, by determining whether the probability of correctly recognizing the evaluation image exceeds a predetermined threshold. If the model fails, the user may again input a training image generation instruction prompt to the training image creation device 3 to create a new training image, and then use that training image to perform additional training on the trained model.

[0060] Furthermore, each trained model is saved to storage 52, and version control of each model is possible, including trained models built through additional training. For example, in a case where an additionally trained model currently undergoing accuracy evaluation does not show improvement in recognition probability, it may be better to revert to the previous or earlier version of the trained model and perform additional training. In such cases, a rollback instruction can be issued to revert to any previous version of the trained model, changing the additional training plan and performing additional training (retraining).

[0061] (Hardware configuration) Next, Figure 6 shows an example of the hardware configuration of the image creation device 1, image correction device 2, learning image creation device 3, and learning device 4. The image creation device 1, image correction device 2, learning image creation device 3, and learning device 4 may have the same configuration or some different configurations, as will be explained below. The processor 50 of the image creation device 1, image correction device 2, learning image creation device 3, and learning device 4 is connected to memory 51, storage 52, drive 53, user interface 54, and communication device 55 via bus 56.

[0062] The processor 50 executes the image creation program, image modification program, training image creation program, or training program stored in the storage 52, while temporarily recording data in the memory 51. The storage 52 is a storage device such as an SSD, and in addition to the operating system, images and various data, it also stores the image creation program, image modification program, training image creation program, or training program. The image creation program, image modification program, training image creation program, or training program works in cooperation with the operating system to perform its functions.

[0063] Drive 53 refers to optical drives, USB memory sticks, etc., and is used for reading programs, transferring data, etc.

[0064] The user interface 54 is a device that allows the user (administrator) to input necessary information and display calculation results, and specifically includes a display, keyboard, mouse, touch panel, etc.

[0065] The communication device 55 is a device for connecting to a network such as the Internet or a LAN. The image creation device 1, image correction device 2, learning image creation device 3, and learning device 4 may be configured together with the user terminal 6 operated by the user, or they may be configured as server devices connected to the user terminal 6 via a network.

[0066] (Overall flow) Next, the operation of the learning image creation system 100 of this disclosure will be explained in accordance with the flowchart in Figure 20. First, the processor 50 of the learning image creation system 100 receives task input from the user (step S1). A task is something the user wants to do based on the created image, and is, for example, something the user inputs to the learning image creation system 100 as text information. For example, a task is input to the learning image creation system 100 as text information, such as "I want to detect people who are looking around nervously in the city."

[0067] Next, the processor 50 generates an image (representative image) to fulfill the task entered by the user (step S2). Details of step S2 will be described later.

[0068] Next, the processor 50 displays the generated representative image on the display (step S3). The user checks the displayed image, determines whether the generated image matches their image, and inputs the result into the learning image creation system 100 (step S4).

[0069] If the representative image does not match the user's image (step S4 is NO), the processor 50 accepts a correction instruction for the representative image from the user (step S5). On the other hand, if the representative image matches the user's image (step S4 is YES), the processor 50 then generates a training image (step S6). Details of steps S5 and S6 will be described later.

[0070] Next, the processor 50 uses the generated training images to train a machine learning model and obtain a performance benchmark (step S7). Model training can also be done using general machine learning methods, such as inputting training data consisting of training images and correct labels into the model, and a trained model is obtained through training. More details about step S7 will be described later.

[0071] Next, the training image creation system 100 inputs images for performance testing into the trained model and calculates the accuracy rate, for example, for image recognition. It then determines whether the accuracy rate meets a predetermined performance standard (step S8).

[0072] If the accuracy rate does not meet the predetermined standard (step S8 is NO), the training image creation system 100 generates training images again (step S6) and repeats the model training. If the accuracy rate meets the predetermined standard (step S8 is YES), the training image creation system 100 outputs a trained model that meets the performance standard (step S9).

[0073] (Representative image creation flow) Next, the process by which the image creation device 1 of this disclosure creates a representative image according to the image creation program (step S2 in Figure 20) will be explained with reference to the flowchart in Figure 21. When the image creation device 1 receives a task from the user as text information, such as "I want to detect people looking around in the city" (step S11), it generates a prompt to be input into a large-scale language model (indicated as LLM in the figure) in order to extract information about the image to be created from this text information (step S12). The prompt is generated by combining the task and predetermined guidance (Figure 7).

[0074] The image creation device 1 inputs a prompt to the large-scale language model 7 and obtains the information extracted by the large-scale language model 7 from the task (step S13). Figure 8 shows an example of the extracted information obtained from the large-scale language model 7.

[0075] Next, the image creation device 1 determines whether all the information necessary to create the image has been extracted (step S14). Specifically, it checks whether there are any "0" values ​​in each item of the extracted information (Figure 8). If there are any "0" values, it determines that the item could not be extracted (step S14 is No), and enters a predetermined initial value for the item with "0" to complete the extracted information (step S15). If all the necessary information has been extracted (step S14 is Yes), the extracted information is used as is.

[0076] Figure 9 shows the extracted information after initial values ​​have been entered in step S15 and all necessary information is available. Initial values ​​have been entered for all items from which information was not extracted in step S13 (items that had "0" entered), and there are no "0"s left.

[0077] Next, the image creation device 1 generates prompts (Figures 10 to 12) to be input to the large-scale language model 7 in order to generate prompts to be input to the person data generation device 10, the motion data generation device 11, and the background data generation device 12 (collectively referred to as the "material data generation device 9") from the extracted information (Figure 9) (step S16).

[0078] Next, the image creation device 1 inputs the prompt generated in step S16 into the large-scale language model 7 and obtains its output (Figures 13 to 15) (step S17).

[0079] Next, the image creation device 1 inputs the parameters acquired in step S17 to the person data generation device 10, the motion data generation device 11, and the background data generation device 12, respectively, to acquire person data (Figure 16), motion data (Figure 17), and background data (Figure 18) (step S18).

[0080] Next, the image creation device 1 inputs the acquired person data, motion data, and background data into the data integration device 13 to obtain an integrated image (Figure 19).

[0081] Next, the image creation device 1 determines the viewpoint of the integrated image (step S20) and displays the image on the display (step S21). The viewpoint may use a value stored by default, or it may be entered by the user.

[0082] (Image correction flow) Next, the process by which the image correction device 2 of this disclosure corrects a representative image according to the image correction program (step S5 in Figure 20) will be explained with reference to the flowchart in Figure 22. If correction of the representative image is necessary, the image correction device 2 receives a correction instruction from the user (step S31). The correction instruction is made by the user inputting text information, for example, "Make the head turn a little more."

[0083] Next, the image correction device 2 acquires the person data parameters, motion data parameters, and background data parameters of the representative image stored in the storage device 5 (step S32). Next, the image correction device 2 acquires parameters that have been modified in part or all of the acquired parameters (step S33). The modified parameters may be obtained by inputting the correction instructions and the parameters of the representative image into the large-scale language model 7, for example, and obtaining the modified parameters from the large-scale language model 7. The modified parameters can be obtained from the large-scale language model 7 by inputting a prompt generated by inserting the correction instructions and parameters into guidance such as "Modify the following parameters based on the following correction instructions."

[0084] Once the corrected person data parameters, motion data parameters, and background data parameters are obtained, the corrected parameters are input to the person data generation device 10, motion data generation device 11, and background data generation device 12, respectively, to obtain person data, motion data, and background data (step S34), similar to steps S18 to S21 in Figure 21. The person data, motion data, and background data are then input to the data integration device 13 to obtain an image (step S35), and the image is displayed (steps S36, S37). This process is repeated until no further corrections are needed.

[0085] (Workflow for creating training images) Next, the process by which the learning image creation device 3 of this disclosure creates a learning image according to the learning image creation program (step S6 in Figure 20) will be described with reference to the flowchart in Figure 23. The learning image creation device 3 acquires the person data parameters, motion data parameters, and background data parameters of a representative image stored in the storage device 5 (step S41). Next, the learning image creation device 3 acquires learning person data parameters, learning motion data parameters, and learning background data parameters by modifying some or all of the acquired parameters (step S42). The learning person data parameters, learning motion data parameters, and learning background data parameters may be obtained by inputting the person data parameters, motion data parameters, and background data parameters of the representative image into the large-scale language model 7 and modifying some or all of them. The large-scale language model 7 can acquire learning parameters by inputting a prompt generated by inserting parameters into guidance such as "Modify some of the following parameters to generate learning parameters."

[0086] Once the training person data parameters, training motion data parameters, and training background data parameters are obtained, the training image creation device 3 inputs the training person data parameters, training motion data parameters, and training background data parameters to the person data generation device 10, motion data generation device 11, and background data generation device 12, respectively, to acquire person data, motion data, and background data (step S43).

[0087] The learning image creation device 3 inputs person data, motion data, and background data into the data integration device 13 to obtain an image in which a person with the appearance of the person data moves according to the motion data on top of the background data (step S44).

[0088] Finally, the learning image creation device 3 displays the created learning images as needed (steps S45, S46). The learning image creation device 3 repeats the above process until the required number of learning images are obtained.

[0089] (Learning flow) Next, the process by which the learning device 4 of this disclosure learns a machine learning model according to the learning program (step S7 in Figure 20) will be explained with reference to the flowchart in Figure 24. The learning device 4 acquires tasks stored in the storage device 5 (step S51). Next, the learning device 4 selects an appropriate machine learning model as the initial model according to the task (step S52). The initial model is selected according to the type of task (action recognition, type determination, etc.).

[0090] Next, the learning device 4 trains an initial model using the training images created by the training image creation device 3 (step S53). Then, the learning device 4 evaluates the accuracy of the trained model using evaluation images (step S54). The evaluation may be performed, for example, by comparing the data output by the trained model with pre-prepared ground truth data for each evaluation image, determining correctness, and calculating the accuracy rate.

[0091] Next, the learning device 4 determines whether the evaluation result (accuracy) meets a predetermined standard (step S55). The determination result may be, for example, a pass if the accuracy rate meets a predetermined standard (threshold), and a fail if it does not. Then, the learning device 4 outputs the determination result (step S56).

[0092] When a user inputs task information from the user terminal 6, they may, for example, interact with the learning image creation system 100 in an interactive manner to input the information. Figures 25 and 26 show an example of a screen when predetermined items are input interactively. First, the learning image creation system 100 asks the user, "What kind of data do you want to create?" (71). In response, the user inputs a task (72). Once the task is input, the learning image creation system 100 asks the user about the variations of the image to be created (73). In response, the user inputs their request for what variations of the image they want to generate (74).

[0093] Based on the task and variation information, the learning image creation system 100 creates and displays a representative image (75, 76). The user looks at the displayed representative image and enters correction instructions as needed (77). When correction instructions are entered, the learning image creation system 100 corrects the representative image and displays the corrected representative image (78, 79).

[0094] After reviewing the corrected representative image, the user inputs instructions to create training images and train the model (80). Upon receiving the instructions, the training image creation system 100 creates training images and trains the machine learning model (81). At this time, the training image creation system 100 displays the progress as appropriate in response to the user's requests (82, 83, 84, 85).

[0095] Once the machine learning model has finished training, the training image creation system 100 evaluates the performance of the trained model and displays the results (86, 87). If the user gives instructions for additional training based on the results (88), the training image creation system 100 creates additional training images, performs training and performance evaluation again, and displays the results (89, 90). If the user is satisfied with the performance evaluation results, they input that information (91).

[0096] In addition to the interactive format described above, the means by which users input task and other information from their user terminal may also be configured to allow users to select and input predetermined items using buttons.

[0097] As described above, the training image creation system 100 can create training images that meet the user's intent based on the user's input task, "I want to detect people looking around in the city," and acquire a highly accurate machine learning model by repeatedly performing performance evaluations and generating training images.

[0098] This disclosure includes the following aspects.

[0099] (1) An image creation device that performs the process of creating an image using a processor, The aforementioned processor, We accept tasks requested by users. Using a large-scale language model, extract image information necessary to perform the task from the task, Based on the aforementioned information, the image is acquired from the image generation device. Image creation device. (2) The aforementioned processor, A prompt is generated by combining the aforementioned task and guidance that defines the output format of the aforementioned information. The prompt is input to the large-scale language model and the information is obtained from the large-scale language model. The image creation device described in (1) above. (3) The aforementioned processor, Based on the aforementioned information, parameters to be input to the image generation device are obtained from the large-scale language model. The parameters are input to the image generation device and the image is obtained from the image generation device. The image creation device described in (1) above. (4) The aforementioned processor, A prompt is generated by combining the aforementioned information and guidance defining the output format of the aforementioned parameters. The prompt is input to the large-scale language model and the parameters are obtained from the large-scale language model. The image creation device described in (3) above. (5) The image generation apparatus comprises a person data generation apparatus for generating person data that constitutes an image, a motion data generation apparatus for generating person motion data, a background data generation apparatus for generating background data, and a data integration apparatus for integrating person data, motion data, and background data to generate an image. The aforementioned processor, Based on the aforementioned information, the following are obtained from the large-scale language model: person data parameters, motion data parameters, and background data parameters to be input to the person data generation device, the motion data generation device, and the background data generation device. The person data parameters, motion data parameters, and background data parameters are input to the person data generation device, the motion data generation device, and the background data generation device, respectively, to acquire the person data, motion data, and background data. The person data, motion data, and background data are input to the data integration device to acquire an image. The image creation device described in (1) above. (6) The aforementioned processor, The user's instructions for modifying the aforementioned image are received. Using a large-scale language model, information about the image and information about the image modified based on the modification instructions are obtained. Based on the information regarding the corrected image, the corrected image is obtained from the image generating device. The image creation device described in (1) above. [Industrial applicability]

[0100] It can efficiently create training images tailored to the user's requested tasks. [Explanation of Symbols]

[0101] 1. Image creation device 2 Image correction device 3. Image creation device for learning 4. Learning device 5 Storage device 6. User terminals 7. Large-scale language models 8 Image generation device 100 Training Image Creation System

Claims

1. An image creation device that performs the process of creating an image using a processor, The aforementioned processor, We accept tasks requested by users. Using a large-scale language model, extract image information necessary to perform the task from the task, Based on the aforementioned information, the image is acquired from the image generation device. Image creation device.

2. The aforementioned processor, A prompt is generated by combining the aforementioned task and guidance that defines the output format of the aforementioned information. The prompt is input to the large-scale language model and the information is obtained from the large-scale language model. The image creation apparatus according to claim 1.

3. The aforementioned processor, Based on the aforementioned information, parameters to be input to the image generation device are obtained from the large-scale language model. The parameters are input to the image generation device and the image is obtained from the image generation device. The image creation apparatus according to claim 1.

4. The aforementioned processor, A prompt is generated by combining the aforementioned information and guidance defining the output format of the aforementioned parameters. The prompt is input to the large-scale language model and the parameters are obtained from the large-scale language model. The image creation apparatus according to claim 3.

5. The image generation apparatus comprises a person data generation apparatus for generating person data that constitutes the image, a motion data generation apparatus for generating person motion data, a background data generation apparatus for generating background data, and a data integration apparatus for integrating the person data, motion data, and background data to generate the image. The aforementioned processor, Based on the aforementioned information, the following are obtained from the large-scale language model: person data parameters, motion data parameters, and background data parameters to be input to the person data generation device, the motion data generation device, and the background data generation device. The person data parameters, motion data parameters, and background data parameters are input to the person data generation device, the motion data generation device, and the background data generation device, respectively, to acquire the person data, motion data, and background data. The person data, motion data, and background data are input to the data integration device to acquire the image. The image creation apparatus according to claim 1.

6. The aforementioned processor, The user's instructions for modifying the aforementioned image are received. Using a large-scale language model, information about the image and information about the image modified based on the modification instructions are obtained. Based on the information regarding the corrected image, the corrected image is obtained from the image generating device. The image creation apparatus according to claim 1.

Images