Program, information processing device, and information processing method

Abstract

[Problem] To efficiently acquire data used for learning. [Solution] Provided is a program that causes a computer to function as a control unit for performing: processing for acquiring, from a captured moving image in which is captured a person performing the movement, expression, or utterance determined on a time series by script data, each skeleton data, each expression data, or each voice data of the person in the captured moving image, by referring to the time information of each movement prescribed by the script data; and processing for adding respective attribute information to the acquired each skeleton data, each expression expression data, or each voice data.

Description

Program, information processing device, and information processing method

[0001] The present disclosure relates to a program, an information processing device, and an information processing method.

[0002] In recent years, technology for generating image generation models using machine learning has become widespread. When performing machine learning, it is necessary to input a large amount of training data into the generated model.

[0003] For example, Patent Document 1 below discloses a technique for generating training data using a computer graphics (CG) model. In particular, it discloses that images based on the parameters of a camera or shooting conditions are acquired as training data.

[0004] International Publication No. 2021 / 177324

[0005] However, the above-mentioned Patent Document 1 uses a CG model to acquire an artificial image, and there is still no improvement in collecting training data from real people.

[0006] Therefore, the present disclosure proposes a program, an information processing device, and an information processing method that enable efficient acquisition of data used for learning.

[0007] According to the present disclosure, a program is provided that causes a computer to function as a control unit that performs the following processes: acquires, from a captured video of a person making movements, facial expressions, or vocalizations defined in a time series in script data, each skeletal data, each facial expression data, or each audio data of the person in the captured video by referencing the time information of each movement defined in the script data; and assigns attribute information to each of the acquired skeletal data, each facial expression data, or each audio data.

[0008] Furthermore, according to the present disclosure, there is provided an information processing device including a control unit that performs the following processes: acquires, from a captured video of a person making movements, facial expressions, or vocalizations defined in a time series in script data, each skeletal data, each facial expression data, or each audio data of the person in the captured video by referring to time information of each movement defined in the script data; and assigns attribute information to each of the acquired skeletal data, each facial expression data, or each audio data.

[0009] Furthermore, according to the present disclosure, an information processing method is provided, which includes a processor acquiring, from a captured video of a person making movements, facial expressions, or vocalizations defined in a time series in script data, each skeletal data, each facial expression data, or each voice data of the person in the captured video by referring to time information of each movement defined in the script data, and assigning attribute information to each of the acquired skeletal data, each facial expression data, or each voice data.

[0010] 1 is a diagram illustrating an overview of an AI advertisement generation service according to an embodiment of the present disclosure. FIG. 1 is a diagram illustrating an overall configuration of an information processing system 1 (AI advertisement generation system) according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating an example of the configuration of a server 20 according to the present embodiment. FIG. 3 is a sequence diagram illustrating an example of the flow of an image generation AI construction process in the information processing system 1 according to the present embodiment. FIG. 4 is a diagram illustrating an example of a screen when shooting a video on a model terminal 10 according to the present embodiment. FIG. 5 is a diagram illustrating an example of a video data transmission screen according to the present embodiment. FIG. 6 is a diagram illustrating data analysis according to the present embodiment. FIG. 7 is a diagram illustrating an example of the data configuration of skeletal data 401 with attribute information according to the present embodiment. FIG. 8 is a diagram illustrating an example of the data configuration of facial expression data 402 with attribute information according to the present embodiment. FIG. 9 is a diagram illustrating an example of the data configuration of voice data 403 with attribute information according to the present embodiment. FIG. 10 is a sequence diagram illustrating an example of the flow of an AI advertisement image generation process in the information processing system 1. FIG. 11 is a diagram illustrating a model selection screen 500 for an AI advertisement image generation request according to the present embodiment. FIG. 12 is a diagram illustrating an AI image generation screen 510 according to the present embodiment. FIG. 13 is a diagram illustrating an example of generation of an AI advertisement image based on a prompt according to the present embodiment. FIG. 14 is a flowchart illustrating an example of the flow of a script update process according to the present embodiment. FIG. 15 is a diagram illustrating patterns P1 and P2 of content to be provided in the information processing system 1 according to the present embodiment. It is a figure explaining the pattern P3 and the pattern P4 of the content to be provided in the information processing system 1 according to this embodiment. It is a figure explaining the system configuration according to this embodiment. It is a figure explaining the operation processing of the system according to this embodiment.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The explanation will be given in the following order: 1. Overview 2. Configuration 2-1. System configuration 2-2. Configuration of server 20 3. Building image generation AI 4. Generation of AI advertising images 5. Feedback to script data 6. Patterns of content provided 7. Use of judgment AI 8. Supplementary information

[0013] 1. Overview An information processing system according to an embodiment of the present disclosure will be specifically described with reference to the drawings.

[0014] First, referring to FIG. 1 , an overall service in which an information processing system according to this embodiment can be utilized will be described. The information processing system according to this embodiment relates to an image generation AI that generates images through machine learning. Images generated by the image generation AI are referred to as AI images. Furthermore, as used herein, "image" includes still images and videos. "Video" may also include audio. That is, the image generation AI according to this embodiment is a generation AI that also includes the functionality of audio generation AI, and may generate videos that include audio, such as singing videos, music videos (MVs), and commercial messages (CMs). In the embodiment described below, an AI advertising generation system that generates advertising images, which are an example of AI images, will be described. Advertising images are images (still images, videos (which may include audio)) for advertising products, services, etc. Note that in this embodiment, the image generation AI includes audio generation functionality, and audio generation AI that generates only audio through machine learning may be provided separately from the image generation AI.

[0015] 1 is a diagram illustrating an overview of an AI advertising generation service according to an embodiment of the present disclosure. As shown in FIG. 1, the AI ​​advertising generation system according to this embodiment collects data used to generate AI advertising images using image generation AI from a model (fashion model) who is the subject of an image or a model agency to which the model belongs, receives a request to generate AI advertising images from an advertising business such as an advertising agency that undertakes advertising production requests from advertisers (companies or individuals), and provides the AI ​​advertising images generated by the image generation AI to the advertising business.

[0016] The main flow of the AI ​​advertisement generation service according to this embodiment will be described.

[0017] First, a model or a model agency to which multiple models belong shoots a video of predetermined movements, facial expressions, and sounds according to a sample video provided by the AI ​​advertising generation system, and the video data is provided to the AI ​​advertising generation system (Step 1). The sample video is a video in which the model performs, for example, poses, actions, facial expressions, and vocalizations (lines, phrases, singing, etc.) in rhythm. The sample video may include music or audio instructions to make it easier for the model to pose, etc. In this embodiment, the poses, actions, facial expressions, and vocalizations to be performed by the model are determined by script data. The sample video may be filmed in advance according to the script data, or may be automatically generated based on a CG model. The acquisition, presentation, and filming of such sample videos may be performed by an application installed on a smartphone or the like used by the model or model agency staff.

[0018] The AI ​​advertising generation system according to this embodiment performs predetermined processing on video data provided by models or model agencies to generate a dataset to be used for training the image generation AI, and stores the dataset in dataset storage. In this embodiment, for example, video data of thousands to tens of thousands of models belonging to a model agency is acquired to generate the dataset.

[0019] Next, the AI ​​advertising generation system performs machine learning using the dataset stored in the dataset storage to build an image generation AI (Step 2). In this embodiment, it is assumed that the image generation AI is built using an AI model (text-to-image model) that generates images based on input text. Note that the AI ​​advertising generation system according to this embodiment may build the image generation AI using a method (so-called fine tuning) in which the above dataset is additionally trained on an already trained text-to-image model.

[0020] Meanwhile, the advertising business requests the AI ​​advertising generation system to generate an AI advertising image (Step 3). At this time, the advertising business may transmit various instructions, such as the image of the advertisement, product information, and the model they would like to use. Communication with the AI ​​advertising generation system may be performed via an application installed on a smartphone or the like. Instructions from the advertising business may be input by so-called prompts, which are used to input instructions to the AI.

[0021] Next, the AI ​​ad generation system generates an AI ad image using the image generation AI (Step 4). The generated AI ad image is expected to be a still image or a video (which may include audio). If the prompt includes a proper noun model (model specification), the AI ​​ad generation system may provide compensation (reward) for the model.

[0022] Next, the AI ​​advertising generation system provides the generated AI advertising image to the advertising company, and the advertising company acquires the AI ​​advertising image (Step 5). Note that the advertising company pays the AI ​​advertising generation system a fee in accordance with the use of the AI ​​advertising image generation service according to this embodiment, such as acquisition of the AI ​​advertising image.

[0023] Next, the advertising business uses the AI ​​advertising image as advertising material (Step 6) or begins making arrangements to shoot an actual advertisement based on the AI ​​advertising image (Step 7). Requests for generating advertising images from advertising businesses are expected to include requests for generating advertising images intended to be used as advertising material as is, and requests for generating advertising proposal images intended to confirm and share the final image with the advertiser before actually shooting the advertisement. For example, if an advertiser has a model they would like to use, they can check and consider in advance the final result when photographed with that model using the AI ​​advertising image.

[0024] The advertising business requests the model or the model agency to which the model belongs to appear in an advertising shoot (Step 8), and if the model agrees to appear (Step 9), the shoot can take place (Step 10). The model can be paid a fee or other compensation. The AI ​​advertising image is provided to the model when the model is requested to appear in order to share the finished advertising image, or is provided to the filming staff, such as the director and cameraman, during the shoot to share the finished advertising image.

[0025] The main flow of the AI ​​advertisement generation service according to this embodiment has been described above. Note that the service flow shown in Fig. 1 is an example of use of the AI ​​advertisement generation system according to this embodiment, and this embodiment is not limited to this.

[0026] For example, in the service format shown in Figure 1, it is assumed that the model agency and the advertising agency are separate entities, but this is not limiting; it is also possible to assume an entity that has the functions of both a model agency and an advertising agency, such as an entity that registers models and undertakes the production of advertisements using the registered models. In this case, such an entity can use the AI ​​advertising generation system to provide advertisers (companies and individuals) with AI advertising images that use models registered with the advertiser, in response to requests from the advertiser.

[0027] Furthermore, compensation (incentives) may be provided to the model or the AI ​​ad generation system as appropriate during the course of the service. Compensation may be provided to the model or the AI ​​ad generation system as appropriate, not only when the AI ​​ad image is generated or acquired as described above, but also depending on, for example, the number of plays, the number of displays, and advertising effectiveness (sales of products, etc.) of the AI ​​ad image, and the advertising effectiveness of commercials and the like shot based on the AI ​​ad image.

[0028] Furthermore, the subject in this embodiment is not limited to a fashion model, but may also be an influencer, a famous person, or an entertainer.

[0029] The configuration of the AI ​​advertisement generation system according to this embodiment, which can be utilized for the AI ​​advertisement generation service described above, will be specifically described below.

[0030] <2. Configuration> <<2-1. System Configuration>> Figure 2 is a diagram showing the overall configuration of an information processing system 1 (AI advertising generation system) according to one embodiment of the present disclosure. The information processing system 1 according to this embodiment includes a model terminal 10 used by a model, a model agency terminal 12 used by a model agency, a server 20 (an example of an information processing device) that generates advertising images using image generation AI, and an advertising agent terminal 30 used by an advertising agent. The model terminal 10, the model agency terminal 12, the server 20, and the advertising agent terminal 30 are communicatively connected to a network 40, and each transmits and receives data.

[0031] The model terminal 10, the model office terminal 12, and the advertising agent terminal 30 can be realized by a smartphone, a tablet terminal, a PC (personal computer), or the like.

[0032] Video data of the model, filmed in accordance with the sample video, is transmitted from the model terminal 10 or the model agency terminal 12 to the server 20. Models who work independently may transmit video data to the server 20 from their own information processing terminal, the model terminal 10. On the other hand, models registered with a model agency may film using their own model terminal 10 and transmit video data to the server 20, or video data filmed in a studio arranged by the agency to which they belong may be transmitted from the model agency terminal 12 to the server 20. There are no particular limitations on the method by which the server 20 obtains the video data of each model.

[0033] The advertising agent terminal 30 can communicate with the server 20 to request the generation of an AI advertising image and acquire the AI ​​advertising image. These operations may be performed by an application installed on the advertising agent terminal 30, or may be performed via an AI advertising image generation screen provided by the server 20 and displayed on a web browser.

[0034] The details of the server 20 will be explained next.

[0035] 3 is a block diagram showing an example of the configuration of the server 20 according to this embodiment. As shown in FIG. 3, the server 20 includes a communication unit 210, a control unit 220, and a storage unit 230.

[0036] (Communication Unit 210) The communication unit 210 has a transmission unit that transmits data to an external device and a reception unit that receives data from an external device. The communication unit 210 according to this embodiment may be communicatively connected to an external device or the Internet using, for example, a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), or the like.

[0037] (Control Unit 220) The control unit 220 functions as an arithmetic processing unit and a control device, and controls the overall operation of the server 20 in accordance with various programs. The control unit 220 is realized by an electronic circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a microprocessor. The control unit 220 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0038] The control unit 220 in this embodiment can also function as a data separation unit 221, a skeletal data analysis unit 222, a facial expression data analysis unit 223, a voice data analysis unit 224, a dataset storage control unit 225, an image generation AI construction unit 226, and an AI advertising image generation unit 227.

[0039] The data separation unit 221 has the function of separating video data of the model (subject) into motion images, facial images, and audio data. The model's video data is transmitted, for example, from the model terminal 10 or the model agency terminal 12 and stored in the video data storage 232. The motion images are images containing the model's entire body movements (poses, actions, etc.) and are used to analyze the skeleton data in the skeleton data analysis unit 222 (described later). It is assumed that the model sequentially performs predetermined poses and actions within the video. The facial images are images containing the model's face and are used to analyze facial data in the facial expression data analysis unit 223 (described later). It is assumed that the model sequentially performs predetermined facial expressions within the video. Note that it is also assumed that the model may perform a specified pose with a specified facial expression, so motion images and facial images may be separated from the same frame. The audio is audio data containing the model's speaking voice and is used to analyze the audio data in the audio data analysis unit 224 (described later). It is assumed that the model reads or sings predetermined lines within the video.

[0040] The skeletal data analysis unit 222 analyzes skeletal data for motion images in which poses or actions specified in the script data are being performed, among the time-series motion images output from the data separation unit 221, assigns attribute information, and outputs each piece of skeletal data (with attribute information assigned) to the dataset storage control unit 225. Assigning attribute information is a so-called annotation process. The assigned attribute information serves as a correct label when a dataset including the skeletal data is subjected to machine learning in the construction of an image generation AI, which will be described later.

[0041] The skeletal data analysis unit 222 first refers to the script data associated with the captured video data and analyzes the skeletal data for each pose and action. The script data can be acquired from the script data storage 231. For example, in the case of video data captured using a sample video based on script A as a sample, script A can be considered the associated script data. The script data indicates the instructions for each pose, action, facial expression, and speech in chronological order. By referencing the script data, information on what pose or facial expression was performed at what time in the video can be acquired. The skeletal data analysis unit 222 acquires time information (e.g., the elapsed time from the start) at which each pose or action should be performed from the script data, analyzes the motion image (frame) at the time (timestamp) indicated by the time information, and acquires skeletal data. The skeletal data analysis method is not particularly limited. For example, the skeletal data analysis unit 222 can acquire time-series position coordinate information of each part of the human body as skeletal data.

[0042] The skeletal data analysis unit 222 then assigns attribute information to the acquired skeletal data. The assigned attribute information includes information acquired from the script data and information about the subject model. The information acquired from the script data is information indicating the content of the movement (type of pose or action). For example, "raising one hand," "raising one leg," "raising both hands," "walking," "jumping," etc. Examples of information about the model include name (model ID), gender, height, weight, country of origin, race, age, and face type. The information about the model may be acquired from the model terminal 10, for example, when acquiring video data. Note that the attribute information assigned to the skeletal data is not limited to the above-described examples, and the skeletal data analysis unit 222 may further assign identification information (script ID) of script data associated with the video data from which the skeletal data was acquired.

[0043] The facial expression data analysis unit 223 analyzes facial expression data for facial images showing expressions specified in the script data among the time-series facial images output from the data separation unit 221, assigns attribute information, and outputs each piece of facial expression data (with attribute information assigned) to the dataset storage control unit 225. The facial expression data analysis unit 223 first references the script data associated with the captured video data and analyzes the facial expression data for each expression. As described above, the script data can be used to acquire information about what expression was performed at what time in the video. The facial expression data analysis unit 223 acquires time information (e.g., the elapsed time from the start) at which each expression should be performed from the script data, analyzes the facial image at the time (timestamp) indicated by the time information, and acquires the facial expression data. The facial expression data analysis method is not particularly limited. For example, the facial expression analysis unit 223 may detect feature points of the face (expression) and acquire the detected feature points as facial expression data. The facial expression data analysis unit 223 then assigns attribute information to the acquired facial expression data. The assigned attribute information includes information acquired from the script data and information about the subject model. The information about the subject model is as described above. The information acquired from the script data is information about facial expressions, and specifically includes the content of the facial expressions (e.g., "laughing," "angry," "crying," "wink," etc.). Note that the attribute information assigned to the facial expression data is not limited to the examples described above, and the facial expression data analysis unit 223 may further assign identification information of the script data associated with the video data from which the facial expression data was acquired.

[0044] The audio data analysis unit 224 acquires audio data containing utterances (lines, phrases, etc.) specified in the script data from the time-series audio data output from the data separation unit 221, assigns attribute information to the audio data, and outputs each piece of audio data (with attribute information assigned) to the dataset storage control unit 225. The audio data analysis unit 224 first references the script data associated with the captured video data and acquires information on what lines, phrases, etc. were spoken at what time in the video. The audio data analysis unit 224 acquires time information (e.g., the elapsed time from the start) at which each line, phrase, etc. should be spoken from the script data, and acquires audio data at the time (timestamp) indicated by the time information (sound source separation). The audio data analysis unit 224 then assigns attribute information to the acquired audio data. The assigned attribute information includes information acquired from the script data and information on the subject model. The information on the subject model is as described above. The information acquired from the script data is information related to the content of the speech, such as lines, phrases, singing, low voices, high voices, laughter, etc. Note that the attribute information assigned to the audio data is not limited to the examples described above, and the audio data analysis unit 224 may further assign identification information of the script data associated with the video data from which the audio data was acquired.

[0045] In this embodiment, it has been described that the data separation unit 221 separates the video data of the model (subject) into motion image, facial image, and audio data, but the present disclosure is not limited to this, and the skeletal data analysis unit 222, facial expression data analysis unit 223, and audio data analysis unit 224 may each perform processing to directly separate the target data from the video data.

[0046] The dataset storage control unit 225 generates a dataset including the skeletal data (with attribute information) output from the skeletal data analysis unit 222, the facial expression data (with attribute information) output from the facial expression data analysis unit 223, and the audio data (with attribute information) output from the audio data analysis unit 224, and controls the storage of the dataset in the dataset storage 233. Here, a dataset obtained from video data of a specific subject, such as a fashion model, is referred to as a proper noun dataset. The dataset storage control unit 225 controls the storage of the proper noun dataset in the dataset storage 233. The attribute information of each piece of data included in the proper noun dataset is assigned the proper noun of the subject, i.e., the name of the subject (model) or identification information (model ID) associated with the name. The proper noun dataset may be organized by gender or country (race). Furthermore, the dataset generated by this embodiment is not limited to a proper noun dataset, and datasets obtained from video data mainly featuring ordinary people as subjects are also contemplated. Such datasets are referred to as general-purpose datasets.

[0047] The image generation AI construction unit 226 has the function of constructing (generating) an image generation AI by machine learning the datasets stored in the dataset storage 233. As described above, in this embodiment, it is assumed that an image generation AI equipped with an AI model (text-to-image model) that generates images based on input text is constructed. The image generation AI construction unit 226 may also construct the image generation AI by additionally training the dataset on an already trained text-to-image model (so-called fine tuning). In this embodiment, by training a dataset (proper noun dataset) obtained from video data of a specific subject, such as a fashion model, it is possible to construct an image generation AI capable of generating AI images corresponding to the subject. The generated image generation AI is stored in the image generation AI storage 234.

[0048] Note that a dataset used to train an image generation AI may have information added to it indicating which image generation AI it was used to train (association with the image generation AI). This information can be referenced when providing feedback to the subject (model) from which the dataset was obtained, in response to the use of the image generation AI by an advertising company (generation of an AI image at the request of the advertising company) or the provision of an API (Application Programming Interface), etc.

[0049] The AI ​​advertising image generation unit 227 has a function of generating an AI advertising image based on input text (so-called prompts) using an image generation AI. In this embodiment, an AI advertising image is given as an example of an AI image generated using the image generation AI. The AI ​​advertising image may be a still image or a video (which may include audio). Note that the AI ​​images that can be generated by this embodiment are not limited to AI advertising images, and may also be, for example, dance videos, music videos (including singing by a synthetic voice generated by AI), commercial films (CFs), commercials, etc. To generate a singing video, an automatic composition AI may also be used, or a specified song, melody, etc. may be used.

[0050] The input text may include a designation of an advertising target (product, service, etc.). The input text may also include a designation of a real person (the model that is the subject of the image). This allows the AI ​​advertising image generation unit 227 to generate an AI advertising image corresponding to the designated real person. The AI ​​advertising image generation unit 227 stores the generated AI advertising image in the generated image storage 235. In this embodiment, it is assumed that a designation of a model is input by an advertising company (or advertiser), but this is not limited thereto. The AI ​​advertising image generation unit 227 may select an optimal model according to conditions included in the input text (prompt) and generate an AI advertising image of the model.

[0051] Note that information indicating the generated AI image may be added to the dataset used to train the image generation AI used to generate the AI ​​image (association with the AI ​​image). Such information may be referenced when providing feedback to the subject (model) from which the dataset was obtained, in the form of incentives based on the sales or advertising effectiveness of the AI ​​image.

[0052] (Storage Unit 230) The storage unit 230 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 220, and a RAM that temporarily stores parameters that change as needed.

[0053] The memory unit 230 stores a script data storage 231, a video data storage 232, a dataset storage 233, an image generation AI storage 234, and a generated image storage 235.

[0054] Although the configuration of the server 20 has been specifically described above, the configuration of the server 20 according to the present disclosure is not limited to the example shown in Fig. 3. For example, the server 20 does not necessarily have to have all of the components shown in Fig. 3. Furthermore, the server 20 may be realized by multiple devices.

[0055] 3. Construction of Image Generation AI Next, construction of image generation AI according to this embodiment will be described.

[0056] 4 is a sequence diagram showing an example of the flow of the image generation AI construction process in the information processing system 1 according to this embodiment. Here, the description will be given assuming that video data is captured by the model terminal 10.

[0057] As shown in FIG. 4, first, an application (hereinafter also referred to as an app) is started in the model terminal 10 in response to an operation by a user (for example, the model himself / herself) (step S103).

[0058] Next, the model terminal 10 accesses the server 20 using the application function, requests transmission of a sample video, and acquires the sample video transmitted from the server 20 (step S106).

[0059] Next, the model terminal 10 plays the sample video using the app's functions and also shoots a video of the model (step S109). Specifically, the sample video is played on the display of the model terminal 10, and an imaging unit (in-camera) provided on the display side of the model terminal 10 captures a video of the model performing the movements, facial expressions, and speech (lines, singing, etc.) determined in accordance with the sample video. FIG. 5 is a diagram showing an example of a screen displayed when a video is being shot on the model terminal 10 according to this embodiment. As shown in FIG. 5, a sample video 300 is displayed on the display unit 120 of the model terminal 10. The video is shot using the imaging unit 130, which is the in-camera, so that the model M can imitate the movements, facial expressions, etc. shown in the sample video 300 while watching the sample video 300. For the lines, text may be displayed on the display unit 120, and the model M may read the text aloud. As for singing, lyrics (for example, one phrase) may be displayed on the display unit 120, and a melody (for example, one phrase) and sample singing may be played in the sample video 300, so that the model M can easily sing. The display unit 120 displays an image (through image) captured by the imaging unit 130, so that the model M can take a photo while checking her own movements, facial expressions, etc.

[0060] Next, the model terminal 10 transmits the captured video data to the server 20 (step S112). The transmission of video data may be performed in response to a user operation. FIG. 6 is a diagram showing an example of a video data transmission screen according to this embodiment. As shown in FIG. 6, for example, thumbnails of captured videos are displayed on the display unit 120 of the model terminal 10. The user can transmit or delete any video data. When generating new video data, the user may, for example, tap the add field 310 to transition to the screen shown in FIG. 5, where the user can start capturing.

[0061] Next, the server 20 separates the video data received from the model terminal 10 into motion image data, facial image data, and audio data (step S115).

[0062] Next, the server 20 performs data analysis on each of the separated data by referring to the script data (step S118). Specifically, the server 20 acquires data corresponding to the movements, facial expressions, and voices defined in the script data from each of the data, and performs processing to assign attribute information to each of the data (skeleton data, facial expression data, and voice data). This processing will be described below with reference to FIG. 7.

[0063] 7 is a diagram illustrating data analysis according to this embodiment. As shown in FIG. 7, the server 20 separates motion images D1, facial images D2, and audio D3 from video data D, and acquires from each data skeletal data of poses (movements), facial expression data, and audio data of utterances defined in time series in the script data, and further assigns attribute information. The timing (time) of poses, actions, facial expressions, utterances, etc. in the video data D can be acquired from the script data corresponding to the video data D.

[0064] For example, in the time-series motion image D1, it may be acquired from the script data that a "standing" motion is performed at time t1 and a "sitting" motion is performed at time t2. In this case, the server 20 analyzes the frame corresponding to time t1 among the frames of the motion image D1 to acquire skeletal data for the "standing" motion and assigns attribute information "standing" indicating the content of the motion to the skeletal data. The server 20 also analyzes the frame corresponding to time t2 among the frames of the motion image D1 to acquire skeletal data for the "sitting" motion and assigns attribute information "sitting" indicating the content of the motion to the skeletal data. Note that the times tN (t1, t2, ...) may be information including a certain time width, and the acquired skeletal data is time-series skeletal data (e.g., time-series position coordinate information).

[0065] Furthermore, it may be acquired from the script data that, in the time-series facial image D2, a "laughing" movement is made at time t11 and an "angry" expression is made at time t12. In this case, the server 20 analyzes the frame corresponding to time t11 among the frames of the facial image D2 to acquire facial expression data of the "laughing" face (detecting facial feature points), and assigns attribute information "laughing" indicating the content of the expression to the facial expression data. The server 20 also analyzes the frame corresponding to time t12 among the frames of the facial image D2 to acquire facial expression data of the "angry" face, and assigns attribute information "angry" indicating the content of the expression to the facial expression data. Note that the times tN (t11, t12, ...) may be information including a certain time span, as in the above case, and the acquired facial expression data is time-series facial expression data (e.g., time-series feature point information).

[0066] Furthermore, it may be acquired from the script data that, in the time-series audio D3, "I" is uttered at time t21 and "you" is uttered at time t22. In this case, the server 20 acquires (cuts out) audio data corresponding to time t21 from the audio D3 and assigns attribute information "I" indicating the content of the utterance to the audio data. The server 20 also acquires (cuts out) audio data corresponding to time t22 from the audio D3 and assigns attribute information "you" indicating the content of the utterance to the audio data. Note that the time tN (t21, t22, ...) may be information including a fixed time width, as in the above, and the acquired audio data is audio data having a fixed time width.

[0067] In this way, the server 20 acquires the skeletal data with attribute information 401, the facial expression data with attribute information 402, and the voice data with attribute information 403, and generates a data set 410 including these. The data set 410 is associated with the subject (fashion model) of the video data D.

[0068] Next, the server 20 performs control to store the data set in the storage unit 230 (step S121). Here, the configuration of data included in each data set according to this embodiment will be described with reference to Figs.

[0069] 8 is a diagram showing an example of the data configuration of attribute-information-attached skeletal data 401 according to this embodiment. In the attribute-information-attached skeletal data 401, a data ID, a model ID (identification information of the subject), information about the subject (e.g., country of origin), model attribute 2 (e.g., gender), and model attribute n), a script ID indicating corresponding script data, movement details, and skeletal data (e.g., time-series position coordinate information) are associated with each other. In the illustrated example, specifically, a data ID "P0001," a model ID "M0101," a model attribute 1 "JP," a model attribute 2 "F," a model attribute n "XXX," a script ID "A001" (e.g., identification information of script A), movement details "raise one hand, raise one leg" (a movement of raising one hand and one leg), and skeleton data "skeleton 0001" are associated with each other. 8, "model ID" and "model attributes 1 to n" are attribute information assigned from the subject information, and "movement details" are attribute information assigned from the corresponding script data (script A in this example). In this manner, in this embodiment, skeletal data on multiple movements of one subject can be collected from one video data D, as shown in FIG.

[0070] FIG. 9 is a diagram showing an example of the data structure of attribute information-attached facial expression data 402 according to this embodiment. In the attribute information-attached facial expression data 402, a data ID, a model ID, a model attribute 1, a model attribute 2, a model attribute n, a script ID, facial expression content, and facial expression data (e.g., feature point data) are associated with each other. In the illustrated example, specifically, a data ID "Q0001," a model ID "M0101," a model attribute 1 "JP," a model attribute 2 "F," a model attribute n "XXX," a script ID "A001," facial expression content "smile," and facial expression data "face 0001" are associated with each other. In the example of FIG. 9 , the "model ID" and "model attributes 1 to n" are attribute information assigned from the subject information, and the "facial expression content" is attribute information assigned from the corresponding script data (here, script A). In this way, in this embodiment, a large number of facial expression data for one subject can be collected from one video data D, as shown in FIG. 9 .

[0071] FIG. 10 is a diagram showing an example of the data configuration of attribute information-attached audio data 403 according to this embodiment. In the attribute information-attached audio data 403, a data ID, a model ID, a model attribute 1, a model attribute 2, a model attribute n, a script ID, vocal content, and audio data (e.g., lines, phrases, and singing data) are associated with each other. In the illustrated example, specifically, a data ID "R0001," a model ID "M0101," a model attribute 1 "JP," a model attribute 2 "F," a model attribute n "XXX," a script ID "A001," vocal content "I," and audio data "audio 0001" are associated with each other. In the example of FIG. 10 , the "model ID" and "model attributes 1 to n" are attribute information assigned from the subject information, and the "vocal content" is attribute information assigned from the corresponding script data (here, script A). In this manner, in this embodiment, a large number of pieces of audio data of one subject can be collected from one piece of video data D, as shown in FIG.

[0072] Then, the server 20 uses the above-mentioned dataset to construct an image generation AI (step S124). In this embodiment, by having the AI ​​learn a dataset (proper noun dataset) acquired from video data of a specific subject, such as a fashion model, it is possible to construct an image generation AI capable of generating an AI image corresponding to the subject.

[0073] 4. Generation of AI Advertising Images Next, generation of AI advertising images according to this embodiment will be described.

[0074] (Operation Processing) FIG. 11 is a sequence diagram showing an example of the flow of an AI advertising image generation process in the information processing system 1.

[0075] As shown in FIG. 11 , first, the advertiser terminal 30 requests the server 20 to generate an AI advertising image (step S203). Specifically, the advertiser terminal 30 transmits text (prompts) specifying the person to be used in the advertisement, the atmosphere of the advertisement, the scene (location, environment, background), and information about the advertised product. The people who can be specified may be presented in advance by the server 20. The transmitted prompt may be input by the advertiser in response to a request from the advertiser. Note that, while a case where a prompt is input from the advertiser terminal 30 is described here as an example, the present disclosure is not limited thereto. An advertiser such as a company may directly input a prompt from an advertiser terminal (not shown) and transmit it to the server 20.

[0076] Next, the server 20 generates an AI advertising image using the image generation AI based on the prompt (step S206).

[0077] Next, the server 20 transmits the generated AI advertising image to the advertising agent terminal 30 (step S209).

[0078] Then, the advertising agent terminal 30 presents (displays) the AI ​​advertising image received from the server 20 (step S212).

[0079] (Display Screen Example) The prompt input from the advertiser terminal 30 can be performed, for example, from a display screen provided on a web browser by the server 20, or a display screen provided by an application installed on the advertiser terminal 30 by appropriately acquiring information from the server 20. Here, an example of a display screen for generating an AI advertising image according to this embodiment will be described with reference to Figs. 12 and 13 .

[0080] FIG. 12 is a diagram showing a model selection screen 500 for requesting AI advertising image generation according to this embodiment. The model selection screen 500 may be displayed on the display unit of the advertising agent terminal 30. As shown in FIG. 12, the model selection screen 500 displays model information (e.g., a face photo, track record, number of SNS followers, hometown, age, height, etc.). The model selection screen 500 may also be displayed on the webpage of a model agency. When an advertising agent is considering a model to use in an advertisement, it is expected that the advertising agent will search the model agency's webpage. Therefore, the display information for the model selection screen 500 may be provided by a model agency server (not shown).

[0081] An AI generation button 502 for generating an AI image corresponding to the selected model is displayed on the model selection screen 500. When the AI ​​generation button 502 is selected, an AI image generation screen is displayed.

[0082] 13 is a diagram showing an AI image generation screen 510 according to this embodiment. As shown in Fig. 13, the AI ​​image generation screen 510 displays model information 511, a prompt input field 512, an AI image display field 514, and a model search field 515.

[0083] 13 may display model information 511 including a profile of a model selected by a client (specifically, an advertising company or advertiser). The client inputs a prompt in prompt input field 512 and selects AI image generation button 513. This causes the prompt to be sent to server 20, which generates an AI advertising image based on the prompt, and the generated AI advertising image may be displayed in AI image display field 514.

[0084] Examples of prompts that can be input include designation of a person (model), designation of the atmosphere of the advertisement, designation of a scene (location, environment, background), and designation of the product to be advertised (product ID). The name of the designated person (model) displayed in the prompt input field 512 is the name of the selected model and can be automatically input. Note that, although the name of the selected model is displayed in the example shown in FIG. 13 , the name of the model may not be displayed in the prompt input field 512 but may be included in the prompt sent to the server 20.

[0085] Multiple AI advertising images may be generated and displayed in response to a single prompt input. Furthermore, the advertising business can generate AI advertising images multiple times by changing the prompt, etc., until a desired AI advertising image is displayed. Furthermore, the server 20 may receive payment from the advertising business each time it generates and displays an AI advertising image in response to a prompt input.

[0086] The product ID is identification information for product information registered in a pre-prepared product database (not shown). The product database may be stored in the server 20, for example. The client may register product information (such as a product image) in the product database from the advertising agent terminal 30 and input the assigned product ID into the prompt input field 512. This may allow an image to be output in which an AI-generated model is holding the target product in the AI ​​advertising image generation described below. It is also possible to specify how to hold the product using a prompt. Furthermore, how to hold the product may be automatically determined by the server 20 depending on the type of product recognized from the input product ID, etc.

[0087] The model agency server can incorporate the AI ​​advertising image generation function according to this embodiment, provided by server 20, into its own website by using the function via an API. In this case, the model agency can use the AI ​​advertising image generation function according to this embodiment by entering into a usage contract for the function with the system provider that provides the function via server 20 and paying a usage fee to the system provider. Alternatively, server 20 can collect information on affiliated models from the model agency server, create a database, and provide a search function for real models and an AI advertising image generation service for real models on the Web.

[0088] The AI ​​image display field 514 displays an AI advertising image 600 generated based on the prompt. Below the AI ​​advertising image 600, an evaluation button 601 for inputting an evaluation of the AI ​​advertising image 600 and an acquisition button 602 for acquiring (purchasing, downloading) the AI ​​advertising image are displayed. The requester can use the evaluation button 601 to input an intuitive evaluation (good / bad) of the AI ​​advertising image 600. Furthermore, if the requester wants to acquire the generated AI advertising image, the requester can select the acquisition button 602, make payment as necessary, and acquire (download) the AI ​​advertising image. The server 20 may receive payment in exchange for the acquisition (download) of the AI ​​advertising image by the advertising agent terminal 30 (i.e., sale of the AI ​​advertising image).

[0089] The evaluation information from the evaluation button 601 is fed back to the server 20. Furthermore, when an AI advertising image is acquired (i.e., purchased), the server 20 may provide an incentive to the selected model. The incentive may be provided to the model not only when the AI ​​advertising image is acquired (purchased), but also when the AI ​​advertising image is generated.

[0090] 14 is a diagram showing an example of generating an AI advertising image based on a prompt according to this embodiment. As shown in FIG. 14, AI advertising images 600, 610, and 620 are generated based on each input prompt.

[0091] 5. Feedback to script data The AI ​​advertising images described above are generated using an image generation AI constructed by learning a dataset generated from video data of a model (subject). Such a dataset is generated from video data capturing a model performing movements, facial expressions, and vocalizations determined by the script data, but there may be cases where an AI advertising image that cannot be generated from the dataset is to be generated.

[0092] In cases where the poses or facial expressions necessary for learning to generate the desired AI advertising image are insufficient, the control unit 220 of the server 20 can generate new script data including the necessary poses and facial expressions, obtain a data set based on the new script data, and have the image generation AI learn from it, thereby generating the desired AI advertising image.

[0093] A specific description will be given below with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the flow of script update processing according to this embodiment.

[0094] As shown in FIG. 15, first, the server 20 constructs an image generation AI_A using a data set based on the script A (step S303).

[0095] Next, the server 20 uses the image generation AI_A to generate an AI advertising image A and provides it to the advertising agent (step S306).

[0096] Next, the server 20 obtains feedback on the AI ​​advertising image A (step S309). Such feedback may be a prompt newly input by the advertising business after confirming the provided AI advertising image A. The feedback may also be advertising effectiveness data (e.g., click rate, conversion rate, etc.) when the advertising business uses the provided AI advertising image A in an advertisement. "Used in an advertisement" is assumed to mean a case where the provided AI advertising image A is used as an advertisement (on a website, SNS, etc.) or a case where an advertisement is shot based on the provided AI advertising image A.

[0097] Next, the server 20 generates a script B in response to the feedback (step S312). If the image generation AI_A, which has learned a data set based on the script A, cannot generate an AI advertising image in response to the feedback, the server 20 generates a script B to enable the AI ​​to do so. For example, the server 20 generates a new script B that includes missing poses and facial expressions.

[0098] Next, the server 20 acquires a dataset based on script B (step S315). Specifically, for example, the server 20 presents a sample video corresponding to script B to the target model (the subject of the video data from which the dataset used to generate the AI ​​advertising image A was obtained) and requests the acquisition of new video data. The sample video may be automatically generated by a CG model based on script B. Also, as an example here, new video data is acquired from the subject of the video data from which the dataset used to generate the AI ​​advertising image A was obtained, but this is not limiting, and video data may be acquired from other relevant subjects in response to feedback.

[0099] Then, the server 20 constructs an image generation AI_B using a data set based on the script B (step S318), generates an AI advertising image B using the image generation AI_B, and provides it to the advertising company (step S321).

[0100] In this way, in this embodiment, it is possible to generate a new script B that reflects feedback from the AI ​​advertising image A, and to generate an AI advertising image B. Furthermore, the server 20 may reflect feedback in the script data in response to a request from the advertising business, and may receive compensation from the advertising business when providing a new AI advertising image.

[0101] 6. Patterns of Content Provided In the above-described embodiment, it has been described that the server 20 provides and sells the AI ​​advertising image, which is the result, to the advertising company or the advertiser, but the content provided in the information processing system 1 according to this embodiment is not limited to the AI ​​advertising image. The patterns of content provided in the information processing system 1 according to this embodiment will be summarized and explained below.

[0102] FIG. 16 is a diagram illustrating patterns P1 and P2 of the content provided in the information processing system 1 according to this embodiment. As shown in the upper part of FIG. 16 , pattern P1, as described in the above embodiment, includes providing and selling an AI advertising image in response to a request for generation of an AI advertising image (input of a prompt) from the advertiser terminal 30 (or advertiser terminal). At this time, a fee is paid in accordance with the provision or purchase. The server 20 may provide appropriate incentives to a model (the subject of the video data from which the dataset used to train the image generation AI that generated the AI ​​advertising image was obtained) or a model agency specified in the generation of the AI ​​advertising image in accordance with the provision or purchase.

[0103] Furthermore, as shown in the lower part of FIG. 16 , pattern P2 is also assumed to involve a request from the advertiser terminal 30 (or advertiser terminal) to provide an application with an AI advertising image generation function that allows a model (subject) to be specified. The server 20 provides the AI ​​advertising image generation application to the advertiser terminal 30 and receives payment. Use of the AI ​​advertising image generation application may be subject to a fee, and the application may be provided with a usage limit set. Furthermore, the server 20 may keep track of the usage status of the AI ​​advertising image generation application and, as appropriate, provide incentives to models or model agencies specified in the generation of the AI ​​advertising image.

[0104] FIG. 17 is a diagram illustrating patterns P3 and P4 of the content provided in the information processing system 1 according to this embodiment. As shown in the upper part of FIG. 17, pattern P3 is also assumed to include a request from the advertising agent terminal 30 to provide an API for image generation AI. The server 20 provides the API for image generation AI to the advertising agent terminal 30 and receives payment. Use of the API for image generation AI may be subject to a fee, and may be provided with a usage limit. The server 20 may also track the usage status of the API and, as appropriate, provide incentives to models or model agencies designated in the generation of AI advertising images.

[0105] As shown in the lower part of FIG. 17 , pattern P4 is also assumed to involve a request from the advertiser terminal 30 to provide a dataset (proper noun dataset, general-purpose dataset) or image generation AI. The server 20 provides the dataset or image generation AI to the advertiser terminal 30 and receives payment. Use of the dataset or image generation AI may be subject to a fee, or may be provided with a usage limit. The server 20 may also keep track of the usage status of the dataset or image generation AI and, as appropriate, provide incentives to models or model agencies designated in the generation of AI advertising images.

[0106] (Other) The information processing system 1 according to this embodiment may convert the generated AI images (still images, videos, dance videos, music videos, singing scenes, etc.) into NFTs (Non-Fungible Tokens). The information processing system 1 may also provide the NFT-converted AI images to an EC (Electronic Commerce) sales market such as a marketplace, thereby earning appropriate sales. Furthermore, appropriate incentives may be provided to models and model agencies based on sales.

[0107] 7. Utilization of Decision AI In the above-described embodiment, the AI ​​advertising image generated by the server 20 is provided to an advertising company, etc., and the AI ​​advertising image is used for advertising photography or advertising at the discretion of the advertising company, but the information processing system 1 according to this embodiment is not limited to this. For example, by using decision AI based on the collected results of advertising effectiveness data, it is possible to provide an AI advertising image that is more suitable for the user without human judgment on the part of the advertising company.

[0108] 18 is a diagram illustrating the system configuration according to this embodiment. As shown in FIG. 18, the system according to this embodiment includes a server 20 and a determination device 70.

[0109] The server 20 generates one or more AI advertising images and outputs them to the determination device 70. Note that the server 20 may generate a large number of AI advertising images and output them to the determination device 70.

[0110] The determination device 70 has a function of collecting advertising effectiveness data (of the customer segment) for the distributed advertisements, creating a dataset, and conducting machine learning on the dataset to construct (generate) a determination AI that determines the preferences of the customer segment. The determination device 70 uses the determination AI to determine whether the AI ​​advertising image output from the server 20 is preferred by the customer segment.

[0111] The determination device 70 may determine that the AI ​​advertising image to be determined is popular with the customer segment when a numerical value (determination score) indicating the preference level of the customer segment output by the determination AI exceeds a threshold. When the determination device 70 determines that the AI ​​advertising image is popular with the customer segment, it determines that distribution is OK and distributes the AI ​​advertising image. When multiple AI advertising images are output from the server 20, the determination device 70 may distribute the AI ​​advertising image with the highest determination score among the AI ​​advertising images whose determination scores exceed the threshold. Note that the distribution of the AI ​​advertising images may be controlled by another server (not shown). The determination device 70 may also collect advertising effectiveness data (of the customer segment) for the AI ​​advertising images distributed in this manner, create a dataset, and perform machine learning on the dataset to continuously update the determination AI. The determination device 70 may again determine the AI ​​advertising image using the updated determination AI.

[0112] On the other hand, if the judgment score does not exceed the threshold, the judgment device 70 judges that the AI ​​advertising image to be judged is not popular with the customer segment. When the judgment device 70 judges that the AI ​​advertising image is not popular with the customer segment, it judges that distribution is NG and transmits the judgment result to the server 20 to instruct regeneration. The judgment result may include, for example, the judgment score.

[0113] The server 20 regenerates an AI advertising image in accordance with the determination result transmitted from the determination device 70 and outputs the image to the determination device 70. At this time, the server 20 may randomly change the prompt to generate a new AI advertising image, or may change the prompt based on the determination result to generate a new AI advertising image. For example, when the determination device 70 obtains determination scores for a large number of AI advertising images as determination results, the server 20 may change the prompt used for an AI advertising image with a low determination score to a prompt used for an AI advertising image with a high determination score, or the like. The server 20 may learn and estimate prompts for generating an AI advertising image that will result in a higher determination score.

[0114] The server 20 outputs the one or more regenerated AI advertising images to the determination device 70. Then, the determination device 70 can perform a determination again.

[0115] In this way, the judgment device 70 judges the customer preferences for AI advertising images, and the server 20 regenerates AI advertising images according to the judgment results, making it possible to automatically provide more suitable AI advertising images.

[0116] FIG. 19 is a diagram illustrating the operation process of the system according to this embodiment.

[0117] As shown in FIG. 19, first, the determination device 70 controls advertisement distribution (step S403) and collects advertisement effectiveness data (step S405).

[0118] Next, the determination device 70 analyzes the customer segment's preferences for advertisements based on the collected advertising effectiveness data (step S407). Specifically, the determination device 70 extracts the features of advertisements preferred by users analyzed from the collected advertising effectiveness data (for example, the atmosphere, features of people, composition, etc. preferred by users in advertisements) to create a data set for learning.

[0119] Next, the determination device 70 constructs a determination AI by performing machine learning on the dataset (step S409). Note that the determination device 70 may construct the determination AI by additionally learning the dataset to an already trained determination model.

[0120] Next, the determination device 70 performs a determination using the determination AI on one or more AI advertising images transmitted from the server 20 (step S411).

[0121] Then, when the determination device 70 determines that the distribution is OK because the determination score exceeds the threshold value (Yes in step S413), the process returns to step S403 and performs advertisement distribution control (step S403).

[0122] On the other hand, if the determination score does not exceed the threshold and the distribution is determined to be NG (step S413 / No), the determination device 70 transmits the determination result to the server 20 and requests regeneration (step S415).

[0123] The server 20 prepares a prompt for creating an AI advertising image (step S503) and creates an AI advertising image using the image generation AI (step S505). If the server 20 has received a determination result from the determination device 70, the server 20 prepares an appropriate prompt based on the determination result. Specifically, the server 20 analyzes each determination score for one or more AI advertising images included in the determination result, and can learn and estimate appropriate prompts for creating an AI advertising image that suits the preferences of the customer segment.

[0124] Then, the server 20 transmits the newly generated one or more AI advertising images to the determination device 70 (step S507).

[0125] The above has described the operation process of the AI ​​advertising image generation system using the judgment AI according to this embodiment. Note that the steps shown in FIG. 19 are merely examples, and this embodiment is not limited to these. For example, the advertisement distribution control shown in S403, the advertising effectiveness data collection shown in S405, and the analysis shown in S407 may be performed by another server, and the learning data set may be output to the judgment device 70.

[0126] <8. Supplementary Information> Although preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0127] For example, one or more computer programs can be created for hardware such as a CPU, ROM, and RAM built into the server 20 to perform the functions of the server 20. Also provided is a computer-readable storage medium storing the one or more computer programs.

[0128] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0129] The present technology can also be configured as follows. (1) A program that causes a computer to function as a control unit that performs the following processes: acquires, from a captured video of a person making movements, facial expressions, or vocalizations defined in a time series in script data, skeletal data, facial expression data, or audio data of the person in the captured video by referring to time information of each movement defined in the script data; and assigns attribute information to each of the acquired skeletal data, facial expression data, or audio data. (2) The program described in (1), in which the control unit assigns information regarding the movement, facial expression, or vocalization included in the script data as the attribute information. (3) The program described in (2), in which the control unit further assigns information regarding the person as the attribute information. (4) The program described in any one of (1) to (3), in which the control unit generates a dataset including at least one of the skeletal data, facial expression data, and audio data to which the attribute information has been assigned, and constructs an image generation AI that has learned the dataset. (5) The program according to any one of (1) to (3), wherein the control unit generates a dataset including at least one of the skeletal data, the facial expression data, and the voice data to which the attribute information has been assigned, and constructs an image generation AI by additionally training the dataset on a trained model. (6) The program according to any one of (1) to (5), wherein the control unit generates a proper noun dataset including at least the skeletal data, the facial expression data, or the voice data to which the person's proper noun is assigned as the attribute information, and constructs an image generation AI trained with the proper noun dataset. (7) The program according to any one of (4) to (6), wherein the control unit generates an AI image using the image generation AI based on text transmitted from an external device. (8) The program according to (7), wherein the text includes a designation of a person, and the control unit generates the AI ​​image using the image generation AI that has trained a dataset corresponding to the designated person.(9) The program according to (8), wherein the control unit controls output of feedback information for a specified person. (10) The program according to any one of (7) to (9), wherein the control unit controls transmission of the AI ​​image to the external device. (11) The program according to any one of (7) to (10), wherein the control unit generates, as the AI ​​image, an advertising image for promoting a specified product. (12) The program according to any one of (4) to (11), wherein the control unit provides an image generation application that uses the image generation AI to an external device. (13) The program according to any one of (4) to (12), wherein the control unit provides an API (Application Programming Interface) of the image generation AI to an external device. (14) The program according to any one of (4) to (13), wherein the control unit newly generates the script data based on feedback regarding the AI ​​image generated by the image generation AI. (15) The program according to (1), wherein the control unit generates a dataset including the skeletal data, the facial expression data, and the audio data to which the attribute information has been assigned, constructs an image generation AI that has learned the dataset, and generates an AI image using the image generation AI based on text transmitted from an external device. (16) The program according to (15), wherein the control unit generates a singing video as the AI ​​image. (17) An information processing device comprising a control unit that performs the following processes: acquires, from a captured video of a person performing movements, facial expressions, or vocalizations defined in a time series by script data, each of the skeletal data, facial expression data, or audio data of the person in the captured video by referring to time information of each movement defined in the script data; and assigns attribute information to each of the acquired skeletal data, facial expression data, or audio data.(18) An information processing method including: a processor acquiring, from a captured video in which a person is captured making movements, making expressions, or making sounds that are determined in a time series in script data, each piece of skeletal data, each piece of facial expression data, or each piece of voice data of the person in the captured video by referring to time information of each movement determined in the script data; and assigning attribute information to each piece of acquired skeletal data, each piece of facial expression data, or each piece of voice data.

[0130] 1 Information processing system 10 Model terminal 12 Model agency terminal 20 Server 210 Communication unit 220 Control unit 221 Data separation unit 222 Skeleton data analysis unit 223 Facial expression data analysis unit 224 Voice data analysis unit 225 Data set storage control unit 226 Image generation AI construction unit 227 AI advertising image generation unit 230 Storage unit 231 Script data storage 232 Video data storage 233 Data set storage 234 Image generation AI storage 235 Generated image storage 30 Advertising agent terminal 40 Network

Claims

1. A program that causes a computer to function as a control unit that performs the following processes: acquires, from a captured video of a person making movements, facial expressions, or vocalizations defined in a time series in script data, each skeletal data, each facial expression data, or each voice data of the person in the captured video by referencing the time information of each movement defined in the script data; and assigns attribute information to each of the acquired skeletal data, each facial expression data, or each voice data.

2. The program according to claim 1, wherein the control unit assigns information relating to movements, facial expressions, or vocalizations contained in the script data as the attribute information.

3. The program according to claim 2, wherein the control unit further adds information about the person as the attribute information.

4. The program according to claim 1, wherein the control unit generates a dataset including at least one of the skeletal data, the facial expression data, and the voice data to which the attribute information has been added, and constructs an image generation AI that has been trained on the dataset.

5. The program according to claim 1, wherein the control unit generates a dataset including at least one of the skeletal data, the facial expression data, and the voice data to which the attribute information has been added, and constructs an image generation AI by additionally training the dataset on a trained model.

6. The program according to claim 1, wherein the control unit generates a proper noun dataset including at least the skeletal data, the facial expression data, or the voice data to which the person's proper noun is assigned as the attribute information, and constructs an image generation AI trained on the proper noun dataset.

7. The program described in claim 4, wherein the control unit generates an AI image using the image generation AI based on text transmitted from an external device.

8. The program of claim 7, wherein the text includes a designation of a person, and the control unit generates the AI ​​image using the image generation AI that has learned a dataset corresponding to the designated person.

9. The program according to claim 8, wherein the control unit controls output of feedback information for a designated person.

10. The program described in claim 7, wherein the control unit controls the transmission of the AI ​​image to the external device.

11. The program described in claim 7, wherein the control unit generates an advertising image for advertising a specified product as the AI ​​image.

12. The program described in claim 4, wherein the control unit provides an image generation application that utilizes the image generation AI to an external device.

13. The program according to claim 4, wherein the control unit provides an API (Application Programming Interface) of the image generation AI to an external device.

14. The program described in claim 4, wherein the control unit newly generates the script data based on feedback regarding the AI ​​image generated by the image generation AI.

15. The program described in claim 1, wherein the control unit generates a dataset including the skeletal data, the facial expression data, and the voice data to which the attribute information has been added, constructs an image generation AI that has been trained on the dataset, and generates an AI image using the image generation AI based on text transmitted from an external device.

16. The program described in claim 15, wherein the control unit generates a singing video as the AI ​​image.

17. An information processing device comprising a control unit that performs the following processes: acquiring each skeletal data, each facial expression data, or each voice data of a person in a captured video that captures the person making movements, facial expressions, or vocalizations defined in a time series in script data, by referring to time information of each movement defined in the script data; and assigning attribute information to each of the acquired skeletal data, each facial expression data, or each voice data.

18. An information processing method comprising: a processor acquiring, from a captured video of a person making movements, facial expressions, or vocalizations defined in a time series in script data, each skeletal data, each facial expression data, or each voice data of the person in the captured video by referencing time information of each movement defined in the script data; and assigning attribute information to each of the acquired skeletal data, each facial expression data, or each voice data.