Information processing device and information processing method

Abstract

To provide an information processing device capable of generating various video descriptions according to a frame image to which attention is paid.SOLUTION: An information processing device comprises: a video generation unit that generates a learning video on the basis of an image statement dataset containing a pair of a captured image and image descriptions, which are sentences explaining the contents of the captured image; an extraction unit that extracts a frame feature amount for learning, which is a feature amount of each of a plurality of frame images constituting the learning video; a determination unit that determines a learning weight, which is a weight corresponding to each of the plurality of frame images constituting the learning video; and a model generation unit that generates a sentence generation model, which is a machine learning model trained so as to generate a learning video description, which is a sentence describing the contents of the learning video and has a feature corresponding to the frame feature amount for learning weighted by learning weight, on the basis of the frame feature amount for learning and the learning weight.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Conventionally, there are known techniques for generating captions (also called caption sentences, hereinafter referred to as "video description sentences") from videos. For example, there is known a technique for inputting a video captured by a surveillance camera into a multi-layer neural network that outputs elements contained in an image as words, and generating a description of the video. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-101317 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned conventional technology, a video captured by a surveillance camera is input into a multi-layer neural network that outputs elements contained in the image as words, and a description of the video is generated, so it is not necessarily possible to generate a variety of video descriptions that correspond to the frame images of interest.

[0005] An object of the present application is to provide an information processing device and an information processing method that are capable of generating a variety of video descriptions according to a frame image of interest. [Means for solving the problem]

[0006] The information processing device of the present application includes a video generation unit that generates a training video based on an image and text dataset including a pair of a captured image and an image description that is a text that explains the content of the captured image, an extraction unit that extracts training frame features that are features of each of a plurality of frame images that constitute the training video, a determination unit that determines training weights that are weights corresponding to each of a plurality of frame images that constitute the training video, and a model generation unit that generates a text generation model that is a machine learning model trained to generate training video description that is a text that explains the content of the training video based on the training frame features and the learning weights, the training video description having features corresponding to the training frame features weighted by the learning weights.

[0007] In addition, the information processing device of the present application includes an acquisition unit that acquires a sentence generation model, which is a machine learning model trained to generate training video descriptions, which are sentences explaining the content of a training video, based on training frame features, which are features of each of a plurality of frame images constituting a training video generated based on an image and sentence dataset including a pair of a captured image and an image description, which is a sentence explaining the content of the captured image, and training weights, which are weights corresponding to each of the plurality of frame images constituting the training video; an extraction unit that extracts target frame features, which are features of each of a plurality of frame images constituting a target video, which is a video to be processed; a determination unit that determines target weights, which are weights corresponding to each of the plurality of frame images among the plurality of frame images constituting the target video, so that a weight corresponding to a designated frame image designated by a user among the plurality of frame images constituting the target video is greater than weights corresponding to other frame images other than the designated frame image; and a sentence generation unit that inputs the target frame features weighted by the target weight to the sentence generation model based on the target frame features and the target weight to generate a target video description, which is a sentence explaining the content of the target video. Effect of the Invention

[0008] According to one aspect of the embodiment, it is possible to generate a variety of video descriptions according to a frame image of interest. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of information processing related to the pre-learning method according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of information processing related to the first additional learning method according to the embodiment. [Figure 4] FIG. 4 is a diagram for explaining a method of weighting a frame feature amount by a weight according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of information processing related to the second additional learning method according to the embodiment. [Figure 6] FIG. 6 is a diagram for explaining a method for calculating a similarity according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of information processing related to the inference method according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating a conditional generative adversarial network (CGAN) that is an example of a sentence generation model according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating a conditional variational autoencoder (CVAE) that is an example of a sentence generation model according to the first modified example. [Figure 10] FIG. 10 is a diagram showing a conditional diffusion model, which is an example of a sentence generation model according to the second modified example. [Figure 11] FIG. 11 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a detailed description will be given of an information processing device and an information processing method according to the present application (hereinafter, referred to as an "embodiment") with reference to the drawings. Note that the information processing device and the information processing method according to the present application are not limited to the embodiment. In addition, the same components in the following embodiments are given the same reference numerals, and duplicated descriptions are omitted.

[0011] (Embodiment) 1. Introduction Conventionally, a technique for generating a video description, which is a text that explains the contents of a video, from a video is known. For example, a machine learning model that generates a video description from a video is known. Here, it is known that the content of the video description generated by the machine learning model that generates a video description from a video varies depending on which part of the video is focused on.

[0012] Furthermore, in recent years, technology related to generative models has become known. A generative model is a model of the data generation process. A generative model is a machine learning model that can learn from training data and generate data similar to the training data. Furthermore, technology related to conditional generative models has become known. A conditional generative model is a machine learning model that can change the data generation process by changing conditions, and generate diverse, high-quality data.

[0013] Here, the condition in the conditional generative model refers to a condition that the features of the target data to be generated by the conditional generative model must satisfy. In other words, the condition here refers to a condition related to the type and attributes of the target data to be generated by the conditional generative model (e.g., the type and attributes of features appearing in the target data to be generated, etc.). For example, when the target data to be generated by the conditional generative model is an image, the condition may be information indicating the attribute or type of the object contained in the image. Specifically, by inputting a condition vector, which is a vector input as a condition to the conditional generative model, into the conditional generative model as input information, data having features corresponding to the condition vector can be generated. For example, the condition vector may be a vector corresponding to information indicating the condition.

[0014] For example, Conditional GAN ​​(CGAN) is known as an example of a conditional generative model (reference: paper title "Conditional Generative Adversarial Nets", <Internet> https: / / arxiv.org / pdf / 1411.1784.pdf (retrieved February 16, 2023)). CGAN is also called a conditional generative adversarial network, and is a machine learning model that has been improved to allow conditions to be given to GAN (generative adversarial network), which generates specific data from noise.

[0015] In addition, the Conditional Variational Auto Encoder (CVAE) is known as an example of a conditional generative model (reference: Paper title: “Semi-supervised Learning with Deep Generative Models”, <Internet> https: / / proceedings.neurips.cc / paper / 2014 / file / d523773c6b194f37b938d340d5d02232-Paper.pdf (Retrieved February 16, 2023)). CVAE is also called a conditional variational autoencoder, and is a machine learning model that has been improved to allow conditions to be given to a VAE (variational autoencoder) that generates data according to a latent representation.

[0016] Additionally, the Diffusion Model (Reference: Paper title: “Denoising Diffusion Probabilistic Models”, <Internet> https: / / arxiv.org / pdf / 2006.11239.pdf (Retrieved February 16, 2023)) is known as an example of a conditional generative model. The Diffusion Model learns the process of gradually restoring data from noise. The Diffusion Model is generally called the diffusion model, but as an applied use, it is possible to generate data with conditions given.

[0017] Other examples of conditional generative models include GLIDE (reference; paper title: “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models”, <Internet>https: / / arxiv.org / pdf / 2112.10741.pdf(Retrieved February 16, 2023)), DALL-E 2 unCLIP (reference; paper title: “Hierarchical Text-Conditional Image Generation with CLIP Latents”, <Internet>https: / / arxiv.org / pdf / 2204.06125.pdf(Retrieved February 16, 2023)), Imagen (reference; paper title: “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding”, <Internet>https: / / arxiv.org / pdf / 2205.11487.pdf(Retrieved February 16, 2023)), and Parti (reference; paper title: “Scaling Autoregressive Models for "Content-Rich Text-to-Image Generation", <Internet>https: / / arxiv.org / pdf / 2206.10789.pdf (Retrieved February 16, 2023) is known.

[0018] The information processing device according to the present embodiment inputs weighted frame features as conditions to a conditional generative model based on weights corresponding to each of a plurality of frame images constituting a video and frame features that are features of each of a plurality of frame images constituting a video. The information processing device also generates a video description having features corresponding to the weighted frame features. As a result, the information processing device can control which part of a video (which frame image) is emphasized in generating a video description by using the weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model. The information processing device can also generate a variety of video descriptions according to a frame image of interest. The information processing device can also use the weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model, thereby making it possible to reflect time-series information of a video in natural language generation.

[0019] 2. Configuration of Information Processing Device An example of the configuration of an information processing device 100 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment. The information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0020] (Communication unit 110) The communication unit 110 is realized by a NIC (Network Interface Card), an antenna, etc. The communication unit 110 is connected to various networks by wire or wirelessly, and transmits and receives information to and from other information processing devices other than the information processing device 100, for example.

[0021] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various data. For example, the storage unit 120 stores learning data used for learning a sentence generation model. The storage unit 120 also stores various programs. For example, the storage unit 120 stores information related to the sentence generation model generated by the model generation unit 134.

[0022] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0023] The control unit 130 has a video generation unit 131, an extraction unit 132, a determination unit 133, a model generation unit 134, an acquisition unit 135, and a sentence generation unit 136 as functional units, and may realize or execute the information processing action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 1, and may be other configurations as long as they perform the information processing described below. Also, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

[0024] (Video Generation Unit 131) The video generator 131 generates a learning video based on an image and text data set including a pair of a captured image and an image caption that is a sentence that explains the content of the captured image.

[0025] (Extraction part 132) The extraction unit 132 extracts learning frame features, which are features of each of a plurality of frame images constituting the learning moving image. The extraction unit 132 also extracts target frame features, which are features of each of a plurality of frame images constituting a target moving image, which is a moving image to be processed.

[0026] (Decision unit 133) The determination unit 133 determines learning weights, which are weights corresponding to each of the multiple frame images constituting the learning moving image. Also, the determination unit 133 determines target weights, which are weights corresponding to each of the multiple frame images constituting the target moving image, such that the weight corresponding to a designated frame image designated by a user among the multiple frame images constituting the target moving image is made larger than the weights corresponding to other frame images other than the designated frame image.

[0027] (Model generation unit 134) The model generation unit 134 generates a sentence generation model, which is a machine learning model trained to generate a training video description, which is a sentence that describes the content of a training video, based on the training frame features and the training weights, and has features corresponding to the training frame features weighted by the training weights. Here, the sentence generation model may be a conditional generative model. For example, the sentence generation model may be a conditional generative adversarial network (CGAN), a conditional variational autoencoder (CVAE), or a conditional diffusion model.

[0028] Moreover, the model generation unit 134 generates a pre-trained sentence generation model by pre-training the sentence generation model. Next, the model generation unit 134 generates an additionally trained sentence generation model by additionally training the pre-trained sentence generation model. The model generation unit 134 stores information on the generated additionally trained sentence generation model in the storage unit 120. Note that hereinafter, the additionally trained sentence generation model may be simply referred to as a "sentence generation model".

[0029] (Acquisition part 135) The acquisition unit 135 acquires a sentence generation model, which is a machine learning model trained to generate a training video description, which is a sentence that describes the content of a training video, having features corresponding to the training frame features weighted by the training weights, based on training frame features that are features of each of a plurality of frame images constituting a training video generated based on an image and text dataset including a pair of a captured image and an image description that is a sentence that describes the content of the captured image, and training weights that are weights corresponding to each of a plurality of frame images constituting the training video. Specifically, the acquisition unit 135 acquires the sentence generation model generated by the model generation unit 134. For example, the acquisition unit 135 acquires information on the sentence generation model from the storage unit 120.

[0030] (Sentence generation section 136) The sentence generation unit 136 inputs the target frame features weighted by the target weight to a sentence generation model based on the target frame features and the target weight, and generates a target moving image description sentence that is a sentence explaining the content of the target moving image.

[0031] [3. Pre-learning method] A pre-learning method for a sentence generation model according to an embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of information processing related to the pre-learning method according to an embodiment. Here, pre-learning refers to learning of a model that is performed as a preparation before the model is fully trained by the first additional learning (see Fig. 3) or the second additional learning (see Fig. 5) described later.

[0032] As shown in Figure 2, in the pre-learning stage, (1) image features are extracted from each frame image constituting a video included in a video sentence dataset that includes pairs of videos and video descriptions. (2) No weighting is applied to each frame image constituting a video. (3) The image features extracted from each frame image are input as conditions into a sentence generation model, which is a conditional generation model, and the sentence generation model is trained to generate video descriptions having features corresponding to the image features.

[0033] Specifically, the extraction unit 132 may acquire a moving image text data set #1 including a pair of a captured video for pre-learning (hereinafter, may be referred to as "pre-learning video #1") and a video description corresponding to the captured video for pre-learning (hereinafter, may be referred to as "pre-learning video description #1"). For example, the extraction unit 132 may acquire the moving image text data set #1 from an external information processing device via the communication unit 110.

[0034] Next, the extraction unit 132 may extract image features of each of the multiple frame images constituting the pre-learning video #1 from each of the multiple frame images constituting the pre-learning video #1 included in the video sentence dataset #1 (step S11). For example, the image features may be a multidimensional vector. In FIG. 2, for simplicity, a case where there are three frame images constituting the pre-learning video #1 is described, but the number of frame images constituting the pre-learning video #1 may be four or more. In FIG. 2, a first frame image corresponding to the start time of the pre-learning video #1, a second frame image corresponding to a time between the start time and the end time of the pre-learning video #1, and a third frame image corresponding to the end time of the pre-learning video #1 are arranged in chronological order. For example, the extraction unit 132 extracts a feature vector V11 from the first frame image. In addition, the extraction unit 132 extracts a feature vector V12 from the second frame image. In addition, the extraction unit 132 extracts a feature vector V13 from the third frame image. Next, the extraction unit 132 may acquire a vector (V11, V12, V13) of the set of feature vectors V11 to V13 as pre-learning frame feature #1, which is the image feature of each of the multiple frame images that make up the pre-learning video #1.

[0035] For example, the extraction unit 132 may extract image features from each of a plurality of frame images constituting a moving image, using any known technique capable of extracting image features from an image. For example, the extraction unit 132 may include an image encoder and extract image features using the image encoder. For example, the extraction unit 132 may include a convolutional neural network (CNN) as the image encoder. Then, the extraction unit 132 may extract image features from each frame image using the CNN. Also, for example, the extraction unit 132 may include ResNet (Residual Network) (Kaiming He et al., 2015), AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan et al., 2014), GoogLeNet (Szegedy et al., 2014), SENet (Squeeze-and-Excitation Networks) (Jie Hu et al., 2018), EfficientNet (Tan et al., 2019), or ZFNet (Matthew et al., 2013) developed for object recognition as an image encoder. The extraction unit 132 may extract image features from each frame image using ResNet, AlexNet, VGGNet, GoogLeNet, SENet, EfficientNet, or ZFNet. Also, for example, the extraction unit 132 may include, as an image encoder, Faster R-CNN (Shaoqing Ren et al., 2015), YOLO (You Look Only Onse) (Joseph Redmon et al., 2015), or SSD (Single Shot MultiBox Detector) (Wei Liu., 2015), which are developed for object detection. The extraction unit 132 may extract image features from each frame image using Faster R-CNN, YOLO, or SSD.

[0036] In this way, the extraction unit 132 extracts pre-learning frame features #1, which are features of each of the multiple frame images that constitute a captured video included in the video text dataset #1, which includes a pair of a captured video (in the example of Figure 2, pre-learning video #1) and a video description, which is a sentence that explains the content of the captured video (in the example of Figure 2, pre-learning video description #1).

[0037] Next, the model generation unit 134 may input the pre-learning frame feature #1 extracted by the extraction unit 132 to the sentence generation model M1 (step S12). For example, the model generation unit 134 may generate a condition vector #1 based on the pre-learning frame feature #1. Next, the model generation unit 134 may combine the generated condition vector #1 with a noise vector (also called a random vector). For example, the model generation unit 134 may adjust the size of the condition vector #1 and the noise vector so that they are the same using a linear conversion process. Next, the model generation unit 134 may combine the condition vector #1 with the noise vector by adding each element of the condition vector #1 to each element of the noise vector. Alternatively, the model generation unit 134 may combine the condition vector #1 with the noise vector by multiplying each element of the condition vector #1 by each element of the noise vector. Next, the model generation unit 134 may input the combined condition vector #1 and the noise vector as input information to the sentence generation model M1, which is a conditional generation model.

[0038] Next, the model generation unit 134 may acquire the moving image description generated by the sentence generation model M1 in response to the input of the combined condition vector #1 and the noise vector, and output from the sentence generation model M1 as output information (step S13). The model generation unit 134 may train the sentence generation model M1 so as to generate a moving image description having a feature corresponding to the condition vector #1 based on the condition vector #1. For example, the model generation unit 134 may train the sentence generation model M1 using backpropagation (error backpropagation method) or the like so as to reduce an error between the moving image description output from the sentence generation model M1 and the pre-learning moving image description #1 included in the moving image description data set #1. In this way, the model generation unit 134 may train the sentence generation model M1 so as to generate a moving image description having a feature corresponding to the pre-learning frame feature #1 based on the pre-learning frame feature #1.

[0039] In this way, the model generation unit 134 generates a pre-trained sentence generation model M1, which is a machine learning model that has been pre-trained to generate a video description (in Figure 2, pre-training video description #1) having features corresponding to the pre-training frame feature #1, based on the pre-training frame feature #1.

[0040] [4. First additional learning method] A first additional learning method according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of information processing related to the first additional learning method according to the embodiment. The first additional learning refers to full-scale model learning performed after the pre-learning described in Fig. 2.

[0041] As shown in FIG. 3, in the first additional learning stage, (1) a known video generation model is used to generate a video from images included in an image-sentence dataset including a pair of images (still images) and image captions. Hereinafter, the image (included in the image-sentence dataset) from which the video is generated may be referred to as the "original image". The generated video includes the original image in a frame. (2) Among the frame images constituting the generated video, the frame image corresponding to the original image is set as a focus point, and weights corresponding to each frame image constituting the video are determined. In addition, image features are extracted from each frame image constituting the generated video, and the image features extracted from each frame image are weighted by the weights corresponding to each frame image. (3) The weighted image features are input as conditions to a pre-trained sentence generation model M1, which is a conditional generation model, and the pre-trained sentence generation model M1 is re-trained to generate a video caption having features corresponding to the weighted image features, thereby generating a first additionally trained sentence generation model M2.

[0042] Specifically, the video generating unit 131 may generate the learning video #2 based on an image and text data set #2 including a pair of a captured image (hereinafter, sometimes referred to as "image #2") and an image caption (hereinafter, sometimes referred to as "image caption #2") that is a sentence that explains the contents of the captured image. For example, the video generating unit 131 may acquire the image and text data set #2 from an external information processing device via the communication unit 110. Next, the video generating unit 131 may acquire a first video generation model M21, which is a machine learning model that generates a video from an image. For example, the first video generation model M21 may be a known machine learning model that generates a video from an image (Reference: "Generating Videos with Scene Dynamics", Carl Vondrick et al., 2016, <Internet> https: / / arxiv.org / pdf / 1609.02612.pdf (searched on February 16, 2023)). For example, the video generator 131 may acquire the first video generation model M21 from the storage unit 120 that stores information about the first video generation model M21 in advance. Next, the video generator 131 may input the image #2 included in the image-sentence dataset #2 to the first video generation model M21, and generate a learning video #2 including the image #2 in a frame from the image #2 (step S21).

[0043] In this way, the video generation unit 131 uses the first video generation model M21, which is a machine learning model that generates videos from images, to generate a learning video #2 that includes a captured image in a frame from a captured image (image #2 in FIG. 3) included in the image-sentence dataset #2. Hereinafter, the image #2 that is the source from which the learning video #2 is generated may be referred to as the "original image #2."

[0044] The extraction unit 132 may extract image features of each of the frame images constituting the learning video #2 from each of the frame images constituting the learning video #2 generated by the video generation unit 131 (step S22). The method by which the extraction unit 132 extracts image features from each frame image may be the same as the method of extracting image features from each frame image in the pre-learning described in FIG. 2. In the following, the description overlapping with FIG. 2 will be omitted. In FIG. 3, for simplicity, a case in which the learning video #2 includes three frame images will be described, but the number of frame images constituting the learning video #2 may be four or more. In FIG. 3, a first frame image corresponding to the start time of the learning video #2, a second frame image corresponding to a time between the start time and the end time of the learning video #2, and a third frame image corresponding to the end time of the learning video #2 are arranged in chronological order. For example, the extraction unit 132 extracts a feature vector V21 from the first frame image. The extraction unit 132 also extracts a feature vector V22 from the second frame image. The extraction unit 132 also extracts a feature vector V23 from the third frame image. Next, the extraction unit 132 may acquire a vector (V21, V22, V23) of the set of feature vectors V21 to V23 as learning frame feature #2, which is the image feature of each of the multiple frame images that make up the learning video #2.

[0045] In this manner, the extraction unit 132 extracts learning frame features #2, which are features of each of the multiple frame images that make up the learning video #2.

[0046] Furthermore, the determination unit 133 may determine weights corresponding to the multiple frame images constituting the learning video #2 generated by the video generation unit 131 (step S22). Either the process of extracting image features by the extraction unit 132 or the process of determining weights by the determination unit 133 may be performed first, or may be performed simultaneously by the extraction unit 132 and the determination unit 133.

[0047] For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images constituting the learning video #2 such that the weight of the frame image corresponding to the original image #2 is greater than the weights corresponding to the other frame images other than the frame image corresponding to the original image #2. For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images based on the value of a convex function such as a Gaussian function (also called a normal distribution) or a part of a circle, the periphery of which is differentiable. Note that the determination unit 133 may determine weights using any function, not limited to a Gaussian function or a part of a circle, as long as the periphery of the maximal value is differentiable. For example, the determination unit 133 may set a value corresponding to the maximal value of a function whose periphery is differentiable as the weight of the frame image corresponding to the original image #2. Furthermore, the determination unit 133 may set a value corresponding to the periphery of the maximal value of a function whose periphery is differentiable as the weight of the frame image other than the frame image corresponding to the original image #2.

[0048] In FIG. 3, the determination unit 133 may determine weights corresponding to each of a plurality of frame images using values ​​of a Gaussian function with the horizontal axis representing the playback time of the video and the vertical axis representing the weight. For example, the determination unit 133 may determine the time corresponding to the average value of the Gaussian function as the playback time of the second frame image corresponding to the original image #2. The determination unit 133 may also determine "1.0", which is the value of the time corresponding to the average value of the Gaussian function, as the weight #22 of the second frame image corresponding to the original image #2. The determination unit 133 may also determine the time corresponding to a value smaller than the average value of the Gaussian function as the playback time of the first frame image. The determination unit 133 may also determine "0.8", which is the value of the time corresponding to a value smaller than the average value of the Gaussian function, as the weight #21 of the first frame image. The determination unit 133 may also determine the time corresponding to a value larger than the average value of the Gaussian function as the playback time of the third frame image. Furthermore, the determination unit 133 may set "0.8", which is the value at the time corresponding to a value greater than the average value of the Gaussian function, as weight #23 for the third frame image. For example, the determination unit 133 may acquire a vector (weight #21, weight #22, weight #23) = (0.8, 1.0, 0.8) of the set of weight #21 for the first frame image to weight #23 for the second frame image as learning weight #2, which is the weight corresponding to each of the multiple frame images constituting the learning video #2.

[0049] In this way, the determination unit 133 determines the learning weights #2 (in FIG. 3, (weights #21, weights #22, weights #23) = (0.8, 1.0, 0.8)) corresponding to each of the multiple frame images constituting the learning video #2 so that the weight (in FIG. 3, weight #22 of the second frame image, "1.0") corresponding to the captured image (original image #2 in FIG. 3) is greater than the weights corresponding to the other frame images other than the captured image (in FIG. 3, weight #21 of the first frame image, "0.8" and weight #23 of the third frame image, "0.8"). Furthermore, in this way, the determination unit 133 determines the learning weights #2 that are weights corresponding to each of the multiple frame images constituting the learning video #2.

[0050] Furthermore, the model generation unit 134 may weight the learning frame feature #2 extracted by the extraction unit 132 by the learning weight #2 determined by the determination unit 133. The model generation unit 134 may generate a weighted learning frame feature #2', which is the learning frame feature #2 weighted by the learning weight #2 determined by the determination unit 133. In FIG. 3, the model generation unit 134 may generate a feature vector V21' weighted by the weight #21 by multiplying each element of the feature vector V21 by "0.8", which is the weight #21 corresponding to the first frame image constituting the learning video #2. Furthermore, the model generation unit 134 may generate a feature vector V22' weighted by the weight #22 by multiplying each element of the feature vector V22 by "1.0", which is the weight #22 corresponding to the second frame image constituting the learning video #2. Furthermore, the model generation unit 134 may generate a feature vector V23' weighted by weight #23 by multiplying each element of the feature vector V23 by "0.8", which is weight #23 corresponding to the third frame image constituting the learning video #2. In this manner, the model generation unit 134 may generate weighted learning frame feature #2'. In FIG. 3, the model generation unit 134 may generate (weight #21, weight #22, weight #23)*(V21, V22, V23)=(weight #21*V21, weight #22*V22, weight #23*V23)=(V21', V22', V23') as the weighted learning frame feature #2'.

[0051] Next, the model generation unit 134 may input the weighted training frame feature #2' to the pre-trained sentence generation model M1 (hereinafter, may be abbreviated as "sentence generation model M1") (step S23). For example, the model generation unit 134 may generate a condition vector #2 based on the weighted training frame feature #2'. Next, the model generation unit 134 may combine the generated condition vector #2 with a noise vector. Note that the method by which the model generation unit 134 combines the condition vector #2 with the noise vector may be the same as the method by which the condition vector #1 with the noise vector is combined in the pre-training described in FIG. 2. In the following, the description overlapping with FIG. 2 will be omitted. Next, the model generation unit 134 may input the combined condition vector #2 and noise vector as input information to the sentence generation model M1, which is a conditional generation model.

[0052] Next, the model generation unit 134 may acquire a moving image description generated by the sentence generation model M1 in response to the input of the combined condition vector #2 and the noise vector, and output from the sentence generation model M1 as output information (hereinafter, may be referred to as "learning moving image description #2") (step S24). The model generation unit 134 may retrain the sentence generation model M1 so as to generate a moving image description having a feature corresponding to the condition vector #2 based on the condition vector #2. For example, the model generation unit 134 may retrain the sentence generation model M1 using backpropagation (error backpropagation method) or the like so as to reduce an error between the learning moving image description #2 output from the sentence generation model M1 and the image description #2 (image description corresponding to the original image #2) included in the image sentence data set #2. In this way, the model generation unit 134 may retrain the sentence generation model M1 so as to generate a moving image description having a feature corresponding to the weighted learning frame feature #2' based on the weighted learning frame feature #2'.

[0053] In this way, the model generation unit 134 generates the sentence generation model M2 by re-training the pre-trained sentence generation model M1 to generate training video description #2, which is a sentence explaining the content of the training video, based on the training frame feature #2 and the training weight #2, and which has features corresponding to the training frame feature #2' weighted by the training weight #2.

[0054] FIG. 4 is a diagram for explaining a method of weighting a frame feature by a weight according to an embodiment. The frame feature may have a value corresponding to each pixel of an image. In the example shown in FIG. 4, for simplicity, a case will be explained in which the pixels of an image are represented by a 3×3 matrix. In this case, the frame feature may be represented by a 3×3 matrix. Also, for simplicity, the weight value is set to "3". In this case, the model generation unit 134 generates a weighted frame feature by multiplying each element of the frame feature (each element of the 3×3 matrix) by the weight value "3".

[0055] [5. Second additional learning method] A second additional learning method according to the embodiment will be described with reference to Fig. 4. Fig. 5 is a diagram showing an example of information processing related to the second additional learning method according to the embodiment. The second additional learning refers to full-scale model learning performed after the pre-learning described in Fig. 2. Fig. 5 differs from Fig. 3 in that the model generation unit 134 re-learns the pre-trained sentence generation model M1 by the second additional learning instead of the first additional learning.

[0056] As shown in FIG. 5, in the second additional learning stage, (1) a known video generation model is used to generate a video from an image description included in an image-sentence data set including a pair of an image (still image) and an image description. Hereinafter, an image (included in an image-sentence data set) corresponding to the image description from which the video is generated may be referred to as an "original image". (2) A similarity between each frame image constituting the generated video and the original image is calculated, and the calculated similarity is set as a weight corresponding to each frame image constituting the video. In addition, image features are extracted from each frame image constituting the generated video, and the image features extracted from each frame image are weighted by the weight corresponding to each frame image. (3) The weighted image features are input as conditions to a pre-trained sentence generation model M1, which is a conditional generation model, and the pre-trained sentence generation model M1 is re-trained to generate a video description having features corresponding to the weighted image features, thereby generating a second additionally trained sentence generation model M3.

[0057] Specifically, the video generation unit 131 may generate the learning video #3 based on an image and sentence data set #3 including a pair of a captured image (hereinafter, sometimes referred to as "image #3") and an image description (hereinafter, sometimes referred to as "image description #3") that is a sentence that explains the contents of the captured image. For example, the video generation unit 131 may acquire the image and sentence data set #3 from an external information processing device via the communication unit 110. Next, the video generation unit 131 may acquire a second video generation model M31, which is a machine learning model that generates a video from a sentence. For example, the second video generation model M31 may be a known machine learning model that generates a video from a sentence (References; "CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers", Wenyi Hong et al, 2022), <Internet> https: / / arxiv.org / pdf / 2205.15868.pdf (searched February 16, 2023)). For example, the video generating unit 131 may acquire the second video generation model M31 from the storage unit 120 that stores information about the second video generation model M31 in advance. Next, the video generating unit 131 may input the image caption #3 included in the image text data set #3 to the second video generation model M31, and generate a learning video #3 from the image caption #3 (step S31).

[0058] In this way, the video generation unit 131 generates a learning video #3 from the image caption #3 included in the image and sentence dataset #3 using the second video generation model M31, which is a machine learning model that generates videos from sentences. In the following, the image caption #3 from which the learning video #3 is generated and the corresponding image #3 (image #3 included in the image and sentence dataset #3) may be referred to as the "original image #3".

[0059] In addition, the extraction unit 132 may extract image features of each of the multiple frame images constituting the learning video #3 from each of the multiple frame images constituting the learning video #3 generated by the video generation unit 131 (step S32). As in FIG. 3, the method in which the extraction unit 132 extracts image features from each frame image may be the same as the method in which image features are extracted from each frame image in the pre-learning described in FIG. 2. In the following, explanations that overlap with FIG. 2 will be omitted. In FIG. 5, for simplicity, a case in which there are three frame images constituting the learning video #3 will be described, but the number of frame images constituting the learning video #3 may be four or more. In FIG. 5, a first frame image corresponding to the start time of the learning video #3, a second frame image corresponding to a time between the start time and the end time of the learning video #3, and a third frame image corresponding to the end time of the learning video #3 are shown arranged in chronological order. For example, the extraction unit 132 extracts a feature vector V31 from the first frame image. Furthermore, the extraction unit 132 extracts a feature vector V32 from the second frame image. Furthermore, the extraction unit 132 extracts a feature vector V33 from the third frame image. Next, the extraction unit 132 may acquire a vector (V31, V32, V33) of the set of feature vectors V31 to V33 as learning frame feature #3, which is the image feature of each of the multiple frame images that make up the learning video #3.

[0060] In this manner, the extraction unit 132 extracts learning frame features #3 that are features of each of the multiple frame images that make up the learning video #3.

[0061] Furthermore, the determination unit 133 may determine weights corresponding to the multiple frame images constituting the learning video #3 generated by the video generation unit 131 (step S32). As in Fig. 3, the process of extracting image features by the extraction unit 132 and the process of determining weights by the determination unit 133 may be performed either first or simultaneously by the extraction unit 132 and the determination unit 133.

[0062] For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images constituting the learning video #3 based on the similarity between each of the multiple frame images and the original image #3. In FIG. 5, the determination unit 133 calculates the similarity #31 between the first frame image and the original image #3 to be "0.1". Then, the determination unit 133 may set the calculated value of similarity #31, "0.1", as the weight #31 of the first frame image. Furthermore, the determination unit 133 calculates the similarity #32 between the second frame image and the original image #3 to be "0.7". Then, the determination unit 133 may set the calculated value of similarity #32, "0.7", as the weight #32 of the second frame image. Furthermore, the determination unit 133 calculates the similarity #33 between the third frame image and the original image #3 to be "0.2". Next, the determination unit 133 may set the calculated value of similarity #33, "0.2", as weight #33 for the third frame image. For example, the determination unit 133 may obtain a vector (weight #31, weight #32, weight #33) = (similarity #31, similarity #32, similarity #33) = (0.1, 0.7, 0.2) of the set of weight #31 of the first frame image to weight #33 of the second frame image as learning weight #3, which is a weight corresponding to each of the multiple frame images that make up the learning video #3.

[0063] In this way, the determination unit 133 sets information regarding the similarity (in FIG. 5, (similarity #31, similarity #32, similarity #33) = (0.1, 0.7, 0.2)) between each of the multiple frame images (in FIG. 5, the first to third frame images) constituting the learning video #3 and the captured image (in FIG. 5, original image #3) as learning weights #3 corresponding to each of the multiple frame images (in FIG. 5, (weights #31, weights #32, weights #33) = (0.1, 0.7, 0.2)). Furthermore, in this way, the determination unit 133 determines learning weights #3, which are weights corresponding to each of the multiple frame images constituting the learning video #3.

[0064] Furthermore, the model generation unit 134 may weight the learning frame feature #3 extracted by the extraction unit 132 by the learning weight #3 determined by the determination unit 133. The model generation unit 134 may generate a weighted learning frame feature #3', which is the learning frame feature #3 weighted by the learning weight #3 determined by the determination unit 133. In FIG. 5, the model generation unit 134 may generate a feature vector V31' weighted by the weight #31 by multiplying each element of the feature vector V31 by "0.1", which is the weight #31 corresponding to the first frame image constituting the learning video #3. Furthermore, the model generation unit 134 may generate a feature vector V32' weighted by the weight #32 by multiplying each element of the feature vector V32 by "0.7", which is the weight #32 corresponding to the second frame image constituting the learning video #3. Furthermore, the model generation unit 134 may generate a feature vector V33' weighted by weight #33 by multiplying each element of the feature vector V33 by "0.2", which is weight #33 corresponding to the third frame image constituting the learning video #3. In this manner, the model generation unit 134 may generate weighted learning frame feature #3'. In FIG. 5, the model generation unit 134 may generate (weight #31, weight #32, weight #33)*(V31, V32, V33)=(weight #31*V31, weight #32*V32, weight #33*V33)=(V31', V32', V33') as the weighted learning frame feature #3'.

[0065] Next, the model generation unit 134 may input the weighted learning frame feature #3' to the pre-trained sentence generation model M1 (hereinafter, may be abbreviated as "sentence generation model M1") (step S33). For example, the model generation unit 134 may generate a condition vector #3 based on the weighted learning frame feature #3'. Next, the model generation unit 134 may combine the generated condition vector #3 with a noise vector. Note that the method by which the model generation unit 134 combines the condition vector #3 with the noise vector may be the same as the method by which the condition vector #1 with the noise vector is combined in the pre-training described in FIG. 2. In the following, the description overlapping with FIG. 2 will be omitted. Next, the model generation unit 134 may input the combined condition vector #3 and noise vector as input information to the sentence generation model M1, which is a conditional generation model.

[0066] Next, the model generation unit 134 may acquire a moving image description (hereinafter, may be referred to as "learning moving image description #3") that is generated by the sentence generation model M1 in response to the input of the combined condition vector #3 and the noise vector and is output from the sentence generation model M1 as output information (step S34). The model generation unit 134 may retrain the sentence generation model M1 so as to generate a moving image description having a feature corresponding to the condition vector #3 based on the condition vector #3. For example, the model generation unit 134 may retrain the sentence generation model M1 using backpropagation (error backpropagation method) or the like so as to reduce an error between the learning moving image description #3 output from the sentence generation model M1 and the image description #3 (image description corresponding to the original image #3) included in the image sentence data set #3. In this way, the model generation unit 134 may retrain the sentence generation model M1 so as to generate a moving image description having a feature corresponding to the weighted learning frame feature #3' based on the weighted learning frame feature #3'.

[0067] In this way, the model generation unit 134 generates a sentence generation model M3 by re-training the pre-trained sentence generation model M1 to generate training video description #3, which is a sentence explaining the content of training video #3, based on training frame feature #3 and training weight #3, and which has features corresponding to training frame feature #3' weighted by training weight #3.

[0068] FIG. 6 is a diagram for explaining a method of calculating similarity according to an embodiment. A frame image and an original image #3 have pixel values ​​corresponding to each pixel of the image. In the example shown in FIG. 6, for simplicity, a case where the pixels of an image are expressed as a 3×3 matrix will be explained. The left side of FIG. 6 shows one frame image among a plurality of frame images constituting a learning video #3. The right side of FIG. 6 shows an original image #3. At this time, the determination unit 133 may calculate a cosine similarity between the one frame image and the original image #3 as the similarity between the one frame image and the original image #3. For example, the determination unit 133 may calculate a cosine similarity between the one frame image and the original image #3 according to the following mathematical formula (1).

[0069]

number

[0070] For example, the determination unit 133 calculates the cosine similarity between one frame image and the original image #3 according to the above formula (1) as "{(1.2*0.2)+(2.4*7.2)+((-2.3)*0.9)+(0.8*(-2.4))+((-1.3)*(-3.9))+((-1.2)*(-3.6))+(2.0*6.0)+((-3.2)*9.6)+(0.3*1.9)} / {1.2 2 +2.4 2 +((-2.3)) 2 +0.8 2 +((-1.3)) 2 +((-1.2)) 2 +2.0 2 +((-3.2))2 +0.3 2} 1 / 2 {0.2 2 +7.2 2 +0.9 2 +((-2.4)) 2 +((-3.9)) 2 +((-3.6)) 2 +6.0 2 +9.6 2 +1.9 2} 1 / 2 =0.05".

[0071] [6. Reasoning method] The inference method according to the embodiment will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of information processing related to the inference method according to the embodiment. In the inference stage, a target video description, which is a text that describes the contents of a target video that is a video to be processed, is generated using the first additionally trained text generation model M2 described in FIG. 3 or the second additionally trained text generation model M3 described in FIG. 5. FIG. 7 describes a case where the text generation unit 136 generates the target video description using the first additionally trained text generation model M2 (hereinafter, may be abbreviated as "text generation model M2"). Note that the text generation unit 136 may generate the target video description using the second additionally trained text generation model M3 (hereinafter, may be abbreviated as "text generation model M3") instead of the text generation model M2.

[0072] As shown in Figure 7, in the inference stage, (1) image features are extracted from each frame image constituting the target video. (2) A frame image of interest (hereinafter sometimes referred to as a "designated frame image") is specified by the user, and weights corresponding to each of the multiple frame images are determined so that the weight corresponding to the designated frame image is maximized. (3) The image features extracted from each frame image are weighted by the weight corresponding to each frame image. The weighted image features are input as conditions to sentence generation model M2, which is a conditional generation model. (4) A description of the target video is generated using sentence generation model M2.

[0073] Specifically, the extraction unit 132 may acquire the target video #4, which is a video to be processed. For example, the extraction unit 132 may acquire the target video #4 from an information processing device used by a user via the communication unit 110. Next, the extraction unit 132 may extract image features of each of the multiple frame images constituting the target video #4 from each of the multiple frame images constituting the target video #4 (step S41). As in FIG. 3 and FIG. 5, the method in which the extraction unit 132 extracts image features from each frame image may be the same as the method in which the image features are extracted from each frame image in the pre-learning described in FIG. 2. In the following, descriptions that overlap with FIG. 2 will be omitted. In FIG. 7, for simplicity, a case in which the target video #4 has three frame images will be described, but the number of frame images constituting the target video #4 may be four or more. FIG. 7 shows a first frame image corresponding to the start time of the target moving image #4, a second frame image corresponding to a time between the start time and end time of the target moving image #4, and a third frame image corresponding to the end time of the target moving image #4, arranged in chronological order. For example, the extraction unit 132 extracts a feature vector V41 from the first frame image. The extraction unit 132 also extracts a feature vector V42 from the second frame image. The extraction unit 132 also extracts a feature vector V43 from the third frame image. Next, the extraction unit 132 may acquire a vector (V41, V42, V43) of the set of feature vectors V41 to V43 as target frame feature #4, which is the image feature of each of the multiple frame images constituting the target moving image #4.

[0074] In this manner, the extraction section 132 extracts target frame feature amounts #4 that are feature amounts of each of a plurality of frame images that make up target moving image #4, which is the moving image to be processed.

[0075] Furthermore, the determination unit 133 may acquire a frame image designated by a user from among a plurality of frame images constituting the target moving image #4 (hereinafter, may be referred to as a "designated frame image #4") and the target moving image #4. For example, the determination unit 133 may acquire the designated frame image #4 and the target moving image #4 from an information processing device used by the user via the communication unit 110. Next, the determination unit 133 may determine a weight corresponding to each of a plurality of frame images constituting the target moving image #4 (step S41). Note that, as in the case of FIG. 3 and FIG. 5, either the process of extracting image features by the extraction unit 132 or the process of determining weights by the determination unit 133 may be performed first, or may be performed simultaneously by the extraction unit 132 and the determination unit 133.

[0076] For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images such that the weight of the frame image corresponding to the designated frame image #4 among the multiple frame images constituting the target moving image #4 is greater than the weights corresponding to the other frame images other than the frame image corresponding to the designated frame image #4. For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images based on the value of a convex function such as a Gaussian function or a part of a circle, the periphery of which is differentiable around the maximum value. In FIG. 7, the determination unit 133 may determine weights corresponding to each of the multiple frame images using the value of a Gaussian function with the horizontal axis representing the playback time of the moving image and the vertical axis representing the weight, as in FIG. 3. For example, the determination unit 133 may determine the time corresponding to the average value of the Gaussian function as the playback time of the second frame image corresponding to the designated frame image #4. Furthermore, the determination unit 133 may determine the weight #42 of the second frame image corresponding to the designated frame image #4 to be "1.0", which is the value of the time corresponding to the average value of the Gaussian function. Furthermore, the determination unit 133 may determine the time corresponding to a value smaller than the average value of the Gaussian function as the playback time of the first frame image. Furthermore, the determination unit 133 may determine "0.8", which is the value of the time corresponding to a value smaller than the average value of the Gaussian function, as the weight #41 of the first frame image. Furthermore, the determination unit 133 may determine the time corresponding to a value larger than the average value of the Gaussian function, as the playback time of the third frame image. Furthermore, the determination unit 133 may determine "0.8", which is the value of the time corresponding to a value larger than the average value of the Gaussian function, as the weight #43 of the third frame image. For example, the determination unit 133 may acquire a vector (weight #41, weight #42, weight #43) = (0.8, 1.0, 0.8) of the set of weight #41 of the first frame image to weight #43 of the second frame image as target weight #4, which is the weight corresponding to each of the multiple frame images constituting the target moving image #4.

[0077] In this way, the determination unit 133 determines target weights #4, which are weights corresponding to each of the multiple frame images that make up the target moving image #4, so that the weight corresponding to the designated frame image #4 designated by the user (in FIG. 7, weight #42 of the second frame image, "1.0") is greater than the weights corresponding to the other frame images other than the designated frame image (in FIG. 7, weight #41 of the first frame image, "0.8", and weight #43 of the third frame image, "0.8"). Also, in this way, the determination unit 133 determines target weights #4, which are weights corresponding to each of the multiple frame images that make up the target moving image #4.

[0078] Furthermore, the sentence generation unit 136 may weight the target frame feature #4 extracted by the extraction unit 132 by the target weight #4 determined by the determination unit 133. The sentence generation unit 136 may generate a weighted target frame feature #4', which is the target frame feature #4 weighted by the target weight #4 determined by the determination unit 133. In FIG. 7, the sentence generation unit 136 may generate a feature vector V41' weighted by the weight #41 by multiplying each element of the feature vector V41 by "0.8", which is the weight #41 corresponding to the first frame image constituting the target moving image #4. Furthermore, the sentence generation unit 136 may generate a feature vector V42' weighted by the weight #42 by multiplying each element of the feature vector V42 by "1.0", which is the weight #42 corresponding to the second frame image constituting the target moving image #4. Furthermore, the sentence generation unit 136 may generate a feature vector V43' weighted by weight #43 by multiplying each element of the feature vector V43 by "0.8", which is weight #43 corresponding to the third frame image constituting the target moving image #4. In this manner, the sentence generation unit 136 may generate a weighted target frame feature #4'. In FIG. 7, the sentence generation unit 136 may generate (weight #41, weight #42, weight #43)*(V41, V42, V43)=(weight #41*V41, weight #42*V42, weight #43*V43)=(V41', V42', V43') as the weighted target frame feature #4'.

[0079] Furthermore, the acquiring unit 135 may acquire the sentence generation model M2. For example, the acquiring unit 135 may acquire the sentence generation model M2 from the storage unit 120 that stores information about the sentence generation model M2.

[0080] In this way, the acquisition unit 135 acquires the sentence generation model M2, which is a machine learning model trained to generate training video description #2, which is a sentence explaining the content of the training video #2, having features corresponding to the training frame feature #2 weighted by the training weight #2, based on training frame feature #2, which is a feature of each of the multiple frame images constituting the training video #2 generated based on the image and sentence dataset #2 including a pair of a captured image (in Figure 7, image #2 described in Figure 3) and an image description (in Figure 7, image description #2 described in Figure 3), which is a sentence explaining the content of the captured image, and training weight #2, which is a weight corresponding to each of the multiple frame images constituting the training video #2.

[0081] Furthermore, the sentence generation unit 136 may input the weighted target frame feature #4' to the sentence generation model M2 acquired by the acquisition unit 135 (step S42). For example, the sentence generation unit 136 may generate a condition vector #4 based on the weighted target frame feature #4'. Next, the sentence generation unit 136 may combine the generated condition vector #4 with a noise vector. The method by which the sentence generation unit 136 combines the condition vector #4 with the noise vector may be the same as the method by which the sentence generation unit 136 combines the condition vector #1 with the noise vector in the pre-learning described in FIG. 2. In the following, the description overlapping with FIG. 2 will be omitted. Next, the sentence generation unit 136 may input the combined condition vector #4 and noise vector as input information to the sentence generation model M2, which is a conditional generation model.

[0082] Next, the sentence generation unit 136 may acquire a video description (hereinafter, sometimes referred to as "target video description #4") generated by the sentence generation model M2 in response to the input of the combined condition vector #4 and the noise vector, and output as output information from the sentence generation model M2 (step S43).

[0083] In this way, based on the target frame feature #4 and the target weight #4, the sentence generation unit 136 inputs the target frame feature #4' weighted by the target weight into the sentence generation model M2 to generate a target moving image description sentence #4, which is a sentence that explains the content of the target moving image #4.

[0084] [7. Example of a sentence generation model] Fig. 8 is a diagram showing a conditional generative adversarial network (CGAN) which is an example of a sentence generation model according to an embodiment. As shown in Fig. 8, the sentence generation model may be a conditional generative adversarial network including a generator network G1 and a classifier network D1. Fig. 8 describes the first additional learning described in Fig. 3 or the second additional learning described in Fig. 5.

[0085] The generator network G1 shown in Fig. 8 may be a machine learning model suitable for generating text, which is time-series data. For example, the generator network G1 may be a recurrent neural network (RNN), a gated recurrent unit (GRU), a long short-term memory (LSTM), a Transformer (Ashish Vaswani et al., 2017), a Bidirectional Encoder Representations from Transformers (BERT) based on the Transformer, Generative Pre-Training 3 (GPT-3), or a Text-to-Text Transfer Transformer (T5), etc.

[0086] First, the model generation unit 134 may train a conditional generative adversarial network including the generator network G1 and the discriminator network D1 by the pre-learning described in FIG. 2. For example, the model generation unit 134 may generate a pre-learning adversarial condition vector V50, which is a condition vector based on the pre-learning frame feature #50. Next, the model generation unit 134 may train the generator network G1 to generate a video description having a feature corresponding to the pre-learning adversarial condition vector V50 when the pre-learning adversarial condition vector V50 and a noise vector are input to the generator network G1 as input information. For example, the model generation unit 134 may train the generator network G1 using backpropagation (error backpropagation method) or the like so that the error between the video description output from the generator network G1 and the pre-learning video description #50 included in the video text dataset #50 is reduced. In this way, the model generation unit 134 may generate a pre-trained generator network G1 that has been pre-trained to generate video description text having features corresponding to the pre-training frame feature #50, based on the pre-training frame feature #50.

[0087] In addition, the model generation unit 134 may train the classifier network D1 so that, when the pre-learning adversarial condition vector # and the pre-learning video description #50 are input as input information to the classifier network D1, the classifier network D1 outputs information (e.g., the number "1") indicating that the pre-learning video description #50 is a true video description and a video description corresponding to the pre-learning adversarial condition vector V50 as output information. Note that the model generation unit 134 may adjust the size of the pre-learning adversarial condition vector V50 and the pre-learning video description #50 to be the same, for example, using a linear conversion process. Next, the model generation unit 134 may combine the pre-learning adversarial condition vector V50 and the pre-learning video description #50, and input the combined pre-learning adversarial condition vector V50 and the pre-learning video description #50 as input information to the classifier network D1. In addition, the model generation unit 134 may train the classifier network D1 so that when the pre-training adversarial condition vector V50 and the fake video description generated by the generator network G1 are input as input information to the classifier network D1, the classifier network D1 outputs information (e.g., the number "0") other than information indicating that the fake video description generated by the generator network G1 is a true video description and corresponds to the pre-training adversarial condition vector V50 as output information. In this way, the model generation unit 134 may generate the pre-trained classifier network D1 that has been trained in advance based on the pre-training frame feature #50.

[0088] In addition, the model generation unit 134 may re-train the pre-trained generator network G1 (hereinafter, may be abbreviated as "generator network G1") and the pre-trained classifier network D1 (hereinafter, may be abbreviated as "classifier network D1") by the first additional learning described in Figure 3 or the second additional learning described in Figure 5.

[0089] In FIG. 8, the model generation unit 134 may generate a first training adversarial condition vector V51 (hereinafter, may be abbreviated as "first adversarial condition vector V51") which is a condition vector based on the training frame feature weighted by the training weight. Next, the model generation unit 134 may retrain the generator network G1 so as to generate a video description (training video description #51 in FIG. 8) having a feature corresponding to the first adversarial condition vector V51 when the first adversarial condition vector V51 and the noise vector N1 are input to the generator network G1 as input information. For example, the model generation unit 134 may retrain the generator network G1 using backpropagation (error backpropagation method) or the like so as to reduce an error between the training video description #51 output from the generator network G1 and the image description #51 (image description corresponding to the original image #51) included in the image description dataset #51. In this manner, the model generation unit 134 may retrain the generator network G1 to generate video description sentences having features corresponding to the weighted training frame features based on the weighted training frame features. In this manner, the model generation unit 134 may generate the first additionally trained or second additionally trained generator network G1.

[0090] In addition, the model generation unit 134 may retrain the classifier network D1 so that when the first hostile condition vector V51 and the training video description #51, which is a false video description generated by the generator network G1, are input as input information to the classifier network D1, the classifier network D1 outputs information other than information indicating that the training video description #51 is a true video description and a video description corresponding to the first hostile condition vector V51 (for example, the number "0") as output information. In addition, the model generation unit 134 may generate a second training hostile condition vector V52 (hereinafter, sometimes abbreviated as "second hostile condition vector V52") that is a condition vector based on pre-training frame feature #52, which is a feature of each of a plurality of frame images constituting the pre-training video #52 included in the video text data set #52 including the pair of the pre-training video #52 and the pre-training video description #52. Next, the model generation unit 134 may retrain the classifier network D1 so that, when the second adversarial condition vector V52 and the pre-learning video description #52 are input as input information to the classifier network D1, the classifier network D1 outputs information (e.g., the number "1") indicating that the pre-learning video description #52 is a true video description and is a video description corresponding to the second adversarial condition vector V52 as output information. In this manner, the model generation unit 134 may generate the first additionally trained or second additionally trained classifier network D1.

[0091] Furthermore, the sentence generation unit 136 may generate an adversarial condition vector V53, which is a condition vector based on the target frame feature weighted by the target weight. Next, the model generation unit 134 may input the adversarial condition vector V53 and the noise vector N2 as input information to the generator network G1 to generate a target video description, which is a sentence that describes the contents of the target video. In this way, the sentence generation unit 136 may input the target frame feature weighted by the target weight to the first additionally trained or second additionally trained generator network G1 based on the target frame feature and the target weight to generate a target video description, which is a sentence that describes the contents of the target video.

[0092] 8. Modifications The process according to the embodiment described above may be implemented in various different forms other than the above embodiment.

[0093] 8-1. First Modified Example In the above-described embodiment, the sentence generation model is a conditional generative adversarial network including a generator network and a classifier network. In the first modified example, the sentence generation model is a conditional variational autoencoder including an encoder and a decoder.

[0094] Fig. 9 is a diagram showing a conditional variational autoencoder (CVAE) which is an example of a sentence generation model according to the first modified example. As shown in Fig. 9, the sentence generation model may be a conditional variational autoencoder including an encoder EN1 and a decoder DE1. In Fig. 9, the pre-learning described in Fig. 2 will be described.

[0095] First, the model generation unit 134 may train a conditional variational autoencoder including the encoder EN1 and the decoder DE1 by the pre-learning described in FIG. 2. For example, the model generation unit 134 may generate a pre-learning variational condition vector V60 (hereinafter, may be abbreviated as "variational condition vector V60") that is a conditional vector based on the pre-learning frame feature #60. Next, the model generation unit 134 may train the encoder EN1 so that, when the variational condition vector V60 and the pre-learning video description #60 are input to the encoder EN1 as input information, the encoder EN1 outputs a mean vector μ and a variance vector σ in a multivariate normal distribution as output information. In addition, the model generation unit 134 may determine a latent vector z that is a sample that follows a multivariate normal distribution based on the mean vector μ and the variance vector σ. The model generation unit 134 may introduce a random variable ε obtained by randomly sampling from a standard normal distribution and use this to determine the latent vector z. Next, the model generation unit 134 may train the decoder DE1 so that when the latent vector z and the variation condition vector V60 are input to the decoder DE1 as input information, the decoder DE1 outputs a video description (pre-learning video description #60 in FIG. 9) having a feature corresponding to the variation condition vector V60 as output information. For example, the model generation unit 134 may train the encoder EN1 and the decoder DE1 using backpropagation (error backpropagation method) or the like so that the error between the pre-learning video description #60 input to the encoder EN1 and the video description output from the decoder DE1 is reduced. In this way, the model generation unit 134 may generate a pre-trained decoder DE1 that has been trained in advance to generate a video description having a feature corresponding to the pre-learning frame feature #60 based on the pre-learning frame feature #60.

[0096] In addition, the model generation unit 134 may re-train the pre-trained decoder DE1 (hereinafter sometimes abbreviated as "decoder DE1") by the first additional learning described in FIG. 3 or the second additional learning described in FIG. 5.

[0097] For example, the model generation unit 134 may determine a latent vector z, which is a sample that follows a multivariate normal distribution based on the mean vector μ and variance vector σ output from the encoder EN1. The model generation unit 134 may also generate a training variation condition vector V61 (hereinafter, may be abbreviated as "variation condition vector V61"), which is a condition vector based on training frame features weighted by training weights. Next, the model generation unit 134 may retrain the decoder DE1 so that, when the latent vector z and the variation condition vector V61 are input to the decoder DE1 as input information, the decoder DE1 outputs a video description having a feature corresponding to the variation condition vector V61 as output information. For example, the model generation unit 134 may retrain the decoder DE1 using backpropagation (error backpropagation method) or the like so as to reduce an error between the training video description #61 output from the decoder DE1 and the image description #61 (image description corresponding to the original image #61) included in the image description dataset #61. In this manner, the model generation unit 134 may retrain the decoder DE1 so as to generate, based on the weighted training frame features, video description text having features corresponding to the weighted training frame features. In this manner, the model generation unit 134 may generate the first additionally trained or second additionally trained decoder DE1.

[0098] Furthermore, the sentence generation unit 136 may generate a variation condition vector V62, which is a condition vector based on the target frame feature weighted by the target weight. Next, the model generation unit 134 may input the latent vector z and the variation condition vector V62 as input information to the decoder DE1 to generate a target moving image description, which is a sentence that describes the contents of the target moving image. In this way, the sentence generation unit 136 may input the target frame feature weighted by the target weight to the first additionally trained or second additionally trained decoder DE1 based on the target frame feature and the target weight to generate a target moving image description, which is a sentence that describes the contents of the target moving image.

[0099] 8-2. Second Modified Example In the above-described embodiment, the sentence generation model is a conditional generative adversarial network including a generator network and a discriminator network. In the first modified example, the sentence generation model is a conditional variational autoencoder including an encoder and a decoder. In the second modified example, the sentence generation model is a conditional diffusion model.

[0100] Fig. 10 is a diagram showing a conditional diffusion model, which is an example of a sentence generation model according to the second modified example. As shown in Fig. 10, the sentence generation model may be a conditional diffusion model. In Fig. 10, the first additional learning described in Fig. 3 or the second additional learning described in Fig. 5 will be described.

[0101] FIG. 10 shows a plurality of noise-added video descriptions in which noise has been gradually added to an initial video description x0, as an example of data used in the learning process of the conditional diffusion model. The model generation unit 134 trains the conditional diffusion model using learning data including the plurality of noise-added video descriptions shown in FIG. 10. In FIG. 10, the initial video description x0 is at stage #0 in terms of noise addition. In other words, it is a video description to which no noise has been added. The model generation unit 134 gradually adds Gaussian noise to the initial video description x0, and finally generates pure Gaussian noise x T In the process of obtaining the video description x0, a small amount of noise is added to the initial video description x0. t-1 Generate a noisy video description x t-1 is the stage #t-1 in terms of noise addition. In other words, it is a moving image description to which t-1 stages of noise have been added to the initial moving image description x0. Next, the model generation unit 134 adds noise-added moving image description x t-1 Video description with noise x with minute noise added t Generate a noisy video description x t is a video description with noise x t-1 This is a video description with one more level of noise added to it. tThe noise-adding stage is #t. That is, the video description is a video description with t stages of noise added to the initial video description x0. For example, t is a video description with noise x t-1 The noise addition process adds noise to the q(x t |x t-1 ) is a video description with noise x t-1 Video description with noise from x t The transition probability for transitioning to is shown.

[0102] Next, the model generator 134 generates a pure Gaussian noise x T In the process of gradually removing Gaussian noise from x0, we finally obtain the video description x0 without noise (the de-diffusion process). t Remove minute noise from the noisy video description x t-1 For example, the model generation unit 134 trains a conditional diffusion model that generates a noise-added video description x t The input is the noisy video description x t Video description with noise removed by one step from x t-1 The conditional diffusion model is trained to output p θ (x t-1 |x t ,V71) is a video description with noise x t-1 Video description with noise from x t The transition probability of transitioning to p θ (x t-1 |x t , V71) is the output of a neural network with a parameter θ determined by learning. In this way, the model generation unit 134 generates a pure Gaussian noise x T We train a conditional diffusion model, a machine learning model that takes the above as input, gradually removes noise, and finally generates a video description x0 without added noise.

[0103] First, the model generation unit 134 may train a conditional diffusion model by the pre-learning described in FIG. 2. For example, the model generation unit 134 may generate a pre-learning diffusion condition vector V70 (hereinafter, may be abbreviated as "diffusion condition vector V70") that is a condition vector based on the pre-learning frame feature #70. Next, the model generation unit 134 may train the conditional diffusion model so that, when the diffusion condition vector V70 is input as input information to the conditional diffusion model, a mean vector and a variance vector in a multivariate normal distribution to which the noise vector follows are output as output information. For example, the model generation unit 134 may train the conditional diffusion model so that, when the diffusion condition vector V70 and a pure Gaussian noise vector are input as input information to the conditional diffusion model, a video description having a feature corresponding to the diffusion condition vector V70 is generated. For example, the model generation unit 134 may use backpropagation or the like to train the conditional diffusion model so as to reduce an error between a video description output from the conditional diffusion model and a pre-training video description #70 included in the video description dataset #70. In this manner, the model generation unit 134 may generate a pre-trained conditional diffusion model that is trained in advance to generate a video description having a feature corresponding to the pre-training frame feature #70, based on the pre-training frame feature #70.

[0104] In addition, the model generation unit 134 may retrain the pre-trained conditional diffusion model (hereinafter, may be abbreviated as "conditional diffusion model") by the first additional learning described in FIG. 3 or the second additional learning described in FIG. 5. For example, the model generation unit 134 may generate a learning diffusion condition vector V71 (hereinafter, may be abbreviated as "diffusion condition vector V71") which is a condition vector based on the learning frame feature weighted by the learning weight. Next, the model generation unit 134 may train the conditional diffusion model so that when the diffusion condition vector V71 which is a condition vector based on the learning frame feature weighted by the learning weight is input as input information to the conditional diffusion model, the model generation unit 134 outputs a mean vector and a variance vector in the multivariate normal distribution to which the noise vector follows as output information. For example, when the diffusion condition vector V71 and pure Gaussian noise are input as input information to the conditional diffusion model, the model generation unit 134 may retrain the conditional diffusion model so as to generate a video description having characteristics corresponding to the diffusion condition vector V71. For example, the model generation unit 134 may use backpropagation or the like to retrain the conditional diffusion model so that an error between the training video description #71 output from the conditional diffusion model and the image description #71 (image description corresponding to the original image #71) included in the image description dataset #71 is reduced. In this manner, the model generation unit 134 may retrain the conditional diffusion model so as to generate a video description having features corresponding to the weighted training frame features based on the weighted training frame features. In this manner, the model generation unit 134 may generate a first additionally trained or second additionally trained conditional diffusion model.

[0105] Further, the sentence generation unit 136 generates a video description by estimating a noise vector using a trained conditional diffusion model (hereinafter, sometimes abbreviated as "conditional diffusion model") and removing the noise vector from the noised video description feature. For example, the sentence generation unit 136 may generate a diffusion condition vector V72, which is a condition vector based on the target frame feature weighted by the target weight. Next, the model generation unit 134 may input the diffusion condition vector V72 and a pure Gaussian noise vector as input information to the conditional diffusion model to generate a target video description, which is a sentence that describes the contents of the target video. In this way, the sentence generation unit 136 may input the target frame feature weighted by the target weight to the first additionally trained or second additionally trained conditional diffusion model based on the target frame feature and the target weight to generate a target video description, which is a sentence that describes the contents of the target video.

[0106] [9. Effects] As described above, the information processing device 100 according to the embodiment includes the video generation unit 131, the extraction unit 132, the determination unit 133, and the model generation unit 134. The video generation unit 131 generates a learning video based on an image and sentence data set including a pair of a captured image and an image description that is a sentence that describes the content of the captured image. The extraction unit 132 extracts learning frame features that are features of each of a plurality of frame images that constitute the learning video. The determination unit 133 determines learning weights that are weights corresponding to each of a plurality of frame images that constitute the learning video. The model generation unit 134 generates a sentence generation model that is a machine learning model trained to generate a learning video description that is a sentence that describes the content of the learning video and has features corresponding to the learning frame features weighted by the learning weights, based on the learning frame features and the learning weights.

[0107] As a result, the information processing device 100 can control which part of the video (which frame image) to emphasize when generating a video description by using weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model. The information processing device 100 can also generate a variety of video descriptions according to the frame image of interest. The information processing device 100 can also generate a variety of video descriptions according to the frame image of interest, which can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure and promote innovation and promote innovation." The information processing device 100 can also use weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model, which can enable the time-series information of the video to be reflected in natural language generation.

[0108] Furthermore, the video generating unit 131 generates a learning video including captured images in frames from captured images included in the image sentence dataset using a first video generation model that is a machine learning model that generates videos from images. The determining unit 133 determines learning weights corresponding to each of the multiple frame images constituting the learning video such that the weight corresponding to the captured image is larger than the weights corresponding to the other frame images other than the captured image.

[0109] This enables the information processing device 100 to generate a video description that places more importance on the captured image that was the source of generating the video than on the other frame images, among the multiple frame images that constitute the video.

[0110] In addition, the video generation unit 131 generates a learning video from the image caption included in the image sentence dataset using a second video generation model, which is a machine learning model that generates a video from a sentence. The determination unit 133 sets information on the similarity between each of the multiple frame images constituting the learning video and the captured image as a learning weight corresponding to each of the multiple frame images.

[0111] This enables the information processing device 100 to generate a video description that places emphasis on frame images that have a high similarity to the captured image corresponding to the image description from which the video was generated, rather than frame images that have a low similarity to the captured image corresponding to the image description from which the video was generated.

[0112] The extraction unit 132 also extracts pre-training frame features, which are features of each of a plurality of frame images constituting the captured video included in a video text data set including a pair of a captured video and a video description that is a text that describes the content of the captured video. The model generation unit 134 generates a pre-trained text generation model, which is a machine learning model that has been trained in advance to generate a video description having features corresponding to the pre-training frame features based on the pre-training frame features, and generates a text generation model by re-training the pre-trained text generation model to generate a training video description that is a text that describes the content of the training video, which has features corresponding to the training frame features weighted by the training weight, based on the training frame features and the training weights.

[0113] This enables the information processing device 100 to generate a video description having features corresponding to the frame features.

[0114] The information processing device 100 further includes an acquisition unit 135 and a sentence generation unit 136. The acquisition unit 135 acquires a sentence generation model, which is a machine learning model trained to generate a learning video description, which is a sentence explaining the content of the learning video, having a feature corresponding to the learning frame feature weighted by the learning weight, based on a learning frame feature that is a feature of each of a plurality of frame images constituting the learning video generated based on an image and sentence dataset including a pair of a captured image and an image description that is a sentence explaining the content of the captured image, and a learning weight that is a weight corresponding to each of a plurality of frame images constituting the learning video. The extraction unit 132 extracts a target frame feature that is a feature of each of a plurality of frame images constituting a target video that is a video to be processed. The determination unit 133 determines a target weight that is a weight corresponding to each of a plurality of frame images such that the weight corresponding to a designated frame image designated by a user among the plurality of frame images constituting the target video is larger than the weight corresponding to other frame images other than the designated frame image. The sentence generation unit 136 inputs the target frame features weighted by the target weight to a sentence generation model based on the target frame features and the target weight, and generates a target moving image description sentence that is a sentence explaining the content of the target moving image.

[0115] As a result, the information processing device 100 can control which part of the video (which frame image) to emphasize when generating a video description by using weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model. The information processing device 100 can generate a variety of video descriptions according to the frame image of interest. The information processing device 100 can also generate a variety of video descriptions according to the frame image of interest, which can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure and promote innovation and promote innovation." The information processing device 100 can also use weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model, which can enable the time-series information of the video to be reflected in natural language generation.

[0116] The sentence generation model is a conditional adversarial generative network including a generator network and a classifier network, and includes a generator network trained to output a training video description as output information when a first adversarial condition vector, which is a condition vector based on training frame features weighted by training weights, and a noise vector are input as input information to the generator network, and a second adversarial condition vector, which is a condition vector based on pre-training frame features that are features of each of a plurality of frame images constituting a captured video included in a video text dataset including a captured video and a video description that is a text that explains the content of the captured video, and a noise vector, which is a condition vector based on pre-training frame features that are features of each of a plurality of frame images constituting a captured video included in a video text dataset including a captured video and a video description that is a text that explains the content of the captured video, A machine learning model including: a classifier network trained to output, as output information, information indicating that the video description is a true video description and that it corresponds to the second adversarial condition vector when a video description is input as input information to the classifier network; and a classifier network trained to output, as output information, information other than information indicating that the training video description is a true video description and that it corresponds to the first adversarial condition vector when a first adversarial condition vector and a training video description, which is a false video description generated by the generator network, are input as input information to the classifier network.

[0117] This enables the information processing device 100 to generate a variety of video description sentences according to a frame image of interest, using a conditional generative adversarial network.

[0118] In addition, the sentence generation model is a conditional variational autoencoder including an encoder and a decoder, and is a machine learning model that includes an encoder that is trained to output a mean vector and a variance vector in a multivariate normal distribution as output information when a first variational condition vector, which is a condition vector based on pre-training frame features that are features of each of a plurality of frame images that constitute a captured video included in a video text dataset that includes a pair of a captured video and a video description, which is a text that explains the content of the captured video, and the video description are input as input information to the encoder, and a decoder that is trained to output the training video description as output information when a latent vector, which is a sample that follows a multivariate normal distribution based on the mean vector and the variance vector, and a second variational condition vector, which is a condition vector based on the training frame features weighted by the training weight, are input as input information to the decoder.

[0119] This enables the information processing device 100 to generate a variety of video description sentences according to a frame image of interest, using a conditional variational autoencoder.

[0120] In addition, the sentence generation model is a conditional diffusion model, and is a machine learning model that estimates the noise vector using the conditional diffusion model that has been trained to output, as output information, a mean vector and a variance vector in the multivariate normal distribution that the noise vector follows when a noisy video description, which is a training video description including a noise vector, and a diffusion condition vector, which is a condition vector based on training frame features weighted by training weights, are input into the conditional diffusion model as input information, and the noise vector is generated by removing the noise vector from the noisy video description.

[0121] This allows the information processing device 100 to generate a variety of video description texts according to a frame image of interest using the conditional diffusion model.

[0122] [10. Hardware Configuration] Moreover, the information processing device 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 11, for example. Fig. 11 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0123] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.

[0124] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0125] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0126] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0127] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0128] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.

[0129] [11. Others] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0130] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0131] For example, in the above-mentioned embodiment, the case where the information processing device 100 has the moving image generating unit 131, the extracting unit 132, the determining unit 133, the model generating unit 134, the acquiring unit 135, and the sentence generating unit 136 as functional units has been described, but each unit can be configured to be distributed in a separate device. For example, the information processing device 100 can have the acquiring unit 135 and the sentence generating unit 136 as functional units as an information processing device that generates sentences. In addition, an information processing device other than the information processing device 100 (hereinafter, referred to as a "generating device") can have the moving image generating unit 131, the extracting unit 132, the determining unit 133, and the model generating unit 134 as functional units as an information processing device that generates a sentence generation model. At this time, the information processing device 100 and the generating device may be connected to various networks by wire or wirelessly, and may transmit and receive information to each other. For example, the information processing device 100 may receive information about the sentence generation model generated by the generating device from the generating device.

[0132] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0133] 100 Information processing device 110 Communications Department 120 Storage section 130 Control section 131 Video Generation Unit 132 Extraction part 133 Decision Section 134 Model Generation Unit 135 Acquisition Department 136 Sentence generation section

Claims

1. A video generation unit that generates learning videos based on an image and text data set including a pair of a captured image and an image description text that explains the content of the captured image; An extraction unit that extracts learning frame features that are features of each of a plurality of frame images that constitute the learning video; A determination unit that determines learning weights corresponding to each of the plurality of frame images constituting the learning video such that a weight corresponding to the captured image is greater than weights corresponding to other frame images other than the captured image; and a model generation unit that generates a sentence generation model, which is a machine learning model trained to generate a training video description, which is a sentence explaining the content of the training video, based on the training frame features and the training weights, the training video description having features corresponding to the training frame features weighted by the training weights; An information processing device comprising:

2. The video generation unit is generating the learning video including the captured image in a frame from the captured image included in the image-sentence dataset using a first video generation model that is a machine learning model that generates a video from an image; The information processing device according to claim 1 .

3. The video generation unit is Using a second video generation model that is a machine learning model that generates videos from text, the training video is generated from the image captions included in the image and text dataset; The determination unit is information regarding a similarity between each of a plurality of frame images constituting the learning video and the captured image is set as the learning weight corresponding to each of the plurality of frame images; The information processing device according to claim 1 .

4. The extraction unit is Extracting pre-learning frame features which are features of each of a plurality of frame images constituting the captured video included in a video text data set including a pair of a captured video and a video description text which is a text explaining the content of the captured video; The model generation unit Based on the pre-training frame features, a pre-trained sentence generation model is generated, which is a machine learning model that has been pre-trained to generate the video description having features corresponding to the pre-training frame features, and based on the training frame features and the training weights, a training video description is a sentence that explains the content of the training video, and the training video description has features corresponding to the training frame features weighted by the training weights, thereby generating the sentence generation model. The information processing device according to claim 1 .

5. an acquisition unit that acquires a sentence generation model, which is a machine learning model trained to generate training video descriptions, which are sentences explaining the content of a training video, based on training frame features, which are features of each of a plurality of frame images constituting a training video generated based on an image and text dataset including a pair of a captured image and an image description, which is a sentence explaining the content of the captured image, and training weights, which are weights corresponding to each of a plurality of frame images constituting the training video, and which has features corresponding to the training frame features weighted by the training weights; an extraction unit that extracts target frame features that are features of each of a plurality of frame images that constitute a target moving image that is a moving image to be processed; a determination unit that determines target weights, which are weights corresponding to each of a plurality of frame images constituting the target moving image, such that a weight corresponding to a designated frame image designated by a user is made larger than weights corresponding to other frame images other than the designated frame image; A sentence generation unit that inputs the target frame feature weighted by the target weight to the sentence generation model based on the target frame feature and the target weight, and generates a target video description sentence that is a sentence that explains the content of the target video; An information processing device comprising:

6. The sentence generation model is a conditional generative adversarial network including a generator network and a classifier network; The generator network is trained to output the training video description as output information when a first adversarial condition vector, which is a condition vector based on the training frame features weighted by the training weights, and a noise vector are input to the generator network as input information; The classifier network is trained to output, as output information, information indicating that the video description is a true video description and corresponds to the second adversarial condition vector, when a second adversarial condition vector, which is a condition vector based on pre-learning frame features that are features of each of a plurality of frame images constituting the captured video included in a video text dataset including a pair of a captured video and a video description that is a text that explains the content of the captured video, and the video description are input as input information to the classifier network, and the classifier network is trained to output, as output information, information indicating that the video description is a true video description and corresponds to the second adversarial condition vector, when the first adversarial condition vector and the training video description, which is a false video description generated by the generator network, are input as input information to the classifier network, 6. The information processing device according to claim 1 or 5.

7. the sentence generation model is a conditional variational autoencoder including an encoder and a decoder; The encoder is trained to output a mean vector and a variance vector in a multivariate normal distribution as output information when a first variational condition vector, which is a condition vector based on pre-learning frame features that are features of each of a plurality of frame images constituting the captured video included in a video text data set including a pair of a captured video and a video description that is a text that explains the content of the captured video, and the video description are input as input information to the encoder; a decoder that is trained to output the training video description as output information when a latent vector that is a sample following the multivariate normal distribution based on the mean vector and the variance vector, and a second variational condition vector that is a condition vector based on the training frame features weighted by the training weights are input to the decoder as input information.

6. The information processing device according to claim 1 or 5.

8. The sentence generation model is a conditional diffusion model, a machine learning model that generates the training video description by estimating the noise vector using the conditional diffusion model that has been trained to output, as output information, a mean vector and a variance vector in a multivariate normal distribution that the noise vector follows when a noisy video description that is the training video description including a noise vector and a diffusion condition vector that is a condition vector based on the training frame features weighted by the training weight are input as input information to the conditional diffusion model, and removing the noise vector from the noisy video description; 6. The information processing device according to claim 1 or 5.

9. An information processing method implemented by a program executed by an information processing device, comprising: A video generation process for generating learning videos based on an image and text data set including a pair of a captured image and an image description text that explains the content of the captured image; An extraction step of extracting learning frame features which are features of each of a plurality of frame images constituting the learning video; A determination step of determining learning weights corresponding to each of the plurality of frame images constituting the learning video such that the weight corresponding to the captured image is greater than the weights corresponding to other frame images other than the captured image; a model generation process for generating a sentence generation model, which is a machine learning model trained to generate a training video description, which is a sentence explaining the content of the training video, based on the training frame features and the training weights, the training video description having features corresponding to the training frame features weighted by the training weights; An information processing method comprising:

10. An information processing method implemented by a program executed by an information processing device, comprising: an acquisition process for acquiring a sentence generation model, which is a machine learning model trained to generate training video description, which is a sentence explaining the content of the training video, based on training frame features, which are features of each of a plurality of frame images constituting the training video, generated based on an image and text dataset including a pair of a captured image and an image description, which is a sentence explaining the content of the captured image, and training weights, which are weights corresponding to each of a plurality of frame images constituting the training video, and which has features corresponding to the training frame features weighted by the training weights; an extraction step of extracting target frame features which are features of each of a plurality of frame images constituting a target moving image which is a moving image to be processed; a determining step of determining target weights, which are weights corresponding to each of a plurality of frame images constituting the target moving image, such that a weight corresponding to a designated frame image designated by a user is made larger than weights corresponding to other frame images other than the designated frame image; A sentence generation process of inputting the target frame feature weighted by the target weight to the sentence generation model based on the target frame feature and the target weight to generate a target video description sentence which is a sentence explaining the content of the target video; An information processing method comprising:

Images