Information processing device and information processing method

Abstract

To enable creation of an explanatory text for a motion picture which reflects time-course change of an action in each of scenes of the motion picture.SOLUTION: An information processing apparatus according to the present application has an estimation unit for estimating movement information relating to a movement of an object included in one of frame images forming a target motion picture as a motion picture to be processed, and a sentence creating unit for, based on a frame image succeeding the one frame image and the movement information, creating an explanatory sentence explaining contents of the succeeding frame image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Conventionally, there is known a technique for generating a caption for a video (also called a video caption, hereinafter referred to as a "video description") from the video. For example, a technique is known in which a video captured by a surveillance camera is input to a multi-layer neural network that outputs elements contained in an image as words, and a description of the video is generated. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-101317 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology simply involves inputting video footage captured by a surveillance camera into a multi-layer neural network that outputs elements contained in the image as words, and then generating a description of the video. Therefore, it is not necessarily possible to generate a video description that reflects changes in action over time in each scene of the video.

[0005] An object of the present application is to provide an information processing device and an information processing method that are capable of generating a video description that reflects temporal changes in the action in each scene of the video. [Means for solving the problem]

[0006] The information processing device of the present application comprises an estimation unit that estimates motion information regarding the motion of an object contained in a frame image that constitutes a target moving image, which is a moving image to be processed, and a sentence generation unit that generates a next image description sentence, which is a sentence explaining the content of the next frame image, based on the next frame image of the one frame image and the motion information.

[0007] In addition, the information processing device of the present application includes an estimation unit that estimates motion information regarding the motion of an object contained in a frame image that constitutes a learning video, and a model generation unit that generates a first sentence generation model, which is a machine learning model trained to output, as output information, a next image description sentence, which is a sentence that explains the content of the next frame image, when a next frame image of the one frame image and the motion information are input as input information.

[0008] The information processing device of the present application also includes an estimation unit that estimates motion information regarding the motion of an object contained in a frame image that constitutes a training video, the encoder that has been trained to output, as output information, an image feature corresponding to the next frame image when the next frame image of the first frame image is input to the encoder as input information, and the decoder that has been trained to weight, when the image feature and a special character string indicating the beginning of a sentence are input to the decoder as input information, a conditional probability distribution regarding the occurrence probability of each character string sequentially output by the decoder so as to increase the occurrence probability of an action character string corresponding to the motion information, and output, as output information, a next image description sentence that is a sentence explaining the content of the next frame image. Effect of the Invention

[0009] According to one aspect of the embodiment, it is possible to generate a video description that reflects the temporal changes in the action in each scene of the video. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment. [Diagram 2] FIG. 2 is a flowchart illustrating an example of a generation process of a sentence generation model according to the embodiment. [Diagram 3] FIG. 3 is a diagram for explaining an example of the motion information estimation process according to the embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of a process for generating a video description according to the embodiment. [Diagram 5] FIG. 5 is a diagram for explaining an example of an image caption generation process according to the comparative technique. [Figure 6] FIG. 6 is a diagram for explaining an example of the image caption generation process according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of a generation process of a sentence generation model according to a modified example. [Figure 8] FIG. 8 is a diagram for explaining an example of a generation process of a sentence generation model according to a modified example. [Figure 9] FIG. 9 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a detailed description will be given of an information processing device and an information processing method according to the present application (hereinafter, referred to as an "embodiment") with reference to the drawings. Note that the information processing device and the information processing method according to the present application are not limited to the embodiment. In addition, the same components in the following embodiments are given the same reference numerals, and duplicated descriptions are omitted.

[0012] (Embodiment) 1. Introduction In recent years, in research into AI (Artificial Intelligence) including machine learning and its applications in industry, the size of AI models (also called machine learning models; hereafter referred to as "machine learning models") has been growing, and it is assumed that large amounts of training data are required to train machine learning models.

[0013] However, it is known that the number of training data (specifically, a dataset including a pair of a video and a video description, hereinafter referred to as a "video description dataset") required for training a machine learning model that generates a video description, which is a sentence that explains the contents of a video, from a video, is still relatively small. For example, it is known that the number of video description datasets is still small compared to the number of training data (specifically, a dataset including a pair of an image and a label corresponding to the image, hereinafter referred to as an "image classification dataset") required for training a machine learning model that classifies images into categories corresponding to the images. It is also known that the number of video description datasets is still small compared to the number of training data (specifically, a dataset including a pair of an image and an image description, hereinafter referred to as a "image description dataset") required for training a machine learning model that generates an image description, which is a sentence that explains the contents of the image, from an image.

[0014] In contrast, it is known that there is a large amount of video data that is not paired with video descriptions (video data without video descriptions) compared to video text datasets, image classification datasets, image text datasets, etc.

[0015] Therefore, the information processing device according to this embodiment uses data of a video that is not paired with a video description (hereinafter, referred to as "video") to generate a pseudo video description from the video. This makes it possible for the information processing device according to this embodiment to generate a large amount of data set (hereinafter, referred to as "pseudo video description data set") including a pair of a video and a pseudo video description. Furthermore, the information processing device according to this embodiment can improve the estimation accuracy of a machine learning model that generates a video description from a video by using a large amount of pseudo video description data set as learning data for the machine learning model that generates a video description from a video.

[0016] Specifically, the information processing device according to this embodiment estimates motion information related to the motion (also called action) of an object included in each frame image constituting a video, and generates an image caption, which is a sentence explaining the content of the frame image following each frame image, based on the motion information of each frame image. This makes it possible for the information processing device according to this embodiment to reflect motion information related to the motion of an object included in the previous frame image in the image caption of each frame image. In other words, the information processing device according to this embodiment can generate an image caption of each frame image that reflects the temporal change of the action in each scene of the video.

[0017] Furthermore, the information processing device according to this embodiment generates a summary that summarizes the image caption corresponding to each frame image that constitutes the video, and generates a pseudo video caption (hereinafter, may be abbreviated as "video caption") based on the summary. In this way, the information processing device according to this embodiment generates a video caption based on the image caption of each frame image that reflects action information regarding the action of the object contained in the previous frame image. In other words, the information processing device according to this embodiment generates a video caption based on the image caption of each frame image that reflects the temporal change in the action in each scene of the video. Therefore, the information processing device according to this embodiment can generate a video caption that reflects the temporal change in the action in each scene of the video.

[0018] 2. Configuration of Information Processing Device An example of the configuration of an information processing device 100 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment. The information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0019] (Communication unit 110) The communication unit 110 is realized by a NIC (Network Interface Card), an antenna, etc. The communication unit 110 is connected to various networks by wire or wirelessly, and transmits and receives information to and from other information processing devices other than the information processing device 100, for example.

[0020] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various data. For example, the storage unit 120 stores information on a learning video, which is a video used for learning a machine learning model. The storage unit 120 also stores various programs. For example, the storage unit 120 stores information on a machine learning model generated by the model generation unit 133. The storage unit 120 also stores information on a voice-to-text conversion model, which is a machine learning model that generates character data corresponding to voice data from voice data.

[0021] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0022] The control unit 130 has an acquisition unit 131, an estimation unit 132, a model generation unit 133, and a sentence generation unit 134 as functional units, and may realize or execute the information processing action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 1, and may be other configurations as long as they perform the information processing described below. Also, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

[0023] (Acquisition part 131) The acquisition unit 131 may acquire various information. For example, the acquisition unit 131 may acquire a learning video that is a video used for learning a machine learning model. The acquisition unit 131 may also acquire a target video that is a video to be processed.

[0024] (Estimation part 132) The estimation unit 132 may estimate motion information related to the motion (also called action) of an object included in an image. Specifically, the estimation unit 132 may estimate motion information related to the motion of an object included in one frame image constituting a video. The estimation unit 132 may estimate motion information related to the motion of an object included in each frame image constituting a video. Here, the object included in the image may be any object that moves. For example, the object may be a person, an animal, or a moving object (for example, a vehicle, a ship, an airplane, etc.). Also, there may be multiple objects. For example, the object may be a second object that appears in the image together with the first object. For example, when the first object is a person, the second object may be a tool used by the person (for example, a cooking tool such as a frying pan, a bat used in sports, a ball, a racket, etc.), etc. Also, when the first object is a person, the second object may be a horse or a vehicle that the person is riding.

[0025] (Model generation unit 133) The model generation unit 133 may generate a sentence generation model trained to generate an image caption, which is a sentence that describes the contents of a predetermined frame, from a predetermined frame image that constitutes a video. Specifically, the model generation unit 133 may generate a sentence generation model trained to output an image caption, which is a sentence that describes the contents of a predetermined frame, as output information when a predetermined frame image that constitutes a video and motion information corresponding to the frame image immediately before the predetermined frame image are input as input information. In other words, the model generation unit 133 may generate a sentence generation model trained to output a next image caption, which is a sentence that describes the contents of the next frame image, as output information when a next frame image of a frame image that constitutes a video and motion information corresponding to the first frame image are input as input information. More specifically, the model generation unit 133 may generate a sentence generation model by retraining a trained machine learning model (hereinafter, referred to as an "existing sentence generation model") that has been trained to generate an image caption, which is a sentence that describes the contents of an image, from an image based on motion information estimated by the estimation unit 132.

[0026] For example, an existing text generation model may include an encoder that outputs image features indicating the characteristics of an image as output information when an image is input as input information. For example, the encoder may be a convolutional neural network (CNN). The encoder may be Residual Network (ResNet) (Kaiming He et al., 2015), AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan et al., 2014), GoogLeNet (Szegedy et al., 2014), Squeeze-and-Excitation Networks (SENet) (Jie Hu et al., 2018), EfficientNet (Tan et al., 2019), or ZFNet (Matthew et al., 2013), which are developed for object recognition. The encoder can also be Faster R-CNN (Shaoqing Ren et al., 2015), YOLO (You Look Only Once) (Joseph Redmon et al., 2015), or SSD (Single Shot MultiBox Detector) (Wei Liu., 2015), which are developed for object detection.

[0027] In addition, the existing text generation model may include a decoder that outputs, as output information, an image description that is a sentence that explains the content of the image input to the encoder when the image feature output from the encoder is input as input information. For example, the decoder may be a recurrent neural network (RNN), a gated recurrent unit (GRU), a long short term memory (LSTM), a Transformer (Ashish Vaswani et al., 2017), a Bidirectional Encoder Representations from Transformers (BERT) based on a Transformer, Generative Pre-Training3 (GPT-3), or a Text-to-Text Transfer Transformer (T5).

[0028] (Sentence generation section 134) The sentence generation unit 134 may generate a plurality of image captions that explain the contents of each of a plurality of frame images that constitute the target moving image, which is the moving image to be processed. Specifically, the sentence generation unit 134 may generate a plurality of image captions that explain the contents of each of a plurality of frame images that constitute the target moving image, using the sentence generation model generated by the model generation unit 133. In addition, the sentence generation unit 134 may generate target character data that corresponds to target audio data of the target moving image.

[0029] In addition, the text generating unit 134 may generate a target video description, which is a text that describes the contents of the target video. Here, the target video description corresponds to the above-mentioned pseudo video description. Specifically, the text generating unit 134 may generate, as the target video description, a summary that summarizes the target character data and a plurality of image descriptions that describe the contents of each of a plurality of frame images that constitute the target video.

[0030] 3. Information Processing Method From here, an example of information processing by the information processing device 100 according to the embodiment will be described with reference to FIGS.

[0031] An example of a generation process of a sentence generation model by the information processing device 100 will be described in detail with reference to Fig. 2. Fig. 2 is a flowchart showing an example of a generation process of a sentence generation model according to an embodiment. In Fig. 2, the acquisition unit 131 acquires a learning video, which is a video used for learning a machine learning model (step S101). Specifically, the acquisition unit 131 may acquire the learning video from the storage unit 120 that stores information related to the learning video.

[0032] The estimation unit 132 also estimates motion information related to the motion of an object included in one frame image constituting the learning video (step S102). Specifically, the estimation unit 132 inputs image features related to one frame image as input information to a classifier, and estimates the motion information based on a motion label corresponding to the image features and output as output information of the classifier.

[0033] An example of the estimation process of the motion information by the information processing device 100 will be described in detail with reference to FIG. 3. FIG. 3 is a diagram for explaining an example of the estimation process of the motion information according to the embodiment. The left side of FIG. 3 shows one frame image G11 constituting a learning video. One frame image G11 shown in FIG. 3 shows a person O1 holding a frying pan O2 in his hand frying vegetables. In FIG. 3, the estimation unit 132 may estimate motion information #11 related to the motion of the person O1, who is an object included in one frame image G11, using a known technique for estimating the motion (action) of an object captured in a still image (Reference literature; Paper title “Understanding action recognition in still images”, <Internet> https: / / openaccess.thecvf.com / content_CVPRW_2020 / papers / w23 / Girish_Understanding_Action_Recognition_in_Still_Images_CVPRW_2020_paper.pdf (searched on February 22, 2023)).

[0034] First, the estimation unit 132 may extract an area in which an object is captured from one frame image G11 using a known object detection technique. In FIG. 3, the estimation unit 132 may extract a first area R1 in which a person O1 is captured, a second area R2 in which a frying pan O2 is captured, an intersection area R3 in which a portion where the person O1 and the frying pan O2 are in contact, and a combined area R4 in which both the person O1 and the frying pan O2 are included from one frame image G11. In FIG. 3, each area extracted by the estimation unit 132 is shown by an area surrounded by a rectangular frame such as a bounding box. Specifically, the intersection area R3 may be an area where the first area R1 and the second area R2 overlap. Also, the combined area R4 may be an area that includes both the first area R1 and the second area R2.

[0035] Next, the estimation unit 132 may extract image features corresponding to each region from each extracted region using the above-mentioned CNN, Faster R-CNN, YOLO, or SSD. In FIG. 3, the estimation unit 132 may extract a first feature that is an image feature corresponding to the first region R1 from the first region R1. Also, the estimation unit 132 may extract a second feature that is an image feature corresponding to the second region R2 from the second region R2. Also, the estimation unit 132 may extract an intersection feature that is an image feature corresponding to the intersection region R3 from an intersection region R3 that is an area where the first region R1 and the second region R2 overlap. Also, the estimation unit 132 may extract a combined feature that is an image feature corresponding to the combined region R4 from a combined region R4 that is an area including both the first region R1 and the second region R2.

[0036] Next, the estimation unit 132 may estimate what features the extracted image feature represents based on the category of the object captured in each extracted region. In FIG. 3, the estimation unit 132 may estimate that the first feature is an image feature related to a "person" since the category of the object captured in the first region R1 is "person". Furthermore, the estimation unit 132 may estimate that the second feature is an image feature related to a frying pan "frying pan" since the category of the object captured in the second region R2 is "frying pan". Furthermore, the estimation unit 132 may estimate that the intersection feature is an image feature related to a "part where a person's hand is holding a frying pan" since the category of the object captured in the intersection region R3 is "part where a person's hand is holding a frying pan". In addition, since the categories of the objects appearing in combined region R4 are “person”, “frying pan”, “part of the person’s hand holding the frying pan”, and “vegetables”, the estimation unit 132 may estimate that the combined features are image features relating to “person”, “frying pan”, “part of the person’s hand holding the frying pan”, and “vegetables”.

[0037] Next, the estimation unit 132 may generate an enriched feature, which is a feature obtained by combining the image features extracted from each region. The estimation unit 132 may generate one feature vector corresponding to the enriched feature by linking feature vectors corresponding to each image feature. In FIG. 3, the estimation unit 132 may generate an enriched feature by linking the first feature, the second feature, the intersection feature, and the combination feature.

[0038] Next, the estimation unit 132 may acquire a classifier, which is a machine learning model previously trained to output an action label corresponding to the enriched feature when the enriched feature is input as input information. For example, the estimation unit 132 may acquire the classifier from the storage unit 120 that stores information about the classifier. Here, the action label may be a character string indicating the action of an object included in an image. For example, the action label may be a character string corresponding to a continuous verb, an instantaneous verb, a state verb, or a fourth type verb in the classification of Japanese verbs. A continuous verb is a verb indicating a type of action that continues for a certain period of time. For example, examples of continuous verbs include verbs that represent human actions such as eating, reading, speaking, singing, walking, writing, laughing, crying, and swimming, and verbs that represent natural phenomena such as scattering, falling, and blooming. An instantaneous verb is a verb that indicates an action that ends in an instant. For example, examples of instantaneous verbs include (lights) turning on, turning off, touching, reaching, deciding, finding, starting, ending, and arriving. Stative verbs are verbs that express a state. For example, some examples of stative verbs are aru (to be), iru (to be), etc. The fourth type of verb does not include the concept of time, and expresses the appearance, quality, shape, or impression of something, like an adjective. For example, some examples of fourth type verbs are niru (to resemble), (to be clear), (to have a sharp corner), and nasu wo te mo (to have a high nose).

[0039] Next, the estimation unit 132 may input the generated enriched feature to a classifier as input information, and estimate an action label corresponding to the enriched feature, which is output as output information of the classifier. Next, the estimation unit 132 may estimate the action information based on the action label. In FIG. 3, the estimation unit 132 may input the generated enriched feature to a classifier as input information, and estimate an action label L11 corresponding to the enriched feature, which is output as output information of the classifier. In FIG. 3, the action label L11 may be a character string corresponding to the verb "fry". Next, the estimation unit 132 may estimate the action information #11 based on the action label L11. For example, the estimation unit 132 may estimate the character string "fry" corresponding to the action label L11 as the action information #11.

[0040] In this manner, the estimation unit 132 inputs, as input information, enriched features generated based on a first feature related to a first region (first region R1 in FIG. 3) in which one object (person O1 in FIG. 3) is captured in one frame image (one frame image G11 in FIG. 3), a second feature related to a second region (second region R2 in FIG. 3) in which another object different from the one object (frying pan O2 in FIG. 3) is captured, an intersection feature related to an intersection region (intersection region R3 in FIG. 3) in which the first region and the second region overlap, and a combined feature related to a combined region (combined region R4 in FIG. 3) in which both the first region and the second region are included, into the classifier, and estimates action information (the character string "fry" corresponding to the action label L11 in FIG. 3) based on the action label corresponding to the enriched feature and output as output information of the classifier (action label L11 in FIG. 3).

[0041] In FIG. 3, the estimation unit 132 inputs the enriched feature to the classifier as input information, and estimates a character string corresponding to the action label output as output information of the classifier as action information. However, the input information input to the classifier by the estimation unit 132 is not limited to the enriched feature. For example, the estimation unit 132 may input the first feature, the second feature, the intersection feature, and the combination feature to the classifier as input information, and estimate a character string corresponding to the action label output as output information of the classifier. Furthermore, the estimation unit 132 may extract an image feature corresponding to one frame image from one frame image. Next, the estimation unit 132 may input the image feature corresponding to one frame image to the classifier as input information, and estimate a character string corresponding to the action label output as output information of the classifier.

[0042] In addition, in FIG. 3, a case has been described in which the image to be processed by the estimation unit 132 is one frame image G11 shown in FIG. 3, but the image to be processed by the estimation unit 132 is not limited to one frame image G11. For example, the image to be processed by the estimation unit 132 may be any frame image included in a plurality of frame images that constitute the learning video. In addition, the image to be processed by the estimation unit 132 may be a frame that constitutes a target moving image, which is a moving image to be processed, which will be described later. For example, the image to be processed by the estimation unit 132 may be any frame image included in a plurality of frame images that constitute the target moving image.

[0043] Also, in FIG. 3, the case where the action information #11 estimated by the estimation unit 132 is a character string corresponding to the verb "stir fry" corresponding to the action label L11 has been described, but the action information estimated by the estimation unit 132 is not limited to a character string corresponding to the verb "stir fry". For example, the action information may be any character string as long as it is a character string indicating the action of an object included in an image. For example, the action information may be any character string corresponding to the above-mentioned continuous verb, momentary verb, state verb, or fourth type verb.

[0044] In addition, the model generation unit 133 generates a sentence generation model M1, which is a machine learning model trained to output a next image description sentence, which is a sentence explaining the contents of the next frame image, as output information when the next frame image of one frame image constituting the learning video and the motion information (hereinafter, may be referred to as "motion information corresponding to one frame image") regarding the motion of the object included in the one frame image estimated by the estimation unit 132 are input as input information (step S103). In the example of FIG. 3, the model generation unit 133 may generate a sentence generation model M1, which is a machine learning model trained to output a next image description sentence CP12, which is a sentence explaining the contents of the next frame image G12, as output information when the next frame image G12 of one frame image G11 constituting the learning video and the motion information #11 (for example, the character string "stir-fry" corresponding to the motion label L11) corresponding to the one frame image G11 estimated by the estimation unit 132 are input as input information.

[0045] Specifically, the model generation unit 133 may generate the sentence generation model M1 by re-learning an existing sentence generation model, which is a machine learning model including an encoder E1 trained to output an image feature corresponding to an image as output information when an image is input to the encoder E1 as input information, and a decoder D1 trained to output an image description as output information when an image feature and a special character string indicating the beginning of a sentence are input to the decoder D1 as input information, based on the motion information estimated by the estimation unit 132. More specifically, the model generation unit 133 may re-learn the existing sentence generation model by adding the motion information estimated by the estimation unit 132 as input information of the existing sentence generation model including the encoder E1 and the decoder D1.

[0046] For example, the model generation unit 133 may generate a vector corresponding to the motion information based on the motion information estimated by the estimation unit 132. For example, the model generation unit 133 may generate a vector V11 corresponding to the motion information #11 corresponding to the one frame image G11 based on the motion information #11 corresponding to the one frame image G11 estimated by the estimation unit 132. For example, the model generation unit 133 may generate a vector V11 corresponding to the character string "fry" corresponding to the motion label L11 based on the character string "fry" corresponding to the motion label L11. Next, the model generation unit 133 may generate a concatenated vector V13 (=V11+V12) that is one concatenated vector by concatenating a vector V12 corresponding to the frame image G12 next to the one frame image G11 and the generated vector V11. Next, the model generation unit 133 may input the generated concatenated vector V13 as input information of the encoder E1 included in the existing sentence generation model, and generate a concatenated feature #13, which is a feature output as output information of the encoder E1 and corresponds to the concatenated vector V13. In this way, the model generation unit 133 may input a concatenated vector V13, which is a concatenated vector of the vector V12 corresponding to the next frame image G12 and the vector V11 corresponding to the motion information #11 corresponding to the one frame image G11, as input information of the encoder E1 included in the existing sentence generation model, and re-train the encoder E1 so as to output the concatenated feature #13, which is a feature corresponding to the concatenated vector V13, as output information. Hereinafter, the re-trained encoder E1 may be referred to as an encoder E1'.

[0047] Next, the model generation unit 133 may input the combined feature #13 output from the encoder E1' as input information of the decoder D1 included in the existing sentence generation model, and generate a next image description CP12 (for example, the sentence "stir-fry vegetables") which is a sentence output as output information of the decoder D1 and corresponds to the combined feature #13. In this way, the model generation unit 133 may input the combined feature output from the encoder E1' as input information of the decoder D1 included in the existing sentence generation model, and re-train the decoder D1 so as to output the next image description which is a sentence corresponding to the combined feature as output information. Hereinafter, the re-trained decoder D1 may be referred to as the decoder D1'. In this way, the model generation unit 133 may generate a sentence generation model M1 which is a machine learning model including the encoder E1' and the decoder D1'.

[0048] An example of a generation process of a video description by the information processing device 100 will be described in detail with reference to Figs. 4 to 6. Fig. 4 is a flowchart showing an example of a generation process of a video description according to an embodiment. In Fig. 4, the acquisition unit 131 acquires a target video which is a video to be processed (step S201). Specifically, the acquisition unit 131 may acquire the target video via the communication unit 110, for example, from an information processing device used by a user.

[0049] Furthermore, the sentence generation unit 134 generates a plurality of image captions that explain the contents of each of the plurality of frame images that constitute the target moving image (step S202). Specifically, the sentence generation unit 134 may use the sentence generation model M1 generated by the model generation unit 133 to generate a plurality of image captions that explain the contents of each of the plurality of frame images that constitute the target moving image.

[0050] Here, an example of the image description generation process according to the comparative technique will be described with reference to FIG. 5. FIG. 5 is a diagram for explaining an example of the image description generation process according to the comparative technique. In FIG. 5, a process of generating an image description from an image using an existing text generation model will be described. In FIG. 5, when a predetermined frame image G22 out of a plurality of frame images constituting a target video is input as input information to an existing text generation model, a text "grilling vegetables" is generated, which is an image description CP22' that explains the contents of the predetermined frame image G22. That is, the existing text generation model generates an image description corresponding to the moment when a person in the predetermined frame image G22 puts vegetables on a frying pan and grills them. In this way, in the comparative technique, the existing text generation model merely generates an image description corresponding to the action of an object appearing in each frame image at that moment from each frame image. Therefore, in the comparative technique, it was sometimes difficult to reflect the action information regarding the action of an object included in a previous frame image in an image description corresponding to a predetermined frame image.

[0051] Next, an example of the image description generation process by the information processing device 100 will be described with reference to FIG. 6. FIG. 6 is a diagram for describing an example of the image description generation process according to the embodiment. In FIG. 6, the estimation unit 132 estimates motion information #21 related to the motion of an object included in one frame image G21 (not shown) constituting a target video, which is a video to be processed. For simplicity, in FIG. 6, a case will be described in which one frame image G21 constituting the target video is the same image as one frame image G11 constituting the learning video described in FIG. 3, but this is not limited thereto. At this time, the estimation unit 132 may estimate a character string "fry" corresponding to the motion label L21 as motion information #21 related to the motion of an object included in one frame image G21 constituting the target video (hereinafter, sometimes referred to as "motion information #21 corresponding to one frame image G21") in the same manner as in FIG. 3.

[0052] Also, in Figure 6, the sentence generation unit 134 may generate the sentence "Stir-fry vegetables", which is the next image description sentence CP22, which is a sentence explaining the content of the next frame image G22, based on the next frame image G22 of a frame image G21 that constitutes the target video and the character string "stir-fry", which is the action information #21 regarding the action of the object contained in the one frame image G21.

[0053] In this way, the sentence generation unit 134 generates a next image description, which is a sentence explaining the contents of the next frame image, based on the next frame image of a frame image that constitutes the target video and the movement information related to the movement of the target object contained in the one frame image.

[0054] Specifically, the sentence generation unit 134 may acquire the sentence generation model M1 from the storage unit 120 that stores information about the sentence generation model M1 generated by the model generation unit 133. Next, the sentence generation unit 134 may input the next frame image G22 of the one frame image G21 constituting the target video and the motion information #21 about the motion of the object included in the one frame image G21 as input information of the sentence generation model M1, and generate a next image description CP22 to be output as output information of the sentence generation model M1. More specifically, the sentence generation unit 134 may input the character string "fry" which is the motion information #21 corresponding to the next frame image G22 of the one frame image G21 constituting the target video and the one frame image G21 as input information of the sentence generation model M1, and generate a sentence "fry vegetables" which is the next image description CP22 to be output as output information of the sentence generation model M1.

[0055] In this way, the sentence generation unit 134 inputs the next frame image of a frame image constituting the target video and movement information regarding the movement of an object contained in the frame image as input information for the sentence generation model M1, which is a machine learning model that generates an image description, which is a sentence that explains the content of the image from the image, and generates a next image description that is output as output information of the sentence generation model M1.

[0056] For example, the sentence generation unit 134 may generate a vector corresponding to the motion information based on the motion information estimated by the estimation unit 132. In FIG. 6, the sentence generation unit 134 may generate a vector V21 corresponding to the motion information #21 corresponding to one frame image G21 based on the motion information #21 corresponding to one frame image G21 estimated by the estimation unit 132. For example, the sentence generation unit 134 may generate a vector V21 corresponding to the character string "fry" corresponding to the motion label L21 based on the character string "fry" corresponding to the motion label L21. Next, the sentence generation unit 134 may generate a concatenated vector V23 (=V21+V22) which is one concatenated vector by concatenating the vector V22 corresponding to the frame image G22 next to the one frame image G21 and the generated vector V21. Next, the sentence generation unit 134 may input the generated link vector V23 as input information of the encoder E1' included in the sentence generation model M1, and generate a link feature #23 that is a feature output as output information of the encoder E1' and corresponds to the link vector V23. Next, the sentence generation unit 134 may input the link feature #23 output from the encoder E1' as input information of the decoder D1' included in the sentence generation model M1, and generate a next image description sentence CP22 (for example, the sentence "Stir-fry vegetables") that is a sentence output as output information of the decoder D1' and corresponds to the link feature #23.

[0057] In this way, when generating an image caption CP22 that explains the contents of a predetermined frame image G22, the information processing device 100 uses the action information #21 (in FIG. 6, the character string "fry" corresponding to the action label L21) related to the action of the object included in the frame image G21 immediately before the predetermined frame image G22. This allows the information processing device 100 to reflect information related to the action performed in the immediately previous frame image G21 in the image caption CP22 corresponding to the predetermined frame image G22. In other words, the information processing device 100 can reflect the continuity of the action performed from the immediately previous frame image G21 to the predetermined frame image G22 in the image caption CP22 corresponding to the predetermined frame image G22. That is, the information processing device 100 can make it easier to reflect the action corresponding to the continuous verb performed from the immediately previous frame image G21 to the predetermined frame image G22 (in FIG. 6, the action corresponding to the continuous verb that a person is frying vegetables in a frying pan) in the image caption CP22 corresponding to the predetermined frame image G22.

[0058] Here, for the first frame image among the multiple frame images constituting the target video, there is no previous frame image, so there is no motion information corresponding to the previous frame image. Therefore, the sentence generation unit 134 may use an existing sentence generation model to generate an image description that describes the contents of the first frame image constituting the target video. Also, the sentence generation unit 134 may use the sentence generation model M1 generated by the model generation unit 133 in the same manner as in FIG. 6 to generate multiple image description sentences that describe the contents of the second and subsequent frame images among the multiple frame images constituting the target video. In this manner, the sentence generation unit 134 may generate multiple image description sentences that describe the contents of each of the multiple frame images constituting the target video.

[0059] Next, the sentence generation unit 134 generates target text data corresponding to the target voice data from the target voice data of the target video using a voice-to-text conversion model, which is a machine learning model that generates text data corresponding to the voice data from the voice data (step S203). Specifically, the voice-to-text conversion model may be Speech to Text (Speech2Text). For example, the sentence generation unit 134 may extract the target voice data from the target video. Next, the sentence generation unit 134 may generate target text data transcribed from the target voice data using the voice-to-text conversion model. Note that either the process of step S202 or the process of step S203 may be performed first, or may be performed simultaneously by the sentence generation unit 134.

[0060] Next, the text generation unit 134 generates a summary text, which is a text summarizing the multiple image captions and the target text data, based on the multiple image captions and the target text data, and generates a target video description text, which is a text explaining the contents of the target video, based on the summary text (step S204). Specifically, the text generation unit 134 may generate a summary text from the multiple image captions and the target text data using an existing text summary model such as an extraction type model or an abstraction type model, and use the generated summary text as the target video description text.

[0061] In this way, the sentence generation unit 134 generates a plurality of image descriptions explaining the contents of each of the plurality of frame images constituting the target moving image and target text data corresponding to the target audio data of the target moving image, generates a summary sentence which is a sentence summarizing the plurality of image descriptions and the target text data based on the plurality of image descriptions and the target text data, and generates a target moving image description which is a sentence explaining the contents of the target moving image based on the summary sentence.

[0062] 4. Modifications The processing according to the above-described embodiment may be implemented in various different forms other than the above embodiment.

[0063] In the above embodiment, the model generation unit 133 generates a sentence generation model M1 that is a machine learning model trained to output a next image description sentence that is a sentence that describes the contents of a next frame image as output information when the next frame image of a frame image constituting a learning video and the action information corresponding to the one frame image are input as input information. In the modified example, the model generation unit 133 generates a sentence generation model M2 that is a machine learning model including a decoder trained to weight a conditional probability distribution regarding the occurrence probability of each character string sequentially output by the decoder so as to increase the occurrence probability of an action character string corresponding to the action information corresponding to a frame image constituting a learning video, and to output a next image description sentence that is a sentence that describes the contents of a next frame image of the one frame image as output information.

[0064] Fig. 7 is a flowchart showing an example of a generation process of a sentence generation model according to a modified example. In Fig. 7, the acquisition unit 131 acquires a learning video, which is a video used for learning a machine learning model (step S301). Specifically, the acquisition unit 131 may acquire the learning video in the same manner as described in step S101 shown in Fig. 2.

[0065] Further, the estimation unit 132 estimates motion information related to the motion of an object included in one frame image constituting the learning video (step S302). Specifically, the estimation unit 132 may estimate the motion information in a manner similar to that of FIG. 3. In the example of FIG. 8 described later, the estimation unit 132 may estimate motion information #31 related to the motion of a person O1, which is an object included in one frame image G31 (not shown) constituting the learning video. Here, for simplicity, a case will be described in which the one frame image G31 is the same image as the one frame image G11 shown in FIG. 3, but this is not limited thereto. For example, the estimation unit 132 may estimate a motion label L31 to be output as output information of the classifier in a manner similar to that of FIG. 3. For example, the motion label L31 may be a character string corresponding to the verb "fry". Next, the estimation unit 132 may estimate the motion information #31 based on the motion label L31. For example, the estimation unit 132 may estimate a character string "fry" corresponding to the action label L31 as the action information #31. Hereinafter, a character string corresponding to the action information may be referred to as an "action character string." For example, the action character string #31 corresponding to the action information #31 is a character string "fry."

[0066] In addition, the model generation unit 133 weights the conditional probability distribution regarding the occurrence probability of each string sequentially output by the decoder so as to increase the occurrence probability of an action string corresponding to action information corresponding to a frame image constituting the training video, and generates a sentence generation model M2, which is a machine learning model including a decoder trained to output a next image description, which is a sentence explaining the content of the next frame image after the first frame image, as output information (step S303).

[0067] An example of the generation process of the sentence generation model according to the modified example will be described in detail with reference to FIG. 8. FIG. 8 is a diagram for explaining an example of the generation process of the sentence generation model according to the modified example. The left side of FIG. 8 shows a frame image G32 next to one frame image G31 (not shown) constituting a learning video. The next frame image G32 corresponds to input information input to the sentence generation model M2. The center of FIG. 8 shows the configuration of the sentence generation model M2. The sentence generation model M2 may include an encoder E2. The next frame image G32 input to the sentence generation model M2 may be input to the encoder E2. When the next frame image G32 is input as input information, the encoder E2 may output an image feature F32 corresponding to the next frame image G32 as output information. The sentence generation model M2 may include a decoder D2. FIG. 8 shows how the decoder D2 generates character strings one by one and inputs the generated character strings to the next step in an autoregressive manner. The decoder D2 extracts the image feature F32 output by the encoder E2 and a special character string indicating the beginning of a sentence (in FIG. 8, <s>" is input as input information, a next image description CP32, which is a sentence explaining the contents of the next frame image G32, may be output as output information. Also, the right side of Fig. 8 shows the next image description CP32, which is a sentence explaining the contents of the next frame image G32. The next image description CP32 corresponds to the output information output from the sentence generation model M2.

[0068] In FIG. 8, the model generation unit 133 is configured to output, as output information, an image feature F32 corresponding to a frame image G32 next to a frame image G31 constituting a learning video, when the frame image G32 next to the frame image G31 is input to the encoder E2 as input information, and to output, as output information, an image feature F32 corresponding to the next frame image G32 and a special character string indicating the beginning of a sentence, <s>" is input to the decoder D2 as input information, the decoder D2 is trained to weight each of the conditional probability distributions PD1 to PD3 regarding the probability of occurrence of each character string sequentially output by the decoder D2 so as to increase the probability of occurrence of the character string "stir-fry", which is action character string #31 corresponding to action information #31 regarding the action of an object included in one frame image G31, and to output as output information a next image description sentence CP32 which is a sentence explaining the content of the next frame image G32.

[0069] 8, for simplicity, a case will be described in which the conditional probability distribution regarding the occurrence probability of each character string sequentially output by the decoder D2 is a conditional probability distribution regarding the occurrence probability of four character strings, "he", "vegetables", "saute", and "wo". In reality, the conditional probability distribution regarding the occurrence probability of each character string sequentially output by the decoder D2 may be, for example, a conditional probability distribution regarding the occurrence probability of each of all character strings included in dictionary data (dictionary data regarding character string candidates output by the decoder D2) previously learned by the sentence generation model M2.

[0070] In FIG. 8, the model generation unit 133 generates the image feature F32 output by the encoder E2 and the special character string " <s>When " is input to the decoder D2 as input information, the conditional probability distribution PD1 regarding the occurrence probability of each of the four character strings "he", "vegetables", "stir fry", and "wo" output by the decoder D2 may be weighted so as to increase the occurrence probability of the character string "stir fry", which is the action character string #31 corresponding to the action information #31. FIG. 8 shows that the occurrence probability of the character string "stir fry" in the conditional probability distribution PD1 is the occurrence probability obtained by adding the occurrence probability P12 weighted by the model generation unit 133 to the original occurrence probability P11 of the character string "stir fry". Also, in FIG. 8, the model generation unit 133 may output the character string "vegetables" as the output information of the decoder D2 since the occurrence probability of the character string "vegetables" is the highest among the occurrence probabilities of the four character strings in the conditional probability distribution PD1.

[0071] Next, when the character string "vegetables" output by the decoder D2 in the previous step is input as input information to the decoder D2, the model generation unit 133 may weight the conditional probability distribution PD2 regarding the occurrence probability of each of the four character strings "he", "vegetables", "stir fry", and "wo" output by the decoder D2 so as to increase the occurrence probability of the character string "stir fry", which is the action character string #31 corresponding to the action information #31. FIG. 8 shows that the occurrence probability of the character string "stir fry" in the conditional probability distribution PD2 is an occurrence probability obtained by adding the occurrence probability P22 weighted by the model generation unit 133 to the original occurrence probability P21 of the character string "stir fry". Also, in FIG. 8, the model generation unit 133 may output the character string "wo" as the output information of the decoder D2 since the occurrence probability of the character string "wo" is the highest among the occurrence probabilities of the four character strings in the conditional probability distribution PD2.

[0072] Next, when the character string "wo" output by the decoder D2 in the previous step is input as input information to the decoder D2, the model generation unit 133 may weight the conditional probability distribution PD3 regarding the occurrence probability of each of the four character strings "he", "vegetables", "stir fry", and "wo" output by the decoder D2 so as to increase the occurrence probability of the character string "stir fry", which is the action character string #31 corresponding to the action information #31. FIG. 8 shows that the occurrence probability of the character string "stir fry" in the conditional probability distribution PD3 is an occurrence probability obtained by adding the occurrence probability P32 weighted by the model generation unit 133 to the original occurrence probability P31 of the character string "stir fry". Also, in FIG. 8, the model generation unit 133 may output the character string "stir fry" as the output information of the decoder D2 since the occurrence probability of the character string "stir fry" is the highest among the occurrence probabilities of the four character strings in the conditional probability distribution PD3.

[0073] Next, the model generation unit 133 may output the next image description sentence CP32, which is the sentence "Stir-fry the vegetables", as output information based on the string "vegetables" which is output first from the decoder D2, the string "wo" which is output second from the decoder D2, and the string "stir-fry" which is output last from the decoder D2.

[0074] In this way, the model generation unit 133 generates a sentence generation model M2, which is a machine learning model that includes an encoder that has been trained to output image features corresponding to the next frame image as output information when a frame image next to one frame image is input to the encoder as input information, and a decoder that has been trained to weight a conditional probability distribution regarding the occurrence probability of each string output sequentially by the decoder when the image features and a special string indicating the beginning of a sentence are input to the decoder as input information, so as to increase the occurrence probability of an action string corresponding to action information, and to output a next image description, which is a sentence that explains the contents of the next frame image, as output information.

[0075] The estimation unit 132 also estimates motion information #41 relating to the motion of an object included in one frame image G41 (not shown) constituting the target moving image, which is the moving image to be processed. For example, the estimation unit 132 may estimate a character string corresponding to the motion label L41 as motion information #41 relating to the motion of an object included in one frame image G41 constituting the target moving image (hereinafter, may be referred to as "motion information #41 corresponding to one frame image G41") in the same manner as in FIG. 3.

[0076] The sentence generation unit 134 may acquire the sentence generation model M2 from the storage unit 120 that stores information about the sentence generation model M2 generated by the model generation unit 133. Next, the sentence generation unit 134 may input a frame image G42 next to one frame image G41 constituting the target moving image as input information of the encoder E2 included in the sentence generation model M2, generate an image feature F42 to be output as output information of the encoder E2, input the image feature F42 and a special character string indicating the beginning of a sentence as input information of the decoder D2 included in the sentence generation model M2, weight the conditional probability distribution regarding the occurrence probability of each character string sequentially output by the decoder D2 so as to increase the occurrence probability of the action character string #41 corresponding to the action information #41, and generate a next image description CP42 to be output as output information of the decoder D2.

[0077] In this way, when generating an image description CP42 that describes the contents of a specific frame image G42, the information processing device 100 weights the conditional probability distribution so as to increase the appearance probability of the action character string #41 corresponding to the action information #41 related to the action of the object included in the frame image G41 one frame before the specific frame image G42. This makes it possible for the information processing device 100 to generate an image description CP42 in which an action character string corresponding to the action performed in the previous frame image G41 is likely to appear. In other words, the information processing device 100 can reflect the continuity of the action performed from the previous frame image G41 to the specific frame image G42 in the image description CP42 corresponding to the specific frame image G42. That is, the information processing device 100 can reflect the action corresponding to the continuous verb performed from the previous frame image G41 to the specific frame image G42 in the image description CP42 corresponding to the specific frame image G42.

[0078] In this way, the sentence generation unit 134 inputs the next frame image as input information for the encoder included in the sentence generation model M2, which is a machine learning model that generates an image description, which is a sentence that explains the contents of an image from an image, generates image features that are output as output information of the encoder, inputs the image features and a special string indicating the beginning of a sentence as input information for the decoder included in the sentence generation model M2, weights the conditional probability distribution regarding the occurrence probability of each string sequentially output by the decoder so as to increase the occurrence probability of the action string corresponding to the action information, and generates the next image description that is output as output information of the decoder.

[0079] 5. Effects As described above, the information processing device 100 according to the embodiment includes the estimation unit 132 and the sentence generation unit 134. The estimation unit 132 estimates motion information related to the motion of a target object included in one frame image constituting a target moving image, which is a moving image to be processed. The sentence generation unit 134 generates a next image description sentence, which is a sentence explaining the content of the next frame image, based on the next frame image of the one frame image and the motion information.

[0080] Thereby, the information processing device 100 can reflect the motion information on the motion of the object included in the previous frame image in the image description of each frame image. That is, the information processing device 100 can generate an image description of each frame image that reflects the temporal change of the action in each scene of the video. Also, the information processing device 100 can generate a video description based on the image description of each frame image that reflects the motion information on the motion of the object included in the previous frame image. That is, the information processing device 100 can generate a video description that reflects the temporal change of the action in each scene of the video based on the image description of each frame image that reflects the temporal change of the action in each scene of the video. Therefore, the information processing device 100 can generate a video description that reflects the temporal change of the action in each scene of the video. Also, the information processing device 100 can generate a video description that reflects the temporal change of the action in each scene of the video, which can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilience and promote innovation and foster innovation."

[0081] In addition, the sentence generation unit 134 inputs the next frame image and action information as input information for a first sentence generation model, which is a machine learning model that generates an image description, which is a sentence that explains the content of the image from the image, and generates a next image description, which is output as output information of the first sentence generation model.

[0082] In this way, when generating an image description that explains the contents of a specific frame image, the information processing device 100 uses motion information related to the motion of an object included in the frame image immediately before the specific frame image. This allows the information processing device 100 to reflect information related to the motion performed in the immediately previous frame image in the image description corresponding to the specific frame image. In other words, the information processing device 100 allows the continuity of the motion performed from the immediately previous frame image to the specific frame image to be reflected in the image description corresponding to the specific frame image. That is, the information processing device 100 allows the motion corresponding to a continuous verb performed from the immediately previous frame image to the specific frame image to be easily reflected in the image description corresponding to the specific frame image.

[0083] In addition, the sentence generation unit 134 inputs the next frame image as input information for an encoder included in a second sentence generation model, which is a machine learning model that generates an image description, which is a sentence that explains the contents of an image from an image, generates image features that are output as output information of the encoder, inputs the image features and a special string indicating the beginning of a sentence as input information for a decoder included in the second sentence generation model, weights the conditional probability distribution regarding the occurrence probability of each string sequentially output by the decoder so as to increase the occurrence probability of an action string corresponding to the action information, and generates a next image description that is output as output information of the decoder.

[0084] In this way, when generating an image description that describes the contents of a specific frame image, the information processing device 100 weights the conditional probability distribution so as to increase the appearance probability of an action string corresponding to action information regarding the action of an object included in a frame image one frame before the specific frame image. This makes it possible for the information processing device 100 to generate an image description in which an action string corresponding to an action performed in the previous frame image is likely to appear. In other words, the information processing device 100 can reflect the continuity of an action performed from the previous frame image to the specific frame image in the image description corresponding to the specific frame image. In other words, the information processing device 100 can make it possible to easily reflect an action corresponding to a continuous verb performed from the previous frame image to the specific frame image in the image description corresponding to the specific frame image.

[0085] In addition, the sentence generation unit 134 generates a plurality of image descriptions explaining the contents of each of the plurality of frame images constituting the target moving image and target text data corresponding to the target audio data of the target moving image, generates a summary sentence which is a sentence summarizing the plurality of image descriptions and the target text data based on the plurality of image descriptions and the target text data, and generates a target moving image description which is a sentence explaining the contents of the target moving image based on the summary sentence.

[0086] This enables the information processing device 100 to generate a target moving image description that comprehensively reflects the content of each of the multiple frame images that constitute the target moving image and the content of the target audio data of the target moving image.

[0087] Furthermore, the sentence generation unit 134 generates target character data from the target voice data using a voice-to-text conversion model, which is a machine learning model that generates character data corresponding to voice data from the voice data.

[0088] This enables the information processing device 100 to efficiently generate target character data from the target voice data.

[0089] In addition, the estimation unit 132 inputs image features related to one frame image as input information to a classifier, and estimates movement information based on an movement label corresponding to the image features and output as output information of the classifier.

[0090] This enables the information processing device 100 to appropriately estimate the motion information.

[0091] Furthermore, the estimation unit 132 inputs enriched features generated based on a first feature related to a first region in which an object is imaged in one frame image, a second feature related to a second region in which an object different from the first object is imaged, an intersection feature related to an intersection region in which the first region and the second region overlap, and a combination feature related to a combination region in which both the first region and the second region are included, into a classifier as input information, and estimates action information based on the action label corresponding to the enriched feature and output as output information of the classifier.

[0092] This enables the information processing device 100 to appropriately estimate the motion information.

[0093] The information processing device 100 also includes an estimation unit 132 and a model generation unit 133. The estimation unit 132 estimates motion information related to the motion of an object included in one frame image constituting the learning video. The model generation unit 133 generates a first sentence generation model, which is a machine learning model trained to output a next image description text that is a text that describes the content of the next frame image, as output information when a next frame image of the one frame image and motion information are input as input information.

[0094] Thereby, the information processing device 100 can reflect the motion information on the motion of the object included in the previous frame image in the image description of each frame image. That is, the information processing device 100 can generate an image description of each frame image that reflects the temporal change of the action in each scene of the video. Also, the information processing device 100 can generate a video description based on the image description of each frame image that reflects the motion information on the motion of the object included in the previous frame image. That is, the information processing device 100 can generate a video description that reflects the temporal change of the action in each scene of the video based on the image description of each frame image that reflects the temporal change of the action in each scene of the video. Therefore, the information processing device 100 can generate a video description that reflects the temporal change of the action in each scene of the video. Also, the information processing device 100 can generate a video description that reflects the temporal change of the action in each scene of the video, which can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build and sustainably innovate and promote innovation."

[0095] The estimation unit 132 estimates motion information related to the motion of the object included in one frame image constituting the learning video. The model generation unit 133 generates a second sentence generation model, which is a machine learning model including an encoder trained to output an image feature corresponding to the next frame image as output information when a frame image next to the one frame image is input to the encoder as input information, and a decoder trained to weight a conditional probability distribution related to the occurrence probability of each character string sequentially output by the decoder so as to increase the occurrence probability of an action character string corresponding to the motion information when the image feature and a special character string indicating the beginning of a sentence are input to the decoder as input information, and to output a next image description, which is a sentence explaining the content of the next frame image, as output information.

[0096] Thereby, the information processing device 100 can reflect the motion information on the motion of the object included in the previous frame image in the image description of each frame image. That is, the information processing device 100 can generate an image description of each frame image that reflects the temporal change of the action in each scene of the video. Also, the information processing device 100 can generate a video description based on the image description of each frame image that reflects the motion information on the motion of the object included in the previous frame image. That is, the information processing device 100 can generate a video description that reflects the temporal change of the action in each scene of the video based on the image description of each frame image that reflects the temporal change of the action in each scene of the video. Therefore, the information processing device 100 can generate a video description that reflects the temporal change of the action in each scene of the video. Also, the information processing device 100 can generate a video description that reflects the temporal change of the action in each scene of the video, which can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build and sustainably innovate and promote innovation."

[0097] [6. Hardware Configuration] Moreover, the information processing device 100 according to the embodiment described above is realized by, for example, a computer 1000 having a configuration as shown in Fig. 9. Fig. 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0098] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.

[0099] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0100] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0101] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0102] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0103] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.

[0104] [7. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0105] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0106] For example, in the above-mentioned embodiment, the case where the information processing device 100 has the acquisition unit 131, the estimation unit 132, the model generation unit 133, and the sentence generation unit 134 as functional units has been described, but each unit can be configured by distributing it to a different device. For example, the information processing device 100 can have the acquisition unit 131, the estimation unit 132, and the sentence generation unit 134 as functional units as an information processing device that generates sentences. In addition, an information processing device other than the information processing device 100 (hereinafter, referred to as a "generation device") can have the acquisition unit 131, the estimation unit 132, and the model generation unit 133 as functional units as an information processing device that generates a sentence generation model. At this time, the information processing device 100 and the generation device may be connected to various networks by wire or wirelessly and may transmit and receive information to each other. For example, the information processing device 100 may receive information about the sentence generation model generated by the generation device from the generation device.

[0107] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0108] 100 Information processing device 110 Communications Department 120 Storage section 130 Control section 131 Acquisition Department 132 Estimation Department 133 Model Generation Unit 134 Sentence generation section< / s> < / s> < / s>

Claims

1. an estimation unit that estimates, from a frame image constituting a target moving image that is a moving image to be processed, motion information regarding a motion of a target object included in the frame image; a text generating unit that generates a next image description text that describes the content of the next frame image based on a next frame image of the one frame image and the action information; An information processing device comprising:

2. The sentence generation unit The next frame image and the action information are input as input information of a first sentence generation model, which is a machine learning model that generates an image caption, which is a sentence that explains the content of the image from the image, and the next image caption is generated as output information of the first sentence generation model. The information processing device according to claim 1 .

3. The sentence generation unit The next frame image is input as input information of an encoder included in a second sentence generation model, which is a machine learning model that generates an image caption, which is a sentence that explains the content of the image, from the image, and an image feature amount is generated that is output as output information of the encoder; inputting the image feature and a special character string indicating the beginning of a sentence as input information of a decoder included in the second sentence generation model, weighting a conditional probability distribution regarding the occurrence probability of each character string sequentially output by the decoder so as to increase the occurrence probability of an action character string corresponding to the action information, and generating the next image description to be output as output information of the decoder; The information processing device according to claim 1 .

4. The sentence generation unit Generate a plurality of image captions explaining the contents of each of a plurality of frame images constituting the target video and target text data corresponding to the target audio data of the target video, generate a summary sentence that is a sentence summarizing the plurality of image captions and the target text data based on the plurality of image captions and the target text data, and generate a target video description that is a sentence explaining the contents of the target video based on the summary sentence. The information processing device according to claim 1 .

5. The sentence generation unit generating the target character data from the target voice data using a voice-to-text conversion model that is a machine learning model that generates character data corresponding to the voice data from the voice data; The information processing device according to claim 4.

6. The estimation unit is inputting an image feature amount relating to the one frame image as input information into a classifier, and estimating the motion information based on a motion label corresponding to the image feature amount and outputted as output information of the classifier; The information processing device according to claim 1 .

7. The estimation unit is a first feature amount related to a first region in which an object is captured, a second feature amount related to a second region in which an object different from the object is captured, an intersection feature amount related to an intersection region in which the first region and the second region overlap, and a combination feature amount related to a combination region in which both the first region and the second region are included are generated based on the first feature amount related to a first region in which an object is captured, the second feature amount related to a second region in which an object different from the object is captured, an intersection feature amount related to an intersection region in which the first region and the second region overlap, and a combination feature amount related to a combination region in which both the first region and the second region are included are input to a classifier as input information; and the motion information is estimated based on a motion label corresponding to the enriched feature amount and output as output information of the classifier. The information processing device according to claim 6.

8. An estimation unit that estimates motion information regarding a motion of an object included in a frame image constituting a learning video from the frame image; a model generation unit that generates a first sentence generation model, which is a machine learning model trained to output, as output information, a next image description sentence that is a sentence explaining the content of the next frame image when a next frame image of the one frame image and the action information are input as input information; An information processing device comprising:

9. An estimation unit that estimates motion information regarding a motion of an object included in one frame image that constitutes the learning video; the encoder is trained to output, as output information, an image feature quantity corresponding to a next frame image of the one frame image when the next frame image of the one frame image is input to the encoder as input information; a model generation unit that generates a second sentence generation model, which is a machine learning model including: a decoder that has been trained to weight a conditional probability distribution regarding the occurrence probability of each character string sequentially output by the decoder when the image feature and a special character string indicating the beginning of a sentence are input as input information to the decoder, so as to increase the occurrence probability of an action character string corresponding to the action information, and to output a next image description, which is a sentence that describes the content of the next frame image, as output information; and An information processing device comprising:

10. An information processing method implemented by a program executed by an information processing device, comprising: an estimation step of estimating, from a frame image constituting a target moving image which is a moving image to be processed, motion information regarding a motion of a target object contained in the frame image; a text generation step of generating an image description text that describes the content of the next frame image based on the next frame image of the one frame image and the action information; An information processing method comprising:

11. An information processing method implemented by a program executed by an information processing device, comprising: An estimation step of estimating, from a frame image constituting a learning video, motion information regarding a motion of an object contained in the frame image; a model generation process for generating a first sentence generation model, which is a machine learning model trained to output, as output information, a next image description sentence that is a sentence explaining the content of the next frame image when a next frame image of the one frame image and the action information are input as input information; An information processing method comprising:

12. An information processing method implemented by a program executed by an information processing device, comprising: An estimation step of estimating motion information regarding a motion of an object included in one frame image constituting the learning video; the encoder is trained to output, as output information, an image feature quantity corresponding to a next frame image of the one frame image when the next frame image of the one frame image is input to the encoder as input information; a model generation process for generating a second sentence generation model, which is a machine learning model including the decoder, which has been trained to weight a conditional probability distribution regarding the occurrence probability of each character string sequentially output by the decoder when the image feature and a special character string indicating the beginning of a sentence are input as input information to the decoder, so as to increase the occurrence probability of an action character string corresponding to the action information, and to output a next image description, which is a sentence explaining the content of the next frame image, as output information; An information processing method comprising:

Images