Information processing device, information processing method, and information processing program
The system addresses the inefficiencies in existing captioning technologies by using a differentiable function and memory vectors to ensure accurate and efficient captioning of events in videos.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- OMRON CORP
- Filing Date
- 2022-03-02
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies require significant computational resources and often result in non-uniform accuracy when attaching captions to events in images and videos due to separate learning processes for event extraction and captioning.
A system that uses a differentiable function, such as the Gumbel-Softmax function, to select event video images accurately and a captioning model to generate captions, while incorporating memory vectors to ensure continuity and accuracy in captioning.
Enables highly accurate captioning of events by optimizing the selection and generation processes, reducing computational overhead and improving caption uniformity.
Smart Images
Figure 0007859083000007 
Figure 0007859083000008 
Figure 0007859083000009
Abstract
Description
Technical Field
[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.
Background Art
[0002] In recent years, in machine learning, there is a technology for detecting events such as objects, actions, and situations from images and moving images, and attaching a caption (explanation text) to the event.
[0003] For example, Patent Document 1 discloses a system for attaching a caption to an event by extracting features from an image and determining words using the features.
[0004] In addition, Non-Patent Document 1 discloses a system for dividing a moving image into various ranges according to an event included in the moving image, selecting a moving image divided so that the ranges of the events do not overlap, and attaching a caption to the event indicated by the selected moving image.
[0005] However, in the technology described in Patent Document 1 above, a huge amount of calculation is required to attach an explanation text to a moving image, and it is not always possible to attach a caption to the moving image.
[0006] In addition, in the technology described in Non-Patent Document 1, captions are attached using the learned models of a model for extracting events so that no overlap occurs and a model for attaching captions to the extracted events, respectively. That is, since the learning for extracting events and the learning for attaching captions are performed individually, the accuracy of the captions for the extracted events may become non-uniform.
Prior Art Documents
Patent Documents
[0007]
Patent Document 1
[0008] [Non-Patent Document 1] Xiong, Y., Dai, B., Lin, D. "Move forward and tell: a progressive generator of video descriptions." In: Proc. ECCV. pp. 489-505(2018) <URL: https: / / www.ecva.net / papers / eccv_2018 / papers_ECCV / papers / Yilei_Xiong_Move_Forward_and_ECCV_2018_paper.pdf> [Overview of the project] [Problems that the invention aims to solve]
[0009] The present invention aims to provide an information processing device, an information processing method, and an information processing program that can add highly accurate captions to events using a model for extracting events and a model for adding captions to the extracted events. [Means for solving the problem]
[0010] The first embodiment of the information processing device includes: an acquisition unit that acquires a video image; a division unit that divides the acquired video image into a plurality of event video images according to the event as candidates to which a caption will be added; an event selection unit that selects an event video image from the divided candidate event video images using a selection model that selects the event video image using a differentiable function, such that there is no excess or deficiency in the range indicated by the event from the plurality of input event video images; and a generation unit that adds a caption to the selected event video image and generates a video image with a caption set, using a captioning model that adds a caption to the event indicated by the input event video image.
[0011] The information processing device of the second embodiment is the information processing device according to the first embodiment, wherein the differentiable function includes the Gumbel-Softmax function.
[0012] The third embodiment of the information processing apparatus is an information processing apparatus according to the first or second embodiment, further comprising a storage unit that stores the characteristics of event videos previously selected from the candidates in a first memory vector, and the event selection unit, when selecting the event video, uses the first memory vector to select an event video that shows a continuation of an event video previously selected.
[0013] The fourth embodiment of the information processing device is an information processing device according to the third embodiment, wherein the storage unit stores the characteristics of the caption assigned to the event video in a second memory vector, and the generation unit uses the second memory vector to select a word indicating the continuation of the assigned caption, and adds the selected word to the event video in addition to the previously assigned caption.
[0014] The information processing device according to the fifth embodiment is an information processing device according to the fourth embodiment, further comprising an update unit that updates a first memory vector stored in the storage unit, the update unit updates the first memory vector using the second memory vector, and the event selection unit selects the next event video from the candidates using the updated first memory vector.
[0015] The information processing device according to the sixth embodiment is an information processing device according to the fourth or fifth embodiment, further comprising an update unit that updates a second memory vector stored in the storage unit, wherein the update unit updates the second memory vector using the first memory vector, and the generation unit uses the updated second memory vector to select the next word for the selected event video.
[0016] The information processing device according to the seventh embodiment is an information processing device according to any one of the first to sixth embodiments, further comprising a learning unit that propagates the learning results learned by the assignment model to the selected model and causes it to learn, or causes the selected model and the assignment model to learn, respectively.
[0017] The information processing device according to the eighth embodiment is an information processing device according to the seventh embodiment, wherein the division unit includes a division model that has been trained to divide the event video from the video, and the learning unit propagates the learning results learned by the selection model to the division model for training.
[0018] The information processing method according to the ninth aspect involves acquiring a video image, dividing the acquired video image into a plurality of event video images according to the event as candidates for captioning, and selecting an event video image from the plurality of input event video images such that there is no excess or deficiency in the range indicated by the event, using the selection model which selects the event video image using a differentiable function, selecting an event video image from the divided candidate event video images, and using a captioning model which adds a caption to the selected event video image for the event indicated by the input event video image, thereby generating a video image with a caption.
[0019] An information processing program according to the tenth embodiment causes a computer to perform the following processes: acquire a video image, divide the acquired video image into a plurality of event video images according to the event as candidates for captioning, select an event video image from the plurality of input event video images so as not to be an excess or deficiency in the range indicated by the event using the selection model which selects the event video image using a differentiable function, select an event video image from the divided candidate event video images, and add a caption to the selected event video image using an assignment model which adds a caption to the event indicated by the input event video image, thereby generating a video image with a caption. [Effects of the Invention]
[0020] According to this disclosure, when adding captions to events using a model for extracting events and a model for adding captions to the extracted events, highly accurate captions can be added. [Brief explanation of the drawing]
[0021] [Figure 1] This is a block diagram showing an example of the hardware configuration of the information processing device according to this embodiment. [Figure 2]It is a block diagram showing an example of the functional configuration of the information processing apparatus according to the present embodiment. [Figure 3] It is a data flow diagram showing an example of the flow of data used for explaining the functional configuration of the information processing apparatus according to the present embodiment. [Figure 4] It is a schematic diagram showing an example of moving image data and candidate data according to the present embodiment. [Figure 5] It is a flowchart showing an example of the learning process according to the present embodiment. [Figure 6] It is a flowchart showing an example of the generation process according to the present embodiment.
Mode for Carrying Out the Invention
[0022] Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the drawings. In the drawings, the same or equivalent components and parts are given the same reference numerals. Also, the dimensional ratios in the drawings may be exaggerated for convenience of explanation and may be different from the actual ratios.
[0023] FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing apparatus 10. As shown in FIG. 1, the information processing apparatus 10 according to the present embodiment includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a monitor 16, and a communication interface (communication I / F) 17. Each of the CPU 11, ROM 12, RAM 13, storage 14, input unit 15, monitor 16, and communication I / F 17 is interconnected by a bus 19.
[0024] The CPU 11 oversees and controls the entire information processing device 10. The ROM 12 stores various programs and data, including the information processing program used in this embodiment. The RAM 13 is memory used as a work area when various programs are executed. The CPU 11 performs learning and generation processes by loading the programs stored in the ROM 12 into the RAM 13 and executing them.
[0025] Storage 14 is, for example, an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory. Input unit 15 is a mouse, touch panel, and keyboard that accepts character input and target selection. Monitor 16 displays characters and images. Communication interface 17 transmits and receives data.
[0026] Next, the functional configuration of the information processing device 10 will be described with reference to Figures 2 and 3. Figure 2 is a block diagram showing an example of the functional configuration of the information processing device 10 according to this embodiment. Figure 3 is a data flow diagram showing an example of the data flow used to explain the functional configuration of the information processing device 10 according to this embodiment.
[0027] As an example, as shown in Figure 2, the information processing device 10 includes an acquisition unit 21, a division unit 22, an event selection unit 23, a generation unit 24, an output unit 25, a storage unit 26, an update unit 27, and a learning unit 28. The CPU 11 executes an information processing program, causing the acquisition unit 21, division unit 22, event selection unit 23, generation unit 24, output unit 25, storage unit 26, update unit 27, and learning unit 28 to function.
[0028] As an example, as shown in Figure 3, the acquisition unit 21 acquires the input video data 30.
[0029] The splitting unit 22 extracts events from the acquired video data 30 and splits it into video data containing those events (hereinafter referred to as "event video"). The splitting unit 22 splits it into multiple event video data (hereinafter referred to as "candidate data") 31 as candidates for event video to which captions will be added, and outputs the split candidate data 31 to the event selection unit 23.
[0030] Here, the splitting unit 22 divides the video data 30 into candidate data 31, each with a different length, so that it includes the extracted event 40, as shown in Figure 4 as an example. In other words, one event 40 included in the video data 30 may be included in multiple event videos 41 from the divided candidate data 31.
[0031] The splitting section, as shown in Figure 3, includes a splitting model 22A that has been trained using machine learning to split the video data 30 into individual event video images 41, each containing an event 40. The splitting model 22A is a MIL-NCE model (Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, IvanLaptev, Josef Sivic, and Andrew Zisserman. "End-to-endlearning of visual representations from uncurated instructionalvideos." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.) that was trained on the dataset Howto100M.
[0032] The event selection unit 23 uses the selection model 23A to select event video 41 from the candidate data 31 input by the division unit 22 so as not to be over- or under-selected within the range of event 40.
[0033] The selection model 23A is an autoregressive trained model (Transformer) that has been trained to select event video 41 from candidate data 31 so that event 40 does not overlap. The selection model 23A outputs the selected event video 41 as a first memory vector to the memory unit 26, which will be described later, and stores it. Using the candidate data 31 and the first memory vector, the selection model 23A selects an event video 41 that shows a continuation of an event video 41 that was previously selected.
[0034] Specifically, the selection model 23A shown in Figure 3 comprises an encoder 23B and a decoder 23C. The encoder 23B adds positional encoding and information about each event 40 (for example, the start and end times of each event) to the input candidate data 31 and extracts features. The encoder 23B then inputs the extracted features to the decoder 23C.
[0035] The decoder 23C extracts feature quantities related to each event video 41 using the input feature quantities and the first memory vector. Using the extracted feature quantities related to each event video 41, the decoder 23C selects an event video 41 that shows a continuation of a previously selected event video 41, and outputs the selected event video (hereinafter, "selected data") 32 to the generation unit 24. The decoder 23C also outputs the selected data 32 as the first memory vector to the storage unit 26 for storage.
[0036] Here, when the selection model 23A selects an event video 41, it applies the feature quantities representing each event video 41 and the first memory vector to a differentiable Gumbel-Softmax function to derive the probability of each event video 41. The selection model 23A outputs the event video 41 with the highest probability as the selected data 32. The probability of each event video 41 is expressed by the following formula.
number
number
[0037] Here, V is a feature of the selected data 32 representing the first memory vector, t is the number of the event to be selected, l is the number of candidate event video images, and L is the last number among the candidate event video images. Also, p is the probability that each event video image 41 is selected, n is the number used to identify the event video image 41, X is the candidate data 31, and T is the last number of the selected event. Furthermore, h is a feature extracted by the decoder 23C from the event video image 41, and i is the number used to identify the extracted feature. Also, max is a function that returns the largest value among the input values.
[0038] In other words, equation (1) above represents the selection of event video 41 that represents the maximum value of the stored first memory vector. Equation (2) above is the Gumbel-Softmax function. Equation (2) above represents the probability that each event video 41 is selected, using the first memory vector and each event video 41. By expressing the probability using a differentiable Gumbel-Softmax function, the selection model 23A can learn the processing of the event selection unit 23 and the generation unit 24 as a series of processes without interrupting the chain rate of differentiation.
[0039] The selection model 23A selects the event video 41 with the highest probability in equation (2) above as the selected data 32 and outputs the selected data 32 to the generation unit 24.
[0040] The generation unit 24 uses the captioning model 24A to add captions to the selected data 32 and generates an event video with captions. Here, the generation unit 24 inputs the selected data 32 and the words assigned to the selected data 32 into the captioning model 24A.
[0041] The caption model 24A is an autoregressive trained model (Transformer) that has been trained to assign captions to events included in the selected data 32. The caption model 24A stores the words assigned as captions as a second memory vector in the memory unit 26, which will be described later. Using the input selected data 32 and the second memory vector, the caption model 24A selects words that follow the assigned words and assigns them to the selected data 32 as captions.
[0042] Specifically, the assignment model 24A shown in Figure 4 includes an encoder 24B and a decoder 24C. The encoder 24B obtains a distributed representation corresponding to the selected data 32, adds the selected data 32 and the position encoding to the distributed representation, and extracts features. The encoder 24B inputs the extracted feature quantities to the decoder 24C.
[0043] The decoder 24C extracts events using the input features and a second memory vector, and selects and assigns words that indicate a continuation of words previously assigned to the events. The assignment model 24A assigns words until the decoder 24C assigns a terminating character, and outputs the assigned words as captions 33. The decoder 24C also outputs the output captions 33 as a second memory vector to the storage unit 26 for storage.
[0044] The output unit 25 concatenates the selected data 32 to which the caption 33 is attached, generates and outputs captioned video data 34.
[0045] The memory unit 26 stores the first memory vector and the second memory vector described above. The memory unit 26 may also store training data for training the partitioning model 22A, the selection model 23A, and the assignment model 24A, or it may store word embeddings used by the assignment model 24A.
[0046] The update unit 27 updates the first memory vector and the second memory vector, respectively, using the first memory vector and the second memory vector. The update unit 27 updates the first memory vector and the second memory vector using the following formula.
number
number
[0047] Here, the hat of Vt (the symbol with a ^ above Vt) is the updated first memory vector, St is the second memory vector, and the hat of St (the symbol with a ^ above St) is the updated second memory vector. Also, f1 and f2 are functions that perform a linear transformation in the selected model 23A, g1 and g2 are functions that perform a linear transformation in the assigned model 24A, and σ is the sigmoid function.
[0048] Equation (3) above is obtained by integrating the result of a linear transformation of Vt with the result of performing a linear transformation of St corresponding to Vt twice, component by component using the Hadamard product. Similarly, equation (4) above is obtained by integrating the result of a linear transformation of St with the result of performing a linear transformation of Vt corresponding to St twice, component by component using the Hadamard product.
[0049] The hat for Vt is stored by updating the first main vector as a new feature and is used when selecting the next event video. The hat for St is stored by updating the second main vector as a new feature and is used when selecting the next word for the selected data 32.
[0050] The learning unit 28 performs training on the partitioned model 22A, the selected model 23A, and the assigned model 24A. The learning unit 28 uses backpropagation to train the partitioned model 22A, the selected model 23A, and the assigned model 24A, respectively, so that the loss function is minimized. The learning unit 28 trains the selected model 23A and the assigned model 24A using the loss function expressed by the following formula.
[0051]
number
[0052]
number
[0053] Here, Le is the loss function for the selection model 23A, C is the set of identifiers that identify the selected event 40, Ls is the loss function for the assignment model 24A, and Y is the set of captions assigned to the event 40.
[0054] The learning unit 28 learns the selected model 23A and the assigned model 24A by concatenating them so that the sum of equations (5) and (6) described above is minimized. By calculating the sum of equations (5) and (6) described above, the learning results of the assigned model 24A are propagated to the selected model 23A, and the learning of the selected model 23A is performed.
[0055] In this embodiment, a configuration in which the selected model 23A and the assigned model 24A are linked and trained has been described. However, the embodiment is not limited to this. Furthermore, the divided model 22A may also be linked and trained. For example, the learning unit 28 may introduce a loss function for the divided model 22A and perform training using the sum of this loss function and the above-described equations (5) and (6).
[0056] Alternatively, the segmented model 22A, the selection model 23A, and the assignment model 24A may be trained individually.
[0057] Furthermore, the learning unit 28 may alternate between training each model in a concatenated manner and training each model individually. This allows each model to learn many patterns of training data, including training data influenced by subsequent models and training data independent of subsequent models.
[0058] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Figures 5 and 6. Figure 5 is a flowchart showing an example of the learning process according to this embodiment. The CPU 11 reads an information processing program from the ROM 12 or storage 14 and executes it, thereby executing the learning process shown in Figure 5. The learning process shown in Figure 5 is executed, for example, when an instruction to execute a learning process is input.
[0059] In step S101, the CPU 11 acquires training data to be used to train each model.
[0060] In step S102, the CPU 11 determines whether to train each of the divided model 22A, the selected model 23A, and the assigned model 24A individually. If each model is to be trained individually (step S102: YES), the CPU 11 proceeds to step S103. On the other hand, if each model is not to be trained individually (the models are to be trained in combination) (step S102: NO), the CPU 11 proceeds to step S104.
[0061] In step S103, the CPU 11 uses the training data corresponding to each model from the acquired training data to perform machine learning in such a way that the loss function corresponding to each model is minimized.
[0062] In step S104, the CPU 11 uses the acquired training data to concatenate each model and perform machine learning. Here, the CPU 11 performs machine learning in such a way that the sum of the loss functions corresponding to each model is minimized.
[0063] In step S105, the CPU 11 verifies the learning results for each model. For example, the CPU 11 verifies the video image data 30 using validation data to which captions have been added as ground truth data.
[0064] In step S106, the CPU 11 determines whether or not to terminate the learning process. If the learning process is to be terminated (step S106: YES), the CPU 11 terminates the learning process. On the other hand, if the learning process is not to be terminated (step S106: NO), the CPU 11 proceeds to step S101 to acquire new learning data.
[0065] Next, with reference to Figure 6, the generation process for generating captioned video data 34 will be described. Figure 6 is a flowchart showing an example of the generation process according to this embodiment. The CPU 11 reads and executes an information processing program from the ROM 12 or storage 14, thereby executing the generation process shown in Figure 6. The generation process shown in Figure 6 is executed, for example, when video data 30 and an instruction to execute the generation process are input.
[0066] In step S201, the CPU 11 acquires the input video data 30.
[0067] In step S202, the CPU 11 divides the acquired video data 30 into candidate data 31 that includes the event 40.
[0068] In step S203, the CPU 11 selects the selected data 32 using the candidate data 31 and the first memory vector.
[0069] In step S204, the CPU 11 stores the selected data 32 as a first memory vector.
[0070] In step S205, the CPU 11 selects a word using the selected data 32 and the second memory vector.
[0071] In step S206, the CPU 11 stores the selected word as a caption 33 to be attached to the selected data 32.
[0072] In step S207, the CPU 11 stores the selected word as a second memory vector.
[0073] In step S208, the CPU 11 determines whether or not a terminator has been selected as a word. If a terminator has been selected (step S208: YES), the CPU 11 proceeds to step S209. On the other hand, if a terminator has not been selected (step S208: NO), the CPU 11 proceeds to step S210.
[0074] In step S209, the CPU 11 adds the stored caption 33 to the selected data 32 and stores it.
[0075] In step S210, the CPU 11 updates the second memory vector using the first memory vector.
[0076] In step S211, the CPU 11 inputs the updated second memory vector into the model 24A. The CPU 11 then proceeds to step S205 to select a word that indicates the continuation of the selected word.
[0077] In step S212, the CPU 11 determines whether or not captions 33 have been added to all events 40 included in the video data 30 up to the last event 40. If captions 33 have been added up to the last event 40 (step S212: YES), the CPU 11 proceeds to step S213. On the other hand, if captions 33 have not been added up to the last event 40 (there are events 40 without captions 33) (step S212: NO), the CPU 11 proceeds to step S214.
[0078] In step S213, the CPU 11 concatenates the selected data 32 to which the caption 33 is attached and outputs it as captioned video data 34.
[0079] In step S214, the CPU 11 updates the first memory vector using the second memory vector.
[0080] In step S215, the CPU 11 inputs the updated first memory vector into the selection model 23A. The CPU 11 then proceeds to step S203 and selects the next selection data 32 that indicates the continuation of the previously selected selection data 32.
[0081] As described above, according to this embodiment, when adding captions to events using a model for extracting events and a model for adding captions to the extracted events, highly accurate captions can be added.
[0082] The embodiments described above are merely illustrative examples of the configurations of the present disclosure. The present disclosure is not limited to the specific forms described above, and various modifications are possible within the scope of its technical concept.
[0083] Furthermore, the control processing that the CPU reads and executes in each of the above embodiments may be executed by various processors other than the CPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits) which have a circuit configuration specifically designed to execute a particular process. In addition, the learning process and generation process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements. It may also be executed using a GPU (Graphics Processing Unit) together with the CPU.
[0084] Furthermore, although the above embodiments describe a configuration in which the information processing program is pre-stored (installed) in the storage 14 or ROM 12, the invention is not limited thereto. The program may be provided in the form of a recording medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the program may be provided in the form of a download from an external device via a network. [Explanation of Symbols]
[0085] 10 Information Processing Devices 11 CPU 12 ROM 13 RAM 14 Storage 15 Input section 16 monitors 17 Communication I / F 19 bus 21 Acquisition Department 22 Division 22A Split Model 23 Event Selection Section 23A Select Model 23B encoder 23C Decoder 24 Generation part 24A-equipped model 24B iOS 24C Decoder 25 Output section 26 Memory section 27 Update section 28 Learning Department 30 Moving image data 31 Candidate Data 32 Selected Data 33 Captions 34. Video data with captions 40 Events 41 Event Videos
Claims
1. An acquisition unit that acquires moving images, A division unit divides the acquired video image into multiple event videos according to the event, as candidates for assigning captions. An event selection unit selects an event video from a divided list of candidate event videos using a selection model that selects an event video from multiple input event videos using a differentiable function, A generation unit generates a video with captions by assigning captions to selected event videos using a captioning model that assigns captions to events shown in input event videos. A storage unit that stores a first memory vector indicating the characteristics of an event video previously selected from the candidates, and a second memory vector indicating the characteristics of a caption assigned to the event video, The system includes an update unit that updates the first memory vector stored in the storage unit using the second memory vector, The event selection unit, when selecting the event video, uses the first memory vector to select an event video that shows a continuation of a previously selected event video, and uses the updated first memory vector to select the next event video from the candidates. The generation unit uses the second memory vector to select a word that indicates a continuation of the assigned caption, and adds the selected word to the event video in addition to the previously assigned caption. Information processing device.
2. The aforementioned differentiable functions include the Gumbel-Softmax function. The information processing apparatus according to claim 1.
3. The update unit updates the second memory vector using the first memory vector, The generation unit uses the updated second memory vector to select the next word for the selected event video. The information processing apparatus according to claim 1 or claim 2.
4. The selection model further comprises a learning unit that propagates the learning results learned by the assignment model to the selection model for further learning. An information processing apparatus according to any one of claims 1 to 3.
5. The division unit includes a division model that has been trained to divide the event video from the video, The learning unit propagates the learning results learned by the selected model to the partitioned model and allows it to learn. The information processing apparatus according to claim 4.
6. Capture video and images, As candidates for captioning, the acquired video image is divided into multiple event videos according to the event, Using a selection model that selects an event video from multiple input event videos using a differentiable function, the event video is selected from the divided candidate event videos. Using a captioning model that assigns captions to events shown in input event videos, captions are assigned to the selected event videos to generate videos with captions. The first memory vector, which is stored in a memory unit that stores a first memory vector indicating the characteristics of an event video previously selected from the candidates, and a second memory vector indicating the characteristics of a caption assigned to the event video, is updated using the second memory vector. When selecting the event video, the first memory vector is used to select an event video that shows a continuation of a previously selected event video, and the updated first memory vector is used to select the next event video from the candidates. In generating a video with the caption set, the second memory vector is used to select a word indicating a continuation of the assigned caption, and the selected word is added to the event video in addition to the previously assigned caption. An information processing method in which an information processing device performs the processing.
7. Capture video and images, As candidates for captioning, the acquired video image is divided into multiple event videos according to the event, Using a selection model that selects an event video from multiple input event videos using a differentiable function, the event video is selected from the divided candidate event videos. Using a captioning model that assigns captions to events shown in input event videos, captions are assigned to the selected event videos to generate videos with captions. The first memory vector, which is stored in a memory unit that stores a first memory vector indicating the characteristics of an event video previously selected from the candidates, and a second memory vector indicating the characteristics of a caption assigned to the event video, is updated using the second memory vector. When selecting the event video, the first memory vector is used to select an event video that shows a continuation of a previously selected event video, and the updated first memory vector is used to select the next event video from the candidates. In generating a video with the caption set, the second memory vector is used to select a word indicating a continuation of the assigned caption, and the selected word is added to the event video in addition to the previously assigned caption. An information processing program that instructs a computer to perform a task.