Model generation method and model generation system

Abstract

To provide a technology that enables the generation of a video extraction model that extracts a matching interval that matches contents of an input sentence from a video and has high generalization performance.SOLUTION: A model generation method according to the present disclosure includes extracting a base matching interval and a sub matching interval in a training video by inputting a base sentence and a sub sentence to a video extraction model. The model generation method further includes: calculating a loss by processing a learning task of reconstructing the base sentence based on a feature value of the training video corresponding to each of a correct answer interval, the base matching interval, and the sub matching interval; and performing machine learning of the video extraction model such that a first loss related to the correct answer interval is smaller than a second loss related to the base matching interval, and the second loss is smaller than a sub loss related to the sub matching interval.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to generating machine learning models. [Background technology]

[0002] In recent years, techniques have been proposed for generating machine learning models that enable appropriate processing of various tasks.

[0003] For example, Patent Document 1 discloses a technique for generating a machine learning model for processing a task in which data of a plurality of different modalities is input and data of a modality different from the input is output.

[0004] Other documents that demonstrate the technical level in this field include Patent Document 2 and Patent Document 3 below. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2021 / 182199 [Patent Document 2] Japanese Patent Publication No. 2022-072444 [Patent Document 3] Patent Publication No. 2021-189892 Summary of the Invention [Problem to be solved by the invention]

[0006] One task is to extract segments from a video that match a query sentence. Conventional machine learning models for this task (hereafter simply referred to as "video extraction models") have not been able to achieve sufficient generalization performance for a variety of query sentences.

[0007] One objective of the present disclosure is to provide a technology that enables generation of an image extraction model with high generalization performance for various input sentences. [Means for solving the problem]

[0008] A first aspect of the present disclosure relates to a model generation method for generating a video extraction model that extracts a matching section that matches the content of an input sentence from a video.

[0009] A model generation method according to a first aspect is executed by a computer and includes inputting a plurality of sentences into a video extraction model and extracting a plurality of matching intervals for each of the plurality of sentences from a training video. The plurality of sentences includes a base sentence and at least one sub-sentence shorter than the base sentence. The at least one sub-sentence includes at least one of a positive example sub-sentence that includes words included in the base sentence and does not include noise words unrelated to the base sentence, and a negative example sub-sentence that includes at least a noise word. The plurality of matching intervals includes a base matching interval for the base sentence and at least one sub-matching interval for at least one sub-sentence. The model generation method further includes calculating a first loss by processing a learning task to reconstruct a base sentence based on features of the training video corresponding to the correct answer section; calculating a second loss by processing a learning task to reconstruct the base sentence based on features of the training video corresponding to the base matching section; calculating at least one sub-loss by processing a learning task to reconstruct the base sentence based on features of the training video corresponding to at least one sub-matching section; and performing machine learning of the video extraction model so that the first loss is smaller than the second loss and the second loss is smaller than at least one sub-loss.

[0010] A second aspect of the present disclosure relates to a model generation system that generates a video extraction model that extracts matching sections that match the content of an input sentence from a video.

[0011] A model generation system according to a second aspect includes one or more processors configured to input a plurality of sentences to a video extraction model and extract a plurality of matching intervals for each of the plurality of sentences from a training video. The plurality of sentences include a base sentence and at least one sub-sentence shorter than the base sentence. The at least one sub-sentence includes at least one of a positive example sub-sentence that includes words included in the base sentence and does not include noise words unrelated to the base sentence, and a negative example sub-sentence that includes at least a noise word. The plurality of matching intervals include a base matching interval for the base sentence and at least one sub-matching interval for at least one sub-sentence. The one or more processors are further configured to execute the following processes: calculating a first loss by processing a learning task to reconstruct a base sentence based on features of the training video corresponding to the correct answer section; calculating a second loss by processing a learning task to reconstruct a base sentence based on features of the training video corresponding to the base matching section; calculating at least one sub-loss by processing a learning task to reconstruct a base sentence based on features of the training video corresponding to at least one sub-matching section; and performing machine learning of the video extraction model so that the first loss is smaller than the second loss and the second loss is smaller than at least one sub-loss. [Effects of the Invention]

[0012] According to the present disclosure, the video extraction model is trained to focus on the content of a portion of the input sentence, thereby generating a video extraction model with high generalization performance for various input sentences. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 2 is a diagram for explaining an overview of the function of a video extraction model according to the present embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of the configuration of a video extraction model at the time of inference. [Figure 3] 1 is a flowchart illustrating a model generation method according to the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of processing executed in the model generation method according to the present embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of processing executed in the model generation method according to the present embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of the configuration of a video extraction model during learning. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, this embodiment will be described with reference to the drawings.

[0015] 1.Video Extraction Model The model generation method according to this embodiment uses a sentence as a query and generates a video extraction model that extracts sections from a video that match the content of the sentence (hereinafter referred to as "matching sections").

[0016] 1(A) and 1(B) are diagrams for explaining an outline of the function of an image extraction model 1 according to this embodiment.

[0017] The video extraction model 1 according to this embodiment functions through processing executed by a computer 100. The computer 100 includes one or more processors 110 (hereinafter simply referred to as "processors 110") and one or more storage devices 120 (hereinafter simply referred to as "storage devices 120"). The processor 110 executes various processes. The storage device 120 is connected to the processor 110 and stores various information required for the processor 110 to execute the processes. The processor 110 is configured, for example, by a CPU (Central Processing Unit) including an arithmetic unit, registers, etc. The storage device 120 is configured by a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive).

[0018] The video extraction model 1 is stored in the storage device 120. The processor 110 reads the video extraction model 1 from the storage device 120 and executes processing, thereby realizing the functions of the video extraction model 1. The video extraction model 1 may be realized as a computer program. In particular, the video extraction model 1 may be stored in a computer-readable recording medium.

[0019] The video extraction model 1 receives as input a query sentence 10 (hereinafter referred to as the "input sentence 10") and a video 20 from which a matching section is to be extracted. The video extraction model 1 then extracts a matching section for the input sentence 10 from the video 20. In FIG. 1(A), the matching section is represented by [SP, EP]. SP is the start point of the matching section, and EP is the end point of the matching section. The video extraction model 1 may be configured to output data of the video 20 corresponding to the matching section.

[0020] Figure 1(B) shows an example of a matching section for the input sentence 10 shown in Figure 1(A). In Figure 1(B), it can be seen that the video 20 corresponding to the matching section matches the content of the input sentence 10. The video extraction model 1 aims to achieve such a function during inference.

[0021] 2 is a diagram showing an example of the configuration of the video extraction model 1 at the time of inference. The video extraction model 1 includes a sentence feature extraction unit 210, a video feature extraction unit 220, and a matching section estimation unit 230.

[0022] The sentence feature extraction unit 210 is composed of a text encoder 211 and a text transformer 212. The text encoder 211 outputs a distributed representation for each word included in the input sentence 10. The text transformer 212 is a transformer model that receives the output of the text encoder 211 as input. The output of the text transformer 212 becomes the output of the sentence feature extraction unit 210.

[0023] The video feature extraction unit 220 is composed of a video encoder 221 and a vision transformer 222. The video encoder 221 outputs a feature amount for each frame included in the video 20. The vision transformer 222 is a transformer model that receives the output of the video encoder 221 as input. The output of the vision transformer 222 becomes the output of the video feature extraction unit 220.

[0024] The matching section estimation unit 230 estimates a matching section from the output of the sentence feature extraction unit 210 and the output of the video feature extraction unit 220. The matching section estimation unit 230 can be configured, for example, with a Transformer model and a fully connected layer. In this case, the Transformer model calculates mutual attention between the output of the sentence feature extraction unit 210 and the output of the video feature extraction unit 220. Then, the fully connected layer calculates a matching section for the input sentence 10 from the feature based on the mutual attention. The fully connected layer may be configured to output the degree of match between the content of the input sentence 10 and each frame of the video 20. In this case, the matching section is represented, for example, by a normalized distribution for each frame of the video 20. In this case, the SP and EP can be given from the center and width of the distribution.

[0025] 2. Model generation method The model generation method according to this embodiment generates a video extraction model 1 by performing machine learning. The model generation method according to this embodiment is executed by a computer 100. The storage device 120 may be configured to store a computer program (hereinafter referred to as a "model generation program") for causing the processor 110 to execute each process in the model generation method according to this embodiment. In this case, the model generation method and model generation system according to this embodiment are realized by the processor 110 reading and executing the model generation program.

[0026] The model generation method according to this embodiment will be described below with reference to Fig. 3. Fig. 3 is a flowchart showing the model generation method according to this embodiment.

[0027] In step S100, the processor 110 acquires training data. The training data is composed of a combination of a training sentence (hereinafter referred to as a "base sentence"), a training video (hereinafter referred to as a "training video"), and a correct answer segment that provides the correct answer for the matching segment for the base sentence. The computer 100 may be configured to acquire the training data via a user interface or communication, and store the acquired training data in the storage device 120. The training data may include multiple data combinations of the base sentence, the training video, and the correct answer segment. The following description focuses on one combination.

[0028] Next, in step S110, processor 110 generates subsentences shorter than the base sentence. In this embodiment, the subsentences are generated to include both positive example subsentences that include words included in the base sentence (hereinafter referred to as "extracted words") but do not include words unrelated to the base sentence (hereinafter referred to as "noise words"), and negative example subsentences that include at least noise words. In particular, in this embodiment, the negative example subsentences are generated to include a first negative example subsentence that includes both the extracted words and the noise words, and a second negative example subsentence that does not include the extracted words but does include the noise words. However, the subsentences may also be generated to include at least one of the positive example subsentences and the negative example subsentences. The subsentences can be considered to be sentences obtained by breaking down part of the content of base sentence 11.

[0029] Here, the noise words may be, for example, words included in separate sentences acquired as training data, or may be managed as data stored in the storage device 120.

[0030] The processor 110 can execute the process of generating a subsentence as follows. FIG. 4A is a diagram illustrating an example of the process of generating a subsentence 12. In the example shown in FIG. 4A, the subsentence 12 is generated from a base sentence 11 and a fill-in sentence 15 in which some words are missing. The fill-in sentence 15 is composed of a template section in which specific words are set and a fill-in section indicating the positions of the missing words. The part of speech of the missing word is also specified in the fill-in section. FIG. 4A shows three fill-in sentences 15. For example, in "The person is [Verb] [Noun].", the template section is "The," "person," and "is," and the fill-in section is [Verb] and [Noun]. The fill-in sentence 15 may be configured to be learnable. For example, the words set in the template section may be represented by learnable tokens. This allows an optimal template section to be configured through learning.

[0031] First, the processor 110 performs part-of-speech tagging on each extracted word in the base sentence 11. An example of an extracted word 14 tagged with a part-of-speech tag is shown in FIG.

[0032] The processor 110 then generates a subsentence 12 by completing a fill-in sentence 15 using extracted words 14 or noise words corresponding to the parts of speech specified in the fill-in section. Specifically, the processor 110 can generate a positive example subsentence by completing a fill-in sentence 15 using only extracted words 14. The processor 110 can also generate a first negative example subsentence by completing a fill-in sentence 15 using both extracted words 14 and noise words. The processor 110 can also generate a second negative example subsentence by completing a fill-in sentence 15 using only noise words.

[0033] FIG. 4(A) shows an example of a subsentence 12 generated for "The person is [Verb] [Noun]." when the extracted words 14 are "cut" and "dog's hair" and the noise words are "add" and "onion." The extracted words 14 corresponding to the parts of speech specified in the fill-in section or the noise words may be selected randomly. The processor 110 may also generate multiple positive example subsentences, first negative example subsentences, and second negative example subsentences.

[0034] Referring again to Figure 3, next in step S120, the processor 110 inputs the base sentence 11 and each of the generated subsentences 12 into the video extraction model 1, and extracts a plurality of matching sections for each sentence from the training video. That is, in this embodiment, the plurality of matching sections include a base matching section estimated for the base sentence 11, a positive example sub-matching section estimated for the positive example subsentence, a first negative example sub-matching section estimated for the first negative example subsentence, and a second negative example sub-matching section estimated for the second negative example subsentence.

[0035] Next, in step S130, the processor 110 calculates the loss for the correct interval based on the base matching interval. Specifically, the processor 110 calculates the regression loss L according to the difference between the base matching interval and the correct interval. reg Calculate.

[0036] Next, in step S140, the processor 110 generates a learning task to reconstruct the base sentence 11. For example, the processor 110 generates a learning task to reconstruct masked words by masking some words from the base sentence 11. FIG. 4B shows an example of the generated learning task 16. In FIG. 4B, the learning task 16 is generated by masking "blue" in the base sentence 11. In this case, the number of words to be masked may be multiple. For example, the learning task 16 may be generated by masking 30% of the words randomly in the base sentence 11. Because the learning task 16 is generated from the base sentence 11, it is a self-supervised learning task.

[0037] Next, in step S150, the processor 110 processes the learning task 16 based on the features of the training video corresponding to each matching segment, thereby calculating a loss for each matching segment. The features of the training video corresponding to the matching segment are calculated, for example, by weighting the features of the training video according to the matching segment. When the matching segment is calculated using a distribution for each frame, the weighting can be performed by multiplying the features of each frame of the training video by the value of the corresponding distribution. Alternatively, the features of the training video corresponding to the matching segment are the features of the training video included in the matching segment. The loss is given according to the error of the reconstructed base sentence 11. For example, when processing the learning task 16 of reconstructing masked words, the loss can be calculated using the cross-entropy error. In other words, in this case, the loss indicates the inference error of the masked words.

[0038] 5 is a diagram showing an example of the configuration of the process relating to step S150. gt , EP gt ],[SP q , EP q ], and [SP p , EP p5, the first negative example sub-matching interval and the second negative example sub-matching interval are omitted.

[0039] The feature amounts of the training video 21 acquired by the video encoder 221 are weighted according to each matching section, thereby acquiring feature amounts of the training video 21 corresponding to each of the correct answer section, the base matching section, the positive example sub-matching section, the first negative example sub-matching section, and the second negative example sub-matching section.

[0040] The task processing unit 241 processes the learning task 16 based on the feature values ​​of the training video 21 corresponding to each matching section. The configuration of the task processing unit 241 may be suitably configured depending on the content of the learning task 16. For example, when processing the learning task 16 of reconstructing masked words, the task processing unit 241 can be configured with a Transformer model and a fully connected layer. In this case, the Transformer model calculates mutual attention between the feature values ​​of the training video 21 corresponding to the matching section and each word included in the masked base sentence 11. Then, the fully connected layer calculates an inference result for the masked word from the feature values ​​based on the mutual attention.

[0041] The loss calculation unit 242 calculates a loss from each processing result of the task processing unit 241. Specifically, a loss (hereinafter referred to as "first loss") L is calculated from the processing result of the learning task 16 based on the feature amount of the training video 21 corresponding to the correct answer section. gt Similarly, a loss (hereinafter referred to as the "second loss") L is calculated from the processing result of the learning task 16 based on the feature amount of the training video 21 corresponding to the base matching section. q Similarly, a loss (hereinafter referred to as a "positive example sub-loss") L is calculated from the processing result of the learning task 16 based on the feature amount of the training video 21 corresponding to the positive example sub-matching section. pSimilarly, a loss (hereinafter referred to as the "first negative example sub-loss") L is calculated from the processing result of the learning task 16 based on the feature amount of the training video 21 corresponding to the first negative example sub-matching section. c Similarly, a loss (hereinafter referred to as the "second negative example sub-loss") L is calculated from the processing result of the learning task 16 based on the feature amount of the training video 21 corresponding to the second negative example sub-matching section. o is calculated.

[0042] Referring again to FIG. 3, next, in step S160, the processor 110 performs machine learning of the video extraction model 1 based on the calculated losses. Specifically, the processor 110 performs machine learning so as to reduce the loss function L in the following equation. L is expressed as L reg ,L rec , and L rank It is composed of the linear sum of the three loss functions. In other words, reducing L reg ,L rec , and L rank This means making each smaller. reg ,λ rec , and λ rank is a hyperparameter that determines the contribution of each loss function to L.

[0043]

number

[0044] L reg As mentioned above, is the regression loss according to the difference between the base matching interval and the correct answer interval. That is, L reg Carrying out machine learning to reduce σ means supervised learning of the correct answer interval.

[0045] L rec is constructed using the first loss, second loss, or positive example sub-loss. For example, L recis the sum of the first loss, the second loss, and the positive example sub-loss. The loss calculated by processing the learning task 16 is one of the indicators for measuring the semantic relevance between the training video 21 of the matching section and the base sentence 11. This is because it is believed that the higher the semantic relevance between the training video 21 of the matching section and the base sentence 11, the higher the accuracy of the base sentence 11 reconstructed from the training video 21 of the matching section. Therefore, L rec By performing machine learning to reduce , the video extraction model 1 can be trained to increase the semantic relevance between the input sentence 10 and the estimated matching section.

[0046] L rank is composed of the losses calculated by processing the learning task 16. In particular, L rank is configured to determine the magnitude of each loss. rank can be constructed using the following formula: m0, m1, m2, and m3 are constants that provide predetermined margins.

[0047]

number

[0048] The first term makes it possible to train video extraction model 1 so that the first loss is smaller than the second loss. The second term makes it possible to train video extraction model 1 so that the second loss is smaller than the positive example sub-loss. The third term makes it possible to train video extraction model 1 so that the positive example sub-loss is smaller than the first negative example sub-loss. The fourth term makes it possible to train video extraction model 1 so that the first negative example sub-loss is smaller than the second negative example sub-loss.

[0049] L rank Since L contains each sub-loss for each sub-matching interval, rank Reducing L means increasing the semantic relevance of the estimated matching interval to the content of a part of the base sentence 11. rank can hierarchically assign the degree of relevance of the training video 21 to the base sentence 11 for the correct answer section, the base matching section, and each sub-matching section. This allows the video extraction model 1 to be trained so that the magnitude relationship of the semantic relevance of each matching section is appropriate. Therefore, L rank By performing machine learning to reduce the size of the input sentence 10, the video extraction model 1 can be trained to pay attention to the content of a part of the input sentence 10.

[0050] Furthermore, L rank In this case, the third or fourth term can be omitted. In this case, the calculation cost for learning can be improved by trading off the estimation accuracy of the matching section. In addition, in this case, the second term, L p may be changed depending on the sub-sentence to be generated. For example, if you want to generate only the first negative example sub-sentence, the second term L p L c That's all we need to do.

[0051] As described above, the processor 110 performs machine learning. The machine learning may be performed by backpropagation. Fig. 6 is a diagram showing an example of the configuration of the video extraction model 1 during learning.

[0052] The subsentence generation unit 201 generates a subsentence from the base sentence 11. The learning task generation unit 202 generates a learning task 16 from the base sentence 11. The task processing execution unit 240 calculates each loss by processing the learning task 16 based on the feature amount of the training video 21 corresponding to each matching section. The loss function calculation unit 250 calculates the loss function L.

[0053] 3.Effects As described above, according to this embodiment, the video extraction model 1 can be trained so as to increase the semantic relevance between the input sentence 10 and the estimated matching section. Furthermore, the video extraction model 1 can be trained so as to increase the semantic relevance even when focusing on the content of a portion of the input sentence 10. This makes it possible to generate a video extraction model 1 with high generalization performance for various input sentences 10. [Explanation of symbols]

[0054] 1 Video extraction model, 10 Input sentence, 11 Base sentence, 12 Sub-sentences, 16 Learning Tasks, 20 Videos, 21 Training Videos, 100 Computer, 110 Processor, 120 Storage Device

Claims

1. A model generation method for generating a video extraction model that extracts a matching section that matches the content of an input sentence from a video, comprising: The model generation method is executed by a computer, The model generation method includes inputting a plurality of sentences into the video extraction model and extracting a plurality of matching segments for each of the plurality of sentences from a training video; the plurality of sentences includes a base sentence and at least one sub-sentence that is shorter than the base sentence; The at least one sub-sentence comprises: a positive example sub-sentence that includes a word included in the base sentence and does not include a noise word unrelated to the base sentence; negative example sub-sentences including at least the noise words; and the plurality of matching sections include a base matching section for the base sentence and at least one sub-matching section for the at least one sub-sentence; The model generation method further comprises: Calculating a first loss by processing a learning task of reconstructing the base sentence based on the feature amount of the training video corresponding to the correct answer section; calculating a second loss by processing the learning task based on the feature of the training video corresponding to the base matching section; calculating at least one sub-loss by processing the learning task based on the feature of the training video corresponding to the at least one sub-matching section; performing machine learning of the video extraction model such that the first loss is smaller than the second loss and the second loss is smaller than the at least one sub-loss; Contains Model generation method.

2. 2. The model generation method of claim 1, the at least one sub-sentence includes both the positive example sub-sentence and the negative example sub-sentence; the at least one sub-matching section includes a positive example sub-matching section for the positive example sub-sentence and a negative example sub-matching section for the negative example sub-sentence; the at least one sub-loss includes a positive example sub-loss calculated by processing the learning task based on features of the training video corresponding to the positive example sub-matching section, and a negative example sub-loss calculated by processing the learning task based on features of the training video corresponding to the negative example sub-matching section; The machine learning is further performed so that the positive example sub-loss is smaller than the negative example sub-loss. Model generation method.

3. 3. The model generation method according to claim 2, The negative example sub-sentence is a first negative example sub-sentence including both a word included in the base sentence and the noise word; a second negative example sub-sentence that does not include a word included in the base sentence but includes the noise word; Including, the negative example sub-matching section includes a first negative example sub-matching section for the first negative example sub-sentence and a second negative example sub-matching section for the second negative example sub-sentence; the negative example sub-loss includes a first negative example sub-loss calculated by processing the learning task based on features of the training video corresponding to the first negative example sub-matching section, and a second negative example sub-loss calculated by processing the learning task based on features of the training video corresponding to the second negative example sub-matching section, The machine learning is further performed so that the first negative example sub-loss is smaller than the second negative example sub-loss. Model generation method.

4. 4. A model generation method according to claim 1, further comprising: The learning task is generated by masking some words from the base sentence, and the task is to infer the masked words. Model generation method.

5. A model generation system for generating a video extraction model that extracts a matching section that matches the content of an input sentence from a video, comprising: one or more processors; the one or more processors are configured to execute a process of inputting a plurality of sentences to the video extraction model and extracting a plurality of matching segments for each of the plurality of sentences from a training video; the plurality of sentences includes a base sentence and at least one sub-sentence that is shorter than the base sentence; The at least one sub-sentence comprises: a positive example sub-sentence that includes a word included in the base sentence and does not include a noise word unrelated to the base sentence; negative example sub-sentences including at least the noise words; and the plurality of matching sections include a base matching section for the base sentence and at least one sub-matching section for the at least one sub-sentence; The one or more processors further A process of calculating a first loss by processing a learning task of reconstructing the base sentence based on the feature amount of the training video corresponding to the correct answer section; a process of calculating a second loss by processing the learning task based on the feature quantities of the training video corresponding to the base matching section; calculating at least one sub-loss by processing the learning task based on the feature of the training video corresponding to the at least one sub-matching section; performing machine learning of the video extraction model such that the first loss is smaller than the second loss and the second loss is smaller than the at least one sub-loss; is configured to run Model generation system.

Images