Representative frame selection device and its program

Abstract

This invention provides a representative frame selection device that can select representative frames from video data using various scales. [Solution] The representative frame selection device 1 includes: a target text generation unit 20 that takes a frame list of video data and instruction text as input and generates target text using a learning model; a partial frame list generation unit 30 that generates a partial frame list by excluding frames one by one from the frame list; a frame importance calculation unit 40 that calculates the importance of frames excluded when generating the partial frame list, with the lower the degree to which the learning model predicts the target text from the partial frame list and instruction text, the higher the importance; and a frame selection unit 70 that selects a representative frame from the frame list based on the importance of each frame.

Description

[Technical Field]

[0001] The present invention relates to a representative frame selection device and a program for selecting a representative frame from video data. [Background technology]

[0002] Video data consists of many frames and is therefore very data-intensive. To reduce the amount of information, it's sometimes necessary to select frames that best represent the video data. For example, when introducing a broadcast program on a website, it's common practice to select a few frames that clearly show the content of the program and include them on the website.

[0003] Several techniques exist for selecting such representative frames. For example, the technique described in Patent Document 1 first detects scene changes in the video and divides it into scenes. Then, the frame with the maximum feature quantity for each scene is designated as the representative frame. Feature quantities used include image brightness, color, edge data, and acoustic data. Furthermore, for example, the technology described in Patent Document 2 primarily aims to select representative frames from action footage, and selects representative frames from the video based on features related to the pose of the captured object. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2000-350156 [Patent Document 2] Japanese Patent Publication No. 2022-104178 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] Conventional feature-based frame selection methods are based on predetermined features or frame selection criteria defined by humans, assuming use in a specific situation. In other words, if the video has features that deviate significantly from the environment assumed beforehand, the accuracy of the predetermined features or frame selection criteria decreases drastically, resulting in the problem of being unable to select the desired frame.

[0006] For example, when using images on a broadcast program's website, frames should be selected based on the criterion that the performers are clearly visible. On the other hand, when using images for sports motion analysis, frames should be selected based on the criterion that the sequence of movements is easy to understand, even if there is some motion blur. Therefore, conventional methods have the problem that, in order to correctly select a representative frame, it is necessary to redesign the features and frame selection criteria according to the various situations depending on what the representative frame will be used for, which is very costly.

[0007] This invention has been made in view of the above problems, and aims to provide a representative frame selection device and program therefor that can easily select a representative frame from video data using various scales. [Means for solving the problem]

[0008] To solve the aforementioned problems, the representative frame selection device according to the present invention is a representative frame selection device that selects a representative frame from video data, and comprises a target text generation unit, a partial frame list generation unit, a frame importance calculation unit, and a frame selection unit.

[0009] In this configuration, the representative frame selection device receives a list of frames constituting the video data and instruction text as input from the target text generation unit, and uses a pre-trained artificial intelligence (AI) learning model to generate target text that indicates the content of the video data corresponding to the instruction text. By changing this instruction text, target text indicating the content of the video data can be generated using various scales.

[0010] Furthermore, the representative frame selection device generates multiple partial frame lists by removing one different frame at a time from the frame list using the partial frame list generation unit. The representative frame selection device, using a frame importance calculation unit, generates a token probability distribution from the subframe list and instruction text using a learning model for each subframe list, and calculates the importance of frames that were excluded when generating the subframe list, with higher importance assigned to frames with a lower degree of prediction of the target text. This necessitates determining the importance of each frame in the frame list that makes up the video data.

[0011] Then, the representative frame selection device, using the frame selection unit, selects a representative frame from the frame list based on the importance of each frame calculated by the frame importance calculation unit. This allows the representative frame selection device to change the importance of frames according to the scale indicated by the instruction text, and select frames with higher importance.

[0012] Furthermore, in order to solve the above problems, the representative frame selection device according to the present invention is a representative frame selection device that selects a representative frame from video data, and comprises a learning data generation unit having a target text generation unit, a partial frame list generation unit, and a frame importance calculation unit, a model learning unit, a second frame importance calculation unit, and a frame selection unit.

[0013] In this configuration, the representative frame selection device generates the importance of each frame corresponding to the training video data as training data using the training data generation unit. In other words, the learning data generation unit receives a frame list and instruction text from the target text generation unit, and uses a pre-trained AI learning model to generate target text that indicates the content of the learning video data corresponding to the instruction text.

[0014] Furthermore, the training data generation unit generates multiple subframe lists by removing one different frame from each frame list using the subframe list generation unit. Then, the training data generation unit, using the frame importance calculation unit, generates a token probability distribution for each subframe list from the subframe list and the instruction text using the trained training model, and calculates the importance of frames that were excluded when generating the subframe list, with the lower the degree of prediction of the target text, the higher the importance. This allows for the generation of importance levels for each frame corresponding to the training video data, which can then be used as training data.

[0015] The representative frame selection device then receives a list of frames constituting the training video data from the model learning unit and trains a second learning model that outputs the importance of each frame generated by the training data generation unit. Then, the representative frame selection device, using the second frame importance calculation unit, calculates the importance of each frame from the frame list that constitutes the video data using the second learning model.

[0016] Then, the representative frame selection device, using the frame selection unit, selects a representative frame from the list of frames that make up the video data, based on the importance of each frame calculated by the second frame importance calculation unit. This allows the representative frame selection device to generate a small (lightweight) second learning model that does not use the instruction text, based on the scale indicated by the instruction text, and then select a representative frame. Furthermore, the representative frame selection device can be operated using a program that allows the computer to function as a representative frame selection device. [Effects of the Invention]

[0017] According to the present invention, representative frames can be easily selected from video data based on various scales. [Brief explanation of the drawing]

[0018] [Figure 1] This is a block diagram showing the configuration of a representative frame selection device according to the first embodiment of the present invention. [Figure 2] This is an explanatory diagram to illustrate the method for calculating the importance of frames. [Figure 3] This is a block diagram showing a modified configuration of a representative frame selection device according to the first embodiment of the present invention. [Figure 4] This is a flowchart showing the operation of the representative frame selection device according to the first embodiment of the present invention. [Figure 5] This flowchart shows another example of the operation of the representative frame selection device according to the first embodiment of the present invention. [Figure 6] This flowchart shows another example of the operation of the representative frame selection device according to the first embodiment of the present invention. [Figure 7] This is a block diagram showing the configuration of a representative frame selection device according to a second embodiment of the present invention. [Figure 8] This is a structural diagram showing an example of an importance predictor, which is the second learning model that the model learning unit learns. [Figure 9] This is a flowchart showing the operation of learning the second learning model of the representative frame selection device according to the second embodiment of the present invention. [Figure 10]This is a flowchart showing the operation of selecting a representative frame using a second learning model of a representative frame selection device according to a second embodiment of the present invention. [Modes for carrying out the invention]

[0019] Embodiments of the present invention will be described below with reference to the drawings. <First Embodiment> [Configuration of the representative frame selection device] Referring to Figure 1, the configuration of the representative frame selection device 1 according to the first embodiment of the present invention will be described.

[0020] The representative frame selection device 1 selects a representative frame from the video data to be selected. Here, the representative frame selection device 1 selects the instruction text T inst The program takes a frame list FL of video data as input and outputs one or more representative frames as a representative frame list RFL.

[0021] Instruction text T inst This is text that requests the content of the video data to be input into the learning model (VisionLLM) described later, and is such as "Please explain the content of this video in detail." This instruction text T inst For example, when using a representative frame on a broadcast program's website, it might be used as a criterion for selecting a representative frame, such as "Please describe the main subject," or when using it for sports motion analysis, "Please describe the person's movements."

[0022] The frame list FL is a list of sequentially numbered frames that make up the video data to be selected as the representative frame. That is, the frame list FL is {f i |i=1,2,…,N frame}. Here, N frame This is the total number of frames in the frame list FL. As shown in Figure 1, the representative frame selection device 1 comprises a model storage unit 10, a target text generation unit 20, a partial frame list generation unit 30, a frame importance calculation unit 40, and a frame selection unit 70.

[0023] The model storage unit 10 stores the learning model M to be used in the learning model calculation units 21 and 40, which will be described later. This model storage unit 10 can be made of a general storage medium such as semiconductor memory.

[0024] The learning model M is a pre-trained model that takes video data and instruction text (which asks for the content of the video data) as input, and outputs the instructed content as text (target text). This model is an AI (Artificial Intelligence) model called VisionLLM (Vision Large Language Model) that integrates vision and a large-scale language model (LLM).

[0025] For example, VisionLLM can utilize common learning models such as Phi-3.5-vision (https: / / huggingface.co / microsoft / Phi-3.5-vision-instruct), InternVL2.0 (https: / / huggingface.co / OpenGVLab / InternVL2-Llama3-76B), and LLaVA-NeXT-Video (https: / / huggingface.co / llava-hf / LLaVA-NeXT-Video-34B-hf).

[0026] The target text generation unit 20 takes a frame list that constitutes the video data and instruction text as input and uses a pre-trained learning model (VisionLLM) to generate target text that indicates the content of the video data corresponding to the instruction text. Here, the target text generation unit 20 comprises a learning model calculation unit 21 and a decoding unit 22.

[0027] The learning model operation unit 21 inputs a token sequence together with the frame list FL, and calculates the learning model M (VisionLLM) stored in the model storage unit 10, thereby obtaining the probability distribution P for the tokens following the input token sequence. target It generates this. A token is the smallest unit of text segmentation defined in advance for easy processing by the LLM / AI. Generally, it is automatically determined using an algorithm such as BPE (Byte-Pair Encoding). The learning model operation unit 21 first inputs the tokenized instruction text T inst (input token sequence) and generates the probability distribution P for the subsequent tokens. target The learning model operation unit 21 outputs the generated probability distribution P target to the decoding unit 22. Note that the learning model operation unit 21 sequentially inputs the decoded tokens from the decoding unit 22 and repeats the operation of generating the probability distribution P for the subsequent tokens. target

[0028] The decoding unit 22 decodes (deciphers) the probability distribution P generated by the learning model operation unit 21 into a token sequence. target The decoding unit 22 inputs the probability distribution P of tokens from the learning model operation unit 21 and determines one subsequent token using a general decoding method. For example, when using the greedy method (Greedy Decoding), the token with the highest probability is determined as the subsequent token. target The decoding unit 22 outputs the token sequence obtained by concatenating the input token sequence and the subsequent token to the learning model operation unit 21.

[0029] The learning model operation unit 21 and the decoding unit 22 repeat the operation of determining the tokens following the input token sequence to generate a token sequence, that is, text. The termination condition for a loop is either that the subsequent token becomes a special token that indicates the end of a statement (as defined in advance), or that the number of tokens generated reaches a predetermined number. The target text T is a sequence of generated tokens at the end of the iteration (excluding the initial input token sequence). target That is the case. In addition to the greedy algorithm, other common decoding methods such as beam search and random sampling can be used in the decoding unit 22. The decoding unit 22 processes the target text T, which is the decoded result. target This is output to the frame importance calculation unit 40. As a result, the target text generation unit 20 outputs the instruction text T to the frame list FL. inst Target text T is the answer to the instructions given. target It can generate [this].

[0030] Furthermore, the target text generation unit 20 selects a new frame list FL from the frame selection unit 70, which will be described later. NEW If entered, a new frame list FL NEW and instruction text T inst From the new target text T target It generates and outputs to the frame importance calculation unit 40.

[0031] The partial frame list generation unit 30 generates multiple partial frame lists by removing one different frame from each frame list. Specifically, the partial frame list generation unit 30 generates the frame list FL{f i |i=1,2,…,N frame By removing the i-th frame (initial value i=1) from} one frame at a time, we can create a partial frame list PFL{f j |j≠i,j=1,2,…,N frame} is generated. That is, the partial frame list generation unit 30 generates N frameMinutes, generate a partial frame list (PFL). The partial frame list generation unit 30 outputs the generated partial frame lists PFL to the frame importance calculation unit 40.

[0032] The partial frame list generation unit 30 generates a new frame list FL from the frame selection unit 70, which will be described later. NEW If entered, a new frame list FL NEW and instruction text T inst A new partial frame list (PFL) is generated from this and output to the frame importance calculation unit 40.

[0033] The frame importance calculation unit 40 calculates the frame importance for each subframe list PFL using the learning model M (VisionLLM) and the instruction text T. inst The probability distribution of tokens is calculated from the target text T. target The lower the degree of prediction, the higher the importance, which is calculated as the importance of each frame that was excluded when generating the partial frame list (PFL). Here, the frame importance calculation unit 40 comprises a learning model calculation unit 41 and an importance calculation unit 42.

[0034] The learning model calculation unit 41 receives a token sequence along with a partial frame list PFL, and generates a probability distribution of subsequent tokens by calculating the learning model M (VisionLLM) stored in the model storage unit 10. The learning model M used by the learning model calculation unit 41 is the same as the learning model used by the learning model calculation unit 21.

[0035] The learning model calculation unit 41 calculates the target text T for each partial frame list PFL. target The probability distribution of the k-th token {P i,k |i=1,2,…,N frame ,k=1,2,…,N tokens Perform the operation on}. Note that N frame N is the number of frames in the partial frame list PFL. tokens The target text is Ttarget This is the number of tokens. In this case, the probability distribution P i,k This is the target text T of the partial frame list PFL obtained by removing the i-th frame from the frame list FL. target This represents the probability distribution of the k-th token. The learning model calculation unit 41 calculates the probability distribution P which is the result of the calculation. i,k The result is output to the importance calculation unit 42.

[0036] The importance calculation unit 42 calculates the probability distribution P calculated by the learning model calculation unit 41. i,k and target text T target Therefore, the probability distribution P i,k This calculates the importance of frames that were excluded during generation. The importance calculation unit 42 calculates the importance of the probability distribution P i,k and target text T target The importance level is determined by which the value increases as the relationship with the other factor becomes stronger.

[0037] Here, the importance calculation unit 42 uses perplexity as an example of importance. Perplexity is an indicator of how accurately a probability distribution predicts a sample. In other words, the importance calculation unit 42 calculates perplexity from the probability distribution P i,k Target text T target It is used as an indicator to show how accurately the prediction is made. The importance calculation unit 42 calculates the importance I(I) using the following formula (1). i ) is calculated as perplexity.

[0038]

number

[0039] Here, N tokens The target text is T target This is the number of tokens. P i,k This is the k-th probability distribution output by the learning model calculation unit 41, where P i,k (t k ) is the target text Ttarget This is the probability of generating the kth token. The smaller this importance I (perplexity), the smaller the probability distribution P. i,k The target text is T target This is close to the same thing. In other words, if the importance I is small, the impact of removing the j-th frame from the frame list FL will be small. Conversely, if the importance I is large, the impact of removing the j-th frame from the frame list FL will be large.

[0040] Note that importance level I is determined by probability distribution P i,k and target text T target Any indicator that shows the relationship with the other is acceptable. For example, the importance calculation unit 42 can use cross-entropy or other indicators of importance in addition to perplexity.

[0041] Here, with reference to Figure 2, we will schematically explain the method for calculating frame importance. For the sake of brevity, we will assume that the frame list FL, which is the video data, contains only three frames (F1, F2, F3). Frame F1 contains a banana, and frames F2 and F3 contain apples. Also, instruction text T inst For example, let's consider the question, "What is being shown?"

[0042] In this case, the target text generation unit 20 uses the learning model M (VisionLLM) to generate the frame list FL and the instruction text T. inst Therefore, target text T target The probability distribution P of the tokens target By generating and decoding the target text T target Generates the instruction text T. inst Target text T target As an example, the message "Bananas and apples." is shown.

[0043] Meanwhile, the partial frame list generation unit 30 generates multiple partial frame lists PFL1, PFL2, ... by sequentially removing one frame at a time from the frame list FL. In the example in Figure 2, partial frame list PFL1 excludes frame F1, which contains only one banana, from frame list FL, resulting in an image containing only apples. On the other hand, partial frame list PFL2 excludes frame F2, which contains one of two apples, resulting in an image containing both bananas and apples.

[0044] The frame importance calculation unit 40 uses the learning model M to calculate the importance of the partial frame list PFL1 and the instruction text T. inst The token probability distribution P1 is generated from the and . Similarly, the frame importance calculation unit 40 uses the learning model M to calculate the partial frame list PFL2 and the instruction text T inst The token probability distribution P2 is generated from this. The same applies to the subframe list PFL3, which is not shown in the diagram.

[0045] The frame importance calculation unit 40 then calculates the probability distribution P1 and the target text T. target From this, the importance I1 (perplexity) of the frame F1 that was excluded when generating the probability distribution P1 is calculated. Similarly, the frame importance calculation unit 40 calculates the importance of the probability distribution P2 and the target text T target From this, we calculate the importance I2 of the frames F2 that were excluded when generating the probability distribution P2. The same procedure is followed for the probability distribution P3, which is not shown in the diagram.

[0046] In this case, from the partial frame list PFL1, the target text T target As a result, "bananas and apples" are not generated, and the importance I1 of frame F1 is high (here, I1=2.8). On the other hand, from the partial frame list PFL2, the target text T target This means that the probability of generating "bananas and apples" is high, and the importance I2 of frame F2 is low (here, I2 = 1.8). In this way, the frame importance calculation unit 40 can calculate importance I1, I2, ... corresponding to each frame F1, F2, ... and generate an importance list IL. Returning to Figure 1, we will continue our explanation of the configuration of the representative frame selection device 1.

[0047] The importance calculation unit 42 calculates the importance I of frame i for each partial frame list PFL using formula (1). i Arrange them frame by frame, and assign importance to the list IL{I i |i=1,2,…,N frame This is output to the frame selection unit 70.

[0048] The frame selection unit 70 selects a representative frame from the frame list based on the importance of each frame calculated by the frame importance calculation unit 40. Here, the frame selection unit 70 selects a predetermined number of frames C from the frame list as representative frames based on their importance.

[0049] For example, the frame selection unit 70 selects a representative frame by following the procedure below. The frame selection unit 70 removes the frame with the lowest importance from the importance list IL and creates a new frame list FL. NEW It generates a new frame list FL. Then, the frame selection unit 70 generates a new frame list FL. NEW The number of frames in the frame list FL is replaced with a new frame list FL sequentially until the number of frames C specified by the selected frame list is reached. NEW Using this, the target text generation unit 20, the partial frame list generation unit 30, and the frame importance calculation unit 40 are operated to recalculate the importance. Then, the frame selection unit 70 sets the frame list that remains without being excluded as the representative frame list RFL, which consists of representative frames.

[0050] In this case, the frame selection unit 70 recalculated importance by excluding the frame with the lowest importance. However, it is also possible to select a specified number of frames C from the frame list FL in order of importance, starting with the most important, and use them as representative frames. This can reduce the cost of importance calculation. However, since importance varies greatly depending on the frames that make up the frame, it is preferable to recalculate importance to improve the accuracy of selecting high-importance frames.

[0051] In this example, we have selected a specified number of frames C. However, when the frame selection unit 70 excludes the frame with the lowest importance in the importance list IL, it may retain the importance of the frame that was excluded earlier. If the change in importance exceeds a predetermined threshold, the remaining frame list, which was not excluded, may be used as the representative frame list RFL, which consists of representative frames. Thus, the frame selection unit 70 can also automatically determine the number of frames to select.

[0052] With the configuration described above, the representative frame selection device 1 selects the instruction text T inst With a simple change, you can select representative frames from video data based on various scales.

[0053] In this case, a representative frame is selected from the entire frame list of the video data. However, the content of neighboring frames in a video often does not change much. Therefore, in order to reduce the number of frames of the target video data in advance, a configuration further including a sampling unit 80 may be used, as shown in Figure 3.

[0054] In other words, the representative frame selection device 1' is configured to include a sampling unit 80 as the foremost stage for inputting the frame list FL. The other configurations are the same as those of the representative frame selection device 1 described in Figure 1. The sampling unit 80 periodically samples (extracts) frames from the frame list FL. For example, the sampling unit 80 samples one frame every 10 to 30 frames in the frame list FL. This means that for video with a frame rate of 30fps (frames per second), it samples 1 to 3 frames per second. This allows the representative frame selection device 1' to speed up the process of selecting a representative frame.

[0055] [Operation of the representative frame selection device] Next, with reference to Figure 4 (and Figure 1 as appropriate for the configuration), the operation of the representative frame selection device 1 according to the first embodiment of the present invention will be described.

[0056] In step S1, the representative frame selection device 1 receives instruction text T from an external source. inst Enter the frame list FL and the number of frames to select C. In step S2, the target text generation unit 20 uses a pre-trained learning model M (VisionLLM) to generate the instruction text T inst From the frame list FL, the target text T target Generates.

[0057] Here, the learning model calculation unit 21 calculates the token sequence (initially the instruction text T) along with the frame list FL. inst By inputting ) and operating the learning model M, the probability distribution P of the tokens following the input token sequence is obtained. target Generates. Then, the decoding unit 22 processes the probability distribution P target Decode the result and determine one subsequent token. Then, the target text generation unit 20 repeats the operations of the learning model calculation unit 21 and the decoding unit 22 to generate the target text T, which is a sequence of tokens. target Generates.

[0058] In step S3, the partial frame list generation unit 30 initializes the value of the variable i for counting up the frame list FL (i=1). In step S4, the partial frame list generation unit 30 generates a partial frame list PFL obtained by excluding the i-th frame from the frame list FL. i is generated.

[0059] In step S5, the frame importance calculation unit 40 uses the learning model M to calculate the importance of the i-th frame excluded in step S4 from the partial frame list PFL i generated in step S4 and the instruction text T inst input in step S1. Here, the learning model operation unit 41 calculates the probability distribution {P target of the k-th token of the target text T for each partial frame list PFL i,k |i = 1, 2, …, N frame , k = 1, 2, …, N tokens}. Then, the importance calculation unit 42 calculates the importance I of the i-th frame as the perplexity according to the formula (1).

[0060] In step S6, the frame importance calculation unit 40 increments the value of the variable i (i = i + 1). In step S7, the frame importance calculation unit 40 determines whether the value of the variable i has exceeded the number of frames in the frame list FL. Here, if the value of the variable i has not yet exceeded the number of frames in the frame list FL (No in step S7), the representative frame selection device 1 returns to step S4 and continues to operate.

[0061] On the other hand, when the value of the variable i exceeds the number of frames in the frame list FL (Yes in step S7), in step S8, the frame selection unit 70 excludes the frame with the lowest importance from the frame list FL and generates a new frame list FL NEW .

[0062] In step S9, the frame selection unit 70 uses the frame list FL NEWDetermine whether the number of frames (frame list length) is equal to the selected number of frames C input in step S1. Here, when the frame list length does not match the selected number of frames C (No in step S9), the representative frame selection device 1 returns to step S2 and continues to operate using the new frame list FL. NEW Continue to operate using it.

[0063] On the other hand, when the frame list length matches the selected number of frames C (Yes in step S9), in step S10, the frame selection unit 70 outputs the frame list that remained without being excluded in step S8 as a representative frame list RFL composed of representative frames.

[0064] Through the above operations, the representative frame selection device 1 can select representative frames from video data based on various metrics with an easy operation that only changes the instruction text T. inst Can select representative frames from video data based on various metrics with an easy operation that only changes the instruction text T. Note that the operation of the representative frame selection device 1 is not limited to the operation described in FIG. 3. Hereinafter, the operation of the modification example will be described.

[0065] (Modification Example Operation: Part 1) In FIG. 4, in step S8, the frame selection unit 70 excluded the frame with the lowest importance, recalculated the importance, and selected the representative frame. However, the frame selection unit 70 may select representative frames in order from the highest importance without recalculating the importance. That is, the representative frame selection device 1 may operate as shown in the flowchart of FIG. 5.

[0066] FIG. 5 is a flowchart in which steps S8, S9, and S10 in FIG. 4 are replaced with steps S8A and S10A. The operations in steps S1 to S7 are the same as those described in FIG. 4. In step S8A, the frame selection unit 70 selects C frames from the frame list FL starting from the frame with the highest importance. In step S10A, the frame selection unit 70 uses the frame selected in step S8A as the representative frame and outputs a representative frame list RFL composed of that representative frame.

[0067] (Modified operation: Part 2) Furthermore, in Figure 4, in step S9, the frame selection unit 70 repeatedly excluded frames of low importance until the remaining frames reached the number of selected frames. However, the frame selection unit 70 may also be configured to automatically determine the number of selected frames. In other words, the representative frame selection device 1 may operate according to the flowchart shown in Figure 6.

[0068] Figure 6 is a flowchart in which steps S1 and S9 in Figure 4 are replaced with steps S1A and S9A. In step 1A, the representative frame selection device 1 receives instruction text T from an external source. inst Enter the frame list FL. The operations in steps S2 to S8 are the same as those described in Figure 4. In step S9A, when the frame selection unit 70 removes the frame with the lowest importance from the frame list in step S8, it retains the importance of the frame that was removed earlier and determines whether the amount of change in importance has become greater than a predetermined threshold. Here, if the change in importance is below the threshold (No in step S9A), the representative frame selection device 1 returns to step S2 and creates a new frame list FL. NEW Continue the operation using this method.

[0069] On the other hand, if the change in importance is greater than the threshold (Yes in step S9A), the representative frame selection device 1 proceeds to step S10. Step S10 is the same operation as described in Figure 4. This allows the frame selection unit 70 to automatically determine the number of frames to select.

[0070] <Second Embodiment> [Configuration of the representative frame selection device] Next, with reference to Figure 7, the configuration of the representative frame selection device 1B according to the second embodiment of the present invention will be described.

[0071] The representative frame selection device 1B selects a representative frame from the video data to be selected. Here, the representative frame selection device 1B uses instruction text T inst The frame list TFL of the training video data is input, and a second training model M2 (importance predictor), which is lighter than the training model M (VisionLLM), is trained in advance. Then, the representative frame selection device 1B uses the second learning model M2 to input the frame list FL of the video data to be selected and the number of frames to be selected C, and outputs one or more representative frames as the representative frame list RFL.

[0072] Instruction text T inst This is the same as the input to the representative frame selection device 1 explained in Figure 1. The training video data (frame list TFL) is the video data used to train the second training model M2.

[0073] As shown in Figure 7, the representative frame selection device 1B comprises a model storage unit 10B, a learning data generation unit 50, a model learning unit 60, a frame importance calculation unit 40B, and a frame selection unit 70B.

[0074] The model storage unit 10B stores the learning model M and the second learning model M2. This model storage unit 10B can be made of a general storage medium such as semiconductor memory. The trained model M is the pre-trained VisionLLM as explained in Figure 1. The second learning model M2 is an AI model that takes the frame list learned by the model learning unit 60 (described later) as input and outputs the importance of each frame as an importance list.

[0075] The learning data generation unit 50 generates the importance of each frame corresponding to the learning video data (frame list TFL) as learning data. The learning data generation unit 50 includes a target text generation unit 20, a partial frame list generation unit 30, and a frame importance calculation unit 40.

[0076] The target text generation unit 20, the partial frame list generation unit 30, and the frame importance calculation unit 40 have the same configuration as described in Figure 1. The only difference is whether the input data is video data (frame list FL) or training video data (frame list TFL). Since only the input data is different, the target text generation unit 20, the partial frame list generation unit 30, and the frame importance calculation unit 40 are given the same reference numerals as in Figure 1, and their explanations are omitted. The training data generation unit 50 outputs the generated importance list IL to the model training unit 60.

[0077] The model learning unit 60 trains the second learning model M2 (importance predictor). The model learning unit 60 receives the training video data (frame list TFL) as input and uses the importance list IL generated by the training data generation unit 50 as the ground truth data to optimize the second learning model M2 (specifically its model parameters) in a way that eliminates the error with the output of the second learning model M2 (so that the error is less than or equal to a predetermined value).

[0078] The optimization method for the second learning model M2 can be a general optimization method such as stochastic gradient descent (SGD) or Adam. The loss function used to measure the error can be a general loss function such as mean squared error (MSE) or ranking loss. The model learning unit 60 writes and stores the learned second learning model M2 in the model storage unit 10B.

[0079] Now, referring to Figure 8, we will explain an example of the configuration of the second learning model M2 that the model learning unit 60 learns. The second learning model M2 can be implemented with the configuration shown in Figure 8. The second learning model M2 converts each frame of the video data into a feature vector and outputs an estimated importance value for each frame from a vector created by concatenating the feature vectors of all frames.

[0080] The input to the second learning model M2 is the frames (F1, F2, ..., F) that make up the training video data (frame list TFL). N Here, each frame is an example of a 3-channel RGB color image [3,H,W] with height H pixels × width W pixels. The importance I output by the second learning model M2 is a list (importance list) for the number of input frames (N). The second learning model M2 consists of multiple feature extractors MF, a coupler CN, and an importance extractor MI. The feature extractors MF and importance extractor MI are the models to be trained.

[0081] A feature extractor (MF) converts a frame into a feature vector. This feature extractor (MF) can be constructed, for example, using a common Vision Transformer. The feature extractor MF uses frames F1, F2, ..., F N These are 512-dimensional feature vectors FV1, FV2, ..., FV N Convert to. The feature extractor MF learns sequentially, one for each frame, depending on the number of input frames. However, the model learning unit 60 learns multiple feature extractors MF as a single feature extractor MF by adding or averaging the parameter values ​​of each learned feature extractor MF.

[0082] The coupler CN concatenates the feature vectors of all frames output from the feature extractor MF. Here, the coupler CN concatenates the feature vectors FV1, FV2, ..., FV for N frames. N These are concatenated as a single feature vector FV (here, [N, 512]).

[0083] The importance extractor MI estimates an importance list IL2[N,1] consisting of the importance of each frame from a concatenated feature vector FV of N frames. This importance extractor MI can be constructed, for example, using a general transformer encoder and a multi-layer perceptron (MLP).

[0084] Furthermore, the second learning model M2 is smaller (lighter) than the learning model M because it only takes a frame list (video data) as input and does not use text. As a result, the second learning model M2 can be generated as a model with no limit on the number of input frames, or even one that takes more input frames than the learning model M, and can select representative frames even from video data with a long duration. Returning to Figure 7, we will continue our explanation of the configuration of the representative frame selection device 1B.

[0085] The frame importance calculation unit (second frame importance calculation unit) 40B uses the second learning model M2 (importance predictor) to calculate the importance of each frame from the frame list FL that constitutes the selected video data. In other words, the frame importance calculation unit 40B inputs the frame list FL for each frame into the second learning model M2, as explained in Figure 8, and calculates the importance list IL2, which is the output. The frame importance calculation unit 40B outputs the calculated importance list IL2 to the frame selection unit 70.

[0086] The frame selection unit 70B selects a representative frame from the frame list that constitutes the video data to be selected, based on the importance of each frame (importance list) calculated by the frame importance calculation unit 40B. The frame selection unit 70B selects frames from the frame list FL in order of importance, starting with the most important frames, based on the importance list IL2 calculated by the frame importance calculation unit 40B, and designates these frames as representative frames.

[0087] With the configuration described above, the representative frame selection device 1B can select a representative frame using a second learning model M2 that is smaller in scale and less computationally intensive than the representative frame selection device 1. If you wish to change the criterion for selecting a representative frame, the representative frame selection device 1B will use instruction text T. inst This can be achieved with a simple process of just changing the second learning model M2 and retraining it.

[0088] [Operation of the representative frame selection device] Next, with reference to Figures 9 and 10 (and Figure 7 as appropriate for the configuration), the operation of the representative frame selection device 1B according to the second embodiment of the present invention will be described. First, referring to Figure 9, we will explain the operation of the representative frame selection device 1B in learning the second learning model M2.

[0089] In step S20, the representative frame selection device 1B receives an instruction text T from an external source. inst Input the frame list TFL of the training video data. In step S21, the target text generation unit 20 of the learning data generation unit 50 uses the pre-trained learning model M (VisionLLM) to generate the instruction text T inst The target text is generated from the frame list TFL.

[0090] In step S22, the subframe list generation unit 30 of the learning data generation unit 50 initializes the value of the variable i for counting up the frame list TFL (i=1). In step S23, the partial frame list generation unit 30 generates a partial frame list PFL by excluding the i-th frame from the frame list TFL. i Generates. In step S24, the frame importance calculation unit 40 of the learning data generation unit 50 uses the learning model M to calculate the partial frame list PFL generated in step S23. i And the instruction text T entered in step S20 inst From this, the importance of the i-th frame that was excluded in step S23 is calculated. In step S25, the frame importance calculation unit 40 stores the importance of the i-th frame calculated in step S24 as training data in a memory unit (not shown in the figure). In step S26, the frame importance calculation unit 40 increments the value of variable i (i = i + 1).

[0091] In step S27, the frame importance calculation unit 40 determines whether the value of variable i exceeds the number of frames in the frame list TFL. If the value of variable i has not yet exceeded the number of frames in the frame list TFL (No in step S27), the representative frame selection device 1B returns to step S23 and continues its operation.

[0092] On the other hand, if the value of variable i exceeds the number of frames in the frame list TFL (Yes in step S27), in step S28, the model learning unit 60 takes the frame list TFL of the training video data as input and trains the second learning model M2 (importance predictor) to output the importance of each frame stored as training data in step 25. Then, the model learning unit 60 writes and stores the learned second learning model M2 in the model storage unit 10B. Through the above operations, the representative frame selection device 1B can learn a second learning model M2 that takes video data as input and outputs the importance of each frame.

[0093] Next, referring to Figure 10, we will explain how the representative frame selection device 1B selects a representative frame from the frame list that constitutes the video data using the second learning model M2.

[0094] In step S30, the representative frame selection device 1B receives a frame list FL and the number of frames to be selected from an external source. In step S31, the frame importance calculation unit 40B uses the second learning model M2 stored in the model storage unit 10B to calculate the importance of each frame from the frame list FL input in step S30.

[0095] In step S32, the frame selection unit 70B selects frames numbered C from the frame list FL, starting with the most important frames. In step S33, the frame selection unit 70B uses the frame selected in step S32 as the representative frame and outputs a representative frame list RFL composed of that representative frame. Through the above operations, the representative frame selection device 1B can select a representative frame from the video data according to the scale indicated by the instruction text.

[0096] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and includes design changes and the like that that do not depart from the spirit of the present invention. Furthermore, although the above-described embodiments assumed that the representative frame selection devices 1(1') and 1B are independent hardware, the present invention is not limited thereto. For example, the present invention can also be realized by a program that causes hardware resources such as the CPU, memory, and hard disk of a computer to function as the representative frame selection devices 1(1') and 1B. This program may be distributed via a communication line, or it may be written to a recording medium such as a CD-ROM or flash memory and distributed. [Explanation of Symbols]

[0097] 1.1B Representative Frame Selection Device 10 Model Memory Unit 20 Target Text Generation Unit 21 Learning Model Calculation Unit 22 Decoding section 30. Partial Frame List Generation Unit 40 Frame Importance Calculation Unit 40B Frame Importance Calculation Unit (Second Frame Importance Calculation Unit) 41 Learning Model Calculation Unit 42 Importance calculation part 50. Training Data Generation Unit 60 Model Learning Department 70, 70B Frame Selection Section 80 Sampling section M Learning Model (VisionLLM) M2 Second Learning Model (Importance Predictor)

Claims

1. A representative frame selection device for selecting a representative frame from video data, A target text generation unit takes a frame list and instruction text that constitute the video data as input, and uses a pre-trained learning model to generate target text that indicates the content of the video data corresponding to the instruction text. A subframe list generation unit generates multiple subframe lists by removing one different frame at a time from the aforementioned frame list, A frame importance calculation unit that, for each subframe list, generates a token probability distribution from the subframe list and the instruction text using the learning model, and calculates a frame importance as the importance of frames excluded when generating the subframe list, with the lower the degree of prediction of the target text, the higher the importance. A frame selection unit selects the representative frame from the frame list based on the importance of each frame calculated by the frame importance calculation unit, A representative frame selection device characterized by comprising the following features.

2. The representative frame selection device according to claim 1, characterized in that the frame selection unit removes the frame with the lowest importance from the frame list to form a new frame list, operates the target text generation unit, the partial frame list generation unit, and the frame importance calculation unit sequentially using the new frame list until the number of frames in the new frame list reaches the specified number of selected frames, and uses the frames in the frame list that remain without being removed as the representative frames.

3. The representative frame selection device according to claim 1, characterized in that the frame selection unit selects a specified number of frames from the frame list in order of importance, and designates them as representative frames.

4. The representative frame selection device according to claim 1, characterized in that the frame selection unit removes the frame with the lowest importance from the frame list to form a new frame list, and operates the target text generation unit, the partial frame list generation unit, and the frame importance calculation unit sequentially using the new frame list until the amount of change in importance of the removed frame becomes greater than a predetermined threshold, and the frames in the frame list that remain without being removed are designated as the representative frames.

5. The representative frame selection device according to claim 1, characterized in that it includes a sampling unit for pre-sampling the frame list.

6. A representative frame selection device for selecting a representative frame from video data, A target text generation unit takes a frame list and instruction text as input to the training video data, and uses a pre-trained learning model to generate target text that indicates the content of the training video data corresponding to the instruction text. A subframe list generation unit generates multiple subframe lists by removing one different frame at a time from the aforementioned frame list, The system includes: a frame importance calculation unit that generates a probability distribution of tokens from the subframe list and the instruction text using the learning model for each subframe list, and calculates a frame importance that is higher the lower the degree of prediction of the target text, as the importance of the frame excluded when generating the subframe list; and a learning data generation unit that generates the importance of each frame corresponding to the learning video data as learning data. A model learning unit that learns a second learning model that takes a frame list constituting the aforementioned learning video data as input and outputs the importance of each frame generated by the learning data generation unit, A second frame importance calculation unit calculates the importance of each frame from the frame list constituting the video data using the second learning model, A frame selection unit selects the representative frame from the frame list that constitutes the video data based on the importance of each frame calculated by the second frame importance calculation unit, A representative frame selection device characterized by comprising the following features.

7. A program for causing a computer to function as a representative frame selection device according to any one of claims 1 to 6.

Images