Video description generation method and system based on grid chart, and storage medium

Abstract

The invention provides a grid chart-based video description generation method. The grid chart-based video description generation method comprises the following steps: S101, acquiring an original video V; s102, extracting k frames of first images from the obtained original video V at equal intervals; wherein k should meet the square root; the size of each frame of first image is W * H * C; s103, an empty graph I with the size of # imgabs0 # W * # imgabs1 # H * C is constructed, and the empty graph I is divided into # imgabs2 # blocks with the size of W * H * C; s104, the k frames of first images are sequentially placed in k blocks with the size of W * H * C of the blank image I from left to right and from top to bottom, and second images with the size of # imgabs3 # W * # imgabs4 # H * C are obtained; s105, adjusting the obtained second image with the size of # imgabs5 # W * # imgabs6 # H * C into a third image with the size of T * T * C; and S106, inputting the third image of which the size is T * T * C and the generation type text into the LVLM model, and outputting the generated video description. According to the method, the calculation complexity of video description generation is reduced, and the generation quality of the video description is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of video description, and in particular to a method, system and storage medium for generating video description based on a grid graph. Background Art

[0002] Video description generation refers to the process of automatically generating text descriptions for videos by computers. Existing video description generation solutions are all based on the LVLM model (Large Visual Language Model), that is, the video frame is fed into the LVLM model and a generated text promptP is attached. The LVLM model will output the corresponding video description.

[0003] The general steps of existing video description generation schemes are as follows: Extract from video k Frame image, each frame image size is W × H × C , adjust each frame image to T × T × C The image ( T Usually 336 / 448), and spliced ​​into a video frame sequence, the size is k × T × T × C The video image is used as the input of the LVLM model, and then k × T × T × C The video image and the generated class text promptP are input into the LVLM model to obtain the generated video description.

[0004] Existing video description generation schemes have the following defects: Extract from video k Frame images and stitch them into k × T × T × C The video image is used as the input of the LVLM model. As the number of frames extracted k The increase will lead to a significant increase in the input data of the LVLM model, which will greatly increase the computational complexity, computing resource consumption, and long generation time. k images will cause the time span between frames to be too long, resulting in incoherent information between frames. Summary of the invention

[0005] The present invention proposes a video description generation method, system and storage medium based on a grid graph to solve the technical problems of existing video description generation computational complexity, computational resource consumption, long generation time and incoherent information between frames.

[0006] One aspect of the present invention is to provide a method for generating video description based on a grid graph, the method comprising the following steps: S101. Obtaining original video V ; S102, from the original video obtained V Medium pitch extraction k Frame first image; wherein, k It should satisfy the square root; The size of the first image in each frame is W × H × C ;in, W Indicates the length of the first image in each frame, H Indicates the width of the first image in each frame, C Represents the original video V The number of channels; S103, using the extracted k Frame first image, calculate the original video V If the optical flow score is lower than the preset threshold, the original video is discarded. V , re-obtain the original video V ; If the optical flow score is higher than the preset threshold, a size of W × H × C Empty map I , the empty map I Divide into The size is W × H × C of blocks; among them, k The original video obtained from V The number of frames of the first image extracted with medium spacing; S104, will k The first image of the frame is placed in the empty image in order from left to right and from top to bottom. I of k The size is W × H × C In the block of W × H ×C a second image of S105, will be the size of W × H × C The second image is resized to T × T × C A third image of ; wherein, T represents the length and width of the third image; S106, the size is T × T × C The third image and the generated class text are input into the LVLM model together, and the generated video description is output.

[0007] In a preferred embodiment, in S102, the original video is obtained. V Medium pitch extraction k =9 frames or k =16 frames of the first image.

[0008] In a preferred embodiment, it is characterized in that, in S102, the original video is obtained V Medium pitch extraction k =25 frames of the first image.

[0009] In a preferred embodiment, in step S103, the original video is calculated by the following method: V The optical flow score is: S1031, extract k The first image of the frame is input into the deep learning model for optical flow calculation, and we get k-1 The frame size is W × H ×2 optical flow feature map; in, W Indicates the length of the first image in each frame, H Indicates the width of the first image of each frame. The number of channels of the optical flow feature map is 2, which respectively represents the horizontal and vertical directions of the optical flow feature map; S1032, k-1 The frame size is W × H ×2 optical flow feature map is normalized to [-1,1], k-1 The weighted average of the optical flow feature map after frame normalization is calculated to obtain the original video V The optical flow score.

[0010] In a preferred embodiment, in step S1032, a weighted average value is calculated for the normalized optical flow feature map by the following method: right k-1 The positions greater than 0 in the optical flow feature map after frame normalization are multiplied by a first coefficient α greater than 1. k- 1 The positions less than 0 in the optical flow feature map after frame normalization are multiplied by a second coefficient β less than 1 to obtain k-1 The weighted average of the normalized optical flow feature maps of the frames.

[0011] In a preferred embodiment, in step S1032, directly calculate k-1 The average value of the optical flow feature map after frame normalization is used to obtain the original video V The optical flow score.

[0012] In a preferred embodiment, in step S103, the coordinates are used to represent the empty image in order from left to right and from top to bottom. I of k The size is W × H × C The location of the block.

[0013] Another aspect of the present invention is to provide a grid-graph-based video description generation system, wherein the video description generation system is used to execute a grid-graph-based video description generation method provided by the present invention.

[0014] Another aspect of the present invention is to provide a computer storage medium, wherein the computer storage medium is used to store computer execution instructions, and the computer execution instructions are used to execute a video description generation method based on a grid graph provided by the present invention.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a method, system and storage medium for generating video description based on a grid graph, by constructing an empty graph I , the empty map I Divide into The size is W × H × C The blocks will be extracted k The first image of the frame is placed in the empty image in order from left to right and from top to bottom. I of k The size is W × H × C In blocks of size W × H × C The second grid-like image will be of size W × H × C The second grid-like image is resized to 1 image size T × T × C The third image (video image) is input into the LVLM model, so that the size of the third image (video image) becomes 1× T × T × C , the computational cost is calculated from the original k video images are reduced to 1 / k video images, greatly reducing the computational complexity, computing resource consumption and generation time of video description generation.

[0016] The present invention proposes a method, system and storage medium for generating video description based on a grid graph, because the extracted k The first image of the frame is placed in the empty image in order from left to right and from top to bottom. I of k The size is W × H × C In blocks of size W × H × C The grid-shaped second image is made so that the number of frames of the first image is extracted k In the case of less, it can also ensure that the span time between frames is reduced and the incoherence of inter-frame information is avoided, so that the size is W × H × C The second grid-like image is resized to 1 image size T × T × C The third image is used as the input of the LVLM model, which reduces the computational complexity, computing resource consumption and generation time of video description generation, while retaining the video information as much as possible, ensuring the generation quality of the video description.

[0017] The present invention proposes a method, system and storage medium for generating video description based on a grid graph, using the extracted k The optical flow score of the original video is calculated for the first image of the frame. The original video without obvious motion is eliminated through the optical flow score of the original video, further increasing the quality of the original video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the following briefly introduces the drawings required for use in the specific implementation or the prior art description. Obviously, the drawings described below are some implementations of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a flow chart of a method for generating video description based on a grid graph of the present invention. DETAILED DESCRIPTION

[0020] In order to make the above and other features and advantages of the present invention more clear, the present invention is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0021] Combination Figure 1 According to an embodiment of the present invention, a method for generating video description based on a grid graph is provided. Video description generation refers to the process in which a computer automatically generates a text description of a video.

[0022] The present invention provides a method for generating video description based on a grid graph, comprising the following steps: Step S101: Obtaining the original video V .

[0023] Step S102: From the original video V Medium pitch extraction k Frame first image; wherein, k It should satisfy the square root.

[0024] The size of the first image in each frame is W × H × C ;in, W Indicates the length of the first image in each frame, H Indicates the width of the first image in each frame, C Represents the original video V The number of channels of the original video. V Number of channels C= 3.

[0025] In one embodiment, the original video is obtained from V Medium pitch extraction k =9 frames or k = 16 frames of the first image. The number of frames of the first image extracted k =9 frames or k=16 frames all satisfy the square root requirement.

[0026] In this embodiment, the original video obtained is exemplified V Medium pitch extraction k =9 frames of the first image, such as Figure 1 As shown, the extracted k =9 frames of first images are: the 1st frame first image, the 2nd frame first image, the 3rd frame first image, the 4th frame first image, the 5th frame first image, the 6th frame first image, the 7th frame first image, the 8th frame first image and the 9th frame first image.

[0027] In another embodiment, in order to reduce the span time between frames and avoid inter-frame information incoherence, it is preferred to obtain the original video V Medium pitch extraction k = 25 frames of the first image. The number of frames of the first image extracted k =25 frames satisfy the square root.

[0028] Step S103: Using the extracted k Frame first image, calculate the original video V The optical flow score.

[0029] Specifically, the original video is calculated by the following method V The optical flow score is: Step S1031: extract k The first image of the frame is input into the deep learning model for optical flow calculation, and we get k-1 The frame size is W × H ×2 optical flow feature map G (( k-1 )× W × H ×2 optical flow feature map G ).

[0030] in, W Indicates the length of the first image of each frame (optical flow feature map of each frame G length), H Indicates the width of the first image of each frame (the optical flow feature map of each frame G width), optical flow feature map G The number of channels is 2, representing the optical flow feature map G both horizontal and vertical directions.

[0031] The deep learning model of the present invention is a deep learning model for optical flow estimation. The specific deep learning model can be reasonably selected by those skilled in the art according to the specific circumstances.

[0032] Step S1032:k-1 The frame size is W × H ×2 optical flow feature map G Normalized to [-1,1], k-1 The weighted average of the optical flow feature map after frame normalization is calculated to obtain the original video V The optical flow score.

[0033] Furthermore, the weighted average value of the normalized optical flow feature map is calculated by the following method: right k-1 The positions greater than 0 in the optical flow feature map after frame normalization are multiplied by a first coefficient α greater than 1. k- 1 The positions less than 0 in the optical flow feature map after frame normalization are multiplied by a second coefficient β less than 1 to obtain k-1 The weighted average of the normalized optical flow feature maps of the frames.

[0034] Preferably, the value of the first coefficient α is α=1.05; the value of the second coefficient β is β=0.95.

[0035] Since there is no obvious motion in the original video V of k The first image of the frame is basically the same, lacking the temporal characteristics of the video modality. The present invention uses optical flow to represent the original video. V Motion information between frames, original video V have k The first image of the frame has k -1 frame optical flow feature map G , the larger the optical flow value, the more motion information the pixel has. k -1 frame optical flow feature map G Calculate the original video V The optical flow score of the original video is V The whole has certain motion characteristics. k The optical flow score of the original video is calculated for the first image of the frame. The original video without obvious motion is eliminated through the optical flow score of the original video, further increasing the quality of the original video data.

[0036] The present invention performs weighted average calculation on the normalized optical flow feature map to make the original video V The weight of the motion information of the original video is increased to more accurately evaluate the original video V of sportiness.

[0037] In some other embodiments, the original video is calculated V When the optical flow fraction is k-1The optical flow feature map after frame normalization can be directly calculated without calculating the weighted average k-1 The average value of the optical flow feature map after frame normalization is used to obtain the original video V The optical flow score.

[0038] According to an embodiment of the present invention, if the optical flow score is lower than a preset threshold γ , the original video is discarded V , re-obtain the original video V。

[0039] If the optical flow score is higher than the preset threshold γ , then construct a size of W × H × C Empty map I , the empty map I Divide into The size is W × H × C of blocks; among them, k The original video obtained from V The frame number of the first image extracted with equal spacing.

[0040] like Figure 1 As shown, in this embodiment, the original video obtained V Medium pitch extraction k = 9 frames of the first image, thereby constructing a size of 3 W ×3 H × C Empty map I , that is, the empty graph I The length is 3 W , with a width of 3 H .

[0041] The empty graph that will be constructed I Divided into 9 sizes W × H × C Blocks, using coordinates from left to right and from top to bottom to represent empty graphs I of k =9 size W × H × C The location of the block.

[0042] In this embodiment, 9 sizes are W × H × CThe blocks are: block (1), block (2), block (3), block (4), block (5), block (6), block (7), block (8) and block (9). Figure 1 shown.

[0043] Step S104: k The first image of the frame is placed in the empty image in order from left to right and from top to bottom. I of k The size is W × H × C In the block of W × H × C The second image (grid map).

[0044] In this embodiment, k = The first image of the 9 frames is placed in the blank image from left to right and from top to bottom. I of k =9 size W × H × C in the block.

[0045] That is, the first image of the first frame is placed in block (1), the first image of the second frame is placed in block (2), the first image of the third frame is placed in block (3), the first image of the fourth frame is placed in block (4), the first image of the fifth frame is placed in block (5), the first image of the sixth frame is placed in block (6), the first image of the seventh frame is placed in block (7), the first image of the eighth frame is placed in block (8), and the first image of the ninth frame is placed in block (9), resulting in a size of 3 W ×3 H × C The second image (grid map) is as follows Figure 1 shown.

[0046] The present invention will extract k The first image of the frame is placed in the empty image in order from left to right and from top to bottom. I of k The size is W × H × C In blocks of size W × H × C The second image (grid map) has the same number of grids in length and width. The second image (grid map) contains the extracted kThe first image of the frame contains all the information, and the second image (grid map) contains the same number of grids in length and width. k The timing information of the first image of the frame makes the number of frames of the first image extracted k In less cases, it can also ensure that the span time between frames is reduced and the incoherence of information between frames is avoided.

[0047] Step S105: Get the size of W × H × C The second image is resized to T × T × C A third image of ; wherein, T Indicates the length and width of the third image.

[0048] In this embodiment, the size obtained is 3 W ×3 H × C The second image is resized to T × T × C The third image ( T Usually 336 / 448).

[0049] Step S106: T × T × C The third image and the generated class text are input into the LVLM model together, and the generated video description is output.

[0050] For example, in this embodiment, the generated text promptP input into the LVLM model is: Please generate the video description.

[0051] The size obtained in step S105 is T × T × C The third image and the generated class text promptP (for example: please generate a description of this video) are input into the LVLM model, and the generated video description is output.

[0052] The present invention will extract k The first image of the frame is placed in the empty image in order from left to right and from top to bottom. I of k The size is W × H × C In blocks of size W × H ×C The second grid-like image (grid map) will be of size W × H × C The second grid-like image is resized to 1 image size T × T × C The third image (video image) is input into the LVLM model, so that the size of the third image (video image) becomes 1× T × T × C , in a third image (video image) also contains the extracted k All information of the first image of the frame and k The timing information of the first image of the frame, the calculation cost is calculated by the original k video images are reduced to 1 / k video images, greatly reducing the computational complexity, computing resource consumption and generation time of video description generation.

[0053] In some embodiments, if the original video is obtained V The number of frames of the first image extracted with medium spacing k cannot be squared, for example k =8, then k = 8 frames of the first image are directly stitched in sequence, and the size is kW × H × C or W × k × C The second image will be of size kW × H × C or W × k × C The second image is resized to T × T × C The third image (video image) is input into the LVLM model. At this time, the computational complexity, computational resource consumption and generation time of video description generation can also be reduced. However, since the second image is not a grid map, it may cause a long span time between frames and incoherent information between frames.

[0054] It should be noted that the present invention obtains the original video V The number of frames of the first image extracted with medium spacing k The case where the square root cannot be taken (e.g. k =8) is a second-best solution. The best solution should be to obtain the original videoV The number of frames of the first image extracted with medium spacing k Should satisfy the square root (e.g. k =9), so that the extracted k The first image of the frame is placed in the empty image in order from left to right and from top to bottom. I of k The size is W × H × C In blocks of size W × H × C The second grid-like image (grid map) is of size W × H × C The second grid-like image is resized to 1 image size T × T × C The third image is used as the input of the LVLM model, which reduces the computational complexity, computing resource consumption and generation time of video description generation, while retaining the video information as much as possible, ensuring the generation quality of the video description.

[0055] According to an embodiment of the present invention, a grid graph-based video description generation system is provided, which is used to execute a grid graph-based video description generation method provided by the present invention.

[0056] According to an embodiment of the present invention, a computer storage medium is provided for storing computer execution instructions, and the computer execution instructions are used to execute a video description generation method based on a grid graph provided by the present invention.

[0057] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A video description generation method based on a grid graph, characterized in that: The video description generation method comprises the following steps: S101. Obtaining original video V ; S102, from the original video obtained V Medium pitch extraction k Frame first image; in, k It should satisfy the square root; The size of the first image in each frame is W × H × C ;in, W Indicates the length of the first image in each frame, H Indicates the width of the first image in each frame, C Represents the original video V The number of channels; S103, using the extracted k Frame first image, calculate the original video V The optical flow score of If the optical flow score is lower than the preset threshold, the original video is discarded V , re-obtain the original video V ; If the optical flow score is higher than the preset threshold, a size W × H × C Empty map I , the empty map I Divide into The size is W × H × C of blocks; among them, k The original video obtained from V The number of frames of the first image extracted with medium spacing; S104, will k The first image of the frame is placed in the empty image in order from left to right and from top to bottom. I of k The size is W × H × C In the block of W × H × C a second image of S105, the size is W × H × C The second image is resized to T × T × C A third image of ; wherein, T represents the length and width of the third image; S106, the size is T × T × C The third image and the generated class text are input into the LVLM model together, and the generated video description is output.

2. The video description generation method according to claim 1, characterized in that: In S102, the original video is obtained V Medium pitch extraction k =9 frames or k =16 frames of the first image.

3. The video description generation method according to claim 1, characterized in that: In S102, the original video is obtained V Medium pitch extraction k =25 frames of the first image.

4. The video description generation method according to claim 1, characterized in that: In step S103, the original video is calculated by the following method V The optical flow score is: S1031, extract k The first image of the frame is input into the deep learning model for optical flow calculation, and we get k-1 The frame size is W × H ×2 optical flow feature map; in, W Indicates the length of the first image in each frame, H Indicates the width of the first image of each frame. The number of channels of the optical flow feature map is 2, which respectively represents the horizontal and vertical directions of the optical flow feature map; S1032, k-1 The frame size is W × H ×2 optical flow feature map is normalized to [-1,1], k-1 The weighted average of the optical flow feature map after frame normalization is calculated to obtain the original video V The optical flow score.

5. The video description generation method according to claim 4, characterized in that: In step S1032, a weighted average value is calculated for the normalized optical flow feature map by the following method: right k-1 The positions greater than 0 in the optical flow feature map after frame normalization are multiplied by a first coefficient α greater than 1. k-1 The positions less than 0 in the optical flow feature map after frame normalization are multiplied by a second coefficient β less than 1 to obtain k-1 The weighted average of the normalized optical flow feature maps of the frames.

6. The video description generation method according to claim 4, characterized in that: In step S1032, directly calculate k- 1 The average value of the optical flow feature map after frame normalization is used to obtain the original video V The optical flow score.

7. The video description generation method according to claim 1, characterized in that: In step S103, the coordinates are used to represent the empty image from left to right and from top to bottom. I of k The size is W × H × C The location of the block.

8. A video description generation system based on a grid graph, characterized in that: The video description generation system is used to execute the video description generation method according to any one of claims 1 to 7.

9. A computer storage medium, characterized in that: The computer storage medium is used to store computer-executable instructions, and the computer-executable instructions are used to execute the video description generation method according to any one of claims 1 to 7.

Images