Method and device for generating video description, equipment and storage medium

Abstract

The invention relates to a method and device for generating video description, equipment and a storage medium. The method comprises the following steps: acquiring a long video to be identified; identifying a shot video segment and a scene video segment in the long video; according to the shot video segment and a pre-trained video description generation model, determining shot description information corresponding to the shot video segment; generating scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment; and generating target description information of the long video according to the scene description information. The method can generate description information of the long video.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for generating a video description. Background Art

[0002] In the era of the steady development of the Internet and big data, the demand for multimedia information has exploded. Traditional information processing technology can no longer cope with the needs of multimedia data in tasks such as annotation and description. For example, with the explosive growth of the number of Internet videos, the demand for video description is increasing. Video description is a technology that generates content description information for videos. In the field of artificial intelligence, video description generation models are usually used to automatically generate video descriptions for videos.

[0003] However, due to the limited processing capabilities of current video description generation models, video description generation can only process videos of a few minutes or even a few seconds, and cannot process videos of tens of minutes or even longer. Therefore, a method for generating video descriptions is urgently needed. Summary of the invention

[0004] The present application provides a method, apparatus, device and storage medium for generating video description, which can generate description information of a long video.

[0005] In a first aspect, the present application provides a method for generating a video description, the method comprising:

[0006] Get the long video to be identified;

[0007] Identifying shot video segments and scene video segments in the long video;

[0008] Determining shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model;

[0009] Generate scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment;

[0010] Generate target description information of the long video according to the scene description information.

[0011] Optionally, the identifying the shot video segments in the long video includes:

[0012] Obtaining a pre-trained first model, where the first model is used to identify shot transition points in the long video;

[0013] Inputting the long video into the first model to obtain a long video including at least one shot transition point;

[0014] The long video is segmented according to at least one shot transition point to obtain a plurality of shot video segments.

[0015] Optionally, the identifying the scene video segment in the long video includes:

[0016] Obtaining a pre-trained second model, where the second model is used to identify scene transition points in the long video;

[0017] Inputting the long video into the second model to obtain a long video including at least one scene transition point;

[0018] The long video is segmented according to at least one scene transition point to obtain a plurality of scene video segments.

[0019] Optionally, the determining, according to the shot video segment and a pre-trained video description generation model, shot description information corresponding to the shot video segment includes:

[0020] Obtaining subtitle content corresponding to the shot video segment;

[0021] The shot video segment and the subtitle content are input into a pre-trained video description generation model to obtain shot description information corresponding to the shot video segment.

[0022] Optionally, before obtaining the subtitle content corresponding to the shot video segment, the method further includes:

[0023] Detecting whether the duration of the shot video segment is greater than a preset duration;

[0024] When the duration of the shot video segment is greater than a preset duration, key video frames are extracted from the shot video segment according to preset rules, and the key video frames are combined according to the time point corresponding to each key video frame to obtain a new shot video frame.

[0025] Optionally, the preset rules include:

[0026] Determine an optical flow of a first video frame and a second video frame in the shot video segment, wherein the first video frame and the second video frame are two consecutive video frames;

[0027] Detecting whether the optical flow is greater than a preset value;

[0028] When the optical flow is greater than a preset value, the first video frame and the second video frame are determined as the key video frames.

[0029] Optionally, the preset rules include:

[0030] Detecting whether the shot video segment is a shot video segment of a person;

[0031] When the shot video segment is a shot video segment of a person, sampling the shot video segment using a first video sampling frequency to obtain the key video frame;

[0032] When the shot video segment is not a shot video segment of a person, sampling the shot video segment using a second video sampling frequency to obtain the key video frame;

[0033] The first video sampling frequency is lower than the second video sampling frequency.

[0034] In a second aspect, the present application provides a device for generating a video description, the device comprising:

[0035] An acquisition unit, used for acquiring a long video to be identified;

[0036] An identification unit, used to identify the shot video segments and the scene video segments in the long video;

[0037] A determination unit, configured to determine shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model;

[0038] A first generating unit, configured to generate scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment;

[0039] The second generating unit is used to generate target description information of the long video according to the scene description information.

[0040] Optionally, the identification unit is used to:

[0041] Obtaining a pre-trained first model, where the first model is used to identify shot transition points in the long video;

[0042] Inputting the long video into the first model to obtain a long video including at least one shot transition point;

[0043] The long video is segmented according to at least one shot transition point to obtain a plurality of shot video segments.

[0044] Optionally, the identification unit is used to:

[0045] Obtaining a pre-trained second model, where the second model is used to identify scene transition points in the long video;

[0046] Inputting the long video into the second model to obtain a long video including at least one scene transition point;

[0047] The long video is segmented according to at least one scene transition point to obtain a plurality of scene video segments.

[0048] Optionally, determine the unit for:

[0049] Obtaining subtitle content corresponding to the shot video segment;

[0050] The shot video segment and the subtitle content are input into a pre-trained video description generation model to obtain shot description information corresponding to the shot video segment.

[0051] Optionally, the device further comprises a detection unit, wherein the detection unit is used to:

[0052] Detecting whether the duration of the shot video segment is greater than a preset duration;

[0053] When the duration of the shot video segment is greater than a preset duration, key video frames are extracted from the shot video segment according to preset rules, and the key video frames are combined according to the time point corresponding to each key video frame to obtain a new shot video frame.

[0054] Optionally, the device further includes a first extraction unit, wherein the first extraction unit is configured to:

[0055] Determine an optical flow of a first video frame and a second video frame in the shot video segment, wherein the first video frame and the second video frame are two consecutive video frames;

[0056] Detecting whether the optical flow is greater than a preset value;

[0057] When the optical flow is greater than a preset value, the first video frame and the second video frame are determined as the key video frames.

[0058] Optionally, the device further includes a second extraction unit, wherein the second extraction unit is configured to:

[0059] Detecting whether the shot video segment is a shot video segment of a person;

[0060] When the shot video segment is a shot video segment of a person, sampling the shot video segment using a first video sampling frequency to obtain the key video frame;

[0061] When the shot video segment is not a shot video segment of a person, sampling the shot video segment using a second video sampling frequency to obtain the key video frame;

[0062] The first video sampling frequency is lower than the second video sampling frequency.

[0063] In a third aspect, the present application provides a device for generating a video description, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to:

[0064] Get the long video to be identified;

[0065] Identifying shot video segments and scene video segments in the long video;

[0066] Determining shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model;

[0067] Generate scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment;

[0068] Generate target description information of the long video according to the scene description information.

[0069] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned method for generating a video description when executed by a processor.

[0070] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: in the embodiment of the present application, a long video to be identified is obtained; the shot video segment and the scene video segment in the long video are identified; the shot description information corresponding to the shot video segment is determined according to the shot video segment and the pre-trained video description generation model; the scene description information corresponding to the scene video segment is generated according to the scene video segment and the shot description information corresponding to the shot video segment; the target description information of the long video is generated according to the scene description information. In the embodiment of the present application, since the duration of the shot video segment is generally very short, the video description generation model can be used to generate the shot description information of the shot video segment, and then the description information of the long video is generated based on the shot description information, that is, the present application can generate the description information of the long video. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0073] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0074] Figure 1 A flowchart of a method for generating a video description provided in an embodiment of the present application;

[0075] Figure 2 A flowchart of a method for obtaining a video segment provided in an embodiment of the present application;

[0076] Figure 3 A flowchart of a method for obtaining a video segment provided in an embodiment of the present application;

[0077] Figure 4 A flowchart of a method for generating description information provided in an embodiment of the present application;

[0078] Figure 5 A flowchart of another method for video segment detection provided in an embodiment of the present application;

[0079] Figure 6 A flowchart of a method for determining a key frame provided in an embodiment of the present application;

[0080] Figure 7 A flowchart of another method for determining a key frame provided in an embodiment of the present application;

[0081] Figure 8 A schematic diagram of a flow chart of a device for generating a video description provided in an embodiment of the present application;

[0082] Fig. 9 A schematic diagram of a device for generating video description provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0084] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0085] In the era of the steady development of the Internet and big data, the demand for multimedia information has exploded. Traditional information processing technology can no longer cope with the needs of multimedia data in tasks such as annotation and description. For example, with the explosive growth of the number of Internet videos, the demand for video description is increasing. Video description is a technology that generates content description information for videos. In the field of artificial intelligence, video description generation models are usually used to automatically generate video descriptions for videos.

[0086] In the embodiment of the present application, the long video is segmented by the shot transition points to obtain shot video segments. Since the shot video segments are generally short in duration, the shot description information of the shot video segments can be obtained based on the shot video segments and the pre-stored video description generation model. After that, the target description information of the long video can be generated based on the shot description information. It can be seen that the embodiment of the present application can obtain the description information of the long video.

[0087] In summary, the embodiment of the present application provides a method for generating a video description, which can generate description information of a long video, such as Figure 1 As shown, the specific steps include:

[0088] Step 101, obtaining a long video to be identified.

[0089] The duration of the long video exceeds the preset duration, for example, the duration of the long video exceeds 30 minutes. Specifically, the long video can be a movie, a video recording, or the like.

[0090] In this step, the terminal can obtain the long video to be identified and perform identification processing on the video.

[0091] Step 102: Identify shot video segments and scene video segments in a long video.

[0092] Among them, all video frames in the shot video segment correspond to the same shot, and a shot is a continuous picture segment shot by the camera between startup and shutdown. All video frames in the scene video segment are in the same scene. Since a scene includes at least one shot, a scene video segment includes at least one shot video segment.

[0093] In this step, the terminal can identify at least one shot transition point in the long video, and divide the long video according to the at least one shot transition point to obtain multiple shot video segments. Similarly, the terminal can identify at least one scene transition point in the long video, and divide the long video according to the at least one scene transition point to obtain multiple scene video segments.

[0094] In the above process, the technician can establish a model for identifying shot transition points and a model for identifying scene transition points, and then identify the shot transition points and scene transition points in the long video based on these two models. Alternatively, the technician can mark the shot transition points and scene transition points in the long video based on experience.

[0095] Step 103: Determine shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model.

[0096] The video description generation model is a machine learning model, such as a GPT-4o model. The video description generation model is used to generate description information of the video. The shot description information is the description information generated by the video description generation model based on the shot video segment. The description information is used to describe the video content, for example, to describe the objects in the video and the actions of the characters in the video.

[0097] In this step, the terminal can directly input the shot video segment into a pre-trained video description generation model to obtain shot description information corresponding to the shot video segment.

[0098] In order to prevent the shot video segment from not meeting the input requirements of the video description generation model, the shot video segment can be processed first to reduce the video frames in the shot video segment so that the processed shot video segment can be input into the video description generation model.

[0099] Step 104: Generate scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment.

[0100] The scene description information is generated based on the shot description information in the scene, and is used to describe the information in the scene.

[0101] In this step, since the description information in the same scene is relatively close, the scene description information generated based on the description information of the shot video segment in the same scene can more accurately describe the scene. In order to generate the scene description information, the terminal can sort and combine the shot description information corresponding to each shot video segment according to the order in which each shot video segment appears in the scene video segment to obtain the scene description information corresponding to the scene video segment. Alternatively, a pre-trained first description integration model is provided in the terminal, and the scene description information is input into the first description integration model to obtain the target description information of the long video. This step can also generate the description information of the long video based on other methods, which are not limited here.

[0102] Step 105, generating target description information of the long video according to the scene description information.

[0103] Among them, the target description information is used to describe the entire long video.

[0104] In this step, the terminal can sort and combine the scene description information corresponding to each scene video segment according to the order in which each scene video segment appears in the long video, and obtain the target description information of the long video. Alternatively, the terminal is provided with a pre-trained second description integration model, and the scene description information is input into the second description integration model to obtain the target description information of the long video. This step can also generate the description information of the long video based on other methods, which are not limited here.

[0105] In actual use, if the first description integration model and the second description integration model are involved, the video description generation model, the first description integration model and the second description integration model can be trained together to obtain a trained video description generation model, a trained first description integration model and a trained second description integration model. The specific training process is: the technician pre-sets a sample set, and there are multiple samples in the sample set, each sample includes a sample video and a benchmark description information. A sample is randomly selected from the sample set, and the predicted description information is generated according to the sample video, the video description generation model, the first description integration model and the second description integration model in the sample. According to the degree of difference between the benchmark description information and the predicted description information, these models are adjusted. Afterwards, the adjusted model is trained using other samples in the sample set until the degree of difference between the benchmark description information and the predicted description information meets the preset conditions.

[0106] Of course, the embodiment of the present application can also train the video description generation model, the first description integration model and the second description integration model separately. The training method is similar to the above training method and will not be repeated here.

[0107] In an embodiment of the present application, a long video to be identified is obtained; the shot video segment and the scene video segment in the long video are identified; the shot description information corresponding to the shot video segment is determined based on the shot video segment and a pre-trained video description generation model; the scene description information corresponding to the scene video segment is generated based on the scene video segment and the shot description information corresponding to the shot video segment; and the target description information of the long video is generated based on the scene description information. In an embodiment of the present application, since the duration of the shot video segment is generally very short, the video description generation model can be used to generate the shot description information of the shot video segment, and then the description information of the long video is generated based on the shot description information, that is, the present application can generate the description information of the long video.

[0108] In the embodiment of the present application, a pre-trained first model is stored in the terminal, and then a plurality of lens video segments are obtained according to the first model. Therefore, the embodiment of the present application provides a method for obtaining video segments, which is a further limitation of step 102, such as Figure 2 As shown, the specific steps are:

[0109] Step 201, obtaining a pre-trained first model.

[0110] Among them, the first model is a machine learning model, which is used to identify shot transition points in long videos.

[0111] In this step, a pre-trained first model is stored in the terminal, and when it is necessary to identify shot transition points in a long video, the terminal can obtain the pre-stored first model.

[0112] Furthermore, the embodiment of the present application can also use a sample set to train the first model. The sample set includes multiple samples, each sample includes a sample video and a reference video, and the reference video is a sample video with at least one shot transition point marked. The process of training the first model based on the sample set is: randomly select a sample in the sample set, input the sample video in the sample into the first model, and obtain a predicted video, and the predicted video is a sample video with at least one shot transition point marked. Afterwards, the degree of difference between the predicted video and the reference video is calculated based on the predicted video and the reference video. When the degree of difference does not meet the preset conditions, the first model is adjusted based on the predicted video and the reference video. Then, the adjusted first model is adjusted using other samples in the sample set until the degree of difference between the predicted video and the reference video meets the preset conditions.

[0113] Step 202: input the long video into the first model to obtain the long video including at least one shot transition point.

[0114] In this step, the terminal inputs the long video into the first model so that the first model marks the shot transition points in the long video to obtain a long video including at least one shot transition point.

[0115] Step 203: segment the long video according to at least one shot transition point to obtain a plurality of shot video segments.

[0116] In this step, the terminal may segment the long video according to at least one shot transition point in the long video to obtain a plurality of shot video segments.

[0117] In the embodiment of the present application, a pre-trained second model is stored in the terminal, and then multiple scene video segments are obtained according to the second model. Therefore, the embodiment of the present application provides a video segment acquisition method, which is a further limitation of step 102, such as Figure 3 As shown, the specific steps are:

[0118] Step 301, obtaining a pre-trained second model.

[0119] Among them, the second model is used to identify scene transition points in long videos.

[0120] In this step, a pre-trained second model is stored in the terminal, and when it is necessary to identify scene transition points in a long video, the terminal can obtain the pre-stored second model.

[0121] Furthermore, the embodiment of the present application can also use a sample set to train the second model. The sample set includes multiple samples, each sample includes a sample video and a reference video, and the reference video is a sample video with at least one scene transition point marked. The process of training the second model based on the sample set is: randomly select a sample in the sample set, input the sample video in the sample into the second model, and obtain a predicted video, and the predicted video is a sample video with at least one scene transition point marked. Afterwards, the degree of difference between the predicted video and the reference video is calculated based on the predicted video and the reference video. When the degree of difference does not meet the preset conditions, the second model is adjusted based on the predicted video and the reference video. Then use other samples in the sample set to adjust the adjusted second model until the degree of difference between the predicted video and the reference video meets the preset conditions.

[0122] Step 302: input the long video into the second model to obtain a long video including at least one scene transition point.

[0123] In this step, the terminal inputs the long video into the second model so that the second model marks the scene transition points in the long video to obtain a long video including at least one scene transition point.

[0124] Step 303: segment the long video according to at least one scene transition point to obtain a plurality of scene video segments.

[0125] In this step, the terminal may segment the long video according to at least one scene transition point in the long video to obtain a plurality of scene video segments.

[0126] In the embodiment of the present application, in order to make the output shot description information more accurate, the terminal also needs the subtitle content of the shot video segment, and inputs the shot video segment and the subtitle content into the pre-trained video description generation model to determine the shot description information corresponding to the shot video segment. Therefore, the embodiment of the present application provides a method for generating description information, which is a further limitation of step 103, such as Figure 4 As shown, the specific steps include:

[0127] Step 401, obtaining the subtitle content corresponding to the shot video segment.

[0128] In this step, the terminal can use OCR technology or ASR technology to recognize the text in the lens video segment and obtain the subtitle content corresponding to the lens video segment. Alternatively, the terminal uses OCR technology or ASR technology in advance to recognize the long video and obtain the subtitle content of the long video. When the terminal needs to obtain the subtitle content corresponding to the lens video segment, the terminal intercepts the subtitle content corresponding to the long video according to the position of the lens video segment in the long video to obtain the subtitle content corresponding to the lens video segment.

[0129] Step 402: Input the shot video segment and the subtitle content into a pre-trained video description generation model to obtain shot description information corresponding to the shot video segment.

[0130] In this step, the terminal may input the shot video segment and the subtitle content into a pre-trained video description generation model to obtain shot description information corresponding to the shot video segment.

[0131] In the embodiment of the present application, before obtaining the lines corresponding to the shot video segment, the terminal can also detect whether the shot video segment can be input into the video description generation model. When the shot video segment cannot be input into the video description generation model, a new shot video segment is generated based on the shot video segment. Therefore, the embodiment of the present application provides a method for video segment detection. Figure 5 As shown, the specific steps include:

[0132] Step 501, detecting whether the duration of the shot video segment is greater than a preset duration.

[0133] Among them, the preset duration can be set according to the input requirements of the video description generation model. Specifically, the maximum duration and frame rate of the input video of the video description generation model are obtained, and the number of video frames that can be input is calculated according to the maximum duration and frame rate. Then, the preset duration is calculated according to the number of video frames and the frame rate of the long video to be detected.

[0134] In this step, in order to determine whether the shot video segment can be input into the video description generation model, it is necessary to pre-detect whether the duration of the shot video segment is greater than the preset duration. If not, it means that the shot video segment can be input into the video description generation model. If yes, it means that the shot video segment cannot be input into the video description generation model and the shot video segment needs to be processed first.

[0135] Step 502, when the duration of the shot video segment is greater than the preset duration, key video frames are extracted from the shot video segment according to preset rules, and the key video frames are combined according to the time point corresponding to each key video frame to obtain a new shot video frame.

[0136] In this step, when the duration of the shot video segment is greater than the preset duration, the terminal reduces the video frames in the shot video segment so that the shot video segment meets the input requirements of the video description generation model. The specific steps are as follows: the terminal extracts video frames with relatively important content in the shot video segment according to the preset rules, uses them as key frames, and combines the key video frames according to the time points corresponding to each key video frame to obtain a new shot video frame.

[0137] In the embodiment of the present application, the terminal can also detect the optical flow of two consecutive video frames, and then use the optical flow to determine the degree of difference between the two video frames. When the terminal detects that the content of the two video frames is too different, the terminal can obtain the two video frames and determine the two video frames as key video frames. Therefore, the embodiment of the present application provides a method for determining key frames, such as Figure 6 As shown, the specific steps include:

[0138] Step 601: determine the optical flows of a first video frame and a second video frame in a shot video segment.

[0139] Optical flow defines the movement of objects in an image, which can be caused by camera movement or object movement. Specifically, it refers to the movement of pixels representing the same object in a certain video frame to the next frame. The first video frame and the second video frame are any two consecutive video frames in the video segment of the shot.

[0140] In this step, for any two consecutive video frames in the lens video segment, the terminal may determine the video frame at the earlier time point as the first video frame, and the video frame at the later time point as the second video frame. Of course, the terminal may also determine the video frame at the later time point as the first video frame, and the video frame at the earlier time point as the second video frame. Afterwards, the terminal may use the optical flow calculation method to calculate the optical flow of the first video frame and the second video frame.

[0141] Step 602, detecting whether the optical flow is greater than a preset value.

[0142] In this step, the preset value can be set by a technician based on experience.

[0143] In this step, the terminal may compare the optical flow with a preset value to determine whether the optical flow is greater than the preset value.

[0144] Step 603: When the optical flow is greater than a preset value, the first video frame and the second video frame are determined as key video frames.

[0145] In this step, when the terminal determines that the optical flow is greater than a preset value, the terminal may extract the two video frames and determine them as key video frames.

[0146] In the embodiment of the present application, the video frames in the shot video segment are under the same shot, and when the shot video segment is a close-up of a person, the close-up of the person generally only undergoes slight changes. For such a shot video segment, a lower video sampling frequency can be used to sample it. For other shot video segments, a higher video sampling frequency can be used to sample them. Therefore, the embodiment of the present application provides a method for determining key frames, such as Figure 7 As shown, the specific steps include:

[0147] Step 701: Detect whether the shot video segment is a shot video segment of a person.

[0148] In this step, since the area occupied by the person in the close-up shot of the person is relatively large, it is possible to detect whether there is a person in the lens video segment and whether the area occupied by the person exceeds a preset value. If so, it is determined that there is a close-up shot of the person in the lens video segment, and if not, it is determined that there is no close-up shot of the person in the lens video segment. Alternatively, after obtaining the lens video segment, it is manually determined whether there is a close-up shot of the person in the lens video segment. Afterwards, the terminal determines whether the lens video segment is a lens video segment of the person by detecting whether there is a close-up shot of the person in the lens video segment.

[0149] Step 702: When the shot video segment is a shot video segment of a person, the shot video segment is sampled using a first video sampling frequency to obtain a key video frame.

[0150] In this step, when the shot video segment is a character shot video segment, it means that generally only slight changes will occur between the video frames included in the video segment. The terminal can use a lower video sampling frequency, that is, the first video sampling frequency, to sample the shot video segment to obtain key video frames.

[0151] For example, when the shot video segment is a shot video segment of a person, the shot video segment is sampled once every 1 second.

[0152] Step 703: When the shot video segment is not a shot video segment of a person, the shot video segment is sampled using the second video sampling frequency to obtain a key video frame.

[0153] The first video sampling frequency is lower than the second video sampling frequency.

[0154] In this step, when the shot video segment is not a character shot video segment, it means that the changes between the video frames included in the video segment are relatively large. The terminal can use a higher video sampling frequency, that is, the second video sampling frequency, to sample the shot video segment to obtain a key video frame.

[0155] For example, when the shot video segment is not a shot video segment of a person, the shot video segment is sampled once every 4 seconds.

[0156] like Figure 8 As shown, the embodiment of the present application provides a device for generating a video description, which corresponds to the method embodiment and specifically includes:

[0157] An acquisition unit 801 is used to acquire a long video to be identified;

[0158] An identification unit 802, configured to identify a shot video segment and a scene video segment in the long video;

[0159] A determining unit 803, configured to determine shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model;

[0160] A first generating unit 804 is configured to generate scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment;

[0161] The second generating unit 805 is used to generate target description information of the long video according to the scene description information.

[0162] Optionally, the identification unit 802 is used to:

[0163] Obtaining a pre-trained first model, where the first model is used to identify shot transition points in the long video;

[0164] Inputting the long video into the first model to obtain a long video including at least one shot transition point;

[0165] The long video is segmented according to at least one shot transition point to obtain a plurality of shot video segments.

[0166] Optionally, the identification unit 802 is used to:

[0167] Obtaining a pre-trained second model, where the second model is used to identify scene transition points in the long video;

[0168] Inputting the long video into the second model to obtain a long video including at least one scene transition point;

[0169] The long video is segmented according to at least one scene transition point to obtain a plurality of scene video segments.

[0170] Optionally, the determining unit 803 is configured to:

[0171] Obtaining subtitle content corresponding to the shot video segment;

[0172] The shot video segment and the subtitle content are input into a pre-trained video description generation model to obtain shot description information corresponding to the shot video segment.

[0173] Optionally, the device further includes a detection unit 806, and the detection unit 806 is used to:

[0174] Detecting whether the duration of the shot video segment is greater than a preset duration;

[0175] When the duration of the shot video segment is greater than a preset duration, key video frames are extracted from the shot video segment according to preset rules, and the key video frames are combined according to the time point corresponding to each key video frame to obtain a new shot video frame.

[0176] Optionally, the device further includes a first extraction unit 808, and the first extraction unit 808 is used to:

[0177] Determine an optical flow of a first video frame and a second video frame in the shot video segment, wherein the first video frame and the second video frame are two consecutive video frames;

[0178] Detecting whether the optical flow is greater than a preset value;

[0179] When the optical flow is greater than a preset value, the first video frame and the second video frame are determined as the key video frames.

[0180] Optionally, the device further includes a second extraction unit 807, and the second extraction unit 807 is used to:

[0181] Detecting whether the shot video segment is a shot video segment of a person;

[0182] When the shot video segment is a shot video segment of a person, sampling the shot video segment using a first video sampling frequency to obtain the key video frame;

[0183] When the shot video segment is not a shot video segment of a person, sampling the shot video segment using a second video sampling frequency to obtain the key video frame;

[0184] The first video sampling frequency is lower than the second video sampling frequency.

[0185] like Fig. 9 As shown, the embodiment of the present application provides a device for generating video description, including a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0186] Memory 903, used for storing computer programs;

[0187] In one embodiment of the present application, the processor 901 is configured to execute a program stored in the memory 903 to implement the method for generating a video description provided by any of the aforementioned method embodiments, including:

[0188] Get the long video to be identified;

[0189] Identifying shot video segments and scene video segments in the long video;

[0190] Determining shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model;

[0191] Generate scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment;

[0192] Generate target description information of the long video according to the scene description information.

[0193] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for generating a video description provided in any of the aforementioned method embodiments are implemented.

[0194] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0196] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0197] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating a video description, characterized in that: The method comprises: Get the long video to be identified; Identifying shot video segments and scene video segments in the long video; Determining shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model; Generate scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment; Generate target description information of the long video according to the scene description information.

2. The method according to claim 1, characterized in that: Identifying the shot video segments in the long video includes: Obtaining a pre-trained first model, where the first model is used to identify shot transition points in the long video; Inputting the long video into the first model to obtain a long video including at least one shot transition point; The long video is segmented according to at least one shot transition point to obtain a plurality of shot video segments.

3. The method according to claim 1, characterized in that: Identifying a scene video segment in the long video, including: Obtaining a pre-trained second model, where the second model is used to identify scene transition points in the long video; Inputting the long video into the second model to obtain a long video including at least one scene transition point; The long video is segmented according to at least one scene transition point to obtain a plurality of scene video segments.

4. The method according to claim 1, characterized in that: The step of determining shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model includes: Obtaining subtitle content corresponding to the shot video segment; The shot video segment and the subtitle content are input into a pre-trained video description generation model to obtain shot description information corresponding to the shot video segment.

5. The method according to claim 4, characterized in that: Before obtaining the subtitle content corresponding to the shot video segment, the method further includes: Detecting whether the duration of the shot video segment is greater than a preset duration; When the duration of the shot video segment is greater than a preset duration, key video frames are extracted from the shot video segment according to preset rules, and the key video frames are combined according to the time point corresponding to each key video frame to obtain a new shot video frame.

6. The method according to claim 5, characterized in that: The preset rules include: Determine an optical flow of a first video frame and a second video frame in the shot video segment, wherein the first video frame and the second video frame are two consecutive video frames; Detecting whether the optical flow is greater than a preset value; When the optical flow is greater than a preset value, the first video frame and the second video frame are determined as the key video frames.

7. The method according to claim 5, characterized in that: The preset rules include: Detecting whether the shot video segment is a shot video segment of a person; When the shot video segment is a shot video segment of a person, sampling the shot video segment using a first video sampling frequency to obtain the key video frame; When the shot video segment is not a shot video segment of a person, sampling the shot video segment using a second video sampling frequency to obtain the key video frame; The first video sampling frequency is lower than the second video sampling frequency.

8. A device for generating a video description, characterized in that: The device comprises: An acquisition unit, used for acquiring a long video to be identified; An identification unit, used to identify the shot video segments and the scene video segments in the long video; A determination unit, configured to determine shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model; A first generating unit, configured to generate scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment; The second generating unit is used to generate target description information of the long video according to the scene description information.

9. A device for generating a video description, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to: Get the long video to be identified; Identifying shot video segments and scene video segments in the long video; Determining shot description information corresponding to the shot video segment according to the shot video segment and a pre-trained video description generation model; Generate scene description information corresponding to the scene video segment according to the scene video segment and the shot description information corresponding to the shot video segment; Generate target description information of the long video according to the scene description information.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a video description according to any one of claims 1 to 7 is implemented.

Images