Video end recognition method, device, electronic device and readable storage medium

Abstract

The present invention provides a method, device, electronic device and readable storage medium for video end recognition, and belongs to the field of computer technology. The method obtains text detection data of a preset text area of ​​a video object; obtains the number of consecutive frames of the text detection data in a preset playback time period of the video object; if the number of consecutive frames of the text data determines that the preset playback time period scene is a multi-line or single-line text data display scene; and determines the playback time period where the consecutive frames are located as the end appearance time period; and determines the start time of the end of the video object according to the deep learning features of each frame image in the end appearance time period. The method realizes automatic recognition of the start time of the end of the video data according to the frame-level change features of the text data in the video, reduces the labor cost of manually marking the location of the video end, and thus significantly improves the efficiency of end recognition.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a video end recognition method, device, electronic device and readable storage medium. Background Art

[0002] In the video content playback scenario on the video playback platform, most users often choose to automatically skip the end of the video content. Therefore, a method that can accurately identify the end position of the video content is particularly important on the video playback platform.

[0003] In the prior art, video content is usually manually annotated with the location of the video ending. However, for a large number of video endings of various types, using the manual annotation method one by one is very time-consuming, which undoubtedly has the problems of high cost and low efficiency for massive video content. Summary of the invention

[0004] The present invention provides a video end credits recognition method, device, electronic device and readable storage medium, which to a certain extent solve the problem of high cost and low efficiency caused by manually setting the beginning and ending positions of massive video data.

[0005] According to a first aspect of the present invention, a method for identifying a video end is provided, the method comprising:

[0006] Obtain text detection data of a preset text area of ​​a video object;

[0007] Acquire the number of consecutive frames of the text detection data during a preset playback time period of the video object;

[0008] If the number of consecutive frames of the text data is less than the frame number threshold, the preset playback time period scene is determined to be a single-line text data display scene; otherwise, the preset playback time period scene is determined to be a multi-line text data display scene; in the single-line text data display scene, the playback time period where the consecutive frames are located is determined to be the end credits appearance time period; in the multi-line text data display scene, the playback time period where the multi-line text data appears is determined to be the end credits appearance time period;

[0009] Determine the start time of the ending of the video object according to the deep learning features of each frame image in the time period when the ending appears.

[0010] According to a second aspect of the present invention, a video end recognition device is provided, the device comprising:

[0011] A text detection data acquisition module, used to acquire text detection data of a preset text area of ​​a video object;

[0012] A continuous frame number determination module, used to obtain the continuous frame number of the text detection data in a preset playback time period of the video object;

[0013] The module for determining the time period for the appearance of the ending credits is used to determine that the preset playback time period scenario is a single-line text data display scenario if the number of consecutive frames of the text data is less than a frame number threshold; otherwise, determine that the preset playback time period scenario is a multi-line text data display scenario; in the single-line text data display scenario, determine the playback time period in which the consecutive frame number is located as the time period for the appearance of the ending credits; in the multi-line text data display scenario, determine that the playback time period in which the multi-line text data appears is the time period for the appearance of the ending credits;

[0014] The module for determining the start time of the ending is used to determine the start time of the ending of the video object according to the deep learning features of each frame image in the time period when the ending appears.

[0015] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-described methods.

[0016] In yet another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-described methods.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] The text detection data of the preset text area of ​​the video object is obtained; in the preset playback time period of the video object, the number of consecutive frames of the text detection data is obtained; if the number of consecutive frames of the text data is less than the frame number threshold, the preset playback time period scene is determined to be a single-line text data display scene; otherwise, the preset playback time period scene is determined to be a multi-line text data display scene; in the single-line text data display scene, the playback time period where the consecutive frames are located is determined as the end credits appearance time period; in the multi-line text data display scene, the playback time period where the multi-line text data appears is determined to be the end credits appearance time period; according to the deep learning features of each frame image in the end credits appearance time period, the end credits start time of the video object is determined. The automatic identification of the end credits start time of the video data according to the frame-level change features of the text data in the video greatly reduces the labor cost caused by manually marking the location of the video end credits, thereby significantly improving the end credits recognition efficiency.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0021] Figure 1 is a flowchart of a method for identifying the end of a video provided by an embodiment of the present invention;

[0022] Figure 2 is a flowchart of a method for identifying a video end credit provided by another embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of a time instance flow chart provided by an embodiment of the present invention;

[0024] Figure 4 is a block diagram of a video end recognition device provided by an embodiment of the present invention;

[0025] Figure 5 is a block diagram of a video end recognition device provided by another embodiment of the present invention;

[0026] Figure 6 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0028] Figure 1 is a flowchart of a method for identifying a video end credit provided by an embodiment of the present invention. Figure 1 As shown, the method may include:

[0029] Step 101, obtaining text detection data of a preset text area of ​​a video object;

[0030] In the embodiment of the present invention, for the massive video data in the network platform, when identifying the end positions of all videos, it is necessary to extract the text detection data in the preset text area of ​​each video object. The preset text area refers to the display area of ​​the lines in the display screen of the video object, from which the text detection data is obtained through text filtering and text tracking technology.

[0031] It can be understood that the preset text area is usually a fixed text display area. Generally, a video includes an upper display area or a lower display area, which is located at a preset distance below the upper boundary or above the lower boundary of the display scene. The preset distance between the preset text area and the video boundary, the size of the preset text area, and the number of characters of text (lines) displayed at one time in the preset text area are default parameters of the video object, and the embodiment of the present invention does not impose specific restrictions on this.

[0032] The acquired text detection data mainly includes the filtered and tracked dialogue text data in the detected video object.

[0033] Specifically, the filtering of text detection data mainly includes extracting text data in a preset text area, and text data outside the preset text area is not used.

[0034] Furthermore, the filtering of text detection data is achieved through a specific text filtering tool. After the text filtering, the non-dialogue text is filtered and only the dialogue text is retained for business needs.

[0035] Tracking of text detection data mainly refers to tracking the changes of text data in the preset text area in each frame of the video object. The text data in the preset text area is the lines of the video object. Since the lines change with the change of video playback time, tracking the preset text area of ​​each frame of the video object can obtain the characteristics of text changes. In the scene where the lines of the video object scroll up and down or left and right, the text in the preset text area will definitely change after a few frames, that is, when one line changes to another, the text characteristics in the preset text area will change.

[0036] It can be understood that the dialogue display of the video object can be a single line or multiple lines. In the scrolling scene of single-line dialogue or multiple lines of dialogue, the dialogue text tracking method described above is also applicable.

[0037] In actual applications, since the lines corresponding to the video objects are stored separately in the web pages of their respective video objects, the line tracking uses crawler technology to analyze each web page containing independent content. The website not only contains the lines, just like the script, but also includes the scene information and action information indicated in italics, etc. This information is usually in the corresponding tag of the line. <em>Under the label or , and through this part of the information, the display scene and display area information of the lines can be obtained. That is to say, through this part of the line information, the preset text area, the line information within the preset text area, the frame number in the video object where the line is located, and the display scene of the line (multiple lines, single line, scrolling up and down, scrolling left and right), etc. can be obtained.

[0038] It can be understood that the specific methods for filtering and tracking text detection data are not limited to the above description, and the embodiments of the present invention do not make specific limitations.

[0039] Step 102: Obtain the consecutive frame numbers of the text detection data during the preset playback time period of the video object.

[0040] In the embodiments of the present invention, for the tracking method of text data in the preset text area described in step 101, when the text data in the preset text area of the video object undergoes frame-level changes, that moment is regarded as the line transition moment of the video object. In this way, the frame numbers corresponding to the frames of each line of text detection data in the preset text area are tracked. Since the frame numbers are continuous, by knowing the consecutive frame numbers of a line of text, the display frame number of this line of text can be obtained through the difference between the start frame number and the end frame number of this line of text. Through this method, the display frame number of each line of text in the video object can be obtained.

[0041] Among them, the display frame number of each line of text, that is, the consecutive frame numbers of each line of text, usually, when a line of text lasts for about twenty frames, the next line of text will be switched.

[0042] Of course, in actual applications, the duration of the line display in different videos is different, and the embodiments of the present invention do not make specific limitations on this.

[0043] Step 103: If the consecutive frame numbers of the text data are less than the frame number threshold, determine that the preset playback time period scene is a single-line text data display scene; otherwise, determine that the preset playback time period scene is a multi-line text data display scene; in the single-line text data display scene, determine the playback time period where the consecutive frame numbers are located as the end-credit appearance time period; in the multi-line text data display scene, determine the playback time period when the multi-line text data appears as the end-credit appearance time period.

[0044] In the embodiments of the present invention, each line of text is tracked during the playback time period of the video object, and the presentation time of the line text is usually very short (about 20 frames). If the consecutive frame numbers corresponding to the display time of a line of text are less than the preset frame number threshold, then it can be determined that the time for the conversion of this line of text is too short and may be the end-credit appearance time period.

[0045] In practical applications, the presentation of the dialogue text includes single-line or multi-line. The frame number threshold is set to 10 frames. When the dialogue text data in the preset text area of ​​the video object has a continuous frame number of less than 10 frames, the dialogue text data is first determined to be a single-line text data display scene. Otherwise, the dialogue text data is determined to be a multi-line text data display scene.

[0046] Of course, in actual applications, the frame number threshold is specifically set according to the number of characters presented in a single line of different video dialogue texts and the dialogue content of a specific video object, and the embodiment of the present invention does not specifically limit this.

[0047] Specifically, when it is determined that the dialogue text data is a single-line text data display scene, the time period in which the single-line dialogue text data appears is determined as the end credits appearance time period. That is, when the dialogue text of the video data is a single-line scrolling, then when tracking the single-line dialogue tracking data in the video playback data, if a scene with a continuous frame number of less than 10 frames appears, then the playback time period in which the continuous frame number is located is determined as the end credits appearance time period.

[0048] Likewise, when it is determined that the dialogue text data is a display scene of multiple lines of text data, the time period in which the multiple lines of dialogue text data appear is determined as the ending appearance time period.

[0049] Step 104: Determine the start time of the end of the video object according to the deep learning features of each frame image in the time period when the end appears.

[0050] In an embodiment of the present invention, in selecting a time period for the appearance of the ending credits, to determine the start time of the ending credits, the first frame where the ending credits start can be determined first. Therefore, to find the frame corresponding to the start time of the ending credits, it is necessary to obtain the image features of each frame of the image in the time period where the ending credits start.

[0051] Among them, the image features include deep learning features of each frame of the image.

[0052] It can be understood that when the video ending appears, the background area of ​​the video object is the same, such as a solid color or patterned area, and the preset text area corresponding to the previous and next frames of this area no longer displays text data, and after the ending appears, the image features of the preset text area remain unchanged.

[0053] Therefore, by extracting the deep learning features of each frame image in the time period when the ending appears, the ending start time in the time period when the ending starts can be determined.

[0054] Preferably, the deep learning features include residual network features.

[0055] Specifically, deep learning features include residual network features, where residual network features are data training features in the deep residual network (ResNet). The deep residual network solves the degradation problem of deep neural networks, that is, deeper networks can be trained to achieve higher accuracy of deep network learning effects as the network level deepens.

[0056] In practical applications, the residual network features of each frame of the video image at the beginning of the tail end are extracted and input into the trained deep residual network model. The deep residual network model outputs the frame number of the frame corresponding to the time when the tail end appears, and then the position of the starting frame of the tail end is obtained, that is, the start time of the tail end.

[0057] In an embodiment of the present invention, the text detection data of the preset text area of ​​the video object is obtained; in the preset playback time period of the video object, the continuous frame number of the text detection data is obtained; if the continuous frame number of the text data is less than the frame number threshold, the preset playback time period scene is determined to be a single-line text data display scene; otherwise, the preset playback time period scene is determined to be a multi-line text data display scene; in the single-line text data display scene, the playback time period where the continuous frame number is located is determined as the end credits appearance time period; in the multi-line text data display scene, the playback time period where the multi-line text data appears is determined to be the end credits appearance time period; according to the deep learning features of each frame image in the end credits appearance time period, the end credits start time of the video object is determined. The automatic identification of the end credits start time of the video data according to the changing features of the text data at the frame level in the video solves the problem of high labor costs caused by manually marking the location of the video end credits, and significantly improves the end credits recognition efficiency.

[0058] Figure 2 is a flowchart of a method for identifying a video end credit provided by another embodiment of the present invention. Figure 2 As shown, the method may include:

[0059] Step 201: Obtain text detection data of a preset text area of ​​a video object.

[0060] This step is the same as step 101 and will not be described in detail here.

[0061] Step 202: Detect the start frame number and the end frame number of the text detection data during the preset playback time period of the video object.

[0062] In an embodiment of the present invention, when a text change is detected at the frame level in a preset text area in a video object, the start frame of the currently tracked text is obtained, and at the next text change moment, the end frame of the current text is obtained. In this way, the start frame number and end frame number of all texts in the video object can be obtained, and the frame number is used to identify and mark each text frame in the video object.

[0063] Step 203: Obtain the number of consecutive frames of the text detection data according to the difference between the starting frame number and the ending frame number.

[0064] In the embodiment of the present invention, after obtaining the starting frame number and the ending frame number of each line of dialogue text data, the number of consecutive frames of each line of dialogue text data can be obtained through the difference between the starting frame number and the ending frame number.

[0065] Step 204: If the number of consecutive frames of the text data is less than the frame number threshold, determine that the preset playback time period scene is a single-line text data display scene; otherwise, determine that the preset playback time period scene is a multi-line text data display scene; in the single-line text data display scene, determine the playback time period where the consecutive frame number is located as the ending appearance time period.

[0066] This step is the same as step 103 and will not be described in detail here.

[0067] Step 205: In the multi-line text data display scene, if the number of consecutive frames of the multi-line text data is less than the frame number threshold and greater than the storage frame number threshold, the consecutive frames are stored in a frame buffer list.

[0068] In the embodiment of the present invention, in the video playback data, the ending time of the film generally appears after 80% of the overall duration of the video data, so the ending selected time period is usually determined based on the line tracking data after 80% of the video duration.

[0069] If the result of tracking a single line of dialogue data in the video data shows that there is no scene in which the number of consecutive frames of dialogue text data is less than 10, then the data played after 80% of the overall duration of the video data is determined as the end credits candidate time period.

[0070] It is understandable that although the probability of the ending appearing after 80% of the overall length of the video data is relatively high, it does not mean that all video data follow this rule. Different time proportions are selected according to the actual situation of the specific video object in actual application, and the embodiment of the present invention does not impose specific restrictions on this.

[0071] Among them, in the end selection time period of the video object, if the number of consecutive frames of the multi-line text data is less than the frame number threshold, it is further determined whether the number of consecutive frames of the multi-line text data is greater than the storage frame number threshold. For example, if the multi-line text has 8 consecutive frames, which is less than the frame number threshold of 10 frames and greater than the storage frame number threshold of 5 frames, then the 8 consecutive frame numbers are stored in the frame cache list.

[0072] The storage frame number threshold is smaller than the frame number threshold.

[0073] It can be understood that in the selected time period of the ending, there may be multiple segments of multi-line text data with a continuous frame number less than the frame number threshold. By further determining whether the continuous frame number of each segment of the multi-line text data is greater than the storage frame number threshold, the multi-line text lines that are displayed too short can be filtered out, and the multi-line text data that meets the storage conditions can be stored in the frame cache list, so as to further determine in which segment of these multi-line text data the ending start time is located.

[0074] In practical applications, the multi-line text data appearing in each frame image starting from the 80% total frame number is tracked. If the number of consecutive frames of the multi-line text data is less than 10 frames, it is determined whether the number of consecutive frames (connect_num) is greater than the storage frame number threshold (CONNECT_THRESH). If connect_num>CONNECT_THRESH, if it is greater, the frames of the current consecutive frame number are stored in the frame cache list anchor_frm_nums_list in the form of frame groups with the starting frame number and the ending frame number as identifiers.

[0075] It can be understood that when further determining the frame corresponding to the start time of the ending, it is only necessary to determine from each frame group in the frame cache list anchor_frm_nums_list, thereby filtering out multiple lines of text data that appear when the number of consecutive frames is too short, thereby making the determination of the frame corresponding to the ending time more efficient.

[0076] It can be understood that in actual applications, the selection ratio of the tail selection time period, the frame number threshold and the storage frame number threshold are different in different application scenarios and for different video objects. The present invention does not limit the specific setting method of the above values.

[0077] Step 206: Obtain the arrangement dispersion of the multiple lines of text data in each of the consecutive frames in the frame buffer list.

[0078] In an embodiment of the present invention, a plurality of continuous frame groups may be stored in the frame cache list anchor_frm_nums_list, which means that in the tail selection time period, there may be a plurality of continuous frames of multi-line text data whose number is less than the frame number threshold but greater than the storage frame number threshold, then the dispersion of the multi-line text data in the above frame group is further determined.

[0079] It can be understood that the greater the interval between the multiple lines of subtitles scrolling on the continuous frames in the frame group, the greater the dispersion. The arrangement dispersion of the multiple lines of text data can be determined by the distance between the lines of subtitles and the interval frames of the subtitles.

[0080] Specifically, in the display scenario of the preset text area for the multi-line text detection data, that is, the multi-line dialogues in the video object, the number of dialogues displayed in the preset text area has a maximum limit (for example, three lines). If in the current video playback stage (for example, within one minute), the number of dialogues is relatively small, and the number of dialogues displayed in the multi-line dialogue display scenario cannot reach the maximum limit, then in the area of ​​the preset text area that can display up to three lines of dialogues, only one or two lines need to be displayed. Then, based on the ratio of the area of ​​the preset text area to the number of dialogues, the arrangement dispersion of the multi-line text data can be obtained, that is, the larger the ratio of the area of ​​the preset text area to the number of dialogues, the higher the arrangement dispersion of the multi-line text data, and vice versa.

[0081] Among them, the area of ​​the preset text area and the maximum limit value of the number of lines of dialogue displayed in the preset text area are included in the dialogue-related parameters obtained by the technician through the dialogue tracking of the video object.

[0082] It is understandable that the area of ​​the preset text area and the maximum limit of the number of lines displayed in the preset text area can be adjusted by relevant technicians of the video object display scene, and the embodiment of the present invention does not limit the specific values.

[0083] Step 207: Determine the playback time period of the continuous frames where the arrangement dispersion of the multiple lines of text data is the largest as the ending time period.

[0084] In the embodiment of the present invention, the time period of the continuous frame group having the largest text data dispersion among the multiple continuous frame groups is determined as the end credits appearance time period.

[0085] It can be understood that the continuous frame group with the largest dispersion can reflect that the interval between the lines is longer, and is most likely to be the time period for the ending to appear, so the time period where the continuous frame group with the largest dispersion is located is determined as the time period for the ending to appear.

[0086] Step 208: Select a target frame from each frame image in the time period in which the ending appears in a time sequence, and extract a first deep learning feature corresponding to the target frame.

[0087] In the embodiment of the present invention, a target frame is selected in a time period in which the ending appears according to the time sequence, and a first deep learning feature of the image of the target frame is extracted.

[0088] Step 209: determine reference frames of a preset number of frames before and after the target frame, and extract second deep learning features of each of the reference frames.

[0089] In an embodiment of the present invention, deep learning features of a target frame image are extracted, and second deep learning features of a reference frame of a preset number of frames of the target frame image are simultaneously extracted. For example, if the preset number of frames is N, deep learning features of images corresponding to the first N frames and the last N frames of the target frame are extracted as second deep learning features.

[0090] It can be understood that the preset number of frames is set by relevant technical personnel according to the application scenario of the specific video object, and the embodiment of the present invention is not limited to this.

[0091] Step 210: If the average matching degree between the first deep learning feature and the plurality of the second deep learning features exceeds a preset matching value, the time at which the target frame is located is determined as the start time of the end of the video object; otherwise, continue to execute step 208.

[0092] In an embodiment of the present invention, the similarity between the first deep learning feature and the second deep learning feature is calculated. If the similarity is greater than a similarity threshold, it is considered that the deep learning features of the target frame image and the corresponding N frames before and after are the same, and the target frame selected above is the ending start frame.

[0093] Furthermore, if the similarity between the first deep learning feature and the second deep learning feature is not greater than the similarity threshold, the second frame image is selected in chronological order from the time period starting from the end of the film as the target frame, and steps 208 to 210 are repeated, and so on, until a target frame is found whose similarity between the first deep learning feature and the second deep learning feature is greater than the similarity threshold as the frame corresponding to the start time of the end of the film.

[0094] That is to say, the first frame image in the time period when the ending appears is taken as the target frame. If the target frame is not determined to be the frame corresponding to the beginning time of the ending, the next frame image in the sequence of the target frame is judged, and so on.

[0095] It can be understood that in practical applications, the deep learning feature extraction of frame images is carried out through a deep learning model, and when the deep learning feature is a residual network feature, the deep learning model includes a deep residual network model, and the extracted frame image is input into a pre-trained deep residual network model to output the residual network feature of the frame image.

[0096] In practical applications, the similarity calculation of deep learning features can use commonly used image feature similarity calculation methods such as weighted average. The specific method can be selected by relevant technical personnel according to the specific application scenario. The embodiment of the present invention does not impose specific restrictions on this.

[0097] like Figure 3 As shown, the method of determining the start time of the ending is further explained by an example:

[0098] First, perform text detection on the video object to filter out the text data (lines) in the specific line display area, i.e., line filtering. Then, detect the change characteristics of the text data in the specific line display area according to the time sequence, i.e., line tracking. Usually, the last 1 / 4 time frame of the video is selected for line tracking, and the number of consecutive frames of a line is detected. Among them, the lines are detected in the frame where the video object is located, and the features of different frames in the same position of the continuous frame image of the video object are used to determine whether the lines have changed. For scenes where the number of display frames of the same line is less than 10, it is considered that the line is changing, and the line is in a single-line display scene, otherwise it is in a multi-line display scene. For the single-line display scene, the resnet feature matching of 10% frames before and after the target frame is used. If the match is successful, it is determined as the starting frame of the end credits;

[0099] Again, for the multi-line display scene of the lines, the frames where the multi-line text appears are detected, and the resnet features are used to find the starting frame of the ending. That is, the multi-line text is detected in each frame image starting from the "0.8*total number of frames" of the video object. If the frame where the multi-line text data is located satisfies the requirement that the number of consecutive frames is less than 10 and is greater than the preset storage frame value, the current continuously tracked frame number is added to anchor_frm_nums_list (a temporary list), and the continuously tracked frame with the most dispersed lines in anchor_frm_nums_list is selected as the frame corresponding to the time period where the ending appears. The starting frame of the continuously tracked frame number is used to search forward and backward for other frames with a preset number of frames, and the resnet features of the starting frame and the front and back search frames are used for matching. If the match is successful, the frame is determined to be the frame corresponding to the starting moment of the ending. Otherwise, the next frame in the continuous frame group is selected in chronological order, and the search is continued until the matching is successful to determine the starting frame of the ending.

[0100] In an embodiment of the present invention, the text detection data of the preset text area of ​​the video object is first detected. After the text data is detected, the starting frame number and the ending frame number of the text detection data in the preset playback time period of the video object are further determined, and the number of consecutive frames of the text detection data is obtained according to the difference between the starting frame number and the ending frame number. The text data display scene is further judged by the number of consecutive frames, that is, in the multi-line text data display scene, if the number of consecutive frames where the multi-line text data is located is less than the frame number threshold and greater than the storage frame number threshold, the continuous frames are stored in the frame cache list; the arrangement dispersion of the multi-line text data in each of the continuous frames in the frame cache list is obtained; and the playback time period where the continuous frame with the largest arrangement dispersion of the multi-line text data is located is determined as the end appearance time period. Finally, a target frame is selected from each frame image in the end time period in chronological order, and the first deep learning feature corresponding to the target frame is extracted; the reference frames of the preset number of frames before and after the target frame are determined, and the second deep learning features of each reference frame are extracted; if the average matching degree of the first deep learning feature and the plurality of the second deep learning features exceeds the preset matching value, the moment of the target frame is determined as the end start time of the video object; otherwise, the step of selecting a target frame from each frame image in the end time period in chronological order and extracting the first deep learning feature corresponding to the target frame is continued until the frame corresponding to the end start time is found. By detecting the deep learning features of the dialogue text data in the video, and matching the deep learning features of each frame of the dialogue text data with the deep learning features of the dialogue text data in the images corresponding to the specific number of frames before and after it, the purpose of fully automating the recognition of the start time of the video end is achieved, and the beneficial effect of improving the efficiency of video end recognition is achieved.

[0101] Figure 4 is a block diagram of a video end recognition device provided by an embodiment of the present invention, such as Figure 4 As shown, the device 220 may include:

[0102] A text detection data acquisition module 2201 is used to acquire text detection data of a preset text area of ​​a video object;

[0103] A continuous frame number determination module 2202 is used to obtain the continuous frame number of the text detection data in a preset playback time period of the video object;

[0104] The module 2203 for determining the time period for which the ending credits appear is used to determine that the preset playback time period scenario is a single-line text data display scenario if the number of consecutive frames of the text data is less than a frame number threshold; otherwise, to determine that the preset playback time period scenario is a multi-line text data display scenario; in the single-line text data display scenario, to determine the playback time period in which the consecutive frame number is located as the time period for which the ending credits appear; in the multi-line text data display scenario, to determine that the playback time period in which the multi-line text data appears is the time period for which the ending credits appear.

[0105] The module 2204 for determining the start time of the end of the video object is used to determine the start time of the end of the video object according to the deep learning features of each frame image in the time period in which the end of the video object appears.

[0106] Figure 5 is a block diagram of a video end recognition device provided by another embodiment of the present invention. Figure 5 As shown, the device 220 may include:

[0107] A text detection data acquisition module 2201 is used to acquire text detection data of a preset text area of ​​a video object;

[0108] A continuous frame number determination module 2202 is used to obtain the continuous frame number of the text detection data in a preset playback time period of the video object;

[0109] The module 2203 for determining the time period for which the ending credits appear is used to determine that the preset playback time period scenario is a single-line text data display scenario if the number of consecutive frames of the text data is less than a frame number threshold; otherwise, to determine that the preset playback time period scenario is a multi-line text data display scenario; in the single-line text data display scenario, to determine the playback time period in which the consecutive frame number is located as the time period for which the ending credits appear; in the multi-line text data display scenario, to determine that the playback time period in which the multi-line text data appears is the time period for which the ending credits appear.

[0110] The module 2204 for determining the start time of the end of the video object is used to determine the start time of the end of the video object according to the deep learning features of each frame image in the time period in which the end of the video object appears.

[0111] Preferably, the continuous frame number determination module 2202 includes:

[0112] A frame number detection submodule 021, used to detect the start frame number and the end frame number of the text detection data in a preset playback time period of the video object;

[0113] The continuous frame number acquisition submodule 022 is used to obtain the continuous frame number of the text detection data according to the difference between the starting frame number and the ending frame number.

[0114] The module 023 for determining the start time of the end of the video object is used to determine the start time of the end of the video object according to the deep learning features of each frame image in the time period when the end of the video object appears.

[0115] Preferably, the end credits appearance time period determination module 2203 includes:

[0116] The frame buffer submodule 031 is used for storing the continuous frames in a frame buffer list if the number of continuous frames of the multiple lines of text data is less than a frame number threshold and greater than a storage frame number threshold in the multiple lines of text data display scene;

[0117] Arrangement dispersion acquisition submodule 032, used to acquire the arrangement dispersion of the plurality of lines of text data in each of the continuous frames in the frame buffer list;

[0118] The submodule 033 for determining the time period for the appearance of the ending credits is used to determine the playing time period of the continuous frames where the arrangement dispersion of the multiple lines of text data is the largest as the time period for the appearance of the ending credits.

[0119] Preferably, the end credits start time determination module 2204 specifically includes:

[0120] A first deep learning feature acquisition submodule 041 is used to select a target frame from each frame image in the time period in which the end of the film appears in a time sequence, and extract a first deep learning feature corresponding to the target frame;

[0121] A second deep learning feature acquisition submodule 042 is used to determine reference frames of a preset number of frames before and after the target frame, and extract a second deep learning feature of each of the reference frames;

[0122] The matching submodule 043 is used to determine the time when the target frame is located as the start time of the end of the video object if the average matching degree between the first deep learning feature and multiple second deep learning features exceeds a preset matching value; otherwise, continue to run the first deep learning feature acquisition submodule.

[0123] In summary, the video end recognition device provided by the embodiment of the present invention obtains the text detection data of the preset text area of ​​the video object; obtains the continuous frame number of the text detection data in the preset playback time period of the video object; if the continuous frame number of the text data is less than the frame number threshold, the preset playback time period scene is determined to be a single-line text data display scene; otherwise, the preset playback time period scene is determined to be a multi-line text data display scene; in the single-line text data display scene, the playback time period where the continuous frame number is located is determined as the end appearance time period; in the multi-line text data display scene, the playback time period where the multi-line text data appears is determined to be the end appearance time period; according to the deep learning features of each frame image in the end appearance time period, the end start time of the video object is determined. It realizes the automatic recognition of the end start time of the video data according to the frame-level change features of the text data in the video, greatly reduces the high labor cost caused by manually marking the video position where the video end is located, and significantly improves the end recognition efficiency.

[0124] As for the above-mentioned device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0125] The embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0126] Memory 303, used for storing computer programs;

[0127] The processor 301 is used to execute the program stored in the memory 303 to implement the following steps:

[0128] Obtaining the reasons for recommending the video based on the comment text generated by the user for the video;

[0129] Using a preset click rate prediction model, based on the text features of the candidate recommendation reason of the video and the user features of the user to be recommended, a predicted click rate corresponding to the candidate recommendation reason is generated;

[0130] The candidate recommendation reason whose corresponding predicted click rate meets the preset condition is determined as the target recommendation reason for the video.

[0131] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0132] The communication interface is used for communication between the above terminal and other devices.

[0133] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0134] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0135] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the computer-readable storage medium is executed on a computer, the computer executes the video ending recognition method described in any of the above embodiments.

[0136] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the video ending recognition method described in any one of the above embodiments.

[0137] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0138] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0139] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.< / em>

Claims

1. A video end recognition method, characterized in that: The method comprises: Obtain text detection data of a preset text area of ​​a video object; Acquire the number of consecutive frames of the text detection data during a preset playback time period of the video object; If the number of consecutive frames showing the text data is less than the frame number threshold, the preset playback time period scene is determined to be a single-line text data display scene; otherwise, the preset playback time period scene is determined to be a multi-line text data display scene; in the single-line text data display scene, the playback time period where the consecutive frames are located is determined to be the end credits appearance time period; in the multi-line text data display scene, the playback time period where the multi-line text data appears is determined to be the end credits appearance time period; Determine the start time of the end of the video object according to the deep learning features of each frame image in the time period when the end appears; The step of determining the start time of the end of the video object according to the deep learning features of each frame image in the end time period includes: Selecting a target frame from each frame image in the time period in which the ending appears in a time sequence, and extracting a first deep learning feature corresponding to the target frame; Determine reference frames of a preset number of frames before and after the target frame, and extract second deep learning features of each of the reference frames; If the average matching degree between the first deep learning feature and the plurality of the second deep learning features exceeds a preset matching value, determining the time at which the target frame is located as the start time of the end of the video object; Otherwise, continue to execute the steps of selecting a target frame from each frame image in the time period in which the ending appears in a time sequence, and extracting a first deep learning feature corresponding to the target frame.

2. The method according to claim 1, characterized in that The step of obtaining the number of consecutive frames of the text detection data during a preset playback time period of the video object includes: In a preset playback time period of the video object, detecting a start frame number and an end frame number of the text detection data; The number of consecutive frames of the text detection data is obtained according to the difference between the starting frame number and the ending frame number.

3. The method according to claim 1, characterized in that In the multi-line text data display scenario, determining the playback time period in which the multi-line text data appears as the end credits appearance time period includes: In the multi-line text data display scenario, if the number of consecutive frames where the multi-line text data is located is less than the frame number threshold and greater than the storage frame number threshold, the consecutive frames are stored in the frame buffer list; Obtaining the arrangement dispersion of the multiple lines of text data in each of the continuous frames in the frame buffer list; The playing time period of the continuous frames where the arrangement dispersion of the multiple lines of text data is the largest is determined as the ending appearance time period.

4. The method according to claim 1, characterized in that The deep learning features include residual network features.

5. A video end recognition device, characterized in that: The device comprises: A text detection data acquisition module, used to acquire text detection data of a preset text area of ​​a video object; A continuous frame number determination module, used to obtain the continuous frame number of the text detection data in a preset playback time period of the video object; The module for determining the time period for the appearance of the ending credits is used to determine that the preset playback time period scene is a single-line text data display scene if the number of consecutive frames displaying the text data is less than a frame number threshold; otherwise, determine that the preset playback time period scene is a multi-line text data display scene; in the single-line text data display scene, determine the playback time period in which the consecutive frames are located as the time period for the appearance of the ending credits; in the multi-line text data display scene, determine that the playback time period in which the multi-line text data appears is the time period for the appearance of the ending credits; An end credits start time determination module, used to determine the end credits start time of the video object according to the deep learning features of each frame image in the end credits appearance time period; The module for determining the time when the ending starts is also used to select a target frame from each frame image in the time period when the ending appears in a time sequence, and extract a first deep learning feature corresponding to the target frame; determine reference frames of a preset number of frames before and after the target frame, and extract a second deep learning feature of each of the reference frames; if the average matching degree between the first deep learning feature and multiple second deep learning features exceeds a preset matching value, the time when the target frame is located is determined as the time when the ending starts for the video object; otherwise, continue to select a target frame from each frame image in the time period when the ending appears in a time sequence, and extract the first deep learning feature corresponding to the target frame.

6. The device according to claim 5, characterized in that The continuous frame number determination module comprises: A frame number detection submodule, used to detect the start frame number and the end frame number of the text detection data in a preset playback time period of the video object; The continuous frame number acquisition submodule is used to obtain the continuous frame number of the text detection data according to the difference between the starting frame number and the ending frame number.

7. The device according to claim 5, characterized in that The module for determining the time period for the end credits to appear includes: A frame buffer submodule, used for storing the continuous frames in a frame buffer list if the number of continuous frames of the multiple lines of text data is less than a frame number threshold and greater than a storage frame number threshold in the multiple lines of text data display scene; An arrangement dispersion degree acquisition submodule, used for acquiring an arrangement dispersion degree of the plurality of lines of text data in each of the continuous frames in the frame buffer list; The submodule for determining the time period for the appearance of the ending credits is used to determine the playing time period of the continuous frames where the arrangement dispersion of the multiple lines of text data is the largest as the time period for the appearance of the ending credits.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-4 when executing a program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images