A video detection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a video detection method and device, electronic equipment and medium, comprising: acquiring a video file and determining a detection target for the video file; acquiring a plurality of continuous video clips from the video file and determining a plurality of audio-video feature information of the plurality of video clips respectively; inputting the plurality of audio-video feature information into a pre-trained classification model respectively to obtain a plurality of corresponding output results; determining a candidate video clip from the plurality of video clips according to the plurality of output results; performing text recognition on image frames of the candidate video clip to obtain a text recognition result; and determining a target image frame where the detection target is located according to the text recognition result. According to the embodiments of the present application, the candidate video clip where the detection target is located is determined by combining picture information and audio information, and text recognition is performed within the range of the clip, so as to locate the image frame where the accurate detection target is located.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video detection method, a video detection device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] In order to improve the user viewing experience in video streaming services, specific time points in the video are marked. By marking specific time points, users can quickly locate the video content corresponding to the time points, or specific services can be provided at the time points. For example, the end of the trailer time point can be marked. On the one hand, it can provide a skip function to help users quickly enter the main content, and on the other hand, it can insert a preview of the story at this point to enrich the user experience. In addition, the start of the trailer time point can be marked to provide a similar movie recommendation function after the movie is played to increase the user's stay time in the movie. Video streaming services can also mark the start of the Easter egg time point in the movie to provide a jump function to help users quickly enter the Easter egg content.

[0003] The marked time points in the video are edited and produced by the video content producer, and the marked positions of different video time points are different. The traditional marking method either uses manual marking, which is low in efficiency, or uses a unified image template for matching marking. This marking method cannot solve the problem of deviation of the time points in different videos and the flexible setting of some time points, and the marking flexibility is poor. SUMMARY

[0004] In view of the above problems, the embodiments of the present application are proposed to provide a video detection method and corresponding video detection device, electronic device and computer readable storage medium which overcome the above problems or at least partially solve the above problems.

[0005] The embodiments of the present application disclose a video detection method, which comprises:

[0006] Obtaining a video file and determining a detection target for the video file; the detection target comprises at least one of trailer end mark information, trailer start mark information and trailer end mark information;

[0007] Obtaining a plurality of continuous video clips from the video file and determining a plurality of audio and video feature information of the plurality of video clips respectively;

[0008] Inputting the plurality of audio and video feature information into a pre-trained classification model respectively to obtain a plurality of output results corresponding thereto;

[0009] determine a candidate video segment from the multiple video segments according to the multiple output results;

[0010] perform text recognition on image frames of the candidate video segment to obtain a text recognition result;

[0011] determine a target image frame in which the detection target is located according to the text recognition result.

[0012] The embodiment of the application further discloses a video detection device, and the device comprises:

[0013] a first obtaining and determining module configured to obtain a video file and determine a detection target for the video file; the detection target comprises at least one of end-of-episode mark information, start-of-episode mark information and end-of-episode mark information;

[0014] a second obtaining and determining module configured to obtain multiple continuous video segments from the video file and determine multiple audio-video feature information of the multiple video segments respectively;

[0015] an input and output module configured to input the multiple audio-video feature information into a pre-trained classification model respectively to obtain corresponding multiple output results;

[0016] a first determining module configured to determine a candidate video segment from the multiple video segments according to the multiple output results;

[0017] a text recognition module configured to perform text recognition on image frames of the candidate video segment to obtain a text recognition result;

[0018] a second determining module configured to determine a target image frame in which the detection target is located according to the text recognition result.

[0019] The embodiment of the application further discloses an electronic device, which comprises a processor, a memory and a computer program stored on the memory and capable of running on the processor, and the computer program implements the steps of the video detection method when executed by the processor.

[0020] The embodiment of the application further discloses a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program implements the steps of the video detection method when executed by a processor.

[0021] The embodiment of the application has the following advantages:

[0022] In the embodiment of the present application, the audio and video feature information of the plurality of continuous video clips in the video file can be used to determine the candidate video clip that may contain the detection target in the plurality of video clips, and the image frame of the candidate video clip is subjected to text recognition to determine the target image frame where the detection target is located. By using the above method, the picture information and audio information of the video file are combined, the deep learning model is used to determine the candidate video clip where the detection target is located, and text recognition is performed in the range of the clip, so as to locate the image frame where the accurate detection target is located, which can improve the recognition accuracy of the video clip end time point, the clip tail start time point and the Easter egg start time point. The detection method does not need to perform image template matching, and does not need to rely on manual operation, and the point recognition method is flexible and has high recognition efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a step flow chart of a video detection method according to an embodiment of the present application;

[0024] Figure 2 is a step flow chart of another video detection method according to an embodiment of the present application;

[0025] Figures 2A-2G is a sub-step flow chart of another video detection method according to an embodiment of the present application;

[0026] Figure 3 is a processing process schematic diagram of audio and video feature information according to an embodiment of the present application;

[0027] Figure 4 is a flow chart of a video detection method according to an embodiment of the present application;

[0028] Figure 5 is a flow chart of another video detection method according to an embodiment of the present application;

[0029] Figure 6 is a flow chart of another video detection method according to an embodiment of the present application;

[0030] Figure 7 is a structural block diagram of a video detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned objects, features and advantages of the present application more apparent and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0032] In order to improve the user viewing experience in a video streaming service, a start time point of a trailer, an end time point of a trailer, and a start time point of an Easter egg are marked. Since the time points marked in the video are edited and produced by a video content producer, the marking positions of different time points are different.

[0033] A traditional time point marking method is either manually marked, which is not suitable for massive videos and cannot quickly determine the position of the point, and the marking efficiency is low, or a unified image template is used for matching, which cannot solve the deviation of the point in different videos and the flexible setting of part of the point.

[0034] One of the core ideas of the embodiments of the present application is that the audio and video feature information of the continuous multiple video clips in the video file can be used to determine the candidate video clip that may contain the detection target in the multiple video clips, and the image frame of the candidate video clip is subjected to text recognition to determine the target image frame where the detection target is located. By using the above method, the picture information and audio information of the video file are combined, and a deep learning model is used to determine the candidate video clip where the detection target is located, and text recognition is performed in the range of the clip, so as to locate the image frame where the accurate detection target is located, which can improve the recognition accuracy of the end time point of the trailer, the start time point of the trailer, and the start time point of the Easter egg. The detection method does not need to perform image template matching and does not need to rely on manual operation, and the point recognition method is flexible and has high recognition efficiency.

[0035] Referring to Figure 1 , a step flowchart of a video detection method according to an embodiment of the present application is shown, which can specifically include the following steps:

[0036] Step 101, a video file is obtained, and a detection target for the video file is determined.

[0037] The detection target includes at least one of trailer end mark information, trailer start mark information, and trailer end mark information.

[0038] In the embodiments of the present application, the video file can be a multimedia video file such as a television series video file, a variety show video file, and a movie video file. The video file can include one or more of trailer information, main film information, trailer information, and Easter egg information.

[0039] The playing order of the trailer information and the main film information can be flexibly set. In one example, the trailer information can be played first, and then the main film information can be played. In another example, a part of the main film information can be played first, then the trailer information can be played, and then another part of the main film information can be played. The playing order of the trailer information and the main film information is not specifically limited in the embodiments of the present application.

[0040] The opening and closing credits markers can indicate the end of the opening sequence; the closing credits markers can indicate the beginning or end of the closing sequence. For a single episode of a TV series video file, the opening and closing credits markers can be the episode number or release number. For a movie video file, the closing credits markers can be text boxes.

[0041] By detecting the end marker information of the intro in a video file, the location of the end time point of the intro can be determined. By detecting the start marker information of the end time in a video file, the location of the start time point of the end time can be determined. By detecting the end marker information of the end time in a video file, the location of the start time point of the Easter egg can be determined. This allows users to quickly jump to the corresponding time point or to provide targeted service functions at the corresponding time point.

[0042] Step 102: Obtain multiple consecutive video segments from the video file, and determine multiple audio and video feature information of the multiple video segments respectively.

[0043] In this embodiment of the invention, a video file can be divided into multiple consecutive video segments, and multiple audio and video feature information of each video segment can be determined. By combining video features and audio features, the main content (including Easter egg content), the opening content, and the closing content can be distinguished.

[0044] Step 103: Input the multiple audio and video feature information into the pre-trained classification model to obtain the corresponding multiple output results.

[0045] In this embodiment of the invention, a classification model is pre-built. This model is used to determine whether a corresponding video segment is an opening sequence, a main feature segment, or an ending sequence based on the input audio and video feature information. Specifically, for movie video files, the main feature segment played after the ending sequence is typically considered a bonus scene.

[0046] Step 104: Based on the multiple output results, determine candidate video segments from the multiple video segments.

[0047] By inputting multiple audio and video feature information into a pre-trained classification model, multiple corresponding output results can be obtained. These multiple output results can be analyzed to determine candidate video segments that may contain the target being detected.

[0048] Step 105: Perform text recognition on the image frames of the candidate video segment to obtain the text recognition result.

[0049] In the embodiment of the present application, after the candidate video segment possibly containing the detection target is determined, the image frame of the candidate video segment can be subjected to character recognition to obtain a corresponding character recognition result.

[0050] In step 106, the target image frame in which the detection target is located is determined according to the character recognition result.

[0051] According to the character recognition result obtained by the character recognition, the target image frame in which the detection target is located is determined, so that the position of the end time point of the head of the video file or the position of the start time point of the tail of the video file or the position of the start time point of the Easter egg can be located, and the location accuracy reaches the frame level.

[0052] In summary, in the embodiment of the present application, the audio and video feature information of the continuous multiple video segments in the video file is used to determine the candidate video segment possibly containing the detection target in the multiple video segments, and the image frame of the candidate video segment is subjected to character recognition to determine the target image frame in which the detection target is located. By using the above method, the picture information and the audio information of the video file are combined, and the deep learning model is used to determine the candidate video segment in which the detection target is located, and the character recognition is performed in the segment range, so that the image frame in which the accurate detection target is located is located, the recognition accuracy of the end time point of the head of the video file, the start time point of the tail of the video file and the start time point of the Easter egg can be improved, the detection method does not need to perform image template matching, and does not need to rely on manual operation, the point recognition mode is flexible and the recognition efficiency is high.

[0053] Referring to Figure 2 , another step flowchart of a video detection method according to an embodiment of the present application is shown, which can specifically include the following steps:

[0054] In step 201, a video file is obtained, and a detection target for the video file is determined.

[0055] The detection target includes at least one of the end mark information of the head, the start mark information of the tail and the end mark information of the tail.

[0056] In the embodiment of the present application, a video file to be detected and analyzed can be obtained, and a detection target for the video file is determined. The detection target can be one or more of the end mark information of the head, the start mark information of the tail and the end mark information of the tail.

[0057] In an example, the detection target can be determined according to the video type of the video file. For example, if the video file is a television series, the end-of-act information and the start-of-act information can be the detection target of the video file. If the video file is a movie, the end-of-act information, the start-of-act information and the end-of-act information can be the detection target of the video file.

[0058] In step 202, a plurality of continuous video clips are obtained from the video file, and a plurality of audio-video feature information of the plurality of video clips are determined respectively.

[0059] In the embodiment of the present application, the detection target is detected by obtaining a plurality of continuous video clips from the video file. The plurality of continuous video clips means that the plurality of video clips are continuous in the playing time of the video file, and the next video clip is played immediately after the playing of the previous video clip. After the plurality of continuous video clips are obtained, a plurality of audio-video feature information of the plurality of video clips are determined respectively.

[0060] In an optional embodiment of the present application, the step of obtaining a plurality of continuous video clips from the video file in step 202 can include the following sub-steps:

[0061] In sub-step S11, the interception time period is determined according to the detection target.

[0062] In sub-step S12, the video file is intercepted according to the interception time period to obtain an intercepted video clip.

[0063] In sub-step S13, the intercepted video clip is divided equally to obtain a plurality of continuous video clips.

[0064] According to different detection targets, different interception time periods can be determined, and the intercepted video clip can be obtained from the complete video file according to the interception time period, and then the intercepted video clip can be divided into a plurality of continuous video clips.

[0065] For example, the first video clip in the time range of t0-t1 can be intercepted as the intercepted video clip, and the first video clip can be divided equally into T video clips, each of which contains 64 frames of images and 12.8 seconds of audio. By intercepting a part of the video file for analysis, the analysis efficiency can be improved.

[0066] In an example, if the end-of-act information or the start-of-act information of the movie video file needs to be detected, the last 30 minutes of the movie video file can be intercepted as the intercepted video clip for analysis.

[0067] After obtaining the plurality of continuous video clips, the audio and video feature information of each video clip can be determined respectively, and the video features and audio features are combined to distinguish the main content (including the Easter egg content), the head content and the tail content.

[0068] In an optional embodiment of the present application, the step of determining the audio and video feature information of each video clip in step 202 can include the following sub-steps:

[0069] In sub-step S21, for each video clip, the pre-trained super-resolution test sequence VGG model is used to extract the corresponding audio feature information, and the pre-trained dual-flow inflated three-dimensional convolutional network I3D model is used to extract the corresponding video feature information. The audio feature information and the video feature information are combined to obtain the audio and video feature information corresponding to the video clip.

[0070] The audio information in the video clip is input into the audio feature extraction model for feature extraction, and the image information in the video clip is input into the video feature extraction model for feature extraction.

[0071] The audio feature extraction model can be a super-resolution test sequence VGG (visual geometry group) model trained on a public data set. The super-resolution test sequence VGG model uses repeatedly stacked 3*3 small convolution kernels and increases the depth of the network. For example, 12.8s audio of a video clip can be input into the super-resolution test sequence VGG model, and 8*128-dimensional audio features can be output.

[0072] The video feature extraction model can be a dual-flow inflated three-dimensional convolutional network I3D model trained on a public data set. The dual-flow inflated three-dimensional convolutional network I3D model is an enhanced version based on a 2D convolutional network, which expands the convolution kernel and the pooling kernel of the convolution classification to 3D. For example, 64 frames of images of a video clip can be input into the dual-flow inflated three-dimensional convolutional network I3D model, and 6*1024-dimensional video features can be output.

[0073] For a video clip, after obtaining the audio feature information and the video feature information corresponding to the video clip, the audio feature information and the video feature information can be combined to obtain the audio and video feature information corresponding to the video clip.

[0074] Reference Figure 3As shown, it is a processing process schematic diagram of audio and video feature information of an embodiment of the present application. The video file to be detected is obtained, continuous video frames can be extracted from the video file to obtain video frame images, the video frame images can include T video frame image sequences (equivalent to T video segments) of video frame image sequence 1, video frame image sequence 2, …, video frame image sequence T; and the audio file corresponding to the video frame image can be extracted from the video file, the audio file can be divided into T audio file segments corresponding to audio file segment 1, audio file segment 2, …, audio file segment T; T video frame image sequences are subjected to video feature extraction respectively to obtain T video features of video feature 1, video feature 2, …, video feature T; and T audio file segments are subjected to audio feature extraction respectively to obtain T audio features of audio feature 1, audio feature 2, …, audio feature T; the corresponding audio features and video features are merged, that is, T audio and video features of audio and video feature 1, audio and video feature 2, …, audio and video feature T can be obtained.

[0075] In an optional embodiment of the present application, the step of merging the audio feature information and the video feature information in sub-step S21 to obtain the audio and video feature information corresponding to the video segment can specifically include the following sub-steps:

[0076] Sub-step S211, attention calculation is performed on the audio feature information and the video feature information based on the shift attention mechanism to obtain corresponding attention audio feature information and attention video feature information.

[0077] Sub-step S212, the attention audio feature information and the attention video feature information are spliced to obtain the corresponding audio and video feature information.

[0078] The merging method of audio feature information and video feature information adopts the shift attention mechanism. It is found through experiments that simply splicing the audio features and the video features cannot train a satisfactory classification model because the modalities of video pictures and audio are different and the meanings of feature vectors and expression values are also different. In order to solve this problem, the shift attention mechanism is adopted, attention units are added on each modality and shift operation is performed to improve the feature expression of audio / video. The attention calculation formula is:

[0079]

[0080] Wherein, X is the input audio feature or video feature; alpha and beta are both learnable parameters; a is the attention weighting vector; N is the number of attention units; and v is the output attention audio feature or attention video feature.

[0081] As can be seen from the above formula, after linear transformation operation is performed on the input audio features or video features by using learnable parameters, L2 regularization processing (||aX+β||2) is performed, the attention audio feature information and the attention video feature information after the shift attention mechanism transformation are obtained, and then the audio-video feature information is obtained. The number of attention units is set through experiments, and the audio features and the video features can both be set to 8.

[0082] In step 203, the plurality of audio-video feature information is input into the classification model respectively, and a plurality of confidence results corresponding thereto are obtained.

[0083] The confidence result is used to represent the confidence that the corresponding video segment belongs to the head segment / normal segment / tail segment.

[0084] In an optional embodiment of the present application, the classification model can be trained in the following manner:

[0085] A sample video segment set for training is obtained; the sample video segment set includes a plurality of continuous sample video segments; the plurality of sample video segments are respectively labeled as head segments, normal segments or tail segments; a plurality of sample audio-video feature information of the plurality of sample video segments is determined respectively; the plurality of sample audio-video feature information is used for model training, and the classification model for identifying the head segment, the normal segment or the tail segment is obtained.

[0086] The sample video segment set for model training includes a plurality of continuous sample video segments, each sample video segment is labeled as a head segment, a normal segment or a tail segment, a plurality of sample audio-video feature information of the plurality of sample video segments can be determined, and then the plurality of sample audio-video feature information is input into the model training system for model training, so that the classification model with the head segment / normal segment / tail segment identification capability is obtained. In an example, the classification model can be a fully connected FC (Fully Connected) classification model.

[0087] In the embodiment of the present application, the audio-video feature information of a video segment is input into the classification model, and the confidence result output therefrom can be used to determine whether the video segment belongs to a head segment, a normal segment or a tail segment.

[0088] Exemplarily, the confidence result output by the classification model can be a probability score value. In an example, the audio-video feature information of a certain video segment is input into the trained classification model, and a probability score value of the video segment belonging to a head segment, a normal segment or a tail segment can be obtained.

[0089] At step 204, a candidate video segment is determined from the plurality of video segments according to the plurality of output results.

[0090] In an embodiment of the present application, the candidate video segment that is likely to contain the detection target can be determined from the plurality of video segments according to the confidence results output by the classification model.

[0091] In an alternative embodiment of the present application, step 204 can specifically include the following sub-steps:

[0092] At sub-step S31, the plurality of confidence results are compared with a preset confidence threshold respectively to obtain a plurality of comparison results.

[0093] At sub-step S32, the candidate video segment is determined from the plurality of video segments according to the plurality of comparison results.

[0094] In an implementation, the head segment has a corresponding first confidence threshold, if the confidence result is greater than the first confidence threshold, the corresponding video segment can be determined as the head segment; the main segment has a corresponding second confidence threshold, if the confidence result is greater than the second confidence threshold, the corresponding video segment can be determined as the main segment; the tail segment has a corresponding third confidence threshold, if the confidence result is greater than the third confidence threshold, the corresponding video segment can be determined as the tail segment.

[0095] In another implementation, only the first confidence threshold corresponding to the head segment and the third confidence threshold corresponding to the tail segment can be set, if the confidence result is greater than the first confidence threshold, the corresponding video segment can be determined as the head segment, otherwise as the main segment; if the confidence result is greater than the third confidence threshold, the corresponding video segment can be determined as the tail segment, otherwise as the main segment.

[0096] In an embodiment of the present application, the segment types of the plurality of video segments are determined according to the plurality of comparison results, and the candidate video segment that is likely to contain the detection target can be determined from the plurality of video segments according to the segment types of the plurality of video segments and the playing order of the plurality of video segments.

[0097] In an alternative embodiment of the present application, the candidate video segment includes a first candidate video segment for searching for the head end marker information of the detection target, and sub-step S32 can specifically include the following sub-steps:

[0098] At sub-step S321, if the detection target is the head end marker information, the plurality of video segments are classified into the head segment and the main segment respectively according to the plurality of comparison results.

[0099] If there is one prologue segment and one episode segment adjacent in the playing order in the plurality of video segments, and the one episode segment plays after the one prologue segment plays, the one prologue segment and the one episode segment are determined as the first candidate video segments which may contain the prologue end mark information.

[0100] If the detection target is the prologue end mark information, the plurality of video segments can be classified as the prologue segment and the episode segment according to the plurality of comparison results respectively. If there is one prologue segment and one episode segment adjacent in the playing order in the plurality of video segments, and the one episode segment plays after the one prologue segment plays, the one prologue segment and the one episode segment are determined as the first candidate video segments which may contain the prologue end mark information.

[0101] For example, the first confidence threshold can be set as 0.8. When the confidence result of a video segment is greater than the first confidence threshold 0.8, the video segment can be considered as the prologue segment, otherwise, the video segment can be considered as the episode segment. Assuming that there are continuous T=10 video segments {t1, …, t10}, and the confidence results corresponding to the video segments are [1.0, 0.9, 0.9, 0.9, 0.9, 0.9, 0.7, 0.4, 0.3, 0.1] respectively, the plurality of confidence results are compared with the first confidence threshold respectively, it can be determined that t1-t6 belong to the prologue segment, and t7-t10 belong to the episode segment, that is, the prologue is t1-t6, the episode is t7-t10, t6 and t7 are one prologue segment and one episode segment adjacent in the playing order, and t7 plays after t6 plays, so t6 and t7 can be selected as the first candidate video segments.

[0102] In an optional embodiment of the present application, the candidate video segments include second candidate video segments for searching the detection target which is the trailer start mark information. The sub-step S32 can specifically include the following sub-steps:

[0103] If the detection target is the trailer start mark information, the plurality of video segments can be classified as the episode segment and the trailer segment according to the plurality of comparison results respectively.

[0104] If there is one episode segment and one trailer segment adjacent in the playing order in the plurality of video segments, and the one trailer segment plays after the one episode segment plays, the one episode segment and the one trailer segment are determined as the second candidate video segments which may contain the trailer start mark information.

[0105] If the detection target is the end-of-credit information, the multiple video clips can be classified into the credit clip and the end-of-credit clip according to the multiple comparison results. If there is one credit clip and one end-of-credit clip adjacent in the playing order in the multiple video clips, and the one end-of-credit clip is played after the one credit clip is played, the one credit clip and the one end-of-credit clip are determined as the second candidate video clips which may contain the end-of-credit information.

[0106] For example, the third confidence threshold can be set as 0.8. When the confidence result of a video clip is greater than the third confidence threshold 0.8, the video clip can be considered as the end-of-credit clip, otherwise, the video clip is considered as the credit clip. Assuming that there are continuous T=10 video clips {t1, …, t10} and the corresponding confidence results are [0.1, 0.9, 0.9, 0.9, 0.1, 0.1, 0.1, 0.8, 0.8, 0.9], the multiple confidence results are compared with the third confidence threshold, it can be determined that t2-t4 and t8-t10 belong to the end-of-credit clip, t1 and t5-t7 belong to the credit clip, that is, the first credit clip is t1, the first end-of-credit clip is t2-t4, the second credit clip is t5-t7, the second end-of-credit clip is t8-t10, t1 and t2 are one credit clip and one end-of-credit clip adjacent in the playing order, t7 and t8 are also one credit clip and one end-of-credit clip adjacent in the playing order, t2 is played after t1 is played, and t8 is played after t7 is played, so t1 and t2 and t7 and t8 can be selected as the second candidate video clips.

[0107] In an optional embodiment of the present application, the candidate video clips include third candidate video clips for searching the detection target as the end-of-credit information, and the sub-step S32 can specifically include the following sub-steps:

[0108] In the sub-step S325, if the detection target is the end-of-credit information, the multiple video clips are classified into the credit clip and the end-of-credit clip according to the multiple comparison results.

[0109] In the sub-step S326, if there is one end-of-credit clip and one credit clip adjacent in the playing order in the multiple video clips, and the one credit clip is played after the one end-of-credit clip is played, the one end-of-credit clip and the one credit clip are determined as the third candidate video clips.

[0110] The classification model can only identify the credit clip, the end-of-credit clip and the credit clip, and cannot identify the Easter egg clip. If it is identified that one credit clip is played after one end-of-credit clip is played in the multiple video clips which are cut out for analysis, the one credit clip can be considered as the Easter egg clip.

[0111] In the embodiment of the present application, if the detection target is the end-of-credit information, the plurality of video clips can be classified into credit clips and end-of-credit clips according to the plurality of comparison results. If there is one end-of-credit clip and one credit clip adjacent in the playing order in the plurality of video clips, and the one credit clip is played after the one end-of-credit clip is played, the one end-of-credit clip and the one credit clip are determined as the third candidate video clip which may contain the end-of-credit information.

[0112] For example, the third confidence threshold can be set as 0.8. When the confidence result of a video clip is greater than the third confidence threshold 0.8, the video clip can be considered as an end-of-credit clip, otherwise, the video clip is a credit clip. Assuming that there are T=10 continuous video clips {t1, …, t10} and the corresponding confidence results are [1.0, 0.9, 0.9, 0.9, 0.1, 0.1, 0.1, 0.8, 0.8, 0.9], the plurality of confidence results are compared with the third confidence threshold, it can be determined that t1-t4 and t8-t10 belong to end-of-credit clips, t5-t7 belong to credit clips, that is, the first end-of-credit is t1-t4, the credit is t5-t7, the second end-of-credit is t8-t10, t4 and t5 are one end-of-credit clip and one credit clip adjacent in the playing order, and t5 is played after t4 is played, so t4 and t5 can be selected as the third candidate video clip.

[0113] It should be noted that the three pairs of sub-steps S321-S322, S323-S324 and S325-S326 are executed in parallel. If the detection target is the end-of-credit information, the sub-steps S321-S322 are executed; if the detection target is the start-of-credit information, the sub-steps S323-S324 are executed; if the detection target is the start-of-credit information, the sub-steps S325-S326 are executed. There is no execution order constraint relationship between the three pairs of sub-steps.

[0114] In step 205, text recognition is performed on the image frames of the candidate video clip to obtain a text recognition result.

[0115] In the embodiment of the present application, the image frames in the candidate video clip can be input into a pre-trained text recognition model for text recognition. The text recognition model has the ability to detect the text display position area of the image and to identify the text content. Text recognition combined with the text display position area and the text content can improve the recognition accuracy.

[0116] In practice, image frames of candidate video segments can be sampled and input into a text recognition model for text recognition. For example, sampling can be performed every 5 frames, and the sampled image frames can be input into the text recognition model.

[0117] Step 206: Determine the target image frame where the detected target is located based on the text recognition result.

[0118] In an optional embodiment of the present invention, the text recognition result includes a first text recognition result obtained by performing text recognition on the image frames of the first candidate video segment, and step 206 may specifically include the following sub-steps:

[0119] Sub-step S41: Match the text content in the first text recognition result with preset keywords, and after matching an image frame containing the preset keywords, track the image frame containing the preset keywords, and determine the last tracked image frame containing the preset keywords as the target image frame where the title sequence end marker information is located.

[0120] The text recognition result includes the first text recognition result for the first candidate video segment. The first text recognition result may include the recognized text content. The text content can be matched with preset keywords. After matching the text content containing the preset keywords, the image frame corresponding to the text content is determined. The image frame is tracked, and the last tracked image frame containing the preset keywords is determined as the target image frame where the title sequence end marker information is located.

[0121] The text content in the recognized text is matched with preset keywords. The preset keywords can be episode number information or release number information, such as "episode *", "chapter *", and release number.

[0122] The system tracks image frames containing preset keywords. The last image frame containing the preset keyword tracked is the one where pixel jitter exceeds a threshold. Tracking until the pixel jitter exceeds the threshold is to capture the image frame where the font of the preset keyword gradually disappears. For example, if the preset keyword is matched at frame N, and pixel jitter exceeds the threshold at frame N+3, at which point the font of the preset keyword gradually disappears, then frame N+3 is taken as the target image frame.

[0123] In an optional embodiment of the present invention, the text recognition result includes a second text recognition result obtained by performing text recognition on the image frames of the second candidate video segment, and step 206 may specifically include the following sub-steps:

[0124] In the substep S42, the second text recognition result corresponding to each image frame in the second candidate video segment is traversed in time sequence, and if the number of text boxes in the second text recognition result corresponding to continuous image frames is greater than the preset number threshold, the first image frame with the number of text boxes greater than the preset number threshold in the second text recognition result is determined as the target image frame in which the end-of-credit information is located.

[0125] The text recognition result includes the second text recognition result of the second candidate video segment. If the detection target is the end-of-credit information, the second text recognition result can be analyzed, the second text recognition result corresponding to each image frame in the second candidate video segment is traversed in time sequence, and if the number of text boxes in the second text recognition result corresponding to continuous image frames is greater than the preset number threshold, it can be considered that the end of the credit is started, and the first image frame with the number of text boxes greater than the preset number threshold can be determined as the target image frame in which the end-of-credit information is located.

[0126] In an optional embodiment of the present application, the text recognition result includes third text recognition result obtained by performing text recognition on the image frames of the third candidate video segment, and the step 206 can specifically include the following substeps:

[0127] In the substep S43, if the third text recognition result contains a text box, the image frame containing the text box is tracked, and the last image frame containing the text box tracked is determined as the target image frame in which the end-of-credit information is located.

[0128] The text recognition result includes the third text recognition result of the third candidate video segment. If the detection target is the end-of-credit information, the third text recognition result can be analyzed to determine whether the third text recognition result contains a text box. If the text box is contained, the image frame containing the text box is tracked, and the last image frame containing the text box tracked is determined as the target image frame in which the end-of-credit information is located.

[0129] Since the end of the video file usually has a rolling production information, such as the name list of the production personnel, the production company, etc., the image frame of the end of the credit will have a rolling multiple text boxes. The position coordinates of the text box can be detected, the image frame containing the text box is tracked, and the last image frame containing the text box tracked is determined as the target image frame in which the end-of-credit information is located.

[0130] It should be noted that the above sub-steps S41, S42 and S43 are executed in parallel, if the detection target is the end of the episode start flag information, sub-step S41 is executed; if the detection target is the end of the episode start flag information, sub-step S42 is executed; if the detection target is the end of the episode start flag information, sub-step S43 is executed. There is no execution order constraint relationship between the three sub-steps. In addition, sub-step S41 is executed after sub-steps S321-S322, sub-step S42 is executed after sub-steps S323-S324, and sub-step S43 is executed after sub-steps S325-S326.

[0131] Referring to Figure 4 Fig. 1 is a flowchart of a video detection method according to an embodiment of the present application, wherein the detection target is the end of the episode start flag information, and the specific process includes:

[0132] S4a, a video file of a television series can be obtained, and a segment with a time period of T0-T1 in the video file is intercepted as a video segment for analysis. The video segment can be divided into multiple video segments, and multiple image information and multiple audio information of the multiple video segments are obtained respectively.

[0133] S4b, for any video segment, the image information and the audio information corresponding to the video segment are input into a video feature extraction model and an audio feature extraction model respectively to extract features, and video feature information and audio feature information corresponding to the video segment are obtained.

[0134] S4c, for any video segment, the video feature information and the audio feature information corresponding to the video segment are merged to obtain audio-video feature information corresponding to the video segment.

[0135] S4d, for any video segment, the audio-video feature information corresponding to the video segment is input into a trained classification model to obtain a confidence result of the video segment belonging to the episode segment / positive film segment / episode end segment.

[0136] S4e, according to the output multiple confidence results, a first candidate video segment is determined from the multiple video segments. Specifically, according to the output multiple confidence results, the multiple video segments can be classified into episode segments and positive film segments respectively. If there is one episode segment and one positive film segment adjacent in playing order in the multiple video segments, and the one positive film segment is played after the one episode segment is played, the one episode segment and the one positive film segment are determined as the first candidate video segment for searching the end of the episode start flag information.

[0137] S4f, the image frame of the first candidate video segment is input into a text recognition model to obtain a corresponding first text recognition result.

[0138] S4g, match the text content in the first text recognition result with the preset keyword, and after matching to the image frame containing the preset keyword, track the image frame containing the preset keyword, and determine the last tracked image frame containing the preset keyword as the target image frame where the end-of-episode marker information is located. If no image frame containing the preset keyword is matched, the time period of the intercepted video segment in step S4a can be adjusted, for example, the time period of the intercepted video segment can be adjusted to T1-T2 time period at this time, and the intercepted video segment with the intercepted time period T1-T2 is analyzed, and steps S4b-S4g are repeated until the image frame containing the preset keyword is matched or the video file containing the video segment is analyzed.

[0139] Referring to Figure 5 Fig. 5 is a flowchart of another video detection method according to an embodiment of the present application, wherein the detection target is the start-of-episode marker information, and the specific process includes:

[0140] S5a, a movie video file can be obtained, and a segment possibly containing the start-of-episode marker information in the video file is intercepted for analysis. Assuming that the intercepted time period is T9-T 10 The intercepted video segment can be divided into multiple video segments, and multiple image information and multiple audio information of the multiple video segments are obtained.

[0141] S5b, for any video segment, the image information and the audio information corresponding to the video segment are input into the video feature extraction model and the audio feature extraction model respectively to extract features, and the video feature information and the audio feature information corresponding to the video segment are obtained.

[0142] S5c, for any video segment, the video feature information and the audio feature information corresponding to the video segment are merged to obtain the audio-video feature information corresponding to the video segment.

[0143] S5d, for any video segment, the audio-video feature information corresponding to the video segment is input into the trained classification model to obtain the confidence result of the video segment belonging to the start-of-episode segment / feature segment / end-of-episode segment.

[0144] S5e, determining a second candidate video segment from the plurality of video segments according to the output plurality of confidence results, specifically, the plurality of video segments can be classified into a positive film segment and an end film segment according to the output plurality of confidence results, if there is a positive film segment and an end film segment adjacent in playing order in the plurality of video segments, and the end film segment is played after the positive film segment is played, the positive film segment and the end film segment are determined as the second candidate video segment for searching the end film start mark information.

[0145] S5f, inputting the image frame of the second candidate video segment into the text recognition model to obtain a corresponding second text recognition result.

[0146] S5g, traversing the second text recognition result corresponding to each image frame in the second candidate video segment in time sequence, if the number of text boxes in the second text recognition result corresponding to the continuous multiple image frames is greater than the preset number threshold, the first image frame with the number of text boxes in the second text recognition result greater than the preset number threshold is determined as the target image frame where the end film start mark information is located. If there is no continuous multiple image frames corresponding to the second text recognition result with the number of text boxes greater than the preset number threshold, the cutting time period in step S5a can be adjusted, for example, the cutting time period of the cutting video segment can be adjusted to T 10 - 11 at this time. 10 - 11 The cutting video segment with the cutting time period of T

[0147] Referring to Figure 6 , it is a flow chart of another video detection method of the embodiment of the application, wherein the detection target is the end film end mark information, and the specific process includes:

[0148] S6a, a movie video file can be acquired, and a segment in the video file that may contain the end film end mark information is cut off for analysis, assuming that the segment with the cutting time period of T8-T9 is used for analysis as the cutting video segment, the cutting video segment can be divided into a plurality of video segments, and a plurality of image information and a plurality of audio information of the plurality of video segments are acquired respectively.

[0149] S6b, for any video segment, the image information and the audio information corresponding to the video segment are input into the video feature extraction model and the audio feature extraction model respectively to extract features, and the video feature information and the audio feature information corresponding to the video segment are obtained.

[0150] S6c, for any video segment, the video feature information and the audio feature information corresponding to the video segment are merged to obtain audio-video feature information corresponding to the video segment.

[0151] S6d, for any video segment, the audio-video feature information corresponding to the video segment is input into the trained classification model to obtain a confidence result of the video segment belonging to a head segment / a main segment / a tail segment.

[0152] S6e, according to the output multiple confidence results, third candidate video segments are determined from the multiple video segments. Specifically, according to the output multiple confidence results, the multiple video segments can be classified into main segments and tail segments respectively. If there is one tail segment and one main segment adjacent in playing order in the multiple video segments, and the one main segment is played after the one tail segment is played, then the one tail segment and the one main segment are determined as third candidate video segments for searching for tail end marker information.

[0153] S6f, the image frames of the third candidate video segments are input into the text recognition model to obtain corresponding third text recognition results.

[0154] S6g, if the third text recognition result contains a text box, the last image frame containing the text box tracked is determined as a target image frame where the tail end marker information is located. If the third text recognition result does not contain a text box, the time period of the intercepted video segment in step S6a can be adjusted. For example, the time period of the intercepted video segment can be adjusted to T9-T 10 , and the intercepted video segment with the time period of T9-T 10 is analyzed, and steps S6b-S6g are repeated until the third text recognition result contains a text box or all video segments contained in the video file have been analyzed.

[0155] In the embodiments of the present application, the audio-video feature information of the multiple video segments in the video file can be used to determine candidate video segments that may contain a detection target in the multiple video segments, and the image frames of the candidate video segments are subjected to text recognition to determine a target image frame where the detection target is located. By using the above method, the picture information and the audio information of the video file are combined, the deep learning model is used to determine the candidate video segments where the detection target is located, and text recognition is performed in the range of the segments, so as to locate the image frame where the accurate detection target is located. The recognition accuracy of the video head end time point, the tail start time point and the egg start time point can be improved. The detection method does not need to perform image template matching, and does not need to rely on manual operation. The point recognition method is flexible and has high recognition efficiency.

[0156] The film head detection method provided by the application can accurately identify the film head even when the film head is played first and the film is played later in a self-made film.

[0157] The film tail detection method provided by the application first extracts 3D convolution network features of a movie video file, identifies the video picture through a classification network trained to determine whether it is a film or a film tail, takes the end time point of the film as the start time point of the film tail, detects the text in the picture near the end time point of the film, and takes the time point at which multiple text boxes are displayed as the start time point of the film tail.

[0158] The Easter egg detection method provided by the application can automatically detect the start time point of the Easter egg, search for whether there is a video segment of the film type after the first end time point of the film, and take the second start time point of the film as the start time point of the Easter egg if there is a video segment of the film type.

[0159] It should be noted that, for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the application are not limited to the action sequence described, because according to the embodiments of the application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the application.

[0160] Referring to Figure 7 , a structural block diagram of a video detection device according to an embodiment of the application is shown, which can specifically include the following modules:

[0161] The first acquisition and determination module 701 is configured to acquire a video file and determine a detection target for the video file; the detection target includes at least one of film head end mark information, film tail start mark information and film tail end mark information.

[0162] The second acquisition and determination module 702 is configured to acquire a plurality of continuous video segments from the video file and determine a plurality of audio and video feature information of the plurality of video segments, respectively.

[0163] The input and output module 703 is configured to input the plurality of audio and video feature information into a pre-trained classification model respectively, and obtain a plurality of output results corresponding thereto.

[0164] The first determination module 704 is configured to determine a candidate video segment from the plurality of video segments according to the plurality of output results.

[0165] The character recognition module 705 is configured to perform character recognition on the image frames of the candidate video segment to obtain a character recognition result.

[0166] The second determination module 706 is configured to determine a target image frame in which the detection target is located according to the character recognition result.

[0167] In the embodiment of the present application, the input and output module comprises:

[0168] The input and output sub-module is configured to input the plurality of audio-video feature information into the classification model respectively to obtain a plurality of confidence results corresponding thereto; the confidence result is used to represent the confidence degree of the corresponding video segment belonging to the head segment / normal segment / tail segment.

[0169] In the embodiment of the present application, the classification model is trained by the following modules:

[0170] The acquisition module is configured to acquire a sample video segment set for training; the sample video segment set comprises a plurality of continuous sample video segments; the plurality of sample video segments are respectively labeled as a head segment or a normal segment or a tail segment;

[0171] The third determination module is configured to respectively determine a plurality of sample audio-video feature information of the plurality of sample video segments.

[0172] The model training module is configured to perform model training using the plurality of sample audio-video feature information to obtain the classification model for recognizing the head segment / normal segment / tail segment.

[0173] In the embodiment of the present application, the second acquisition and determination module comprises:

[0174] The feature extraction and merging sub-module is configured to, for each video segment, extract corresponding audio feature information by using a pre-trained super-resolution test sequence VGG model, extract corresponding video feature information by using a pre-trained dual-flow dilated three-dimensional convolutional network I3D model, and merge the audio feature information and the video feature information to obtain the audio-video feature information corresponding to the video segment.

[0175] In the embodiment of the present application, the feature extraction and merging sub-module comprises:

[0176] The attention calculation unit is configured to perform attention calculation on the audio feature information and the video feature information respectively based on a shift attention mechanism to obtain corresponding attention audio feature information and attention video feature information.

[0177] The splicing unit is configured to splice the attention audio feature information and the attention video feature information to obtain the audio-video feature information.

[0178] In the embodiment of the present application, the first determining module comprises:

[0179] The comparison sub-module is configured to compare the plurality of confidence results with a preset confidence threshold respectively to obtain a plurality of comparison results.

[0180] The first determining sub-module is configured to determine the candidate video segment from the plurality of video segments according to the plurality of comparison results.

[0181] In the embodiment of the present application, the candidate video segment comprises a first candidate video segment for searching the detection target as the end mark information of the head of a film, and the first determining sub-module comprises:

[0182] The first classification unit is configured to classify the plurality of video segments as a head-of-film segment and a main film segment according to the plurality of comparison results if the detection target is the end mark information of the head of a film.

[0183] The first determining unit is configured to determine one head-of-film segment and one main film segment as the first candidate video segment if there is a head-of-film segment and a main film segment adjacent in playing order in the plurality of video segments, and the main film segment is played after the head-of-film segment is played.

[0184] In the embodiment of the present application, the candidate video segment comprises a second candidate video segment for searching the detection target as the start mark information of the end of a film, and the first determining sub-module comprises:

[0185] The second classification unit is configured to classify the plurality of video segments as a main film segment and an end-of-film segment according to the plurality of comparison results if the detection target is the start mark information of the end of a film.

[0186] The second determining unit is configured to determine one main film segment and one end-of-film segment as the second candidate video segment if there is a main film segment and an end-of-film segment adjacent in playing order in the plurality of video segments, and the end-of-film segment is played after the main film segment is played.

[0187] In the embodiment of the present application, the candidate video segment comprises a third candidate video segment for searching the detection target as the end mark information of the end of a film, and the first determining sub-module comprises:

[0188] the third classification unit is configured to, if the detection target is the end-of-credit information, classify the plurality of video clips into credit clips and end-of-credit clips according to the plurality of comparison results respectively;

[0189] the third determination unit is configured to, if there is one end-of-credit clip and one credit clip adjacent in playing order in the plurality of video clips, and the one credit clip is played after the one end-of-credit clip is played, determine the one end-of-credit clip and the one credit clip as the third candidate video clip.

[0190] In the embodiment of the present application, the text recognition result includes a first text recognition result obtained by performing text recognition on the image frames of the first candidate video clip, and the second determination module includes:

[0191] The second determination submodule is configured to match the text content in the first text recognition result with a preset keyword, and track the image frames containing the preset keyword after the image frames containing the preset keyword are matched, and determine the last image frame containing the preset keyword tracked as the target image frame where the end-of-credit information is located.

[0192] In the embodiment of the present application, the text recognition result includes a second text recognition result obtained by performing text recognition on the image frames of the second candidate video clip, and the second determination module includes:

[0193] The third determination submodule is configured to traverse the second text recognition results corresponding to the image frames in the second candidate video clip in time sequence, and if the number of text boxes in the second text recognition results corresponding to a plurality of continuous image frames is greater than a preset number threshold, determine the first image frame with the number of text boxes greater than the preset number threshold in the second text recognition result as the target image frame where the start-of-credit information is located.

[0194] In the embodiment of the present application, the text recognition result includes a third text recognition result obtained by performing text recognition on the image frames of the third candidate video clip, and the second determination module includes:

[0195] The fourth determination submodule is configured to, if the third text recognition result contains a text box, track the image frames containing the text box, and determine the last image frame containing the text box tracked as the target image frame where the end-of-credit information is located.

[0196] In the embodiment of the present application, the second acquisition and determination module includes:

[0197] The fifth determination submodule is configured to determine a time period for interception according to the detection target.

[0198] a clipping sub-module, configured to clip the video file according to the clipping time period to obtain a clipped video segment;

[0199] a splitting sub-module, configured to split the clipped video segment evenly to obtain a plurality of continuous video segments.

[0200] To sum up, in the embodiment of the present application, the audio and video feature information of the plurality of continuous video segments in the video file is used to determine the candidate video segment that may contain the detection target in the plurality of video segments, and the image frame of the candidate video segment is subjected to text recognition to determine the target image frame where the detection target is located. By using the above method, the picture information and the audio information of the video file are combined, the deep learning model is used to determine the candidate video segment where the detection target is located, and text recognition is performed in the range of the segment, so as to locate the image frame where the accurate detection target is located, which can improve the recognition accuracy of the video segment end time point, the segment start time point and the Easter egg start time point. The detection method does not need to perform image template matching and does not need to rely on manual operation, and the point recognition mode is flexible and has high recognition efficiency.

[0201] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0202] The embodiment of the present application also provides an electronic device, which comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, when the computer program is executed by the processor, each process of the above-mentioned video detection method embodiment is realized, and the same technical effect is achieved. To avoid repetition, it will not be repeated here.

[0203] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, when the computer program is executed by a processor, each process of the above-mentioned video detection method embodiment is realized, and the same technical effect is achieved. To avoid repetition, it will not be repeated here.

[0204] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0205] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other.

[0206] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0207] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0208] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0210] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0211] Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0212] The above provides a detailed description of the video detection method, the video detection device, the electronic device and the computer readable storage medium provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A video detection method, characterized by, The method comprises: acquiring a video file and determining a detection target for the video file; the detection target comprises at least one of end-of-credit information, start-of-credit information and end-of-credit information; the video file comprises at least one of credit information, main-credit information, end-of-credit information and Easter egg information, and the playing order of the credit information and the main-credit information is flexibly set; the end-of-credit information is used to determine the position of an Easter egg start time point; dividing the video file into a plurality of continuous video clips and respectively determining a plurality of audio-video feature information of the plurality of video clips; respectively inputting the plurality of audio-video feature information into a pre-trained classification model to obtain a plurality of corresponding output results; determining a candidate video clip from the plurality of video clips according to the plurality of output results; the candidate video clip comprises at least one of a first candidate video clip for searching for the detection target as the end-of-credit information, a second candidate video clip for searching for the detection target as the start-of-credit information, and a third candidate video clip for searching for the detection target as the end-of-credit information; the first candidate video clip comprises one credit clip and one main-credit clip adjacent in playing order, and the main-credit clip is played after the credit clip is played; the second candidate video clip comprises one main-credit clip and one end-of-credit clip adjacent in playing order, and the end-of-credit clip is played after the main-credit clip is played; and the third candidate video clip comprises one end-of-credit clip and one main-credit clip adjacent in playing order, and the end-of-credit clip is played after the main-credit clip is played; performing text recognition on image frames of the candidate video clip to obtain a text recognition result; determining a target image frame where the detection target is located according to the text recognition result.

2. The method of claim 1, wherein, The method comprises: respectively inputting the plurality of audio-video feature information into the classification model to obtain a plurality of corresponding confidence results; the confidence result is used to represent the confidence that the corresponding video clip belongs to a credit clip / main-credit clip / end-of-credit clip.

3. The method of claim 1, wherein, The classification model is trained in the following manner: acquiring a sample video clip set for training; the sample video clip set comprises a plurality of continuous sample video clips; the plurality of sample video clips are respectively labeled as credit clips or main-credit clips or end-of-credit clips; respectively determining a plurality of sample audio-video feature information of the plurality of sample video clips; using the plurality of sample audio-video feature information to train a model to obtain the classification model for recognizing credit clips / main-credit clips / end-of-credit clips.

4. The method of claim 1, wherein, The method comprises: For each video segment, a pre-trained super-resolution test sequence VGG model is used to extract corresponding audio feature information, and a pre-trained two-stream inflated three-dimensional convolutional network I3D model is used to extract corresponding video feature information, the audio feature information and the video feature information are merged to obtain the audio-video feature information corresponding to the video segment.

5. The method of claim 4, wherein, The merging of the audio feature information and the video feature information to obtain the audio-video feature information corresponding to the video segment comprises: Based on the shift attention mechanism, attention calculation is performed on the audio feature information and the video feature information respectively to obtain corresponding attention audio feature information and attention video feature information; The attention audio feature information and the attention video feature information are spliced to obtain the audio-video feature information.

6. The method of claim 2, wherein, The method comprises the steps of: The plurality of confidence results are compared with a pre-set confidence threshold to obtain a plurality of comparison results; According to the plurality of comparison results, the candidate video segment is determined from the plurality of video segments.

7. The method of claim 6, wherein, The method comprises the steps of: If the detection target is the end of the head mark information, the plurality of video segments are classified into head segments and main segments according to the plurality of comparison results; If there is a head segment and a main segment adjacent in playing order in the plurality of video segments, and the main segment is played after the head segment is played, the head segment and the main segment are determined as the first candidate video segment.

8. The method of claim 6, wherein, The method comprises the steps of: If the detection target is the start of the tail mark information, the plurality of video segments are classified into main segments and tail segments according to the plurality of comparison results; If there is a main segment and a tail segment adjacent in playing order in the plurality of video segments, and the tail segment is played after the main segment is played, the main segment and the tail segment are determined as the second candidate video segment.

9. The method of claim 6, wherein, The method comprises the steps of: If the detection target is the end of the tail mark information, the plurality of video segments are classified into main segments and tail segments according to the plurality of comparison results; If there is a tail segment and a main segment adjacent in playing order in the plurality of video segments, and the main segment is played after the tail segment is played, the tail segment and the main segment are determined as the third candidate video segment.

10. The method of claim 7, wherein, The text recognition result includes a first text recognition result obtained by performing text recognition on image frames of the first candidate video segment, and the target image frame in which the detection target is located is determined according to the text recognition result, including: Text content in the first text recognition result is matched with a preset keyword, and after an image frame containing the preset keyword is matched, the image frame containing the preset keyword is tracked, and the last image frame containing the preset keyword tracked is determined as the target image frame in which the end-of-lead-in mark information is located.

11. The method of claim 8, wherein, The text recognition result includes a second text recognition result obtained by performing text recognition on image frames of the second candidate video segment, and the target image frame in which the detection target is located is determined according to the text recognition result, including: The second text recognition results corresponding to the image frames in the second candidate video segment are traversed in time sequence, and if the number of text boxes in the second text recognition results corresponding to a plurality of continuous image frames is greater than a preset number threshold, the first image frame in which the number of text boxes in the second text recognition result is greater than the preset number threshold is determined as the target image frame in which the start-of-lead-out mark information is located.

12. The method of claim 9, wherein, The text recognition result includes a third text recognition result obtained by performing text recognition on image frames of the third candidate video segment, and the target image frame in which the detection target is located is determined according to the text recognition result, including: If the third text recognition result contains a text box, the image frame containing the text box is tracked, and the last image frame containing the text box tracked is determined as the target image frame in which the end-of-lead-out mark information is located.

13. The method of claim 1, wherein, The video file is divided into a plurality of continuous video segments, including: A time period is determined according to the detection target; The video file is intercepted according to the time period to obtain an intercepted video segment; The intercepted video segment is evenly divided to obtain the plurality of continuous video segments.

14. A video detection apparatus characterized by comprising: The device includes: A first acquisition and determination module is configured to acquire a video file and determine a detection target for the video file; the detection target includes at least one of end-of-lead-in mark information, start-of-lead-out mark information, and end-of-lead-out mark information; the video file includes at least one of lead-in information, main information, lead-out information, and Easter egg information, and the playing order of the lead-in information and the main information is flexibly set; the end-of-lead-out mark information is used to determine the location of an Easter egg start time point; A second acquisition and determination module is configured to acquire a plurality of continuous video segments from the video file and determine a plurality of audio and video feature information of the plurality of video segments; An input and output module is configured to input the plurality of audio and video feature information into a pre-trained classification model to obtain a plurality of corresponding output results. The first determining module is configured to determine a candidate video segment from the multiple video segments according to the multiple output results; the candidate video segment comprises at least one of a first candidate video segment for searching the detection target as the prologue end mark information, a second candidate video segment for searching the detection target as the epilogue start mark information, and a third candidate video segment for searching the detection target as the epilogue end mark information; the first candidate video segment comprises a prologue segment and a main segment which are adjacent in playing order, and the main segment is played after the prologue segment is played; the second candidate video segment comprises a main segment and an epilogue segment which are adjacent in playing order, and the epilogue segment is played after the main segment is played; and the third candidate video segment comprises an epilogue segment and a main segment which are adjacent in playing order, and the epilogue segment is played after the main segment is played; The character recognition module is configured to perform character recognition on image frames of the candidate video segment to obtain a character recognition result. The second determining module is configured to determine a target image frame in which the detection target is located according to the character recognition result.

15. An electronic device, comprising: The computer program is stored in the computer readable storage medium and is executed by the processor to implement the steps of the video detection method according to any one of claims 1-13. The computer program is stored in the computer readable storage medium and is executed by the processor to implement the steps of the video detection method according to any one of claims 1-13.

16. A computer readable storage medium characterized by: ​

Images