A frame clamp detection method, system and electronic equipment

Abstract

The present application discloses a method, system and electronic device for detecting interframes. After determining a target frame segment in a target video, each two adjacent image frames in the target frame segment are detected respectively, so as to determine whether there are interframes between each two adjacent image frames based on the optical flow vector information of sparse points in each two adjacent image frames, thereby realizing detection of interframes in the target video, so that after the interframes are detected, the interframes can be processed in time to avoid the problem of picture incoherence or visual jumps in the video. Moreover, the interframes in the video are detected during the video processing process, so as to ensure the high efficiency of interframe detection and improve the user experience.

Description

Technical Field

[0001] The present application relates to the field of video detection, and in particular to a frame clipping detection method, system and electronic equipment. Background Art

[0002] In the field of online video editing, which pursues both efficiency and creativity, accurately integrating various scenes to weave a coherent and fascinating story has become the pursuit of every creator.

[0003] However, multi-scene video fusion will produce frame insertion, which means that the pixel difference or brightness difference between adjacent frames in the video stream exceeds the threshold. The frame insertion phenomenon will cause picture incoherence and visual jitter. Summary of the invention

[0004] In view of this, the present application provides a frame clamping detection method, system and electronic device, and the specific scheme thereof is as follows:

[0005] A frame clipping detection method, comprising:

[0006] Obtaining a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in a processing process;

[0007] Obtaining sparse points of every two adjacent image frames in the target frame segment;

[0008] Determine the optical flow vector information of the sparse points of each two adjacent image frames;

[0009] It is determined whether there is a sandwiched frame between each of the two adjacent image frames based on the optical flow vector information of the sparse points of each of the two adjacent image frames.

[0010] Furthermore, the determining whether there is a sandwiched frame between each two adjacent image frames based on the optical flow vector information of the sparse points of each two adjacent image frames includes:

[0011] Determine the change information of the matched sparse points and the number of unmatched sparse points in each of the two adjacent image frames based on the optical flow vector information of the sparse points in each of the two adjacent image frames;

[0012] Whether the latter image frame of each of the two adjacent image frames is a sandwich frame is determined based on the change information of the matched sparse points in each of the two adjacent image frames and the number of the unmatched sparse points.

[0013] Further, the determining whether the latter image frame of each of the two adjacent image frames is a sandwiched frame based on the change information of the matched sparse points in each of the two adjacent image frames and the number of unmatched sparse points includes:

[0014] If it is determined based on the change information of the matched sparse points in the two adjacent image frames that the number of sparse points whose change rate of the matched sparse points is greater than the target change rate reaches a first threshold, determining the latter image frame of the two adjacent image frames as a sandwich frame;

[0015] If it is determined that the number of unmatched sparse points in the two adjacent image frames reaches a second threshold, the latter image frame of the two adjacent image frames is determined to be a sandwich frame.

[0016] Further, obtaining the target frame segment in the target video includes:

[0017] Obtaining a first jump time of a jump point in the target video determined by a user in the target video;

[0018] A first specific number of continuous image frames in the target video is determined based at least on the first jump time, and the first specific number of continuous image frames is determined as a target frame segment.

[0019] Further, determining a first specific number of continuous image frames in the target video at least based on the first jump time includes:

[0020] Determining a frame rate of the target video;

[0021] Determining a first frame stamp of the jump point based on the first jump time and the frame rate of the target video;

[0022] determining a second frame stamp based on the first frame stamp;

[0023] Determining a second jump time of the target video based on the second frame stamp and the frame rate of the target video;

[0024] The target image frame corresponding to the second jump time in the target video and a second specific number of image frames adjacent to the target image frame in the target video are determined as the first specific number of continuous image frames, wherein the first specific number is the sum of the second specific number and 1.

[0025] Further, determining the second frame stamp based on the first frame stamp includes:

[0026] Determining whether the first frame stamp is an integer;

[0027] If the first frame stamp is an integer, determining the first frame stamp as the second frame stamp;

[0028] If the first frame stamp is a non-integer, the first frame stamp is rounded down to obtain the second frame stamp.

[0029] Further, obtaining the target frame segment in the target video includes:

[0030] Determine a time period of a target frame segment in the target video;

[0031] Loading the target frame segment into a virtual file system based on a time period of the target frame segment in the target video;

[0032] The target frame segment loaded into the virtual file system is converted into specific address information so as to call the target frame segment based on the specific address information.

[0033] Furthermore, it also includes:

[0034] If it is determined that the target frame image in the target frame segment is a sandwiched frame, marking the target frame image;

[0035] Each image frame in the target frame segment is rendered, and the rendered target frame image in the target frame segment and a previous frame image adjacent to the target frame image are displayed, so that a user can determine whether the target frame image is a sandwiched frame.

[0036] A frame clamp detection system, comprising:

[0037] A first obtaining unit is used to obtain a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in a processing process;

[0038] A second obtaining unit, used for obtaining sparse points of every two adjacent image frames in the target frame segment;

[0039] A first determining unit, configured to determine optical flow vector information of the sparse points of each two adjacent image frames;

[0040] The second determining unit is configured to determine whether there is an intervening frame between each two adjacent image frames based on the optical flow vector information of the sparse points of each two adjacent image frames.

[0041] An electronic device, comprising:

[0042] a memory storing a computer program;

[0043] The processor reads the computer program stored in the memory to execute any of the methods described above.

[0044] It can be seen from the above technical scheme that the frame insertion detection method, system and electronic device disclosed in the present application obtain a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in the processing process; obtain sparse points of each two adjacent image frames in the target frame segment; determine the optical flow vector information of the sparse points of each two adjacent image frames; and determine whether there is a frame insertion in each two adjacent image frames based on the optical flow vector information of the sparse points of each two adjacent image frames. After determining the target frame segment in the target video, the scheme detects each two adjacent image frames in the target frame segment respectively, so as to determine whether there is a frame insertion in each two adjacent image frames based on the optical flow vector information of the sparse points in each two adjacent image frames, thereby realizing the detection of frame insertion in the target video, so that the frame insertion can be processed in time after the frame insertion is detected, so as to avoid the problem of picture incoherence or visual jump in the video, and the frame insertion in the video is detected during the video processing process, so as to ensure the efficiency of frame insertion detection and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a frame clamping detection method disclosed in an embodiment of the present application;

[0047] Figure 2 A flowchart of a frame clamping detection method disclosed in an embodiment of the present application;

[0048] Figure 3 A flowchart of a frame clamping detection method disclosed in an embodiment of the present application;

[0049] Figure 4 A flowchart of a frame clamping detection method disclosed in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the structure of a frame clamping detection system disclosed in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0053] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0054] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0055] The present application discloses a method for detecting a frame clip, and its flow chart is as follows: Figure 1 As shown, including:

[0056] Step S11, obtaining a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in a processing process;

[0057] Step S12, obtaining sparse points of every two adjacent image frames in the target frame segment;

[0058] Step S13, determining the optical flow vector information of the sparse points of every two adjacent image frames;

[0059] Step S14: determining whether there is an intermediate frame between every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames.

[0060] In the field of online video editing, which pursues both efficiency and creativity, accurately integrating various scenes to weave a coherent and fascinating story has become the pursuit of every creator.

[0061] However, multi-scene video fusion will produce frame insertion, which means that the pixel difference or brightness difference between adjacent frames in the video stream exceeds the threshold. The frame insertion phenomenon will cause picture incoherence and visual jitter.

[0062] Based on this, the present embodiment discloses a frame-binding detection method, which is applied to the online video editing process. For the video being edited, that is, the target video, the target frame segment is determined, and based on the optical flow vector information of the sparse points of each two adjacent image frames in the target frame segment, it is determined whether there are frame-binding frames between each two adjacent image frames, thereby realizing the detection of frame-binding frames in the target video during the editing process, achieving the characteristics of high efficiency and real-time, so as to ensure that the target video is optimized at the first time, avoid rework in the post-production process of the target video, and improve work efficiency.

[0063] Specifically, during the process of processing the target video, the target frame segment is obtained to detect the frame insertion, which ensures that the frame insertion can be discovered in time during the process of processing the target video so that it can be processed in time, thereby ensuring the efficiency of the frame insertion processing in the video.

[0064] The processing of the target video may include: performing a production process such as editing and splicing on the target video, and the target frame segment is a first specific number of continuous image frames in the target video, that is, the target frame segment is a continuous video segment in the target video, and the continuous video segment includes the first specific number of image frames. The target frame segment may be any frame segment in the target video, or a frame segment in the target video selected by a user, or a frame segment selected from the target video according to a specific rule.

[0065] After determining the target frame segment, every two adjacent image frames in the target frame segment may be compared. For example, if the target frame segment includes 10 image frames, the first image frame is compared with the second image frame, the second image frame is compared with the third image frame, the third image frame is compared with the fourth image frame, and so on until the ninth image frame is compared with the tenth image frame, to ensure that except for the first image frame, the other image frames can be compared with the previous image frame to determine whether they are sandwiched frames relative to the previous image frame.

[0066] Specifically, the sparse points of every two adjacent image frames in the target frame segment are determined, and the sparse points are key feature points in the image frames. The optical flow vector information of each sparse point is determined by using the sparse optical flow method. Based on the optical flow vector information of each sparse point of every two adjacent image frames, it is determined whether there is a sandwiched frame between the two adjacent image frames.

[0067] That is, first determine the sparse points of each image frame in two adjacent image frames, then determine the matching sparse points on the two adjacent image frames, and determine whether there is an obvious and huge change between the two image frames based on the matched sparse points, that is, match the sparse points on the two image frames, and calculate the one-to-one correspondence, movement direction, speed and other data while matching. Based on these data, it can be determined whether the latter image frame of the two image frames has obvious pixel differences or brightness differences relative to the previous image frame. If so, it can be determined that the latter image frame of the two image frames is an intervening frame. If not, it can be determined that there is no intervening frame between the two image frames.

[0068] You can first introduce the OpenCv.js library to read and display image frames. The OpenCv.js library is an open source, cross-platform computer vision library that allows developers to use JavaScript in a web browser for image processing and computer vision tasks.

[0069] The Lucas-Kanade algorithm can also be used to calculate the sparse optical flow vector field of each two adjacent image frames. The Lucas-Kanade algorithm is an algorithm used in the fields of image alignment, optical flow, target tracking, image stitching, and face detection. The sparse optical flow vector field is a two-dimensional vector field used to represent the motion pattern of a specific set of points in an image, that is, between two consecutive frames of images, the motion pattern of an object or scene is described by calculating the movement vector of a selected specific point.

[0070] The target frame segment includes a first specific number of continuous image frames, so whether there are intervening frames can be judged starting from the first image frame and the second image frame in the target frame segment. After the first image frame and the second image frame are detected to determine that there are no intervening frames, the second image frame and the third image frame are judged to determine whether there are intervening frames. That is, the continuous image frames in the target frame segment are judged in sequence. Even after it is determined that there are intervening frames, subsequent judgments are continued to avoid missing intervening frames.

[0071] In addition, it can also be: since it is necessary to determine whether there is an intervening frame between every two adjacent image frames in the target frame segment based on the optical flow vector information of sparse points, it is necessary to determine whether there is an intervening frame for multiple groups of image frames based on the optical flow vector information of sparse points. Therefore, it is possible to simultaneously determine whether there is an intervening frame for multiple groups of image frames separately, so as to improve the efficiency of intervening frame detection.

[0072] When it is detected that there are interframes in the target video, prompts can be given in time so that users can deal with the interframes in time during the editing process of the target video, avoiding rework after production is completed, thereby improving the creation efficiency and quality of video files.

[0073] The frame insertion detection method disclosed in the present embodiment obtains a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in the processing process; obtains sparse points of every two adjacent image frames in the target frame segment; determines the optical flow vector information of the sparse points of every two adjacent image frames; and determines whether there is a frame insertion in every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames. After determining the target frame segment in the target video, this scheme detects every two adjacent image frames in the target frame segment respectively, so as to determine whether there is a frame insertion in every two adjacent image frames based on the optical flow vector information of the sparse points in every two adjacent image frames, thereby realizing the detection of frame insertion in the target video, so that after the frame insertion is detected, the frame insertion can be processed in time to avoid the problem of picture incoherence or visual jump in the video, and the frame insertion in the video is detected during the video processing process, which ensures the efficiency of frame insertion detection and improves the user experience.

[0074] This embodiment discloses a frame clamping detection method, and its flow chart is as follows: Figure 2 As shown, including:

[0075] Step S21, obtaining a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in a processing process;

[0076] Step S22, obtaining sparse points of every two adjacent image frames in the target frame segment;

[0077] Step S23, determining the optical flow vector information of the sparse points of every two adjacent image frames;

[0078] Step S24, determining the change information of the matched sparse points and the number of unmatched sparse points in each two adjacent image frames based on the optical flow vector information of the sparse points in each two adjacent image frames;

[0079] Step S25: determining whether the latter image frame between each two adjacent image frames is a sandwich frame based on the change information of the matched sparse points between each two adjacent image frames and the number of unmatched sparse points.

[0080] In the process of editing the target video, a target frame segment is determined, and based on the optical flow vector information of the sparse points of each two adjacent image frames in the target frame segment, it is determined whether there is a sandwiched frame between each two adjacent image frames.

[0081] Specifically, after determining the sparse points on each of two adjacent image frames, the sparse points on the two image frames need to be matched. For example, if the previous image frame has the first, second and third sparse points, and the next image frame has the fourth, fifth and sixth sparse points, the sparse points in the previous image frame need to be matched with the sparse points in the next image frame, such as: the first sparse point matches the fourth sparse point, the second sparse point matches the sixth sparse point, and the third sparse point matches the fifth sparse point.

[0082] Determine the change information of the sparse points that can be matched in two adjacent image frames and the number of the sparse points that cannot be matched.

[0083] If a sparse point in the previous image frame matches a sparse point in the next image frame, it is necessary to determine the change of the sparse point in the next image frame relative to the sparse point in the previous image frame, such as brightness change, position change, etc., determine its rate of change, and determine whether this point is a frame-interlaced point based on the rate of change.

[0084] If the change rate is greater than the target change rate, it indicates that the change rate of the sparse point is too large, and it can be determined as a sandwich point. The number of sandwich points among the matching sparse points in the two image frames is determined. If the number of sandwich points reaches a first threshold, the latter image frame of the two image frames can be determined as a sandwich frame.

[0085] When there are too many sandwiched frame points in the matched sparse points in two adjacent image frames, that is, there are too many sparse points with a large change rate, it means that the gap between the latter image frame and the previous image frame is large. At this time, the latter image frame can be determined as a sandwiched frame.

[0086] That is, if the number of sparse points whose change rate of matched sparse points is greater than the target change rate reaches a first threshold based on the change information of the matched sparse points in two adjacent image frames, the latter image frame of the two adjacent image frames is determined to be a sandwich frame.

[0087] After determining the change information of the matched sparse points in two adjacent image frames, it is also necessary to determine the number of unmatched sparse points in the two adjacent image frames. If the number of unmatched sparse points reaches a second threshold, the latter image frame of the two adjacent image frames can be directly determined as the sandwiched frame.

[0088] Furthermore, it can be specifically as follows: if the number of unmatched sparse points in two adjacent image frames reaches a second threshold, and the matching rate exceeds a predetermined pixel grid, the latter image frame of the two adjacent image frames is directly determined as a sandwich frame.

[0089] Alternatively, if the number of sparse points whose change rate of matched sparse points in two adjacent image frames is greater than the target change rate does not reach the first threshold, the number of unmatched sparse points can be used to jointly determine whether the latter image frame is an intermediate frame, such as: if the number of sparse points whose change rate of matched sparse points in two adjacent image frames is greater than the target change rate does not reach the first threshold, but reaches the third threshold, and the number of unmatched sparse points does not reach the second threshold, but reaches the fourth threshold, then it can also be determined that the latter image frame between the two adjacent image frames is an intermediate frame.

[0090] The present embodiment discloses a method for detecting intervening frames, which obtains a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in the process of being processed; obtains sparse points of every two adjacent image frames in the target frame segment; determines the optical flow vector information of the sparse points of every two adjacent image frames; determines the change information of the matched sparse points and the number of unmatched sparse points in every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames; determines whether the latter image frame in every two adjacent image frames is an intervening frame based on the change information of the matched sparse points and the number of unmatched sparse points in every two adjacent image frames. This scheme can determine the change information of the matched sparse points and the number of unmatched sparse points in the two adjacent image frames by comparing the optical flow vector information of the sparse points of the two adjacent image frames, thereby determining whether the latter image frame in the two adjacent image frames is an intervening frame, and ensures the accuracy of the intervening frame detection while realizing the detection of intervening frames in the process of processing the target video.

[0091] This embodiment discloses a frame clamping detection method, and its flow chart is as follows: Figure 3 As shown, including:

[0092] Step S31, obtaining a first jump time of a jump point in a target video determined by a user in the target video;

[0093] Step S32: determining a first specific number of continuous image frames in the target video based at least on the first jump time, and determining the first specific number of continuous image frames as the target frame segment;

[0094] Step S33, obtaining sparse points of every two adjacent image frames in the target frame segment;

[0095] Step S34, determining the optical flow vector information of the sparse points of every two adjacent image frames;

[0096] Step S35 : determining whether there is an intermediate frame between every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames.

[0097] When determining the target frame segment in the target video, it can be determined based on the jump point selected by the user, that is, the user can select a jump point in the target video during the editing process of the target video. The jump point can be a position where the user believes that there may be a frame clip. After determining the jump point selected by the user, the continuous image frames near the jump point are determined as the target frame segment.

[0098] Specifically, after determining the jump point in the target video, the time of the jump point in the target video, that is, the first jump time, is determined. The determined target frame segment is a continuous image frame surrounding the first jump time. For example, if the target frame segment is 20 continuous image frames, the target frame segment includes 10 image frames before the first jump time and 10 image frames after the first jump time.

[0099] Further, the target frame segment in the target video is determined based on the first jump time, which can be:

[0100] Determine the frame rate of the target video; determine the first frame stamp of the jump point based on the first jump time and the frame rate of the target video; determine the second frame stamp based on the first frame stamp; determine the second jump time of the target video based on the second frame stamp and the frame rate of the target video; determine the target image frame corresponding to the second jump time in the target video and a second specific number of image frames adjacent to the target image frame in the target video as a first specific number of continuous image frames, wherein the first specific number is the sum of the second specific number and 1.

[0101] Among them, the first jump time corresponding to the jump point selected by the user may not fall on an integer frame stamp and does not correspond to an image frame. Therefore, after determining the first jump time corresponding to the jump point, the first jump time needs to be adjusted to obtain the second jump time, so as to realize automatic correction of the jump point to ensure consistency on different decoders during preview, and determine a continuous first specific number of image frames based on the second jump time.

[0102] The frame rate of the target video can be determined first, such as fps25 frames / second. After obtaining the first jump time (in seconds) corresponding to the jump point in the target video, the first jump time is multiplied by the frame rate of the target video to obtain the first frame stamp corresponding to the jump point. The first frame stamp may not be an integer. For example, if the frame rate is fps25 frames / second and the first jump time is 00:00:01.233, the first frame stamp can be calculated as 00:00:01.233×25=30.825, which is not an integer and cannot correspond to an image frame. Therefore, the first frame stamp needs to be adjusted to obtain the second frame stamp, and the second jump time is obtained based on the second frame stamp to determine the first specific number of consecutive image frames.

[0103] Among them, determining the second frame stamp based on the first frame stamp can be specifically as follows: determining whether the first frame stamp is an integer; if the first frame stamp is an integer, determining the first frame stamp as the second frame stamp; if the first frame stamp is a non-integer, rounding the first frame stamp down to obtain the second frame stamp.

[0104] That is, if the first frame stamp is an integer, then the first frame stamp does not need to be adjusted, the first frame stamp is determined as the second frame stamp, and the second jump time is directly obtained by using the first frame stamp. At this time, the second jump time is the same as the first jump time, and a first specific number of continuous image frames around the first jump time is determined;

[0105] If the first frame stamp is a non-integer, such as: the first frame stamp is 30.825, the first frame stamp is adjusted to the second frame stamp. The adjustment can be specifically rounding the first frame stamp down, and the obtained value is the second frame stamp. For example: 30.825 is rounded down to 30, and the second frame stamp is determined to be 30.

[0106] After obtaining the second frame stamp, divide the second frame stamp by the frame rate of the target video to obtain the quotient, which is the second jump time. For example, 30 / 25=00:00:01.200, and the second jump time is 00:00:01.200.

[0107] After obtaining the second jump time, the second jump time corresponds to an image frame in the target video, and the image frame corresponding to the second jump time in the target video is determined as the target image frame. Then, several consecutive image frames before the target image frame are selected, and then several consecutive image frames after the target image frame are selected. The continuous image frame obtained by combining the selected target image frame, several image frames before the target image frame, and several image frames after the target image frame can be determined as the target frame segment.

[0108] For example: the first specific number is 21, that is, the target frame segment is a video clip including 21 consecutive image frames, select the target image frame in the target video, then select the 10 consecutive image frames before the target image frame and the 10 consecutive image frames after the target image frame, and combine the selected image frames to obtain a continuous video clip including 21 image frames, which is the target frame segment.

[0109] In the frame insertion detection method disclosed in this embodiment, when determining the target frame segment in the target video, the jump point selected by the user is automatically corrected so that the target image frame finally used to determine the target frame segment falls on an integer frame stamp, ensuring consistency on different decoders during preview.

[0110] The frame-interval detection method disclosed in the present embodiment obtains the first jump time of the jump point in the target video determined by the user in the target video; determines the first specific number of continuous image frames in the target video based at least on the first jump time, and determines the first specific number of continuous image frames as the target frame segment; obtains the sparse points of every two adjacent image frames in the target frame segment; determines the optical flow vector information of the sparse points of every two adjacent image frames; and determines whether there is a frame-interval between every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames. In this scheme, when determining the target frame segment in the target video, the target frame segment is determined based on the jump point in the target video determined by the user to ensure that the target frame segment is a frame segment where frame-intervals may exist, avoids frame-interval detection for frame segments where no frame-intervals exist, and improves the efficiency of frame-interval detection.

[0111] This embodiment discloses a frame clamping detection method, and its flow chart is as follows: Figure 4 As shown, including:

[0112] Step S41, determining the time period of the target frame segment in the target video;

[0113] Step S42, loading the target frame segment into the virtual file system based on the time period of the target frame segment in the target video;

[0114] Step S43, converting the target frame segment loaded into the virtual file system into specific address information, so as to call the target frame segment based on the specific address information;

[0115] Step S44, obtaining sparse points of every two adjacent image frames in the target frame segment;

[0116] Step S45, determining the optical flow vector information of the sparse points of every two adjacent image frames;

[0117] Step S46: Determine whether there is an intermediate frame between every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames.

[0118] The frame insertion detection method disclosed in this embodiment can be applied to a browser, that is, when splicing, editing, etc. a video, the browser is used to perform frame insertion detection on the video to determine whether there are frame insertions in the video.

[0119] When performing frame insertion detection on a target frame segment, the target frame segment can be directly sent to a main thread that performs frame insertion detection so that the main thread can perform frame insertion detection on its target frame segment. Alternatively, the target frame segment is not directly sent to the main thread, but only the address of the target frame segment is sent to the main thread, so as to reduce the cost of image frame transmission between threads and improve the efficiency of frame insertion detection.

[0120] Specifically, when determining the target frame segment, it is necessary to determine the time period of the target frame segment in the target video so as to use the target video within this time period to determine the address. The ffmpeg.wasm library can be introduced, and multi-threading support can be enabled to extract and process each image frame in parallel. In order not to interrupt user interaction, multi-threading can be used to avoid blocking the main thread.

[0121] Among them, the ffmpeg.wasm library is a WebAssembly module that allows the use of FFmpeg functions in web browsers. ffmpeg.wasm compiles FFmpeg's libav library into a WebAssembly format file, allowing developers to directly call FFmpeg (an open source audio and video processing tool that supports conversion and editing of various audio and video formats) in the browser environment. The powerful multimedia processing function; through WebAssembly technology, ffmpeg.wasm can efficiently perform video and audio processing tasks in the browser, such as transcoding, editing, and streaming media transmission.

[0122] Afterwards, the target frame segment is loaded into the virtual file system using the time period of the target frame segment in the target video. FFmpeg can be used to load the target frame segment into the virtual file system. Due to the security restrictions of the browser, the code cannot directly read the resource, but the browser provides a virtual file system, which can realize the normal reading and writing of files through FFmpeg.

[0123] Specifically, after determining the time period of the target frame segment in the target video, the frame stamp of each image frame in the target frame segment is determined, and FFmpeg is executed using multiple threads to retrieve data from the virtual file system, parse the image frames, and then write the image frames into the virtual file system; thereafter, the image frames in the virtual file system are converted into specific address information so that the main thread can display and call it, wherein the specific address information is: a url (uniform resource locator) address that can be directly displayed in a browser.

[0124] After converting the target frame segment loaded into the virtual file system into specific address information, the main thread that performs frame clamping detection can directly call the specific address information when performing frame clamping detection, thereby reducing the problem of slow transmission efficiency caused by operations such as directly copying the target frame segment when detecting the target frame segment, thereby reducing the efficiency of frame clamping detection.

[0125] The frame-interval detection method disclosed in the present embodiment determines the time period of the target frame segment in the target video; loads the target frame segment into the virtual file system based on the time period of the target frame segment in the target video; converts the target frame segment loaded into the virtual file system into specific address information so as to call the target frame segment based on the specific address information; obtains the sparse points of every two adjacent image frames in the target frame segment; determines the optical flow vector information of the sparse points of every two adjacent image frames; determines whether there is a frame-interval between every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames. In this scheme, after determining the target frame segment in the target video, during the frame-interval detection process for the target frame segment, only the address of the target frame segment is sent to the main thread for frame-interval detection, so that the target frame segment is directly determined based on the address when performing frame-interval detection on the target frame segment, and frame-interval detection is performed, without the need to perform operations such as transmission of the target frame segment, thereby simplifying the operation process and improving the operation efficiency.

[0126] Furthermore, the frame clamping detection method disclosed in this embodiment may further include:

[0127] If it is determined that the target frame image in the target frame segment is an intervening frame, the target frame image is marked; each image frame in the target frame segment is rendered, and the target frame image in the rendered target frame segment and the previous frame image adjacent to the target frame image are displayed, so that the user can determine whether the target frame image is an intervening frame.

[0128] Based on the optical flow vector information of the sparse points of each two adjacent image frames in the target frame segment in the target video, it is determined whether there is an intermediate frame between each two adjacent image frames. If it is determined that there is an intermediate frame, such as: determining that the target frame image in the target video is an intermediate frame, at this time, the user can be directly prompted that the target frame image is an intermediate frame, such as: marking the target frame image and outputting prompt information so that the user can process the marked target frame image.

[0129] Alternatively, the target frame image may not be directly determined as an intervening frame, but the target frame image may be determined as a suspicious intervening frame, and the suspicious intervening frame may be marked and output for further confirmation by the user to determine whether the target frame image can be determined as an intervening frame.

[0130] The target frame image can be marked, and each image frame in the target frame segment can be rendered and displayed on the video axis after rendering so that the user can confirm whether the target frame image can be determined as a clip frame, wherein the video axis is the axis corresponding to the time-frame image when the user performs editing, splicing, etc. on the target video.

[0131] Specifically, each image frame in the target frame segment can be drawn on a canvas as a base map, and the frame-binding points in the target frame image can be rendered on the corresponding canvas; the rendered target frame segment can be displayed on the video axis so that the user can confirm each image frame displayed on the video axis to determine whether the target frame image is a frame-binding point; or, the rendered target frame segment can be displayed on the video axis, and when the user clicks on the target frame image, the target frame image and the previous frame image of the target frame image can be displayed so that the user can accurately determine whether the two adjacent image frames are very different, and the user can finally determine whether the target frame image is a frame-binding point; or, instead of directly drawing each image frame in the target frame segment on the canvas as a base map, the specific address information corresponding to each image frame in the target frame segment can be directly drawn using the canvas as a base map.

[0132] Among them, canvas is an HTML element that provides a canvas in which you can draw.

[0133] In the frame clipping detection method disclosed in this embodiment, a step is added in which the user further confirms whether the target frame image is a frame clipping, thereby improving the accuracy of frame clipping detection and ensuring user experience.

[0134] This embodiment discloses a frame clamping detection system, and its structural schematic diagram is shown in FIG. Figure 5 As shown, including:

[0135] A first obtaining unit 51 , a second obtaining unit 52 , a first determining unit 53 and a second determining unit 54 .

[0136] The first obtaining unit 51 is used to obtain a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in the processing process;

[0137] The second obtaining unit 52 is used to obtain the sparse points of every two adjacent image frames in the target frame segment;

[0138] The first determining unit 53 is used to determine the optical flow vector information of the sparse points of every two adjacent image frames;

[0139] The second determining unit 54 is used to determine whether there is an intermediate frame between every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames.

[0140] Further, the second determining unit is used for:

[0141] Based on the optical flow vector information of the sparse points of each two adjacent image frames, the change information of the matched sparse points and the number of unmatched sparse points in each two adjacent image frames are determined; based on the change information of the matched sparse points and the number of unmatched sparse points in each two adjacent image frames, it is determined whether the latter image frame of each two adjacent image frames is a sandwich frame.

[0142] Further, the second determining unit is used for:

[0143] If it is determined based on the change information of the matched sparse points in two adjacent image frames that the number of sparse points whose change rate of the matched sparse points is greater than the target change rate reaches a first threshold, the latter image frame of the two adjacent image frames is determined to be a sandwich frame; if it is determined that the number of unmatched sparse points in the two adjacent image frames reaches a second threshold, the latter image frame of the two adjacent image frames is determined to be a sandwich frame.

[0144] Furthermore, the first obtaining unit is used for:

[0145] A first jump time of a jump point in a target video determined by a user is obtained in the target video; a first specific number of continuous image frames in the target video is determined based at least on the first jump time, and the first specific number of continuous image frames is determined as a target frame segment.

[0146] Furthermore, the first obtaining unit is used for:

[0147] Determine the frame rate of the target video; determine the first frame stamp of the jump point based on the first jump time and the frame rate of the target video; determine the second frame stamp based on the first frame stamp; determine the second jump time of the target video based on the second frame stamp and the frame rate of the target video; determine the target image frame corresponding to the second jump time in the target video and a second specific number of image frames adjacent to the target image frame in the target video as a first specific number of continuous image frames, wherein the first specific number is the sum of the second specific number and 1.

[0148] Furthermore, the first obtaining unit is used for:

[0149] Determine whether the first frame stamp is an integer; if the first frame stamp is an integer, determine the first frame stamp as the second frame stamp; if the first frame stamp is a non-integer, round the first frame stamp down to obtain the second frame stamp.

[0150] Furthermore, the first obtaining unit is used for:

[0151] Determine a time period of a target frame segment in a target video; load the target frame segment into a virtual file system based on the time period of the target frame segment in the target video; convert the target frame segment loaded into the virtual file system into specific address information so as to call the target frame segment based on the specific address information.

[0152] Furthermore, the frame clamping detection system disclosed in this embodiment may also include:

[0153] The display unit is used to mark the target frame image when determining that the target frame image in the target frame segment is an intervening frame; render each image frame in the target frame segment, and display the target frame image in the rendered target frame segment and the previous frame image adjacent to the target frame image, so that the user can determine whether the target frame image is an intervening frame.

[0154] The frame clamping detection system disclosed in this embodiment is implemented based on the frame clamping detection method disclosed in the above embodiment, which will not be described in detail here.

[0155] The frame-interval detection system disclosed in the present embodiment obtains a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in the processing process; obtains sparse points of every two adjacent image frames in the target frame segment; determines the optical flow vector information of the sparse points of every two adjacent image frames; and determines whether there is a frame-interval between every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames. After determining the target frame segment in the target video, the present scheme detects every two adjacent image frames in the target frame segment respectively, so as to determine whether there is a frame-interval between every two adjacent image frames based on the optical flow vector information of the sparse points in every two adjacent image frames, thereby realizing the detection of frame-interval in the target video, so that the frame-interval can be processed in time after being detected, so as to avoid the problem of picture incoherence or visual jump in the video, and the frame-interval in the video is detected during the video processing process, so as to ensure the efficiency of frame-interval detection and improve the user experience.

[0156] This embodiment discloses an electronic device, and its structural diagram is as follows: Figure 6 As shown, including:

[0157] Processor 61 and memory 62.

[0158] The memory 62 stores computer programs.

[0159] The processor 61 is used to read the computer program stored in the memory 62 to execute the frame clamping detection method disclosed in the above embodiment.

[0160] For example: the processor reads a computer program stored in a memory to obtain a target frame segment in a target video, the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in the processing process; obtains sparse points of every two adjacent image frames in the target frame segment; determines optical flow vector information of the sparse points of every two adjacent image frames; and determines whether there is an intervening frame between every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames.

[0161] The electronic device disclosed in this embodiment is implemented based on the frame clamping detection method disclosed in the above embodiment, which will not be described in detail here.

[0162] The electronic device disclosed in this embodiment obtains a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video in the processing process; obtains sparse points of every two adjacent image frames in the target frame segment; determines the optical flow vector information of the sparse points of every two adjacent image frames; and determines whether there is an intervening frame in every two adjacent image frames based on the optical flow vector information of the sparse points of every two adjacent image frames. After determining the target frame segment in the target video, this scheme detects every two adjacent image frames in the target frame segment respectively, so as to determine whether there is an intervening frame in every two adjacent image frames based on the optical flow vector information of the sparse points in every two adjacent image frames, thereby realizing the detection of intervening frames in the target video, so that the intervening frames can be processed in time after being detected, so as to avoid the problem of incoherent picture or visual jump in the video, and the intervening frames in the video are detected during the video processing process, so as to ensure the efficiency of intervening frame detection and improve the user experience.

[0163] The embodiment of the present application further provides a readable storage medium on which a computer program is stored. The computer program is loaded and executed by a processor to implement the steps of the above-mentioned frame clamping detection method. The specific implementation process can refer to the description of the corresponding part of the above-mentioned embodiment, which will not be repeated in this embodiment.

[0164] The present application also proposes a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device executes the method provided in various optional implementations of the above-mentioned frame clamping detection method or frame clamping detection system. The specific implementation process can refer to the description of the above-mentioned corresponding embodiment, and will not be repeated here.

[0165] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0166] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0167] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0168] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A frame-interval detection method, characterized in that: include: Obtaining a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video that is being edited or spliced; Obtaining sparse points of every two adjacent image frames in the target frame segment; Determine the optical flow vector information of the sparse points of each two adjacent image frames; Determine whether there is a sandwiched frame between each of the two adjacent image frames based on the optical flow vector information of the sparse points of each of the two adjacent image frames, and when it is detected that there is a sandwiched frame in the target video, prompt it in time so that the user can process the sandwiched frame in time during the editing or splicing process of the target video; Among them, the determining whether there is an intermediate frame between each two adjacent image frames based on the optical flow vector information of the sparse points of each two adjacent image frames includes: if it is determined based on the optical flow vector information that the change rate of the matched sparse points in each two adjacent image frames is greater than the target change rate, the sparse points are determined to be intermediate frame points; if it is determined that the number of intermediate frame points among the matched sparse points in each two adjacent image frames reaches a first threshold, the latter image frame of each two adjacent image frames is determined to be an intermediate frame; if the number of sparse points whose change rate of the matched sparse points in the two adjacent image frames is greater than the target change rate does not reach the first threshold but reaches the third threshold, and the number of unmatched sparse points does not reach the second threshold but reaches the fourth threshold, the latter image frame of the two adjacent image frames is determined to be an intermediate frame.

2. The method according to claim 1, characterized in that The step of obtaining a target frame segment in a target video includes: Obtaining a first jump time of a jump point in the target video determined by a user in the target video; A first specific number of continuous image frames in the target video is determined based at least on the first jump time, and the first specific number of continuous image frames is determined as a target frame segment.

3. The method according to claim 2, characterized in that The determining a first specific number of continuous image frames in the target video based at least on the first jump time comprises: Determining a frame rate of the target video; Determining a first frame stamp of the jump point based on the first jump time and the frame rate of the target video; determining a second frame stamp based on the first frame stamp; Determining a second jump time of the target video based on the second frame stamp and the frame rate of the target video; The target image frame corresponding to the second jump time in the target video and a second specific number of image frames adjacent to the target image frame in the target video are determined as the first specific number of continuous image frames, wherein the first specific number is the sum of the second specific number and 1.

4. The method according to claim 3, characterized in that The determining a second frame stamp based on the first frame stamp comprises: Determining whether the first frame stamp is an integer; If the first frame stamp is an integer, determining the first frame stamp as the second frame stamp; If the first frame stamp is a non-integer, the first frame stamp is rounded down to obtain the second frame stamp.

5. The method according to claim 1, characterized in that The step of obtaining a target frame segment in a target video includes: Determine a time period of a target frame segment in the target video; Loading the target frame segment into a virtual file system based on a time period of the target frame segment in the target video; The target frame segment loaded into the virtual file system is converted into specific address information so as to call the target frame segment based on the specific address information.

6. The method according to claim 1, characterized in that Also includes: If it is determined that the target frame image in the target frame segment is a sandwiched frame, marking the target frame image; Each image frame in the target frame segment is rendered, and the rendered target frame image in the target frame segment and a previous frame image adjacent to the target frame image are displayed, so that a user can determine whether the target frame image is a sandwiched frame.

7. A frame clamp detection system, characterized in that: include: A first obtaining unit is used to obtain a target frame segment in a target video, wherein the target frame segment includes a first specific number of continuous image frames in the target video, and the target video is a video that is being edited or spliced; A second obtaining unit, used for obtaining sparse points of every two adjacent image frames in the target frame segment; A first determining unit, configured to determine optical flow vector information of the sparse points of each two adjacent image frames; A second determining unit is used to determine whether there is a sandwiched frame between each two adjacent image frames based on the optical flow vector information of the sparse points of each two adjacent image frames, and when it is detected that there is a sandwiched frame in the target video, prompt it in time so that the user can process the sandwiched frame in time during the editing or splicing process of the target video; Wherein, the second determination unit is used for: if it is determined based on the optical flow vector information that the change rate of the matched sparse points in each two adjacent image frames is greater than the target change rate, the sparse points are determined to be sandwich points; if it is determined that the number of sandwich points among the matched sparse points in each two adjacent image frames reaches a first threshold, the latter image frame of each two adjacent image frames is determined to be a sandwich frame; if the number of sparse points whose change rate of the matched sparse points in two adjacent image frames is greater than the target change rate does not reach the first threshold but reaches the third threshold, and the number of unmatched sparse points does not reach the second threshold but reaches the fourth threshold, the latter image frame of the two adjacent image frames is determined to be a sandwich frame.

8. An electronic device, characterized in that: include: a memory storing a computer program; A processor reads the computer program stored in the memory to execute the method according to any one of claims 1 to 6.

Images