A method, device, and storage medium for detecting lens boundaries

Abstract

An embodiment of the present application provides a method, device, and storage medium for detecting lens boundaries. In the embodiment of the present application, a lens boundary detection request for a target video can be received; spatio-temporal slices are extracted from the target video; according to the spatio-temporal slices, video frames suspected of being lens boundaries in the target video are determined as candidate frames; from the candidate frames, target frames that meet the preset local feature matching requirements are selected as the lens boundary frames in the target video. Accordingly, in the embodiment of the present application, by combining spatio-temporal slices and local feature matching, video frames suspected of being lens boundaries can be preliminarily screened based on the spatio-temporal slices, and the true lens boundary frames can be further selected through local feature matching, which can greatly reduce the amount of calculation and can search for lens boundary frames at multiple levels. Therefore, the efficiency, recall rate, and accuracy of lens boundary detection can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of video processing technologies, and in particular, to a method, device, and storage medium for detecting shot boundaries. Background Art

[0002] As the basic semantic unit of a video, the detection of shot boundaries is the basis for video segmentation and video indexing, and is also the key to semantic acquisition and content analysis in video retrieval.

[0003] Currently, deep learning methods are increasingly used to solve the problem of shot boundary detection. However, the operations of deep learning are very time-consuming, and there are many forms of shot transitions in videos. Especially with the popularity of short videos, various new transition effects emerge in an endless stream, resulting in high implementation costs and insufficient detection performance of the detection model. Summary of the Invention

[0004] Multiple aspects of this application provide a method, device, and storage medium for detecting shot boundaries to improve the efficiency and / or accuracy of shot boundary detection.

[0005] An embodiment of this application provides a method for detecting shot boundaries, including:

[0006] Receiving a shot boundary detection request for a target video;

[0007] Extracting spatio-temporal slices from the target video;

[0008] Based on the spatio-temporal slices, determining video frames suspected of shot boundaries in the target video as candidate frames;

[0009] Selecting target frames that meet the preset local feature matching requirements from the candidate frames as the shot boundary frames in the target video.

[0010] An embodiment of this application also provides a computing device, including a memory and a processor;

[0011] The memory is used to store one or more computer instructions;

[0012] The processor is coupled to the memory and is used to execute the one or more computer instructions for:

[0013] Receiving a shot boundary detection request for a target video;

[0014] Extracting spatio-temporal slices from the target video;

[0015] Based on the spatio-temporal slices, determining video frames suspected of shot boundaries in the target video as candidate frames;

[0016] From the candidate frames, select a target frame that meets the preset local feature matching requirements as the shot boundary frame in the target video.

[0017] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the foregoing shot boundary detection method.

[0018] In an embodiment of the present application, a shot boundary detection request for a target video may be received; spatio-temporal slices are extracted from the target video; according to the spatio-temporal slices, video frames suspected of being shot boundaries are determined in the target video as candidate frames; from the candidate frames, a target frame that meets the preset local feature matching requirements is selected as the shot boundary frame in the target video. Accordingly, in the embodiment of the present application, a combination of spatio-temporal slices and local feature matching is adopted. Based on the spatio-temporal slices, video frames suspected of being shot boundaries can be initially screened out, and through local feature matching, the true shot boundary frames can be further selected. This can greatly reduce the computational amount and can search for shot boundary frames at multiple levels. Therefore, the efficiency, recall rate, and accuracy of shot boundary detection can be effectively improved. Description of the Drawings

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0020] Figure 1 is a flowchart of a shot boundary detection method provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a logical diagram of a shot boundary detection solution provided by an exemplary embodiment of the present application

[0022] Figure 3 is a flowchart of an implementation manner of a shot boundary detection provided by an exemplary embodiment of the present application;

[0023] Figure 4 is a logical diagram of another implementation manner of the shot boundary detection method provided by an exemplary embodiment of the present application;

[0024] Figure 5 is a structural diagram of a computing device provided by another exemplary embodiment of the present application. Detailed Description of the Embodiment

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments of this application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0026] In view of the technical problems of large computational time consumption and low accuracy in existing deep learning methods for shot boundary detection, in some embodiments of the embodiments of this application: a shot boundary detection request for a target video can be received; spatio-temporal slices can be extracted from the target video; according to the spatio-temporal slices, video frames suspected of being shot boundaries are determined in the target video as candidate frames; from the candidate frames, target frames that meet the preset local feature matching requirements are selected as the shot boundary frames in the target video. Accordingly, in the embodiments of this application, a method combining spatio-temporal slices and local feature matching is adopted. Based on the spatio-temporal slices, video frames suspected of being shot boundaries can be preliminarily screened out, and through local feature matching, the real shot boundary frames can be further selected. This can greatly reduce the amount of calculation and can search for shot boundary frames at multiple levels. Therefore, the efficiency, recall rate, and accuracy of shot boundary detection can be effectively improved.

[0027] Before introducing the technical solutions proposed in this application, several technical concepts will be explained first:

[0028] A shot is a segment composed of several consecutive video frames in time and is a basic unit for describing a continuous scene in a video. Usually, a video contains multiple shots, there are boundaries between shots, and the shots can be switched with each other.

[0029] Shot boundary detection refers to the process of segmenting and extracting shots from a video, and usually shot boundary frames can be used to represent the boundaries of shots.

[0030] The technical solutions provided by each embodiment of this application will be described in detail below in conjunction with the drawings.

[0031] Figure 1 It is a schematic flowchart of a shot boundary detection method provided for an exemplary embodiment of this application. Figure 2 It is a schematic logical diagram of a shot boundary detection scheme provided for an exemplary embodiment of this application. The shot boundary detection method provided in this embodiment can be executed by a shot boundary detection device, and the shot boundary detection device can be implemented as software or as a combination of software and hardware, and the shot boundary detection device can be integrally arranged in a computing device. As Figure 1 shown, the method includes:

[0032] Step 100, receive a shot boundary detection request for a target video;

[0033] Step 101: Extract spatio-temporal slices from the target video;

[0034] Step 102: Determine video frames suspected of being shot boundaries in the target video based on the spatio-temporal slices as candidate frames;

[0035] Step 103: Select target frames that meet the preset local feature matching requirements from the candidate frames as the shot boundary frames in the target video.

[0036] The shot boundary detection method provided in this embodiment can be applied to various scenarios that require shot boundary detection, such as video segmentation, video indexing, video content analysis, etc. Of course, it can also be used in video teaching, excellent video source teaching, reflecting the transition connection art, etc. This embodiment does not limit the application scenario.

[0037] In step 100, a shot detection request for the target video can be received. Among them, the target video can be a video in any scenario that requires shot boundary detection. This embodiment does not limit the attributes such as the specifications and formats of the target video and the video content in the target video.

[0038] Based on this, in step 101, spatio-temporal slices can be extracted from the target video. Among them, the spatio-temporal slice is a two-dimensional image composed of pixel lines extracted at fixed positions in a continuous video frame sequence. The pixel line refers to the combination of pixel points on a line in the video frame. In this embodiment, L rows (columns) of pixels can be extracted from the same position of at least one video frame included in the target video frame to form a two-dimensional image to obtain the spatio-temporal slice. Among them, in this embodiment, the value of L can be set according to actual needs. For example, L can be set to 3. In this way, 3 rows (columns) of pixels can be extracted at the specified position of a single video frame, so as to form a two-dimensional image with the pixels extracted from other video frames.

[0039] Based on this, in step 102, video frames suspected of being shot boundaries in the target video can be determined based on the spatio-temporal slices as candidate frames. Among them, the spatio-temporal slice may include slice features such as color and texture. The inventor found during the research process that since the video frames within a single shot are usually continuous in time, space, and image structure, therefore, on the spatio-temporal slice, it is usually manifested as the continuity of the slice features. When the shot is switched, the performance on the spatio-temporal slice is usually manifested as an obvious discontinuity of the slice features.

[0040] To this end, in this embodiment, in step 102, video frames that conform to the characteristics of shot transitions can be searched according to the continuity of the slice features in the spatio-temporal slice, and such video frames can be determined as video frames suspected of being shot boundaries. It should be understood that in step 102, shot boundary detection can be performed based on the spatio-temporal slice, but the accuracy of the shot boundaries detected in this way is insufficient. Therefore, in this embodiment, the results of shot boundary detection based on the spatio-temporal slice are described as video frames suspected of being shot boundaries.

[0041] In this embodiment, the video frames suspected of being shot boundaries determined in the target video according to the spatio-temporal slice can be used as candidate frames. Among them, the number of candidate frames can be multiple. It is worth noting that in practical applications, the starting station in the shot can be used as the shot boundary frame to represent the shot boundary. Therefore, in this embodiment, the video frames suspected of being shot boundaries determined in the target video according to the spatio-temporal slice can select the video frames suspected of being the start of the shot, that is, the suspected shot start frames. Of course, this embodiment does not limit this, and the end frame of the shot can also be used to represent the shot boundary.

[0042] In this embodiment, at least one video frame in the target video can be traversed to determine whether at least one video frame can be used as a candidate frame respectively. Since the analysis process for each video frame is similar, the process of determining candidate frames will be described below taking the first video frame in the target video as an example. It should be understood that the first video frame can be any one of at least one video frame in the target video.

[0043] In this embodiment, on the spatio-temporal slice, if the slice features corresponding to the first video frame meet the preset continuity requirements, the first video frame can be determined as a video frame suspected of being a shot boundary, that is, a candidate frame.

[0044] In an alternative implementation, the difference degree between the slice features of the first video frame and its previous frame can be calculated. If the difference degree is greater than the preset threshold, it can be determined that the slice features corresponding to the first video frame meet the preset continuity requirements and be used as a candidate frame. Of course, the difference degree between the slice features of the first video frame and its next frame can also be calculated. If the difference degree is greater than the preset threshold, the first video frame can be determined as a candidate frame.

[0045] In this implementation, based on the difference degree of the slice features between each video frame in the target video and its adjacent frames, the difference degrees greater than the preset threshold can be selected from them, so as to directly screen out the candidate frames. However, the inventor found in the research process that the accuracy of the candidate frames screened by this implementation is insufficient, and the order of magnitude of the candidate frames is still relatively large.

[0046] In another alternative implementation, the first video frame and its corresponding N previous frames and M subsequent frames can be obtained to form a first frame image sequence, where N and M are positive integers; calculate the difference degree between the first video frame and the slice features corresponding to its adjacent previous frame, as well as the difference degrees between the slice features corresponding to other adjacent frames in the first frame image sequence; perform mutation detection on the difference degrees under the first frame image sequence; if a difference degree mutation is detected on the first video frame, it is determined that the slice features corresponding to the first video frame meet the preset continuity requirements and are used as candidate frames.

[0047] Figure 3 The flowchart of an implementation of a lens boundary detection provided by an exemplary embodiment of the present application. Refer to Figure 3 , in this implementation, a frame image queue can be maintained, and in practical applications, a first-in-first-out queue or the like can be used. When the first video frame is traversed, the first video frame and its corresponding N previous frames and M subsequent frames can be loaded into the frame image queue in sequence, that is, the first frame image sequence is loaded into the frame image queue. For example, N can be set to 6 and M can be set to 7. In this way, the first frame image sequence can contain 14 video frames and be loaded into the frame image queue. When starting the analysis process of the next video frame of the first video frame, the video frames in the frame image queue can be replaced with the previous frames and subsequent frames of the next video frame.

[0048] Refer to Figure 3 , in this implementation, a difference degree queue can also be maintained. Continuing with the above example, for the first video frame, under the first frame image sequence, 13 difference degrees can be obtained, and these 13 difference degrees can be loaded into the difference degree queue, and mutation detection can be performed in the difference degree queue.

[0049] In the process of performing mutation detection on the difference degrees under the first frame image, a difference degree change curve can be constructed according to the difference degrees under the first frame image sequence; if the difference degree corresponding to the first video frame is located in the peak region of the difference degree change curve, it is determined that a difference degree mutation is detected on the first video frame. Among them, the peak region not only includes the peak, but also can include the region near the peak. For example, if in a specified coordinate system, the horizontal axis represents the video frame and the vertical axis represents the difference degree, then from the perspective of the difference degree change curve, if the first video frame just lies at the peak, it can be determined that the first video frame is a candidate frame; of course, if the frame distance between the first video and the video frame at the peak is less than a preset distance threshold, for example, the frame distance is 2, which is less than the preset distance of 5, it can also be determined that the first video frame is a candidate frame.

[0050] Accordingly, in this implementation, the first video frame and the sequence of frame images formed by its previous frame and subsequent frame can be used as the analysis unit, and the change state of the difference degree of the slice features of each adjacent frame can be analyzed in the sequence of frame images. If a mutation is detected in the first video frame, the first video frame can be determined as a candidate frame. In this way, a broader reference frame can be used to detect whether the first video frame has the characteristics of a shot boundary. Therefore, compared with the method of directly analyzing the absolute value of the difference degree between the first video frame and its adjacent frame in the previous implementation, higher analysis accuracy can be obtained, which can effectively reduce the number of candidate frames and improve the accuracy rate of candidate frames.

[0051] On the basis of determining the candidate frames, referring to Figures 1-3 , in step 103, a target frame that meets the preset local feature matching requirements can be selected from the candidate frames as the shot boundary frame in the target video.

[0052] In this embodiment, the local features can adopt fast oriented rotation ORB (Oriented Fast and Rotated Brief) features, accelerated robust SURF (speed up robust feature) features, scale invariant feature transform SIFT (Scale Invariant Feature Transform) features, etc. This embodiment does not limit the type of local features.

[0053] The following will take the ORB feature as an example to exemplarily illustrate the process of local feature matching. Since the processing process of each candidate frame is similar, the following will take the first candidate frame as an example to illustrate the process of determining local feature matching. It should be understood that the first candidate frame can be any one of the candidate frames.

[0054] In one implementation, the previous frame of the first candidate frame can be determined in the target video; the local features in the first candidate frame and its previous frame are extracted respectively; if the local features between the first candidate frame and its previous frame match, the first candidate frame is determined as the target frame. By performing local feature matching on the candidate frames, a secondary judgment of the candidate frames can be realized to determine whether the candidate frames are shot boundary frames, which can effectively improve the recall rate and accuracy rate of shot boundary detection.

[0055] Among them, in step 103, the frame image queue mentioned in the previous text can be continued to be used. Therefore, the previous frame of the first candidate frame can be quickly determined in the frame image queue.

[0056] On this basis, ORB local features can be extracted from the first candidate frame and its previous frame. In practical applications, the feature points in the first candidate frame and its previous frame can be determined first. For example, the FAST feature point detection method can be used to detect the feature points, and then the Harris corner measurement method can be used to select P feature points with the largest Harris corner response value from the FAST feature points. After that, BRIEF can be used as the feature description method to generate the BRIEF descriptors of each feature point. For example, the BRIEF descriptor can be a binary code string with a length of n.

[0057] In this implementation manner, a feature matching scheme based on Grid-based Motion Statistics (GMS) can be adopted to determine whether the local features between the first candidate frame and its previous frame match. In practical applications, a preliminary comparison can be performed based on the BRIEF descriptors of the feature points determined above. For example, each feature point in the first candidate frame can be traversed to find the matching points in its previous frame respectively based on the BRIEF descriptor. There will be many matching pairs determined by the preliminary comparison. After that, the first candidate frame can be divided into several grids, and each matching pair can be traversed. Taking the first matching pair as an example, the number of matching pairs existing in the grid where the first matching pair is located can be judged. If the number is large enough, it can be determined that the first matching pair is correct, so as to find the correct matching pairs in the preliminary comparison result. If the number of correct matching pairs is large enough, it can be determined that the local features of the first candidate frame and its previous frame match.

[0058] It should be noted that the above ORB local feature matching scheme is only exemplary, and this embodiment is not limited thereto. In addition, the specific matching process for other local features will not be elaborated here.

[0059] In this embodiment, annotation information can also be configured for the shot boundary frames determined in the target video to prompt the user of the shot boundary frames. In practical applications, when the target video is played to the shot boundary frame, the corresponding annotation information can be displayed. In this way, the user can timely discover the shot boundary frames, which can provide assistance for video switching, video editing, etc.

[0060] In this embodiment, different shot boundary detection schemes can also be determined according to different application requirements, that is, different processing parameters can be configured for the above spatio-temporal slicing method and local feature matching method to obtain shot boundary frames that meet different application requirements.

[0061] Accordingly, in this embodiment, on the one hand, since only the spatio-temporal slices in the target video are analyzed instead of the overall frame images, the efficiency of shot boundary detection can be effectively improved; only the local features of the candidate frames in the target video are analyzed, which can also reduce the order of magnitude of local feature analysis, thereby improving the efficiency of shot boundary detection. On the other hand, based on the local features, the candidate frames are rejudged, and the shot boundary frames in the target video can be determined more accurately. Therefore, the recall rate and accuracy of shot boundary detection can be effectively improved.

[0062] Figure 4 It is a logical schematic diagram of another implementation manner of the shot boundary detection method provided by an exemplary embodiment of the present application. Refer to Figure 4 , in the above or following embodiments, if the K consecutive video frames before the first video frame are all non-shot boundary frames, the difference degree between the first video frame and the previous shot boundary frame is calculated, where K is a positive integer; if the difference degree between the first video frame and the previous shot boundary frame is greater than the preset difference degree threshold, it is determined that the first video frame is the boundary frame of the dissolve / wipe shot.

[0063] Among them, in this embodiment, the shot boundary frame mentioned refers to the boundary frame corresponding to the ordinary shot, for example, the cut shot, etc. This embodiment can also support the boundary detection of special shots, such as the above-mentioned dissolve / wipe shot and the fade-in / fade-out shot mentioned in the subsequent embodiments.

[0064] Refer to Figure 4 , in this embodiment, K can be set to 15. In this way, if the 15 consecutive frames before the first video frame are not shot boundary frames, the first video frame can be compared with the previous shot boundary frame. If the difference between the first video frame and the previous shot boundary frame is large enough, it can be determined that the first video frame is the boundary frame of the dissolve / wipe shot.

[0065] In addition, refer to Figure 4 , if the number of the previous frames of the first video frame satisfies N and / or the number of the subsequent frames satisfies M, the mutation detection scheme provided in the previous embodiment can be used to determine whether there is a mutation on the first video frame; if the number of the previous frames of the first video frame is less than N and / or the number of the subsequent frames is less than M, the method proposed in this embodiment can be used to determine whether there is a mutation on the first video frame by referring to the previous shot boundary frame.

[0066] For example, when N is set to 6 and M is set to 7, if the first video frame is the 3rd frame in the target video, then the number of previous frames of the first video frame is less than 7, and the frame image queue is not full. In this case, the mutation detection scheme in the foregoing embodiments may not be implemented or the result may be inaccurate due to insufficient data basis. In this case, if the previous shot boundary frame is the 1st frame in the target video, the difference degree between the 3rd frame and the 1st frame in the target video can be calculated. If the difference degree is large enough, it can be determined that there is a mutation in the slice feature on the 3rd frame.

[0067] Accordingly, the detection of fade-in / fade-out or wipe shots can be realized.

[0068] Reference Figure 4 , in this embodiment, the spatio-temporal slice segment corresponding to the first frame image sequence can also be determined; the variance between the slice pixels of each video frame under the spatio-temporal slice segment is calculated; if the variance is less than a preset variance threshold, it is determined that the first video frame is the boundary frame of the fade-in / fade-out shot.

[0069] Accordingly, the detection of fade-in / fade-out shots can be realized.

[0070] Continuing to refer to Figure 4 , in the process of performing local feature matching on the first candidate frame, if the local feature matching between the first candidate frame and its previous frame is unsuccessful, the histograms of the first candidate frame and its previous frame are respectively extracted; if the histogram matching between the first candidate frame and its previous frame is successful, it is determined that the first candidate frame is the shot boundary frame.

[0071] Accordingly, in this embodiment, the boundary detection of various types of shots can be realized. Moreover, after the secondary judgment based on local features, further judgment can be performed based on the histogram to avoid missing the shot boundary frame, which can effectively expand the applicable scenarios of shot boundary detection and improve the recall rate and accuracy of shot boundary detection.

[0072] It should be noted that the execution subject of each step of the method provided in the foregoing embodiments may be the same device, or the method may also be executed by different devices as the execution subject. For example, the execution subject of steps 101 to 103 may be device A; for another example, the execution subject of steps 101 and 102 may be device A, and the execution subject of step 103 may be device B; and so on.

[0073] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first", "second", etc. in this document are used to distinguish different video frames, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are of different types.

[0074] Figure 5 The structural schematic diagram of a computing device provided by another exemplary embodiment of the present application. As Figure 5 shown, the computing device includes: a memory 50 and a processor 51.

[0075] The memory 50 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of these data include instructions, messages, pictures, videos, etc. for any application or method operating on the computing platform.

[0076] The memory 50 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0077] The processor 51 is coupled to the memory 50 and is used to execute the computer program in the memory 50 for:

[0078] Receiving a shot boundary detection request for a target video;

[0079] Extracting spatio-temporal slices from the target video;

[0080] Determining video frames suspected of being shot boundaries in the target video according to the spatio-temporal slices as candidate frames;

[0081] Selecting target frames that meet the preset local feature matching requirements from the candidate frames as the shot boundary frames in the target video.

[0082] In an optional embodiment, when the processor 51 determines video frames suspected of being shot boundaries in the target video according to the spatio-temporal slices, it is used for:

[0083] On the spatio-temporal slice, if the slice feature corresponding to the first video frame meets the preset continuity requirement, determine the first video frame as the video frame suspected of being a shot boundary;

[0084] Wherein, the first video frame is any one of at least one video frame included in the target video.

[0085] In an optional embodiment, the processor 51 is further configured to:

[0086] Obtain the first video frame and its corresponding N previous frames and M subsequent frames to form a first frame image sequence, where N and M are positive integers;

[0087] Calculate the difference degree between the slice feature corresponding to the first video frame and the previous adjacent frame, and the difference degree between the slice features corresponding to other adjacent frames in the first frame image sequence;

[0088] Perform mutation detection on the difference degree under the first frame image sequence;

[0089] If a difference degree mutation is detected on the first video frame, determine that the slice feature corresponding to the first video frame meets the preset continuity requirement.

[0090] In an optional embodiment, when performing mutation detection on the difference degree under the first frame image sequence, the processor 51 is configured to:

[0091] Construct a difference degree change curve according to the difference degree under the first frame image sequence;

[0092] If the difference degree corresponding to the first video frame is located in the peak region of the difference degree change curve, determine that a difference degree mutation is detected on the first video frame.

[0093] In an optional embodiment, the processor 51 is further configured to:

[0094] If the K consecutive video frames before the first video frame are all non-shot boundary frames, calculate the difference degree between the first video frame and the previous shot boundary frame, where K is a positive integer;

[0095] If the difference degree between the first video frame and the previous shot boundary frame is greater than the preset difference degree threshold, determine that the first video frame is the boundary frame of a dissolve / wipe shot.

[0096] In an optional embodiment, the processor 51 is further configured to:

[0097] Determine the spatio-temporal slice segment corresponding to the first frame image sequence;

[0098] Calculate the variance between the slice pixels of each video frame under the spatio-temporal slice segment;

[0099] If the variance is less than a preset variance threshold, determine the first video frame as the boundary frame of the fade-in / fade-out shot.

[0100] In an alternative embodiment, when the processor 51 selects a target frame that meets the preset local feature matching requirements from the candidate frames, it is used for:

[0101] In the target video, determine the frame preceding the first candidate frame;

[0102] Extract the local features in the first candidate frame and its preceding frame respectively;

[0103] If the local features between the first candidate frame and its preceding frame match, determine the first candidate frame as the target frame;

[0104] Wherein, the first candidate frame is any one of the candidate frames.

[0105] In an alternative embodiment, the processor 51 is further used for:

[0106] Adopt a feature matching scheme based on grid motion statistics to determine whether the local features between the first candidate frame and its preceding frame match.

[0107] In an alternative embodiment, the processor 51 is further used for:

[0108] If the local feature matching between the first candidate frame and its preceding frame is unsuccessful, extract the histograms of the first candidate frame and its preceding frame respectively;

[0109] If the histograms between the first candidate frame and its preceding frame match, determine the first candidate frame as the target frame.

[0110] In an alternative embodiment, the processor 51 is further used for:

[0111] Configure annotation information for the shot boundary frame to prompt the user of the shot boundary frame.

[0112] Furthermore, as Figure 5 shown, the computing device further includes: a communication component 52, a power supply component 53 and other components. Figure 5 Only some components are schematically shown in Figure 5 and it does not mean that the computing device only includes

[0113] It should be noted that for the technical details in the above embodiments of the computing device, reference can be made to the relevant descriptions in the foregoing method embodiments. For the sake of brevity, they will not be repeated here, but this should not cause loss of the protection scope of this application.

[0114] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed, it can implement each step executable by a computing device in the above method embodiment.

[0115] The above Figure 5 The communication component therein is configured to facilitate communication between the device where the communication component is located and other devices in a wired or wireless manner. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on technologies such as Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0116] The above Figure 5 The power supply component therein provides power for various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device where the power supply component is located.

[0117] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0118] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0119] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction means that implements the function specified in one or more of the processes Figure 1 or boxes Figure 1 specified in one or more of the processes and / or boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing steps for implementing the function specified in one or more of the processes Figure 1 or boxes Figure 1 specified in one or more of the boxes and / or processes.

[0121] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0122] The memory may include non-permanent memory in the computer-readable medium, random access memory (RAM) and / or non-volatile memory such as read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

[0123] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0124] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising said element.

[0125] The above are only examples of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A method for detecting shot boundaries, characterized in that, it includes: Receiving a shot boundary detection request for a target video; Extracting spatio-temporal slices from the target video; Based on the spatio-temporal slices, determining video frames suspected of shot boundaries in the target video as candidate frames; In the target video, determining the previous frame of a first candidate frame, where the first candidate frame is any one of the candidate frames; Respectively extracting local features in the first candidate frame and its previous frame; If the local features between the first candidate frame and its previous frame match, determining the first candidate frame as a shot boundary frame in the target video.

2. The method according to claim 1, characterized in that, The determining, based on the spatio-temporal slices, of video frames suspected of shot boundaries in the target video includes: On the spatio-temporal slices, if there is a slice feature corresponding to a first video frame that meets a preset continuity requirement, determining the first video frame as a video frame suspected of a shot boundary; wherein, the first video frame is any one of at least one video frame included in the target video.

3. The method according to claim 2, characterized in that, it further includes: Obtaining the first video frame and its corresponding N previous frames and M subsequent frames to form a first frame image sequence, where N and M are positive integers; Calculating the difference degree between the slice feature corresponding to the first video frame and its adjacent previous frame and the difference degrees between the slice features corresponding to other adjacent frames in the first frame image sequence; Performing mutation detection on the difference degrees under the first frame image sequence; If a difference degree mutation is detected on the first video frame, determining that the slice feature corresponding to the first video frame meets the preset continuity requirement.

4. The method according to claim 3, characterized in that, The performing of mutation detection on the difference degrees under the first frame image sequence includes: Constructing a difference degree change curve based on the difference degrees under the first frame image sequence; If the difference degree corresponding to the first video frame is located in the peak region of the difference degree change curve, determining that a difference degree mutation is detected on the first video frame.

5. The method according to claim 3, characterized in that, it further includes: If K consecutive video frames before the first video frame are all non-shot boundary frames, calculating the difference degree between the first video frame and the previous shot boundary frame, where K is a positive integer; If the difference degree between the first video frame and the previous shot boundary frame is greater than a preset difference degree threshold, determining the first video frame as a boundary frame of a dissolve / wipe shot.

6. The method according to claim 3, characterized in that, it further includes: Determining the spatio-temporal slice segment corresponding to the first frame image sequence; Calculating the variance between the slice pixels of each video frame under the spatio-temporal slice segment; If the variance is less than a preset variance threshold, determining the first video frame as a boundary frame of a fade-in / fade-out shot.

7. The method according to claim 1, characterized in that, it further includes: Adopting a feature matching scheme based on grid motion statistics to determine whether the local features between the first candidate frame and its previous frame match.

8. The method according to claim 1, wherein, it further includes: if the local feature matching between the first candidate frame and its previous frame is unsuccessful, then extract the histograms of the first candidate frame and its previous frame respectively; if the histogram matching between the first candidate frame and its previous frame is successful, then determine the first candidate frame as the shot boundary frame.

9. The method according to claim 1, wherein, the local feature includes the Fast Oriented Rotated BRIEF (ORB) feature, the Speeded Up Robust Features (SURF) feature or the Scale-Invariant Feature Transform (SIFT) feature.

10. The method according to claim 1, wherein, it further includes: configuring annotation information for the shot boundary frame to prompt the user of the shot boundary frame.

11. A computing device, wherein, it includes a memory and a processor; the memory is used to store one or more computer instructions; the processor is coupled with the memory and is used to execute the one or more computer instructions for: receiving a shot boundary detection request for a target video; extracting spatio-temporal slices from the target video; determining, according to the spatio-temporal slices, video frames suspected of being shot boundaries in the target video as candidate frames; determining, in the target video, the previous frame of a first candidate frame, where the first candidate frame is any one of the candidate frames; extracting local features in the first candidate frame and its previous frame respectively; if the local feature matching between the first candidate frame and its previous frame is successful, then determine the first candidate frame as the shot boundary frame in the target video.

12. The device according to claim 11, wherein, when determining, according to the spatio-temporal slices, video frames suspected of being shot boundaries in the target video, the processor is used for: on the spatio-temporal slices, if there is a slice feature corresponding to a first video frame that meets a preset continuity requirement, then determine the first video frame as the video frame suspected of being a shot boundary; wherein, the first video frame is any one of at least one video frame included in the target video.

13. The device according to claim 12, wherein, the processor is further used for: acquiring the first video frame and its corresponding N previous frames and M subsequent frames to form a first frame image sequence, where N and M are positive integers; calculating the difference degree between the slice feature corresponding to the first video frame and the slice feature corresponding to its adjacent previous frame and the difference degrees between the slice features corresponding to other adjacent frames in the first frame image sequence; performing mutation detection on the difference degrees under the first frame image sequence; if a difference degree mutation is detected on the first video frame, then determine that the slice feature corresponding to the first video frame meets the preset continuity requirement.

14. The device according to claim 13, wherein, when performing mutation detection on the difference degrees under the first frame image sequence, the processor is used for: constructing a difference degree change curve according to the difference degrees under the first frame image sequence; if the difference degree corresponding to the first video frame is located in the peak region of the difference degree change curve, then determine that a difference degree mutation is detected on the first video frame.

15. The device according to claim 13, wherein, the processor is further configured to: if K consecutive video frames before the first video frame are all non-shot boundary frames, calculate the difference degree between the first video frame and the previous shot boundary frame, where K is a positive integer; if the difference degree between the first video frame and the previous shot boundary frame is greater than a preset difference degree threshold, determine that the first video frame is a boundary frame of a dissolve / wipe shot.

16. The device according to claim 13, wherein, the processor is further configured to: determine the spatio-temporal slice segment corresponding to the first frame image sequence; calculate the variance between the slice pixels of each video frame under the spatio-temporal slice segment; if the variance is less than a preset variance threshold, determine that the first video frame is a boundary frame of a fade-in / fade-out shot.

17. The device according to claim 11, wherein, the processor is further configured to: adopt a feature matching scheme based on grid motion statistics to determine whether the local features between the first candidate frame and its previous frame match.

18. The device according to claim 11, wherein, the processor is further configured to: if the local features between the first candidate frame and its previous frame do not match successfully, respectively extract the histograms of the first candidate frame and its previous frame; if the histograms of the first candidate frame and its previous frame match, determine that the first candidate frame is a shot boundary frame.

19. A computer-readable storage medium storing computer instructions, wherein, when the computer instructions are executed by one or more processors, the one or more processors are caused to execute the shot boundary detection method according to any one of claims 1-10.

Images