Scene switching detection method and apparatus, computer-readable storage medium, terminal

Abstract

The application discloses a scene switching detection method and device, a computer readable storage medium and a terminal. The method comprises the following steps: extracting multiple video frames from a video to be detected; extracting a foreground target image block and a background image block from each video frame; calculating a foreground similarity score between the foreground target image block of a current frame and the foreground target image block of at least part of previous video frames, and calculating a background similarity score between the background image block of the current frame and the background image block of at least part of the previous video frames; obtaining a similarity fusion score based on the foreground similarity score and the background similarity score; comparing the similarity fusion score with a preset score; and determining that the video frame is after scene switching until the similarity fusion score is smaller than the preset score. The application can reduce the operation complexity and improve the detection efficiency.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a scene switching detection method and apparatus, a computer-readable storage medium, and a terminal. Background Technology

[0002] In existing technologies, scene transition detection technology is used to detect frames in a video segment where scene changes occur and to determine the position of those frames. This is an important task in computer vision.

[0003] However, in existing traditional scene transition detection methods, scene transition detection can be performed based on grayscale images, edge contours, or motion. However, these methods require manually setting features, are limited by user experience, and fail to meet performance standards when the data volume is large.

[0004] Another existing method for scene transition detection is based on deep learning. However, this method requires a specific dataset and has low computational efficiency.

[0005] There is an urgent need for a scene switching detection method that can reduce computational complexity and improve detection efficiency. Summary of the Invention

[0006] The technical problem solved by this invention is to provide a scene switching detection method and device, a computer-readable storage medium, and a terminal, which reduces computational complexity and improves detection efficiency.

[0007] To address the aforementioned technical problems, this application provides the following technical solutions:

[0008] A first aspect provides a scene transition detection method, the method comprising: extracting multiple video frames from a video to be detected; extracting foreground target image blocks and background image blocks from each video frame, wherein the foreground target image blocks are used to indicate the area occupied by one or more foreground targets, and the background image blocks are the remaining areas after removing each foreground target image block from the video frame; calculating a foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames, and calculating a background similarity score between the background image block of the current frame and the background image blocks of at least a portion of the previous video frames; obtaining a similarity fusion score based on the foreground similarity score and the background similarity score; comparing the similarity fusion score with a preset score; if the similarity fusion score is greater than or equal to the preset score, then comparing the similarity fusion score of the next video frame with the preset score, until the similarity fusion score is less than the preset score, and determining that it is a video frame after a scene transition.

[0009] Optionally, calculating the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames frame by frame includes: selecting at least a portion of the video frames located before the current frame, denoted as comparison frames; selecting the foreground target image patch of the current frame one by one, and determining whether each foreground target image patch has a valid mapped image patch in each comparison frame; if so, determining the image patch similarity between each valid mapped image patch and the foreground target image patch; and determining the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patch of at least a portion of the previous video frames based on the image patch similarity of each valid mapped image patch.

[0010] Optionally, selecting at least a portion of video frames preceding the current frame includes selecting some or all video frames from the video frames following the previous scene change.

[0011] Optionally, determining whether each foreground target image block has a valid mapped image block in each comparison frame includes: determining the mapped position of the center position of the foreground target image block in the current frame in each comparison frame; and determining foreground target image blocks in each comparison frame whose distance from the mapped position is less than a preset distance as valid mapped image blocks in that comparison frame.

[0012] Optionally, the preset distance is determined based on the time difference between the comparison frame and the current frame and a preset distance threshold; wherein, the preset distance threshold is the distance difference obtained by moving the foreground target within a single frame duration using a first speed, and the first speed is less than a preset moving speed.

[0013] Optionally, the preset distance threshold can be determined using the following formula:

[0014] R = (nq) × TH F

[0015] Where R represents the preset distance threshold, and n represents the current frame t. n The frame number q is used to represent the comparison frame t. q Frame number, TH F Used to indicate a preset distance threshold.

[0016] Optionally, determining the image patch similarity between each valid mapped image patch and the foreground target image patch includes: extracting feature vectors of the valid mapped image patch and the foreground target image patch respectively; determining the product of the feature vectors of the valid mapped image patch and the foreground target image patch, and the product of the moduli of the feature vectors of the valid mapped image patch and the foreground target image patch; and determining the image patch similarity between each valid mapped image patch and the foreground target image patch based on the quotient between the product and the moduli.

[0017] Optionally, the image patch similarity between each valid mapped image patch and the foreground target image patch can be determined using the following formula:

[0018]

[0019] Among them, S intra Used to represent image patch similarity Used to represent the current frame t n The i-th foreground target image patch f i eigenvectors, Used to represent the t-th q Frame comparison: the k-th valid mapped image block f in the frame k eigenvectors, Image block f used to represent foreground target i With effective mapped image block f k The product of the moduli of the eigenvectors, L, is used to represent the product of the moduli of the t-th eigenvectors. q The number of valid mapped image blocks in a frame is compared, where 1 ≤ k ≤ L.

[0020] Optionally, determining the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames, based on the image block similarity of each valid mapped image block, includes: summing and averaging the image block similarities of each valid mapped image block in each comparison frame to obtain the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames.

[0021] Optionally, the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames can be determined using the following formula:

[0022]

[0023] Among them, S F Used to represent foreground similarity scores, Used to represent image patch similarity Used to represent the current frame t n The i-th foreground target image patch f i eigenvectors, Used to represent the t-th q Frame comparison: the k-th valid mapped image block f in the frame k The feature vector, Q, is used to represent the number of at least some of the previous video frames, and 1≤q≤Q<n-1, m is used to represent the current frame t. n Foreground target image patch f i The number of , and 1≤i≤m.

[0024] Optionally, calculating the background similarity score between the background image patch of the current frame and the background image patches of at least a portion of the previous video frames includes: selecting some or all of the video frames after the previous scene switch; calculating the difference in the average grayscale value of the background image patch between each selected first video frame and the previous video frame, from the selected first video frame to the current frame; and determining the average of the absolute values ​​of each difference as the background similarity score between the background image patch of the current frame and the background image patch of at least a portion of the previous video frames.

[0025] Optional, current frame t n The frame number is n, and the number of previous video frames is n-1. The background similarity score between the background image patch of the current frame and the background image patch of the previous at least part of the video frames is determined using the following formula:

[0026]

[0027] D(h)=|H(t h )-H(t h-1 )|

[0028] Among them, S B H(t) is used to represent the background similarity score. h ) is used to represent the t-th h The average grayscale value of a video frame, H(t) h-1 ) is used to represent the t-th h-1 The average grayscale value of a video frame, D(h), is used to represent the absolute value of the difference between the h-th average grayscale values, and 2≤h≤n.

[0029] Optionally, a weighted average calculation is used to obtain a similarity fusion score based on the foreground similarity score and the background similarity score; wherein, the first weight is less than the second weight.

[0030] Optionally, when extracting foreground target image blocks and background image blocks in each video frame, a foreground sequence and a background sequence are constructed; whenever a scene change is determined, the foreground target image blocks in the foreground sequence are cleared, and the background image blocks in the background sequence are cleared, and the foreground target image blocks of the video frame after the scene change are used as the first batch of foreground target image blocks in the reset foreground sequence, and the background image blocks of the video frame after the scene change are used as the first batch of background image blocks in the reset background sequence.

[0031] Optionally, extracting foreground target image blocks and background image blocks in each video frame includes: using a target detection network to extract foreground target image blocks in each video frame; removing the foreground target image blocks in each video frame, and using the remaining area as the background image block.

[0032] Secondly, a scene transition detection device is provided, comprising: a video frame extraction module for extracting multiple video frames from a video to be detected; an image block extraction module for extracting foreground target image blocks and background image blocks from each video frame, wherein the foreground target image block indicates the area occupied by one or more foreground targets, and the background image block is the remaining area after removing each foreground target image block from the video frame; a calculation module for calculating, frame by frame, the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames, and calculating the background similarity score between the background image block of the current frame and the background image blocks of at least a portion of the previous video frames; a fusion module for obtaining a similarity fusion score based on the foreground similarity score and the background similarity score; a comparison module for comparing the similarity fusion score with a preset score; and a scene transition judgment module for comparing the similarity fusion score of the next video frame with the preset score when the similarity fusion score is greater than or equal to the preset score, until the similarity fusion score is less than the preset score, and then judging it as a video frame after a scene transition.

[0033] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes any of the scene switching detection methods provided in the first aspect to be executed.

[0034] Fourthly, a terminal is provided, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes any of the scene switching detection methods provided in the first aspect when running the computer program.

[0035] Fifthly, a computer program product is provided, the computer program product including a computer program, which, when run on a computer, causes the computer to execute any of the scene switching detection methods provided in the first aspect.

[0036] In a sixth aspect, a chip is provided, on which a computer program is stored, which, when executed by the chip, causes any of the scene switching detection methods provided in the first aspect to be executed.

[0037] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects:

[0038] In this embodiment of the invention, foreground target image blocks and background image blocks are extracted from each video frame. Foreground similarity scores and background similarity scores are then calculated separately, fused, and the comparison between the fused similarity score and a preset score is used as the basis for scene transition detection. This scheme determines scene transitions by comparing calculated values, requiring less computation. Compared to manually setting features, it effectively improves detection accuracy. Compared to model training, it effectively reduces computational complexity and improves detection efficiency.

[0039] Furthermore, an object detection network is used to extract foreground target image blocks in each video frame. Compared to manually dividing image blocks, which results in fixed positions, the foreground target image blocks obtained by the object detection network extraction method in this embodiment of the invention can move with the foreground target, improving the flexibility of image segmentation.

[0040] Furthermore, a comparison frame is selected, and it is determined whether each foreground target image block has a valid mapped image block in each comparison frame. If it does, the image block similarity between each valid mapped image block and the foreground target image block is determined, and a foreground similarity score is determined based on the image block similarity. Compared with using conventional image correlation algorithms or image similarity algorithms to determine the similarity at the same position between different frames, the scheme of this embodiment can determine the valid mapped image block based on fully considering the mobility of the foreground target between different frames, thereby improving the accuracy of scene switching detection.

[0041] Furthermore, the center position of the foreground target image block in the current frame is determined as the mapping position in each comparison frame; foreground target image blocks in each comparison frame whose distance from the mapping position is less than a preset distance are determined as valid mapped image blocks in that comparison frame. When determining valid mapped image blocks, the mobility of the foreground target between different frames can be considered, and the search range of valid mapped image blocks can be appropriately expanded by setting a preset distance, while limiting the size of the expanded area, thereby improving the determination effectiveness.

[0042] Furthermore, feature vectors of the effective mapped image patch and the foreground target image patch are extracted respectively. The product of the feature vectors of the effective mapped image patch and the foreground target image patch, as well as the product of the moduli of the feature vectors of the effective mapped image patch and the foreground target image patch, are determined. Based on the quotient between the product and the moduli, the image patch similarity between each effective mapped image patch and the foreground target image patch is determined. Compared with using conventional image correlation algorithms or image similarity algorithms to determine the similarity of different image patches, the scheme of this embodiment of the invention calculates the product of feature vectors and the product of moduli, and uses the quotient of the two to determine the image patch similarity, which can further improve the accuracy of determining the image patch similarity.

[0043] Furthermore, the image patch similarity of each effective mapped image patch in each comparison frame is summed and averaged to obtain the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames. The summation and averaging operation can remove the influence of extreme values ​​and improve the accuracy of the calculated foreground similarity score.

[0044] Furthermore, from the selected first video frame to the current frame, the difference in the average grayscale value of the background image block between each video frame and the previous video frame is calculated frame by frame. The average value of the absolute values ​​of each difference is determined as the background similarity score between the background image block of the current frame and the background image blocks of at least a portion of the previous video frames. Compared with using conventional image correlation algorithms or image similarity algorithms to determine the similarity of different background image blocks, since the background area is mostly a static area, the difference in background information between adjacent frames is very small. The scheme of this embodiment can introduce the average grayscale value and its difference, reduce the amount of calculation, and improve the computational efficiency.

[0045] Furthermore, a weighted average calculation is used to obtain a similarity fusion score based on the foreground similarity score and the background similarity score, wherein the first weight is less than the second weight. Since foreground targets are prone to significant deformation, the solution of this embodiment of the invention sets the foreground similarity weight to a low level, thereby avoiding the influence of extreme values ​​that could erroneously increase the similarity fusion score and cause missed scene transition frames.

[0046] Furthermore, whenever a scene change is determined, the foreground target image blocks in the foreground sequence and the background image blocks in the background sequence are cleared. The foreground target image blocks of the video frame after the scene change are used as the first batch of foreground target image blocks in the reset foreground sequence, and the background image blocks of the video frame after the scene change are used as the first batch of background image blocks in the reset background sequence. By employing the technical solution of this embodiment of the invention, the influence of the previous scene can be removed after a scene change, thereby improving the accuracy of scene determination. Attached Figure Description

[0047] Figure 1 This is a flowchart of a scene switching detection method according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a video frame in an embodiment of the present invention where the scene has not been switched;

[0049] Figure 3 This is a schematic diagram of video frames before and after a scene switch in an embodiment of the present invention;

[0050] Figure 4 yes Figure 1A flowchart of a specific implementation of step S13;

[0051] Figure 5 This is a flowchart of another scene switching detection method in an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram of the structure of a scene switching detection device according to an embodiment of the present invention. Detailed Implementation

[0053] As mentioned earlier, existing scene transition detection methods suffer from problems such as high computational complexity or low detection efficiency.

[0054] The inventors of this invention discovered through research that in a prior art, scene switching detection can be performed based on grayscale images, edge contours, or motion. The above methods have advantages such as low computational complexity and good performance when the amount of data is small. However, they require manually setting features, and when the amount of data is large, the performance often fails to meet the standards due to insufficient coverage of manually set features.

[0055] The inventors of this invention also discovered through research that in another existing technology, images can be stitched together according to channels based on deep learning methods, or image pairs can be used as input to solve the scene transition detection problem using an encoder-decoder approach. This type of method does not require manual feature setting and has high accuracy, but it requires specific datasets to train and optimize the model in advance. Furthermore, the computational load of different fusion methods varies greatly, and there is a problem of low computational efficiency.

[0056] In this embodiment of the invention, foreground target image blocks and background image blocks are extracted from each video frame. Foreground similarity scores and background similarity scores are then calculated separately, fused, and the comparison between the fused similarity score and a preset score is used as the basis for scene transition detection. This scheme determines scene transitions by comparing calculated values, requiring less computation. Compared to manually setting features, it effectively improves detection accuracy. Compared to model training, it effectively reduces computational complexity and improves detection efficiency.

[0057] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] Reference Figure 1 , Figure 1 This is a flowchart of a scene transition detection method according to an embodiment of the present invention. The scene transition detection method may include steps S11 to S16:

[0059] Step S11: Extract multiple video frames from the video to be detected;

[0060] Step S12: Extract foreground target image blocks and background image blocks in each video frame. The foreground target image block is used to indicate the area occupied by one or more foreground targets, and the background image block is the remaining area after removing each foreground target image block from the video frame.

[0061] Step S13: Calculate the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patch of at least a portion of the previous video frames frame by frame, and calculate the background similarity score between the background image patch of the current frame and the background image patch of at least a portion of the previous video frames.

[0062] Step S14: Obtain the similarity fusion score based on the foreground similarity score and the background similarity score;

[0063] Step S15: Compare the similarity fusion score with a preset score;

[0064] Step S16: If the similarity fusion score is greater than or equal to the preset score, then the similarity fusion score of the next video frame is compared with the preset score until the similarity fusion score is less than the preset score, at which point it is determined to be a video frame after the scene switch.

[0065] It is understood that, in specific implementation, the method can be implemented using a software program that runs in a processor integrated within the chip or chip module.

[0066] In the specific implementation of step S11, the video can be converted into a video frame sequence, for example, by extracting frame by frame or by extracting at certain intervals.

[0067] The extraction interval can be kept constant, or multiple extraction intervals can be used for extraction.

[0068] In the specific implementation of step S12, foreground target image blocks and background image blocks can be extracted from each video frame.

[0069] The foreground target image block can be used to indicate the area occupied by one or more foreground targets, while the background image block can be the remaining area after removing each foreground target image block from the video frame.

[0070] Specifically, the bounding box coordinates of the foreground target can be extracted in each video frame. The lines connecting adjacent bounding box coordinates can form a certain shape, forming a foreground target image block.

[0071] The foreground target image block and the background image block can be determined in the following two ways:

[0072] Method 1, the steps of extracting foreground target image blocks and background image blocks in each video frame may include: using a target detection network to extract foreground target image blocks in each video frame; removing the foreground target image blocks in each video frame, and using the remaining area as the background image block.

[0073] In this embodiment of the invention, a target detection network is used to extract foreground target image blocks in each video frame. Compared with manually dividing image blocks, which results in fixed positions, the foreground target image blocks obtained by the target detection network extraction method in this embodiment of the invention can follow the movement of the foreground target, improving the flexibility of image block division.

[0074] Furthermore, the target detection network can be selected from: a backbone network (Mobilenet_v3) to build a detection network (YOLOv4).

[0075] Among them, Mobilenet_v3 can be a high-precision backbone network with low computational cost and few parameters, suitable for deep convolutional networks deployed on mobile devices, while YOLOv4 can be an object detection network that achieves an optimal balance between accuracy and speed.

[0076] In a non-limiting embodiment, the area occupied by one or more foreground targets can be of a preset shape, wherein the preset shape can be a rectangle, and the bounding coordinates can be, for example, (x1, y1, x2, y2), and then a rectangular foreground target image block can be cropped according to the coordinate position.

[0077] In this embodiment of the invention, by setting a preset shape and cropping the foreground target image block to obtain the preset shape, it is beneficial to compare the image blocks with similar shapes and sizes when comparing the similarity of the foreground target image blocks in the future, thereby improving the effectiveness of the comparison.

[0078] Method 2 uses conventional target object extraction methods. This is non-restrictive; for example, one could first outline the foreground target, then extract the region within that outline as the foreground target image block, while the background image block is the remaining region after removing each foreground target image block from the video frame.

[0079] Combined with reference Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of a video frame in an embodiment of the present invention where the scene has not been switched. Figure 3 This is a schematic diagram of video frames before and after a scene switch in an embodiment of the present invention.

[0080] like Figure 2As shown, solid lines, dashed lines, and dotted lines can be used to represent three different foreground target image blocks, and the remaining area after removing these three boxes is the background image block.

[0081] Depend on Figure 2 It can be seen that when the scene does not change (i.e. both are background 1), the foreground similarity between the foreground target image blocks of the two video frames is high, and the background similarity between the background image blocks of the two video frames is also high.

[0082] like Figure 3 As shown, a solid-line frame can be used to represent the foreground target image block, and the remaining area after removing the solid-line frame is the background image block.

[0083] Depend on Figure 3 It can be seen that when the scene changes (i.e., from background 1 to background 2), the foreground target often changes as well. The foreground similarity between the foreground target image blocks of the two video frames is low, and the background similarity between the background image blocks of the two video frames is also low.

[0084] Furthermore, when extracting foreground target image blocks and background image blocks in each video frame, a foreground sequence and a background sequence can be constructed. Whenever a scene change is determined, the foreground target image blocks in the foreground sequence and the background image blocks in the background sequence are cleared. The foreground target image blocks of the video frame after the scene change are used as the first batch of foreground target image blocks in the reset foreground sequence, and the background image blocks of the video frame after the scene change are used as the first batch of background image blocks in the reset background sequence.

[0085] In a non-restrictive example, the foreground sequence could, for example, be P. F express, The background sequence can be, for example, P. B express, Used to represent the t-th i Background image blocks of video frames.

[0086] Among them, t1 to t n It can represent time-domain information, t n This indicates that the current frame is the t-th frame. n Frame, F is used to represent the foreground target detected in this frame, P F ti Used to represent the t-th element in the current scene j A collection of foreground object image patches in a video frame. Since there may be multiple foreground objects in each frame, m is used to represent the number of foreground target image blocks in the frame. For cases where each foreground target has its own foreground target image block, m can also be used to represent the number of foreground targets.

[0087] Each P F and P B By jointly storing the content of a scene, and by clearing the foreground target image blocks in the foreground sequence and the background image blocks in the background sequence when a scene transition occurs, P can be optimized. F and P B Initialize to an empty set, that is, reset to empty.

[0088] In this embodiment of the invention, whenever a scene change is determined, the foreground target image blocks in the foreground sequence and the background image blocks in the background sequence are cleared. The foreground target image blocks of the video frame after the scene change are used as the first batch of foreground target image blocks in the reset foreground sequence, and the background image blocks of the video frame after the scene change are used as the first batch of background image blocks in the reset background sequence. By employing the technical solution of this embodiment of the invention, the influence of the previous scene can be removed after a scene change, thereby improving the accuracy of scene determination.

[0089] Continue to refer to Figure 1 In the specific implementation of step S13, the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames is calculated frame by frame, and the background similarity score between the background image block of the current frame and the background image blocks of at least a portion of the previous video frames is calculated.

[0090] This can be achieved by selecting at least a portion of the video frames from the previous scene transition, thus avoiding the influence of the scene before the transition. Then, within the selected video frames, foreground similarity scores and background similarity scores are calculated frame by frame.

[0091] Reference Figure 4 , Figure 4 yes Figure 1 A flowchart of a specific implementation of step S13 is provided. The step of calculating the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames may include steps S41 to S44, which are described below.

[0092] In step S41, at least a portion of video frames preceding the current frame are selected and denoted as comparison frames.

[0093] Furthermore, the step of selecting at least a portion of the video frames preceding the current frame may include: selecting some or all of the video frames after the previous scene change.

[0094] In practice, an appropriate number of video frames can be selected based on the video length and the estimated switching frequency. Furthermore, by selecting from the video frames after the previous scene switch, the influence of the scene before the switch can be avoided.

[0095] In step S42, foreground target image blocks of the current frame are selected one by one to determine whether each foreground target image block has a valid mapped image block in each comparison frame.

[0096] according to Figure 2 As can be seen, since the foreground target located in the solid frame is in a moving state, there is a possibility of misjudging it as a dashed frame or dotted frame in the next frame. It is necessary to determine the consistent foreground target in the two video frames by setting up a judgment step for valid mapped image blocks.

[0097] Continue to refer to Figure 4 In step S42, the step of determining whether each foreground target image block has a valid mapped image block in each comparison frame may include: determining the mapping position of the center position of the foreground target image block in the current frame in each comparison frame; and determining the foreground target image block in each comparison frame whose distance from the mapping position is less than a preset distance as a valid mapped image block in that comparison frame.

[0098] Specifically, the center position of the foreground target image block in the current frame and its mapping position in each comparison frame can be positions with the same coordinates. By selecting the foreground target image block that is closer to the mapping position, it is beneficial to determine the same foreground target.

[0099] Furthermore, the preset distance is determined based on the time difference between the comparison frame and the current frame and a preset distance threshold; wherein, the preset distance threshold is the distance difference obtained by moving the foreground target within a single frame duration using a first speed, and the first speed is less than a preset moving speed.

[0100] Specifically, by using a predetermined distance difference based on a faster first velocity as a basis to determine a preset distance threshold, the motion characteristics of the foreground target can be considered more fully, thereby determining the foreground target image block within a larger range of the comparison frames and improving the accuracy of determining the same foreground target.

[0101] Furthermore, the preset distance threshold can be determined using the following formula:

[0102] R = (nq) × TH F

[0103] Where R represents the preset distance threshold, and n represents the current frame t. n The frame number q is used to represent the comparison frame t. qFrame number, TH F Used to indicate a preset distance threshold.

[0104] In this embodiment of the invention, the center position of the foreground target image block in the current frame is determined as the mapping position in each comparison frame; foreground target image blocks in each comparison frame whose distance from the mapping position is less than a preset distance are determined as valid mapping image blocks in that comparison frame. When determining valid mapping image blocks, the mobility of the foreground target between different frames can be considered, and the search range of valid mapping image blocks can be appropriately expanded by setting a preset distance, while limiting the size of the expanded area, thereby improving the determination effectiveness.

[0105] In step S43, if present, the image patch similarity between each valid mapped image patch and the foreground target image patch is determined.

[0106] Specifically, image patch similarity can be determined in the following two ways:

[0107] Method 1, the step of determining the image patch similarity between each valid mapped image patch and the foreground target image patch may include: extracting feature vectors (also called feature maps) of the valid mapped image patch and the foreground target image patch respectively; determining the product of the feature vectors of the valid mapped image patch and the foreground target image patch, and the product of the modulus of the feature vectors of the valid mapped image patch and the foreground target image patch; and determining the image patch similarity between each valid mapped image patch and the foreground target image patch based on the quotient between the product and the modulus product.

[0108] In this invention, conventional methods can be used to extract the feature vectors of image blocks. For example, the feature vectors (feature maps) of image blocks composed of the coordinates of each border can be extracted. In this embodiment of the invention, there are no restrictions on the specific feature vector extraction.

[0109] In this embodiment of the invention, feature vectors of the effective mapped image block and the foreground target image block can be extracted respectively. The product of the feature vectors of the effective mapped image block and the foreground target image block, as well as the product of the modulus of the feature vectors of the effective mapped image block and the foreground target image block, are determined. Based on the quotient between the product and the modulus product, the image block similarity between each effective mapped image block and the foreground target image block is determined. Compared with using conventional image correlation algorithms or image similarity algorithms to determine the similarity of different image blocks, the scheme of this embodiment of the invention, which calculates the product of feature vectors and the product of modulus, and uses the quotient of the two to determine the image block similarity, can further improve the accuracy of determining the image block similarity.

[0110] Furthermore, the image patch similarity between each valid mapped image patch and the foreground target image patch can be determined using the following formula:

[0111]

[0112] Among them, S intra Used to represent image patch similarity Used to represent the current frame t n The i-th foreground target image patch f i eigenvectors, Used to represent the t-th q Frame comparison: the k-th valid mapped image block f in the frame k eigenvectors, Image block f used to represent foreground target i With effective mapped image block f k The product of the moduli of the eigenvectors, L, is used to represent the product of the moduli of the t-th eigenvectors. q The number of valid mapped image blocks in a frame is compared, where 1 ≤ k ≤ L. It can be used to represent cosine distance.

[0113] Alternatively, conventional image correlation or image similarity algorithms can be used to determine the similarity between the effectively mapped image patch and the foreground target image patch, such as the Structural Similarity (SSIM) similarity algorithm.

[0114] In step S44, based on the image patch similarity of each valid mapped image patch, the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames is determined.

[0115] The foreground similarity score can be determined in the following two ways:

[0116] Method 1, the step of determining the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames based on the image block similarity of each valid mapped image block, may include: summing and averaging the image block similarities of each valid mapped image block in each comparison frame to obtain the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames.

[0117] In this embodiment of the invention, the image block similarity of each valid mapped image block in each comparison frame is summed and averaged to obtain the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames. The summation and averaging operation can remove the influence of extreme values ​​and improve the accuracy of the calculated foreground similarity score.

[0118] Furthermore, the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames can be determined using the following formula:

[0119]

[0120] Among them, S F Used to represent foreground similarity scores, Used to represent image patch similarity Used to represent the current frame t n The i-th foreground target image patch f i eigenvectors, Used to represent the t-th q Frame comparison: the k-th valid mapped image block f in the frame k The feature vector, Q, is used to represent the number of at least some of the previous video frames, and 1≤q≤Q<n-1, m is used to represent the current frame t. n Foreground target image patch f i The number of , and 1≤i≤m.

[0121] Alternatively, other appropriate operations can be performed on the image patch similarity of each valid mapped image patch in each comparison frame, such as median value operation, to obtain the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames.

[0122] In this embodiment of the invention, a comparison frame is selected, and it is determined whether each foreground target image block has a valid mapped image block in each comparison frame. When it does, the image block similarity between each valid mapped image block and the foreground target image block is determined, and the foreground similarity score is determined based on the image block similarity. Compared with using conventional image correlation algorithms or image similarity algorithms to determine the similarity at the same position between different frames, the scheme of this embodiment of the invention can determine the valid mapped image block based on fully considering the mobility of the foreground target between different frames, thereby improving the accuracy of scene switching detection.

[0123] Continue to refer to Figure 1 In the specific implementation of step S13, it is also necessary to calculate the background similarity score between the background image block of the current frame and the background image blocks of at least some of the previous video frames.

[0124] The background similarity score can be determined in the following two ways:

[0125] Method 1, the step of calculating the background similarity score between the background image patch of the current frame and the background image patches of at least a portion of the previous video frames may include: selecting some or all video frames in the video frames after the previous scene switch; calculating the difference in the average grayscale value of the background image patch between each selected first video frame and the previous video frame from the current frame; and determining the average of the absolute values ​​of each difference as the background similarity score between the background image patch of the current frame and the background image patch of at least a portion of the previous video frames.

[0126] In this embodiment of the invention, from the selected first video frame to the current frame, the difference in the average grayscale value of the background image block between each video frame and the previous video frame is calculated frame by frame. The average value of the absolute values ​​of each difference is determined as the background similarity score between the background image block of the current frame and the background image blocks of at least a portion of the previous video frames. Compared with using conventional image correlation algorithms or image similarity algorithms to determine the similarity of different background image blocks, since the background area is mostly a static area, the difference in background information between adjacent frames is very small. By adopting the scheme of this embodiment of the invention, the average grayscale value and its difference can be introduced, reducing the amount of calculation and improving the computational efficiency.

[0127] Furthermore, the current frame t n The frame number is n, and the number of previous video frames is n-1. The background similarity score between the background image patch of the current frame and the background image patch of the previous at least part of the video frames can be determined using the following formula:

[0128]

[0129] D(h)=|H(t h )-H(t h-1 )|

[0130] Among them, S B H(t) is used to represent the background similarity score. h ) is used to represent the t-th h The average grayscale value of a video frame, H(t) h-1 ) is used to represent the t-th h-1 The average grayscale value of a video frame, D(h), is used to represent the absolute value of the difference between the h-th average grayscale values, and 2≤h≤n.

[0131] Alternatively, conventional image correlation or image similarity algorithms can be used to determine the similarity between the effectively mapped image patch and the foreground target image patch, such as the Structural Similarity (SSIM) similarity algorithm.

[0132] In the specific implementation of step S14, a similarity fusion score is obtained based on the foreground similarity score and the background similarity score.

[0133] The similarity fusion score can be determined in the following two ways:

[0134] Method 1 uses a weighted average calculation to obtain a similarity fusion score based on the foreground similarity score and the background similarity score; where the first weight is less than the second weight.

[0135] In this embodiment of the invention, a weighted average calculation is used to obtain a similarity fusion score based on the foreground similarity score and the background similarity score, wherein the first weight is less than the second weight. Since foreground targets are prone to significant deformation, the solution in this embodiment of the invention sets a lower foreground similarity weight, thereby avoiding the influence of extreme values ​​that could erroneously increase the similarity fusion score and lead to missed scene transition frames.

[0136] Furthermore, the similarity fusion score can be obtained based on the foreground similarity score and the background similarity score using the following formula:

[0137] S=αS F +βS B

[0138] Where S represents the similarity fusion score, α represents the first weight, and S F The value is used to represent the foreground similarity score, β is used to represent the second weight, and S... B Used to represent background similarity scores.

[0139] Alternatively, other appropriate calculation methods can be used, such as weighted average calculation, to obtain a similarity fusion score based on the foreground similarity score and the background similarity score.

[0140] In the specific implementation of step S15, the similarity fusion score is compared with a preset score.

[0141] The preset score can be pre-set based on experience values, or it can be adjusted in real time based on the judgment results of previous scenarios.

[0142] In the specific implementation of step S16, if the similarity fusion score is greater than or equal to the preset score, the similarity fusion score of the next video frame is compared with the preset score until the similarity fusion score is less than the preset score, at which point it is determined to be a video frame after scene switching.

[0143] Specifically, when the similarity fusion score is less than the preset score, it can be determined to be a video frame after a scene switch. It can also output indication information of the video in the video to be detected, such as outputting a sequence number, so as to record the scene switch situation.

[0144] In this embodiment of the invention, foreground target image blocks and background image blocks are extracted from each video frame. Foreground similarity scores and background similarity scores are then calculated separately, fused, and the comparison between the fused similarity score and a preset score is used as the basis for scene transition detection. This scheme determines scene transitions by comparing calculated values, requiring less computation. Compared to manually setting features, it effectively improves detection accuracy. Compared to model training, it effectively reduces computational complexity and improves detection efficiency.

[0145] Reference Figure 5 , Figure 5 This is a flowchart of another scene transition detection method in an embodiment of the present invention. The other scene transition detection method may include steps S501 to S509, which are described below.

[0146] In step S501, video frames can be extracted.

[0147] In step S502, foreground target image blocks can be extracted.

[0148] In step S503, background image blocks can be extracted.

[0149] In step S504, a foreground similarity score can be calculated between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames.

[0150] In step S505, a background similarity score can be calculated between the background image block of the current frame and the background image blocks of at least a portion of the previous video frames.

[0151] In step S506, the similarity fusion score can be obtained.

[0152] In step S507, it can be determined whether the similarity fusion score is less than a preset score. If the determination result is yes, then step S508 can be executed. Otherwise, if the determination result is no, then step S510 can be executed.

[0153] In step S508, it can be determined that it is a video frame after the scene switch.

[0154] In step S509, the sequence number of the video frame in the video to be detected can be output.

[0155] In step S510, the next frame after the current frame can be selected.

[0156] Specifically, the next video frame can be selected as the current frame for the next round of judgment.

[0157] For more detailed information regarding steps S501 to S510 in the specific implementation, please refer to the preceding text and... Figures 1 to 4 The description is as follows, and will not be repeated here.

[0158] Reference Figure 6 , Figure 6 This is a schematic diagram of a scene transition detection device according to an embodiment of the present invention. The scene transition detection device may include:

[0159] The video frame extraction module 61 is used to extract multiple video frames from the video to be detected.

[0160] The image block extraction module 62 is used to extract foreground target image blocks and background image blocks in each video frame. The foreground target image block is used to indicate the area occupied by one or more foreground targets, and the background image block is the remaining area after removing each foreground target image block in the video frame.

[0161] The calculation module 63 is used to calculate the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a portion of the previous video frames, and to calculate the background similarity score between the background image block of the current frame and the background image blocks of at least a portion of the previous video frames.

[0162] The fusion module 64 is used to obtain a similarity fusion score based on the foreground similarity score and the background similarity score;

[0163] Comparison module 65 is used to compare the similarity fusion score with a preset score;

[0164] The scene switching judgment module 66 is used to compare the similarity fusion score of the next video frame with the preset score when the similarity fusion score is greater than or equal to the preset score, until the similarity fusion score is less than the preset score, and then judge it as a video frame after scene switching.

[0165] In specific implementations, the aforementioned device may correspond to a chip with data processing function in a terminal; or to a chip module in a terminal that includes a chip with data processing function; or to a terminal itself.

[0166] For the principle, implementation and beneficial effects of the scene transition detection device, please refer to the previous description of the scene transition detection method, which will not be repeated here.

[0167] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, the aforementioned scene switching detection method is performed. The computer-readable storage medium may include read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. The computer-readable storage medium may also include non-volatile memory or non-transitory memory, etc.

[0168] Specifically, in this embodiment of the invention, the processor can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0169] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0170] This invention also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the above-described method. The terminal includes, but is not limited to, terminal devices such as mobile phones, computers, tablets, servers, and cloud platforms.

[0171] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0172] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0173] In the embodiments of this application, "multiple" refers to two or more.

[0174] The descriptions of "first," "second," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.

[0175] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A scene transition detection method, characterized in that, include: Extract multiple video frames from the video to be detected; Extract foreground target image blocks and background image blocks in each video frame. Foreground target image blocks are used to indicate the area occupied by one or more foreground targets, and background image blocks are the remaining areas after removing each foreground target image block in the video frame. Calculate the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames, and calculate the background similarity score between the background image patch of the current frame and the background image patches of at least a portion of the previous video frames, frame by frame. A similarity fusion score is obtained based on foreground similarity score and background similarity score; The similarity fusion score is compared with a preset score; If the similarity fusion score is greater than or equal to the preset score, the similarity fusion score of the next video frame is compared with the preset score until the similarity fusion score is less than the preset score, at which point it is determined to be a video frame after a scene switch. The calculation of the foreground similarity score between the foreground target image patch in the current frame and the foreground target image patches in at least a portion of the previous video frames includes: Select at least a subset of video frames preceding the current frame, and denote them as comparison frames; Select foreground target image blocks one by one in the current frame, and determine whether each foreground target image block has a valid mapped image block in each comparison frame; If it exists, determine the image patch similarity between each valid mapped image patch and the foreground target image patch; Based on the image patch similarity of each valid mapped image patch, determine the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames; Calculating the background similarity score between the background image patch of the current frame and the background image patches of at least a portion of the previous video frames includes: Select some or all of the video frames from the video frames following the previous scene change; From the first selected video frame to the current frame, calculate the difference in the average grayscale value of the background image block between each video frame and the previous video frame. The average of the absolute values ​​of each difference is determined as the background similarity score between the background image patch of the current frame and the background image patches of at least a portion of the previous video frames.

2. The scene switching detection method according to claim 1, characterized in that, Selecting at least a subset of video frames preceding the current frame includes: Select some or all of the video frames from the video frames after the previous scene change.

3. The scene switching detection method according to claim 1, characterized in that, Determining whether each foreground target image patch has a valid mapped image patch in each comparison frame includes: Determine the center position of the foreground target image patch in the current frame and its mapped position in each comparison frame; In each comparison frame, foreground target image blocks whose distance to the mapping position is less than a preset distance are identified as valid mapped image blocks in that comparison frame.

4. The scene switching detection method according to claim 3, characterized in that, The preset distance is determined based on the time difference between the comparison frame and the current frame, as well as a preset distance threshold. The preset distance threshold is the distance difference obtained by moving the foreground target within a single frame using a first speed, where the first speed is less than the preset moving speed.

5. The scene switching detection method according to claim 4, characterized in that, The preset distance threshold is determined using the following formula: Where R represents the preset distance threshold, and n represents the current frame t. n The frame number q is used to represent the comparison frame t. q Frame number, TH F Used to indicate a preset distance threshold.

6. The scene switching detection method according to claim 1, characterized in that, Determining the image patch similarity between each valid mapped image patch and the foreground target image patch includes: Extract the feature vectors of the effective mapped image patch and the foreground target image patch respectively; Determine the product of the feature vectors of the effective mapped image block and the foreground target image block, and the product of the moduli of the feature vectors of the effective mapped image block and the foreground target image block; The image patch similarity between each valid mapped image patch and the foreground target image patch is determined based on the quotient between the product and the product of the modulus.

7. The scene switching detection method according to claim 6, characterized in that, The image patch similarity between each valid mapped image patch and the foreground target image patch is determined using the following formula: Among them, S intra Used to represent image patch similarity Used to represent the current frame t n The i-th foreground target image patch f i eigenvectors, Used to represent the t-th q Frame comparison: the k-th valid mapped image block f in the frame k eigenvectors, Image block f used to represent foreground target i With effective mapped image block f k The product of the moduli of the eigenvectors, L, is used to represent the product of the moduli of the t-th eigenvectors. q The number of valid mapped image blocks in a frame is compared, where 1 ≤ k ≤ L.

8. The scene switching detection method according to claim 1, characterized in that, Based on the image patch similarity of each valid mapped image patch, the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a subset of previous video frames is determined, including: The image patch similarity of each valid mapped image patch in each comparison frame is summed and averaged to obtain the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patches of at least a portion of the previous video frames.

9. The scene switching detection method according to claim 8, characterized in that, The foreground similarity score between a foreground target image patch in the current frame and foreground target image patches in at least a subset of previous video frames is determined using the following formula: Among them, S F Used to represent foreground similarity scores, Used to represent image patch similarity Used to represent the current frame t n The i-th foreground target image patch f i eigenvectors, Used to represent the t-th q Frame comparison: the k-th valid mapped image block f in the frame k The feature vector, Q, is used to represent the number of at least some of the previous video frames, and 1≤q≤Q<n-1, m is used to represent the current frame t. n Foreground target image patch f i The number of , and 1≤i≤m.

10. The scene switching detection method according to claim 1, characterized in that, Current frame t n The frame number is n, and the number of the preceding video frames is n-1; The background similarity score between the background image patch of the current frame and the background image patches of at least a portion of the previous video frames is determined using the following formula: Among them, S B Used to represent background similarity scores, Used to represent the t-th h The average grayscale value of a video frame. Used to represent the t-th h-1 The average grayscale value of a video frame, D(h), is used to represent the absolute value of the difference between the h-th average grayscale values, and 2≤h≤n.

11. The scene switching detection method according to claim 1, characterized in that, A weighted average calculation is used to obtain a similarity fusion score based on the foreground similarity score and the background similarity score; The first weight is less than the second weight.

12. The scene switching detection method according to claim 1, characterized in that, When extracting foreground target image blocks and background image blocks in each video frame, construct foreground sequence and background sequence; Whenever a scene change is determined, the foreground target image blocks in the foreground sequence are cleared, and the background image blocks in the background sequence are cleared. The foreground target image blocks of the video frame after the scene change are used as the first batch of foreground target image blocks in the reset foreground sequence, and the background image blocks of the video frame after the scene change are used as the first batch of background image blocks in the reset background sequence.

13. The scene switching detection method according to claim 1, characterized in that, Extracting foreground target image patches and background image patches from each video frame includes: An object detection network is used to extract foreground object image blocks in each video frame; The foreground target image block is removed from each video frame, and the remaining area is used as the background image block.

14. A scene switching detection device, characterized in that, include: The video frame extraction module is used to extract multiple video frames from the video to be detected. The image block extraction module is used to extract foreground target image blocks and background image blocks in each video frame. The foreground target image block is used to indicate the area occupied by one or more foreground targets, and the background image block is the remaining area after removing each foreground target image block in the video frame. The calculation module is used to calculate the foreground similarity score between the foreground target image patch of the current frame and the foreground target image patch of at least a portion of the previous video frames, and to calculate the background similarity score between the background image patch of the current frame and the background image patch of at least a portion of the previous video frames. The fusion module is used to obtain a similarity fusion score based on the foreground similarity score and the background similarity score; The comparison module is used to compare the similarity fusion score with a preset score; The scene switching judgment module is used to compare the similarity fusion score of the next video frame with the preset score when the similarity fusion score is greater than or equal to the preset score, until the similarity fusion score is less than the preset score, and then judge it as a video frame after scene switching. The calculation module executes the following steps: Select at least a subset of video frames preceding the current frame, denoted as comparison frames; sequentially select foreground target image blocks in the current frame, and determine whether each foreground target image block has a valid mapped image block in each comparison frame; if it does, determine the image block similarity between each valid mapped image block and the foreground target image block; based on the image block similarity of each valid mapped image block, determine the foreground similarity score between the foreground target image block of the current frame and the foreground target image blocks of at least a subset of the previous video frames; In the video frames after the previous scene switch, select some or all of the video frames; from the first selected video frame to the current frame, calculate the difference in the average grayscale value of the background image block between each video frame and the previous video frame; determine the average of the absolute values ​​of each difference as the background similarity score between the background image block of the current frame and the background image blocks of at least some of the previous video frames.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of the scene switching detection method according to any one of claims 1 to 13.

16. A terminal comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the scene switching detection method according to any one of claims 1 to 13.

Images