Video splitting method and device, terminal and storage medium

Abstract

The invention relates to a video splitting method and device, a terminal and a storage medium, and relates to the technical field of video processing, and the method comprises the steps: obtaining an image frame from a to-be-processed video; under the condition that the two image frames are similar, index coordinate points are constructed according to the two indexes corresponding to the two adjacent image frames; classifying the index coordinate points according to whether the index coordinate points are continuous on the coordinate axis, and generating an initial continuous line segment; combining the two initial continuous line segments of which the coordinate difference value of the index coordinate points is smaller than or equal to a first threshold value to generate a target continuous line segment; and obtaining an index of an end point of the target continuous line segment to obtain a target index, taking a timestamp of an image frame corresponding to the target index as a segmentation time point of the to-be-processed video, and splitting the to-be-processed video according to the segmentation time point to obtain a sub-lens segment corresponding to the to-be-processed video. Through the method and the device, the technical problems of low video shot recognition accuracy and influence on the splitting effect in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video splitting method, device, terminal and storage medium. Background Art

[0002] Video shot recognition technology uses image processing and pattern recognition technology to detect shot boundaries and transitions in videos, and achieves structured analysis of video content. The original technology relied on image processing methods such as edge detection and histogram analysis to detect shot boundaries; with the introduction of machine learning algorithms, especially the rapid development of deep learning technology, the accuracy and efficiency of video shot recognition have been significantly improved; in recent years, deep learning models such as convolutional neural networks (CNNs) have also been applied to detect shot boundaries. These methods mainly split videos by detecting the similarity between the previous and next image frames, while factors such as the viewing angle and distance of the object, the rapid movement of the object, and the noise and artifacts of the image will affect the recognition accuracy of the image frame, and thus affect the accuracy of video segmentation. Summary of the invention

[0003] The embodiments of the present application provide a video splitting method, device, terminal and storage medium to solve the technical problem in the prior art that the accuracy of video shot recognition is low and the splitting effect is affected.

[0004] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions: In a first aspect, an embodiment of the present application provides a video splitting method, the method comprising: Obtaining an image frame from the video to be processed, and assigning an index to the image frame, wherein the index is used to identify the image frame, and each of the indexes is arranged in sequence according to the timestamp of the image frame to obtain a set of continuous natural numbers; Obtaining a comparison result of each two adjacent image frames, and when the comparison result indicates that the two adjacent image frames are similar, constructing an index coordinate point according to two indexes corresponding to the two adjacent image frames; Classifying the index coordinate points according to whether the index coordinate points are continuous on the coordinate axis to generate initial continuous line segments, wherein each of the initial continuous line segments is composed of a plurality of index coordinate points that are continuous on the coordinate axis; Merge the two initial continuous line segments whose coordinate difference of the index coordinate point is less than or equal to the first threshold to generate a target continuous line segment; Obtain the index of the endpoint of the target continuous line segment to obtain the target index, use the timestamp of the image frame corresponding to the target index as the segmentation time point of the video to be processed, and split the video to be processed at the segmentation time point to obtain the storyboard fragments corresponding to the video to be processed.

[0005] In combination with the first aspect, in a possible design, the index coordinate points are classified according to whether the index coordinate points are continuous on the coordinate axis to generate initial continuous line segments, including: Sorting the index coordinate points according to their order on any coordinate axis, and sequentially performing continuity detection on each of the sorted index coordinate points, wherein the continuity detection is used to determine whether the index coordinate point and its adjacent index coordinate points are on the same straight line; If they are on the same straight line, the index coordinate point is merged into the adjacent index coordinate point to generate an initial continuous line segment; If they are not on the same straight line, the index coordinate point is used as the starting point of a new initial continuous line segment.

[0006] In combination with the first aspect, in a possible design manner, merging the two initial continuous line segments whose coordinate difference of the index coordinate point is less than or equal to the first threshold to generate a target continuous line segment includes: If the values ​​of two adjacent index coordinate points on any coordinate axis are equal and the coordinate difference on the other coordinate axis is less than or equal to the first threshold, the two adjacent index coordinate points are connected to obtain a target continuous line segment formed by merging the two initial continuous line segments.

[0007] In combination with the first aspect, in a possible design manner, after generating the initial continuous line segments, the method further includes: Initial continuous line segments whose line segment length is less than a second threshold are eliminated.

[0008] In combination with the first aspect, in a possible design manner, the line segment length is represented by the number of index coordinate points on the initial continuous line segment, and the initial continuous line segment whose line segment length is less than a second threshold is eliminated, including: Initial continuous line segments whose number of index coordinate points is less than a second threshold are eliminated.

[0009] In combination with the first aspect, in a possible design, obtaining the comparison result of each two adjacent image frames includes: Extracting pixel content of the image frame to obtain pixel feature information of the image frame; Obtain the similarity distance between the pixel feature information of each two adjacent image frames; The similarity distance is compared with a third threshold to generate a comparison result of the two adjacent image frames.

[0010] In combination with the first aspect, in a possible design manner, obtaining an image frame from the video to be processed includes: According to a preset time interval, image frames are extracted from the video to be processed; or according to a preset sampling number, image frames are extracted from the video to be processed.

[0011] In a second aspect, an embodiment of the present application provides a video splitting device, including: An image acquisition module, used to acquire image frames from the video to be processed, and assign indexes to the image frames, wherein the indexes are used to identify the image frames, and the indexes are arranged in sequence according to the timestamps of the image frames to obtain a set of continuous natural numbers; A similarity comparison module, used to obtain a comparison result of each two adjacent image frames, and when the comparison result indicates that the two adjacent image frames are similar, construct an index coordinate point according to two indexes corresponding to the two adjacent image frames; A continuity detection module, used to classify the index coordinate points according to whether the index coordinate points are continuous on the coordinate axis, and generate initial continuous line segments, wherein each of the initial continuous line segments is composed of a plurality of index coordinate points that are continuous on the coordinate axis; A line segment merging module, used for merging two of the initial continuous line segments whose coordinate difference of the index coordinate point is less than or equal to a first threshold value to generate a target continuous line segment; A video splitting module is used to obtain the index of the endpoint of the target continuous line segment to obtain the target index, use the timestamp of the image frame corresponding to the target index as the segmentation time point of the video to be processed, and split the video to be processed at the segmentation time point to obtain the storyboard fragments corresponding to the video to be processed.

[0012] In a third aspect, an embodiment of the present application provides a terminal, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method of the first aspect and possible design methods thereof.

[0013] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method of the first aspect and its possible design method when running, and the display screen is configured to display the second video and audio processed by the processor.

[0014] Compared with the prior art, a video splitting method, device, terminal and storage medium provided in an embodiment of the present application obtain image frames from a video to be processed and assign indexes to the image frames, wherein the indexes are used to identify the image frames, and the indexes are arranged in sequence according to the timestamps of the image frames to obtain a group of continuous natural numbers; the comparison results of each two adjacent image frames are obtained, and when the comparison results indicate that the two adjacent image frames are similar, index coordinate points are constructed according to the two indexes corresponding to the two adjacent image frames; the index coordinate points are classified according to whether they are continuous on the coordinate axis, and initial continuous line segments are generated, wherein each initial continuous line segment is composed of a plurality of index coordinate points that are continuous on the coordinate axis; two continuous line segments whose coordinate difference of the index coordinate points is less than a first threshold are merged to generate a target continuous line segment; the indexes of the endpoints of the target continuous line segment are obtained to obtain the target index, and the timestamps of the image frames corresponding to the target indexes are used as the segmentation time points of the video to be processed, and the video to be processed is segmented at the segmentation time points to obtain storyboard fragments corresponding to the video to be tested. In the embodiment of the present application, when two adjacent image frames are identified as being similar, index coordinate points corresponding to the indexes of the two adjacent image frames are constructed, and each index coordinate point is classified in an iterative manner, further subdividing the lens segmentation points. Each initial continuous line segment after subdivision represents a segment of the lens, and two similar segments of the lens are merged into one segment by merging two adjacent initial continuous line segments, so that the lens recognition result can still be corrected in the scene where the image frame recognition accuracy is poor, thereby solving the technical problem of low accuracy in video lens recognition and affecting the splitting effect, so that each lens segmentation point identified is more accurate, and the effect of splitting the video based on the lens segmentation point is better.

[0015] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A hardware structure block diagram of a terminal provided in an embodiment of the present application is shown; Figure 2 A flowchart of a video splitting method provided by an embodiment of the present application is shown; Figure 3 A schematic diagram showing a line segment merging result provided in an embodiment of the present application is shown; Figure 4 A schematic diagram showing another line segment merging result provided in an embodiment of the present application is shown; Figure 5 A schematic diagram of a video splitting method provided in an embodiment of the present application is shown; Figure 6 A structural block diagram of a video splitting device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0018] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0019] Before describing the solution of the present application, the terms that may be involved are first explained.

[0020] Image processing technology: including histogram comparison method, image difference method, edge detection method, and motion vector analysis method, which are explained below in turn.

[0021] Histogram comparison method: By comparing the pixel value histograms between consecutive frames, significant changes in pixel distribution are detected to identify shot boundaries.

[0022] Image difference method: Calculate the difference image between adjacent frames. If the difference exceeds a certain threshold, it is considered that a shot switch has occurred.

[0023] Edge detection method: Use the Sobel operator, Canny edge detection and other methods to identify changes in edge strength, and regard the points where the edge strength suddenly increases as the points where the lens changes.

[0024] Motion vector analysis method: Motion vectors can reveal the movement of objects, so the sudden change of motion vectors between consecutive frames is considered to indicate shot switching.

[0025] Statistical and machine learning methods: including cluster analysis, support vector machine, and hidden Markov model, which are explained below one by one.

[0026] Cluster analysis method: Use clustering algorithms such as K-means or DBSCAN to classify the feature vectors of consecutive frames, and frames in the same class are considered to belong to the same shot.

[0027] Support Vector Machine (SVM) trains an SVM model to classify whether consecutive frames belong to the same shot, based on the similarity or difference of feature vectors.

[0028] Hidden Markov Model (HMM): Establish a probabilistic model to predict the shot sequence. HMM can effectively capture the probabilistic characteristics of shot transitions.

[0029] ResNet50: A deep residual network with 50 layers, designed to solve the problem of gradient vanishing or exploding in deep neural networks. It introduces jump connections of identity mapping to enable the network to learn residual blocks, effectively promote information flow, simplify the optimization process, significantly improve the depth and performance of the model, and is widely used in image recognition and other fields.

[0030] OpenCV (Computer Vision Library): Often used in Python as the cv2 module, it is an efficient computer vision and machine learning software library. It provides hundreds of image and video analysis functions, such as image processing, object detection, face recognition, etc., supports cross-platform operations, and is widely used in research, development, and commercial products. It is an indispensable tool in the field of computer vision.

[0031] numpy.linalg: A submodule in the NumPy library dedicated to linear algebra operations. It provides a series of efficient and convenient functions, such as matrix decomposition, inversion, determinant calculation, eigenvalue problem solving, etc., which are suitable for mathematical operations on vectors and matrices.

[0032] Video shot recognition technology stems from the demand for efficient management of massive video content and technological progress. With the popularization of digital media and the development of the Internet, video data has grown exponentially. Early video management and retrieval mainly relied on manual annotation, which was inefficient and costly. Therefore, automated video analysis technology has become a research hotspot. In recent years, significant progress has been made in image processing technology and pattern recognition algorithms in the field of computer vision. Methods such as edge detection and histogram analysis are used to detect shot changes. At the same time, the improvement of hardware performance enables complex algorithms to be applied in real time. These technical backgrounds have promoted the development of video shot recognition technology, enabling it to quickly and accurately segment videos into independent shots, laying the foundation for subsequent content analysis, indexing, and retrieval.

[0033] The development of video shot recognition technology began in the late 1990s and early 21st century, with the advancement of computer vision and image processing technology. The initial technology relied on image processing methods such as edge detection and histogram analysis to detect shot boundaries. Subsequently, with the introduction of machine learning algorithms, especially the rapid development of deep learning technology, the accuracy and efficiency of video shot recognition have been significantly improved. In recent years, deep learning models such as convolutional neural networks (CNNs) have been widely used in video analysis, achieving more accurate shot detection and content understanding. These technological advances have not only improved the efficiency of video management, but also opened up broad prospects for applications such as video summary generation and content retrieval.

[0034] However, there are some problems with image processing technology. First, the same object will have a large difference in appearance at different viewing angles and distances, which increases the difficulty of recognition. Second, occlusion and background interference will also affect the recognition accuracy, resulting in feature loss or misdetection. In addition, fast-moving objects may cause image blur, affecting the clarity and accuracy of features. In addition, noise in the image acquisition process and artifacts in the compression process will affect the image quality, which in turn affects the recognition effect.

[0035] There are also some problems with statistical and machine learning methods. The first is the problem of overfitting and underfitting. That is, the model may perform well on the training set, but poorly on unseen data. This is a manifestation of overfitting. On the contrary, if the model performs poorly on the training set, it may be underfitting. Or class imbalance may cause the model to be biased towards the majority class, and the recognition ability of the minority class will decrease.

[0036] As can be seen from the above, the shot recognition method used in the prior art is highly dependent on the recognition accuracy of the image. When the recognition accuracy is poor, the accuracy of shot recognition is low, which in turn affects the accuracy of video segmentation.

[0037] In view of this, the embodiment of the present application provides a video splitting method, which constructs index coordinate points corresponding to the indexes of the two adjacent image frames when two adjacent image frames are identified to be similar, and classifies each index coordinate point in an iterative manner, further subdividing the shot segmentation points. Each initial continuous line segment after subdivision represents a shot, and two similar shots are merged into one by merging two adjacent initial continuous line segments, so that in the scenario where the image frame recognition accuracy is poor, the shot recognition result can still be corrected, so that each shot segmentation point identified is more accurate, and the effect of splitting the video based on the shot segmentation point is better.

[0038] The method can be executed in a terminal or a computer device. Taking running on a terminal as an example, Figure 1 FIG. 1 shows a hardware structure block diagram of a terminal provided in an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 The terminal may further include a transmission device 106 and an input / output device 108 for communication functions. It will be appreciated by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0039] The memory 104 can be used to store computer programs, for example, software programs and modules of application software. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, the above-mentioned method is realized. The memory 104 can be used to store data, for example, videos to be processed, feature information, index coordinate points, etc. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0041] Figure 2 A flow chart of a watermark adding method provided in an embodiment of the present application is shown. Figure 2 As shown, the method includes steps S201 to S205.

[0042] Step S201: Obtain image frames from the video to be processed, and assign indexes to the image frames, wherein the indexes are used to identify the image frames, and the indexes are arranged in sequence according to the timestamps of the image frames to obtain a group of continuous natural numbers.

[0043] In this step, the image frame may be each frame in the video to be processed, or may be a part of the frames in the video to be processed. When the image frame is a part of the frames in the video to be processed, the image frame may be obtained from the video to be processed by frame extraction, and the frame extraction includes: extracting the image frame from the video to be processed according to a preset time interval; or extracting the image frame from the video to be processed according to a preset sampling number.

[0044] The frame extraction process in this step is explained below through a specific example. If the duration of the video to be processed is 12 seconds, 60 image frames can be extracted from the video to be processed at a preset time interval of extracting an image frame every 0.2 seconds. Alternatively, the preset sampling number can be determined to be 100. Assuming that the duration of the video to be processed is 50 seconds, 100 image frames can be extracted from the video to be processed at a sampling frequency of 100 / 50=2 frames / second, wherein the preset time interval and the preset sampling number can be designed according to the actual application scenario, and the embodiment of the present application does not limit this. By way of example and not limitation, OpenCV can be used to perform frame extraction processing on the video to be processed.

[0045] Frame extraction reduces the total data volume of the video by reducing the number of frames in the video to be processed, thereby saving storage space and computing costs. The frame extraction sampling method at preset time intervals can not only ensure the content integrity of the video to be processed, but also reduce the resources consumed by computing, transmission, and storage.

[0046] In this step, the arrangement of the indexes is related to the order of the timestamps of the image frames, that is, the indexes corresponding to two adjacent image frames are also adjacent. Exemplarily, 10 image frames are obtained from the video to be processed, and the image frames are arranged in the order of the timestamps of the image frames as image frame A, image frame B...image frame J, the index of image frame A is 1, the index of image frame B is 2...image frame J is 10, where 1, 2...10 are a set of continuous natural numbers. Alternatively, the index of image frame A is 100, the index of image frame B is 99...image frame J is 98, where 100, 99...98 are a set of continuous natural numbers.

[0047] The index is used to identify the image frame, that is, the image frame corresponding to the index can be located through the index, and then the timestamp, pixel information, etc. of the image frame can be obtained. The advantage of setting the index to identify the image frame is that the continuity of the image frame can be represented by the continuity of the index in the future, and the continuity of the index is easier to represent on the coordinate axis, which simplifies the subsequent calculation process.

[0048] Step S202: Obtain the comparison result of each two adjacent image frames. When the comparison result indicates that the two adjacent image frames are similar, construct index coordinate points according to two indexes corresponding to the two adjacent image frames.

[0049] This step screens the image frames, removes dissimilar image frames, retains similar image frames, and represents the image frames with indexes and constructs index coordinate points. For example, the index coordinate point is [n, n-1], which means that the image frame with index n is similar to the image frame with index n-1.

[0050] In some embodiments, step S202 includes: extracting the pixel content of the image frame to obtain the pixel feature information of the image frame; obtaining the similarity distance between the pixel feature information of each two adjacent image frames; comparing the similarity distance with the third threshold value to generate the comparison result of the two adjacent image frames. Specifically, this embodiment identifies the pixel feature information of the image through a model, and determines the similarity between the image frames based on the comparison result of the similarity distance between each pixel feature information and the third threshold value, wherein the larger the similarity distance, the less similar the two are, and vice versa, the smaller the similarity distance, the more similar the two are. The model can be the ResNet50 model mentioned above or other similarity detection models, and this application does not limit this. In addition to comparing the similarity distance, the similarity can also be compared. The larger the similarity, the more similar the two are, and vice versa, the smaller the similarity, the more dissimilar the two are.

[0051] After comparing the similarity of all image frames in the video to be processed, a set of index coordinate points is generated. A set of index coordinate points can be represented by an array, such as np.array([[1, 1], [2, 2], [3, 3], [5, 4], [6, 5], [7, 6], [9, 8], [10, 9], [12, 11], [4, 3], [3, 2]]).

[0052] In the related art, image recognition technology is relied upon to identify shots. Then, when two image frames are identified as being similar, it is considered that the two image frames belong to the same shot. This is not suitable for identifying complex videos. Therefore, after identifying the comparison results of the image frames, this embodiment constructs index coordinate points, and uses subsequent steps to perform continuity detection on the index coordinate points to find the time point of the shot change in similar image frames, thereby correcting the comparison results of the image frames to make the identified shot segmentation points more accurate. For details, see the description of steps S203 to S204 below.

[0053] Step S203: classify the index coordinate points according to whether they are continuous on the coordinate axis, and generate initial continuous line segments, wherein each initial continuous line segment is composed of a plurality of index coordinate points that are continuous on the coordinate axis.

[0054] In this step, it is determined whether the index coordinate points are continuous on the coordinate axis, which may be an XY coordinate axis. The index coordinate points A, B, C, and D are used as examples for explanation. If the index coordinate point C is continuous on the coordinate axis, it means that the index coordinate point C is on the straight line generated by the index coordinate point A and the index coordinate point B, that is, the index coordinate point C is determined to be in the same category as the index coordinate point A and the index coordinate point B by the points on the line.

[0055] If the index coordinate point D is not continuous on the coordinate axis, it means that the index coordinate point D is not on the straight line generated by the index coordinate point A, the index coordinate point B, and the index coordinate point C. That is, the point is not on the line to determine that the index coordinate point D is not classified into the same category as the index coordinate point A, the index coordinate point B, and the index coordinate point C. Then, a new category is created to classify the index coordinate point D.

[0056] The index coordinate points classified into one category can generate an initial continuous line segment, and the initial continuous line segment is composed of at least two points, including two endpoints, or also including a middle point. Continuing with the example shown above, in the initial continuous line segment generated by index coordinate point A, index coordinate point B, and index coordinate point C, index coordinate point A and index coordinate point C are the two endpoints of the line segment, and index coordinate point B is the middle point of the line segment. Since the index identifies the image frame, it can be understood that image frame A is the starting frame of this storyboard segment, image frame B is the middle frame of this storyboard segment, and image frame C is the ending frame of this storyboard segment.

[0057] In some embodiments, the difference between the two preceding and following index coordinate points is calculated to determine whether the latter index coordinate point is continuous. Specifically, the index coordinate points are classified according to whether they are continuous on the coordinate axis to generate an initial continuous line segment, including: sorting the index coordinate points according to their order on any coordinate axis, and performing continuity tests on each sorted index coordinate point in turn, wherein the continuity test is used to determine whether the index coordinate point and its adjacent index coordinate points are on the same straight line; if they are on the same straight line, the index coordinate point is merged into the adjacent index coordinate points to generate an initial continuous line segment; if they are not on the same straight line, the index coordinate point is used as the starting point of a new initial continuous line segment.

[0058] In this embodiment, the index coordinate points are sorted according to the X-axis, which means that the index coordinate points are sorted from large to small or from small to large according to the X-axis coordinates, such as [1, 1], [2, 2], [3, 3], [5, 4], [6, 5], [7, 6], [9, 8], [10, 9], [12, 11], [4, 3], [3, 2] are sorted as [1, 1], [2, 2], [3, 2], [3, 3], [4, 3], [5, 4], [6, 5], [7, 6], [9, 8], [10, 9], [12, 11]. After sorting, the difference between the two index coordinate points before and after is calculated. If all (diff == 1) or all (diff == [1, 1]), the two coordinate points are classified as the same segment. Otherwise, a new segment is started and the loop comparison continues until the end. In this process, the coordinate point difference value can be dynamically controlled to improve the anti-interference ability of special video recognition. Among them, all() is a built-in function in Python, which is used to check whether all elements in iterable objects (such as lists, tuples, sets, etc.) are True. Here, all(diff==1) or all(diff==[1,1]) is used to determine whether the two index coordinate points are on the same line segment. Since the index identifies the image frame, the index coordinate points on the same line segment means that the two image frames are continuous, so the two image frames can be classified into one shot. In other words, this step transforms the judgment of whether the image frames belong to the same shot into the judgment of whether the index coordinate points are continuous, which is simpler in calculation.

[0059] In some embodiments, after step S203, the method further includes: removing initial continuous line segments whose line segment length is less than a second threshold.

[0060] Specifically, the line segment length can be the unit length of the line segment, or the number of index coordinate points on the initial continuous line segment. Take the number of index coordinate points as an example: if the line segment length of the initial continuous line segment is too short, it may be similar to image noise and artifacts. Removing these initial continuous line segments that will interfere with recognition is equivalent to filtering out false changes caused by common video interference factors such as lighting changes, camera shake, and fast motion, reducing false alarms, and helping to improve the recognition rate of real shot switching points.

[0061] Step S204: merge two initial continuous line segments whose coordinate difference of the index coordinate points is less than or equal to the first threshold to generate a target continuous line segment.

[0062] When the index coordinate points are sorted by the coordinate size of the X-axis or Y-axis, the change trends of the generated initial continuous line segments are consistent, either all are rising trends or all are falling trends. Therefore, when comparing the coordinate difference of the index coordinate points in two initial continuous line segments, only the two closest index coordinate points of the two initial continuous line segments can be compared to speed up the processing speed. Exemplarily, when the index coordinate points are sorted by the ascending order of the X-axis, the coordinate difference of the end point of the previous initial continuous line segment is compared with the starting point of the next initial continuous line segment. The coordinate difference refers to the difference on the coordinate axis, such as the X-axis difference and the Y-axis difference. A first threshold can be preset for comparison with the coordinate difference. If the coordinate difference is greater than the first threshold, no merging is performed. If the coordinate difference is less than or equal to the first threshold, merging is performed. The first threshold can be a natural number, such as 1, 2, 3, 4, 5, etc. The first threshold can also be a coordinate point composed of natural numbers, such as (1, 1), (2, 2), etc. It can be dynamically controlled according to the actual application scenario, and the embodiments of the present application do not limit this.

[0063] Reference below Figure 3 and Figure 4 The process of line segment merging and the setting of the first threshold are explained.

[0064] Figure 3 A schematic diagram showing a line segment merging result provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the horizontal axis is the X axis, the vertical axis is the Y axis, and the coordinate unit is 1. Each set of index coordinate points is Figure 3 Corresponding to a point in Figure 3 There are three initial continuous line segments, denoted as L1, L2, and L3. Here, the number of index coordinate points in the line segment is used to represent the line segment length. The line segment length of L1 is 3, which is less than or equal to the second threshold (3), so L1 is eliminated. Here, the coordinate difference is represented by the size of the Y-axis coordinates of the two closest points in L2 and L3. The coordinate difference between L2 and L3 is 1, which is equal to the threshold 1, so L2 and L3 are merged.

[0065] Figure 4 A schematic diagram showing another line segment merging result provided by an embodiment of the present application is shown. Figure 4 As shown, the horizontal axis is the X axis, the vertical axis is the Y axis, and the unit of the coordinate is 1. The coordinate difference between L4 and L5 is (1, 1), and the coordinate difference is equal to the threshold (1, 1), so L4 and L5 are merged.

[0066] In this step, the cutting strategy is dynamically adjusted according to the complexity of the video content, which means that the algorithm can still maintain high performance when processing videos of different types or qualities. The two shots that are not recognized by image recognition can be separated through a more reasonable first threshold, which has the effect of correcting the shot recognition result.

[0067] In some of the embodiments, the above-mentioned two continuous line segments whose coordinate difference of index coordinate points is less than a first threshold are merged to generate a target continuous line segment, including: if the values ​​of two adjacent index coordinate points on any coordinate axis are equal, and the coordinate difference on the other coordinate axis is less than the first threshold, the two adjacent index coordinate points are connected to obtain a target continuous line segment formed by the merger of the two initial continuous line segments.

[0068] In this embodiment, two initial continuous line segments are connected and merged into a target continuous line segment through two adjacent index coordinate points. It can be understood that when the line segment length of the continuous line segment is expressed by the number of index coordinate points, the line segment length of a target continuous line segment = the line segment length of the two initial continuous line segments - 1. For example, the line segment length of the first initial continuous line segment is 5, and the line segment length of the second initial continuous line segment is 4, then the line segment length of the target continuous line segment obtained by merging the two line segments is 5+4-1=8.

[0069] The method implemented by step S204 does not rely solely on a rough comparison of the feature vectors of the picture frames, but gradually refines the boundary detection through an iterative process, that is, it is able to detect the shot switching points more accurately, and thus shows higher accuracy in identifying fast-switching or gradually changing shots.

[0070] Step S205, obtain the index of the endpoint of the target continuous line segment to obtain the target index, use the timestamp of the image frame corresponding to the target index as the segmentation time point of the video to be processed, and split the video to be processed according to the segmentation time point to obtain the storyboard fragments corresponding to the video to be processed.

[0071] In this step, the target index is the coordinate of the endpoint of the target continuous line segment in the index. Taking the target continuous line segment composed of index coordinate point A, index coordinate point B, index coordinate point C, and index coordinate point E as an example, the target index is two indexes in index coordinate point A (recorded as the first group of indexes), and two indexes in index coordinate point E (recorded as the second group of indexes). As can be seen from the above, the image frame corresponding to the index can be located through the index, and then the timestamp, pixel information, etc. of the image frame can be obtained. Therefore, after obtaining the target index, the timestamp of the image frame corresponding to the target index is obtained. Then any time point between the timestamps of the two image frames corresponding to the first group of indexes is used as the segmentation time point of the video to be processed, and any time point between the timestamps of the two image frames corresponding to the second group of indexes is used as the segmentation time point of the video to be processed. Then, the video to be processed is divided into three segments, and the segment between the two segmentation time points is the storyboard segment corresponding to the video to be processed. By segmenting the video to be processed in this way, multiple storyboard segments can be obtained, and each storyboard segment is a segment of video under the same scene lens in the video to be processed.

[0072] In the embodiment described in the above steps S201 to S205, image frames are obtained from the video to be processed, and indexes are assigned to the image frames, wherein the indexes are used to identify the image frames, and the indexes are arranged in sequence according to the timestamps of the image frames to obtain a group of continuous natural numbers; the comparison results of each two adjacent image frames are obtained, and when the comparison results indicate that the two adjacent image frames are similar, index coordinate points are constructed according to the two indexes corresponding to the two adjacent image frames; the index coordinate points are classified according to whether they are continuous on the coordinate axis, and initial continuous line segments are generated, wherein each initial continuous line segment is composed of a plurality of index coordinate points that are continuous on the coordinate axis; two continuous line segments whose coordinate difference of the index coordinate points is less than a first threshold are merged to generate a target continuous line segment; the indexes of the endpoints of the target continuous line segment are obtained to obtain the target index, and the timestamps of the image frames corresponding to the target indexes are used as the segmentation time points of the video to be processed, and the video to be processed is split at the segmentation time points to obtain storyboard fragments corresponding to the video to be tested. In the embodiment of the present application, when two adjacent image frames are identified as being similar, index coordinate points corresponding to the indexes of the two adjacent image frames are constructed, and each index coordinate point is classified in an iterative manner to further subdivide the lens segmentation points. Each initial continuous line segment after subdivision represents a segment of the lens, and two similar segments of the lens are merged into one segment by merging two adjacent initial continuous line segments, so that in a scenario where the image frame recognition accuracy is poor, the lens recognition result can still be corrected, so that each lens segmentation point identified is more accurate, and the effect of splitting the video based on the lens segmentation point is better. In addition, the first threshold for merging similar initial continuous line segments can be adjusted dynamically, so that the algorithm can still maintain a high performance when processing videos of different types or qualities, and dynamically adjusting the cutting strategy according to the complexity of the video content also makes the algorithm more flexible.

[0073] The video splitting method provided in the embodiment of the present application is described and illustrated by means of specific examples below.

[0074] Figure 5 A schematic diagram of a video splitting method provided in an embodiment of the present application is shown, Figure 5 As shown, the method includes: Step S501, obtaining basic video information through the cv2 module.

[0075] In this step, the basic video information includes various parameters related to the video to be processed, such as the video to be processed above, the timestamp of each image frame, the shooting location information, the camera parameters, etc.

[0076] Step S502: Perform video frame extraction processing at specified intervals.

[0077] The frame extraction process can refer to the description of step S201 above, which will not be repeated here.

[0078] Step S503: extract vectors using the ReSNet50 model.

[0079] The vector is equivalent to the pixel feature information mentioned above. For the step of extracting the vector, reference may be made to the description of the embodiment of recognizing an image through a model in step S202 above.

[0080] Step S504: enumerate and iterate the video frame vector group.

[0081] Step S505: Calculate the distance between the front and back frame vectors.

[0082] Step S506: determine whether the vector distance is less than 0.8.

[0083] Step S507a: If yes, save the previous and next frame index coordinates.

[0084] Among them, 0.8 is the threshold of the similarity distance, which is equivalent to the third threshold mentioned above. The third threshold can be adjusted dynamically, and this step is only an example and is not intended to limit the value of the third threshold.

[0085] Step S507b: If not, discard the index coordinate.

[0086] Steps S504 to S507b may refer to the description of step S202 above, and will not be described in detail here.

[0087] Step S508: Perform continuity detection on the sorted coordinate groups and classify them based on the detection results.

[0088] Step S509: determine whether the difference between the front and rear coordinates is 1 or [1, 1].

[0089] Among them, 1 or [1, 1] is equivalent to the first threshold above.

[0090] Step S510a: If yes, classify the coordinate difference as a line segment.

[0091] Step S510b: If not, create a new line segment to store the coordinate difference.

[0092] For steps S508 to S510b, reference may be made to the description of step S203 above, and will not be repeated here.

[0093] Step S511, determine whether the length of the line segment is greater than 3.

[0094] This step removes image frames (or video frames) that may be affected by noise and artifacts, which is equivalent to filtering out image frames that cause false changes. That is, the segmentation caused by noise and artifacts does not mean that the camera has switched. The image frame corresponding to the changed line segment and its adjacent line segment is still an image under one camera. The "3" in this step is equivalent to the second threshold above.

[0095] Step S512a: If yes, merge the line segments that are close at the beginning and the end.

[0096] Step S512b: If not, delete the line segment.

[0097] Step S513: convert the line segment into video shot information.

[0098] For this step, please refer to the description of step S205 above, which will not be described in detail here.

[0099] Step S514: Use MySQL to store data.

[0100] A video splitting device is also provided in the embodiments of the present application, and the device is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0101] Figure 6 A structural block diagram of a video splitting device provided in an embodiment of the present application is shown. Figure 6 As shown, the device comprises: The image acquisition module 61 is used to acquire image frames from the video to be processed and assign indexes to the image frames, wherein the indexes are used to identify the image frames, and the indexes are arranged in sequence according to the timestamps of the image frames to obtain a set of continuous natural numbers.

[0102] The similarity comparison module 62 is used to obtain the comparison result of each two adjacent image frames, and when the comparison result indicates that the two adjacent image frames are similar, construct an index coordinate point according to two indexes corresponding to the two adjacent image frames.

[0103] The continuity detection module 63 is used to classify the index coordinate points according to whether the index coordinate points are continuous on the coordinate axis, and generate initial continuous line segments, wherein each initial continuous line segment is composed of a plurality of index coordinate points that are continuous on the coordinate axis.

[0104] The line segment merging module 64 is used to merge two initial continuous line segments whose coordinate difference of the index coordinate points is less than or equal to the first threshold value to generate a target continuous line segment.

[0105] The video splitting module 65 is used to obtain the index of the endpoint of the target continuous line segment, obtain the target index, use the timestamp of the image frame corresponding to the target index as the segmentation time point of the video to be processed, and split the video to be processed at the segmentation time point to obtain the storyboard fragments corresponding to the video to be processed.

[0106] In some embodiments, the image acquisition module 61 is further used to extract image frames from the video to be processed according to a preset time interval; or to extract image frames from the video to be processed according to a preset sampling number.

[0107] In some of the embodiments, the similarity comparison module 62 is also used to extract pixel content of the image frame to obtain pixel feature information of the image frame; Obtain the similarity distance between the pixel feature information of each two adjacent image frames; The similarity distance is compared with the third threshold to generate a comparison result of two adjacent image frames.

[0108] In some of the embodiments, the continuity detection module 63 is further used to sort the index coordinate points according to their order on any coordinate axis, and perform continuity detection on each sorted index coordinate point in turn, wherein the continuity detection is used to determine whether the index coordinate point and its adjacent index coordinate points are on the same straight line; if they are on the same straight line, the index coordinate point is merged into the adjacent index coordinate points to generate an initial continuous line segment; if they are not on the same straight line, the index coordinate point is used as the starting point of a new initial continuous line segment.

[0109] In some embodiments, the continuity detection module 63 is further configured to remove initial continuous line segments whose line segment length is less than a second threshold.

[0110] In some embodiments, the continuity detection module 63 is further configured to remove initial continuous line segments on which the number of index coordinate points is less than a second threshold.

[0111] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0112] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0113] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0114] Optionally, in this embodiment, the processor may be configured to execute the following steps S1 to S5 through a computer program.

[0115] S1. Obtain image frames from the video to be processed, and assign indexes to the image frames, wherein the indexes are used to identify the image frames, and the indexes are arranged in sequence according to the timestamps of the image frames to obtain a group of continuous natural numbers.

[0116] S2. Obtain the comparison result of each two adjacent image frames. When the comparison result indicates that the two adjacent image frames are similar, construct index coordinate points according to two indexes corresponding to the two adjacent image frames.

[0117] S3. Classify the index coordinate points according to whether they are continuous on the coordinate axis, and generate initial continuous line segments, wherein each initial continuous line segment is composed of a plurality of index coordinate points that are continuous on the coordinate axis.

[0118] S4. Merge two initial continuous line segments whose coordinate difference of the index coordinate points is less than or equal to the first threshold to generate a target continuous line segment.

[0119] S5. Obtain the index of the endpoint of the target continuous line segment to obtain the target index, use the timestamp of the image frame corresponding to the target index as the segmentation time point of the video to be processed, and split the video to be processed according to the segmentation time point to obtain the storyboard fragments corresponding to the video to be processed.

[0120] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0121] In addition, in combination with the method provided in the above embodiment, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the video splitting methods in the above embodiment is implemented.

[0122] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes each function or step executed by the processor in the above method embodiment.

[0123] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0124] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0125] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0126] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A video splitting method, characterized in that: The method comprises: Obtaining an image frame from the video to be processed, and assigning an index to the image frame, wherein the index is used to identify the image frame, and each of the indexes is arranged in sequence according to the timestamp of the image frame to obtain a set of continuous natural numbers; Obtaining a comparison result of each two adjacent image frames, and when the comparison result indicates that the two adjacent image frames are similar, constructing an index coordinate point according to two indexes corresponding to the two adjacent image frames; Classifying the index coordinate points according to whether the index coordinate points are continuous on the coordinate axis to generate initial continuous line segments, wherein each of the initial continuous line segments is composed of a plurality of index coordinate points that are continuous on the coordinate axis; Merge the two initial continuous line segments whose coordinate difference of the index coordinate point is less than or equal to the first threshold to generate a target continuous line segment; Obtain the index of the endpoint of the target continuous line segment to obtain the target index, use the timestamp of the image frame corresponding to the target index as the segmentation time point of the video to be processed, and split the video to be processed at the segmentation time point to obtain the storyboard fragments corresponding to the video to be processed.

2. The video splitting method according to claim 1, characterized in that: Classifying the index coordinate points according to whether the index coordinate points are continuous on the coordinate axis to generate initial continuous line segments includes: Sorting the index coordinate points according to their order on any coordinate axis, and sequentially performing continuity detection on each of the sorted index coordinate points, wherein the continuity detection is used to determine whether the index coordinate point and its adjacent index coordinate points are on the same straight line; If they are on the same straight line, the index coordinate point is merged into the adjacent index coordinate point to generate an initial continuous line segment; If they are not on the same straight line, the index coordinate point is used as the starting point of a new initial continuous line segment.

3. The video splitting method according to claim 1, characterized in that: The step of merging the two initial continuous line segments whose coordinate difference of the index coordinate points is less than or equal to the first threshold to generate a target continuous line segment includes: If the values ​​of two adjacent index coordinate points on any coordinate axis are equal and the coordinate difference on the other coordinate axis is less than or equal to the first threshold, the two adjacent index coordinate points are connected to obtain a target continuous line segment formed by merging the two initial continuous line segments.

4. The video splitting method according to claim 1, characterized in that: After generating the initial continuous line segments, the method further includes: Initial continuous line segments whose line segment length is less than a second threshold are eliminated.

5. The video splitting method according to claim 4, characterized in that: The line segment length is represented by the number of index coordinate points on the initial continuous line segment, and the initial continuous line segment whose line segment length is less than a second threshold is eliminated, including: Initial continuous line segments whose number of index coordinate points is less than a second threshold are eliminated.

6. The video splitting method according to any one of claims 1 to 5, characterized in that: The obtaining of the comparison result of each two adjacent image frames includes: Extracting pixel content of the image frame to obtain pixel feature information of the image frame; Obtain the similarity distance between the pixel feature information of each two adjacent image frames; The similarity distance is compared with a third threshold to generate a comparison result of the two adjacent image frames.

7. The video splitting method according to any one of claims 1 to 5, characterized in that: The step of obtaining an image frame from the video to be processed includes: According to a preset time interval, image frames are extracted from the video to be processed; or according to a preset sampling number, image frames are extracted from the video to be processed.

8. A video splitting device, characterized in that: include: An image acquisition module, used to acquire image frames from the video to be processed, and assign indexes to the image frames, wherein the indexes are used to identify the image frames, and the indexes are arranged in sequence according to the timestamps of the image frames to obtain a set of continuous natural numbers; A similarity comparison module, used to obtain a comparison result of each two adjacent image frames, and when the comparison result indicates that the two adjacent image frames are similar, construct an index coordinate point according to two indexes corresponding to the two adjacent image frames; A continuity detection module, used to classify the index coordinate points according to whether the index coordinate points are continuous on the coordinate axis, and generate initial continuous line segments, wherein each of the initial continuous line segments is composed of a plurality of index coordinate points that are continuous on the coordinate axis; A line segment merging module, used for merging two of the initial continuous line segments whose coordinate difference of the index coordinate point is less than or equal to a first threshold value to generate a target continuous line segment; A video splitting module is used to obtain the index of the endpoint of the target continuous line segment to obtain the target index, use the timestamp of the image frame corresponding to the target index as the segmentation time point of the video to be processed, and split the video to be processed at the segmentation time point to obtain the storyboard fragments corresponding to the video to be processed.

9. A terminal comprising a memory, a processor and a display screen, characterized in that: A computer program is stored in the memory, the processor is configured to run the computer program to execute the video splitting method described in any one of claims 1 to 7, and the display screen is configured to display the second video and audio processed by the processor.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the video splitting method according to any one of claims 1 to 7 when running.

Images