Data processing method and device, equipment, storage medium and computer program product

Abstract

The application provides a data processing method and device, equipment, a storage medium and a computer program product, which can be applied to the fields or scenes of artificial intelligence, vehicle-mounted, etc. The method comprises the following steps: acquiring a to-be-processed video frame sequence comprising M video frames, performing global feature extraction and local feature extraction on each of the M video frames to obtain a global feature vector and a local feature vector of each video frame; constructing a global similarity matrix and a local similarity matrix of the M video frames according to the obtained M global feature vectors and M local feature vectors; determining a time sequence feature sequence corresponding to the to-be-processed video frame sequence according to the global similarity matrix and the local similarity matrix, performing time sequence enhancement processing on the time sequence feature sequence, processing the time sequence feature sequence after the enhancement processing, and determining a scene segmentation point detection result of each video frame. Through the method, the accuracy of video scene segmentation can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods, data processing apparatus, computer equipment, computer-readable storage media, and computer program products. Background Technology

[0002] Scene segmentation is a crucial technique in video editing. Its purpose is to break down a video into segments with different story backgrounds based on scene changes. The core of this technique is determining the scene segmentation points within the video (i.e., identifying different scenes). Currently, clustering-based scene segmentation methods are quite popular. These methods extract video frame features using feature models and then use clustering and other methods to group segments with similar features together, thereby identifying scene segments.

[0003] However, clustering-based methods are limited by backend feature clustering, and their accuracy in determining scene segmentation points in videos with complex scenes is not high enough. Therefore, how to more accurately determine scene segmentation points in videos has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides data processing methods, apparatus, devices, storage media, and computer program products that can improve the accuracy of video scene segmentation.

[0005] One embodiment of this application discloses a data processing method, the method comprising:

[0006] Obtain the video frame sequence to be processed from the target video, wherein the video frame sequence to be processed includes M video frames, where M is a positive integer greater than 1;

[0007] Global feature extraction and local feature extraction are performed on each of the M video frames to obtain the global feature vector and local feature vector of each video frame;

[0008] Based on the obtained M global feature vectors, construct the global similarity matrix of the M video frames, and based on the obtained M local feature vectors, construct the local similarity matrix of the M video frames.

[0009] The temporal feature sequence corresponding to the video frame sequence to be processed is determined based on the global similarity matrix and the local similarity matrix. The temporal feature sequence is then subjected to temporal enhancement processing to obtain the enhanced temporal feature sequence.

[0010] The enhanced temporal feature sequence is processed to determine the scene segmentation point detection result for each of the M video frames.

[0011] One embodiment of this application discloses a data processing apparatus, which includes:

[0012] The acquisition unit is used to acquire a sequence of video frames to be processed from the target video, wherein the sequence of video frames to be processed includes M video frames, where M is a positive integer greater than 1;

[0013] Processing unit, used for:

[0014] Global feature extraction and local feature extraction are performed on each of the M video frames to obtain the global feature vector and local feature vector of each video frame;

[0015] Based on the obtained M global feature vectors, construct the global similarity matrix of the M video frames, and based on the obtained M local feature vectors, construct the local similarity matrix of the M video frames.

[0016] The temporal feature sequence corresponding to the video frame sequence to be processed is determined based on the global similarity matrix and the local similarity matrix. The temporal feature sequence is then subjected to temporal enhancement processing to obtain the enhanced temporal feature sequence.

[0017] The enhanced temporal feature sequence is processed to determine the scene segmentation point detection result for each of the M video frames.

[0018] One aspect of this application discloses a computer device, which includes a processor adapted to implement one or more computer programs; and a computer-readable storage medium storing one or more computer programs adapted to be loaded by the processor and executed in accordance with the above-described data processing method.

[0019] One aspect of this application discloses a computer-readable storage medium storing one or more computer programs adapted to be loaded by a processor and executed by the above-described data processing method.

[0020] One embodiment of this application discloses a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned data processing method.

[0021] In this embodiment, the data processing method mainly includes: acquiring a sequence of video frames to be processed from the target video, the sequence of video frames to be processed including M video frames; performing global feature extraction and local feature extraction on each of the M video frames to obtain a global feature vector and a local feature vector for each video frame; then constructing a global similarity matrix of the M video frames based on the obtained M global feature vectors, and constructing a local similarity matrix of the M video frames based on the obtained M local feature vectors; determining the temporal feature sequence corresponding to the sequence of video frames to be processed based on the global similarity matrix and the local similarity matrix, combining global and local features, not only referring to the global similarity between image frames, but also referring to the local similarity between image frames, combining the two can more accurately determine whether the image frames have the same scene, and then performing temporal enhancement processing on the temporal feature sequence to obtain an enhanced temporal feature sequence, which is equivalent to adding time sequence to the feature sequence to avoid scene segmentation point recognition errors caused by temporal disorder; and then processing the enhanced temporal feature sequence to determine the scene segmentation point detection result of each of the M video frames. This method can determine whether each video frame is a scene segmentation frame. Specifically, this application uses multiple similarity matrices, combines local and global features, and adds temporal sequence to the features to make the similarity confirmation between video frames more accurate, thereby improving the accuracy of judging whether a video frame is a scene segmentation frame, which in turn improves the accuracy of video scene segmentation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the architecture of a data processing system disclosed in an embodiment of this application;

[0024] Figure 2 This is a flowchart illustrating a data processing method disclosed in an embodiment of this application;

[0025] Figure 3a This is a schematic diagram of the structure of a timing deformer disclosed in an embodiment of this application;

[0026] Figure 3b This is a schematic diagram of the structure of a deformable encoder disclosed in an embodiment of this application;

[0027] Figure 4This is a data processing system framework diagram disclosed in an embodiment of this application;

[0028] Figure 5 This is a flowchart illustrating another data processing method disclosed in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the structure of a data processing device disclosed in an embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the structure of a computer device disclosed in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] To effectively improve the accuracy of similarity comparison between video frames and thus accurately identify scene segmentation frames in the video, this application proposes a data processing method. This method can solve the above-mentioned problems. The data processing method provided by this application involves artificial intelligence technology and its sub-technologies such as computer vision and machine learning. The relevant technologies are described below:

[0033] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0034] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0035] This application relates to computer vision technology and machine learning, both branches of artificial intelligence. Computer vision (CV) is the science that studies how machines "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. Machine learning is the core of artificial intelligence, a fundamental approach to enabling computers to possess intelligence. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0036] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a data processing system according to an embodiment of this application, such as... Figure 1 As shown, the data processing system 100 may include at least a terminal device 101 and a computer device 102. The terminal device 101 and the computer device 102 can be connected via a wired or wireless connection, which is not limited here. In the specific implementation, the terminal device 101 is mainly used to display the results. In this application, it can display the scene segmentation point detection results of each of the M video frames, or it can be used to display video segments output based on the segmentation points. The computer device 102 is mainly used to acquire the sequence of video frames to be processed from the target video, and to process the sequence of video frames to be processed, including feature extraction, construction of a similarity matrix, and temporal enhancement processing, thereby obtaining the scene segmentation point detection results of each of the M video frames.

[0037] In one possible implementation, the terminal device 101 mentioned above includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.; the computer device 102 mentioned above can be a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Figure 1 This is merely an illustrative representation of the architecture of a data processing system and is not intended to be limiting. For example, Figure 1 The computer device 102 can be deployed as a node in the blockchain network, or the computer device 102 can be connected to the blockchain network, so that the computer device 102 can upload the video frame sequence to be processed of the target video and the scene segmentation point detection results of each video frame in M ​​video frames to the blockchain network for storage, so as to prevent the internal data from being tampered with and thus ensure data security.

[0038] In conjunction with the aforementioned data processing system, the data processing method of this application embodiment can generally include: the computing device 102 obtains a sequence of video frames to be processed from the terminal device 101, the sequence of video frames to be processed including M video frames; global feature extraction and local feature extraction are performed on each of the M video frames to obtain a global feature vector and a local feature vector for each video frame; then, based on the obtained M global feature vectors, a global similarity matrix of the M video frames is constructed, and based on the obtained M local feature vectors, a local similarity matrix of the M video frames is constructed; the temporal feature sequence corresponding to the sequence of video frames to be processed is determined based on the global similarity matrix and the local similarity matrix; the temporal feature sequence is subjected to temporal enhancement processing to obtain an enhanced temporal feature sequence; this is equivalent to adding a time sequence to the feature sequence; the enhanced temporal feature sequence is processed to determine the scene segmentation point detection result of each of the M video frames; the computing device 102 can also send the segmentation point detection result to the terminal device 101 so that the terminal device 101 can perform operations such as video content creation based on the scene segmentation point detection result. This method can determine whether each video frame is a scene segmentation point. Furthermore, by using multiple similarity matrices and adding temporal sequence to the features, the similarity judgment between video frames can be made more accurate. Based on the more accurate similarity matrix, the accuracy of judging whether a video frame is a scene segmentation frame can be improved, which means that the accuracy of video scene segmentation can be improved.

[0039] The data processing methods mentioned in this application can be applied to any video editing scenario. Typical scenarios include: 1) In video creation assistance tasks, user-input videos are split into scene-independent segments to meet users' needs for montage editing and extraction of highlights; 2) In copyright scenarios, long videos can be split into scene-independent video segments for similarity comparison, improving the accuracy and efficiency of similarity judgment; 3) In video creation assistance tasks, each scene-independent video segment is labeled with fine-grained tags to meet users' needs for precise retrieval; 4) In video content distribution tasks, corresponding descriptions are added to the split video scene segments, and they are automatically distributed.

[0040] The data processing method mentioned in this application can also be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The data processing system can also be deployed in a blockchain network. It should be noted that in the specific embodiments of this application, the data involved, including the target video and the video sequence to be processed from the target video, are all authorized by the user. When the above embodiments of this application are applied to specific products or technologies, the data used must be authorized or agreed to by the user, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0041] Based on the above description of the data processing system architecture, this application discloses a data processing method. Please refer to [link to relevant documentation]. Figure 2 This is a flowchart illustrating a data processing method disclosed in an embodiment of this application. This data processing method can be executed by a computer device and specifically includes steps S201 to S205:

[0042] S201. Obtain the video frame sequence to be processed from the target video. The video frame sequence to be processed includes M video frames, where M is a positive integer greater than 1.

[0043] The video frame sequence to be processed includes M video frames. The M video frames are arranged in chronological order, with the first video frame being earlier in time and closer to the beginning of the video, and the last video frame being later in time and closer to the end of the video.

[0044] In one possible implementation, before determining the video frame sequence to be processed, a target video can be acquired first. The target video can be of any type, either captured in real-time by the user or retrieved from a database. After acquiring the target video, it is processed by equally spaced frame extraction to obtain the video frames to be processed. This application considers the balance between accuracy and time / computing resources, employing a sparse frame extraction method. Specifically, it can be equally spaced frame extraction, such as 1 frame per second. Since most scene changes will not be less than 1 second, the 1-frame-per-second extraction method will not lead to missed or missed scenes at segmentation points. For example, for a target video of (1600, 25fps), using a 1fps (frame rate, 1 frame per second) sampling method, 64 video frames can be obtained. Alternatively, for a T-second target video, using a 1-frame-per-second extraction method, a video frame sequence to be processed can be obtained, comprising a total of T video frames, including {I1, I2, ..., I...}. T Of course, dense sampling can achieve more accurate segmentation. However, in practical applications, due to limitations in model processing time, sparse frame extraction is generally preferred. In some cases, video frames in the video frame sequence to be processed can be replaced by other keyframes in the video, or video frames obtained by a learning-based frame extraction method. Alternatively, all video frames of the target video can be processed (which is time-consuming and complex). This application does not impose any limitations on this approach.

[0045] S202. Perform global feature extraction and local feature extraction on each of the M video frames to obtain the global feature vector and local feature vector of each video frame.

[0046] In this application, both global and local feature extraction are performed simultaneously for each video frame. Global feature extraction refers to the entire content of a video frame, while local feature extraction refers to a specific object within that frame. Each video frame extracts one global feature vector and one local feature vector. This application exemplifies this with M video frames, resulting in M ​​global feature vectors and M local feature vectors.

[0047] In one possible implementation, before extracting local feature vectors, it's necessary to first determine what the local information is, that is, to identify the local objects, and then extract local features based on these objects. In a video frame, an image includes foreground and background. For example, in a landscape video containing people, the foreground is usually people, and the background is usually scenery. Changes in the people often indicate changes in the scene. Therefore, on the one hand, local objects can be identified based on the foreground people, but some special cases cannot be ruled out. Therefore, on the other hand, local objects can be identified based on the heatmap of the video frame; the object with the highest heat value can be used as the local object for local feature extraction. Alternatively, during training, local objects can be pre-labeled, and the labels can be used to identify local objects for local feature extraction.

[0048] In one possible implementation, a CLIP pre-trained model can be used as a feature extractor. For each video frame, the CLIP model transforms the video frame into fixed-size blocks, which are then processed by a transformer to obtain image features. Besides the above method, other image feature extraction algorithms can also be used, such as Scale-invariant Feature Transform (SIFT) and Histogram of Oriented Gradients (HOG), to extract features from video frames. Alternatively, feature extraction can be performed using deep machine learning-based neural networks, such as Convolutional Neural Networks (CNNs). The specific choice depends on the actual situation, and this application does not limit the specific implementation.

[0049] When performing global or local feature extraction, one can extract a single feature corresponding to each video frame as a feature vector using one feature extraction method, or extract multiple single feature vectors corresponding to each video frame using multiple feature extraction methods and then combine them to obtain a feature vector.

[0050] S203. Based on the obtained M global feature vectors, construct the global similarity matrix of M video frames, and based on the obtained M local feature vectors, construct the local similarity matrix of M video frames.

[0051] The similarity matrix is ​​constructed based on the distance between feature vectors. The global similarity matrix and the local similarity matrix can be determined in one way (i.e., one distance calculation method) or in multiple ways (i.e., multiple distance calculation methods).

[0052] In one possible implementation, taking multiple distance calculation methods as an example, the distance calculation methods include cosine distance and Euclidean distance. Therefore, constructing a global similarity matrix for M video frames can specifically include: calculating the cosine distance and Euclidean distance between a first global feature vector and a second global feature vector. The first global feature vector is any one of the M global feature vectors, and the second global feature vector is any one of the M global feature vectors. During the calculation process, for any global feature vector, it is also necessary to calculate the cosine distance and Euclidean distance between that global feature vector and itself. Based on the calculated cosine distance between the global feature vectors, the first global similarity matrix is ​​determined. Then, based on the calculated Euclidean distance between the global feature vectors, the second global similarity matrix is ​​determined. Finally, the first global similarity matrix and the second global similarity matrix are used to determine the global similarity matrix for the M video frames.

[0053] Similarly, constructing a local similarity matrix for M video frames can specifically include: calculating the cosine distance and Euclidean distance between a first local feature vector and a second local feature vector, where the first local feature vector is any one of the M local feature vectors, and the second local feature vector is any one of the M local feature vectors. During the calculation process, for any local feature vector, it is also necessary to calculate the cosine distance and Euclidean distance between that local feature vector and itself; determining the first local similarity matrix based on the calculated cosine distance between the local feature vectors; determining the second local similarity matrix based on the calculated Euclidean distance between the local feature vectors; and finally, using the first local similarity matrix and the second local similarity matrix as the local similarity matrix for the M video frames.

[0054] The cosine distance is calculated using formula (1):

[0055]

[0056] The Euclidean distance is calculated using formula (2):

[0057] D(i,j)=‖F i ||-||F j || (2)

[0058] Formulas (1) and (2) are both calculations between vectors. Specifically, the first and second global similarity matrices can be normalized separately and then concatenated to obtain the global similarity matrix. Similarly, the first and second local similarity matrices can also be normalized separately and then concatenated to obtain the local similarity matrix. The normalization process here standardizes the similarity values ​​between video frames within a certain range to facilitate subsequent processing.

[0059] In this application, both the global similarity matrix and the local similarity matrix between video frames are obtained simultaneously, which allows for a more accurate determination of whether video frames are similar. For example, for two images with the same background and both displaying a person, if only global similarity is used for similarity determination, these two images are very likely to be identified as the same frame. However, if local features, such as a person's hair and clothing, are added, and the local features are similar, the two frames can be determined to be similar more accurately. As a result, the similarity determination becomes more accurate.

[0060] S204. Determine the temporal feature sequence corresponding to the video frame sequence to be processed based on the global similarity matrix and the local similarity matrix, and perform temporal enhancement processing on the temporal feature sequence to obtain the enhanced temporal feature sequence.

[0061] In one possible implementation, the global similarity matrix and the local similarity matrix are first concatenated to obtain a concatenated similarity matrix. Then, an inter-frame similarity processing network is used to process the concatenated similarity matrix to obtain a temporal feature sequence. The temporal feature sequence includes M temporal features, and each temporal feature corresponds to one video frame among the M video frames.

[0062] In one possible implementation, when the global similarity matrix includes a first global similarity matrix and a second similarity matrix, and the local similarity matrix includes a first local similarity matrix and a second local similarity matrix, obtaining the temporal feature sequence can specifically include: concatenating the first global similarity matrix, the second global similarity matrix, the first local similarity matrix, and the second local similarity matrix to obtain a concatenated similarity matrix; the concatenation here can be a simple concatenation; then using an inter-frame similarity processing network to process the concatenated similarity matrix to obtain the temporal feature sequence, the inter-frame similarity processing network can be a fully connected layer in a neural network; wherein, the temporal feature sequence includes M temporal features, each temporal feature corresponding to one video frame among the M video frames, or the temporal feature sequence can be understood as a temporal feature matrix, where each row of the matrix represents a video frame.

[0063] Furthermore, after determining the temporal feature sequence, a temporal enhancement network is used to perform temporal enhancement processing on the temporal feature sequence to obtain an enhanced temporal feature sequence. The enhanced temporal feature sequence includes M enhanced temporal features, each of which corresponds to one video frame among the M video frames. The temporal enhancement network can specifically refer to a temporal deformer, which can enhance the relationship between temporal features through autocorrelation and cross-correlation. This application adopts a standard transformer module, which contains several encoder layers. After all sequence features are processed through attention and MLP operations, they are transformed into the same feature sequence output, and then global pooling is used to obtain the final feature output.

[0064] The specific timing deformer structure diagram can be seen as follows: Figure 3a As shown, it mainly includes an embedding layer (Linear Projection of Flattened Patches), a deformation encoder (Transformer Ecoder), and an MLP Head (for classification). Based on the structure of the temporal deformer, the processing can be divided into steps S1-S5:

[0065] S1: Divide the image into patches; for this application, this means splitting a temporal feature sequence, and splitting it a certain number of times depending on the number of temporal feature sequences.

[0066] S2: The patches split in step S1 are transformed into embeddings. This can be achieved by first flattening the patches to reduce their dimensionality, and then performing a linear mapping on the dimensionality-reduced patches to obtain the embeddings. Besides using linear layers for linear mapping, convolution can also be used to obtain the embeddings. In this application, the linear mapping method is used to obtain the patch embedding corresponding to each patch in the temporal feature sequence.

[0067] S3: Generate positional codes for all sequences to obtain positional embeddings, and add the positional embeddings to the embeddings from step S2. In this application, when splitting the temporal feature sequence, its corresponding positional embedding can be generated, and then the positional embeddings and the embeddings corresponding to the patches are combined to obtain the combined embedding.

[0068] S4: The combined embeddings are input into the TRM (Transformer Encoder) model for processing, resulting in the input feature sequence, which is then integrated. The Transformer Encoder architecture diagram is shown below. Figure 3b As shown, the processing mainly utilizes an attention mechanism, comprising L modules. Each module includes a first normalization layer, an attention layer, a second normalization layer, and a multilayer perceptron (MLP, commonly understood as a convolutional layer). In this application, the combined embedding is input into the TRM, processed layer by layer, and finally subjected to global pooling to obtain the enhanced temporal feature sequence.

[0069] S5: Categorization and processing, see step S205 for details.

[0070] In this application, this step may specifically include: for each feature element in the temporal feature sequence, where each feature element can be considered as a set of vectors representing a video frame, performing block processing on each feature element to obtain block information for each feature element, and determining the position embedding information of each feature element based on each feature element and its block information; performing feature transformation processing on the block information of each feature element to obtain feature embedding information for each feature element; combining the feature embedding information and position embedding information of each feature element to obtain combined embedding information for each feature element; and finally inputting the combined embedding information into a deformer for processing to obtain an enhanced temporal feature sequence, with each video frame corresponding to one enhanced temporal feature sequence.

[0071] S205. Process the enhanced temporal feature sequence to determine the scene segmentation point detection result for each of the M video frames.

[0072] The enhanced temporal feature sequence still includes M features, which are M enhanced temporal feature sequences.

[0073] In one possible implementation, a classification network is used to process the enhanced temporal feature sequence to obtain a classification probability value corresponding to each enhanced temporal feature. Then, each classification probability value is processed to obtain scene segmentation point detection results for M video frames. This classification probability value is mainly used to indicate whether the corresponding video frame is a scene segmentation point. Since the model's classification results may have biases, this application employs a non-maximum suppression method to process the classification probability values ​​to obtain more accurate segmentation points, thereby making the classification results more accurate.

[0074] Specifically, non-maximum suppression (NMS) merges any two classification probability values ​​whose difference is within a set range, and compares the merged classification probability value with a probability threshold. Based on the comparison result, the scene segmentation point detection result for each of the M video frames is determined. NMS is an algorithm for removing non-maximum values, commonly used in edge detection and object recognition in computer vision. The main algorithm flow is as follows: Given an image and many candidate bounding boxes for object detection (each box may represent a certain object), but these boxes may overlap. The goal is to retain only the optimal bounding box. Assume there are N bounding boxes, and each box is scored Si by the classifier, where 1 <= i <= N. First, a set H is constructed to store the candidate bounding boxes to be processed, initialized to contain all N boxes. Simultaneously, a set M is constructed to store the optimal bounding boxes, initialized to an empty set. 1. Sort all boxes in set H, select the box m with the highest score, and move it from set H to set M; 2. Iterate through the boxes in set H, and calculate the intersection-over-union (IoU) with box m for each box. If the IoU is higher than a certain threshold (usually 0 to 0.5), the box is considered to overlap with m, and the box is removed from set H; 3. Return to step 1 and iterate until set H is empty, and the boxes in set M are the result. In this application, to ensure that the output corresponds to the number of input video frames, for example, for three video frames, the corresponding classification probability values ​​are [0.9, 0.8, 0.1]. As described above, assuming the difference is set to a range of 0 to 0.5, the difference between 0.8 and 0.9 is 0.1, so 0.8 and 0.9 are merged. Since the difference between 0.1 and 0.9 is too large, they are not merged. The result obtained is [0.9, 0.9, 0.1]. Then, it is binarized (that is, the classification probability value obtained after merging is compared with the probability threshold), and the final result is [1, 1, 0]. Based on this result, it can be determined that the first and second frames are scene segmentation frames, and the third frame is a non-scene segmentation frame.

[0075] Regarding the data processing method of this application, taking a video as an example, the specific data processing system framework diagram can be as follows: Figure 4 As shown, the process includes preprocessing, encoding, determining similarity frames, classifier processing, and post-processing. Specifically, it can be summarized into the following steps: First, for the input target video (1600, 25fps), frame sampling is performed (1 frame per second), and features are extracted from it using an encoder to obtain the feature vector for each frame. Second, the similarity between all frames is calculated (64 frames in the figure, resulting in a 64*64 matrix). Then, this similarity matrix is ​​normalized to obtain a 64-dimensional temporal feature sequence. Figure 4The temporal feature sequence shown in the diagram is a 64*512 matrix, where 512 represents the length of a sequence. The length is not fixed; a larger value results in longer processing time, while a smaller value leads to insufficient precision. Therefore, a moderate value is chosen, and this application uses 512. The temporal features are input into a transformer for further feature enhancement, resulting in an enhanced temporal feature sequence. Finally, the enhanced temporal feature sequence undergoes classification and post-processing to obtain the final scene segmentation point detection results.

[0076] This application primarily describes the acquisition of a sequence of video frames to be processed from a target video. This sequence comprises M video frames. Global and local feature extraction is performed on each of the M video frames to obtain a global feature vector and a local feature vector for each video frame. Based on the obtained M global feature vectors, a global similarity matrix is ​​constructed for the M video frames, and based on the obtained M local feature vectors, a local similarity matrix is ​​constructed for the M video frames. The temporal feature sequence corresponding to the sequence of video frames to be processed is determined based on the global and local similarity matrices. Combining global and local features, not only the global similarity between image frames but also the local similarity between image frames is considered. Combining both allows for a more accurate determination of whether image frames share the same scene. Then, temporal enhancement processing is performed on the temporal feature sequence to obtain an enhanced temporal feature sequence, which essentially adds temporal sequence order to the feature sequence, avoiding errors in scene segmentation point recognition due to temporal disorder. Finally, the enhanced temporal feature sequence is further processed to determine the scene segmentation point detection result for each of the M video frames. This method can determine whether each video frame is a scene segmentation frame. Specifically, this application uses multiple similarity matrices, combines local and global features, and adds temporal sequence to the features to make the similarity confirmation between video frames more accurate, thereby improving the accuracy of judging whether a video frame is a scene segmentation frame, which in turn improves the accuracy of video scene segmentation.

[0077] Based on the above embodiments, this application discloses another data processing method. Please refer to [link to relevant documentation]. Figure 5 , Figure 5 This is a flowchart illustrating another data processing method disclosed in an embodiment of this application. This data processing method can be executed by a computer device, and specifically may include steps S501 to S508:

[0078] S501, Obtain the target video.

[0079] S502. Perform equal-interval frame extraction on the target video to obtain a sequence of video frames to be processed, which includes M video frames.

[0080] S503. Perform global feature extraction and local feature extraction on each of the M video frames to obtain the global feature vector and local feature vector of each video frame.

[0081] S504. Based on the obtained M global feature vectors, construct the global similarity matrix of M video frames, and based on the obtained M local feature vectors, construct the local similarity matrix of M video frames.

[0082] S505. Determine the temporal feature sequence corresponding to the video frame sequence to be processed based on the global similarity matrix and the local similarity matrix, and perform temporal enhancement processing on the temporal feature sequence to obtain the enhanced temporal feature sequence.

[0083] S506. Process the enhanced temporal feature sequence to determine the scene segmentation point detection result for each of the M video frames.

[0084] Among them, steps S501 to S506 are Figure 2 The relevant embodiments shown have been described in detail, and will not be repeated here. The foregoing description can be regarded as both model prediction and model training. In this application, during the model training process, image features are not extracted offline and then used as subsequent inputs. Instead, the model is trained as part of a complete training framework. During training, the pre-trained model parameters are modified based on the dataset adjustment to improve accuracy. For a given video frame sequence {I1, I2, ..., In}, a set of feature sequences {F1, F2, ..., Fn} can be obtained. The model training steps are the same as described above. Therefore, based on the aforementioned steps, a data processing model can be determined. This data processing model can process any video to obtain the detection results of whether each video frame after frame extraction is a scene segmentation frame.

[0085] Based on this, this application prepared a dataset containing 10,000 videos of film and television programs and UGC (User Generated Content) under different scene variations, and manually labeled the scene segmentation points. 8,000 videos were used as the training set, and the remaining 2,000 were used for testing. Compared with shot-based methods, offline feature classification methods, and end-to-end methods, the data processing method in this application shows a significant improvement in accuracy, as detailed in Table 1:

[0086] Table 1

[0087] accuracy Recall rate F1 score Lens-based approach 0.7370 0.7157 0.7262 Offline feature classification 0.8478 0.7515 0.7968 end-to-end method 0.9064 0.8135 0.8575 This application 0.9256 0.8442 0.8830

[0088] As shown in Table 1, the method of this application exhibits better performance in terms of accuracy, recall, and F1-score. The F1-score is the harmonic mean of accuracy (P) and recall (R).

[0089] S507. For any video frame among M video frames, obtain the previous video frame from the target video.

[0090] Since this application processes a sequence of video frames after frame extraction, and does not include all video frames of the target video, the scene segmentation point detection results can still be verified.

[0091] In one possible implementation, after determining the scene segmentation point detection results of M video frames, for any video frame among the M video frames, the preceding video frame of any video frame is obtained from the target video. Here, the preceding video frame is immediately adjacent to any video frame and precedes any video frame in time.

[0092] S508. If the scene segmentation point detection result of any video frame indicates that any video frame is a scene segmentation point, and the scene indicated by the previous video frame is inconsistent with that indicated by any video frame, then any video frame is determined to be a scene segmentation point.

[0093] In one possible implementation, after acquiring any video frame, it is first determined whether the video frame is a scene segmentation point based on the scene segmentation point detection result. If so, the scene of the video frame is compared with the scene of the previously acquired video frame. If the scenes are inconsistent, the video frame is determined to be a scene segmentation frame. This increases the accuracy of the scene segmentation point detection result.

[0094] As mentioned above, the data processing method proposed in this application can be applied to various scenarios. Taking tagging as an example, it can specifically include: dividing the target video into multiple video segments based on the scene segmentation point detection results; generating a video tag set for the multiple video segments, with different video tags corresponding to different video segments; receiving a video acquisition request and parsing the request to obtain the requested video tag; retrieving the target video tag that matches the requested video tag from the video tag set and returning the video segment corresponding to the target video tag. The video acquisition request can be a keyword / phrase directly entered by the user during retrieval, and the corresponding target video segment can be directly output based on the keyword / phrase to meet the user's needs.

[0095] In another possible implementation, the data processing method is also used in video editing scenarios. For example, after a user uploads a target video, the computer device determines the scene segmentation point detection results based on the data processing method, and then splits the target video into independent video segments based on the scene segmentation point detection results, and directly outputs multiple independent video segments. Users can perform video montage based on these video segments, select the video segments they like, and discard the video segments that are not good.

[0096] In this embodiment, in addition to determining whether a video frame is a scene segmentation point, the scene segmentation point detection results are verified, and experimental results are demonstrated. This application mainly utilizes an end-to-end training framework and combines similarity matrices of different dimensions to improve the similarity between image frames. Then, temporal enhancement of features can improve the accuracy of determining whether a video frame is a scene segmentation frame, thus improving the accuracy of video scene segmentation. Finally, non-maximum suppression is used to determine the final scene segmentation point detection result, further improving accuracy. Compared to other scene segmentation methods, the data processing method proposed in this application yields scene segmentation point detection results with higher accuracy and wider applicability.

[0097] Based on the above method embodiments, this application also provides a schematic diagram of the structure of a data processing apparatus. See also... Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 6 The data processing device 600 shown can operate the following units:

[0098] The acquisition unit 601 is used to acquire a sequence of video frames to be processed from the target video, wherein the sequence of video frames to be processed includes M video frames, where M is a positive integer greater than 1;

[0099] Processing unit 602 is used for:

[0100] Global feature extraction and local feature extraction are performed on each of the M video frames to obtain the global feature vector and local feature vector of each video frame;

[0101] Based on the obtained M global feature vectors, construct the global similarity matrix of the M video frames, and based on the obtained M local feature vectors, construct the local similarity matrix of the M video frames.

[0102] The temporal feature sequence corresponding to the video frame sequence to be processed is determined based on the global similarity matrix and the local similarity matrix. The temporal feature sequence is then subjected to temporal enhancement processing to obtain the enhanced temporal feature sequence.

[0103] The enhanced temporal feature sequence is processed to determine the scene segmentation point detection result for each of the M video frames.

[0104] In one possible implementation, when the processing unit 602 determines the temporal feature sequence corresponding to the video frame sequence to be processed based on the global similarity matrix and the local similarity matrix, it is specifically used for:

[0105] The global similarity matrix and the local similarity matrix are concatenated to obtain a concatenated similarity matrix;

[0106] The spliced ​​similarity matrix is ​​processed using an inter-frame similarity processing network to obtain a temporal feature sequence; wherein, the temporal feature sequence includes M temporal features, each temporal feature corresponding to one video frame among the M video frames.

[0107] In one possible implementation, when the processing unit 602 constructs the global similarity matrix of the M video frames based on the obtained M global feature vectors, it is specifically used for:

[0108] Calculate the cosine distance and Euclidean distance between the first global feature vector and the second global feature vector, where the first global feature vector is any one of the M global feature vectors and the second global feature vector is any one of the M global feature vectors;

[0109] Based on the cosine distance between the calculated global feature vectors, a first global similarity matrix is ​​determined. Based on the Euclidean distance between the calculated global feature vectors, a second global similarity matrix is ​​determined. The first global similarity matrix and the second global similarity matrix are then used as the global similarity matrix for the M video frames.

[0110] Specifically, when the processing unit 602 constructs the local similarity matrix of the M video frames based on the obtained M local feature vectors, it is used for:

[0111] Calculate the cosine distance and Euclidean distance between the first local feature vector and the second local feature vector, where the first local feature vector is any one of the M local feature vectors and the second local feature vector is any one of the M local feature vectors;

[0112] Based on the cosine distance between the calculated local feature vectors, a first local similarity matrix is ​​determined, and based on the Euclidean distance between the calculated local feature vectors, a second local similarity matrix is ​​determined. The first local similarity matrix and the second local similarity matrix are then used as the local similarity matrices for the M video frames.

[0113] In one possible implementation, when the processing unit 602 concatenates the global similarity matrix and the local similarity matrix to obtain a concatenated similarity matrix, it is specifically used for:

[0114] The first global similarity matrix, the second global similarity matrix, the first local similarity matrix, and the second local similarity matrix are concatenated to obtain a concatenated similarity matrix.

[0115] In one possible implementation, when the processing unit 602 performs time-series enhancement processing on the time-series feature sequence to obtain the enhanced time-series feature sequence, it is specifically used for:

[0116] The temporal feature sequence is subjected to temporal enhancement processing using a temporal enhancement network to obtain an enhanced temporal feature sequence; wherein, the enhanced temporal feature sequence includes M enhanced temporal features, and each enhanced temporal feature corresponds to one video frame among the M video frames;

[0117] When the processing unit 602 processes the enhanced temporal feature sequence to determine the scene segmentation point detection result for each of the M video frames, it specifically performs the following:

[0118] The enhanced temporal feature sequence is processed using a classification network to obtain the classification probability value corresponding to each enhanced temporal feature;

[0119] Merge any two classification probability values ​​whose differences are within a set range from multiple classification probability values, and compare the merged classification probability value with a probability threshold.

[0120] Based on the comparison results, the scene segmentation point detection result of each of the M video frames is determined.

[0121] In one possible implementation, when the acquisition unit 601 acquires the sequence of video frames to be processed from the target video, it is specifically used for:

[0122] Acquire the target video;

[0123] The target video is subjected to equally spaced frame extraction to obtain the video frame sequence to be processed.

[0124] In one possible implementation, the processing unit 602 is further configured to:

[0125] For any video frame among the M video frames, obtain the previous video frame from the target video;

[0126] If the scene segmentation point detection result of any video frame indicates that any video frame is a scene segmentation point, and the scene indicated by the previous video frame is inconsistent with that of any video frame, then any video frame is determined to be a scene segmentation point.

[0127] According to one embodiment of this application, Figure 2 The data processing method shown can involve various steps that are... Figure 6 This is performed by each unit in the data processing apparatus shown. For example, Figure 2 Step S201 in the data processing method shown can be performed by... Figure 6 The acquisition unit 601 in the data processing device shown is used to execute steps S202-S205, which can be performed by... Figure 6 The data processing unit 602 in the shown data processing apparatus performs the execution. For example, Figure 5 Steps S501 and S507 in the data processing method shown can be derived from... Figure 6 The acquisition unit 601 in the data processing device shown executes the steps S502-S506 and S508, which can be performed by... Figure 6 The data processing unit 602 in the data processing device shown executes the operation.

[0128] According to another embodiment of this application, Figure 6 The data processing apparatus shown can be composed of individual or combined units into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0129] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 2 as well as Figure 5 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 6 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computer device via the computer-readable storage medium, and run therein.

[0130] In this embodiment, the acquisition unit 601 acquires a sequence of video frames to be processed from the target video, which includes M video frames. The processing unit 602 performs global feature extraction and local feature extraction on each of the M video frames to obtain a global feature vector and a local feature vector for each video frame. Then, based on the obtained M global feature vectors, a global similarity matrix is ​​constructed for the M video frames, and based on the obtained M local feature vectors, a local similarity matrix is ​​constructed for the M video frames. Finally, the temporal feature sequence corresponding to the sequence of video frames to be processed is determined based on the global similarity matrix and the local similarity matrix. This method combines global and local similarities, considering not only the global similarity between image frames but also their local similarities. This combination allows for a more accurate determination of whether image frames share the same scene. The processing unit 602 then performs temporal enhancement processing on the temporal feature sequence, resulting in an enhanced temporal feature sequence. This enhances the sequence by adding temporal order, preventing errors in scene segmentation point identification due to temporal discrepancies. The processing unit 602 further processes this enhanced temporal feature sequence to determine the scene segmentation point detection result for each of the M video frames. This method can determine whether each video frame is a scene segmentation frame. Specifically, this application uses multiple similarity matrices, combining local and global features, and adding temporal order to the features, making the similarity confirmation between video frames more accurate. This improves the accuracy of determining whether a video frame is a scene segmentation frame, thus enhancing the accuracy of video scene segmentation.

[0131] Based on the above methods and apparatus embodiments, this application provides a computer device, see [link to relevant documentation]. Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 7 The computer device 700 shown includes at least a processor 701, an input interface 702, an output interface 703, a computer storage medium 704, and a memory 705. The processor 701, input interface 702, output interface 703, computer storage medium 704, and memory 705 can be connected via a bus or other means.

[0132] The computer storage medium 704 can be stored in the memory 705 of the computer device 700. The computer storage medium 704 is used to store computer programs, which include program instructions. The processor 701 is used to execute the program instructions stored in the computer storage medium 704. The processor 701 (or CPU (Central Processing Unit)) is the computing and control core of the computer device 700. It is suitable for implementing one or more instructions, specifically for loading and executing one or more computer instructions to achieve corresponding method flows or corresponding functions.

[0133] This application embodiment also provides a computer-readable storage medium (Memory), which can be a memory device in a computer device 700 for storing programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device 700 and extended storage media supported by the computer device 700. The computer-readable storage medium provides storage space that stores the operating system of the computer device 700. Furthermore, this storage space also stores one or more computer programs (including program code) suitable for loading and execution by the processor 701. It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0134] In one embodiment, the computer-readable storage medium may be loaded and executed by processor 701 of one or more computer programs stored in the computer-readable storage medium to achieve the aforementioned related functions. Figure 2 as well as Figure 5 The corresponding steps of the data processing method. In specific implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 701 and executed as follows:

[0135] Obtain the video frame sequence to be processed from the target video, wherein the video frame sequence to be processed includes M video frames, where M is a positive integer greater than 1;

[0136] Global feature extraction and local feature extraction are performed on each of the M video frames to obtain the global feature vector and local feature vector of each video frame;

[0137] Based on the obtained M global feature vectors, construct the global similarity matrix of the M video frames, and based on the obtained M local feature vectors, construct the local similarity matrix of the M video frames.

[0138] The temporal feature sequence corresponding to the video frame sequence to be processed is determined based on the global similarity matrix and the local similarity matrix. The temporal feature sequence is then subjected to temporal enhancement processing to obtain the enhanced temporal feature sequence.

[0139] The enhanced temporal feature sequence is processed to determine the scene segmentation point detection result for each of the M video frames.

[0140] In one possible implementation, when the processor 701 determines the temporal feature sequence corresponding to the video frame sequence to be processed based on the global similarity matrix and the local similarity matrix, it is specifically used for:

[0141] The global similarity matrix and the local similarity matrix are concatenated to obtain a concatenated similarity matrix;

[0142] The spliced ​​similarity matrix is ​​processed using an inter-frame similarity processing network to obtain a temporal feature sequence; wherein, the temporal feature sequence includes M temporal features, each temporal feature corresponding to one video frame among the M video frames.

[0143] In one possible implementation, when the processor 701 constructs the global similarity matrix of the M video frames based on the obtained M global feature vectors, it is specifically used for:

[0144] Calculate the cosine distance and Euclidean distance between the first global feature vector and the second global feature vector, where the first global feature vector is any one of the M global feature vectors and the second global feature vector is any one of the M global feature vectors;

[0145] Based on the cosine distance between the calculated global feature vectors, a first global similarity matrix is ​​determined. Based on the Euclidean distance between the calculated global feature vectors, a second global similarity matrix is ​​determined. The first global similarity matrix and the second global similarity matrix are then used as the global similarity matrix for the M video frames.

[0146] Specifically, when the processor 701 constructs the local similarity matrix of the M video frames based on the obtained M local feature vectors, it is used for:

[0147] Calculate the cosine distance and Euclidean distance between the first local feature vector and the second local feature vector, where the first local feature vector is any one of the M local feature vectors and the second local feature vector is any one of the M local feature vectors;

[0148] Based on the cosine distance between the calculated local feature vectors, a first local similarity matrix is ​​determined, and based on the Euclidean distance between the calculated local feature vectors, a second local similarity matrix is ​​determined. The first local similarity matrix and the second local similarity matrix are then used as the local similarity matrices for the M video frames.

[0149] In one possible implementation, the processor 701 concatenates the global similarity matrix and the local similarity matrix to obtain a concatenated similarity matrix, specifically for the following purposes:

[0150] The first global similarity matrix, the second global similarity matrix, the first local similarity matrix, and the second local similarity matrix are concatenated to obtain a concatenated similarity matrix.

[0151] In one possible implementation, when the processor 701 performs time-series enhancement processing on the time-series feature sequence to obtain the enhanced time-series feature sequence, it is specifically used for:

[0152] The temporal feature sequence is subjected to temporal enhancement processing using a temporal enhancement network to obtain an enhanced temporal feature sequence; wherein, the enhanced temporal feature sequence includes M enhanced temporal features, and each enhanced temporal feature corresponds to one video frame among the M video frames;

[0153] When the processor 701 processes the enhanced temporal feature sequence to determine the scene segmentation point detection result for each of the M video frames, it specifically performs the following operations:

[0154] The enhanced temporal feature sequence is processed using a classification network to obtain the classification probability value corresponding to each enhanced temporal feature;

[0155] Merge any two classification probability values ​​whose differences are within a set range from multiple classification probability values, and compare the merged classification probability value with a probability threshold.

[0156] Based on the comparison results, the scene segmentation point detection result of each of the M video frames is determined.

[0157] In one possible implementation, when the processor 701 acquires the sequence of video frames to be processed from the target video, it is specifically used for:

[0158] The target video is acquired; the target video is subjected to equal-interval frame extraction to obtain the video frame sequence to be processed.

[0159] In one possible implementation, the processor 701 is further configured to:

[0160] For any video frame among the M video frames, obtain the previous video frame from the target video; if the scene segmentation point detection result of any video frame indicates that the video frame is a scene segmentation point, and the scene indicated by the previous video frame is inconsistent with that indicated by the video frame, then determine that the video frame is a scene segmentation point.

[0161] According to one aspect of this application, an embodiment of this application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor 701 reads the computer program from the computer-readable storage medium and executes the computer program, causing a computer device 700 to perform... Figure 2 as well as Figure 5 Data processing methods.

[0162] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the video frame sequence to be processed from the target video, wherein the video frame sequence to be processed includes M video frames, where M is a positive integer greater than 1; Global feature extraction and local feature extraction are performed on each of the M video frames to obtain the global feature vector and local feature vector of each video frame; Based on the obtained M global feature vectors, construct the global similarity matrix of the M video frames, and based on the obtained M local feature vectors, construct the local similarity matrix of the M video frames. The global similarity matrix and the local similarity matrix are concatenated to obtain a concatenated similarity matrix. An inter-frame similarity processing network is used to process the concatenated similarity matrix to obtain a temporal feature sequence. A temporal enhancement network is then used to perform temporal enhancement processing on the temporal feature sequence to obtain an enhanced temporal feature sequence. The enhanced temporal feature sequence includes M enhanced temporal features, each corresponding to one of the M video frames. The temporal feature sequence is used to indicate the similarity and temporal sequence among the M video frames. The enhanced temporal feature sequence is processed using a classification network to obtain the classification probability value corresponding to each enhanced temporal feature; any two classification probability values ​​whose difference is within a set range are merged, and the merged classification probability value is compared with a probability threshold; the scene segmentation point detection result of each of the M video frames is determined based on the comparison result. For any video frame among the M video frames, obtain the previous video frame from the target video. If the scene segmentation point detection result of any video frame indicates that the video frame is a scene segmentation point, and the scene indicated by the previous video frame is inconsistent with that indicated by the video frame, then determine that the video frame is a scene segmentation point.

2. The method according to claim 1, characterized in that, The step of constructing the global similarity matrix of the M video frames based on the obtained M global feature vectors includes: Calculate the cosine distance and Euclidean distance between the first global feature vector and the second global feature vector, where the first global feature vector is any one of the M global feature vectors and the second global feature vector is any one of the M global feature vectors; Based on the cosine distance between the calculated global feature vectors, a first global similarity matrix is ​​determined, and based on the Euclidean distance between the calculated global feature vectors, a second global similarity matrix is ​​determined. The first global similarity matrix and the second global similarity matrix are then used as the global similarity matrix for the M video frames. The step of constructing the local similarity matrix of the M video frames based on the obtained M local feature vectors includes: Calculate the cosine distance and Euclidean distance between the first local feature vector and the second local feature vector, where the first local feature vector is any one of the M local feature vectors and the second local feature vector is any one of the M local feature vectors; Based on the cosine distance between the calculated local feature vectors, a first local similarity matrix is ​​determined, and based on the Euclidean distance between the calculated local feature vectors, a second local similarity matrix is ​​determined. The first local similarity matrix and the second local similarity matrix are then used as the local similarity matrices for the M video frames.

3. The method according to claim 2, characterized in that, The step of concatenating the global similarity matrix and the local similarity matrix to obtain a concatenated similarity matrix includes: The first global similarity matrix, the second global similarity matrix, the first local similarity matrix, and the second local similarity matrix are concatenated to obtain a concatenated similarity matrix.

4. The method according to claim 1, characterized in that, The process of obtaining the video frame sequence to be processed from the target video includes: Acquire the target video; The target video is subjected to equally spaced frame extraction to obtain the video frame sequence to be processed.

5. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire a sequence of video frames to be processed from the target video, wherein the sequence of video frames to be processed includes M video frames, where M is a positive integer greater than 1; Processing unit, used for: Global feature extraction and local feature extraction are performed on each of the M video frames to obtain the global feature vector and local feature vector of each video frame; Based on the obtained M global feature vectors, construct the global similarity matrix of the M video frames, and based on the obtained M local feature vectors, construct the local similarity matrix of the M video frames. The global similarity matrix and the local similarity matrix are concatenated to obtain a concatenated similarity matrix. An inter-frame similarity processing network is used to process the concatenated similarity matrix to obtain a temporal feature sequence. A temporal enhancement network is then used to perform temporal enhancement processing on the temporal feature sequence to obtain an enhanced temporal feature sequence. The enhanced temporal feature sequence includes M enhanced temporal features, each corresponding to one of the M video frames. The temporal feature sequence is used to indicate the similarity and temporal sequence among the M video frames. The enhanced temporal feature sequence is processed using a classification network to obtain the classification probability value corresponding to each enhanced temporal feature; any two classification probability values ​​whose difference is within a set range are merged, and the merged classification probability value is compared with a probability threshold; the scene segmentation point detection result of each of the M video frames is determined based on the comparison result. For any video frame among the M video frames, obtain the previous video frame from the target video. If the scene segmentation point detection result of any video frame indicates that the video frame is a scene segmentation point, and the scene indicated by the previous video frame is inconsistent with that indicated by the video frame, then determine that the video frame is a scene segmentation point.

6. A computer device, characterized in that, The computer device includes an input interface and an output interface, and the computer device further includes: A processor, suitable for implementing one or more computer programs; and, A computer-readable storage medium storing one or more computer programs adapted to be loaded by the processor and executed as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more computer programs, which are adapted to be loaded by a processor and executed as described in any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, a processor of a computer device reading the computer program from the computer-readable storage medium, and the processor executing the computer program to cause the computer device to perform the data processing method as described in any one of claims 1-4.

Images