Video scene segmentation method, device and equipment and computer readable storage medium

Abstract

The application provides a video scene segmentation method and device, equipment and computer readable storage medium; it relates to artificial intelligence technology, and the method comprises: frame extraction is carried out to the video to be handled, and a video frame set is obtained;And the feature extraction is carried out to each video frame in the video frame set, and the feature vector corresponding to each video frame is obtained;Based on the feature vector, the interframe similarity matrix is constructed;The interframe similarity matrix represents the similarity between video frames;Scene change prediction is carried out based on the interframe similarity matrix, and the scene segmentation frame is determined from the video frame set;Based on the scene segmentation frame, the scene segmentation is carried out to the video to be handled, and the scene segmentation result is obtained. Through the application, the accuracy of video scene segmentation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence technology, and in particular to a video scene segmentation method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] At present, for the video scene segmentation method, the related technology usually first segments the video through a video segmentation algorithm; then clusters the video segments according to feature similarity, uses a neural network to extract the features of the video segments, and learns the features of the video segments, relying on the assumption that the video segments in the same scene are more similar, clusters the features of the video segments, and gathers the segments in the same scene together as a video scene, thereby realizing the scene segmentation of the video. It can be seen that the method of the related technology usually needs to introduce a special algorithm model to obtain the video segments, thereby causing additional processing cost and reducing the efficiency of scene segmentation; and the accuracy of scene segmentation is easily affected by the video quality of the video segments, thereby reducing the accuracy of video scene segmentation. SUMMARY

[0003] The embodiments of the present application provide a video scene segmentation method, device, equipment and computer readable storage medium, which can improve the accuracy of video scene segmentation.

[0004] The technical solutions of the embodiments of the present application are as follows:

[0005] The embodiments of the present application provide a video scene segmentation method, comprising:

[0006] Frame extraction is performed on the to-be-processed video to obtain a video frame set; and feature extraction is performed on each video frame in the video frame set to obtain a feature vector corresponding to each video frame;

[0007] Based on the feature vector, an inter-frame similarity matrix is constructed; the inter-frame similarity matrix represents the similarity between video frames;

[0008] Scene change prediction is performed based on the inter-frame similarity matrix, and a scene segmentation frame is determined from the video frame set;

[0009] Scene segmentation is performed on the to-be-processed video based on the scene segmentation frame to obtain a scene segmentation result.

[0010] The embodiments of the present application provide a video scene segmentation device, comprising:

[0011] The extraction module is configured to extract frames from the to-be-processed video to obtain a video frame set; and extract features from each video frame in the video frame set to obtain a feature vector corresponding to each video frame;

[0012] a calculating module configured to construct an inter-frame similarity matrix based on the feature vector; the inter-frame similarity matrix representing a similarity degree between video frames;

[0013] a predicting module configured to perform scene change prediction based on the inter-frame similarity matrix to determine a scene split frame from the set of video frames;

[0014] a splitting module configured to perform scene splitting on the video to be processed based on the scene split frame to obtain a scene splitting result.

[0015] In the apparatus, the extracting module is further configured to perform feature extraction on each video frame in at least one mode to obtain at least one single feature corresponding to each video frame; and perform feature fusion on the at least one single feature to obtain a feature vector corresponding to each video frame.

[0016] In the apparatus, the extracting module is further configured to map the at least one single feature to a unified feature dimension to obtain at least one mapped single feature; perform feature splicing and average pooling processing on the at least one mapped single feature to obtain an average feature; calculate a fusion weight corresponding to each mapped single feature in the at least one mapped single feature based on an attention mechanism according to the average feature; and perform weighting and splicing on the at least one mapped single feature according to the fusion weight corresponding to each mapped single feature to obtain the feature vector.

[0017] In the apparatus, the calculating module is further configured to calculate a distance between the feature vector of each video frame and the feature vectors of all video frames in the set of video frames to obtain at least one feature distance corresponding to each video frame; construct a feature distance matrix according to the at least one feature distance corresponding to each video frame; and perform normalization processing on the feature distance matrix to obtain the inter-frame similarity matrix.

[0018] In the apparatus, the predicting module is further configured to perform convolution processing and classification prediction on the inter-frame similarity matrix by using a scene split detection network to obtain a scene split probability corresponding to each video frame; and determine the scene split frame according to the scene split probability.

[0019] In the apparatus, the scene split detection network comprises a classification module and at least two convolution modules; and the predicting module is further configured to perform multi-layer convolution processing on the inter-frame similarity matrix by using the at least two convolution modules to obtain a frame embedding vector; and perform classification prediction based on the frame embedding vector by using the classification module to obtain the scene split probability corresponding to each video frame.

[0020] In the apparatus, the at least two convolution modules include a first convolution module and at least one second convolution module, the corresponding feature dimension of the first convolution module is higher than the corresponding feature dimension of the second convolution module; the prediction module is further configured to: perform convolution processing on the inter-frame similarity matrix by using the first convolution module to obtain a multi-dimensional feature vector corresponding to each matrix element in the inter-frame similarity matrix, thereby obtaining a first feature matrix; perform dimension transformation on the first feature matrix to obtain a second feature matrix; and perform convolution processing on the second feature matrix by using the at least one second convolution module to obtain the frame embedding vector.

[0021] In the apparatus, the segmentation module is further configured to: obtain a segmentation time point corresponding to the scene segmentation frame in the to-be-processed video; perform frame extraction on the to-be-processed video according to a preset time range containing the segmentation time point to obtain an updated video frame set; the frame extraction frequency of the updated video frame set is higher than that of the video frame set; determine an updated segmentation frame corresponding to the scene segmentation frame from the updated video frame set according to the pixel color value contained in each updated video frame of the updated video frame set; and perform scene segmentation on the to-be-processed video according to the updated segmentation frame to obtain a scene segmentation result.

[0022] In the apparatus, the segmentation module is further configured to: calculate a pixel color distance between each updated video frame and two adjacent updated video frames corresponding to the updated video frame; and determine the updated segmentation frame from the updated video frame set according to the pixel color distance.

[0023] In the apparatus, the segmentation module is further configured to: before the calculation of the pixel color distance between each updated video frame and two adjacent updated video frames corresponding to the updated video frame, convert the pixel color space of each updated video frame into a perceptual color space; and the spatial model of the perceptual color space is represented by hue, saturation and brightness.

[0024] In the apparatus, the segmentation module is further configured to: perform mean value processing on the pixel color distance as the color change degree corresponding to each video frame; and determine, in the updated scene video frame set, an updated video frame corresponding to the maximum color change degree as the updated scene segmentation frame.

[0025] An electronic device is provided in an embodiment of the present application, and the electronic device includes:

[0026] a memory configured to store executable instructions;

[0027] a processor configured to execute the executable instructions stored in the memory to implement the video scene segmentation method as described above.

[0028] The embodiment of the present application provides a computer readable storage medium, which stores executable instructions, and the executable instructions are executed by a processor to realize the video scene segmentation method provided by the embodiment of the present application.

[0029] The embodiment of the present application provides a computer program product, which comprises a computer program or instructions, and the computer program or instructions are executed by a processor to realize the video scene segmentation method provided by the embodiment of the present application.

[0030] The embodiment of the present application has the following beneficial effects:

[0031] The embodiment of the present application extracts features at a frame level, records similarity information between video frames by constructing an inter-frame similarity matrix, and then performs scene change prediction based on the inter-frame similarity matrix to obtain scene segmentation frames, so that the scene segmentation problem based on dividing video segments and clustering in the related art is converted into a detection problem of scene segmentation frames, thereby reducing the influence of poor video segment quality on scene segmentation accuracy and improving the accuracy of scene segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is an optional structural schematic diagram of a video scene segmentation system architecture provided by the embodiment of the present application;

[0033] Figure 2 is an optional structural schematic diagram of a video scene segmentation device provided by the embodiment of the present application;

[0034] Figure 3 is an optional flow schematic diagram of a video scene segmentation method provided by the embodiment of the present application;

[0035] Figure 4 is an optional flow schematic diagram of a video scene segmentation method provided by the embodiment of the present application;

[0036] Figure 5 is an optional flow schematic diagram of a video scene segmentation method provided by the embodiment of the present application;

[0037] Figure 6 is an optional flow schematic diagram of a video scene segmentation method provided by the embodiment of the present application;

[0038] Figure 7 is an optional flow schematic diagram of a video scene segmentation method provided by the embodiment of the present application;

[0039] Figure 8is an optional structural schematic diagram of a scene segmentation detection network provided by an embodiment of the present application;

[0040] Figure 9 is an optional flow schematic diagram of a video scene segmentation method provided by an embodiment of the present application;

[0041] Figure 10 is an optional flow schematic diagram of a video scene segmentation method provided by an embodiment of the present application;

[0042] Figure 11 is an optional flow schematic diagram of a video scene segmentation method in an actual application scenario provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0044] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0045] In the following description, the terms "first\second\third" are only to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0047] The related data collection processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0048] Before the embodiments of the present application are further described in detail, the terms and terms related to the embodiments of the present application are explained, and the terms and terms related to the embodiments of the present application are applicable to the following explanations.

[0049] 1) Artificial Intelligence (AI) is the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0050] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, automatic driving, intelligent transportation, etc.

[0051] 2) Computer Vision (CV) Computer vision is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics so that the computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, etc. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0052] 3) Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and example-based learning technologies.

[0053] With the research and progress of artificial intelligence technology, artificial intelligence technology is researched and applied in many fields, such as common smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned driving, automatic driving, unmanned aerial vehicle, robot, smart medical treatment, smart customer service, Internet of vehicles, automatic driving, intelligent transportation and the like. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important value.

[0054] The scheme provided by the embodiments of the present application relates to computer vision technology of artificial intelligence, which is specifically explained by the following embodiments.

[0055] The embodiments of the present application provide a video scene segmentation method, device and equipment and computer readable storage medium, which can improve the accuracy of video scene segmentation. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a smart television, a set-top box, a smart vehicle and intelligent transportation device, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device and various types of terminals, but is not limited thereto; it can also be implemented as a server. When the electronic device is implemented as a server, it can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms, etc. The embodiments of the present application are not limited. The following will illustrate an exemplary application when the electronic device is implemented as a server.

[0056] Referring to Figure 3 , Figure 3 is an optional architecture schematic diagram of a video scene segmentation system 100 provided by the embodiments of the present application, which includes a terminal 400 and a server 200. The terminal 400 is connected to the server 200 through a network 300, which can be a wide area network or a local area network, or a combination of the two.

[0057] The terminal 400 is configured to upload a video to be processed to the server 200 through a video scene segmentation application 410.

[0058] The server 200 is configured to receive the to-be-processed video uploaded by the terminal 400, frame extraction is performed on the to-be-processed video to obtain a video frame set, feature extraction is performed on each video frame in the video frame set to obtain a feature vector corresponding to each video frame, an inter-frame similarity matrix is constructed based on the feature vectors, the inter-frame similarity matrix represents the similarity between each two frames in the video frame set, scene change prediction is performed based on the inter-frame similarity matrix, and a scene segmentation frame is determined from the video frame set; the scene segmentation point is a video frame in the video frame set located between different scenes; and scene segmentation is performed on the to-be-processed video according to the scene segmentation frame to obtain a scene segmentation result.

[0059] The server 200 is further configured to send the scene segmentation result to the terminal 400, so that the terminal 400 performs video content production and the like based on the scene segmentation result through the video scene segmentation application 410.

[0060] Referring to Figure 2 , Figure 2 is a structural schematic diagram of the server 200 provided in the embodiments of the present application, Figure 2 The server 200 shown in FIG. 1 includes at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the server 200 are coupled together by a bus system 240. It can be understood that the bus system 240 is used to realize the connection and communication between the components. The bus system 240 includes a data bus, a power bus, a control bus, and a status signal bus in addition to the data bus. However, for the purpose of clear illustration, all the buses are marked as the bus system 240 in Figure 2 .

[0061] The processor 210 can be an integrated circuit chip having a signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0062] The user interface 230 includes one or more output devices 231 enabling presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0063] The memory 250 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 250 optionally includes one or more storage devices remotely located from the processor 210.

[0064] The memory 250 includes volatile memory or nonvolatile memory, and can also include both volatile and nonvolatile memory. Nonvolatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, the memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0066] The operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0067] The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 420 include Bluetooth, wireless compatibility certification (Wi-Fi), and universal serial bus (USB, Universal Serial Bus), and the like;

[0068] The presentation module 253 is used to enable the presentation of information via one or more output devices 231 associated with the user interface 230 (e.g., display screen, speaker, and the like) (e.g., user interface for operating peripheral devices and displaying content and information);

[0069] The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0070] In some embodiments, the device provided by the embodiments of the present application can be realized in software, Figure 2 A video scene segmentation device 255 stored in the memory 250 is shown, which can be software in the form of programs and plug-ins, including the following software modules: extraction module 2551, calculation module 2552, prediction module 2553 and segmentation module 2554, these modules are logical, so they can be combined or further split according to the functions implemented.

[0071] The functions of the various modules will be described below.

[0072] In some embodiments, the device provided by the embodiments of the present application can be implemented in a hardware manner. For example, the device provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video scene segmentation method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0073] The video scene segmentation method provided by the embodiments of the present application will be described in conjunction with an exemplary application and implementation of the server provided by the embodiments of the present application. In addition, the video scene segmentation method provided by the embodiments of the present application can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation, and vehicle-mounted devices.

[0074] Referring to Figure 3 , Figure 3 is an optional flowchart of the video scene segmentation method provided by the embodiments of the present application. The steps shown in Figure 3 will be described.

[0075] In S101, a frame is extracted from a to-be-processed video to obtain a video frame set, and a feature is extracted from each video frame in the video frame set to obtain a feature vector corresponding to each video frame.

[0076] The video scene segmentation method provided by the embodiments of the present application can be applied to scenes such as video splitting and video key scene extraction to meet the needs of video content machine creation and material retrieval. Alternatively, the video scene segmentation method provided by the embodiments of the present application can also be applied to the scene of generating video tags. Since most of the current video tags are for the entire video, there is a lack of fine-grained tags for different scenes in the video. Based on the video scene segmentation method provided by the embodiments of the present application, the granularity of the video tag can be refined from the video level to the scene level, and the semantic analysis degree of the video can be deepened. It can also be used in other application scenarios of scene segmentation of videos based on artificial intelligence. The specific selection is based on the actual situation, and the embodiments of the present application are not limited.

[0077] In the embodiments of the present application, the server can frame extraction on the to-be-processed video, and split the to-be-processed video into a video frame set. It can be understood that the more dense the video frame extraction is, the more information in the to-be-processed video can be completely retained. In actual application, considering the balance between precision and time and computing resources, sparse sampling can be adopted to frame extraction on the to-be-processed video. Exemplarily, one frame per second can be adopted to frame extraction on the to-be-processed video. That is, for a to-be-processed video with a time length of T seconds, a video frame set containing T video frames is obtained . Wherein, represents a single video frame.

[0078] In the embodiments of the present application, the server extracts features from each video frame in the video frame set to obtain a feature vector corresponding to each video frame. Here, the server can extract features from the video frame through image feature extraction algorithms such as Scale invariant feature Transform (SIFT), Histogram of Oriented Gradient (HOG), etc.; or through neural networks based on deep machine learning such as Convolutional Neural Networks (CNN), etc. The specific selection is made according to the actual situation, and the embodiments of the present application are not limited.

[0079] Here, the server can extract a single feature corresponding to each video frame as a feature vector through a feature extraction method when performing feature extraction, or can extract multiple single feature vectors corresponding to each video frame and combine them to obtain a feature vector, which will be described in the following embodiments.

[0080] S102, based on the feature vector, constructing an inter-frame similarity matrix; the inter-frame similarity matrix represents the similarity between video frames.

[0081] In the embodiments of the present application, the server can calculate the similarity between each video frame in the video frame set based on the feature vector of each video frame, thereby constructing an inter-frame similarity matrix. The matrix elements in the inter-frame similarity matrix can represent the similarity between any two frames in the video frame set. Since the similarity between two frames in the same scene is higher than the similarity between two frames in different scenes, the inter-frame similarity matrix can be used to predict the scene segmentation point, i.e. the scene segmentation frame, between different scenes.

[0082] In some embodiments, based on Figure 3 For example, Figure 4As shown, S102 can be implemented by performing S1021-S1023, which will be described in conjunction with the respective steps.

[0083] S1021, respectively calculate the distance between each video frame and the feature vector of all video frames in the video frame set, to obtain at least one feature distance corresponding to each video frame.

[0084] In the embodiments of the present application, for each video frame in the video frame set, the server can respectively calculate the distance between the video frame and the feature vector of all video frames in the video frame set, to obtain at least one feature distance corresponding to the video frame.

[0085] Here, the server can obtain the feature distance in the form of calculating the cosine distance, or use other distance calculation methods, such as two-norm distance, etc., which is selected according to the actual situation, and the embodiments of the present application are not limited.

[0086] S1022, according to at least one feature distance corresponding to each video frame, a feature distance matrix is constructed.

[0087] In the embodiments of the present application, based on the at least one feature distance corresponding to each video frame obtained above, the server can construct the at least one feature distance corresponding to each video frame in the video frame set into a matrix form, to obtain a feature distance matrix. Wherein, the at least one feature distance corresponding to each video frame can correspond to a row or a column in the feature distance matrix, and the specific construction form is not limited in the embodiments of the present application.

[0088] In some embodiments, the corresponding feature vector set in the video frame set can be represented as The server can calculate at least one feature distance corresponding to each video frame by formula (1), and construct a feature distance matrix, as follows:

[0089] (1)

[0090] In formula (1), and respectively represent the feature vector of the i-th frame and the j-th frame in the video frame set; represents a scale control factor, used to adjust the scale of the feature distance, to avoid and are close, and the distance value calculated by directly calculating the distance, such as calculating the two-norm distance, is small, which makes it difficult for the subsequent neural network model to process the too small value. represents the feature distance of the i-th frame and the j-th frame, represents the feature distance matrix, wherein .

[0091] S1023. Normalize the feature distance matrix to obtain the inter-frame similarity matrix.

[0092] In this embodiment of the application, the server can perform feature distance matrix analysis. Normalization is performed to map each feature distance in the feature distance matrix to the same value range, resulting in the inter-frame similarity matrix.

[0093] In some embodiments, the server can normalize the feature distance matrix using formula (2) to obtain the inter-frame similarity matrix, as follows:

[0094] (2)

[0095] Here, formula (2) is the exponential normalization function. Represents the i-th frame in the feature distance matrix At least one feature distance corresponding to it. ; The server applies an exponential normalization function to the feature distance matrix. After normalization, the resulting inter-frame similarity matrix... .

[0096] In some embodiments, the server may also employ other normalization methods, such as linear normalization, mean normalization, or variance normalization, to process the feature distance matrix to obtain the inter-frame similarity matrix. The specific method chosen depends on the actual situation, and this application embodiment does not impose limitations.

[0097] S103. Based on the inter-frame similarity matrix, predict scene changes and determine the scene segmentation frames from the video frame set.

[0098] In this embodiment of the application, since the inter-frame similarity matrix represents the degree of similarity between any two frames in the video frame set, the server can infer the context relationship between each two frames based on the inter-frame similarity matrix, thereby predicting whether the video frames belong to the same scene, and then determining the scene segmentation frame from the video frame set.

[0099] In this embodiment, the server can construct an AI-based scene segmentation point detection network to process inter-frame similarity, predict and output the probability that each video frame corresponds to a scene segmentation frame, thereby obtaining scene segmentation frames. In some embodiments, based on Figure 3 or Figure 4 ,like Figure 5 As shown, S103 can be achieved by executing S1031-S1032, and each step will be explained in detail.

[0100] S1031. Using a scene segmentation detection network, perform convolution processing and classification prediction on the inter-frame similarity matrix to obtain the scene segmentation probability corresponding to each video frame.

[0101] In this embodiment, the server can input a similarity matrix containing the feature distances between video frames into a pre-trained scene segmentation and detection network. The scene segmentation and detection network then performs multi-layer convolution processing on the inter-frame similarity matrix to combine the adjacent video frames of each video frame. Based on the feature distance, the correlation between video frames is automatically learned, and the changes between the current video frame and the preceding and following video frames are compared. Based on the learned feature information, classification and prediction are performed to obtain the probability score of each video frame as a scene segmentation frame, which is used as the scene segmentation probability.

[0102] In some embodiments, the scene segmentation and detection network includes: a classification module and at least two convolutional modules; based on Figure 5 ,like Figure 6 As shown, S1031 can be implemented through S201-S202, which will be explained in conjunction with each step.

[0103] S201. Perform multi-layer convolution processing on the inter-frame similarity matrix through at least two convolution modules to obtain the frame embedding vector; the frame embedding vector contains the context information corresponding to each video frame.

[0104] In this embodiment, the server can perform multi-layer convolution processing on the inter-frame similarity matrix using at least two convolutional modules. Each matrix element representing the feature distance in the inter-frame similarity matrix is ​​used as local information. Pre-trained weights in the convolutional modules are then used to perform layer-by-layer mapping and function fitting to obtain a feature vector representing the global relevance of each video frame, which serves as the frame embedding vector. Here, the frame embedding vector contains globally relevant contextual information corresponding to each video frame.

[0105] In some embodiments, at least two convolutional modules include: a first convolutional module and at least one second convolutional module, wherein the feature dimension corresponding to the first convolutional module is higher than the feature dimension corresponding to the second convolutional module; based on Figure 6 ,like Figure 7 As shown, S201 can be implemented through S2011-S2013, which will be explained in conjunction with each step.

[0106] S2011. The first convolution module performs convolution processing on the inter-frame similarity matrix to obtain the multi-dimensional feature vector corresponding to each matrix element in the inter-frame similarity matrix, thereby obtaining the first feature matrix.

[0107] In S2011, the server can perform convolution processing on the inter-frame similarity matrix by using the first convolution module to obtain a multi-dimensional feature vector corresponding to each matrix element in the inter-frame similarity matrix, thereby obtaining a first feature matrix.

[0108] In some embodiments, the first convolution module can be a two-dimensional convolution module. The server can perform two-dimensional convolution processing on the inter-frame similarity matrix with dimensions of by using the first convolution module to obtain a D-dimensional feature vector corresponding to each matrix element, thereby obtaining a first feature matrix with dimensions of where D is a positive integer greater than or equal to 1.

[0109] In S2012, the server can perform dimension transformation on the first feature matrix to obtain a second feature matrix.

[0110] In S2012, the server can perform dimension transformation on the first feature matrix by using dimension merging or stretching, or the like, to transform the dimension of the first feature matrix to a feature dimension that can be processed by the second convolution module, thereby obtaining the second feature matrix.

[0111] In some embodiments, the server can transform the dimension of the first feature matrix with dimensions of to dimensions of , and treat as a feature dimension , to transform the dimension of the first feature matrix, thereby obtaining the second feature matrix with dimensions of

[0112] In S2013, the server can perform convolution processing on the second feature matrix by using at least one second convolution module to obtain a frame embedding vector.

[0113] In the embodiments of the present application, the server can perform convolution processing on the second feature matrix by using at least one second convolution module on a low-dimensional space to obtain an embedding vector corresponding to each video frame as the frame embedding vector.

[0114] In some embodiments, the second convolution module can be a one-dimensional convolution module. The server can perform convolution processing on the second feature matrix with dimensions of by using at least one second convolution module to obtain an embedding vector corresponding to each video frame in the T video frames, i.e., the frame embedding vector.

[0115] In S202, the server can perform classification prediction based on the frame embedding vector by using a classification module to obtain a scene segmentation probability corresponding to each video frame.

[0116] ​​In this embodiment of the application, the classification module can be a neural network module with classification function. The server can use the classification module to perform classification prediction on the embedding vector corresponding to each video frame in the frame embedding vector to obtain the probability score of each video frame as a scene segmentation frame, which is used as the scene segmentation probability.

[0117] In some embodiments, such as Figure 8 As shown, the scene segmentation and detection network can include a first convolutional module 80, a dimension transformation module 81, and four second convolutional modules 82. Figure 8 The diagram shows the second convolutional modules 82-1 to 82-4 and the classification module 83. Each convolutional module, including the first and second convolutional modules, contains a set of convolutional layers, activation layers, and pooling layers. The pooling layer output of the first convolutional module 80 is connected to a dimension transformation module 81. The dimension transformation module 81 transforms the dimension of a feature matrix output from the pooling layer of the first convolutional module 80 through methods such as dimension merging or stretching, transforming its dimension to the feature dimension that the second convolutional module can process, thus obtaining the second feature matrix. Figure 8 As shown, the output of the dimension transformation module 81 is connected to the input of the convolutional layer of the second convolutional module 82-1, and the output of the pooling layer of the second convolutional module 82-1 is connected to the input of the convolutional layer of the second convolutional module 82-2 (not shown in the diagram)... The four second convolutional modules 82 are connected sequentially, and the output of the last second convolutional module 82-4 is connected to the classification module 83. The classification module 83 includes a pooling layer and a classification layer. From the... The convolutional module to the th The convolution processing of the convolution module of the layer can be as shown in formula (3), as follows:

[0118] (3)

[0119] In formula (3), Indicates the first The output of each convolutional module, For the first The output of the convolutional layer in +1 convolutional module, Indicates the first The convolution kernel parameter matrix of the k-th filter in +1 convolutional modules Indicates the first The bias vector of the kth filter in +1 convolutional modules; Representing the The output of the activation function in +1 convolutional module, where max is the maximum value operation; Representing the +1 output of a convolutional module, This refers to the pooling operation of the pooling layer.

[0120] Figure 8 In some embodiments, the classification layer can be configured to perform classification prediction by using a binary classifier with the pooling layer and the classification layer. In some embodiments, the process of classification prediction can be shown in Equation (4) as follows:

[0121] (4)

[0122] wherein, pi represents the probability of the i-th frame being a scene split point, i.e., the scene split probability of the i-th frame, and w represents the network parameter of the classification layer in the classification module.

[0123] In some embodiments, the server can first construct an initial scene split detection network, train the initial scene split detection network by using a labeled data set, and obtain the scene split detection network. Exemplarily, the labeled data set can use a data set containing 10,000 videos, including 5,000 training videos and 5,000 test videos. Each video contains 2 to 6 scenes, and the scene split frame corresponds to a video dense frame level label. The server inputs the inter-frame similarity matrix M into the first convolution module in the initial scene split detection network, i.e., , and obtains the output of the last classification layer by layer-by-layer forward propagation, wherein, pi represents the probability of the i-th frame being a scene split point. The server can use as a loss function, optimize the loss function by using an error back propagation algorithm, and obtain the trained scene split detection network until a preset training target is reached. Then, the server can use the trained scene split detection network to predict the probability of each video frame being a scene split frame, i.e., the scene split probability corresponding to each video frame.

[0124] S1032. Determine the scene split frame according to the scene split probability.

[0125] In the embodiments of the present application, the scene split probability represents the probability of each video frame being a scene split frame. The server can determine the scene split frame from the video frame set according to a pre-set threshold combined with the scene split probability, or sort the scene split frames and determine the video frame with a larger scene split probability as the scene split frame. The actual situation is selected, and the embodiments of the present application are not limited.

[0126] In the embodiments of the present application, the time accuracy of the scene split frame corresponds to the sampling rate of the frame extraction processing in S101. Exemplarily, if the sampling rate of the video frame obtained by the frame extraction in S101 is 1 frame per second, the time accuracy of the scene split frame is 1 second.

[0127] S104, scene segmentation is performed on the video to be processed based on the scene segmentation frame to obtain a scene segmentation result.

[0128] In the embodiments of the present application, the server can directly perform scene segmentation on the video to be processed based on the scene segmentation frame to obtain video clips corresponding to different scenes in the video to be processed as the scene segmentation result.

[0129] In some embodiments, the server can also perform scene segmentation on the video to be processed based on the scene segmentation frame in combination with video information in other dimensions in the video to be processed, such as text information, audio information, etc., to obtain the scene segmentation result.

[0130] By way of example, the text information described above can be iconic text information contained in the video to be processed. The server can adjust the preliminary segmentation result corresponding to the scene segmentation frame based on the scene segmentation frame in combination with iconic text information in the video to be processed, such as building iconic text, background decoration text, etc. in the main picture of the video to be processed, and take video frames containing the same text information as video clips in the same scene to obtain the scene segmentation result.

[0131] By way of example, the text information described above can also be subtitle information contained in the video to be processed. The server can adjust the preliminary segmentation result corresponding to the scene segmentation frame based on the scene segmentation frame in combination with context semantic understanding and analysis of the subtitle information in the video to be processed by using a natural language processing model based on artificial intelligence, and take video frames belonging to the context relationship of the subtitle information as video clips in the same scene to obtain the scene segmentation result.

[0132] By way of example, the audio information described above can be background sound in the video to be processed. The server can take video frames of the same background sound, such as background music or background environmental sound, as video clips in the same scene.

[0133] By way of example, the audio information described above can also be the speech characteristics of a character in the video. The server can take adjacent video frames of the same speech characteristics, such as continuous speech of the same character, as video clips in the same scene.

[0134] In some embodiments, the server can also obtain the motion trajectory of a person or object in a video frame by using detection of changes in the spatial position of a pixel point, and perform scene segmentation in combination with the scene segmentation frame according to features such as motion trajectory that are strongly associated with changes in time, which is selected according to actual conditions and is not limited in the embodiments of the present application.

[0135] It can be understood that the embodiments of the present application perform feature extraction at the frame level, record the similarity information between each video frame by constructing an inter-frame similarity matrix, and then perform scene change prediction based on the inter-frame similarity matrix to obtain scene segmentation frames, which converts the scene segmentation problem based on dividing video segments and clustering in the related art into a detection problem of scene segmentation frames, thereby reducing the influence of poor video segment quality on scene segmentation accuracy and improving the accuracy of scene segmentation. Moreover, the embodiments of the present application directly extract and predict the similarity information of the video frames, thereby reducing the workload of generating video segments and improving the efficiency of scene segmentation.

[0136] In some embodiments, based on any one of Figures 3-7 , as shown in Figure 9 , S101 can be implemented by performing S1011-S1012, which will be described in combination with each step.

[0137] S1011, performing feature extraction on each video frame in at least one way to obtain at least one single feature corresponding to each video frame.

[0138] In the embodiments of the present application, the features obtained by a single feature extraction method are often only applicable to a video scene with relatively single change. After the server obtains a video frame set by frame extraction on a to-be-processed video, the server can perform feature extraction on each video frame in the video frame set in at least one feature extraction manner to obtain a single feature corresponding to each video frame and each feature extraction manner, thereby obtaining at least one single feature corresponding to each video frame.

[0139] In some embodiments, the server can perform feature extraction on each video frame by at least one feature extraction model to obtain at least one single feature. Exemplarily, the at least one feature extraction model can include a CLIP model, a SWIN model and an EfficientNet model. For a given video frame I, the server can input it into the CLIP model, the SWIN model and the EfficientNet model respectively, perform feature extraction on the video frame I by using the feature extraction branches in the CLIP model, the SWIN model and the EfficientNet model, and obtain three single features, the process being as shown in formula (6) as follows:

[0140] (6)

[0141] In formula (6), , and respectively represent the corresponding network model, represents the single feature obtained by the model on the video frame I, representations single features extracted by the model from the video frame I, representations single features extracted by the model from the video frame I. Among them, , respectively represent the dimensions of the corresponding single features. That is, the feature dimensions of the single features extracted by different feature extraction methods, such as different feature extraction models, may be different.

[0142] Here, The model and The model is trained in a semantic and image dual-modal manner, so The model and The single features extracted by the model carry high-level semantic features, while The model focuses on extracting low-level features of the image itself, so that through the feature extraction of the above multiple models, more complex scenarios in the video frame can be covered.

[0143] Here, it should be noted that the server can determine at least one feature extraction method according to actual needs, such as performance, speed, or feature accuracy requirements, etc. The at least one feature extraction method is not limited to using the three feature extraction models in the above examples for extraction, and can be any at least one feature extraction method or a combination of extraction methods corresponding to the feature extraction model, thereby improving the flexibility and scalability of the at least one feature extraction method.

[0144] S1012, feature fusion is performed on the at least one single feature to obtain a feature vector corresponding to each video frame.

[0145] In the embodiments of the present application, the server can merge at least one single feature into a feature vector by performing feature fusion on the at least one single feature.

[0146] In some embodiments, the server can perform feature fusion on the at least one single feature with the same weight, and in some embodiments, the server can also perform weighted fusion with different weights for each single feature. The specific selection is made according to the actual situation, and the embodiments of the present application are not limited.

[0147] In some embodiments, the server can obtain the weight of each single feature for fusion through an attention mechanism. The server can map the at least one single feature to a unified feature dimension to obtain at least one mapped single feature; perform feature splicing and average pooling processing on the at least one mapped single feature to obtain an average feature; calculate the fusion weight corresponding to each mapped single feature in the at least one mapped single feature based on the attention mechanism according to the average feature; and perform weighting and splicing on the at least one mapped single feature according to the fusion weight corresponding to each mapped single feature to obtain a feature vector.

[0148] Exemplarily, the server can fuse at least one single feature by using an SE (Squeeze-and-Excitation) network model. The SE network model itself can modulate information of different regions in an image through a spatial attention mechanism. In the embodiments of the present application, the SE network model can be used to modulate the fusion weights of the three single model features, so that the single features extracted by different models can be adaptively fused.

[0149] Exemplarily, the at least one single feature can be a single feature extracted by a model , a single feature extracted by a model , and a single feature extracted by a model . The server can map the at least one single feature to a unified feature dimension by formula (7) to obtain at least one mapped single feature, as follows:

[0150] (7)

[0151] In formula (7), represents a full connection layer, , and are the corresponding mapped single features, wherein , and . , , .

[0152] In some embodiments, based on formula (7), the server can perform feature splicing and average pooling processing on the at least one mapped single feature by formula (8) to obtain an average feature, as follows:

[0153] (8)

[0154] In formula (8), , represent a splicing layer and an average pooling layer respectively, represents the result of feature splicing, represents the average feature.

[0155] In some embodiments, based on formula (8), the server can calculate the fusion weight corresponding to each of the at least one mapped single feature according to the average feature by formula (9), and weight the at least one mapped single feature according to the fusion weight corresponding to each of the at least one mapped single feature. As follows:

[0156] (9)

[0157] In formula (9), denotes a small neural network; in some embodiments, may be implemented by a network containing two fully connected layers. 、 denote denote According to the average feature Each mapping single feature output 、 denote corresponding fusion weights, denote the single features obtained after reweighting.

[0158] In some embodiments, based on formula (9), the server can concatenate the weighted at least one mapping single feature to obtain a feature vector by formula (10) as follows:

[0159] (10)

[0160] In formula (10), denotes feature concatenation, is the final fusion feature obtained by concatenating the weighted mapping single model features, i.e., the feature vector.

[0161] It can be understood that by performing feature extraction and fusion in at least one way on the video frames, the feature vector obtained can contain more feature information, so as to adapt to the complex change scene of the video in the actual application scene, thereby improving the robustness of the scene segmentation detection network using the feature vector, and the accuracy of video scene segmentation.

[0162] At present, in the related art, in order to take into account the speed, the frequency of video frame sampling is not too high, which limits the detection accuracy of the scene segmentation frame. The applicant observed in a large number of experiments that within a local time range, the video frames near the actual scene segmentation frame change more dramatically in brightness, color and other attributes than the non-scene segmentation frames. Based on this observation, the embodiment of the present application provides a scene segmentation method based on pixel color, which can fine-tune the coarse segmentation frame with large error to the fine segmentation frame with small error, thereby improving the accuracy of scene segmentation. This will be described below.

[0163] In some embodiments, based on any one of Figures 3-7 As shown in Figure 10 S104 can be implemented by performing S1041-S1044, which will be described in conjunction with each step.

[0164] S1041, acquire a corresponding segmentation time point of the scene segmentation frame in the to-be-processed video.

[0165] S1042, perform frame extraction on the to-be-processed video according to a preset time range containing the segmentation time point, to obtain an updated video frame set; the frame extraction frequency of the updated video frame set is higher than that of the video frame set.

[0166] In the embodiments of the present application, the server can take the time point of the scene segmentation frame in the to-be-processed video as the segmentation time point, acquire the segmentation time point, and perform frame extraction on the video near the segmentation time point at a higher sampling rate, to obtain an updated video frame set.

[0167] In some embodiments, the scene segmentation frame can be a video frame obtained through sparse sampling once per second, and the corresponding segmentation time point of the scene segmentation frame is the tthsecond. The server can take each of the 0.5 seconds before and after the tthsecond as a preset time range, take the original frame rate of the to-be-processed video as the frame extraction frequency, extract the dense video frames in each of the 0.5 seconds before and after the tthsecond as updated video frames, and assume that the original frame rate of the to-be-processed video is 25 frames per second, then 25 updated video frames can be extracted, and the time accuracy of each updated video frame is 0.04 seconds, thereby obtaining an updated video frame set of 25 frames. It can be seen that the frame extraction frequency and time accuracy of the updated video frame set are higher than those of the video frame set, and therefore, on the basis of the coarse positioning of the scene segmentation frame, a more fine segmentation frame can be positioned through the updated video frame set, to improve the accuracy of scene segmentation.

[0168] S1043, determine an updated segmentation frame corresponding to the scene segmentation frame from the updated video frame set according to the pixel color values contained in each updated video frame of the updated video frame set.

[0169] S1044, perform scene segmentation on the to-be-processed video according to the updated segmentation frame, to obtain a scene segmentation result.

[0170] In the embodiments of the present application, the server can input the updated video frame into a neural network, and predict the updated segmentation frame from the updated video frame through a large number of network operations. In order to reduce the amount of operation, the server can find the updated video frame with a dramatic color change from the updated video frame set through the calculation of the pixel color distance according to the pixel color values contained in each updated video frame, and determine the updated video frame as the updated segmentation frame corresponding to the scene segmentation frame.

[0171] In some embodiments, the server can calculate the pixel color distance according to the pixel color values corresponding to the same pixel position in adjacent updated video frames, evaluate the change of the pixel color in the time dimension based on the pixel color distance, and obtain the updated segmentation frame. ​

[0172] In some embodiments, in order to make the result of color segmentation more consistent with the intuitive perception of human eyes, the server can convert the pixel color space of each updated video frame, such as the original RGB color space, into an intuitive color space; here, the spatial model of the intuitive color space is represented by hue, saturation and brightness. Exemplarily, the intuitive color space can be the HSV (Hue, Saturation, Value) space. Compared with the RGB color space, the HSV color space directly depicts the hue, saturation and brightness attributes of a picture, and is more consistent with the perception of human eyes. Assuming that the RGB value of a certain pixel on the ith updated video frame is , the HSV value converted into the HSV space is .

[0173] In some embodiments, taking the pixel color distance in the HSV space as an example, for each updated video frame, the server can calculate the pixel color distance between the updated video frame and its corresponding two adjacent updated video frames according to the pixel color value by formula (11); as follows:

[0174] (11)

[0175] In formula (11), for each pixel position, is the HSV pixel value corresponding to the pixel position in the ith updated video frame, is the HSV pixel value corresponding to the pixel position in the (i-1)th updated video frame, is the HSV pixel value corresponding to the pixel position in the (i+1)th updated video frame; is the pixel color distance between the (i-1)th updated video frame and the ith updated video frame at the pixel position; is the pixel color distance between the ith updated video frame and the (i+1)th updated video frame at the pixel position; the server can perform the same processing on each pixel position by formula (11) to obtain the pixel color distance at each pixel position, and finally obtain the pixel color distance between each updated video frame and its corresponding two adjacent updated video frames by means of mean value processing of the pixel color distance at each pixel position.

[0176] It should be noted that the above method of obtaining the pixel color distance by calculating the difference between the pixel color values of each updated video frame and its corresponding two adjacent updated video frames is an exemplary calculation method of pixel color distance according to pixel color values, and other distance calculation methods can also be used in actual application, which is selected according to actual situation, and the embodiments of the present application are not limited.

[0177] In some embodiments, based on the pixel color distance between each updated video frame and its corresponding two adjacent updated video frames calculated above and The server can average the pixel color distance as the color change degree corresponding to each video frame. For example, the method of average processing can be shown in formula (12) as follows:

[0178] (12)

[0179] In formula (12), is the color change degree corresponding to the ith updated video frame. Here, the server can also use other average methods, such as variance average, exponential average, etc. to obtain the color change degree corresponding to each video frame, which is selected according to the actual situation, and the embodiments of the present application are not limited.

[0180] In some embodiments, in the updated scene video frame set, the server can determine the updated video frame corresponding to the maximum color change degree as the updated scene segmentation frame.

[0181] It can be understood that, compared to inputting each frame of picture into the neural network to calculate the probability of each frame being a segmentation point, the method directly operates on the video frame by calculating the transformation of pixel values between adjacent frames, which saves the process of inputting the video frame into the neural network, participating in the calculation by the neural network, and predicting the segmentation point, thereby greatly reducing the calculation amount and improving the efficiency of scene segmentation. Moreover, since the calculation amount of inter-frame color change is small, dense sampling can be used to improve the accuracy of the updated scene segmentation frame. Further, by converting the pixel color space from the RGB space to the HSV space, the scene segmentation result can be more consistent with the intuitive experience of the human eye, thereby improving the accuracy of scene segmentation.

[0182] In the following, the exemplary application of the embodiments of the present application in some actual scenarios will be introduced in combination with the processes of S301-S304 shown in Figure 11 .

[0183] S301, using a plurality of feature extraction models, extracting features of each frame of picture in the to-be-processed video.

[0184] In S301, the server can sample the to-be-processed video to frame, input the sampled frame pictures, i.e. video frames, into a plurality of feature extraction models for single feature extraction, and fuse the extracted single features to obtain the features of each frame of picture in the to-be-processed video, i.e. the feature vector corresponding to each video frame.

[0185] Here, the process description of S301 is consistent with the process description of S1011-S1012 described above, and will not be repeated here.

[0186] S302, constructing an inter-frame similarity matrix from the features of each frame of picture.

[0187] In S302, the process description of S302 is consistent with the process description of S102 described above, and will not be repeated here.

[0188] S303, inputting the inter-frame similarity matrix into a scene cut point detection network to obtain a coarse segmentation point.

[0189] In S303, the scene cut point detection network is equivalent to the scene segmentation detection network, and the coarse segmentation point is equivalent to the scene segmentation frame. The process of S303 is consistent with the process description of S102 described above, and will not be repeated here.

[0190] S304, a post-processing strategy based on HSV color space change detection is used to confirm the final scene segmentation point.

[0191] In S304, the server can confirm the final scene segmentation point, i.e., update the scene segmentation frame, by detecting the maximum color change point near the coarse segmentation point in the HSV color space. The process of S304 is consistent with the process description of S1041-S1044, and will not be repeated here.

[0192] After S304, the server can complete video scene segmentation according to the final scene segmentation point.

[0193] As can be seen, the embodiments of the present application construct an inter-frame similarity matrix representing the inter-frame similarity information of the video based on the feature vector of each video frame, and process the inter-frame similarity information of the video through a scene cut point detection network to predict the segmentation point where the scene changes. The segmentation precision of the existing video segment is improved to frame-level segmentation, and the calculation amount of additional video segment division is not required, thereby improving the efficiency and accuracy of scene segmentation. Furthermore, in order to adapt to the complex change scene of online video, multiple advanced features are fused, so that the scene segmentation can cover more complex change scenes. Further, through the coarse detection based on pixel color calculation + fine adjustment strategy, the time cost caused by processing dense sampling data is avoided, the requirements of precision and time efficiency are met, and the efficiency and accuracy of scene segmentation are further improved.

[0194] In some embodiments, the effect of the scene segmentation method in various embodiments of the present application can be further illustrated by experiments. The following experiments are performed on a dataset of 5000 training videos and 5000 test videos, and are evaluated using the Mean Intersection over Union (MIoU) index. The experimental results are shown in Table 1 as follows:

[0195]

[0196] Table 1

[0197] In Table 1, the baseline method is a method that does not include the multi-feature fusion and post-processing strategy based on pixel color values described above. The MIoU of the baseline method is about 82%. After adding the post-processing strategy, the MIoU is improved to 86.1%, indicating that the post-processing strategy significantly improves the detection accuracy of the scene segmentation frame. After adding multi-feature fusion, the MIoU is improved to 89.2%, indicating that using multi-feature fusion has obvious advantages over single feature and can further improve the accuracy of scene segmentation.

[0198] The following continues to illustrate an exemplary structure of the video scene segmentation apparatus 255 provided by the embodiments of the present application as a software module. In some embodiments, as shown in FIG. 25, the software module stored in the video scene segmentation apparatus 255 of the memory 250 can include an extraction module 2551, a calculation module 2552, a prediction module 2553, and a segmentation module 2554, wherein, Figure 2

[0199] The extraction module 2551 is configured to frame extraction on a to-be-processed video to obtain a video frame set, and perform feature extraction on each video frame in the video frame set to obtain a feature vector corresponding to each video frame.

[0200] The calculation module 2552 is configured to construct an inter-frame similarity matrix based on the feature vector, wherein the inter-frame similarity matrix represents the similarity between video frames.

[0201] The prediction module 2553 is configured to perform scene change prediction based on the inter-frame similarity matrix to determine a scene segmentation frame from the video frame set.

[0202] The segmentation module 2554 is configured to perform scene segmentation on the to-be-processed video based on the scene segmentation frame to obtain a scene segmentation result.

[0203] In some embodiments, the extraction module 2551 is further configured to perform feature extraction on each video frame in at least one way to obtain at least one single feature corresponding to each video frame, and perform feature fusion on the at least one single feature to obtain a feature vector corresponding to each video frame.​

[0204] In some embodiments, the extraction module 2551 is further configured to map the at least one single feature to a unified feature dimension to obtain at least one mapped single feature; perform feature splicing and average pooling processing on the at least one mapped single feature to obtain an average feature; calculate a fusion weight corresponding to each mapped single feature in the at least one mapped single feature based on an attention mechanism according to the average feature; and perform weighting and splicing on the at least one mapped single feature according to the fusion weight corresponding to each mapped single feature to obtain the feature vector.

[0205] In some embodiments, the calculation module 2552 is further configured to calculate a distance between the feature vector of each video frame and the feature vectors of all video frames in the set of video frames respectively to obtain at least one feature distance corresponding to each video frame; construct a feature distance matrix according to the at least one feature distance corresponding to each video frame; and perform normalization processing on the feature distance matrix to obtain the inter-frame similarity matrix.

[0206] In some embodiments, the prediction module 2553 is further configured to perform convolution processing and classification prediction on the inter-frame similarity matrix by using a scene segmentation detection network to obtain a scene segmentation probability corresponding to each video frame; and determine the scene segmentation frame according to the scene segmentation probability.

[0207] In some embodiments, the scene segmentation detection network comprises a classification module and at least two convolution modules; and the prediction module 2553 is further configured to perform multi-layer convolution processing on the inter-frame similarity matrix by the at least two convolution modules to obtain a frame embedding vector; the frame embedding vector contains context information corresponding to each video frame; and perform classification prediction based on the frame embedding vector by the classification module to obtain the scene segmentation probability corresponding to each video frame.

[0208] In some embodiments, the at least two convolution modules comprise a first convolution module and at least one second convolution module, a feature dimension corresponding to the first convolution module is higher than a feature dimension corresponding to the second convolution module; and the prediction module 2553 is further configured to perform convolution processing on the inter-frame similarity matrix by the first convolution module to obtain a multi-dimensional feature vector corresponding to each matrix element in the inter-frame similarity matrix, thereby obtaining a first feature matrix; perform dimension transformation on the first feature matrix to obtain a second feature matrix; and perform convolution processing on the second feature matrix by the at least one second convolution module to obtain the frame embedding vector.

[0209] In some embodiments, the segmentation module 2554 is further configured to acquire a segmentation time point corresponding to the scene segmentation frame in the to-be-processed video; perform frame extraction on the to-be-processed video according to a preset time range containing the segmentation time point, to obtain an updated video frame set; the frame extraction frequency of the updated video frame set is higher than that of the video frame set; determine an updated segmentation frame corresponding to the scene segmentation frame from the updated video frame set according to the pixel color value contained in each updated video frame of the updated video frame set; and perform scene segmentation on the to-be-processed video according to the updated segmentation frame, to obtain a scene segmentation result.

[0210] In some embodiments, the segmentation module 2554 is further configured to calculate a pixel color distance between each updated video frame and two adjacent updated video frames corresponding to the updated video frame; and determine the updated segmentation frame from the updated video frame set according to the pixel color distance.

[0211] In some embodiments, the segmentation module 2554 is further configured to, before the calculation of the pixel color distance between each updated video frame and two adjacent updated video frames corresponding to the updated video frame, convert the pixel color space of each updated video frame into a perceptual color space; and the spatial model of the perceptual color space is represented by hue, saturation and brightness.

[0212] In some embodiments, the segmentation module 2554 is further configured to perform mean value processing on the pixel color distance, as the color change degree corresponding to each video frame; and determine, in the updated scene video frame set, an updated video frame corresponding to the maximum color change degree as the updated scene segmentation frame.

[0213] It should be noted that the above description of the device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0214] The embodiments of the present application provide a computer program product, which includes a computer program or executable instructions, and when the computer program or executable instructions run on an electronic device, the electronic device performs a video scene segmentation method as described in the embodiments of the present application.

[0215] The embodiments of the present application provide a computer readable storage medium storing executable instructions, wherein the executable instructions are stored, and when the executable instructions are executed by a processor, the processor will execute the video scene segmentation method provided by the embodiments of the present application, for example, the method as shown in Figures 3-7 , or Figures 9-11 .

[0216] In some embodiments, the computer-readable storage media can be memory such as a FRAM, a ROM, a PROM, an EPROM, an EEPROM, a flash memory, a magnetic surface memory, an optical disk, or a CD-ROM, etc.; or various devices including one or any combination of the above memories.

[0217] In some embodiments, the executable instructions can be in the form of a program, software, a software module, a script, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0218] As an example, the executable instructions can, but need not, correspond to a file in a file system, can be stored in a part of a file that holds other programs or data, for example, in one or more scripts stored in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, sub programs, or portions of code.

[0219] As an example, the executable instructions can be deployed to execute on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected through a communication network.

[0220] In summary, the embodiments of the present application perform feature extraction at frame level granularity, and record the similarity information between each video frame by constructing an inter-frame similarity matrix, and then can perform scene change prediction based on the inter-frame similarity matrix to obtain scene segmentation frames, which realizes the conversion of the scene segmentation problem in the related art based on dividing video segments and clustering into the detection problem of scene segmentation frames, thereby reducing the influence of poor video segment quality on scene segmentation accuracy, and improving the accuracy of scene segmentation. Moreover, the embodiments of the present application directly perform similarity information extraction and prediction processing on video frames, thereby reducing the workload of generating video segments, and improving the efficiency of scene segmentation. Moreover, by performing feature extraction and fusion in at least one way on video frames, the obtained feature vectors can contain more feature information, thereby being able to adapt to complex change scenes of videos in actual application scenarios, thereby being able to improve the robustness of the scene segmentation detection network using the feature vectors, and the accuracy of video scene segmentation. Moreover, by calculating the transformation of pixel values between adjacent frames, direct operation is performed on video frames, which saves the process of inputting video frames into a neural network, participating in calculation by the neural network, and predicting segmentation points, thereby greatly reducing the amount of calculation, and improving the efficiency of scene segmentation. Moreover, since the calculation amount of inter-frame color change is small, dense sampling can be used to improve the accuracy of updating scene segmentation frames. Further, by converting the pixel color space from the RGB space to the HSV space, the scene segmentation result can be more consistent with the intuitive experience effect of the human eye, thereby improving the accuracy of scene segmentation.

[0221] The above merely describes the embodiments of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A video scene segmentation method, characterized in that, include: The video to be processed is frame-stripped to obtain a set of video frames; and features are extracted from each video frame in the set of video frames to obtain at least one single feature corresponding to each video frame. Map at least one of the single features to a unified feature dimension to obtain at least one mapped single feature; The at least one mapped single feature is subjected to feature concatenation and average pooling to obtain the average feature; Based on the average features, the fusion weights corresponding to each of the at least one mapped single features are calculated using an attention mechanism; Based on the fusion weights corresponding to each mapping single feature, the at least one mapping single feature is weighted and concatenated to obtain a feature vector; Based on the feature vectors, an inter-frame similarity matrix is ​​constructed; the inter-frame similarity matrix represents the degree of similarity between video frames. Scene change prediction is performed based on the inter-frame similarity matrix, and scene segmentation frames are determined from the video frame set; The scene segmentation results are obtained by segmenting the video to be processed based on the scene segmentation frames.

2. The method according to claim 1, characterized in that, The construction of the inter-frame similarity matrix based on the feature vector includes: Calculate the distance between each video frame and the feature vectors of all video frames in the video frame set to obtain at least one feature distance for each video frame. A feature distance matrix is ​​constructed based on at least one feature distance corresponding to each video frame; The feature distance matrix is ​​normalized to obtain the inter-frame similarity matrix.

3. The method according to claim 1, characterized in that, The step of predicting scene changes based on the inter-frame similarity matrix and determining scene segmentation frames from the video frame set includes: Using a scene segmentation detection network, the inter-frame similarity matrix is ​​convolutionally processed and classified for prediction to obtain the scene segmentation probability corresponding to each video frame; The scene segmentation frame is determined based on the scene segmentation probability.

4. The method according to claim 3, characterized in that, The scene segmentation and detection network includes a classification module and at least two convolutional modules; the process of using the scene segmentation and detection network to perform convolution processing and classification prediction on the inter-frame similarity matrix to obtain the scene segmentation probability corresponding to each video frame includes: The frame embedding vector is obtained by performing multi-layer convolution processing on the inter-frame similarity matrix through the at least two convolution modules. The classification module performs classification prediction based on the frame embedding vector to obtain the scene segmentation probability corresponding to each video frame.

5. The method according to claim 4, characterized in that, The at least two convolutional modules include: a first convolutional module and at least one second convolutional module, wherein the feature dimension corresponding to the first convolutional module is higher than the feature dimension corresponding to the second convolutional module; the step of performing multi-layer convolution processing on the inter-frame similarity matrix through the at least two convolutional modules to obtain a frame embedding vector includes: The first convolution module performs convolution processing on the inter-frame similarity matrix to obtain the multi-dimensional feature vector corresponding to each matrix element in the inter-frame similarity matrix, thereby obtaining the first feature matrix. The first feature matrix is ​​transformed to obtain the second feature matrix; The second feature matrix is ​​convolved by the at least one second convolution module to obtain the frame embedding vector.

6. The method according to any one of claims 1 to 5, characterized in that, The step of performing scene segmentation on the video to be processed based on the scene segmentation frames to obtain scene segmentation results includes: Obtain the segmentation time point corresponding to the scene segmentation frame in the video to be processed; Based on a preset time range including the segmentation time points, the video to be processed is frame-sampling to obtain an updated video frame set; the frame-sampling frequency of the updated video frame set is higher than the frame-sampling frequency of the video frame set. Based on the pixel color values ​​contained in each updated video frame of the updated video frame set, the updated segmentation frame corresponding to the scene segmentation frame is determined from the updated video frame set; Based on the updated segmentation frame, scene segmentation is performed on the video to be processed to obtain the scene segmentation result.

7. The method according to claim 6, characterized in that, The step of determining the updated segmentation frame corresponding to the scene segmentation frame from the updated video frame set based on the pixel color values ​​contained in each updated video frame of the updated video frame set includes: Based on the pixel color value, calculate the pixel color distance between each updated video frame and its two adjacent updated video frames; The updated segmented frame is determined from the updated video frame set based on the pixel color distance.

8. The method according to claim 7, characterized in that, Before calculating the pixel color distance between each updated video frame and its two adjacent updated video frames, the method further includes: The pixel color space of each updated video frame is converted into an intuitive color space; the spatial model of the intuitive color space is represented by hue, saturation, and brightness.

9. The method according to claim 7, characterized in that, Determining the updated segmented frame from the updated video frame set based on the pixel color distance includes: The pixel color distance is averaged to obtain the color change degree for each video frame. In the set of updated video frames, the updated video frame corresponding to the largest color change is determined as the updated segmented frame.

10. A video scene segmentation device, characterized in that, include: The extraction module is used to extract frames from the video to be processed to obtain a set of video frames; and to extract features from each video frame in the set of video frames to obtain at least one single feature corresponding to each video frame; and to map at least one single feature to a unified feature dimension to obtain at least one mapped single feature. The at least one mapped single feature is subjected to feature concatenation and average pooling to obtain an average feature; based on the average feature, the fusion weight corresponding to each mapped single feature in the at least one mapped single feature is calculated based on an attention mechanism; based on the fusion weight corresponding to each mapped single feature, the at least one mapped single feature is weighted and concatenated to obtain a feature vector; The calculation module is used to construct an inter-frame similarity matrix based on the feature vectors; The inter-frame similarity matrix represents the degree of similarity between video frames; The prediction module is used to predict scene changes based on the inter-frame similarity matrix and determine scene segmentation frames from the video frame set. The segmentation module is used to perform scene segmentation on the video to be processed based on the scene segmentation frame to obtain the scene segmentation result.

11. The apparatus according to claim 10, characterized in that, The computing module is specifically used for: Calculate the distance between each video frame and the feature vectors of all video frames in the video frame set to obtain at least one feature distance for each video frame. A feature distance matrix is ​​constructed based on at least one feature distance corresponding to each video frame; The feature distance matrix is ​​normalized to obtain the inter-frame similarity matrix.

12. The apparatus according to claim 10, characterized in that, The prediction module is specifically used for: Using a scene segmentation detection network, the inter-frame similarity matrix is ​​convolutionally processed and classified for prediction to obtain the scene segmentation probability corresponding to each video frame; The scene segmentation frame is determined based on the scene segmentation probability.

13. The apparatus according to claim 12, characterized in that, The scene segmentation and detection network includes: a classification module and at least two convolutional modules; the prediction module is further used for: The frame embedding vector is obtained by performing multi-layer convolution processing on the inter-frame similarity matrix through the at least two convolution modules. The classification module performs classification prediction based on the frame embedding vector to obtain the scene segmentation probability corresponding to each video frame.

14. The apparatus according to claim 13, characterized in that, The at least two convolutional modules include: a first convolutional module and at least one second convolutional module, wherein the feature dimension corresponding to the first convolutional module is higher than the feature dimension corresponding to the second convolutional module; the prediction module is further configured to: The first convolution module performs convolution processing on the inter-frame similarity matrix to obtain the multi-dimensional feature vector corresponding to each matrix element in the inter-frame similarity matrix, thereby obtaining the first feature matrix. The first feature matrix is ​​transformed to obtain the second feature matrix; The second feature matrix is ​​convolved by the at least one second convolution module to obtain the frame embedding vector.

15. The apparatus according to any one of claims 10 to 14, characterized in that, The segmentation module is specifically used for: Obtain the segmentation time point corresponding to the scene segmentation frame in the video to be processed; Based on a preset time range including the segmentation time points, the video to be processed is frame-sampling to obtain an updated video frame set; the frame-sampling frequency of the updated video frame set is higher than the frame-sampling frequency of the video frame set. Based on the pixel color values ​​contained in each updated video frame of the updated video frame set, the updated segmentation frame corresponding to the scene segmentation frame is determined from the updated video frame set; Based on the updated segmentation frame, scene segmentation is performed on the video to be processed to obtain the scene segmentation result.

16. The apparatus according to claim 15, characterized in that, The segmentation module is further configured to: Based on the pixel color value, calculate the pixel color distance between each updated video frame and its two adjacent updated video frames; The updated segmented frame is determined from the updated video frame set based on the pixel color distance.

17. The apparatus according to claim 16, characterized in that, The segmentation module is further configured to: Before calculating the pixel color distance between each updated video frame and its two adjacent updated video frames, the pixel color space of each updated video frame is converted into an intuitive color space; the spatial model of the intuitive color space is represented by hue, saturation, and brightness.

18. The apparatus according to claim 16, characterized in that, The segmentation module is further configured to: The pixel color distance is averaged to obtain the color change degree for each video frame. In the set of updated video frames, the updated video frame corresponding to the largest color change is determined as the updated segmented frame.

19. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method of any one of claims 1 to 9.

20. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the method of any one of claims 1 to 9.

21. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method of any one of claims 1 to 9.

Images