Intelligent video source switching system and method for director switching station

Abstract

The invention relates to the technical field of video source switching, and discloses an intelligent video source switching system and method for a director switching station. The method comprises the following steps: acquiring an input video stream of a current video source through a director switching station, and performing N-gram feature vector calculation and inter-frame difference analysis to obtain video frame segmentation data; performing scene analysis on the video frame segmentation data to obtain a scene feature signature; performing spatial-temporal feature extraction and fusion on each video frame in the input video stream according to the scene feature signature to obtain a scene spatial-temporal fusion feature set; inputting the scene space-time fusion feature set into an LSTM-based lightweight time sequence convolutional network model to carry out video switching behavior prediction so as to obtain video switching behavior prediction data; and inputting the video switching behavior prediction data into a double-depth Q learning model based on a Markov decision process to carry out video source switching decision to obtain a video source switching decision result, thereby improving the intelligent video source switching accuracy of the director switching station.

Description

Technical Field

[0001] The present application relates to the technical field of video source switching, and in particular to an intelligent video source switching system and method for a director switcher. Background Art

[0002] With the continuous development of TV program production and live broadcast technology, the director switcher plays an increasingly important role in video source switching and picture synthesis. Traditional director switching operations mainly rely on manual experience, which not only requires operators to have rich professional knowledge and quick response capabilities, but is also easily affected by subjective factors, resulting in unstable switching effects. At the same time, with the increase in multi-camera shooting and complex scenes, the pressure and risk of error in manual operation also increase.

[0003] In recent years, artificial intelligence technology has made significant progress in the field of video processing, providing new ideas for solving the above problems. However, existing intelligent video switching methods still have some limitations. The understanding of video content and scene analysis are not deep enough, and it is difficult to accurately capture key information and changes in complex scenes. Secondly, it is difficult to achieve a good balance between real-time and decision-making accuracy, and it cannot meet the needs of high-quality live broadcasts. Summary of the invention

[0004] The present application provides an intelligent video source switching system and method for a director switcher, which are used to improve the accuracy of the intelligent video source switching of the director switcher.

[0005] In a first aspect, the present application provides an intelligent video source switching method for a director switcher, the intelligent video source switching method for a director switcher comprising:

[0006] The input video stream of the current video source is obtained through the director switcher, and the N-gram feature vector calculation and inter-frame difference analysis are performed on the input video stream to obtain video frame segmentation data;

[0007] Performing scene analysis on the video frame segmentation data based on a regular expression generation algorithm of a dictionary tree to obtain a scene feature signature;

[0008] Extract and fuse spatiotemporal features of each video frame in the input video stream according to the scene feature signature to obtain a scene spatiotemporal fusion feature set;

[0009] Inputting the scene spatiotemporal fusion feature set into a LSTM-based lightweight temporal convolutional network model to predict video switching behavior, thereby obtaining video switching behavior prediction data;

[0010] The video switching behavior prediction data is input into a dual-depth Q learning model based on a Markov decision process to make a video source switching decision, and a video source switching decision result is obtained.

[0011] In a second aspect, the present application provides an intelligent video source switching system for a director switcher, the intelligent video source switching system for a director switcher comprising:

[0012] An acquisition module is used to acquire the input video stream of the current video source through the director switcher, and perform N-gram feature vector calculation and inter-frame difference analysis on the input video stream to obtain video frame segmentation data;

[0013] An analysis module, used for performing scene analysis on the video frame segmentation data based on a regular expression generation algorithm of a dictionary tree to obtain a scene feature signature;

[0014] A fusion module, used for extracting and fusing spatiotemporal features of each video frame in the input video stream according to the scene feature signature to obtain a scene spatiotemporal fusion feature set;

[0015] A prediction module, used for inputting the scene spatiotemporal fusion feature set into a lightweight temporal convolutional network model based on LSTM to predict video switching behavior and obtain video switching behavior prediction data;

[0016] The decision module is used to input the video switching behavior prediction data into a dual-depth Q learning model based on a Markov decision process to make a video source switching decision and obtain a video source switching decision result.

[0017] A third aspect of the present application provides an intelligent video source switching device for a director switch station, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the intelligent video source switching device for the director switch station executes the above-mentioned intelligent video source switching method for the director switch station.

[0018] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned intelligent video source switching method for a director switcher.

[0019] In the technical solution provided by the present application, through N-gram feature vector calculation and inter-frame difference analysis, the refined segmentation of video content is realized, which helps to improve the accuracy and coherence of video source switching. The regular expression generation algorithm based on the dictionary tree can adaptively generate scene matching rules, greatly improve the flexibility and robustness of scene recognition, and enable the system to better adapt to different types of program content. The spatiotemporal feature extraction and fusion technology fully considers the temporal information and spatial characteristics of the video, so that the system can understand the video content more comprehensively, thereby making more reasonable switching decisions. The lightweight temporal convolutional network model based on LSTM effectively combines long-term and short-term memory capabilities and local feature extraction capabilities, can accurately predict video switching behavior, and improve the real-time and prediction accuracy of the system. The dual-depth Q learning model based on the Markov decision process is used for video source switching decisions, which realizes adaptive learning in complex environments and can make the optimal switching strategy according to different scenarios. The multi-level feature extraction and decision-making process make the system highly interpretable, while ensuring the switching quality, greatly reducing the workload of manual operation, and improving the work efficiency and automation of the director switcher. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 A schematic diagram of an embodiment of an intelligent video source switching method for a director switcher in an embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of an embodiment of an intelligent video source switching system for a director switcher in an embodiment of the present application. DETAILED DESCRIPTION

[0023] Embodiments of the present application provide an intelligent video source switching system and method for a director switcher. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 , an embodiment of an intelligent video source switching method for a director switcher in an embodiment of the present application includes:

[0025] Step S101: obtaining the input video stream of the current video source through the director switch, and performing N-gram feature vector calculation and inter-frame difference analysis on the input video stream to obtain video frame segmentation data;

[0026] It is understandable that the execution subject of the present application may be an intelligent video source switching system for a director switcher, or a terminal or a server, which is not limited here. The present application embodiment is described by taking a server as the execution subject as an example.

[0027] Specifically, the input video stream of the current video source is obtained through the input interface of the director switcher. The director switcher can receive multiple video sources in real time and select one of them as the current input video stream. The input video stream is subjected to frame extraction processing, and the video stream is extracted into several video frames at fixed time intervals to form a continuous frame sequence. The color feature vector of each video frame is calculated by the color histogram, and the color histogram is obtained based on the color distribution statistics of the pixel, which can effectively reflect the color characteristics of the image. The color feature vector is converted into an N-gram feature vector. The N-gram feature vector is a feature expression method based on sequence information, which can capture the local continuity characteristics in the video frame sequence. The inter-frame difference is quantified by calculating the cosine similarity between adjacent video frames. The cosine similarity is a commonly used similarity measurement method. The closer its value is to 1, the more similar the two vectors are, and vice versa, the greater the difference. After the inter-frame difference value is obtained by using the cosine similarity calculation, the difference value is processed by the adaptive threshold segmentation method to determine the initial segmentation point set. The adaptive threshold segmentation can dynamically adjust the threshold according to the statistical characteristics of the inter-frame difference value to achieve more accurate segmentation point detection. For the initial segmentation point set, audio feature extraction is performed on the video frame sequence to obtain an audio feature vector. Audio feature extraction is based on the audio signal in the video stream. Commonly used methods include MFCC (Mel-Frequency Cepstrum Coefficient) and STFT (Short-Time Fourier Transform). The audio feature vector can reflect the audio features corresponding to the video frame, and it is weighted and fused with the difference value between frames to form a fused feature sequence. Dynamic programming is performed on the fused feature sequence to obtain the target segmentation point set. Dynamic programming is an optimization algorithm that can find the optimal segmentation solution in a global range. According to the target segmentation point set, the video frame sequence is segmented and tested to ensure that the video frames in each segment have high similarity and the differences between segments are large, so as to obtain accurate video frame segmentation data.

[0028] Step S102: Perform scene analysis on the video frame segmentation data based on a regular expression generation algorithm of a dictionary tree to obtain a scene feature signature;

[0029] Specifically, multi-scale feature extraction is performed on the video frame segmentation data to obtain a hierarchical feature descriptor, and a multi-level scene representation is constructed based on the hierarchical feature descriptor. The multi-scale feature extraction method can capture feature information of different scales in the video frame, making the scene representation richer and more accurate. By constructing a multi-level scene representation, these feature information can be better organized and managed. The multi-level scene representation is segmented to obtain an initial scene boundary, and the correlation analysis of adjacent scenes is performed based on the initial scene boundary to obtain a scene correlation matrix. The determination of the initial scene boundary is based on the change of feature descriptors. By analyzing the correlation between adjacent scenes, a scene correlation matrix is ​​constructed to reflect the similarity and connection between different scenes. The initial scene boundary is optimized for scene segmentation based on the scene correlation matrix to obtain an optimized scene segmentation result. By analyzing the information in the scene correlation matrix, the initial scene boundary is adjusted and optimized. The key frame sequence of each scene is extracted based on the optimized scene segmentation result to obtain a scene representative frame set, and a scene feature dictionary tree of the scene representative frame set is dynamically updated based on the regular expression generation algorithm of the dictionary tree. The extraction of the key frame sequence is based on the scene segmentation result, and the scene representative frame set is constructed by selecting representative frames. The regular expression generation algorithm based on the dictionary tree can effectively organize and manage the feature information in the scene representative frame set and construct a dynamically updated scene feature dictionary tree. The dictionary tree can be dynamically adjusted as the video content changes to keep the scene features updated and accurate. An adaptive regular expression template is generated based on the dynamically updated scene feature dictionary tree to obtain a scene matching rule set, and a scene recognition strategy is generated based on the scene matching rule set. The adaptive regular expression template can be adjusted according to the changes in scene features to generate an accurate scene matching rule set. Based on the matching rule set, a scene recognition strategy is formulated to ensure accurate recognition and classification of different scenes. Feature matching and encoding of video frame segmentation data are performed according to the scene recognition strategy, and the feature information in the video frame segmentation data is converted into a scene feature signature to form a complete scene recognition and classification system.

[0030] Step S103, extracting and fusing spatiotemporal features of each video frame in the input video stream according to the scene feature signature to obtain a scene spatiotemporal fusion feature set;

[0031] Specifically, multi-level feature construction and adaptive feature screening are performed on each video frame in the input video stream according to the scene feature signature to obtain a scene-related feature set. The multi-level feature construction method can capture feature information of various scales and levels in the video frame, and the adaptive feature screening can screen out scene-related features by evaluating the importance of features. Dynamic convolution operation and feature extraction are performed on the scene-related feature set to obtain a scene-adaptive feature map. The dynamic convolution operation captures the changes of video frames in the time dimension by applying convolution kernels in different time periods, and the feature extraction further processes these convolution results to generate an adaptive feature map reflecting the scene characteristics. Temporal difference calculation is performed on the scene-adaptive feature map to obtain inter-frame change features. Temporal difference calculation reveals the dynamic features of video frames changing over time by comparing the feature changes between consecutive frames. The dynamic weight matrix is ​​calculated according to the inter-frame change features, and the scene-adaptive feature map and the dynamic weight matrix are weighted fused to obtain enhanced spatiotemporal features. The dynamic weight matrix is ​​calculated based on the inter-frame change features to reflect the importance and relevance of video frames at different time points. The expressiveness of spatiotemporal features is enhanced by weighted fusion of scene adaptive feature maps and dynamic weight matrices. A temporal feature sequence is generated based on the enhanced spatiotemporal features, and multi-scale spatiotemporal feature aggregation is performed on the temporal feature sequence to obtain a scene spatiotemporal fusion feature set. Multi-scale spatiotemporal feature aggregation enhances the expressiveness of the feature set by integrating temporal features at different scales, ensuring that it has good adaptability and expressiveness at different time and space scales.

[0032] Step S104: input the scene spatiotemporal fusion feature set into the LSTM-based lightweight temporal convolutional network model to predict the video switching behavior, and obtain the video switching behavior prediction data;

[0033] Specifically, the scene spatiotemporal fusion feature set is input into the LSTM-based lightweight temporal convolutional network model, and the scene spatiotemporal fusion feature set is processed as an input tensor to obtain a T×D-dimensional input tensor, where T is the time step and D is the feature dimension. The T×D-dimensional input tensor is input into the multi-channel one-dimensional convolution layer, and the convolution operation is performed through F convolution kernels of different sizes to extract the local pattern and change trend of the input feature set in the time dimension, and obtain a T×F-dimensional local temporal feature representation. The T×F-dimensional local temporal feature representation is subjected to a maximum pooling operation to obtain a (T / s)×F-dimensional reduced dimension feature map, where the pooling window size is k and the step size is s. The maximum pooling operation can reduce the feature dimension while retaining important temporal information and reducing computational complexity. The (T / s)×F-dimensional reduced dimension feature map is input into the residual block, and a (T / s)×F-dimensional multi-scale temporal feature is output. The residual block contains two layers of one-dimensional convolution and a skip connection, which can effectively capture multi-scale temporal features, and solve the gradient vanishing problem through skip connections to ensure the training stability of deep networks. Channel attention calculation is performed on the (T / s)×F-dimensional multi-scale temporal features, and an F-dimensional weight vector is generated and multiplied element by element to obtain a (T / s)×F-dimensional weighted feature map. The channel attention mechanism can adaptively adjust the weight according to the importance of the feature, highlight the key features, and improve the feature expression ability. The (T / s)×F-dimensional weighted feature map is reshaped into a (T / s) F-dimensional vector sequence, input into the LSTM unit, and the H-dimensional hidden state is output at each time step to obtain a (T / s)×H-dimensional long-range dependency feature. The LSTM unit can effectively capture long-term dependencies through its long short-term memory mechanism and has excellent performance in modeling time series data. The self-attention mechanism is analyzed for the (T / s)×H-dimensional long-range dependency features, and the (T / s)×(T / s) attention matrix is ​​calculated and multiplied to obtain the (T / s)×H-dimensional context-enhanced features. The self-attention mechanism can capture the dependencies between positions in the sequence, provide global context information, and enhance the feature representation capability. The (T / s)×H-dimensional context-enhanced features are input into the gated linear unit, and the information flow is controlled by the sigmoid gating mechanism to output the (T / s)×H-dimensional candidate switching behavior features. The gated linear unit can filter according to the importance of the input information to improve the accuracy of feature expression. The (T / s)×H-dimensional candidate switching behavior features are analyzed by a multi-task learning head, including N parallel fully connected layers, each fully connected layer outputs a Mi-dimensional prediction result, and obtains N prediction vectors of different dimensions, where i=1, 2,..., N. The multi-task learning head can process multiple related tasks at the same time, improving the generalization ability and prediction accuracy of the model. According to the predefined task weights, the prediction vectors of N different dimensions are weighted and summed, and normalized through the softmax function to obtain the video switching behavior prediction data, so as to achieve accurate prediction of video switching behavior and provide intelligent decision support for the director switcher.

[0034] Step S105: input the video switching behavior prediction data into the dual deep Q learning model based on the Markov decision process to make a video source switching decision, and obtain a video source switching decision result.

[0035] Specifically, the state information of the current video source is obtained, and the video switching behavior prediction data and the state information of the current video source are spliced ​​to obtain an S-dimensional state vector. The S-dimensional state vector is input into the state encoder network, and the E-dimensional encoded state representation is obtained through two layers of fully connected layers and ReLU activation function, ensuring that the high-dimensional features of the input state vector can be effectively compressed and expressed. The E-dimensional encoded state representation is processed by the attention mechanism, and the similarity with the historical state is calculated to obtain the A-dimensional attention weight vector. The attention mechanism generates an attention weight vector by calculating the similarity between the current state and the historical state, so that the model can focus on the historical information most relevant to the current state. The historical state is weighted and summed according to the A-dimensional attention weight vector to obtain the H-dimensional historical context vector. The E-dimensional encoded state representation and the H-dimensional historical context vector are spliced ​​and input into the value network in the dual-depth Q learning model based on the Markov decision process. The scalar state value estimation is obtained through three layers of fully connected layers and LeakyReLU activation function. The value network estimates the value of the current state by comprehensively analyzing the state and historical context, providing a basis for action selection. At the same time, the E-dimensional coded state representation is processed by the action generation network, and a K-dimensional candidate action vector is obtained through two layers of fully connected layers and Tanh activation function. The action generation network generates possible candidate actions by processing the coded state. The K-dimensional candidate action vector and the E-dimensional coded state representation are spliced ​​and input into the advantage function network in the double deep Q learning model based on the Markov decision process. Through three layers of fully connected layers and ELU activation function, the K-dimensional action advantage value is obtained. The advantage function network is used to evaluate the degree of advantage of each candidate action relative to the current state, so as to select the best action among the candidate actions. According to the scalar state value estimate and the K-dimensional action advantage value, the K-dimensional Q value vector is calculated using the double Q learning formula. The double Q learning formula provides a more stable and accurate action value evaluation by combining state value and action advantage. The K-dimensional Q value vector is operated by the Softmax function with a temperature parameter of τ to obtain the K-dimensional action probability distribution. The Softmax function operation can convert the Q value vector into a probability distribution, ensuring that the action selection has a certain degree of exploration and randomness. Importance sampling is performed based on the K-dimensional action probability distribution, the optimal action index is selected, and the corresponding video source switching decision result is obtained by looking up the table. In this way, the model can intelligently switch between different video sources, optimize the operating efficiency of the director switcher and the quality of video switching.

[0036] The K-dimensional candidate action vector and the E-dimensional encoding state representation are concatenated to obtain a (K+E)-dimensional input feature vector, ensuring that the action and state information are represented in the same vector and providing a unified input. The first fully connected layer of the advantage function network is matrix multiplied according to the (K+E)-dimensional input feature vector to obtain an M-dimensional intermediate feature vector. The fully connected layer projects the input feature vector into a high-dimensional space through matrix multiplication to extract the deep features of the input features. The M-dimensional intermediate feature vector is batch normalized to obtain a standardized M-dimensional feature vector. Batch normalization reduces the deviation of data distribution by standardizing the mean and variance of each feature, thereby improving the training stability and convergence speed of the model. In batch normalization, the ELU activation function is calculated according to the standardized M-dimensional feature vector to obtain an M-dimensional nonlinear feature vector. The ELU activation function can introduce nonlinear characteristics, making the network more expressive, and its smoothness in the negative region helps alleviate the gradient vanishing problem. The M-dimensional nonlinear feature vector is dropout-operated to obtain an M-dimensional sparse feature vector. The dropout operation randomly discards some neurons to prevent the model from overfitting and improve its generalization ability. According to the M-dimensional sparse feature vector, the second fully connected layer of the advantage function network is matrix multiplied to obtain an N-dimensional intermediate feature vector. The second fully connected layer further extracts features and performs deep feature transformation on the input. The layer normalization operation is performed on the N-dimensional intermediate feature vector to obtain a normalized N-dimensional feature vector. Layer normalization is similar to batch normalization, but it normalizes each layer of each feature vector to more stably control the distribution of eigenvalues. According to the normalized N-dimensional feature vector, the ELU activation function is calculated to obtain an N-dimensional nonlinear feature vector. By repeatedly using the ELU activation function, it is ensured that the feature vector can capture rich nonlinear information in each layer. The N-dimensional nonlinear feature vector is residually connected and element-wise added to the (K+E)-dimensional input feature vector to obtain an N-dimensional residual feature vector. Residual connection directly adds input information to the output, reduces information loss, and improves the training effect of the network. Especially in deep networks, residual connection can effectively alleviate the gradient disappearance problem. According to the N-dimensional residual feature vector, the third fully connected layer of the advantage function network is matrix multiplied, and the ELU activation function is calculated to obtain the K-dimensional action advantage value. In this way, the advantage function network can accurately evaluate the advantage of the candidate action in the current state and provide a basis for the final action selection.

[0037] In the embodiment of the present application, through N-gram feature vector calculation and inter-frame difference analysis, the refined segmentation of video content is achieved, which helps to improve the accuracy and coherence of video source switching. The regular expression generation algorithm based on the dictionary tree can adaptively generate scene matching rules, greatly improve the flexibility and robustness of scene recognition, and enable the system to better adapt to different types of program content. The spatiotemporal feature extraction and fusion technology fully considers the temporal information and spatial characteristics of the video, so that the system can understand the video content more comprehensively, thereby making more reasonable switching decisions. The lightweight temporal convolutional network model based on LSTM effectively combines long-term and short-term memory capabilities and local feature extraction capabilities, can accurately predict video switching behavior, and improve the real-time and prediction accuracy of the system. The dual-depth Q learning model based on the Markov decision process is used for video source switching decisions, which realizes adaptive learning of complex environments and can make the best switching strategy according to different scenarios. The multi-level feature extraction and decision-making process make the system highly interpretable, while ensuring the switching quality, greatly reducing the workload of manual operation, and improving the work efficiency and automation of the director switcher.

[0038] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0039] (1) Obtain the input video stream of the current video source through the director switcher, and perform frame extraction processing on the input video stream to obtain a video frame sequence;

[0040] (2) Calculating the color histogram of each video frame according to the video frame sequence to obtain a color feature vector, and converting the color feature vector to obtain an N-gram feature vector;

[0041] (3) Calculate the cosine similarity between adjacent video frames based on the N-gram feature vector to obtain the inter-frame difference value, and perform adaptive threshold segmentation on the inter-frame difference value to obtain the initial segmentation point set;

[0042] (4) extracting audio features from the video frame sequence according to the initial segmentation point set to obtain an audio feature vector, and performing weighted fusion of the audio feature vector and the inter-frame difference value to obtain a fused feature sequence;

[0043] (5) Dynamically program the fused feature sequence to obtain a target segmentation point set, and segment and inspect the video frame sequence according to the target segmentation point set to obtain video frame segmentation data.

[0044] Specifically, the input video stream of the current video source is obtained through the director switcher, and the video stream is received and processed in real time through the hardware interface and software interface of the director switcher. The video stream is subjected to frame extraction processing, and the extraction interval is usually a fixed time step, such as 30 frames per second, to obtain a video frame sequence. Assuming there is a video stream, a series of frames are obtained after frame extraction, denoted as {F1, F2, F3, …, F n}, where F i represents the i-th frame. The color histogram of each video frame is calculated according to the video frame sequence. Color histogram is a commonly used image feature representation method that describes the color distribution of an image by counting the number of pixels of different colors in the image. For each video frame F i , calculate its color histogram and get the color feature vector H i . A color histogram is usually divided into several color channels, and each channel is further divided into several color levels. For example, for an RGB image, the histograms are calculated for the three channels R, G, and B. The color feature vector is converted to obtain an N-gram feature vector. The N-gram feature vector is a feature representation method based on sequence information that can capture the relationship between continuous elements in a sequence. For each frame's color feature vector H i , construct an N-gram feature vector by concatenating the color feature vectors of N consecutive frames. For example, for N = 3, the feature vector G i =[H i ,H i+1 ,H i+2 ]. Calculate the cosine similarity between adjacent video frames based on the N-gram feature vectors to obtain the inter-frame difference value. Cosine similarity is a commonly used similarity measurement method used to measure the similarity between two vectors. For two N-gram feature vectors G i and G i11 , the cosine similarity calculation formula is:

[0045]

[0046] Among them, G i ·G i+1 Represents vector dot product, ∥G i ∥ and ∥G i +1∥ represents the L2 norm of the vector. The inter-frame difference value can be expressed as:

[0047] diff(G i ,G i+1 )=1-cos_sin(G i ,G i+1 );

[0048] The difference value between each pair of adjacent video frames is calculated by the above formula. The inter-frame difference value is adaptively thresholded to obtain an initial segmentation point set. The adaptive threshold segmentation method dynamically adjusts the segmentation threshold by analyzing the statistical characteristics of the inter-frame difference value to identify frames with obvious changes in video content. The initial segmentation point set represents possible scene switching points. Audio feature extraction is performed on the video frame sequence based on the initial segmentation point set to obtain an audio feature vector. Audio feature extraction can use common methods such as MFCC (Mel-Frequency Cepstrum Coefficient) or STFT (Short-Time Fourier Transform) to effectively extract the frequency domain features of the audio signal. For each initial segmentation point, extract its corresponding audio feature vector, denoted as A i The audio feature vector is weighted and fused with the inter-frame difference value to obtain a fused feature sequence. The importance of audio features and video features is balanced by setting the weight parameter. Assuming that the weight of the audio feature is α and the weight of the inter-frame difference value is β, the fused feature is expressed as:

[0049] F i =α·A i +β·diff(G i ,G i+1 );

[0050] Dynamic programming is performed on the fusion feature sequence to obtain the target segmentation point set, and the video frame sequence is segmented and tested according to the target segmentation point set to obtain video frame segmentation data. Dynamic programming is an optimization algorithm that can find the optimal segmentation scheme in the global scope, and select the scheme with the lowest cost by comparing the cost functions of different segmentation schemes. The target segmentation point set represents the final scene switching point. According to the target segmentation point set, the video frame sequence is segmented and tested to ensure that the video frames in each segment have high similarity and the differences between segments are large, so as to obtain accurate video frame segmentation data.

[0051] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0052] (1) Perform multi-scale feature extraction on the video frame segmentation data to obtain a hierarchical feature descriptor, and construct a multi-level scene representation based on the hierarchical feature descriptor;

[0053] (2) Segment the multi-level scene representation to obtain the initial scene boundary, and perform correlation analysis on adjacent scenes based on the initial scene boundary to obtain a scene correlation matrix;

[0054] (3) Optimizing the scene segmentation of the initial scene boundary according to the scene correlation matrix to obtain an optimized scene segmentation result;

[0055] (4) extracting the key frame sequence of each scene according to the optimized scene segmentation result to obtain the scene representative frame set, and constructing a dynamically updated scene feature dictionary tree of the scene representative frame set based on the regular expression generation algorithm of the dictionary tree;

[0056] (5) Generate an adaptive regular expression template based on the dynamically updated scene feature dictionary tree to obtain a scene matching rule set, and generate a scene recognition strategy based on the scene matching rule set;

[0057] (6) Perform feature matching and encoding on the video frame segmentation data according to the scene recognition strategy to obtain the scene feature signature.

[0058] Specifically, multi-scale feature extraction is performed on the video frame segmentation data to obtain a hierarchical feature descriptor. Feature extraction is performed on video frames at different scales to capture information at different levels. Multi-scale features are obtained using a convolutional neural network at different pooling layers, and these features are combined to form a hierarchical feature descriptor. For example, suppose there is a sequence of video frames, and feature maps of different scales are obtained after feature extraction by a convolutional neural network, denoted as in, Represents the feature map of the i-th frame at scale s. Combining these feature maps, we get the hierarchical feature descriptor D i . Construct a multi-level scene representation based on the hierarchical feature descriptors. The multi-level scene representation combines features of different scales to form a multi-level feature space, which is used to describe the scene information in the video frame. The hierarchical feature descriptors of all frames are combined to form an overall multi-level scene representation matrix R. The multi-level scene representation is segmented to obtain the initial scene boundary. The segmentation process can be implemented by a clustering algorithm, such as K-means or DBSCAN, which clusters the feature space and segments similar frames into the same scene. Assuming that K means clustering is used, the initial scene boundary can be expressed as {B1,B2,…,B )}, where B i Represents the boundary of the i-th scene. According to the initial scene boundary, the correlation analysis of adjacent scenes is performed to obtain the scene correlation matrix. The scene correlation matrix is ​​used to describe the similarity between adjacent scenes, which is achieved by calculating the cosine similarity between scene boundary features. Assume S i and S + is the feature vector of two adjacent scenes, then their similarity can be expressed as:

[0059]

[0060] By calculating the similarity of all adjacent scene pairs, the scene correlation matrix M is obtained. The scene segmentation is optimized for the initial scene boundary according to the scene correlation matrix to obtain the optimized scene segmentation result. The optimization can be achieved by dynamic programming or graph cut algorithm to minimize the cost function in the similarity matrix to ensure that the internal similarity of each scene is maximized and the difference between different scenes is maximized.

[0061] ″″

[0062] The optimized scene segmentation result can be expressed as {B1,B2,…,B1} ′ Among them, B i represents the boundary of the optimized i-th scene. According to the optimized scene segmentation results, the key frame sequence of each scene is extracted to obtain the scene representative frame set. The key frame refers to the frame that can represent the characteristics of the entire scene. It can be determined by selecting the frame with the largest feature change or the most representative frame in the scene. Assume that the scene S i The key frames are {K1, K2,…, K3}. A regular expression generation algorithm based on a dictionary tree constructs a dynamically updated scene feature dictionary tree for the scene representative frame set. The dictionary tree is an efficient structured data storage method used to store and match feature sequences. By inserting the feature vectors in the scene representative frame set into the dictionary tree and dynamically updating it when a new scene representative frame appears, a dynamically updated scene feature dictionary tree T is formed. An adaptive regular expression template is generated based on the dynamically updated scene feature dictionary tree to obtain a scene matching rule set. The adaptive regular expression template automatically generates a regular expression that can match the scene features by analyzing the feature patterns in the dictionary tree. The scene matching rule set can be expressed as {R1, R2,…, R5}, where R i It is a regular expression that matches the characteristics of a specific scene. A scene recognition strategy is generated based on the scene matching rule set. The scene recognition strategy matches and encodes the input video frame segmentation data by applying a regular expression template to identify the scene category to which each video frame belongs. The video frame segmentation data is feature matched and encoded according to the scene recognition strategy to obtain a scene feature signature. The scene feature signature is a high-level feature representation that can effectively describe the scene information of the video content.

[0063] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0064] (1) Perform multi-level feature construction and adaptive feature screening on each video frame in the input video stream according to the scene feature signature to obtain a scene-related feature set;

[0065] (2) Performing dynamic convolution operation and feature extraction on the scene-related feature set to obtain a scene-adaptive feature map, and performing temporal difference calculation on the scene-adaptive feature map to obtain inter-frame change features;

[0066] (3) Calculate the dynamic weight matrix based on the inter-frame change characteristics, and perform weighted fusion on the scene adaptability feature map and the dynamic weight matrix to obtain enhanced spatiotemporal features;

[0067] (4) Generate a temporal feature sequence based on the enhanced spatiotemporal features, and perform multi-scale spatiotemporal feature aggregation on the temporal feature sequence to obtain a scene spatiotemporal fusion feature set.

[0068] Specifically, multi-level feature construction and adaptive feature screening are performed on each video frame in the input video stream according to the scene feature signature to obtain a scene-related feature set. Multi-level feature construction captures various feature information in the video frame through feature extraction networks at different levels, such as convolutional neural networks. Assume that the input video stream is {V1, V2, V3, …, V n}, where V i Represents the i-th frame. Through the multi-layer convolution operation of the convolutional neural network, the feature map set F of each video frame is obtained. i These feature maps capture different visual features at different levels. Through the adaptive feature screening method, these features are screened and those that are highly correlated with the scene feature signature are selected to form the scene-related feature set S i The scene-related feature set is subjected to dynamic convolution operation and feature extraction to obtain a scene-adaptive feature map. Dynamic convolution operation refers to applying convolution kernels at different time periods to capture the temporal changes of video frames. Assume that the convolution kernel W is used to extract the feature set S. i Perform convolution operation to obtain the scene adaptability feature map C i , its formula is expressed as:

[0069] C i =W*S i ;

[0070] Among them, * represents the convolution operation. Through the dynamic convolution operation, the local pattern and change trend of the video frame in the time dimension are effectively extracted. The scene adaptive feature map is subjected to temporal difference calculation to obtain the inter-frame change characteristics. The temporal difference calculation reveals the dynamic characteristics of the video frame changing over time by comparing the feature changes between consecutive frames. Assume that C i and C i+1 are the scene adaptability feature maps of adjacent frames, and the temporal difference calculation formula is:

[0071] ΔC i =C i+1 -C i ;

[0072] Where, ΔC iRepresents the difference feature between the i-th frame and the i+1-th frame. Through calculation, the inter-frame change feature between each pair of adjacent frames is obtained. The dynamic weight matrix is ​​calculated based on the inter-frame change feature, and the scene adaptive feature map and the dynamic weight matrix are weighted fused to obtain the enhanced spatiotemporal feature. The dynamic weight matrix is ​​used to measure the importance of each feature in the time dimension. Assume that the dynamic weight matrix is ​​W : , then the formula for weighted fusion is:

[0073] E i =W : ·C i ;

[0074] Among them, E i Represents enhanced spatiotemporal features. Through weighted fusion, the features that change significantly in the time dimension are highlighted to obtain more representative spatiotemporal features. A temporal feature sequence is generated based on the enhanced spatiotemporal features, and multi-scale spatiotemporal feature aggregation is performed on the temporal feature sequence to obtain a scene spatiotemporal fusion feature set. The temporal feature sequence is formed by arranging the enhanced spatiotemporal features in chronological order, denoted as {E1, E2, E3, …, E n Multi-scale spatiotemporal feature aggregation aggregates features at different time scales to capture long-term and short-term feature changes. Assuming that a multi-scale aggregation function P is used, the formula for the scene spatiotemporal fusion feature set is:

[0075] F ％＝ =P(E1,E2,…,E n );

[0076] Among them, F ％＝ Represents the spatiotemporal fusion feature set of the scene. The multi-scale aggregation function can be a pooling operation based on a time window, or an aggregation method based on a long short-term memory network (LSTM) or an attention mechanism.

[0077] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0078] (1) Input the scene spatiotemporal fusion feature set into the LSTM-based lightweight temporal convolutional network model, and perform input tensor processing on the scene spatiotemporal fusion feature set to obtain a T×D-dimensional input tensor, where T is the time step and D is the feature dimension;

[0079] (2) The T×D-dimensional input tensor is input into a multi-channel one-dimensional convolutional layer, and convolution operations are performed through F convolution kernels of different sizes to obtain a T×F-dimensional local temporal feature representation;

[0080] (3) Perform a maximum pooling operation on the T×F dimensional local temporal feature representation to obtain a (T / s)×F dimensional reduced dimension feature map, where the pooling window size is k and the step size is s;

[0081] (4) The (T / s)×F dimensional reduced feature map is input into the residual block, and the (T / s)×F dimensional multi-scale temporal features are output. The residual block contains two layers of one-dimensional convolution and one skip connection.

[0082] (5) Perform channel attention calculation on the (T / s)×F dimensional multi-scale time series features, generate an F dimensional weight vector and perform element-by-element multiplication to obtain a (T / s)×F dimensional weighted feature map;

[0083] (6) The (T / s)×F-dimensional weighted feature map is reshaped into a (T / s)F-dimensional vector sequence, which is input into the LSTM unit. The H-dimensional hidden state is output at each time step to obtain a (T / s)×H-dimensional long-range dependency feature.

[0084] (7) Perform self-attention mechanism analysis on the (T / s)×H dimensional long-range dependency features, calculate the (T / s)×(T / s) attention matrix and multiply it to obtain the (T / s)×H dimensional context enhancement features;

[0085] (8) Input the (T / s)×H dimensional context-enhanced features into the gated linear unit, control the information flow through the sigmoid gating mechanism, and output the (T / s)×H dimensional candidate switching behavior features;

[0086] (9) Perform multi-task learning head analysis on the (T / s)×H dimensional candidate switching behavior features, including N parallel fully connected layers, each fully connected layer outputs a Mi-dimensional prediction result, and obtains N prediction vectors of different dimensions, where i=1, 2, ..., N;

[0087] (10) According to the predefined task weights, the prediction vectors of N different dimensions are weighted and summed, and normalized by the softmax function to obtain the video switching behavior prediction data.

[0088] Specifically, the scene spatiotemporal fusion feature set is input into the LSTM-based lightweight temporal convolutional network model, and the scene spatiotemporal fusion feature set is processed as an input tensor to obtain a T×D-dimensional input tensor, where T is the time step and D is the feature dimension. Suppose there is a scene spatiotemporal fusion feature set {F1, F2, …, F T}, each eigenvector F i The dimension of is D, then the input tensor X can be expressed as:

[0089] X=[F1,F2,…,F T ];

[0090] The dimension of X is T×D. The T×D-dimensional input tensor is fed into a multi-channel one-dimensional convolutional layer, and convolution is performed through F convolution kernels of different sizes to obtain a T×F-dimensional local temporal feature representation. The multi-channel convolutional layer extracts local temporal features of different scales through convolution kernels of different sizes. For example, using k1, k2, …, k B These convolution kernels of different sizes perform convolution operations on the input tensor X to obtain a local temporal feature representation C with a dimension of T×F. The T×F-dimensional local temporal feature representation is subjected to a maximum pooling operation to obtain a (T / s)×F-dimensional reduced-dimensional feature map, where the pooling window size is k and the step size is s. The maximum pooling operation reduces the feature dimension by taking the maximum value within the pooling window while retaining important feature information. Assuming the pooling window size is k and the step size is s, the dimension of the reduced-dimensional feature map P is (T / s)×F. The (T / s)×F-dimensional reduced-dimensional feature map is input into the residual block, and the (T / s)×F-dimensional multi-scale temporal features are output. The residual block contains two layers of one-dimensional convolution and a skip connection. The residual connection solves the gradient vanishing problem in the deep network and effectively extracts multi-scale temporal features. Assume that the input of the residual block is P and the output is R, whose dimension is (T / s)×F. The channel attention is calculated for the (T / s)×F dimensional multi-scale time series features, an F dimensional weight vector is generated and element-by-element multiplication is performed to obtain a (T / s)×F dimensional weighted feature map. The channel attention mechanism highlights key features by assigning different weights to each feature channel. Assume that the weight vector is W D , the weighted feature map is A, and its calculation formula is:

[0091] A=R⊙W D ;

[0092] Where ⊙ represents element-by-element multiplication. The (T / s)×F-dimensional weighted feature map is reshaped into a (T / s) F-dimensional vector sequence, input into the LSTM unit, and output the H-dimensional hidden state at each time step to obtain the (T / s)×H-dimensional long-range dependency feature. LSTM captures the long-range dependency relationship in time series data through its memory unit. Assume that the output of the LSTM unit is L, and its dimension is (T / s)×H. The self-attention mechanism is analyzed for the (T / s)×H-dimensional long-range dependency feature, and the (T / s)×(T / s) attention matrix is ​​calculated and multiplied to obtain the (T / s)×H-dimensional context-enhanced feature. The self-attention mechanism enhances the feature representation by calculating the correlation between each position in the sequence. Assume that the attention matrix is ​​A ＝ , the context enhancement feature is E D , and its calculation formula is:

[0093] E D =A ＝ ·L;

[0094] Among them, A＝ The dimension of is (T / s)×(T / s). The (T / s)×H dimensional context enhancement feature is input into the gated linear unit, and the information flow is controlled by the sigmoid gating mechanism, and the (T / s)×H dimensional candidate switching behavior feature is output. The gated linear unit controls the flow of information through the sigmoid function, retains important information, and suppresses irrelevant information. Assume that the output of the gated linear unit is G, and its dimension is (T / s)×H. The (T / s)×H dimensional candidate switching behavior feature is analyzed by a multi-task learning head, including N parallel fully connected layers, each fully connected layer outputs M i dimensional prediction results, and obtain prediction vectors of N different dimensions, where i = 1, 2, ..., N. Assume that the output of each fully connected layer is O i , whose dimension is M i According to the predefined task weights, the prediction vectors of N different dimensions are weighted and summed, and normalized by the softmax function to obtain the video switching behavior prediction data. Assume that the task weight is w i , the weighted sum is O, and the calculation formula is:

[0095]

[0096] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0097] (1) Obtaining the state information of the current video source, and concatenating the video switching behavior prediction data and the state information of the current video source to obtain an S-dimensional state vector;

[0098] (2) Input the S-dimensional state vector into the state encoder network, pass through two fully connected layers and the ReLU activation function, and obtain the E-dimensional encoded state representation;

[0099] (3) Perform attention mechanism processing on the E-dimensional encoded state representation, calculate the similarity with the historical state, and obtain the A-dimensional attention weight vector;

[0100] (4) Perform weighted summation of historical states according to the A-dimensional attention weight vector to obtain an H-dimensional historical context vector;

[0101] (5) The E-dimensional encoded state representation and the H-dimensional historical context vector are concatenated and input into the value network in the dual deep Q-learning model based on the Markov decision process. The scalar state value estimation is obtained through three layers of fully connected layers and LeakyReLU activation function.

[0102] (6) The E-dimensional encoded state representation is processed by the action generation network, and a K-dimensional candidate action vector is obtained through two fully connected layers and the Tanh activation function;

[0103] (7) The K-dimensional candidate action vector and the E-dimensional encoding state representation are concatenated and input into the advantage function network in the dual-depth Q-learning model based on the Markov decision process. Through three layers of fully connected layers and the ELU activation function, the K-dimensional action advantage value is obtained;

[0104] (8) Based on the scalar state value estimate and the K-dimensional action advantage value, the K-dimensional Q-value vector is calculated using the double Q learning formula;

[0105] (9) Performing a Softmax function operation with a temperature parameter of τ on the K-dimensional Q-value vector to obtain a K-dimensional action probability distribution;

[0106] (10) Importance sampling is performed according to the K-dimensional action probability distribution, the optimal action index is selected, and the corresponding video source switching decision result is obtained by looking up the table.

[0107] Specifically, obtain the state information of the current video source, including the frame rate, resolution, current scene features, etc. of the video stream. Concatenate the state information with the video switching behavior prediction data to obtain an S-dimensional state vector. Assuming that the state information of the current video source is V-dimensional and the video switching behavior prediction data is P-dimensional, then S=V+P, and the concatenated state vector is denoted as s. Input the S-dimensional state vector s into the state encoder network, and obtain the E-dimensional encoded state representation through two fully connected layers and the ReLU activation function. The output of the first fully connected layer is:

[0108] h1=ReLU(W1s+b1);

[0109] Among them, W1 and b1 are the weight matrix and bias vector of the first layer respectively. The output of the second fully connected layer is the encoding state representation:

[0110] h2=ReLU(W2h1+b2);

[0111] Among them, W2 and b2 are the weight matrix and bias vector of the second layer respectively. The E-dimensional encoded state representation h2 is processed by the attention mechanism, and the similarity with the historical state is calculated to obtain the A-dimensional attention weight vector. Assume that the historical state is represented by {h 2,1 ,h 2,2 ,…,h 2,T}, where T is the number of historical time steps, the calculation formula of the attention weight vector a is:

[0112]

[0113] Where · represents the vector dot product. The historical states are weighted and summed according to the A-dimensional attention weight vector a to obtain the H-dimensional historical context vector h D The calculation formula is:

[0114]

[0115] The E-dimensional encoded state representation h2 and the H-dimensional historical context vector h D After concatenation, the value network in the dual-depth Q-learning model based on the Markov decision process is input, and the scalar state value estimate is obtained through three layers of fully connected layers and the LeakyReLU activation function. The concatenated vector is denoted as h D\nD]＝ , the output of the first fully connected layer is:

[0116] h3=LeakyReLU(W3h D\nD]＝ +b3);

[0117] The output of the second fully connected layer is:

[0118] h _ =LeakyReLU(W _ h3+b _ );

[0119] The output of the third fully connected layer is a scalar state value estimate V:

[0120] V=W a h _ +b a ;

[0121] At the same time, the E-dimensional encoded state representation h2 is processed by the action generation network, and a K-dimensional candidate action vector is obtained through two layers of fully connected layers and Tanh activation function. The output of the first fully connected layer is:

[0122] h a =Tanh(W b h2+b b );

[0123] The output of the second fully connected layer is the candidate action vector A:

[0124] A=Tanh(W c h a +b c );

[0125] The K-dimensional candidate action vector A and the E-dimensional encoding state representation h2 are concatenated and input into the advantage function network in the dual-depth Q learning model based on the Markov decision process. After passing through three layers of fully connected layers and the ELU activation function, the K-dimensional action advantage value is obtained. The concatenated vector is denoted as h D\nD]＝2 , the output of the first fully connected layer is:

[0126] h b =ELU(W d h D\nD]＝2 +b d );

[0127] The output of the second fully connected layer is:

[0128] h c =ELU(W e h b +b e );

[0129] The output of the third fully connected layer is the action advantage value A f :

[0130] A f =W 1g h c +b 1g ;

[0131] Estimate V and K-dimensional action advantage value A based on scalar state value f , use the double Q learning formula to calculate the K-dimensional Q value vector. The double Q learning formula is:

[0132]

[0133] Among them, Q(s,a) represents the Q value of action a in state s, A f [k] represents the advantage value of the k-th action. The Softmax function with a temperature parameter of τ is performed on the K-dimensional Q value vector to obtain the K-dimensional action probability distribution. The Softmax function formula is:

[0134]

[0135] Among them, P(a i ) indicates action a i The probability of τ is the temperature parameter. According to the K-dimensional action probability distribution, importance sampling is performed, the optimal action index is selected, and the corresponding video source switching decision result is obtained by looking up the table. For example, assuming that the K-dimensional action probability distribution is {P(a1), P(a2), …, P(a j )}, select the action index a with the highest probability through importance sampling * , the corresponding video source switching decision result is D(a * ).

[0136] In a specific embodiment, the execution step concatenates the K-dimensional candidate action vector and the E-dimensional encoding state representation, inputs the advantage function network in the double-depth Q learning model based on the Markov decision process, and obtains the K-dimensional action advantage value through three layers of fully connected layers and the ELU activation function. The process can specifically include the following steps:

[0137] (1) Concatenate the K-dimensional candidate action vector and the E-dimensional encoded state representation to obtain a (K+E)-dimensional input feature vector;

[0138] (2) Perform matrix multiplication on the first fully connected layer of the advantage function network according to the (K+E)-dimensional input feature vector to obtain an M-dimensional intermediate feature vector;

[0139] (3) performing batch normalization on the M-dimensional intermediate feature vector to obtain a standardized M-dimensional feature vector;

[0140] (4) performing ELU activation function calculation based on the standardized M-dimensional feature vector to obtain an M-dimensional nonlinear feature vector;

[0141] (5) Perform a dropout operation on the M-dimensional nonlinear feature vector to obtain an M-dimensional sparse feature vector;

[0142] (6) performing matrix multiplication on the second fully connected layer of the advantage function network according to the M-dimensional sparse feature vector to obtain an N-dimensional intermediate feature vector;

[0143] (7) performing a layer normalization operation on the N-dimensional intermediate feature vector to obtain a normalized N-dimensional feature vector;

[0144] (8) performing ELU activation function calculation based on the normalized N-dimensional feature vector to obtain an N-dimensional nonlinear feature vector;

[0145] (9) performing residual connection on the N-dimensional nonlinear feature vector and performing element-wise addition on the (K+E)-dimensional input feature vector to obtain an N-dimensional residual feature vector;

[0146] (10) According to the N-dimensional residual feature vector, matrix multiplication operation is performed on the third fully connected layer of the advantage function network, and the ELU activation function calculation is performed to obtain the K-dimensional action advantage value.

[0147] Specifically, the K-dimensional candidate action vector and the E-dimensional encoding state representation are concatenated to obtain a (K+E)-dimensional input feature vector. Assuming that the candidate action vector is A, the encoding state representation is H, and the concatenated input feature vector is I, then:

[0148] I = [A; H];

[0149] Among them, the dimension of I is (K+E). According to the (K+E)-dimensional input feature vector I, the first fully connected layer of the advantage function network is matrix multiplied to obtain an M-dimensional intermediate feature vector. Assuming that the weight matrix of the first fully connected layer is W1 and the bias vector is b1, the calculation formula of the intermediate feature vector M1 is:

[0150] M1=W1I+b1;

[0151] Batch normalization is performed on the M-dimensional intermediate feature vector M1 to obtain a standardized M-dimensional feature vector. Batch normalization reduces the deviation of data distribution by standardizing the mean and variance of each feature. The ELU activation function is calculated based on the standardized M-dimensional feature vector to obtain an M-dimensional nonlinear feature vector. The ELU activation function enables the network to better represent complex feature relationships by introducing nonlinearity. The dropout operation is performed on the M-dimensional nonlinear feature vector to obtain an M-dimensional sparse feature vector. Dropout prevents overfitting of the model and improves its generalization ability by randomly discarding some neurons. Matrix multiplication is performed on the second fully connected layer of the advantage function network based on the M-dimensional sparse feature vector to obtain an N-dimensional intermediate feature vector. Assuming that the weight matrix of the second fully connected layer is W2 and the bias vector is b2, the calculation formula for the intermediate feature vector z2 is:

[0152] z2=W2·d1+b2;

[0153] Perform layer normalization on the N-dimensional intermediate feature vector to obtain a normalized N-dimensional feature vector. Layer normalization further improves the training effect of the model by standardizing the feature vector so that its mean is 0 and its variance is 1. Perform ELU activation function calculation based on the normalized N-dimensional feature vector to obtain an N-dimensional nonlinear feature vector. Perform residual connection on the N-dimensional nonlinear feature vector and perform element-wise addition with the (K+E)-dimensional input feature vector to obtain an N-dimensional residual feature vector. Residual connection reduces the gradient vanishing problem by adding the input directly to the output. Perform matrix multiplication on the third fully connected layer of the advantage function network based on the N-dimensional residual feature vector and perform ELU activation function calculation to obtain the K-dimensional action advantage value.

[0154] The above describes the intelligent video source switching method for the director switcher in the embodiment of the present application. The following describes the intelligent video source switching system for the director switcher in the embodiment of the present application. Figure 2 In one embodiment of the present application, an intelligent video source switching system for a director switch station includes:

[0155] The acquisition module 201 is used to acquire the input video stream of the current video source through the director switcher, and perform N-gram feature vector calculation and frame difference analysis on the input video stream to obtain video frame segmentation data;

[0156] An analysis module 202 is used to perform scene analysis on the video frame segmentation data based on a regular expression generation algorithm of a dictionary tree to obtain a scene feature signature;

[0157] A fusion module 203 is used to extract and fuse spatiotemporal features of each video frame in the input video stream according to the scene feature signature to obtain a scene spatiotemporal fusion feature set;

[0158] The prediction module 204 is used to input the scene spatiotemporal fusion feature set into the LSTM-based lightweight temporal convolutional network model to predict the video switching behavior and obtain the video switching behavior prediction data;

[0159] The decision module 205 is used to input the video switching behavior prediction data into the dual deep Q learning model based on the Markov decision process to make a video source switching decision and obtain a video source switching decision result.

[0160] Through the collaboration of the above components, the N-gram feature vector calculation and inter-frame difference analysis can achieve fine segmentation of video content, which helps to improve the accuracy and coherence of video source switching. The regular expression generation algorithm based on the dictionary tree can adaptively generate scene matching rules, greatly improving the flexibility and robustness of scene recognition, so that the system can better adapt to different types of program content. The spatiotemporal feature extraction and fusion technology fully considers the temporal information and spatial characteristics of the video, so that the system can understand the video content more comprehensively and make more reasonable switching decisions. The lightweight temporal convolutional network model based on LSTM effectively combines the long-term and short-term memory capabilities and local feature extraction capabilities, can accurately predict video switching behavior, and improve the real-time and prediction accuracy of the system. The dual-depth Q learning model based on the Markov decision process is used for video source switching decisions, which realizes adaptive learning in complex environments and can make the optimal switching strategy according to different scenarios. The multi-level feature extraction and decision-making process make the system highly interpretable. While ensuring the switching quality, it greatly reduces the workload of manual operations and improves the work efficiency and automation of the director switcher.

[0161] The present application also provides an intelligent video source switching device for a director switch station, the intelligent video source switching device for a director switch station includes a memory and a processor, the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the intelligent video source switching method for a director switch station in the above-mentioned embodiments.

[0162] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the intelligent video source switching method for a director switcher.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0165] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent video source switching method for a director switcher, characterized in that: The method comprises: The input video stream of the current video source is obtained through the director switcher, and the N-gram feature vector calculation and inter-frame difference analysis are performed on the input video stream to obtain video frame segmentation data; Performing scene analysis on the video frame segmentation data based on a regular expression generation algorithm of a dictionary tree to obtain a scene feature signature; Extract and fuse spatiotemporal features of each video frame in the input video stream according to the scene feature signature to obtain a scene spatiotemporal fusion feature set; Inputting the scene spatiotemporal fusion feature set into a LSTM-based lightweight temporal convolutional network model to predict video switching behavior, thereby obtaining video switching behavior prediction data; The video switching behavior prediction data is input into a dual-depth Q learning model based on a Markov decision process to make a video source switching decision, and a video source switching decision result is obtained.

2. The intelligent video source switching method for a director switcher according to claim 1, characterized in that: The step of obtaining the input video stream of the current video source through the director switcher, and performing N-gram feature vector calculation and frame difference analysis on the input video stream to obtain video frame segmentation data includes: The input video stream of the current video source is obtained through the director switcher, and frame extraction processing is performed on the input video stream to obtain a video frame sequence; Calculating a color histogram for each video frame according to the video frame sequence to obtain a color feature vector, and converting the color feature vector to obtain an N-gram feature vector; Calculating the cosine similarity between adjacent video frames according to the N-gram feature vector to obtain an inter-frame difference value, and performing adaptive threshold segmentation on the inter-frame difference value to obtain an initial segmentation point set; Extract audio features from the video frame sequence according to the initial segmentation point set to obtain an audio feature vector, and perform weighted fusion of the audio feature vector and the inter-frame difference value to obtain a fused feature sequence; Dynamic programming is performed on the fused feature sequence to obtain a target segmentation point set, and the video frame sequence is segmented and tested according to the target segmentation point set to obtain video frame segmentation data.

3. The intelligent video source switching method for a director switcher according to claim 1, characterized in that: The regular expression generation algorithm based on the dictionary tree performs scene analysis on the video frame segmentation data to obtain a scene feature signature, including: Performing multi-scale feature extraction on the video frame segmentation data to obtain a hierarchical feature descriptor, and constructing a multi-level scene representation according to the hierarchical feature descriptor; Segmenting the multi-level scene representation to obtain an initial scene boundary, and performing correlation analysis on adjacent scenes according to the initial scene boundary to obtain a scene correlation matrix; Performing scene segmentation optimization on the initial scene boundary according to the scene correlation matrix to obtain an optimized scene segmentation result; Extracting a key frame sequence of each scene according to the optimized scene segmentation result to obtain a scene representative frame set, and constructing a dynamically updated scene feature dictionary tree of the scene representative frame set based on a regular expression generation algorithm of the dictionary tree; Generate an adaptive regular expression template according to the dynamically updated scene feature dictionary tree to obtain a scene matching rule set, and generate a scene recognition strategy according to the scene matching rule set; The video frame segmentation data is feature matched and encoded according to the scene recognition strategy to obtain a scene feature signature.

4. The intelligent video source switching method for a director switcher according to claim 1, characterized in that: The step of extracting and fusing spatiotemporal features of each video frame in the input video stream according to the scene feature signature to obtain a scene spatiotemporal fusion feature set includes: Performing multi-level feature construction and adaptive feature screening on each video frame in the input video stream according to the scene feature signature to obtain a scene-related feature set; Performing dynamic convolution operation and feature extraction on the scene-related feature set to obtain a scene-adaptive feature map, and performing temporal difference calculation on the scene-adaptive feature map to obtain inter-frame change features; Calculating a dynamic weight matrix according to the inter-frame variation feature, and performing weighted fusion on the scene adaptability feature map and the dynamic weight matrix to obtain an enhanced spatiotemporal feature; A temporal feature sequence is generated according to the enhanced spatiotemporal feature, and multi-scale spatiotemporal feature aggregation is performed on the temporal feature sequence to obtain a scene spatiotemporal fusion feature set.

5. The intelligent video source switching method for a director switcher according to claim 1, characterized in that: The step of inputting the scene spatiotemporal fusion feature set into a lightweight temporal convolutional network model based on LSTM to predict video switching behavior and obtain video switching behavior prediction data includes: Inputting the scene spatiotemporal fusion feature set into a lightweight temporal convolutional network model based on LSTM, and performing input tensor processing on the scene spatiotemporal fusion feature set to obtain a T×D-dimensional input tensor, where T is the time step and D is the feature dimension; Inputting the T×D-dimensional input tensor into a multi-channel one-dimensional convolutional layer, performing a convolution operation through F convolution kernels of different sizes, and obtaining a T×F-dimensional local temporal feature representation; Performing a maximum pooling operation on the T×F dimensional local temporal feature representation to obtain a (T / s)×F dimensional reduced dimensionality feature map, where the pooling window size is k and the step size is s; Input the (T / s)×F dimensional reduced dimension feature map into a residual block, and output a (T / s)×F dimensional multi-scale temporal feature, wherein the residual block includes two layers of one-dimensional convolution and one skip connection; Perform channel attention calculation on the (T / s)×F dimensional multi-scale time series features, generate an F dimensional weight vector and perform element-by-element multiplication to obtain a (T / s)×F dimensional weighted feature map; Reshape the (T / s)×F-dimensional weighted feature map into a (T / s) F-dimensional vector sequence, input it into the LSTM unit, output an H-dimensional hidden state at each time step, and obtain a (T / s)×H-dimensional long-range dependency feature; Perform a self-attention mechanism analysis on the (T / s)×H dimensional long-range dependency features, calculate the (T / s)×(T / s) attention matrix and multiply them to obtain the (T / s)×H dimensional context enhancement features; The (T / s)×H dimensional context enhancement features are input into a gated linear unit, the information flow is controlled by a sigmoid gating mechanism, and a (T / s)×H dimensional candidate switching behavior feature is output; The (T / s)×H dimensional candidate switching behavior features are subjected to a multi-task learning head analysis, including N parallel fully connected layers, each fully connected layer outputs M i dimensional prediction results, and obtain prediction vectors of N different dimensions, where i = 1, 2, ..., N; The prediction vectors of the N different dimensions are weighted and summed according to the predefined task weights, and normalized by a softmax function to obtain the video switching behavior prediction data.

6. The intelligent video source switching method for a director switcher according to claim 1, characterized in that: The step of inputting the video switching behavior prediction data into a dual-depth Q learning model based on a Markov decision process to make a video source switching decision, and obtaining a video source switching decision result, includes: Acquire the state information of the current video source, and concatenate the video switching behavior prediction data and the state information of the current video source to obtain an S-dimensional state vector; Input the S-dimensional state vector into the state encoder network, and obtain the E-dimensional encoded state representation through two fully connected layers and ReLU activation function; Performing attention mechanism processing on the E-dimensional encoded state representation, calculating the similarity with the historical state, and obtaining an A-dimensional attention weight vector; Performing weighted summation of historical states according to the A-dimensional attention weight vector to obtain an H-dimensional historical context vector; The E-dimensional encoded state representation and the H-dimensional historical context vector are concatenated and input into the value network in the dual-depth Q-learning model based on the Markov decision process, and a scalar state value estimation is obtained through three layers of fully connected layers and LeakyReLU activation function; Performing action generation network processing on the E-dimensional encoded state representation, obtaining a K-dimensional candidate action vector through two fully connected layers and a Tanh activation function; The K-dimensional candidate action vector and the E-dimensional encoding state representation are concatenated and input into the advantage function network in the double-depth Q learning model based on the Markov decision process, and the K-dimensional action advantage value is obtained through three layers of fully connected layers and ELU activation function; According to the scalar state value estimate and the K-dimensional action advantage value, a K-dimensional Q value vector is calculated using a double Q learning formula; Performing a Softmax function operation with a temperature parameter of τ on the K-dimensional Q-value vector to obtain a K-dimensional action probability distribution; Importance sampling is performed according to the K-dimensional action probability distribution, an optimal action index is selected, and a table is looked up to obtain a corresponding video source switching decision result.

7. The intelligent video source switching method for a director switcher according to claim 6, characterized in that: The K-dimensional candidate action vector and the E-dimensional encoding state representation are concatenated and input into the advantage function network in the double-depth Q learning model based on the Markov decision process, and the K-dimensional action advantage value is obtained through three layers of fully connected layers and ELU activation function, including: Performing a concatenation operation on the K-dimensional candidate action vector and the E-dimensional encoding state representation to obtain a (K+E)-dimensional input feature vector; Performing a matrix multiplication operation on the first fully connected layer of the advantage function network according to the (K+E)-dimensional input feature vector to obtain an M-dimensional intermediate feature vector; Performing batch normalization processing on the M-dimensional intermediate feature vector to obtain a standardized M-dimensional feature vector; Performing ELU activation function calculation according to the standardized M-dimensional feature vector to obtain an M-dimensional nonlinear feature vector; Performing a dropout operation on the M-dimensional nonlinear feature vector to obtain an M-dimensional sparse feature vector; Performing a matrix multiplication operation on the second fully connected layer of the advantage function network according to the M-dimensional sparse feature vector to obtain an N-dimensional intermediate feature vector; Performing a layer normalization operation on the N-dimensional intermediate feature vector to obtain a normalized N-dimensional feature vector; Performing ELU activation function calculation according to the normalized N-dimensional feature vector to obtain an N-dimensional nonlinear feature vector; Performing residual connection on the N-dimensional nonlinear feature vector and performing element-wise addition on the (K+E)-dimensional input feature vector to obtain an N-dimensional residual feature vector; A matrix multiplication operation is performed on the third fully connected layer of the advantage function network according to the N-dimensional residual feature vector, and an ELU activation function calculation is performed to obtain a K-dimensional action advantage value.

8. An intelligent video source switching system for a director switcher, characterized in that: The system is used to execute the intelligent video source switching method for a director switcher according to any one of claims 1 to 7, and comprises: An acquisition module is used to acquire the input video stream of the current video source through the director switcher, and perform N-gram feature vector calculation and inter-frame difference analysis on the input video stream to obtain video frame segmentation data; An analysis module, used for performing scene analysis on the video frame segmentation data based on a regular expression generation algorithm of a dictionary tree to obtain a scene feature signature; A fusion module, used for extracting and fusing spatiotemporal features of each video frame in the input video stream according to the scene feature signature to obtain a scene spatiotemporal fusion feature set; A prediction module, used for inputting the scene spatiotemporal fusion feature set into a lightweight temporal convolutional network model based on LSTM to predict video switching behavior and obtain video switching behavior prediction data; The decision module is used to input the video switching behavior prediction data into a dual-depth Q learning model based on a Markov decision process to make a video source switching decision and obtain a video source switching decision result.

9. An intelligent video source switching device for a director switcher, characterized in that: The intelligent video source switching device for a director switch station comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the intelligent video source switching device for a director switcher executes the intelligent video source switching method for a director switcher as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, an intelligent video source switching method for a director switch station as described in any one of claims 1-7 is implemented.

Images