A director's view angle assisted switching method

Abstract

The application provides a director view angle auxiliary switching method, the method comprises the following steps: collecting video and audio data, pre-processing, and storing in a data set; for the video data in the data set, applying a convolutional neural network to extract video feature information, for the audio data in the data set, applying an LSTM network and combining Mel frequency spectrum coefficient and short-time Fourier transform to extract audio feature information; using the video feature information and the audio feature information to train a decision neural network, and using historical director station data and artificially annotated best switching nodes to optimize the decision neural network; using the trained decision neural network to receive real-time video and audio data to assist the switching of the director view angle.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data processing systems, and particularly relates to a director's perspective assisted switching method. BACKGROUND

[0002] Live, video and audio playback platforms such as recording platforms are popular among the public as mainstream media platforms, and the recording of programs requires the director and subsequent editing to present a better viewing experience to the audience. At present, the director adjusts and switches through a large number of staff at the director's console to change the audio and video perspective, but manual director may face some defects in actual operation:

[0003] Manual director needs to quickly judge and operate when switching perspectives, and in complex or rapidly changing scenes, the best switching opportunity may be missed due to insufficient reaction; long-term high-intensity work can easily lead to director fatigue, making it difficult to concentrate for a long time, thereby increasing the likelihood of operational errors; the director needs to monitor multiple video sources and audio signals at the same time, making quick judgments and switching, which is a great challenge to their multitasking ability, and some information may be ignored; the director's effect largely depends on the experience and intuition of the director, and novice directors may need a long time to learn and practice to reach a proficient level; different directors may have different judgments and preferences for the same scene, resulting in strong subjectivity in perspective switching, which may not meet the expectations or habits of the audience; in the event of unexpected events or accidents, the director may not have enough time to make a comprehensive analysis and decision, thereby affecting the quality of the program. SUMMARY

[0004] Therefore, the present application aims to provide a director's perspective assisted switching method to at least solve some of the above technical problems.

[0005] To achieve the above-mentioned purpose, the technical scheme of the present application is as follows:

[0006] A director's perspective assisted switching method, the method comprising:

[0007] Collecting video and audio data, pre-processing and storing in a data set;

[0008] For video data in the data set, applying a convolutional neural network to extract video feature information, and for audio data in the data set, applying an LSTM network and combining Mel frequency coefficients and short-time Fourier transform to extract audio feature information;

[0009] Using video feature information and audio feature information to train a decision neural network, and using historical director console data and manually annotated best switching nodes to optimize the decision neural network;

[0010] The trained decision neural network receives real-time video and audio data to assist the director in switching the view angle.

[0011] Further, the pre-processing of the video and audio data includes:

[0012] The collected data is classified and packaged according to its source to obtain data sets belonging to different devices, and the data sets of different devices are associated through keywords;

[0013] Duplicate data in the data set is identified and deleted, and the missing data in the data set is filled using the mean method;

[0014] Data with the same keywords in each data set is converted to a unified format.

[0015] Further, the process of extracting video feature information using a convolutional neural network includes:

[0016] The video data is decomposed into a continuous frame sequence, each frame is standardized, and the feature image of each frame is extracted through multi-layer convolution operation stacking, and the dimension of the feature image is reduced through maximum pooling and average pooling to obtain key features;

[0017] The key features are input into a fully connected layer to identify the position of the person, facial expression, and body movement information in each frame, and a specific emotion and action category is identified through a classifier.

[0018] Further, in the process of decomposing the video data into continuous frame data, a corresponding timestamp is added to each frame, and the person movement trajectory is obtained by combining the timestamp and the person position information.

[0019] Further, the process of applying an LSTM network and combining Mel frequency spectrum coefficients and short-time Fourier transform to extract audio feature information includes:

[0020] The audio signal is sampled and quantized, and frame processing is performed;

[0021] STFT processing is performed on each audio frame to obtain a frequency domain representation, the STFT spectrum is converted to a Mel spectrum, and a discrete cosine transform is applied to the Mel spectrum to extract an MFCCs feature sequence;

[0022] The extracted MFCCs feature sequence is input into the LSTM network to obtain audio feature data carrying time sequence features.

[0023] Further, the process of training a decision neural network using video feature information and audio feature information includes:

[0024] The video features and audio features are spliced, and the timing of the video and audio features is aligned to form a joint feature vector, which is input into the constructed decision neural network for training of the decision neural network.

[0025] The prediction results of the model are evaluated using a cross-entropy loss function, and the weights in the decision neural network are updated using an Adam algorithm until the result of the cross-entropy loss function meets the business needs.

[0026] Further, the process of optimizing the decision neural network using the historical director station data and the manually annotated optimal switching nodes includes:

[0027] The director annotates the historical data according to the video content and program requirements, including switching time points, corresponding camera angles, and corresponding camera numbers.

[0028] The unannotated historical data is input into the decision neural network, the output of the decision neural network is compared with the annotated historical data, the decision neural network is evaluated according to the comparison result, and the decision neural network is optimized until the evaluation result meets the requirements.

[0029] Compared with the prior art, the director view angle auxiliary switching method has the following beneficial effects:

[0030] The beneficial effects of the above technical solutions mainly include the following aspects:

[0031] (1) Through the automatic view angle switching auxiliary system, the frequency and complexity of manual switching by the director are reduced, the overall switching efficiency is improved, and the director can focus more on other important decision-making and monitoring tasks.

[0032] (2) The machine learning model is used to comprehensively analyze the video and audio features, which can more accurately identify the optimal switching node, thereby enhancing the switching effect and ensuring that the audience has a smoother and higher quality viewing experience.

[0033] (3) Since the switching decision is based on a large amount of historical data and real-time feature analysis, errors caused by manual operation are reduced, and the accuracy and consistency of switching are improved.

[0034] (4) The system can quickly process and respond to real-time data to ensure timely view angle switching in dynamic scenarios and adapt to different live or recording environment requirements.

[0035] (5) By continuously learning and optimizing from new data, the system can adapt to different types of programs and scenes, gradually improving its decision-making ability and accuracy.

[0036] (6) Through the automatic and intelligent auxiliary tool, the work burden of the director can be significantly reduced, so that the director can work in a more relaxed environment, and the overall work efficiency and satisfaction are improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which form a part of the present description, illustrate the present application and together with the written description serve to explain the application. In the drawings:

[0038] Figure 1 A flowchart of a director's view angle auxiliary switching method according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0040] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0041] A director's view angle auxiliary switching method, the method comprising:

[0042] S1, collecting video and audio data, pre-processing, and storing in a data set;

[0043] S2, for the video data in the data set, applying a convolutional neural network to extract video feature information;

[0044] S3, for the audio data in the data set, applying an LSTM network and combining Mel frequency spectrum coefficient and short-time Fourier transform to extract audio feature information;

[0045] S4, using video feature information and audio feature information to train a decision neural network, and using historical director station data and artificially annotated best switching nodes to optimize the decision neural network;

[0046] S5, using the trained decision neural network to receive real-time video and audio data to assist the switching of the director's view angle.

[0047] The pre-processing process of the video and audio data in step S1 comprises:

[0048] The collected data is classified and packaged according to its source to obtain data sets belonging to different devices, and the data sets of different devices are associated through keywords;

[0049] The repeated data in the data set is identified and deleted, and the missing part of the data in the data set is filled using the mean method;

[0050] Convert the data in each dataset that has the same keyword to a unified format.

[0051] In some embodiments, a specific implementation process of the above-mentioned association of different device datasets by keywords is as follows:

[0052] Analyze the content of each device dataset to identify keywords that can be used to associate different datasets. The keywords are usually certain fields or attributes in the dataset, such as device ID, location, timestamp, sensor type, etc. Ensure that the selected keywords are unique or have sufficient discriminability to accurately associate data.

[0053] Create a keyword dictionary to record the correspondence between each keyword and its related dataset or record. The dictionary is a key-value pair structure, where the key is the keyword and the value is the reference or ID of the data record related to the keyword. Traverse each device dataset, extract the keywords, and add the data records to the keyword dictionary according to the keywords, ensuring that each data record can find its associated item in the dictionary through the keyword.

[0054] Note that one keyword may correspond to multiple datasets (or records), so use a list, set, or many-to-many relationship table in the database to store the association between keywords and datasets.

[0055] According to the keyword dictionary, logically integrate related datasets. Different device data related to the same keyword can be considered as a whole during analysis to ensure that the integrated data structure is reasonable and facilitates subsequent queries and analysis.

[0056] The process of applying a convolutional neural network to extract video feature information in step S2 includes:

[0057] Decompose the video data into a continuous frame sequence, standardize each frame, and extract the feature image of each frame through multi-layer convolution operation stacking. Reduce the dimension of the feature image through maximum pooling and average pooling methods to obtain key features.

[0058] Input the key features into the full connection layer to identify the person's position, facial expression, and body movement information in each frame, and identify specific emotion and action categories through the classifier.

[0059] During the process of decomposing video data into continuous frame data, add a corresponding timestamp to each frame, and combine the timestamp with the person's position information to obtain the person's movement trajectory.

[0060] In some embodiments, a specific implementation process of the above-mentioned step S2 is as follows:

[0061] Acquire video file or live video stream: Use camera or video file to acquire video data, ensure video format compatible, such as MP4 or AVI format, use video processing tool (like OpenCV) to decompose video into continuous static frames, record timestamp of each frame for later analysis.

[0062] Adjust each frame image to uniform size, such as 224x224 pixels, to match model input requirements, and normalize image pixel values, scaling them to 0 to 1, to improve model processing performance.

[0063] Pass normalized image through multiple convolution layers, each using multiple filters to scan image, convolution operation extracts local features like edges, textures through sliding window mechanism, apply nonlinear activation function after each convolution layer to increase model's nonlinear ability, enabling it to learn complex patterns;

[0064] After convolution layer, add pooling layer to reduce feature map size, take maximum value within pooling window to retain significant features, and normalize feature map through batch normalization layer to mitigate internal covariate shift, speed up training process, and improve model stability, flatten multi-dimensional feature map obtained through convolution and pooling operations into one-dimensional vector for further processing by fully connected layer;

[0065] Use multiple convolution blocks (convolution + activation + pooling) stacked, extract higher-level features layer by layer, through deep network, extracted features can represent complex patterns and semantic information of image, finally output processed feature vector, which contains high-level information of input image for subsequent classification or detection tasks.

[0066] Input feature vector extracted from each frame image into classification model, ensure feature vector dimension and format meet model requirements, acquire labeled training dataset containing feature vector and corresponding action, emotion label, select appropriate model architecture according to task requirements, such as support vector machine (SVM), decision tree, random forest, or deep neural network (DNN), input feature vector and label data into model for training, through multiple iterations (epochs), use optimization algorithm (such as SGD, Adam) to adjust model parameters to minimize loss function (such as cross-entropy loss).

[0067] Input feature vector of frame to be recognized into trained model, model calculates probability distribution of each class through forward propagation, according to probability distribution, select class with highest probability as prediction result. Threshold or other decision rules can be used to improve prediction robustness.

[0068] The action and emotion labels corresponding to each frame are output, the numerical labels output by the model are decoded into human-readable category names such as "running", "smiling", etc., the recognition results of consecutive frames are time-smoothed to reduce the influence of short-time noise on the final results, and the recognition results are superimposed on the video to display or generate a chart to show the changes of actions and emotions over time.

[0069] For each frame, the position coordinates of the person in the image are obtained by a detection algorithm, and the movement trajectory of the person in the video is calculated according to the timestamp and position information. The trajectory graph can be generated by recording the change of position in adjacent frames.

[0070] The process of applying the LSTM network and combining the Mel spectrum coefficient and the short-time Fourier transform to extract the audio feature information in step S3 includes:

[0071] The audio signal is sampled and quantized, and frame processing is performed;

[0072] STFT processing is performed on each audio frame to obtain a frequency domain representation, the STFT spectrum is converted into a Mel spectrum, and a discrete cosine transform is applied to the Mel spectrum to extract an MFCCs feature sequence;

[0073] The extracted MFCCs feature sequence is input into the LSTM network to obtain audio feature data carrying time sequence features.

[0074] STFT (Short-Time Fourier Transform) is a short-time Fourier transform, and MFCCs (Mel-Frequency Cepstral Coefficients) are Mel-frequency cepstral coefficients.

[0075] In some embodiments, a specific execution process of the above step S3 is as follows:

[0076] The original audio signal is obtained from a microphone or an audio file, pre-emphasis processing is performed on the original audio signal by a high-pass filter to enhance high-frequency components, and the audio signal is divided into a series of overlapping short-time frames, each frame having a length of 25 milliseconds and a frame shift of 10 milliseconds, ensuring a certain overlap between frames.

[0077] Before performing Fourier transform on each audio frame, a window function is applied to reduce spectral leakage, assuming that the number of samples in each frame is N, and the window function is: ;

[0078] The window function is applied to each frame x(n) to obtain ;

[0079] where N is the number of samples in each frame, n is the index of the current sample, and the range is from 0 to , w(n) is the value of the Hamming window function, x(n) is the nth sample in the original audio signal frame, x w (n) is the nth sample in the windowed signal frame.

[0080] The windowed signal is subjected to a short-time Fourier transform to obtain a spectrum, for each frame, the following is calculated: ; and the power spectrum of each frequency component is calculated: ;

[0081] where X(k) is the complex value of the kth frequency component of the signal frame after FFT transform, k is the index of the frequency component, ranging from 0 to , P(k) is the power spectral density of the kth frequency component, j is the imaginary unit, satisfying .

[0082] The spectrum is passed through a Mel filter bank, and the Mel frequency scale is converted by the following formula:

[0083] ;

[0084] The center frequency and bandwidth of the filter are determined according to the Mel scale, assuming there are M filters, and the energy output by the filter is:

[0085] ;

[0086] where f is the actual frequency in hertz; Mel(f) is the conversion of the actual frequency f to the Mel frequency scale; M is the number of Mel filters; S(m) is the energy output by the mth Mel filter; k min and k max are the effective frequency component ranges defined in the spectrum for each filter m; H m (k) is the gain of the mth Mel filter at the kth frequency component.

[0087] The logarithmic energy output by each Mel filter is calculated (the logarithmic energy output by the mth Mel filter): .

[0088] A discrete cosine transform (DCT) is applied to the Mel spectrum energy to obtain MFCCs, and the first L coefficients are selected as MFCCs, usually L is 12 or 13:

[0089] ;

[0090] where MFCC( l ) is the lth MFCC coefficient; L is the number of extracted MFCC coefficients, usually 12 or 13; m is the index of the Mel filter; The value of the DCT basis function.

[0091] The MFCCs features are normalized, usually using mean normalization or Z-score standardization, to improve robustness. The normalized MFCCs sequence is input to the LSTM network. The LSTM processes the whole sequence, capturing temporal dependencies and contextual information.

[0092] The process of training the decision neural network using video feature information and audio feature information in step S4 includes:

[0093] The video features and audio features are spliced and the timing of the video and audio features is aligned. A joint feature vector is formed and input to the constructed decision neural network to train the decision neural network.

[0094] The prediction results of the model are evaluated using a cross-entropy loss function and the weights in the decision neural network are updated using the Adam algorithm until the result of the cross-entropy loss function meets the business needs.

[0095] In some embodiments, a specific execution process of step S4 is as follows:

[0096] The above-mentioned scheme of step S4 is applied to the director's view angle auxiliary switching process of a sports match, and the best view angle is selected for switching by the following methods:

[0097] Extract the features of each camera video frame, such as the video feature information and audio feature information extracted in the scheme of step S1-3;

[0098] Construct a decision neural network.

[0099] The input layer of the decision neural network includes: a video input layer, the video feature vector of each camera as input; an audio input layer, the audio feature vector as input.

[0100] The hidden layer of the decision neural network includes:

[0101] A) The first hidden layer (low-level feature processing) receives video features (such as edges, color histograms) and audio features (such as spectral intensity), wherein:

[0102] Neuron 1 (video motion detection) receives pixel movement data in the video frame, calculates its motion vector, identifies fast moving areas (e.g. the movement of a football on the field), and outputs motion intensity features;

[0103] Neuron 2 (audio intensity detection) receives spectral data of the audio signal, identifies high-frequency intensity changes in the audio signal (e.g. a sudden increase in audience cheers), and outputs audio intensity features;

[0104] Neuron 3 (Edge Detection) scans the video frame using a convolution kernel, identifies the edges of important objects, and outputs edge features for recognizing the field boundaries or goalposts.

[0105] B) The second hidden layer (Intermediate Feature Combination) receives motion intensity, audio intensity, and edge features from the first layer.

[0106] Neuron 4 (Event Recognition) combines motion intensity and audio intensity features, uses a threshold to determine if an important event has occurred (e.g., when a goal is scored, both motion intensity and audio intensity are high), and outputs an event flag (0 or 1).

[0107] Neuron 5 (Region Recognition) receives edge features and motion intensity features, determines if a specific region (e.g., the goal area) has significant activity, and outputs a region activity flag.

[0108] Neuron 6 (Audio-Video Synchronization) checks the temporal synchronization between audio intensity and video motion, outputs a synchronization flag, and helps determine the timeliness of events.

[0109] C) The third hidden layer (Advanced Decision Support) receives event flags, region activity flags, and synchronization flags.

[0110] Neuron 7 (Priority Assignment) receives event flags and region activity flags, assigns priorities for view angle switching based on event types (e.g., goals), and outputs a priority score.

[0111] Neuron 8 (View Angle Selection) receives priority scores and synchronization flags, selects the most appropriate camera view angle (e.g., close-up shot) based on the scores, and outputs a view angle selection instruction (camera number).

[0112] Neuron 9 (Feedback Adjustment) receives view angle selection instructions, determines if the current selection meets expectations (by comparing with historical data), and outputs an adjustment signal to optimize future selections.

[0113] The output layer of the decision neural network includes output neurons, each corresponding to a camera view angle, and outputs the selection probability of that view angle.

[0114] By combining video and audio information, the system can more accurately identify important game moments and automatically switch view angles in real-time. For example, when the audience's cheers reach a peak, the system can quickly identify and switch to a close-up shot, enhancing the audience's experience. This system, through the layer-by-layer processing of neural networks and the fusion of multi-modal information, can effectively assist the director in switching view angles, improving the intelligence and automation level of live broadcasts.

[0115] The process of optimizing the decision neural network using the history director station data and the manually labeled best switching node in step S4 includes:

[0116] The director labels the historical data according to video content and program requirements, and the labeling includes a switching time point, a corresponding camera view, and a corresponding camera number;

[0117] The historical data that has not been labeled is input into the decision neural network, the output of the decision neural network is compared with the labeled historical data, the decision neural network is evaluated according to the comparison result, and the decision neural network is optimized until the evaluation result meets the requirement.

[0118] Those skilled in the art can realize that the units and method steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0119] In several embodiments provided in the present application, it should be understood that the disclosed method and system can be realized in other ways. For example, the division of the above-described units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above units can be or can not be physically separated, and the components displayed as units can be or can not be physical units, i.e., they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0120] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and description of the present application.

[0121] The above merely provides the preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A director's eye view assisted switching method, characterized by, The method comprises: After collecting video and audio data, pre-processing is performed, and the data is stored in a data set; For video data in the data set, a convolutional neural network is applied to extract video feature information; For audio data in the data set, an LSTM network is applied in combination with Mel spectrum coefficients and short-time Fourier transform to extract audio feature information; A decision neural network is trained using video feature information and audio feature information, and the decision neural network is optimized using historical director station data and manually annotated optimal switching nodes; The trained decision neural network is used to receive real-time video and audio data to assist the director in switching perspectives; The process of applying an LSTM network in combination with Mel spectrum coefficients and short-time Fourier transform to extract audio feature information comprises: Obtain the original audio signal from a microphone or an audio file, pre-emphasize the original audio signal through a high-pass filter to enhance the high-frequency component, and divide the audio signal into a series of overlapping short-time frames; Before performing Fourier transform on each audio frame, a window function is applied to reduce spectral leakage, the number of samples in each frame is N, and the window function is: ； Applying a window function to each frame x(n) gives ; where N is the number of samples per frame, n is the index of the current sample, ranging from 0 to N-1, w(n) is the value of the Hamming window function, x(n) is the nth sample in the original audio signal frame, x w (n) is the nth sample in the windowed signal frame; Perform short-time Fourier transform on the windowed signal to obtain the frequency spectrum, and for each frame, calculate: ； and the power spectrum of each frequency component is calculated: ; where X(k) is the complex value of the kth frequency component of the signal frame after FFT transformation, K is the index of the frequency component, ranging from 0 to N−1, P(k) is the power spectral density of the kth frequency component, J is the imaginary unit, satisfying j 2 =−1. Pass the frequency spectrum through a Mel filter bank, and convert the Mel frequency scale by the following formula: ； The center frequency and bandwidth of the filter are determined according to the Mel scale, assuming there are M filters, and the energy output by the filter is: ； where f is the actual frequency in Hertz; Mel(f) is the conversion of the actual frequency f to the Mel frequency scale; M is the number of Mel filters; S(m) is the energy output of the mth Mel filter; k min and k max is the range of valid frequency components defined in the spectrum for each filter m; H m (k) is the gain of the mth Mel filter at the kth frequency component. The log energy of each Mel filter output (log energy of the mth Mel filter output) is calculated: ; Apply discrete cosine transform (DCT) to the Mel spectrum energy to obtain MFCCs, select the first L coefficients as MFCCs, and usually L is 12 or 13: ； wherein MFCC(l) is the lth MFCC coefficient; L is the number of extracted MFCC coefficients, typically 12 or 13; m is the index of the Mel filter; is the value of the DCT basis function; Normalize the MFCCs feature, usually use mean normalization or Z-score standardization to improve robustness, and input the normalized MFCCs sequence into the LSTM network, which will process the entire sequence to capture time dependence and context information; The process of applying a convolutional neural network to extract video feature information comprises: Decompose the video data into a continuous frame sequence, standardize each frame, stack through multi-layer convolution operations, extract the feature image of each frame, and reduce the dimension of the feature image through maximum pooling and average pooling methods to obtain key features; Input the key features into a fully connected layer to identify the position of the person, facial expression, and body movement information in each frame, and identify specific emotions and action categories through a classifier; During the process of decomposing the video data into continuous frame data, add a corresponding timestamp to each frame, and combine the timestamp and the person position information to obtain the person movement trajectory; The process of applying an LSTM network in combination with Mel spectrum coefficients and short-time Fourier transform to extract audio feature information comprises: Sample and quantize the audio signal and perform frame processing; Perform STFT processing on each audio frame to obtain frequency domain representation, convert the STFT spectrum to Mel spectrum, apply discrete cosine transform to the Mel spectrum, and extract the MFCCs feature sequence; The extracted MFCCs feature sequence is input into the LSTM network to obtain audio feature data carrying time sequence characteristics; The process of training the decision neural network using video feature information and audio feature information comprises: The video features and audio features are spliced and the time sequences of the video and audio features are aligned to form a joint feature vector, and the joint feature vector is input into the constructed decision neural network to train the decision neural network; The prediction result of the model is evaluated using a cross-entropy loss function, and the weights in the decision neural network are updated using an Adam algorithm until the result of the cross-entropy loss function meets the business needs; The input layer of the decision neural network comprises: a video input layer, the video features of each camera as input; and an audio input layer, the audio features as input; The hidden layer of the decision neural network comprises: A) a first hidden layer, the video features comprising edge, color histogram, and the audio features comprising spectral intensity, wherein: Neuron 1 receives pixel movement data in the video frame, calculates its motion vector, identifies a fast moving area, and outputs a motion intensity feature; Neuron 2 receives spectral data of the audio signal, identifies high-frequency intensity changes in the audio signal, and outputs an audio intensity feature; Neuron 3 scans the video frame using a convolution kernel, identifies the edges of important objects, and outputs edge features for identification; B) a second hidden layer receiving motion intensity, audio intensity, and edge features from the first hidden layer; Neuron 4 combines the motion intensity and audio intensity features, uses a threshold to determine whether an important event has occurred, and outputs an event marker 0 or 1; Neuron 5 receives edge features and motion intensity features, determines whether there is significant activity in a specific area, and outputs a region activity marker; Neuron 6 checks the time synchronization between audio intensity and video motion, outputs a synchronization marker, and helps determine the real-time nature of the event; C) a third hidden layer receiving event markers, region activity markers, and synchronization markers; Neuron 7 receives event markers and region activity markers, assigns a priority of view switching according to the type of event, and outputs a priority score; Neuron 8 receives the priority score and the synchronization marker, selects the most suitable camera view according to the score, and outputs a view selection instruction; Neuron 9 receives the view selection instruction, determines whether the current selection meets expectations, and outputs an adjustment signal to optimize future selection; The output layer of the decision neural network comprises: output neurons, each neuron corresponding to a camera view, and outputting a selection probability for that view.

2. The director's eye view assisted switching method of claim 1, wherein, The process of preprocessing the video and audio data comprises: The collected data is classified and packaged according to its source to obtain data sets belonging to different devices, and the data sets of different devices are associated through keywords; Duplicate data in the data sets is identified and deleted, and the missing data in the data sets is filled using the mean method; Data in each data set with the same keyword is converted to a unified format.

3. The director's eye view assisted switching method of claim 1, wherein, The process of optimizing the decision neural network using historical director station data and manually annotated optimal switching nodes comprises: The director annotates historical data according to video content and program requirements, and the annotation includes switching time points, corresponding camera angles and corresponding camera numbers; The historical data without annotation is input into the decision neural network, the output of the decision neural network is compared with the annotated historical data, the decision neural network is evaluated according to the comparison result, and the decision neural network is optimized until the evaluation result meets the requirements.

Images