An intelligent director method and system for language class scenarios

Abstract

This invention relates to an intelligent broadcasting method and system for language-based scenarios, belonging to the field of broadcasting technology, and solves the problems of high cost and complex processes in existing manual broadcasting techniques. The method includes: extracting the current sentence and the corresponding audio and video segments of each camera based on real-time speech recognition technology; performing speaker detection on each audio and video segment using a trained speaker detection model to obtain the speaker detection result for each audio and video segment; extracting semantic features of the current sentence based on a trained semantic classification model; obtaining a comprehensive feature corresponding to the current sentence based on the semantic features, the speaker detection results of each audio and video segment, and the shot overview of each camera; and inputting a comprehensive feature sequence composed of the k sentences preceding the current sentence and the comprehensive feature corresponding to the current sentence into a trained decision model to predict the camera identifier for the next shot. This achieves efficient and smooth intelligent shot switching.

Description

Technical Field

[0001] This invention relates to the field of intelligent broadcasting technology, and in particular to an intelligent broadcasting method and system for language-based scenarios. Background Technology

[0002] In modern television program production, successful programs require close collaboration between multi-camera operators, directors, and other relevant professionals. In this process, the director plays a crucial role, coordinating the cooperation between different departments and using camera switching to present the program content from multiple angles. However, current training models for human directors face problems such as high costs and long development cycles, failing to meet the growing market demand. Furthermore, hiring a professional production team is a challenge for small-scale program productions with limited budgets. Additionally, in scenarios involving frequent speaking, manual operation is prone to camera delays or switching errors. Summary of the Invention

[0003] Based on the above analysis, the embodiments of the present invention aim to provide an intelligent directing method and system for language-related scenarios, in order to solve the problems of high cost and complex process of existing manual directing.

[0004] On one hand, embodiments of the present invention provide an intelligent broadcasting method for language-based scenarios, comprising the following steps:

[0005] Extract the current sentence and the corresponding audio and video segments from each camera based on real-time speech recognition technology;

[0006] Speaker detection results for each audio / video segment are obtained by performing speaker detection on each audio / video segment based on the trained speaker detection model.

[0007] Extract semantic features of the current sentence based on a trained semantic classification model;

[0008] Based on the semantic features, the speaker detection results of each audio and video segment, and the shot summary of each camera, the comprehensive features corresponding to the current sentence are obtained.

[0009] The comprehensive feature sequence, composed of the k statements preceding the current statement and the comprehensive features corresponding to the current statement, is input into the trained decision model to predict the camera identifier for the next shot.

[0010] Based on a further improvement of the above method, the speaker detection model is an improved TalkNet model;

[0011] The improved TalkNet model includes a residual network;

[0012] The residual network includes multiple residual blocks, each residual block including two convolutional layers; a dual-pooling temporal attention module is added after the second convolutional layer; the dual-pooling temporal attention module is used to dynamically calibrate the spatiotemporal features of the input residual network.

[0013] Based on a further improvement of the above method, the dual-pooling temporal attention module dynamically calibrates the spatiotemporal features in the following manner:

[0014] Dimensional rearrangement of spatiotemporal features;

[0015] Global average pooling is applied to the spatial dimension of the rearranged spatiotemporal features to generate a joint channel-time description.

[0016] Calculate channel sensitivity weights based on channel-time joint description;

[0017] Time-sensitive weights are calculated based on channel-time joint description and channel-sensitive weights.

[0018] The rearranged spatiotemporal features are weighted based on time-sensitive weights and restored to their original dimensional order to obtain dynamically calibrated spatiotemporal features.

[0019] Based on the further improvement of the above method, the channel sensitivity weight is calculated using the following formula:

[0020] ;

[0021] in, The channel-time joint description of the c-th channel of the t-th frame represents the spatiotemporal characteristics, where T represents the number of image frames in the audio / video segment. Represents one-dimensional convolution. This indicates the channel-sensitive weight.

[0022] Based on the above method, and with further improvements, the time-sensitive weight is calculated using the following formula, based on the channel-time joint description and channel-sensitive weight:

[0023] ;

[0024] ;

[0025] in, The channel-temporal joint description of the c-th channel of the t-th frame image representing spatiotemporal features. Represents one-dimensional convolution. Indicates channel-sensitive weights. This indicates element-wise multiplication. Represents matrix addition. Represents the ReLU function. This represents the Sigmoid activation function. Indicates time-sensitive weights, Indicates an intermediate variable.

[0026] Based on further improvements to the above method, the semantic classification model includes:

[0027] The BERT encoder is used to semantically encode input text to obtain semantic features;

[0028] The TextCNN module is used to perform feature enhancement on semantic features to obtain enhanced semantic features;

[0029] The classification layer is used for classification based on the enhanced semantic features.

[0030] Based on the above method, the semantic classification model is fine-tuned in the following way:

[0031] Freeze the parameters of the BERT encoder and optimize the TextCNN module and classification layer;

[0032] Starting from the last layer and working down to the first layer, the parameters of the BERT encoder are unfrozen sequentially, and the BERT encoder, the TextCNN module, and the classification layer are optimized.

[0033] On the other hand, embodiments of the present invention provide an intelligent broadcasting system for language-related scenarios, including:

[0034] Multiple cameras are used to send the captured audio and video data to the intelligent broadcast server via a switcher;

[0035] The intelligent director server is used to predict the camera identifier of the next shot using an intelligent director method oriented towards language-based scenarios, and then send the camera identifier information to the switcher.

[0036] A switcher is used to switch shots based on the camera identification information of the next shot.

[0037] Based on further improvements to the above system, the intelligent broadcasting method for language-related scenarios includes the following steps:

[0038] Extract the current sentence and the corresponding audio and video segments from each camera based on real-time speech recognition technology;

[0039] Speaker detection results for each audio / video segment are obtained by performing speaker detection on each audio / video segment based on the trained speaker detection model.

[0040] Extract semantic features of the current sentence based on a trained semantic classification model;

[0041] Based on the semantic features, the speaker detection results of each audio and video segment, and the shot summary of each camera, the comprehensive features corresponding to the current sentence are obtained.

[0042] The comprehensive feature sequence, composed of the k statements preceding the current statement and the comprehensive features corresponding to the current statement, is input into the trained decision model to predict the camera identifier for the next shot.

[0043] Based on further improvements to the above system, the speaker detection model is an improved TalkNet model;

[0044] The improved TalkNet model includes a residual network;

[0045] The residual network includes multiple residual blocks, each residual block including two convolutional layers; a dual-pooling temporal attention module is added after the second convolutional layer; the dual-pooling temporal attention module is used to dynamically calibrate the spatiotemporal features of the input residual network.

[0046] Compared with existing technologies, this invention extracts the current sentence and the corresponding audio and video segments captured by each camera through real-time speech recognition technology; by extracting the semantic features of the current sentence, the speaker detection results of each audio and video segment within the time range of the current sentence, and the shot overview of each camera, it obtains comprehensive features including multimodal information, thereby making shot switching decisions based on multi-dimensional indicators, and predicting the camera identifier of the next shot based on the comprehensive features of the current sentence and k historical sentences, thus achieving efficient and smooth shot switching.

[0047] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0048] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0049] Figure 1 This is a flowchart of an intelligent broadcasting method for language-related scenarios according to an embodiment of the present invention;

[0050] Figure 2 This is a diagram of the backbone network structure of the TalkNet model in an embodiment of the present invention;

[0051] Figure 3 This is a structural diagram of the visual encoder according to an embodiment of the present invention;

[0052] Figure 4 This is the structure of the dual-pooling temporal attention module in an embodiment of the present invention;

[0053] Figure 5 This is a block diagram of an intelligent broadcasting system for language-related scenarios according to an embodiment of the present invention;

[0054] Figure 6 This is a structural block diagram of the semantic classification model in an embodiment of the present invention. Detailed Implementation

[0055] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0056] A specific embodiment of the present invention discloses an intelligent broadcasting method for language-based scenarios, such as... Figure 1 As shown, it includes the following steps:

[0057] S1. Extract the current sentence and the corresponding audio and video segments of each camera based on real-time speech recognition technology;

[0058] S2. Based on the trained speaker detection model, perform speaker detection on each audio and video segment to obtain the speaker detection result for each audio and video segment;

[0059] S3. Extract the semantic features of the current sentence based on the trained semantic classification model;

[0060] S4. Based on the semantic features, the speaker detection results of each audio and video segment, and the lens overview of each camera, the comprehensive features corresponding to the current sentence are obtained;

[0061] S5. Input the comprehensive feature sequence formed by the k statements before the current statement and the comprehensive features corresponding to the current statement into the trained decision model to predict the camera identifier of the next shot.

[0062] In implementation, the shot summary, or shooting description information for each camera, includes camera identification, shot type (close-up, medium shot, full shot, long shot, and no human presence), and coverage area. For example, in a standardized three-camera deployment, CAM1 is fixed to film the guest seating area, using a medium-close shot to show the subjects from chest to upper body, covering 1-2 key guests; CAM3 focuses on the host's seating area, using a close-up to show the subject from shoulders to upper body, with the center of the lens aligned with the host's face; CAM2 is responsible for filming the entire venue, including the stage and part of the audience seating, used for scene transitions and overall atmosphere presentation. The corresponding shot summary would be:

[0063] CAM1: Medium shot; Guest;

[0064] CAM2: Wide shot; Full view;

[0065] CAM3: Close-up; presenter.

[0066] The shot overview is encoded and used as part of the input information for the decision-making module.

[0067] It should be noted that the speaker detection result of the audio and video segment refers to the detection result of whether a speaker exists in each frame of the audio and video segment.

[0068] This invention focuses on language-related scenarios such as meetings, interviews, and news, including small dialogue scenarios (such as academic interviews and personal interviews), emphasizing the capture of one-on-one interactions between the host and guests; roundtable discussion scenarios (1 host + n guests) to simulate multi-role communication situations; and press conference speech scenarios (the host speaks and intersperses questions with shots).

[0069] In language-based scenarios, viewers typically focus on the person speaking. Therefore, speaker detection is crucial in intelligent broadcasting algorithm design, with intelligent broadcasting camera switching in language-based scenarios primarily driven by whether the person being filmed is speaking. However, a single strategy based solely on "whoever is speaking, cut to them" fails to consider the semantic relationship between the speech content and the camera's logic, resulting in poor program engagement and a monotonous, rigid switching style. Therefore, semantic analysis of the speaker's content is necessary, combining context and speaker detection results with other multi-dimensional indicators to make camera switching decisions. Furthermore, during switching, factors such as camera composition and framing are considered to enhance program engagement. This invention achieves automated camera switching through intelligent broadcasting technology, improving the efficiency and experience of multi-camera program production.

[0070] Compared with existing technologies, the intelligent broadcasting method for language-based scenarios provided in this embodiment extracts the current sentence and the corresponding audio and video segments captured by each camera through real-time speech recognition technology; by extracting the semantic features of the current sentence, the speaker detection results of each audio and video segment within the time range of the current sentence, and the shot overview of each camera, a comprehensive feature including multimodal information is obtained, thereby making shot switching decisions based on multi-dimensional indicators, and predicting the camera identifier of the next shot based on the comprehensive features of the current sentence and k historical sentences, thus achieving efficient and smooth shot switching.

[0071] During implementation, the audio and video captured by each camera are acquired in real time. Based on real-time speech recognition technology, the text corresponding to the audio stream can be identified in real time, and the start and end times of each sentence can be returned, thus obtaining the timestamp range corresponding to each sentence.

[0072] For the current statement, based on its timestamp range, extract the audio and video segments within the timestamp range of the current statement from the audio and video captured by each camera, thus obtaining the audio and video segments of each camera.

[0073] Each audio / video segment is input into the trained speaker detection model for speaker detection, that is, to detect whether a speaker exists in each frame of the video segment.

[0074] Most efficient speaker detection models currently employ multimodal information fusion technology, organically combining audio and video features to compensate for the shortcomings of single-modality models. The process involves first detecting face regions in the image, tracking candidate bounding boxes, and then cropping out the face regions to form a face sequence. One audio stream and all candidate faces are then fed into the speaker detection module, which mainly includes a video feature extraction network and an audio feature extraction network. The two-modal networks are then fused, and the probability that each frame of face image and the current audio is a correct match is output.

[0075] The baseline model of this algorithm uses TalkNet, an active speaker detection model based on audio-visual multimodal information fusion. Its core idea is to improve the ability to discriminate speaker activity through long-term temporal feature modeling and cross-modal interaction mechanisms. The model consists of two parts: a feature representation front-end and a back-end classifier, responsible for the spatiotemporal encoding of multimodal features and the learning of cross-modal correlations, respectively. Its main algorithm is as follows: Figure 2 As shown.

[0076] The feature representation front end includes a visual encoder and an audio encoder, which extract long-term temporal features from the video frame sequence and audio signal, respectively. The visual encoder structure consists of a visual frontend and a visual temporal network, as shown in the figure. Figure 3 As shown, the visual front end employs 3D convolutional layers (3D Conv) to capture short-term motion information between adjacent frames, and the extracted spatiotemporal features are further enhanced in terms of feature representation through a residual network (ResNet18). The visual temporal network consists of a video temporal convolutional module (V-TCN), which includes multiple depthwise separable 1D convolutional layers (DS Conv1D), batch normalization (BN), and a ReLU activation function. This module expands the receptive field (e.g., 21 frames, covering approximately 840 ms) by stacking convolutions, thereby modeling long-term facial motion patterns (such as continuous lip movements).

[0077] The audio encoder is based on an improved ResNet34 network, combined with a compression excitation module (SE Block) to enhance feature representation capabilities. The input audio is converted into a temporal spectrogram using MFCC (Mel frequency cepstral coefficients), and frame-level features are extracted via a 2D convolutional network. Through dilated convolution, the temporal resolution of the audio features is aligned with that of the visual features, facilitating subsequent cross-modal interaction.

[0078] The backend classifier achieves multimodal feature fusion and long-term context modeling through cross-modal attention and self-attention mechanisms, ultimately outputting detection results through a classification layer. The cross-modal attention layer dynamically aligns audio and visual features to address synchronization issues caused by temporal discrepancies (such as lip movement and speech delay). The self-attention layer further mines long-term contextual dependencies from joint features to distinguish between speaking and non-speaking frames. Its structure is similar to cross-attention, but the query, key, and value all originate from the same feature. Finally, a fully connected layer and a softmax layer output frame-by-frame classification probabilities, using cross-entropy loss as the loss function.

[0079] Current speaker detection algorithms all suffer from long temporal dependence, meaning that the shorter the analyzed audio and video segment, the lower the accuracy. Taking TalkNet as an example, it requires more than 50 frames of data to ensure accuracy (mAP 89.5%). As shown in Table 1, TalkNet's performance drops significantly with short temporal inputs (5 frames) (mAP only 75.2%). Directing is for live broadcasts, requiring real-time switching capabilities. Therefore, intelligent directing algorithms also need to extract features from video and audio information, detect them, and make switching decisions within a short timeframe. This invention adapts the baseline model for directing work, improving recognition accuracy under time-limited conditions.

[0080] Table 1 Performance comparison of TalkNet with different input frame counts

[0081]

[0082] To improve the accuracy of speaker detection under time-limited conditions, this invention employs an improved TalkNet model. A dual-pooling temporal attention module is introduced into the residual network of the TalkNet model, extracting visual features based on this module. The dual-pooling temporal attention module decouples channel and temporal attention calculations through a dual-pooling strategy and combines lightweight convolutions to achieve dynamic feature calibration, enhancing sensitivity to continuous lip movements. This results in improved recognition accuracy under time-limited conditions.

[0083] Specifically, the residual network includes multiple residual blocks, each of which includes two convolutional layers; a dual-pooling temporal attention module is added after the second convolutional layer; the dual-pooling temporal attention module is used to dynamically calibrate the spatiotemporal features of the input residual network.

[0084] In implementation, the residual network uses ResNet18, which consists of four stages (Stage 1-Stage 4), each composed of multiple stacked basic blocks. Each basic block contains two convolutional layers. To fully integrate the dual-pooling temporal attention module into the ResNet18 of the TalkNet visual encoder, this invention proposes a hierarchical embedding strategy, embedding the dual-pooling temporal attention module after the second convolutional layer of each residual block. This ensures that the dual-pooling temporal attention module can dynamically calibrate the spatiotemporal features of the convolutional output while preserving the gradient propagation path of the skip connections.

[0085] Specifically, the dual-pooling temporal attention module uses the following method to dynamically calibrate spatiotemporal features:

[0086] Dimensional rearrangement of spatiotemporal features;

[0087] Global average pooling is applied to the spatial dimension of the rearranged spatiotemporal features to generate a joint channel-time description.

[0088] Calculate channel sensitivity weights based on channel-time joint description;

[0089] Time-sensitive weights are calculated based on channel-time joint description and channel-sensitive weights.

[0090] The rearranged spatiotemporal features are weighted based on time-sensitive weights and restored to the original dimensions to obtain dynamically calibrated spatiotemporal features.

[0091] In practice, the features output by the second convolutional layer are Where T is the length of the image frame sequence, C is the number of channels, and H and W are the height and width of the feature. To facilitate subsequent operations, the spatiotemporal features are rearranged, i.e., the dimensional order is adjusted:

[0092] ;

[0093] in, This represents the spatiotemporal characteristics before rearrangement. This represents the spatiotemporal characteristics after rearrangement. This represents the permutation function, also known as the rearrangement function.

[0094] Then, global average pooling is performed on the spatial dimension to generate a joint channel-time description. :

[0095] ;

[0096] in, The channel-time joint description of the t-th frame image representing the c-th channel of the spatiotemporal features.

[0097] right Average pooling is performed along the time dimension, and channel-sensitive weights are generated through one-dimensional convolution (Conv1D). The kernel size is 1:

[0098] ;

[0099] in, The channel-time joint description of the c-th channel of the t-th frame represents the spatiotemporal characteristics, where T represents the number of image frames in the audio / video segment. Represents one-dimensional convolution. This indicates the channel-sensitive weight.

[0100] right Using one-dimensional convolution (Conv1D) and gating mechanisms, time-sensitive weights are generated. :

[0101] ;

[0102] ;

[0103] in, The channel-temporal joint description of the c-th channel of the t-th frame image representing spatiotemporal features. Represents one-dimensional convolution. Indicates channel-sensitive weights. Represents the ReLU function. This represents the Sigmoid activation function. Indicates time-sensitive weights, Indicates intermediate variables. Represents matrix addition. This indicates element-wise multiplication. Both operations first align the two tensors according to standard broadcast rules before performing the operation.

[0104] The rearranged spatiotemporal features are weighted using time-sensitive weights and restored to their original dimensional order to obtain dynamically calibrated spatiotemporal features:

[0105] ;

[0106] During implementation, first Broadcast to In relation to Multiply. These are the calibrated spatiotemporal features.

[0107] The structure of the dual-pooling temporal attention module is as follows: Figure 4 As shown.

[0108] By embedding a dual-pooling temporal attention module in each residual block, the model can refine spatiotemporal features layer by layer, making it particularly suitable for progressive modeling of temporal lip movements and micro-expressions. By replacing fully connected layers with lightweight one-dimensional convolutions and global fully connected layers with local convolutional kernels, the model retains local temporal correlations while reducing computational complexity (FLOPs are reduced by about 30%).

[0109] During implementation, the improved TalkNet model can be trained using the existing AVA-ActiveSpeaker dataset.

[0110] It should be noted that the trained improved TalkNet model outputs the probability that each face in each frame of the input audio / video segment is a speaker. During speaker detection, if the probability of a face being a speaker in a frame exceeds a preset threshold, that frame is labeled as 1; otherwise, it is labeled as 0, thus obtaining the speaker detection result for the input video.

[0111] The current statement is input into a trained semantic classification model to extract its semantic features.

[0112] In implementation, semantic classification models, such as Figure 6 As shown, it includes:

[0113] The BERT encoder is used to semantically encode input text to obtain semantic features;

[0114] The TextCNN module is used to perform feature enhancement on semantic features to obtain enhanced semantic features;

[0115] The classification layer is used for classification based on the enhanced semantic features.

[0116] In practice, statements are categorized into six types: emphasis, summary, question, case study, introduction, and statement. These classifications are closely related to switching decisions.

[0117] In implementation, the semantic classification model employs a lightweight BERT-TextCNN fusion architecture. BERT, as the core semantic encoder of the fusion architecture, is essentially a bidirectional language model based on the Transformer encoder, capturing long-distance dependencies in the text through a self-attention mechanism. The TextCNN module extracts local key phrase features from the text through convolutional operations. The classification layer consists of fully connected layers and a softmax layer.

[0118] In practice, the BERT encoder in the semantic classification model is a pre-trained network. The semantic classification model is trained on the shot switching decision dataset.

[0119] Specifically, the semantic classification model is trained using the following method:

[0120] Freeze the parameters of the BERT encoder and optimize the TextCNN module and classification layer;

[0121] Starting from the last layer and working down to the first layer, the parameters of the BERT encoder are unfrozen sequentially, and the BERT encoder, TextCNN module, and classification layer are optimized.

[0122] The first stage is feature freezing. During initial training, all parameters of the BERT encoder are frozen, and optimization is performed only on the TextCNN module and the classification layer. This avoids the loss of general language knowledge due to over-fine-tuning of pre-trained semantic representations, while allowing the model to learn local semantic patterns specific to language-related program scenarios. This stage uses a small learning rate and trains for 3-5 epochs to allow the model to initially adapt to the six-class classification task.

[0123] Next comes the layered unfreezing phase, which gradually unfreezes the top-level network of the BERT encoder (usually starting from the last layer), allowing the model to adjust its high-level semantic representations based on the semantic labels of language-based programs. Since semantic understanding in language-based program scenarios relies more on the contextual association of words, unfreezing the top-level Transformer block of BERT can effectively improve classification accuracy. In this phase, the learning rate is reduced to 5e-5, and training is conducted for 2-3 epochs, aiming to achieve an initial fusion of pre-trained knowledge and task features. Later in training, after layer-by-layer unfreezing, the model training transitions to a full fine-tuning phase, unfreezing all parameters of BERT. To prevent overfitting, gradient clipping and weight decay are introduced. Simultaneously, label smoothing techniques are used to mitigate prediction bias caused by uneven distribution of sample classes. This phase uses dynamic learning rate scheduling to balance convergence speed and generalization ability until the loss function stabilizes, thus completing the final optimization of the model.

[0124] After inputting the current statement into the trained semantic classification model, the enhanced semantic features input from the TextCNN module are extracted as the semantic features of the current statement.

[0125] During implementation, the speaker detection results, semantic features, and shot overview of each shot corresponding to the current sentence are spliced ​​together to obtain the comprehensive features of the current sentence.

[0126] The comprehensive feature sequence, composed of the k statements preceding the current statement and the comprehensive features corresponding to the current statement, is input into the trained decision model to predict the next shot.

[0127] During implementation, a camera switching decision dataset is constructed. This dataset collects recordings of real meetings, interviews, and other programs, and obtains audio and video data from each camera. The annotation information includes a camera shot overview for each camera, the text, timestamp, and semantic classification of each statement, whether the speaker is present in each frame of the corresponding audio and video segment of each camera, and the camera identifier for each shot.

[0128] During implementation, the semantic features of each sentence are extracted from the pre-trained semantic classification model by inputting the text of each sentence. The semantic features of each sentence, the speaker detection results of each corresponding audio and video segment, and the shot summary of each camera are used as the comprehensive features of that sentence.

[0129] In practice, the combined features of k+1 consecutive statements are used as the input of a sample, and the camera identifier of the next statement is used as the label to construct a training sample set.

[0130] In implementation, the decision-making module can employ a temporal neural network, such as a Long Short-Term Memory (LSTM) network. The decision-making module is trained using a camera switching decision dataset to obtain a well-trained module.

[0131] The comprehensive feature sequence, composed of the k statements preceding the current statement and the comprehensive features corresponding to the current statement, is input into the trained decision model to predict the probability of switching to each shot. The shot with the highest probability is selected as the identifier for the current shot to be switched to, which is then used as the final switching decision.

[0132] One specific implementation of this application discloses an intelligent broadcasting system for language-related scenarios, such as... Figure 5 As shown, it includes:

[0133] Multiple cameras are used to send the captured audio and video data to the intelligent broadcast server via a switcher;

[0134] The intelligent director server is used to predict the camera identifier of the next shot using an intelligent director method oriented towards language-based scenarios, and then send the camera identifier information of the next shot to the switcher.

[0135] A switcher is used to switch shots based on the camera identification information of the next shot.

[0136] In practice, the camera and switcher are connected via a specific HDBT cable. This cable can simultaneously power the camera, transmit audio and video signals, and transmit camera control signals, simplifying the system connection. The switcher loops out the audio and video signals contained in the HDBT via an SDI signal and inputs them to the intelligent broadcast server. After intelligently analyzing the audio and video signals, the server generates a switching decision signal, which communicates with the switcher via an RS422 serial port to control the switcher's screen output.

[0137] In practice, after receiving each video stream, the intelligent broadcast server performs speaker detection in parallel by starting a thread corresponding to the number of video streams. Simultaneously, a thread is started for semantic classification.

[0138] A preferred intelligent broadcasting method for language-related scenarios includes the following steps:

[0139] Extract the current sentence and the corresponding audio and video segments from each camera based on real-time speech recognition technology;

[0140] Speaker detection results for each audio / video segment are obtained by performing speaker detection on each audio / video segment based on the trained speaker detection model.

[0141] Extract semantic features of the current sentence based on a trained semantic classification model;

[0142] Based on the semantic features, the speaker detection results of each audio and video segment, and the shot summary of each camera, the comprehensive features corresponding to the current sentence are obtained.

[0143] The comprehensive feature sequence, composed of the k statements preceding the current statement and the comprehensive features corresponding to the current statement, is input into the trained decision model to predict the camera identifier for the next shot.

[0144] Preferably, the speaker detection model is an improved TalkNet model;

[0145] The improved TalkNet model includes a residual network;

[0146] The residual network includes multiple residual blocks, each residual block including two convolutional layers; a dual-pooling temporal attention module is added after the second convolutional layer; the dual-pooling temporal attention module is used to dynamically calibrate the spatiotemporal features of the input residual network.

[0147] The above-described method and system embodiments are based on the same principles, and their related aspects can be referenced from each other to achieve the same technical effects. For specific implementation processes, please refer to the foregoing embodiments, which will not be repeated here.

[0148] Electronic device example:

[0149] One specific implementation of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the intelligent broadcasting method for language-based scenarios in the method embodiment.

[0150] Examples of readable storage media:

[0151] One specific implementation of this application discloses a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the intelligent broadcasting method for language-based scenarios in the method embodiment.

[0152] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0153] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An intelligent broadcasting method for language-based scenarios, characterized in that, Includes the following steps: Extract the current sentence and the corresponding audio and video segments from each camera based on real-time speech recognition technology; Speaker detection results for each audio / video segment are obtained by performing speaker detection on each audio / video segment based on the trained speaker detection model. Extract semantic features of the current sentence based on a trained semantic classification model; Based on the semantic features, the speaker detection results of each audio and video segment, and the shot summary of each camera, the comprehensive features corresponding to the current sentence are obtained. The comprehensive feature sequence, composed of the k statements preceding the current statement and the comprehensive features corresponding to the current statement, is input into the trained decision model to predict the camera identifier for the next shot. The speaker detection model is an improved TalkNet model; The improved TalkNet model includes a residual network; The residual network includes multiple residual blocks, each residual block including two convolutional layers; a dual-pooling temporal attention module is added after the second convolutional layer; The dual-pooling temporal attention module is used to dynamically calibrate the spatiotemporal features of the input residual network; The dual-pooling temporal attention module dynamically calibrates spatiotemporal features using the following method: Rearrange the dimensions of spatiotemporal features; Global average pooling is applied to the spatial dimension of the rearranged spatiotemporal features to generate a joint channel-time description. Calculate channel sensitivity weights based on channel-time joint description; Time-sensitive weights are calculated based on channel-time joint description and channel-sensitive weights. The rearranged spatiotemporal features are weighted based on time-sensitive weights and restored to their original dimensional order to obtain dynamically calibrated spatiotemporal features.

2. The intelligent broadcasting method for language-based scenarios according to claim 1, characterized in that, The channel sensitivity weight is calculated using the following formula: ； in, The channel-time joint description of the c-th channel of the t-th frame represents the spatiotemporal characteristics, where T represents the number of image frames in the audio / video segment. Represents one-dimensional convolution. This indicates the channel-sensitive weight.

3. The intelligent broadcasting method for language-based scenarios according to claim 1, characterized in that, Based on the channel-time joint description and channel-sensitive weights, the time-sensitive weights are calculated using the following formula: ； ； in, The channel-temporal joint description of the c-th channel of the t-th frame image representing spatiotemporal features. Represents one-dimensional convolution. Indicates channel-sensitive weights. This indicates element-wise multiplication. Represents matrix addition. Represents the ReLU function. This represents the Sigmoid activation function. Indicates time-sensitive weights, Indicates an intermediate variable.

4. The intelligent broadcasting method for language-based scenarios according to claim 1, characterized in that, The semantic classification model includes: The BERT encoder is used to semantically encode input text to obtain semantic features; The TextCNN module is used to perform feature enhancement on semantic features to obtain enhanced semantic features; The classification layer is used for classification based on the enhanced semantic features.

5. The intelligent broadcasting method for language-based scenarios according to claim 4, characterized in that, The semantic classification model was fine-tuned using the following method: Freeze the parameters of the BERT encoder and optimize the TextCNN module and classification layer; Starting from the last layer and working down to the first layer, the parameters of the BERT encoder are unfrozen sequentially, and the BERT encoder, the TextCNN module, and the classification layer are optimized.

6. An intelligent broadcasting system for language-based scenarios, characterized in that, include: Multiple cameras are used to send the captured audio and video data to the intelligent broadcast server via a switcher; The intelligent director server is used to predict the camera identifier of the next shot using an intelligent director method oriented towards language-based scenarios, and then send the camera identifier information to the switcher. A switcher is used to switch shots based on the camera identification information of the next shot; The intelligent broadcasting method for language-based scenarios includes the following steps: Extract the current sentence and the corresponding audio and video segments from each camera based on real-time speech recognition technology; Speaker detection results for each audio / video segment are obtained by performing speaker detection on each audio / video segment based on the trained speaker detection model. Extract semantic features of the current sentence based on a trained semantic classification model; Based on the semantic features, the speaker detection results of each audio and video segment, and the shot summary of each camera, the comprehensive features corresponding to the current sentence are obtained. The comprehensive feature sequence, composed of the k statements preceding the current statement and the comprehensive features corresponding to the current statement, is input into the trained decision model to predict the camera identifier for the next shot. The speaker detection model is an improved TalkNet model; The improved TalkNet model includes a residual network; The residual network includes multiple residual blocks, each residual block including two convolutional layers; a dual-pooling temporal attention module is added after the second convolutional layer; the dual-pooling temporal attention module is used to dynamically calibrate the spatiotemporal features of the input residual network. The dual-pooling temporal attention module dynamically calibrates spatiotemporal features using the following method: Rearrange the dimensions of spatiotemporal features; Global average pooling is applied to the spatial dimension of the rearranged spatiotemporal features to generate a joint channel-time description. Calculate channel sensitivity weights based on channel-time joint description; Time-sensitive weights are calculated based on channel-time joint description and channel-sensitive weights. The rearranged spatiotemporal features are weighted based on time-sensitive weights and restored to their original dimensional order to obtain dynamically calibrated spatiotemporal features.

Images