A smart broadcasting system designed for symphony concerts

Abstract

This invention discloses an intelligent directing system for symphony concerts, comprising: an audio / video stream receiving and decoding module for simultaneously decoding the original audio and video data and sending the decoded audio / video data to a feature detection module; a score analysis module for identifying and analyzing the symphony score data and sending the identification results to a switching logic module; a feature detection module for analyzing the decoded audio / video data to identify various instruments appearing in the video frame, the shot size of the current frame, and the image quality, providing this information to the switching logic module; a switching logic module for determining whether the current frame can be switched out according to switching rules; and a directing visualization module for playing the frame switched out by the switching logic module. This invention enables automatic directing of symphony concert performances.

Description

Technical Field

[0001] This invention relates to the field of intelligent broadcasting technology, and in particular to an intelligent broadcasting system for symphony concert scenarios. Background Technology

[0002] With the increasing popularity of symphonic music, more and more people are paying attention to and appreciating this art form. This has led to a gradual expansion of the audience for symphonic performances, and their demands have become more diversified. It is necessary not only to meet the needs of live audiences but also to consider the needs of online viewers. Multi-camera human directors can be supplemented or replaced by artificial intelligence technology in the directing system. Intelligent directing systems, through automated camera switching and processing, provide better visual presentation, enabling audiences to better appreciate and experience symphonic performances.

[0003] Before discussing the introduction of artificial intelligence technology into program switching, it is necessary to first study the switching operations and patterns in traditional television programs (television or internet). In fixed locations and under fixed lighting, such as for guest interviews, concerts, and sporting events, multiple cameras are often used. The program director's responsibility is to select a relatively compelling signal closely related to the program content from a large number of camera signals and present it to the audience. This method allows television programs to present a more comprehensive and engaging effect from multiple angles and scenes. In television program production, different camera positions and different shots produce different effects, directly impacting the presentation and broadcast quality of the program.

[0004] Achieving intelligent directing faces two main challenges: first, selecting appropriate switching points to ensure a smooth transition from one shot to another and maintain the logical coherence between each shot transition in the edited video; and second, selecting the appropriate view from numerous candidate views to ensure the smooth flow of the final edited video story. In traditional production methods, the shooting and broadcasting directing stages are very demanding, putting immense pressure on staff and resulting in high production costs.

[0005] Therefore, using artificial intelligence technology to identify the real-time status of symphony orchestra scenes and combining it with the switching strategies of real broadcasters to achieve automatic or assisted switching of symphony orchestra program broadcasts has significant research significance and application value. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention proposes an intelligent broadcasting system for symphony concerts, which improves broadcasting efficiency, enables automated or manual broadcasting, and reduces costs.

[0007] To achieve the above objectives, the present invention provides an intelligent broadcasting system for symphony concert scenarios, comprising:

[0008] Audio and video stream receiving and decoding module: used to simultaneously decode the original audio and video data, and send the decoded audio and video data to the feature detection module;

[0009] The score analysis module is used to identify and analyze symphony score data and send the identification results to the switching logic module.

[0010] Feature detection module: used to analyze the decoded audio and video data, identify various musical instruments appearing in the video frame, the current scene size and the quality of the image in real time, and provide them to the switching logic module as reference information;

[0011] Switching logic module: Used to determine whether the current screen can be switched out according to the switching judgment rules;

[0012] The director visualization module is used to play the footage switched out by the switching logic module.

[0013] Preferably, the audio / video stream receiving and decoding module includes:

[0014] Decoding unit: used to decode audio and video streams, obtain valid information, and transmit the valid information in units of frames;

[0015] Multi-threaded processing unit: used to establish a communication channel with the feature detection module through a multi-threaded method.

[0016] Preferably, obtaining the valid information includes:

[0017] The address of the stream is stored in a variable, the address information of the stream is obtained, and the request timeout time is determined by setting the timeout parameter. Then, the stream is read asynchronously, the valid audio and video data in the stream is obtained and stored, and then the audio and video data in the stream is decoded frame by frame in a loop to obtain the valid information.

[0018] The multithreading method includes:

[0019] By cyclically reading the addresses and port numbers of all channels in the target file and starting a decoding and data dispatch thread for each channel, a communication channel is established between each decoding unit and the feature detection module.

[0020] Preferably, the total spectrum analysis module includes:

[0021] The score extraction unit is used to identify clefs in the score using the YOLOv5 model and to segment the score using image processing technology to extract the independent scores for each instrument group in the symphony.

[0022] Instrument identification unit: used to crop the instrument identification image before each line of the score according to the clef position definition in the independent score, save the instrument identification image, and perform feature extraction and processing on the instrument identification image based on the Tesseract engine to obtain the identification result of the instrument identification of the score.

[0023] Musical note information recognition unit: used to process the independent musical score using an end-to-end monophonic optical musical score recognition model to obtain the musical note recognition results of the musical score;

[0024] Performance information detection unit: Used to determine the performance status of the instrument in each measure of the independent score by using a pre-detection model to detect the position of the bar lines and whether there is other musical note information between the bar lines;

[0025] Audio start detection unit: used to locate the beginning information of a piece of music by detecting the start point in the audio.

[0026] Preferably, the spectral extraction unit determines the upper and lower boundaries of the spectral segment corresponding to each spectral number through a preset strategy, thereby determining the segmented region, specifically including:

[0027] The boundary segmentation method for the G spectral symbol is as follows:

[0028] y Top =y G_Top

[0029] y Bottom =y G_Bottom

[0030] In the formula, y Top To determine the upper boundary coordinates of the spectrum segment, y Bottom To determine the lower boundary coordinates of the spectrum segment, y G_Top y G_Bottom These are the ordinates of the upper boundary and lower boundary of the spectral symbol, respectively, identified by the G spectral symbol.

[0031] The boundary segmentation method for F and C spectral numbers is as follows:

[0032]

[0033] In the formula, y adjusted_Top The upper boundary coordinates of the segmentation are modified and adjusted according to the clef position definition, y adjusted_Bottom The corresponding spectral number is modified and adjusted according to its spectral position definition to define the lower boundary coordinates of the segmentation, y F_Top y F_Bottom These are the ordinates of the upper and lower boundaries of the clef, respectively, identified by the corresponding clef.

[0034] Preferably, the processing method of the end-to-end monophonic optical score recognition model includes:

[0035] First, features are extracted from the independent musical score images through convolutional layers. Then, the extracted features are used to perform sequence modeling based on a bidirectional long short-term memory network layer through recurrent layers. Finally, the features at each time step are classified through fully connected layers to obtain the final note prediction results.

[0036] The end-to-end monophonic optical score recognition model includes four convolutional layers, each followed by a max pooling layer, and the loss function is the CTC loss function.

[0037] Preferably, the audio start detection unit locates the beginning information of a piece of music based on the symphonic audio start detection method using audio energy and temporal characteristics, specifically including:

[0038] Load the decoded audio frames and obtain the audio data and sampling rate;

[0039] The audio data is subjected to a short-time Fourier transform to obtain the amplitude spectrum. The values ​​of the amplitude spectrum are squared and summed along the time axis to obtain the energy of the audio frame. The audio frame energy curve is plotted, the energy curve is smoothed, and an adaptive energy threshold is set.

[0040] The starting point in the audio data is detected. The audio data and the adaptive energy threshold are used as inputs, and the frame index corresponding to the starting point is output. Then, the frame index is converted into time information.

[0041] Preferably, the feature detection module includes:

[0042] Musical instrument image detection unit: used to analyze the video stream in a symphony concert scene and identify various musical instruments appearing in the video frames in real time;

[0043] Scene recognition unit: Used to detect each frame of the video stream frame by frame using a scene recognition model based on the convolutional neural network ResNet50, and output the corresponding scene label;

[0044] Blur detection unit: used to determine whether the lens is in adjustment state by using a blur detection algorithm, and filter out unusable blurry images; wherein, the blur detection algorithm evaluates the degree of blur of the image by calculating the average amplitude value of high frequency components in the image and comparing it with a preset value.

[0045] Preferably, the instrument image detection unit is used to perform frame-by-frame processing on the video stream, input each processed frame image into the improved YOLOv5 model for instrument detection, detect various types of instruments appearing in the video frame, and mark the position of the instruments in the image.

[0046] The improved YOLOv5 model adds an object detection layer to the feature fusion part of YOLOv5. The object detection layer includes several convolutional operations and a C3 module to extract and detect subtle features.

[0047] Preferably, the switching determination rule includes:

[0048] Instrument priority selection rule: Based on the recognition results of the score analysis module and the recognition analysis results of the instrument image detection unit, prioritize switching the screen containing the instrument being played in the current time period;

[0049] Shot type determination rules: used to determine the next cut to the shot and the time that the current shot can be broadcast based on the current shot type;

[0050] Fuzzy judgment rule: Used to determine whether the current scene can be cut out based on whether the image is blurry.

[0051] Compared with the prior art, the present invention has the following advantages and technical effects:

[0052] (1) This invention processes and identifies the total score and audio of a symphony to obtain the time period information of various instruments in the performance of a symphony concert; an additional detection layer is added on the basic structure of YOLOv5, and a series of convolution operations and C3 modules are used to help extract and detect more subtle features, so that the model can more effectively distinguish and detect small targets for the recognition of small instruments in panoramic images.

[0053] (2) By identifying the feature information in the audio and video stream of a symphony and the instrument performance time in the symphony score, and selecting the signal according to the preset switching logic rules, the present invention realizes the automatic directing or automatic editing of symphony concert performances. This not only reduces the directing cost and the pressure on the directing staff, but also makes it easier and faster to present more performances to a wider audience in video form, and reduces costs. Attached Figure Description

[0054] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is a schematic diagram of an intelligent broadcasting system for symphony concert scenarios according to an embodiment of the present invention;

[0056] Figure 2 This is a flowchart illustrating the implementation of the total spectrum analysis module in an embodiment of the present invention.

[0057] Figure 3These are example diagrams illustrating several spectral segmentation effects according to embodiments of the present invention;

[0058] Figure 4 This is a structural diagram of the end-to-end neural network model according to an embodiment of the present invention;

[0059] Figure 5 This is an example diagram illustrating the performance of musical measures according to an embodiment of the present invention;

[0060] Figure 6 This is a structural diagram of YOLOv5 with a new detection layer added according to an embodiment of the present invention;

[0061] Figure 7 This is a flowchart illustrating the switching logic of an embodiment of the present invention;

[0062] Figure 8 This is an example diagram of the interface in an embodiment of the present invention. Detailed Implementation

[0063] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0064] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0065] This invention proposes an intelligent broadcasting system for symphony concert scenarios, such as... Figure 1 ,include:

[0066] Audio and video stream receiving and decoding module: used to simultaneously decode the original audio and video data, and send the decoded audio and video data to the feature detection module;

[0067] The score analysis module is used to identify and analyze symphony score data and send the identification results to the switching logic module.

[0068] Feature detection module: used to analyze the decoded audio and video data, identify various musical instruments appearing in the video frame, the current scene size and the quality of the image in real time, and provide them to the switching logic module as reference information;

[0069] Switching logic module: Used to determine whether the current screen can be switched out according to the switching judgment rules;

[0070] The director visualization module is used to play the footage switched out by the switching logic module.

[0071] This embodiment identifies the feature information in the audio and video stream of a symphony and the instrument performance periods in the symphony score, and selects signals according to preset switching logic rules. This enables automatic directing or automatic post-production editing of symphony concert performances, which not only reduces directing costs and the pressure on directing personnel, but also allows more performances to be presented to a wider audience in video format more conveniently and quickly, while also reducing costs.

[0072] Furthermore, the audio / video stream receiving and decoding module includes:

[0073] Decoding unit: used to decode audio and video streams, obtain valid information, and transmit the valid information in units of frames;

[0074] Multi-threaded processing unit: used to establish a communication channel with the feature detection module through a multi-threaded method.

[0075] Specifically, the audio / video stream receiving and decoding module has the functions of decoding and transmitting data. After the original audio and video streams enter the system, the audio / video stream receiving and decoding module first starts a data transmission thread to determine the communication ports between various functional modules, opens the communication function, extracts the original data in the decoding thread, and finally sends the decoded data to the feature detection module for feature analysis through the data transmission thread.

[0076] For stream input decoding, this embodiment uses the Python library function PyAV in the decoding unit to decode the audio and video streams to obtain valid information. The decoding process specifically consists of the following steps:

[0077] The `av.open()` function is used to store the stream's address in a variable, retrieve the stream's address information, and set the `timeout` parameter to determine the request timeout period. Then, the `streams.get()` function is called to retrieve and store the valid audio and video data from the stream. Finally, a loop is used to decode the audio and video data frame by frame, and the decoded data is transmitted frame by frame.

[0078] At this point, the system calls a callback function to extract the data obtained during decoding, such as DTS (Decoding Time Stamp), PTS (Display Time Stamp), stream address, and sample rate, and saves it to a new object.

[0079] The broadcast control system needs to handle the selective switching of multiple audio and video streams simultaneously. Therefore, it must be able to transmit multiple channels of audio and video data at the same time, and ensure that each channel is independent and does not interfere with the others. This embodiment uses a multi-threaded processing unit to synchronously enable multi-threading to achieve the decoding and transmission of audio and video for each channel. The detailed steps for enabling multi-threading are as follows:

[0080] First, the system iteratively reads the addresses and port numbers of all channels from the JSON file and starts a decoding and data dispatch thread for each channel. Then, it uses the `socket` method from the ZMQ library and the port numbers in the JSON file to establish communication channels between each decoding unit and the feature detection module. Finally, it opens a new communication port to communicate with the switching logic module and calls a callback function to send the tags obtained by the feature detection module to the switching logic module.

[0081] Using multithreading technology to decode and distribute audio and video streams can make the system's logical structure clearer, and the operation of each thread is independent of each other, which facilitates analysis.

[0082] Furthermore, the total spectrum analysis module, such as Figure 2 As shown, it includes:

[0083] The score extraction unit is used to identify clefs in the score using the YOLOv5 model and to segment the score using image processing technology to extract the independent scores for each instrument group in the symphony.

[0084] Instrument identification unit: used to crop the instrument identification image before each line of the score according to the clef position definition in the independent score, save the instrument identification image, and perform feature extraction and processing on the instrument identification image based on the Tesseract engine to obtain the identification result of the instrument identification of the score.

[0085] Musical note information recognition unit: used to process the independent musical score using an end-to-end monophonic optical musical score recognition model to obtain the musical note recognition results of the musical score;

[0086] Performance information detection unit: Used to determine the performance status of the instrument in each measure of the independent score by using a pre-detection model to detect the position of the bar lines and whether there is other musical note information between the bar lines;

[0087] Audio start detection unit: used to locate the beginning information of a piece of music by detecting the start point in the audio.

[0088] Specifically, after the score is input into the score analysis module, it first identifies and locates the clefs to obtain the positions of each part, segments the score into images, and uses OMR (Optical Music Recognition) to identify the musical notes. Then, it uses OCR to identify the instrument identifiers, and finally combines this with audio start detection technology to obtain the actual start time of the symphony audio corresponding to the score. Next, by combining the performance time information of each instrument in the score, it summarizes the instrument performance information to assist the symphony concert director in switching between instruments. Specifically, this module can determine which instruments are playing and which are not playing within a certain time period. For example, in T... a To T b During a given time period, instrument X is playing, while other instruments are not. This information provides reference data for the switching logic module of the symphony concert directing system, enabling it to switch and guide more accurately.

[0089] Furthermore, the spectral extraction unit determines the upper and lower boundaries of the spectral segment corresponding to each spectral symbol through a preset strategy, thereby determining the segmented region, specifically including:

[0090] The key to score segment recognition is acquiring the feature information and identification location information of each score segment to determine the position of the score segment to be segmented. This embodiment uses the YOLOv5 model to identify the clefs in the score and utilizes image processing technology to segment the score into individual scores. This divide-and-conquer strategy effectively reduces complexity and error rate, improves the accuracy of each score note recognition, and provides a reliable foundation for subsequent instrument information extraction.

[0091] The dataset consists of 100 symphony scores collected from the internet, each with approximately 16 clefs, resulting in a total of about 1600 clefs. This embodiment uses the Labelimg annotation tool to annotate the dataset. During annotation, the principle of ensuring clefs are completely within the annotation box was followed to guarantee accuracy and completeness. After annotation, a YOLO-formatted annotation file was exported to prepare for subsequent model training.

[0092] This embodiment's clef recognition model was trained using 80 symphony score images, with a training set to validation set ratio of 72:8 and a test set of 20 images. The processor used in this embodiment was an 11th Gen Intel(R) Core(TM) i9-11900K @ 3.50GHz, an NVIDIA GeForce RTX 3090 graphics card, 64GB of RAM, and a Windows 10 64-bit operating system. Python version 3.7 was used. The final clef recognition accuracy reached 89.4%.

[0093] In musical notation, the treble clef, bass clef, and alto clef are typically in fixed positions, occupying different locations on the staff to indicate pitch. Generally, their positions are defined as follows:

[0094] The treble clef (G clef) is usually located on the second line of the staff and represents the note G.

[0095] Bass clef (F clef): Usually located on the fourth line of the staff, representing the note F.

[0096] Alto clef (C clef): Usually located on the third line of the staff, representing the note C.

[0097] This embodiment uses a clef recognition model trained with YOLOv5 to locate the clef position before each staff. Based on these position definitions, the upper and lower boundaries of each staff are determined, thereby defining the segmented region. The original image is then cropped to obtain the segmented image of each staff.

[0098] During the segmentation process, this embodiment follows the strategy of determining the upper and lower boundaries of the segment corresponding to each spectral symbol:

[0099] The boundary segmentation method for the G spectral symbol is as follows:

[0100] y Top =y G_Top

[0101] y Bottom =y G_Bottom

[0102] In the formula, y Top To determine the upper boundary coordinates of the spectrum segment, y Bottom To determine the lower boundary coordinates of the spectrum segment, y G_Top y G_Bottom These are the ordinates of the upper boundary and lower boundary of the spectral symbol, respectively, identified by the G spectral symbol.

[0103] The boundary segmentation method for F and C spectral numbers is as follows:

[0104]

[0105] In the formula, y adjusted_top The upper boundary coordinates of the segmentation are modified and adjusted according to the clef position definition, y adjusted_Bottom The corresponding spectral number is modified and adjusted according to its spectral position definition to define the lower boundary coordinates of the segmentation, y F_Top y F_Bottom These represent the ordinates of the upper and lower boundaries of the clef, respectively. The segmentation result is shown in the image below. Figure 3 .

[0106] Furthermore, the instrument identification unit is used to crop the instrument identification image before each line of the score according to the clef position definition in the independent score, save the instrument identification image, and perform feature extraction and processing on the instrument identification image based on the Tesseract engine to obtain the identification result of the instrument identification of the score.

[0107] Specifically, instrument markings refer to the abbreviations or full names preceding each part in the score. They help performers clearly understand the allocation of each part and the performance requirements. These markings can be used in multiple languages, including but not limited to English, French, German, Italian, and Spanish.

[0108] To achieve instrument identification, this embodiment uses Google's Tesseract OCR engine. Tesseract is an open-source OCR engine developed by HP Labs and maintained by Google, specifically designed to convert images into text and supporting multiple languages. Using Tesseract for instrument identification can effectively convert the markings on the musical score into processable text data.

[0109] The instrument identification images in the musical score are obtained from the clef recognition model described earlier, and then the coordinate boundaries of the images to be cut are determined. Simultaneously, based on the clef position definition described earlier, the instrument identification images before each line of the score are cropped and saved for subsequent optical character recognition (OCR). This process ensures that the instrument identifications in the score are accurately extracted and recognized, providing a reliable data foundation for further music analysis and processing. Before recognition, Tesseract performs a series of preprocessing steps on the input musical score identification images to improve recognition accuracy. This includes image binarization, noise reduction, character and text line segmentation, and tilt correction, ensuring that the input image presents clear and accurate features during recognition.

[0110] Tesseract then segments the characters in the text line into individual characters so that each character can be recognized independently. This step involves detecting whitespace between characters or using dividing lines to ensure that each character can be accurately extracted and recognized. During feature extraction, Tesseract uses convolutional neural networks to extract character features, effectively distinguishing the shape and structure of different characters.

[0111] Finally, Tesseract combines the recognized characters to generate editable text output, which is the recognition result of the musical instrument identifiers.

[0112] Furthermore, the musical note information recognition unit is used to process the independent musical score using an end-to-end monophonic musical score optical musical score recognition model to obtain the musical note recognition results of the sub-score;

[0113] Specifically, to achieve accurate sheet music recognition, this embodiment trains a model specifically for sheet music recognition using a neural network. This model takes a single-line sheet music image as input, processes the image, and outputs musical note information. This embodiment selects an end-to-end monophonic optical sheet music recognition model, which integrates convolutional neural networks (CNNs) and recurrent neural networks (RNNs), and uses the CTC loss function during training. The structure of the end-to-end monophonic optical sheet music recognition model is as follows... Figure 4 As shown.

[0114] The model first extracts image features through convolutional layers, then models the sequence of these features using recurrent layers. Specifically, the model consists of four convolutional layers, each followed by a max-pooling layer. After image feature extraction, the model uses two bidirectional long short-term memory (BLSTM) layers to model the sequence. The BLSTM layers effectively capture long-term dependencies in the sequence, helping to accurately predict note sequences in musical scores. Finally, the model classifies the features at each time step through a fully connected layer to obtain the final note prediction. Stochastic gradient descent is used to train the model and optimize the parameters, with the CTC loss function employed.

[0115] Furthermore, the performance information detection unit is used to employ a pre-detection model to determine the performance status of the instrument in each measure of the independent score by detecting the position of the bar lines and whether there is other musical note information between the bar lines;

[0116] Specifically, after obtaining the musical note information for each part of the score, the next crucial step is to determine, based on music theory, whether the instrument is playing in each measure. This embodiment employs a note-based inference method, considering whether the measure contains conventionally played notes, such as quarter notes, eighth notes, and sixteenth notes. Conversely, if the measure only contains rests, whole rests, or other rests without any played notes, this embodiment infers that the instrument corresponding to the part of the score in that measure is not playing. Conversely, if the measure contains played notes, it is inferred that the instrument corresponding to the part of the score in that measure is playing. For example, as... Figure 5 As shown, there is a whole rest in the first measure on the left of the second line, so the instrument corresponding to this measure is not played during this measure. The following three measures all have notes played, so it can be inferred that the instrument corresponding to this measure is played during this measure.

[0117] In this embodiment, the pre-detection model, while traversing the recognition results of the bisection score, detects the position of bar lines and analyzes whether there is other musical note information between the bar lines to determine the performance status of each measure. This process, through the detection and analysis of symbols within the measure, forms a judgment information list. This list records in detail the instrument performance status in the corresponding bisection score under each measure, and appends it to the overall instrument performance analysis data, providing important reference for subsequent director switching operations.

[0118] Furthermore, the audio start detection unit locates the beginning information of a piece of music based on the symphonic audio start detection method using audio energy and temporal features, specifically including:

[0119] Use the Librosa library to load the decoded audio frames and obtain the audio data and sampling rate;

[0120] Using the STFT function of the Librosa library, a short-time Fourier transform is performed on the audio signal to obtain the amplitude spectrum. The values ​​of the amplitude spectrum are squared and summed along the time axis to obtain the energy of the audio frame. The audio frame energy curve is plotted, the energy curve is smoothed, and an adaptive energy threshold is set.

[0121] The onset_detect function in the Librosa library is used to detect the start point in the audio. The audio data and the adaptive energy threshold are taken as input, and the frame index corresponding to the start point is output. Then, the frame index is converted into time information.

[0122] Specifically, this embodiment uses the Librosa library to implement audio start detection. The specific steps are as follows:

[0123] (1) Load audio file:

[0124] First, use the Librosa library to load the audio and obtain the audio data y and the sampling rate sr.

[0125] (2) Calculate the Short-Time Fourier Transform (STFT):

[0126] Using the STFT function from the Librosa library, a short-time Fourier transform is performed on the audio signal to obtain the amplitude spectrum. The STFT can be expressed as:

[0127]

[0128] In the formula, X[m,k] is the result of STFT, x[n] is the input signal, w[nm] is the window function, and e -j2πkn / N is the rotation factor, and N is the window length.

[0129] (3) Calculate energy:

[0130] The energy of an audio frame is obtained by summing the squares of the amplitude spectrum obtained from the short-time Fourier transform along the time axis. The formula for calculating the energy is:

[0131]

[0132] Where E[n] is the energy of the nth frame.

[0133] (4) Smoothing and adaptive energy threshold:

[0134] A smoothing process is performed on the energy curve to reduce peaks and make the energy threshold calculation more stable. To adapt to the characteristics of different symphonic audio frequencies, this embodiment sets an adaptive energy threshold, which can be expressed as:

[0135] threshold=smooth_factor×max(E smooth Where threshold is the adaptive energy threshold, and smooth_factor is a coefficient that adjusts the relationship between the smoothed energy and the threshold.

[0136] (5) Audio start detection:

[0137] The `onset_detect` function from the Librosa library is used to detect the start point in the audio. This function takes audio data and a threshold as input and determines the start point based on an algorithm. This step can be represented as:

[0138] onset_frames=onset_detect(y, sr, on set_envelope=E smooth (threshold, ...)

[0139] Here, `onset_frames` is the frame index corresponding to the detected start point. The frame index is then converted into time information for subsequent analysis.

[0140] onset_times=frames_to_time(onset_frames, sr)

[0141] This process provides an effective method for locating the beginning of symphonic audio in this embodiment, providing a reliable basis for the score analysis module to obtain information about the time intervals of instrument performance.

[0142] Furthermore, the feature detection module includes:

[0143] Musical instrument image detection unit: used to analyze the video stream in a symphony concert scene and identify various musical instruments appearing in the video frames in real time;

[0144] Scene recognition unit: Used to detect each frame of the video stream frame by frame using a scene recognition model based on the convolutional neural network ResNet50, and output the corresponding scene label;

[0145] Blur detection unit: used to determine whether the lens is in adjustment state by using a blur detection algorithm, and filter out unusable blurry images; wherein, the blur detection algorithm calculates the average amplitude value of high frequency components in the image and compares it with a preset value to evaluate the degree of blur of the image.

[0146] Furthermore, the instrument image detection unit is used to perform frame segmentation processing on the video stream, input each processed frame image into the improved YOLOv5 model for instrument detection, detect various types of instruments appearing in the video frame, and mark the position of the instruments in the image.

[0147] The improved YOLOv5 model adds an object detection layer to the feature fusion part of YOLOv5. The object detection layer includes several convolutional operations and a C3 module to extract and detect subtle features.

[0148] Specifically, this embodiment adds a small target detection layer to the feature fusion part of YOLOv5. This modification aims to improve the model's ability to detect small targets. Specifically, the new target detection layer is located in the model's head and extracts and detects features of smaller targets. The design of this layer includes a series of convolutional operations (Conv) and a C3 module (C3) to improve the model's ability to extract features of small targets in dense scenes.

[0149] The newly added detection layer improves the detection performance of small musical instruments in panoramic or wide-angle scenarios primarily through the following aspects: First, a dedicated layer for small object detection is added to the existing feature fusion part, allowing the model to focus more on relevant features when processing small objects, thereby enhancing feature extraction capabilities. This additional detection layer employs a series of convolutional operations and the C3 module, which helps extract and detect more subtle features, enabling the model to more effectively distinguish and detect small objects. Second, in dense symphony orchestra scenes, small objects are often occluded or mixed in with other objects, making accurate detection difficult. By adding a dedicated layer for small objects, the model can better separate and detect these small objects, thus improving detection performance in dense scenes and further enhancing the detection effect in such scenarios. The improved YOLOv5 structure is as follows: Figure 6 As shown.

[0150] Furthermore, the scene recognition unit is used to cyclically detect each frame of the video stream using a scene recognition model based on the ResNet50 convolutional neural network, and output the corresponding scene label;

[0151] Specifically, this embodiment introduces a shot recognition unit, which identifies different shot sizes in the video stream to provide a reference for the director and realize intelligent switching of shot sizes.

[0152] Based on the definition of shot type and industry practice, this embodiment classifies shot types into the following 6 categories:

[0153] ECS (Extreme Close-up): Focuses on the subject's shoulders and above;

[0154] Close-up (CS): A shot that includes the waist and above of a person;

[0155] Medium shot (MS): A shot that covers the area from the knees up to the body of a person;

[0156] Full shot (FS): A shot that shows the entire body of a person;

[0157] Long shot (LS): wide-ranging, with no clearly defined main subject;

[0158] Shot without people (OS): The scope is smaller than that of a long shot, and the main subject is not a person.

[0159] This embodiment employs a scene size recognition model based on the ResNet50 convolutional neural network. This model iteratively detects each frame of the video stream frame by frame and outputs its corresponding scene size label, thereby predicting and recognizing the scene size in a symphony concert scene. The introduction of the scene size recognition unit provides intelligent assistance to the directing system, enabling it to automatically adjust the scene size of the output image according to different contexts and needs of the concert program, thus enhancing the audience's viewing experience. Through the application of this module, the directing team can more flexibly utilize different scene size switching techniques, making the program visuals more vivid and layered, further improving the quality and appeal of the concert program.

[0160] Furthermore, the blur detection unit is used to determine whether the lens is in an adjustment state using a blur detection algorithm, and to filter out unusable blurry images; wherein, the blur detection algorithm calculates the average amplitude value of the high-frequency components in the image and compares it with a preset value to evaluate the degree of blur in the image.

[0161] Furthermore, the switching logic module is used to determine whether the current screen can be switched out according to the switching judgment rules; the switching logic flowchart is as follows. Figure 7 As shown.

[0162] The switching judgment rules include:

[0163] Instrument priority selection rule: Based on the recognition results of the score analysis module and the recognition analysis results of the instrument image detection unit, prioritize switching the screen containing the instrument being played in the current time period;

[0164] Shot type determination rules: used to determine the next cut to the shot and the time that the current shot can be broadcast based on the current shot type;

[0165] Fuzzy judgment rule: Used to determine whether the current scene can be cut out based on whether the image is blurry.

[0166] Specifically, the instrument selection rule employs a preferred instrument selection function (Preferred_instruments). This function combines the results from two modules: one is the performance time interval information for various instruments from the score analysis module, and the other is the instrument type information currently present in each frame from the feature detection module. Based on the combined information from both, the function identifies which frame in the current time interval has an instrument playing and determines the frame that should be prioritized for selection.

[0167] The shot size judgment rule uses a shot size judgment function (Judge_shotlabel) that takes the shot size of the current frame as input and outputs a flag indicating whether a shot size has been found. This flag indicates the optimal choice for the next shot, meaning that a combination of shot sizes will have a better visual effect. This embodiment summarizes the switching probability matrix for shot size combinations by analyzing the switching rules of a large number of concert broadcasts. In the shot size judgment function, firstly, based on the current shot size, the required probability row in the switching probability matrix is ​​extracted. Then, a module that generates random numbers using probability is used to select the next shot size for switching. Finally, the shot size of the current frame is compared with the next shot size. If they are the same, the flag indicating that a shot size has been found is set to True; otherwise, it is set to False, and the flag is returned.

[0168] The fuzzy judgment rule employs a time-based selection function: This embodiment analyzes the switching rules of numerous concert broadcasts to summarize the playback duration probability matrix for different shot types. The time-based selection function receives the current shot type as input and outputs the playback duration. In this function, the required probability rows in the playback duration probability matrix are first extracted based on the current shot type. Then, random playback durations are generated using these probabilities.

[0169] These three functions together constitute the design and implementation of the switching logic, enabling the broadcasting system to intelligently switch scenes based on factors such as instrument performance, shot size, and duration, thereby enhancing the program's watchability and appeal.

[0170] Furthermore, the director visualization module is used to play the footage switched out by the switching logic module.

[0171] Specifically, to facilitate system testing, this embodiment designs a simple user interface for the intelligent broadcasting system. This interface is constructed manually using the PyQt5 framework. Unlike using QtDesigner to generate the interface, the UI code in this embodiment is written directly in Python scripts, constructing the interface by calling PyQt5 classes and methods.

[0172] The main function of the interface is to manage multiple RTMP stream addresses and related port information, and provides functions such as adding channels, deleting channels, and applying modifications. The entire interface layout uses QVBoxLayout and QHBoxLayout, as well as QScrollArea controls to display channel information. Users can add, delete, and modify channels using buttons. Each channel group consists of a QGroupBox, used to display the RTMP stream address information of that channel group. Within the channel group, QLineEdit controls are used to input and display the address, DispatchPort, and ReportPort information for each channel. QPushButton controls are used to implement the functions of adding and deleting channels. Clicking the "Apply Modify" button triggers the processing of the channel information entered in the interface and updates the relevant JSON configuration file. In addition, several child processes are launched to execute other system tasks, such as ConcertDetectWorker.py, Extracor_Concert.py, and Concert_Channel_Handler.py. An example interface diagram is shown below. Figure 8 As shown.

[0173] To address the issue that the accuracy of polyphonic music recognition is not suitable for application in the system, this embodiment proposes a method based on symphony score analysis. By processing and recognizing the full score and audio of a symphony, the time interval information of various instruments in a symphony concert is obtained.

[0174] This invention proposes an instrument image detection model suitable for symphony concert scenes. Addressing the lack of existing instrument image datasets for symphony concert scenes, this embodiment proposes a scheme to construct a custom instrument image dataset and introduces methods for dataset labeling and augmentation. Furthermore, addressing the issue of low accuracy of the YOLOv5 model in detecting small instruments in panoramic and extreme panoramic images of symphony concert scenes, this embodiment adds an additional detection layer to the basic YOLOv5 structure. This layer employs a series of convolutional operations and the C3 module, which helps extract and detect more subtle features, enabling the model to more effectively distinguish and detect small targets.

[0175] This embodiment analyzes the switching rules of a large number of concerts and summarizes the basic switching indicators that directors need to rely on, including not switching when adjusting the shot, smooth transitions between shots, and different playback lengths for shots of different shot sizes.

[0176] Finally, the overall framework of the intelligent broadcasting system was designed, including modules such as decoding, spectrum analysis, feature extraction, and switching selection, to achieve automated video switching.

[0177] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An intelligent broadcasting system for symphony concert scenarios, characterized in that, include: Audio and video stream receiving and decoding module: used to simultaneously decode the original audio and video data, and send the decoded audio and video data to the feature detection module; The score analysis module is used to identify and analyze symphony score data and send the identification results to the switching logic module. Feature detection module: used to analyze the decoded audio and video data, identify various musical instruments appearing in the video frame, the current scene size and the quality of the image in real time, and provide them to the switching logic module as reference information; Switching logic module: Used to determine whether the current screen can be switched out according to the switching judgment rules; The director visualization module is used to play the footage switched out by the switching logic module. The total spectrum analysis module includes: The score extraction unit is used to identify clefs in the score using the YOLOv5 model and to segment the score using image processing technology to extract the independent scores for each instrument group in the symphony. Instrument identification unit: used to crop the instrument identification image before each line of the score according to the clef position definition in the independent score, save the instrument identification image, and perform feature extraction and processing on the instrument identification image based on the Tesseract engine to obtain the identification result of the instrument identification of the score. Musical note information recognition unit: used to process the independent musical score using an end-to-end monophonic optical musical score recognition model to obtain the musical note recognition results of the musical score; Performance information detection unit: Used to determine the performance status of the instrument in each measure of the independent score by using a pre-detection model to detect the position of the bar lines and whether there is other musical note information between the bar lines; Audio start detection unit: used to locate the beginning information of a piece of music by detecting the start point in the audio; The switching determination rules include: Instrument priority selection rule: Based on the playing time interval information of various instruments and the instrument type information currently present in each screen, it finds which screen has an instrument playing in the current time interval and determines the screen that should be cut out first. Shot type determination rule: It is used to extract the probability row required in the switching probability matrix based on the current shot type of the screen, determine the next cut-out shot, and extract the probability row required in the playback duration probability matrix based on the current shot type. It uses probability to generate a random playback duration and determines the time when the current screen can be broadcast. Fuzzy judgment rule: Used to determine whether the current scene can be cut out based on whether the image is blurry.

2. The intelligent broadcasting system for symphony concert scenarios according to claim 1, characterized in that, The audio / video stream receiving and decoding module includes: Decoding unit: used to decode audio and video streams, obtain valid information, and transmit the valid information in units of frames; Multi-threaded processing unit: used to establish a communication channel with the feature detection module through a multi-threaded method.

3. The intelligent broadcasting system for symphony concert scenarios according to claim 2, characterized in that, Obtaining the valid information includes: The address of the stream is stored in a variable, the address information of the stream is obtained, and the request timeout time is determined by setting the timeout parameter. Then, the stream is read asynchronously, the valid audio and video data in the stream is obtained and stored, and then the audio and video data in the stream is decoded frame by frame in a loop to obtain the valid information. The multithreading method includes: By cyclically reading the addresses and port numbers of all channels in the target file and starting a decoding and data dispatch thread for each channel, a communication channel is established between each decoding unit and the feature detection module.

4. The intelligent broadcasting system for symphony concert scenarios according to claim 1, characterized in that, The spectral extraction unit determines the upper and lower boundaries of the spectral segment corresponding to each spectral number through a preset strategy, thereby determining the segmented region, specifically including: The boundary segmentation method for the G spectral symbol is as follows: In the formula, To determine the upper boundary coordinates of the spectrum segment, To determine the lower boundary coordinates of the spectrum segment, These are the ordinates of the upper boundary and lower boundary of the spectral symbol, respectively, identified by the G spectral symbol. The boundary segmentation method for F and C spectral numbers is as follows: In the formula, The upper boundary coordinates of the segment are modified and adjusted according to the clef position definition for the corresponding clef. The lower boundary coordinates of the segment are modified and adjusted according to the definition of the clef position for the corresponding clef. These are the ordinates of the upper and lower boundaries of the clef, respectively, identified by the corresponding clef.

5. The intelligent broadcasting system for symphony concert scenarios according to claim 1, characterized in that, The processing method for the end-to-end monophonic optical score recognition model includes: First, features are extracted from the independent musical score images through convolutional layers. Then, the extracted features are used to perform sequence modeling based on a bidirectional long short-term memory network layer through recurrent layers. Finally, the features at each time step are classified through fully connected layers to obtain the final note prediction results. The end-to-end monophonic optical score recognition model includes four convolutional layers, each followed by a max pooling layer, and the loss function is the CTC loss function.

6. The intelligent broadcasting system for symphony concert scenarios according to claim 1, characterized in that, The audio start detection unit locates the beginning information of a piece of music based on the symphonic audio start detection method using audio energy and temporal characteristics, specifically including: Load the decoded audio frames and obtain the audio data and sampling rate; The audio data is subjected to a short-time Fourier transform to obtain the amplitude spectrum. The values ​​of the amplitude spectrum are squared and summed along the time axis to obtain the energy of the audio frame. The audio frame energy curve is plotted, the energy curve is smoothed, and an adaptive energy threshold is set. The starting point in the audio data is detected. The audio data and the adaptive energy threshold are used as inputs, and the frame index corresponding to the starting point is output. Then, the frame index is converted into time information.

7. The intelligent broadcasting system for symphony concert scenarios according to claim 1, characterized in that, The feature detection module includes: Musical instrument image detection unit: used to analyze the video stream in a symphony concert scene and identify various musical instruments appearing in the video frames in real time; Scene recognition unit: Used to detect each frame of the video stream frame by frame using a scene recognition model based on the convolutional neural network ResNet50, and output the corresponding scene label; Blur detection unit: used to determine whether the lens is in adjustment state by using a blur detection algorithm, and filter out unusable blurry images; wherein, the blur detection algorithm evaluates the degree of blur of the image by calculating the average amplitude value of high frequency components in the image and comparing it with a preset value.

8. The intelligent broadcasting system for symphony concert scenarios according to claim 7, characterized in that, The instrument image detection unit is used to perform frame-by-frame processing on the video stream, input each processed frame image into the improved YOLOv5 model for instrument detection, detect various instruments appearing in the video frame, and mark the position of the instrument in the image. The improved YOLOv5 model adds an object detection layer to the feature fusion part of YOLOv5. The object detection layer includes several convolutional operations and a C3 module to extract and detect subtle features.

Images