Intelligent director system for symphony concert scene

Abstract

The invention discloses an intelligent director system for a symphony concert scene, and the system comprises an audio and video stream receiving and decoding module which is used for decoding original audio and video data at the same time, and transmitting the decoded audio and video data to a feature detection module; the total spectrum analysis module is used for identifying and analyzing the total symphony spectrum data and sending an identification result to the switching logic module; the feature detection module is used for analyzing the decoded video and audio data, identifying various musical instruments appearing in a video frame picture, the scene type of the current picture and the quality of the picture, and providing the identified musical instruments, the scene type and the quality to the switching logic module as reference information; the switching logic module is used for judging whether the current picture can be cut out or not according to a switching judgment rule; and the director visualization module is used for playing the picture cut by the switching logic module. According to the invention, automatic broadcasting of the symphony concert performance is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent directing technology, and in particular to an intelligent directing system for symphony concert scenes. Background Art

[0002] As symphony becomes more popular, more and more people begin to pay attention to and appreciate this art form. This has led to a gradual expansion of the audience group that enjoys symphony performances, and their needs have become more diversified. It is necessary not only to meet the needs of live audiences, but also to consider the needs of online audiences. Multi-camera manual directing can be assisted or replaced by the directing system through the application of artificial intelligence technology. The intelligent directing system provides better visual presentation through automated camera image switching, allowing the audience to better appreciate and experience symphony performances.

[0003] Before discussing the introduction of artificial intelligence technology into director switching, we first need to study the director switching operations and rules in traditional TV programs (TV or Internet). In fixed venues and fixed lighting, such as guest interviews, concerts, sports games, etc., multiple cameras are often used. The responsibility of the program director is to select a relatively exciting signal that is closely related to the program content from a large number of camera signals and show it to the audience. In this way, TV programs can be presented from multiple angles and multiple scenes to present a more comprehensive and fascinating effect. In TV program production, different camera positions and different lenses have different presentation effects, which directly affects the display and broadcast effects of TV programs.

[0004] There are two main challenges in achieving the goal of intelligent directing: one is to select the appropriate switching point to ensure a smooth transition from one shot to another and maintain the logical rationality of each shot switch in the edited video; the other is to select the appropriate view from many candidate views to ensure the smoothness of the final edited video story. Under the traditional production method, the shooting and broadcast directing of the program are very arduous tasks, the staff are under great pressure, and the program production cost is high.

[0005] Therefore, using artificial intelligence technology to identify the real-time status of a symphony scene and combining it with the switching ideas of a real director to achieve automatic switching or assisted switching of symphony program broadcasts has important research significance and application value. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes an intelligent directing system for symphony concert scenes, which improves the broadcast efficiency, realizes automatic directing or assists manual directing, and reduces costs.

[0007] To achieve the above object, the present invention provides an intelligent broadcasting system for symphony concert scenes, comprising:

[0008] Audio and video stream receiving and decoding module: used to decode the original audio and video data simultaneously, and send the decoded audio and video data to the feature detection module;

[0009] Score analysis module: used to identify and analyze the symphony score data and send the identification results to the switching logic module;

[0010] Feature detection module: used to analyze the decoded audio and video data, identify various musical instruments appearing in the video frame, the scene of the current picture and the quality of the picture in real time, and provide them to the switching logic module as reference information;

[0011] Switching logic module: used to determine whether the current screen can be switched out according to the switching judgment rules;

[0012] Director visualization module: used to play the screen cut out by the switching logic module.

[0013] Preferably, the audio and video stream receiving and decoding module includes:

[0014] Decoding unit: used to decode the audio and video streams, obtain valid information, and transmit the valid information in frames;

[0015] Multi-thread processing unit: used to establish a communication channel with the feature detection module through a multi-thread method.

[0016] Preferably, obtaining the valid information includes:

[0017] The address of the stream is stored in a variable, the address information of the stream is obtained, and the request timeout is determined by setting the timeout parameter, and then the stream is asynchronously read to obtain and store the valid audio and video data in the stream, and then the audio and video data in the stream are decoded frame by frame in a loop to obtain the valid information;

[0018] The multithreading method includes:

[0019] The addresses and port numbers of all channels in the target file are read cyclically, and a decoding and data distribution thread is started for each channel, so as to establish a communication channel between each decoding unit and the feature detection module.

[0020] Preferably, the total score analysis module comprises:

[0021] The score extraction unit is used to identify the clefs in the full score through the YOLOv5 model, and use image processing technology to cut the full score to extract the independent score of each instrument group in the symphony;

[0022] The instrument identification unit is used to cut the instrument identification image before each line of the score according to the position definition of the clef in the independent score, save the instrument identification image, extract and process the instrument identification image based on the Tesseract engine, and obtain the recognition result of the instrument identification of the score;

[0023] A musical note information recognition unit is used to process the independent musical score using an end-to-end monophonic musical score optical musical score recognition model to obtain a musical note recognition result for a separate musical score;

[0024] The performance information detection unit is used to use a pre-detection model to determine the performance of the instrument in each measure of the independent score by detecting the position of the bar line and whether there is other musical note information between the bar lines, and obtain the performance status of the instrument;

[0025] Audio start detection unit: used to locate the beginning information of the music by detecting the starting point in the audio.

[0026] Preferably, the spectrum extraction unit determines the upper and lower boundaries of the spectrum corresponding to each clef through a preset strategy, and then determines the segmentation area, which specifically includes:

[0027] The boundary segmentation method for the G clef is:

[0028] y Top =y G_Top

[0029] y Bottom =y G_Bottom

[0030] In the formula, y Top The upper boundary coordinates of the spectrum to be divided, y Bottom is the lower boundary coordinate of the spectrum to be divided, y G_Top ,y G_Bottom They are the vertical axis coordinates of the upper boundary and the lower boundary of the clef identified by the G clef respectively;

[0031] The boundary segmentation method for F clef and C clef is:

[0032]

[0033] In the formula, y adjusted_Top The coordinates of the upper boundary of the segmentation after the corresponding clef is modified and adjusted according to its clef position definition, y adjusted_Bottom The coordinates of the lower boundary of the segmentation after the corresponding clef is modified and adjusted according to its clef position definition, y F_Top ,y F_Bottom They are the vertical axis coordinates of the upper boundary and the lower boundary of the clef identified for the corresponding clef respectively.

[0034] Preferably, the end-to-end processing method of the monophonic optical music score recognition model comprises:

[0035] First, the convolutional layer is used to extract features from the independent music score image. Then, the recurrent layer is used to perform sequence modeling on the extracted features based on the bidirectional long short-term memory network layer. Finally, the fully connected layer is used to classify the features of each time step to obtain the final note prediction result.

[0036] The end-to-end monophonic optical music score recognition model includes four convolutional layers, each convolutional layer is followed by a maximum pooling layer, and the loss function adopts the CTC loss function.

[0037] Preferably, the audio start detection unit locates the beginning information of the music piece based on a symphony audio start detection method based on audio energy and time domain features, specifically comprising:

[0038] Load the decoded audio frame and obtain the audio data and sampling rate;

[0039] Performing short-time Fourier transform on the audio data to obtain an amplitude spectrum, and summing the squares of the amplitude spectrum along the time axis to obtain the energy of the audio frame, and drawing an audio frame energy curve, smoothing the energy curve, and setting an adaptive energy threshold;

[0040] The starting point in the audio data is detected, the audio data and the adaptive energy threshold are used as input, a frame index corresponding to the starting point is output, and the frame index is converted into time information.

[0041] Preferably, the feature detection module comprises:

[0042] Musical instrument image detection unit: used to analyze the video stream of a symphony concert scene and identify various musical instruments appearing in the video frame in real time;

[0043] Scene recognition unit: used to detect each frame of the video stream frame by frame using the scene recognition model based on the convolutional neural network ResNet50, and output the corresponding scene label;

[0044] Blur detection unit: used to use a blur detection algorithm to determine whether the lens is in an adjustment state and filter out unusable blurry images; wherein the blur detection algorithm evaluates the blur level of the image by calculating the average amplitude value of the high-frequency components in the image and comparing it with a preset value.

[0045] Preferably, the musical instrument image detection unit is used to perform frame processing on the video stream, input each processed frame image into the improved YOLOv5 model for musical instrument detection, detect various musical instruments appearing in the video frame, and mark the position of the musical instrument in the image;

[0046] The improved YOLOv5 model adds a target detection layer to the feature fusion part of YOLOv5, and the target detection layer includes several convolution operations and a C3 module to extract and detect subtle features.

[0047] Preferably, the switching judgment rule includes:

[0048] Musical instrument priority selection rule: used to preferentially switch the screen containing the musical instrument being played in the current time period according to the recognition result of the full score analysis module and the recognition and analysis result of the musical instrument image detection unit;

[0049] Scene judgment rule: used to judge the next cut shot and the time when the current scene can be broadcast according to the current scene;

[0050] Blur judgment rule: used to judge whether the current picture can be cut out according to whether the picture is blurry.

[0051] Compared with the prior art, the present invention has the following advantages and technical effects:

[0052] (1) The present invention processes and recognizes the overall score and audio of a symphony to obtain the time period information of various musical instruments in the symphony concert. An additional detection layer is added to the basic structure of YOLOv5, and a series of convolution operations and C3 modules are used to help extract and detect more subtle features, so that the model can more effectively distinguish and detect small targets for small musical instrument recognition in panoramic images.

[0053] (2) The present invention realizes automatic directing or post-production editing of symphony concert performances by identifying characteristic information in the symphony audio and video streams and the instrument playing periods in the symphony score and performing signal selection according to preset switching logic rules. This not only reduces the directing cost and pressure on directors, but also enables more performances to be presented to a wider audience in a more convenient and efficient manner in the form of videos, while reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0055] Figure 1 This is a schematic diagram of the structure of an intelligent broadcast directing system for a symphony concert scene according to an embodiment of the present invention;

[0056] Figure 2 This is a flowchart for implementing the full spectrum analysis module of an embodiment of the present invention;

[0057] Figure 3These are some example diagrams of spectrum segmentation effects according to the embodiments of the present invention;

[0058] Figure 4 is a structural diagram of an end-to-end neural network model according to an embodiment of the present invention;

[0059] Figure 5 An example diagram of a music score section performance according to an embodiment of the present invention;

[0060] Figure 6 A YOLOv5 structure diagram with a new detection layer added according to an embodiment of the present invention;

[0061] Figure 7 is a switching logic flow chart of an embodiment of the present invention;

[0062] Figure 8 This is an example diagram of the interface of an embodiment of the present invention. DETAILED DESCRIPTION

[0063] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0064] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0065] The present invention proposes an intelligent broadcasting system for symphony concert scenes. Figure 1 ,include:

[0066] Audio and video stream receiving and decoding module: used to decode the original audio and video data simultaneously, and send the decoded audio and video data to the feature detection module;

[0067] Score analysis module: used to identify and analyze the symphony score data and send the identification results to the switching logic module;

[0068] Feature detection module: used to analyze the decoded audio and video data, identify various musical instruments appearing in the video frame, the scene of the current picture and the quality of the picture in real time, and provide them to the switching logic module as reference information;

[0069] Switching logic module: used to determine whether the current screen can be switched out according to the switching judgment rules;

[0070] Director visualization module: used to play the screen cut out by the switching logic module.

[0071] This embodiment realizes automatic directing or post-automatic editing of symphony concert performances by identifying characteristic information in the symphony audio and video streams and the instrument performance periods in the symphony score, and performing signal selection according to preset switching logic rules. This not only reduces the directing cost and the pressure on directors, but also enables more performances to be presented to a wider audience in the form of videos more conveniently and quickly, and reduces costs.

[0072] Furthermore, the audio and video stream receiving and decoding module includes:

[0073] Decoding unit: used to decode the audio and video streams, obtain valid information, and transmit the valid information in frames;

[0074] Multi-thread processing unit: used to establish a communication channel with the feature detection module through a multi-thread method.

[0075] Specifically, the audio and video stream receiving and decoding module has the functions of decoding data and sending data. After the original audio and video streams enter the system, the audio and video stream receiving and decoding module first starts the data sending thread to determine the communication port between each functional module, opens the communication function, and extracts the original data in the decoding thread, and finally sends the decoded data to the feature detection module for feature analysis through the data sending thread.

[0076] For stream input decoding, this embodiment uses the Python library function PyAV through the decoding unit to decode the video and audio stream to obtain valid information therein. The decoding process is specifically divided into the following steps:

[0077] Use the av.open() function to store the stream address in a variable, obtain the stream address information, and determine the request timeout by setting the timeout parameter. Then call the streams.get() function to obtain and store the valid audio and video data in the stream. Then use a loop to decode the audio and video data in the stream frame by frame, and transmit the decoded data in frames.

[0078] At this time, the system calls the callback function to extract the data obtained in the decoding, such as DTS (Decoding Time Stamp), PTS (Display Time Stamp), stream address (streamaddress) and sampling rate (sample rate), and saves it to a new object again.

[0079] The director system needs to handle the selective switching of multi-channel audio and video streams at the same time, so it must be able to transmit multi-channel audio and video data at the same time, and ensure that each channel is independent of each other and does not interfere with each other. This embodiment uses a multi-threaded processing unit to use a method of synchronously opening multiple threads to achieve decoding and transmission of audio and video of each channel. The detailed steps of opening multiple threads are as follows:

[0080] First, the addresses and port numbers of all channels in the JSON file are read in a loop, and the decoding and data distribution threads are opened for each channel. Then, the communication channels between each decoding unit and the feature detection module are established using the socket method in the ZMQ library and the port number in the JSON file. Finally, a new communication port is opened to realize communication with the switching logic module, and the callback function is called to transfer the label obtained by the feature detection module to the switching logic module.

[0081] Using multi-threading technology to decode and distribute audio and video streams can make the logical structure of the system clearer, and the operations of each thread are independent of each other, which is easy to analyze.

[0082] Further, the score analysis module, such as Figure 2 As shown, including:

[0083] The score extraction unit is used to identify the clefs in the full score through the YOLOv5 model, and use image processing technology to cut the full score to extract the independent score of each instrument group in the symphony;

[0084] The instrument identification unit is used to cut the instrument identification image before each line of the score according to the position definition of the clef in the independent score, save the instrument identification image, extract and process the instrument identification image based on the Tesseract engine, and obtain the recognition result of the instrument identification of the score;

[0085] A musical note information recognition unit is used to process the independent musical score using an end-to-end monophonic musical score optical musical score recognition model to obtain a musical note recognition result for a separate musical score;

[0086] The performance information detection unit is used to use a pre-detection model to determine the performance of the instrument in each measure of the independent score by detecting the position of the bar line and whether there is other musical note information between the bar lines, and obtain the performance status of the instrument;

[0087] Audio start detection unit: used to locate the beginning information of the music by detecting the starting point in the audio.

[0088] Specifically, after the full score is input into the full score analysis module, the clef recognition and positioning will be performed first to obtain the positions of each score. While cutting out the score pictures, the score note information will be recognized using OMR (Optical Music Recognition). Then, the instrument logo will be recognized using OCR. The actual start time of the symphony audio corresponding to the full score will be obtained using the audio start detection technology. Next, combined with the time information of each instrument in the full score, a summary of the instrument performance can be obtained to assist the director in switching the symphony concert. Specifically, this module can determine which instruments are being played and which are not being played within a certain period of time. For example, at T a To T b During the time period, instrument X was playing, while other instruments were not playing. This information provides reference information for the switching logic module of the symphony concert director system, allowing it to switch and guide more accurately.

[0089] Furthermore, the spectrum extraction unit determines the upper and lower boundaries of the spectrum corresponding to each clef through a preset strategy, and then determines the segmentation area, which specifically includes:

[0090] The key point of the score recognition is to obtain the characteristic information of each line of the score and identify the positioning information to determine the position of the score to be cut. This embodiment uses the YOLOv5 model to identify the clef in the full score and uses image processing technology to cut the full score into each score. This divide-and-conquer strategy effectively reduces the complexity and error rate, improves the accuracy of each score's musical note recognition, and provides a reliable basis for the subsequent extraction of instrument information.

[0091] The data set is a symphony score collected from the Internet, with a total of 100 symphony score images. Each symphony score has about 16 clef logos, that is, the collected clef data set has a total of about 1,600. This embodiment uses the labeling tool Labelimg to label the data set. During the labeling process, the principle that the clef is completely within the labeling box is followed to ensure the accuracy and completeness of the labeling. After the labeling is completed, the labeling file in the YOLO format is exported to prepare for subsequent model training.

[0092] The clef recognition model of this embodiment uses 80 symphony score images for training, where the ratio of training set to validation set is 72:8, and the test set is 20. The processor used in this embodiment is 11thGenIntel(R)Core(TM)i9-11900K@3.50GHz, the graphics card is NVIDIAGeForce RTX 3090, 64GB memory, and the operating system is Windows10, 64-bit. The python version used is python 3.7. The final clef recognition accuracy reached 89.4%.

[0093] In a five-line musical notation, the positions of the treble clef, bass clef, and alto clef are usually fixed, and they are located at different positions on the five-line musical notation to indicate the pitch. In general, their positions are defined as follows:

[0094] Treble clef (G clef): Usually located on the second line of the staff, representing the note G.

[0095] Bass clef (F clef): Usually located on the fourth line of the staff, representing the note F.

[0096] Alto clef (C clef): Usually located on the third line of the staff, representing the note C.

[0097] This embodiment uses the clef recognition model trained by YOLOv5 to locate the clef position in front of each staff, and then determines the upper and lower boundaries of each staff based on these position definitions, thereby determining the segmentation area, and then uses the original image to crop to obtain the score images of each staff.

[0098] During the segmentation process, this embodiment follows the following strategy to determine the upper and lower boundaries of the partial score corresponding to each clef:

[0099] The boundary segmentation method for the G clef is:

[0100] y Top =y G_Top

[0101] y Bottom =y G_Bottom

[0102] In the formula, y Top The upper boundary coordinates of the spectrum to be divided, y Bottom is the lower boundary coordinate of the spectrum to be divided, y G_Top ,y G_Bottom They are the vertical axis coordinates of the upper boundary and the lower boundary of the clef identified by the G clef respectively;

[0103] The boundary segmentation method for F clef and C clef is:

[0104]

[0105] In the formula, y adjusted_top The coordinates of the upper boundary of the segmentation after the corresponding clef is modified and adjusted according to its clef position definition, y adjusted_Bottom The coordinates of the lower boundary of the segmentation after the corresponding clef is modified and adjusted according to its clef position definition, y F_Top ,y F_Bottom The vertical axis coordinates of the upper boundary and the lower boundary of the clef identified by the corresponding clef are shown in the segmentation effect diagram. Figure 3 .

[0106] Furthermore, the musical instrument identification recognition unit is used to cut the musical instrument identification image before each line of the score according to the position definition of the clef in the independent music score, save the musical instrument identification image, extract and process the musical instrument identification image based on the Tesseract engine, and obtain the recognition result of the score instrument identification;

[0107] Specifically, instrument labels refer to the abbreviations or full names before each line of the score, which help performers clearly understand the allocation and performance requirements of each part. These labels can be expressed in multiple languages, including but not limited to English, French, German, Italian, Spanish, etc.

[0108] In order to realize the recognition of instrument logos, this embodiment selects Google's Tesseract OCR engine. Tesseract is an open source OCR engine developed by HP Labs and maintained by Google. It is specially used to convert images into text and supports multiple languages. Using Tesseract to recognize instrument logos can effectively convert the logos on the score into processable text data.

[0109] The image coordinate boundary to be cut after the split score instrument identification image is obtained by the previous clef recognition model, and the instrument identification image before each line of the split score is cut according to the previous clef position definition, and the instrument identification image is saved for subsequent optical character recognition. Such a process can ensure that the instrument identification in the split score is accurately extracted and identified, and will provide a reliable data basis for the next step of music analysis and processing. Before recognition, Tesseract performs a series of preprocessing steps on the input split score instrument identification image to improve the accuracy of recognition. This includes image binarization, denoising, character and text line segmentation, and tilt correction operations, which can ensure that the input image can present clear and accurate features during the recognition process.

[0110] Tesseract then segments the characters in the text line into individual characters so that each character can be recognized independently. This step involves detecting the blank areas between characters or using segmentation lines to ensure that each character can be accurately extracted and recognized. When performing feature extraction, Tesseract uses convolutional neural networks to extract the features of the characters, thereby effectively distinguishing the shapes and structures of different characters.

[0111] Finally, Tesseract combines the recognized characters to generate editable text output, which is the recognition result of the instrument identification of the score.

[0112] Furthermore, the musical note information recognition unit is used to process the independent musical score using an end-to-end monophonic musical score optical musical score recognition model to obtain a musical note recognition result for a separate musical score;

[0113] Specifically, in order to achieve accurate recognition of music scores, this embodiment trains a model specifically for music score recognition by training a neural network. The model takes a single-line music score image as input, processes the image and outputs the musical note information. This embodiment chooses to use an end-to-end monophonic music score optical music score recognition model, which combines a convolutional neural network CNN and a recurrent neural network RNN, and uses a CTC loss function during the training process. The structure of the end-to-end monophonic music score optical music score recognition model is as follows: Figure 4 shown.

[0114] The model first extracts image features through a convolutional layer, and then uses a recurrent layer to model the sequence of the extracted features. Specifically, the model includes four convolutional layers, each followed by a maximum pooling layer. After image feature extraction, the model uses two bidirectional long short-term memory (BLSTM) layers to model the sequence. The BLSTM layer can effectively capture long-term dependencies in the sequence and help to accurately predict the sequence of notes in the score. Finally, the model classifies the features of each time step through a fully connected layer to obtain the final note prediction result. In order to train the model and optimize the parameters, the stochastic gradient descent algorithm is used, and the CTC loss function is used as the loss function.

[0115] Furthermore, the performance information detection unit is used to use a pre-detection model to determine the performance of the instrument in each measure in the independent score by detecting the position of the bar line and whether there is other musical note information between the bar lines, and obtain the performance status of the instrument;

[0116] Specifically, after obtaining the musical note information of each score, the next key step is to determine whether the instrument is being played in each measure based on music theory knowledge. This embodiment adopts a note-based inference method, which considers whether the measure contains regular playing notes, such as quarter notes, eighth notes, and sixteenth notes. On the contrary, if the measure only contains measure rests, whole rests, or other rests, and there are no playing notes, then this embodiment infers that the instrument corresponding to the score corresponding to the measure is not in a playing state in the measure. Conversely, if the measure contains playing notes, it is inferred that the instrument corresponding to the score corresponding to the measure is being played. For example, if Figure 5 As shown, there is a whole rest in the first measure on the left side of the second line, so the instrument corresponding to this measure does not play during this measure. There are notes played in the following three measures, so it is inferred that the instrument corresponding to this measure is played during this measure.

[0117] When traversing the recognition results of the split score, the pre-detection model of this embodiment detects the position of the bar line and analyzes whether there is other musical note information between the bar lines to determine the performance of each bar. This process forms a judgment information list by detecting and analyzing the symbols in the bar. The list records in detail the performance status of the instrument in the corresponding score under each bar, and adds it to the data of the overall instrument performance analysis, providing an important reference for the subsequent director switching operation.

[0118] Furthermore, the audio start detection unit locates the beginning information of the music piece based on the symphony audio start detection method based on audio energy and time domain features, specifically including:

[0119] Use the Librosa library to load the decoded audio frames and obtain the audio data and sampling rate;

[0120] Using the STFT function of the Librosa library, the audio signal is short-time Fourier transformed to obtain an amplitude spectrum, and the values ​​of the amplitude spectrum are squared and summed along the time axis to obtain the energy of the audio frame, and the audio frame energy curve is drawn, the energy curve is smoothed, and an adaptive energy threshold is set;

[0121] The onset_detect function in the Librosa library is used to detect the starting point in the audio, the audio data and the adaptive energy threshold are used as input, the frame index corresponding to the starting point is output, and the frame index is converted into time information.

[0122] Specifically, this embodiment uses the Librosa library to implement audio start detection. The specific steps are as follows:

[0123] (1) Loading audio files:

[0124] First, use the Librosa library to load the audio and obtain the audio data y and sampling rate sr.

[0125] (2) Calculate the short-time Fourier transform (STFT):

[0126] Using the STFT function of the Librosa library, the audio signal is short-time Fourier transformed to obtain the amplitude spectrum. STFT can be expressed as:

[0127]

[0128] Where X[m, k] is the result of STFT, x[n] is the input signal, w[nm] is the window function, and e[n] is the window function. -j2πkn / N is the rotation factor and N is the window length.

[0129] (3) Calculate energy:

[0130] The amplitude spectrum obtained by short-time Fourier transform is squared and summed along the time axis to obtain the energy of the audio frame. The energy calculation formula is:

[0131]

[0132] Where E[n] is the energy of the nth frame.

[0133] (4) Smoothing and adaptive energy threshold:

[0134] Smoothing is performed to smooth the energy curve to reduce the peak of the energy curve, so that the calculation of the energy threshold is more stable. In order to adapt to the characteristics of different symphony audio, this embodiment sets an adaptive energy threshold, which can be expressed as:

[0135] threshold=smooth_factor×max(E smooth )Where threshold is the adaptive energy threshold, smooth_factor is a coefficient that adjusts the relationship between the smoothed energy and the threshold.

[0136] (5) Audio start detection:

[0137] Use the onset_detect function in the Librosa library to detect the onset point in the audio. This function takes the audio data and the threshold as input and determines the onset point based on an algorithm. This step can be expressed as:

[0138] onset_frames=onset_detect(y, sr, on set_envelope=E smooth , threshold, ...)

[0139] Among them, onset_frames is the frame index corresponding to the detected starting point. The frame index is then converted into time information for subsequent analysis.

[0140] onset_times=frames_to_time(onset_frames, sr)

[0141] This process provides an effective method for the present embodiment to locate the beginning of the symphony audio, and provides a reliable basis for the score analysis module to obtain information about the time interval of the instrument performance.

[0142] Furthermore, the feature detection module includes:

[0143] Musical instrument image detection unit: used to analyze the video stream of a symphony concert scene and identify various musical instruments appearing in the video frame in real time;

[0144] Scene recognition unit: used to detect each frame of the video stream frame by frame using the scene recognition model based on the convolutional neural network ResNet50, and output the corresponding scene label;

[0145] Blur detection unit: used to use a blur detection algorithm to determine whether the lens is in an adjustment state and filter out unusable blurry images; wherein the blur detection algorithm calculates the average amplitude value of the high-frequency components in the image and compares it with a preset value to evaluate the blur level of the image.

[0146] Furthermore, the musical instrument image detection unit is used to perform frame processing on the video stream, and input each processed frame image into the improved YOLOv5 model for musical instrument detection, detect various musical instruments appearing in the video frame, and mark the position of the musical instrument in the image;

[0147] Among them, the improved YOLOv5 model adds a target detection layer to the feature fusion part of YOLOv5, and the target detection layer includes several convolution operations and a C3 module to extract and detect subtle features.

[0148] Specifically, this embodiment adds a small target detection layer (detectlayer) to the feature fusion part of YOLOv5. This modification is intended to improve the model's ability to detect small targets. Specifically, the newly added target detection layer is located in the head of the model and extracts and detects features of smaller targets. The design of this layer includes a series of convolution operations (Conv) and C3 modules (C3) to improve the model's ability to extract features of small targets in dense scenes.

[0149] The newly added detection layer mainly improves the detection effect of small-target musical instruments in panoramic or large panoramic situations in the following aspects: First, a layer specifically for small target detection is added to the existing feature fusion part, so that the model can pay more attention to relevant features when processing small targets, thereby enhancing the feature extraction ability. This additional detection layer uses a series of convolution operations and C3 modules to help extract and detect more subtle features, allowing the model to more effectively distinguish and detect small targets. Secondly, in dense symphony scenes, small targets are often obscured or mixed with other objects, making them difficult to detect accurately. By adding a layer specifically for processing small targets, the model can better separate and detect these small targets, thereby improving the detection performance in dense scenes and further improving the detection effect in dense scenes. The improved YOLOv5 structure is as follows: Figure 6 shown.

[0150] Furthermore, the scene recognition unit is used to detect each frame of the video stream frame by frame using a scene recognition model based on a convolutional neural network ResNet50, and output a corresponding scene label;

[0151] Specifically, this embodiment introduces a scene recognition unit, which provides a reference for the director by identifying different scenes in the video stream, thereby realizing intelligent switching of scenes.

[0152] According to the definition of scene type and industry practices, this embodiment divides scene type into the following 6 categories:

[0153] Close-up shot size (ECS): Focus on the shoulders and above of the character;

[0154] Close-up shot (CS): includes the image from the waist up to the top of the character.

[0155] Medium shot (MS): covers the image from the knees to above the character.

[0156] Panoramic view (FS): shows the whole body of the person;

[0157] Long shot (LS): wide range, no clear subject;

[0158] Non-human shot (OS): The range is smaller than the long shot, and the subject is not a person.

[0159] This embodiment uses a scene recognition model based on the convolutional neural network ResNet50. The model detects each frame of the video stream frame by frame in a loop and outputs its corresponding scene label, thereby realizing the prediction and recognition of the scene in the symphony concert scene. The introduction of the scene recognition unit provides intelligent assistance to the director system, enabling the system to automatically adjust the scene of the output image according to the different situations and needs of the concert program to enhance the audience's viewing experience. Through the application of this module, the director team can use different scene switching techniques more flexibly to make the program picture more vivid and layered, further improving the quality and attractiveness of the concert program.

[0160] Furthermore, the blur detection unit is used to use a blur detection algorithm to determine whether the lens is in an adjustment state and filter out unusable blurry images; wherein the blur detection algorithm calculates the average amplitude value of the high-frequency components in the image and compares it with a preset value to evaluate the blur degree of the image.

[0161] Furthermore, the switching logic module is used to determine whether the current screen can be switched out according to the switching judgment rule; the switching logic flow chart is as follows: Figure 7 shown.

[0162] The switching judgment rules include:

[0163] Musical instrument priority selection rule: used to preferentially switch the screen containing the musical instrument being played in the current time period according to the recognition result of the full score analysis module and the recognition and analysis result of the musical instrument image detection unit;

[0164] Scene judgment rule: used to judge the next cut shot and the time when the current scene can be broadcast according to the current scene;

[0165] Blur judgment rule: used to judge whether the current picture can be cut out according to whether the picture is blurry.

[0166] Specifically, the instrument priority selection rule uses the preferred instrument selection function (Preferred_instruments). This function combines the results of two modules to make a judgment. One is the performance time interval information of various instruments from the score analysis module, and the other is the instrument type information currently existing in each screen from the feature detection module. Based on the common information of the two, the function will find which screen has the instrument being played in the current time interval, and determine the screen that should be cut out first.

[0167] The shot judgment rule uses a shot judgment function (Judge_shotlabel) to receive the shot of the current picture as input, and outputs a flag of whether the shot is found. The flag is the optimal choice for the next shot, that is, the shot combination of two shots will have a better visual effect. This embodiment summarizes the switching transfer probability matrix of shot matching by analyzing the director switching rules of a large number of concerts. In the shot judgment function, first, according to the current shot, the required probability rows in the switching transfer probability matrix are extracted. Then, the module that generates random numbers by probability is used to select the next switching shot. Finally, the shot of the current picture is compared with the next shot. If they are the same, the flag of the shot found is set to True; otherwise, it is set to False, and the flag is returned.

[0168] The fuzzy judgment rule uses a duration selection function (Time): This embodiment summarizes the probability matrix of playback duration under different scene levels by analyzing the director switching rules of a large number of concerts. The duration selection function receives the current scene level as input and outputs the playback duration. In this function, first, according to the current scene level, the required probability row in the playback duration probability matrix is ​​extracted. Then, the probability is used to generate a random playback duration.

[0169] These three functions together constitute the design and implementation of the switching logic, enabling the director system to intelligently switch images based on factors such as instrument performance, scene angle and duration, in order to enhance the viewing experience and attractiveness of the program.

[0170] Furthermore, the director visualization module is used to play the pictures cut out by the switching logic module.

[0171] Specifically, in order to facilitate system testing, this embodiment designs a simple user interface for the intelligent director system, which is constructed by manual writing and uses the PyQt5 framework. Different from using the QtDesigner tool to generate the interface, the UI interface code of this embodiment is written directly in the Python script, and the interface is constructed by calling the classes and methods of PyQt5.

[0172] The main function of the interface is to manage multiple RTMP stream addresses and related port information, and provides functions such as adding channels, deleting channels, and applying modifications. The entire interface layout uses QVBoxLayout and QHBoxLayout, as well as QScrollArea controls to display channel information. Users can add, delete, and modify channels through buttons. Each channel group consists of a QGroupBox, which is used to display the RTMP stream address information of the channel group. In the channel group, the QLineEdit control is used to enter and display the address, DispatchPort, and ReportPort information of each channel. At the same time, the QPushButton control is used to implement the functions of adding and deleting channels. The click event of the application modification button will trigger the processing of the channel information entered in the interface and update the relevant JSON configuration file. In addition, several subprocesses will be started to perform other tasks of the system, such as ConcertDetectWorker.py, Extracor_Concert.py, and Concert_Channel_Handler.py. The interface example is shown in the figure below Figure 8 shown.

[0173] To address the problem that the polyphonic music recognition accuracy is not suitable for application in the system, this embodiment proposes a method based on symphony score analysis. By processing and identifying the symphony score and audio, the time period information of various instruments in the symphony concert is obtained.

[0174] The present invention proposes a musical instrument image detection model suitable for symphony concert scenes. In view of the lack of musical instrument image datasets in existing symphony concert scenes, this embodiment proposes a solution for self-constructing a musical instrument image dataset, and introduces methods for marking and enhancing the dataset. At the same time, in view of the low accuracy of the YOLOv5 model in detecting small target musical instruments in panoramic and large panoramic images in symphony concert scenes, this embodiment adds an additional detection layer to the basic structure of YOLOv5. This layer uses a series of convolution operations and C3 modules to help extract and detect more subtle features, so that the model can distinguish and detect small targets more effectively.

[0175] This embodiment analyzes the director switching rules of a large number of concerts and summarizes the basic switching indicators that the director needs to rely on, including not switching when the lens is adjusted, the transition between scenes should be smooth, and the playback time length of shots of different scenes should be different.

[0176] Finally, the overall framework of the intelligent director system was designed, including modules such as decoding, score analysis, feature extraction and switching selection, to achieve automatic screen switching.

[0177] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An intelligent broadcasting system for symphony concert scenes, characterized in that: include: Audio and video stream receiving and decoding module: used to decode the original audio and video data simultaneously, and send the decoded audio and video data to the feature detection module; Score analysis module: used to identify and analyze the symphony score data and send the identification results to the switching logic module; Feature detection module: used to analyze the decoded audio and video data, identify various musical instruments appearing in the video frame, the scene of the current picture and the quality of the picture in real time, and provide them to the switching logic module as reference information; Switching logic module: used to determine whether the current screen can be switched out according to the switching judgment rules; Director visualization module: used to play the screen cut out by the switching logic module.

2. The intelligent broadcasting system for symphony concert scenes according to claim 1 is characterized in that: The audio and video stream receiving and decoding module comprises: Decoding unit: used to decode the audio and video streams, obtain valid information, and transmit the valid information in frames; Multi-thread processing unit: used to establish a communication channel with the feature detection module through a multi-thread method.

3. The intelligent broadcasting system for symphony concert scenes according to claim 2 is characterized in that: Obtaining the valid information includes: The address of the stream is stored in a variable, the address information of the stream is obtained, and the request timeout is determined by setting the timeout parameter, and then the stream is asynchronously read to obtain and store the valid audio and video data in the stream, and then the audio and video data in the stream are decoded frame by frame in a loop to obtain the valid information; The multithreading method includes: The addresses and port numbers of all channels in the target file are read cyclically, and a decoding and data distribution thread is started for each channel, so as to establish a communication channel between each decoding unit and the feature detection module.

4. The intelligent broadcasting system for symphony concert scenes according to claim 1 is characterized in that: The full score analysis module comprises: The score extraction unit is used to identify the clefs in the full score through the YOLOv5 model, and use image processing technology to cut the full score to extract the independent score of each instrument group in the symphony; The instrument identification unit is used to cut the instrument identification image before each line of the score according to the position definition of the clef in the independent score, save the instrument identification image, extract and process the instrument identification image based on the Tesseract engine, and obtain the recognition result of the instrument identification of the score; A musical note information recognition unit is used to process the independent musical score using an end-to-end monophonic musical score optical musical score recognition model to obtain a musical note recognition result for a separate musical score; The performance information detection unit is used to use a pre-detection model to determine the performance of the instrument in each measure of the independent score by detecting the position of the bar line and whether there is other musical note information between the bar lines, and obtain the performance status of the instrument; Audio start detection unit: used to locate the beginning information of the music by detecting the starting point in the audio.

5. The intelligent broadcasting system for symphony concert scenes according to claim 4 is characterized in that: The spectrum extraction unit determines the upper and lower boundaries of the spectrum corresponding to each clef through a preset strategy, and then determines the segmentation area, specifically including: The boundary segmentation method for the G clef is: and Top =and G_Top and Bottom =and G_Bottom In the formula, y Top The upper boundary coordinates of the spectrum to be divided, y Bottom The lower boundary coordinates of the spectrum to be divided, y G_Top ,y G_Bottom They are the vertical axis coordinates of the upper boundary and the lower boundary of the clef identified by the G clef respectively; The boundary segmentation method for F clef and C clef is: In the formula, y adjusted_Top The coordinates of the upper boundary of the segmentation after the corresponding clef is modified and adjusted according to its clef position definition, y adjusted_Bottom The coordinates of the lower boundary of the segmentation after the corresponding clef is modified and adjusted according to its clef position definition, y F_Top ,y F_Bottom They are the vertical axis coordinates of the upper boundary and the lower boundary of the clef identified for the corresponding clef respectively.

6. The intelligent broadcasting system for symphony concert scenes according to claim 4 is characterized in that: The processing method of the end-to-end monophonic optical music score recognition model includes: First, the convolutional layer is used to extract features from the independent music score image. Then, the recurrent layer is used to perform sequence modeling on the extracted features based on the bidirectional long short-term memory network layer. Finally, the fully connected layer is used to classify the features of each time step to obtain the final note prediction result. The end-to-end monophonic optical music score recognition model includes four convolutional layers, each convolutional layer is followed by a maximum pooling layer, and the loss function adopts the CTC loss function.

7. The intelligent broadcasting system for symphony concert scenes according to claim 4 is characterized in that: The audio start detection unit locates the beginning information of the music based on the symphony audio start detection method based on audio energy and time domain characteristics, specifically including: Load the decoded audio frame and obtain the audio data and sampling rate; Performing short-time Fourier transform on the audio data to obtain an amplitude spectrum, and summing the squares of the amplitude spectrum along the time axis to obtain the energy of the audio frame, and drawing an audio frame energy curve, smoothing the energy curve, and setting an adaptive energy threshold; The starting point in the audio data is detected, the audio data and the adaptive energy threshold are used as input, a frame index corresponding to the starting point is output, and the frame index is converted into time information.

8. The intelligent broadcasting system for symphony concert scenes according to claim 1 is characterized in that: The feature detection module comprises: Musical instrument image detection unit: used to analyze the video stream of a symphony concert scene and identify various musical instruments appearing in the video frame in real time; Scene recognition unit: used to detect each frame of the video stream frame by frame using the scene recognition model based on the convolutional neural network ResNet50, and output the corresponding scene label; Blur detection unit: used to use a blur detection algorithm to determine whether the lens is in an adjustment state and filter out unusable blurry images; wherein the blur detection algorithm evaluates the blur level of the image by calculating the average amplitude value of the high-frequency components in the image and comparing it with a preset value.

9. The intelligent broadcasting system for symphony concert scenes according to claim 8, characterized in that: The musical instrument image detection unit is used to perform frame processing on the video stream, input each processed frame image into the improved YOLOv5 model for musical instrument detection, detect various musical instruments appearing in the video frame, and mark the position of the musical instrument in the image; The improved YOLOv5 model adds a target detection layer to the feature fusion part of YOLOv5, and the target detection layer includes several convolution operations and a C3 module to extract and detect subtle features.

10. The intelligent broadcasting system for symphony concert scenes according to claim 1, characterized in that: The switching judgment rule includes: Musical instrument priority selection rule: used to preferentially switch the screen containing the musical instrument being played in the current time period according to the recognition result of the full score analysis module and the recognition and analysis result of the musical instrument image detection unit; Scene judgment rule: used to judge the next cut shot and the time when the current scene can be broadcast according to the current scene; Blur judgment rule: used to judge whether the current picture can be cut out according to whether the picture is blurry.

Images