Dance scene-oriented intelligent broadcasting director system

Abstract

The invention discloses an intelligent broadcasting director system for a dance scene, and the system comprises a pull stream decoding module which is used for obtaining video streams from a plurality of cameras and decoding the video streams; the fuzzy detection module is used for evaluating whether each frame of image in the video stream is clear or not; the dance formation change judgment module is used for analyzing the decoded video frame based on a target tracking algorithm and detecting and judging whether the formation of the dance actor changes or not; the leader dance recognition module is used for recognizing leader dance actors in the decoded video frames through a deep learning human body posture recognition algorithm; the scene switching module is used for obtaining a scene-based shot transition probability matrix and a shot playing duration matrix according to a statistical result of the dance program video data, and is used for guiding the staying duration of a shot and switching channel selection according to scene matching when no formation change exists and the leader dance does not have special actions; and the intelligent director logic switching module is used for automatically executing a video switching instruction according to the demand of the dance scene so as to realize an intelligent director effect. According to the invention, intelligent broadcasting of a dance scene can be realized, and the efficiency and quality of program production are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent directing, and in particular relates to an intelligent directing system for dance scenes. Background Art

[0002] As an important part of the evening party, dance has extremely high requirements for directors, especially in capturing and presenting the content of the picture. In the past, this task mainly relied on manual directing, but manual directing requires a high level of technology and art. It requires staff to coordinate the entire team, implement the artistic conception of the program director, and use various auxiliary means to present the program. Therefore, manual directing is difficult and consumes a lot of human resources.

[0003] With the rapid development of intelligent technology, people's lifestyles and working methods have been greatly facilitated. The combination of technology and directing art has given rise to the intelligent directing system. Intelligent directing is a video production system that uses artificial intelligence technology to achieve more intelligent, automated and unmanned TV directing switching. Intelligent directing provides a new solution for TV program directing of all sizes.

[0004] At present, the article "Smart Director Promotes Innovation of New Media Programs of the 2021 Spring Festival Gala--A Brief Analysis of the Application of Artificial Intelligence Switching Technology" describes the application of smart director in the new media programs of the 2021 Spring Festival Gala. Artificial intelligence technology realizes real-time recognition and intelligent switching, accelerates program production, and increases innovation and attractiveness. The system is mainly based on face recognition and key point detection, combined with sound information for comprehensive consideration, to achieve accurate recognition of the key states of the host, guests and other directors who guide the switching. In addition, in 2019, Tanwi Mallick studied the method of dance video analysis problem with a focus on Indian classical dance. This method combines music information and motion information to extract key postures in dance. In order to help analyze the key elements and techniques in dance works, Stancliffe proposed a new method to solve the analysis ability of dance by introducing video annotation technology. Zhang Y, Park HS proposed a semi-supervised learning framework that uses multi-view image streams to train key point detectors given limited labeled data (usually <4%). An algorithm performs cut, pan, and zoom operations by optimizing the path of a cropping window in the original video while finding salient areas to preserve, re-editing the entire movie in a single shot. For the selection of shot frames, Gschwindt M, Camci E, Bonatti R proposed a learning scheme that exploits the aesthetic features of retrospective shots by selecting the ideal shot mode based on the aesthetic value of the video shot.

[0005] In summary, judging from most of the existing research results, research related to intelligent directing is inseparable from the understanding of the content of the picture, and different methods are used to extract the semantic information of the picture. There are some shortcomings: there is currently a lack of intelligent directing systems for dance scenes, and the semantic analysis of dance scenes usually focuses on the recognition of dance movements, and lacks a strong correlation with the factors that trigger the director switching. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes an intelligent directing system for dance scenes. The directing elements of different scenes are different, and the switching of group dances focuses on dance formations and dancers' body language. It is necessary to extract semantic features that are strongly related to the director switching for dance scenes and develop an intelligent directing system for dance scenes to solve the problems existing in the above-mentioned prior art.

[0007] To achieve the above object, the present invention provides an intelligent broadcasting system for dance scenes, comprising:

[0008] The stream pulling and decoding module is used to obtain video streams from multiple cameras and decode them to obtain video frames;

[0009] Blur detection module, used to evaluate whether each frame of the video stream is clear;

[0010] A dance formation change judgment module is used to analyze the decoded video frames based on the target tracking algorithm to detect and judge whether the dancers' formation has changed;

[0011] The lead dancer recognition module is used to identify the lead dancer in the decoded video frame through the deep learning human posture recognition algorithm and mark the appearance of the lead dancer;

[0012] The scene switching module obtains the scene-based shot transfer probability matrix and the shot playback duration matrix based on the statistical results of the dance program video data. It is used to guide the shot dwell time and switch channel selection based on the scene matching when there is no formation change and the lead dancer has no special movements.

[0013] The intelligent directing logic switching module is used to automatically execute video switching instructions according to the needs of the dance scene to achieve an intelligent directing effect.

[0014] Preferably, the blur detection module adopts a Laplace transform method and a Fourier transform method.

[0015] Preferably, the stream pulling decoding module includes:

[0016] A video stream address acquisition unit, used to acquire video stream address information from different cameras;

[0017] The video frame synchronization unit is used to ensure the temporal consistency of video frames acquired from different cameras.

[0018] Preferably, the dance formation change judgment module includes:

[0019] an object detection and tracking unit for detecting dancers in video frames based on an object tracking algorithm;

[0020] The formation analysis unit is used to analyze the relative positions and movement trajectories of the dancers and to determine whether the formation has changed by setting a dynamic threshold.

[0021] Preferably, the lead dancer identification module comprises:

[0022] A human posture estimation unit, which is used to estimate the key points of the dancer's body based on a deep learning human posture recognition algorithm to obtain a human posture estimation result;

[0023] The lead dancer determination unit is used to identify the lead dancer according to the human body posture estimation result and the context information of the dance scene.

[0024] Preferably, the scene switching module includes:

[0025] The scene classification unit is used to analyze the scene of the current video frame and obtain a classification result of which category the scene belongs to: far, full, medium, near, or special;

[0026] A hidden Markov model unit based on the scene state is used to model the scene transfer and dwell time by taking the scene as the state variable, the transition relationship between scenes as the state transfer matrix, and the shot dwell time at each scene as the emission matrix;

[0027] The scene switching decision unit is used to decide whether to switch the scene according to the classification result and the hidden Markov model based on the scene.

[0028] Preferably, the intelligent broadcast directing logic switching module includes:

[0029] Switching logic design unit, used to design switching logic based on dance scene requirements;

[0030] The switching instruction execution unit is used to execute the video switching operation according to the output of the switching logic design unit.

[0031] Preferably, the intelligent directing system adopts a combination of multi-process and multi-threading.

[0032] Compared with the prior art, the present invention has the following advantages and technical effects:

[0033] The present invention proposes an intelligent directing system for dance scenes. First, in the streaming decoding module, by obtaining and decoding multi-camera video streams, the original video data required for subsequent processing is provided to ensure that the system can process signals from different cameras. Secondly, in the dance formation change judgment module, a target tracking algorithm is used to track the dancers in the video, and the dynamic threshold is set to determine whether the formation has changed, so as to perform corresponding switching when the formation changes. Then, in the lead dance recognition module, a human posture recognition algorithm based on deep learning is used to identify the lead dancer, ensuring that the dance posture can be highlighted when the lead dancer appears. Then, in the scene switching module, according to the statistical results of the dance program video data, the lens transfer probability matrix based on the scene and the lens playback time matrix are obtained to guide the lens stay time and automatic switching. Finally, in the intelligent directing logic switching module, according to the general directing idea of ​​the dance scene, the switching logic is designed and the switching instruction is generated to realize automatic intelligent directing switching and reduce the difficulty of directing.

[0034] The present invention is based on the collaborative work of multiple modules, can realize intelligent directing of dance scenes, improve the efficiency and quality of program production, reduce the consumption of human resources, and bring better viewing experience to the audience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 is a system schematic diagram of an embodiment of the present invention;

[0037] Figure 2 The following is a system switching logic flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Embodiment 1

[0041] like Figure 1 As shown, this embodiment provides an intelligent broadcasting system for dance scenes, including:

[0042] The stream pulling and decoding module is used to obtain video streams from multiple cameras and decode them to obtain video frames;

[0043] Blur detection module, used to evaluate whether each frame of the video stream is clear;

[0044] A dance formation change judgment module is used to analyze the decoded video frames based on the target tracking algorithm to detect and judge whether the dancers' formation has changed;

[0045] The lead dancer recognition module is used to identify the lead dancer in the decoded video frame through the deep learning human posture recognition algorithm and mark the appearance of the lead dancer;

[0046] The scene switching module obtains the scene-based shot transfer probability matrix and the shot playback duration matrix based on the statistical results of the dance program video data. It is used to guide the shot dwell time and switch channel selection based on the scene matching when there is no formation change and the lead dancer has no special movements.

[0047] The intelligent directing logic switching module is used to automatically execute video switching instructions according to the needs of the dance scene to achieve an intelligent directing effect.

[0048] In this embodiment, taking three cameras as an example, feature extraction is performed on the decoded data, including blur detection, judgment of dance formation changes, identification of group dance leaders, and classification of scene types. By analyzing the arrangement of cameras in the venue and combining the test materials, it can be found that the screen of camera No. 1 is mainly panoramic, the screen of camera No. 2 is long-range, and the screen of camera No. 3 is mainly panoramic. Switching judgment is performed in units of video frames. If it is judged to be blurry, the switching qualification of the signal is cancelled. If it is not blurry, the subsequent switching decision is continued. If there is a change in formation, the switching instruction is triggered and the screen is cut to a specific channel. If there is a lead dancer, the screen is cut to a specific channel. Otherwise, it is switched according to the scene type. The system switching logic flow chart is as follows: Figure 2 shown.

[0049] Furthermore, the blur detection module adopts a Laplace transform method and a Fourier transform method.

[0050] Specifically, the blur detection module uses the Laplace transform and Fourier transform methods to identify blur or distortion in the image. For systems such as intelligent broadcasting that require a lot of computing resources, when selecting algorithms, it is necessary to choose simple and fast methods as much as possible while ensuring availability to ensure the real-time performance of the entire system. Both methods are relatively simple and efficient image processing technologies, especially the fast Fourier transform (FFT) algorithm of Fourier transform, which can efficiently process the frequency domain information of the image. This blur detection method has the advantages of being simple and fast, no training required, strong interpretability, good adaptability to specific blur types, visualization, and adjustable parameters.

[0051] Furthermore, the stream pulling and decoding module includes:

[0052] A video stream address acquisition unit, used to acquire video stream address information from different cameras;

[0053] The video frame synchronization unit is used to ensure the temporal consistency of video frames acquired from different cameras.

[0054] Specifically, when building a simulation system, the front-end data transmission can be in the form of virtual machine push streaming, and the Ubuntu terminal FFmpeg command can be used to push multiple video streams to simulate real-time signals. In actual application, the signal pushed by the camera can be directly pulled. In the decoding process, a multi-threaded loop is created to read each video stream frame. After each frame of the image is obtained, the image is converted into an array in rgb24 format, and a data structure containing the image and metadata is constructed. The image data is transmitted to each feature extraction module for analysis.

[0055] Further, the dance formation change judgment module includes:

[0056] an object detection and tracking unit for detecting dancers in video frames based on an object tracking algorithm;

[0057] The formation analysis unit is used to analyze the relative positions and movement trajectories of the dancers and to determine whether the formation has changed by setting a dynamic threshold.

[0058] Specifically, the role of the target tracking module in the system is to extract features and realize the judgment of formation changes in group dance scenes. It completes tasks including target detection, target tracking, average displacement calculation, dynamic threshold update, and formation change judgment. The feature extraction algorithm is called in the form of encapsulation as a function to realize the analysis and processing of video frames and obtain the data required for switching judgment. In the function encapsulation part, the formation change judgment part is encapsulated as a function. The function input data is the video image data obtained by the decoding module. The video frame decoded and output by the system framework is passed into the target tracking algorithm. After image preprocessing, model reasoning and detection, and data analysis, the formation change judgment result is obtained. Set the interface function in the system framework, initialize the target tracking module, perform relevant configuration settings of the experimental model, extract image data from the input image, complete target detection, tracking, and formation change judgment tasks, store and transmit image data and detection data, receive and process displacement data, and define queues for storing displacement data of adjacent frames. As the video plays, the video frame image data is updated, the target position information is updated, the detection result is updated, and the judgment result is updated. Switching judgment is performed and executed according to the detection result.

[0059] Furthermore, the lead dance recognition module includes:

[0060] A human posture estimation unit, which is used to estimate the key points of the dancer's body based on a deep learning human posture recognition algorithm to obtain a human posture estimation result;

[0061] The lead dancer determination unit is used to identify the lead dancer according to the human body posture estimation result and the context information of the dance scene.

[0062] Specifically, the role of the human posture recognition module in the system is to extract human limb movement information and realize the recognition of the lead dancer in the group dance scene based on the extracted human posture information. The system call is also implemented in the form of function encapsulation to realize the analysis and processing of the video frame and obtain the data required for switching judgment. In order to facilitate the connection between the system framework and the human posture recognition module, the lead dancer recognition part is encapsulated by function. The encapsulated function completes the tasks including target detection, key point detection, motion amplitude judgment, different motion amplitude number counting, and lead dancer situation judgment. The function input data is the video image data obtained by the decoding module. The system implementation part first initializes the model in the system framework, and obtains the video frame data through the decoding process of the video stream. The image is processed and analyzed through the function interface, and image processing, model reasoning, key point information processing, motion amplitude judgment, cyclic detection of multiple targets in a single frame picture, motion amplitude number counting of multiple targets and lead dancer situation judgment are completed. As the video frame image data of the video playback is updated, the judgment result is updated, and the switching judgment is performed and executed according to the detection result.

[0063] Further, the scene switching module includes:

[0064] The scene classification unit is used to analyze the scene of the current video frame and obtain a classification result of which category the scene belongs to: far, full, medium, near, or special;

[0065] A hidden Markov model unit based on the scene state is used to model the scene transfer and dwell time by taking the scene as the state variable, the transition relationship between scenes as the state transfer matrix, and the shot dwell time at each scene as the emission matrix;

[0066] The scene switching decision unit is used to decide whether to switch the scene according to the classification result and the hidden Markov model based on the scene.

[0067] Specifically, in the production of dance programs, the reasonable combination of different scenes is crucial. Enriching and reasonably matching these scenes can not only enhance the expressiveness of the program, but also better convey the beauty and emotion of the dance. The alternating use of different scenes can form a distinct visual effect, avoid visual fatigue caused by a single scene, and help meet the diverse needs of the audience. In order to achieve the switching between different scenes, the scene classification module of the feature extraction part first classifies the video screen as the basis for switching. The switching is carried out according to the scene switching model containing the scene transfer probability matrix and the stay time emission matrix. There are six possible scene types for the current playback screen, and there are also six possible scene types for the next playback screen. The next scene type to be switched is determined by the scene transfer probability matrix under the dance scene obtained by statistics. The stay time of each scene is divided into 11 levels, and the playback time is determined according to the stay time emission probability matrix. In summary, the scene type of the next shot to be switched is selected according to the scene transfer probability, and the duration of each shot is selected according to the stay time emission matrix.

[0068] Furthermore, the intelligent broadcast directing logic switching module includes:

[0069] Switching logic design unit, used to design switching logic based on dance scene requirements;

[0070] The switching instruction execution unit is used to execute the video switching operation according to the output of the switching logic design unit.

[0071] Specifically, the intelligent director switching module determines whether switching is required according to the director switching logic based on the data detected by the feature extraction part, and performs the switching operation if the switching conditions are met. The system framework receives information including the video stream address, scene type, fuzzy mark, formation change judgment result, and lead dance recognition result, and determines whether switching is required. According to the switching model, it determines whether the switching conditions are met, including switching the broadcast screen to the distant view when the formation change is detected, and switching the screen to the panoramic screen when the lead dance is detected. If it is detected that switching is required, a switching instruction is sent. The switching implementation method is to construct a command object containing playback related information, which includes the specified RTMP address, the time code indicating the playback time point, the display timestamp (PTS) of the video frame, and the decoding timestamp (DTS) of the video frame. Then the command object is sent to the server through WebSocket. After receiving the command, the server performs the corresponding playback operation.

[0072] Furthermore, the intelligent directing system adopts a combination of multi-process and multi-threading.

[0073] In this embodiment, three processes are started according to the system's functional requirements to process three tasks in parallel, and the stream decoding, feature detection and switching judgment are performed respectively. Because program recording usually involves multi-channel signal input, the stream decoding and feature detection use a multi-threaded approach to process multi-channel video data in parallel. Taking three-channel data as an example in the system testing phase, a total of three threads are started. Data interaction between processes is carried out through ZMQ. (ZeroMQ) is an open source messaging library that uses zmq socket to achieve communication and complete data transmission tasks. The director tool first performs stream decoding on multi-camera signals to obtain image data, then extracts features from the image data, and finally makes switching decisions based on the extracted underlying features to achieve switching.

[0074] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An intelligent broadcasting system for dance scenes, characterized in that: The following steps are involved: The stream pulling and decoding module is used to obtain video streams from multiple cameras and decode them to obtain video frames; Blur detection module, used to evaluate whether each frame of the video stream is clear; A dance formation change judgment module is used to analyze the decoded video frames based on the target tracking algorithm to detect and judge whether the dancers' formation has changed; The lead dancer recognition module is used to identify the lead dancer in the decoded video frame through the deep learning human posture recognition algorithm and mark the appearance of the lead dancer; The scene switching module obtains the scene-based shot transfer probability matrix and the shot playback duration matrix based on the statistical results of the dance program video data. It is used to guide the shot dwell time and switch channel selection based on the scene matching when there is no formation change and the lead dancer has no special movements. The intelligent directing logic switching module is used to automatically execute video switching instructions according to the needs of the dance scene to achieve an intelligent directing effect.

2. The intelligent broadcasting system for dance scenes according to claim 1 is characterized in that: The blur detection module adopts Laplace transform method and Fourier transform method.

3. The intelligent broadcasting system for dance scenes according to claim 1 is characterized in that: The stream pulling decoding module includes: A video stream address acquisition unit, used to acquire video stream address information from different cameras; The video frame synchronization unit is used to ensure the temporal consistency of video frames acquired from different cameras.

4. The intelligent broadcasting system for dance scenes according to claim 1 is characterized in that: The dance formation change judgment module includes: an object detection and tracking unit for detecting dancers in video frames based on an object tracking algorithm; The formation analysis unit is used to analyze the relative positions and movement trajectories of the dancers and to determine whether the formation has changed by setting a dynamic threshold.

5. The intelligent broadcasting system for dance scenes according to claim 1 is characterized in that: The lead dance recognition module comprises: A human posture estimation unit, which is used to estimate the key points of the dancer's body based on a deep learning human posture recognition algorithm to obtain a human posture estimation result; The lead dancer determination unit is used to identify the lead dancer according to the human body posture estimation result and the context information of the dance scene.

6. The intelligent broadcasting system for dance scenes according to claim 1 is characterized in that: The scene switching module includes: The scene classification unit is used to analyze the scene of the current video frame and obtain the classification result of whether the scene belongs to the category of "far, full, medium, near, special"; A hidden Markov model unit based on the scene state is used to model the scene transfer and dwell time by taking the scene as the state variable, the transition relationship between scenes as the state transfer matrix, and the shot dwell time at each scene as the emission matrix; The scene switching decision unit is used to decide whether to switch the scene according to the classification result and the hidden Markov model based on the scene.

7. The intelligent broadcasting system for dance scenes according to claim 1 is characterized in that: The intelligent broadcast directing logic switching module includes: Switching logic design unit, used to design switching logic based on dance scene requirements; The switching instruction execution unit is used to execute the video switching operation according to the output of the switching logic design unit.

8. The intelligent broadcasting system for dance scenes according to claim 1 is characterized in that: The intelligent broadcast directing system adopts a combination of multi-process and multi-threading.

Images