Campus broadcasting system based on video dynamic fusion and fusion method

Abstract

The invention discloses a campus broadcast system based on video dynamic fusion. The campus broadcast system comprises a broadcast background server, a broadcast host, a broadcast playing terminal and a broadcast acquisition server, the broadcast background server side is connected with the broadcast host through broadcast acquisition service; the broadcast background server selects a broadcast playing terminal for broadcast playing; after the broadcast background server issues an acquisition instruction, the broadcast acquisition server acquires audio and video data, carries out video dynamic fusion, and pushes the fused audio and video data to a local server; after the broadcast background server issues a broadcast instruction, the broadcast playing terminal acquires and plays the fused audio and video data; in addition, the invention further discloses a video picture dynamic fusion method, on the basis of audio and video data coding synchronization, multiple paths of video pictures at the same moment are dynamically fused, and the fused audio and video data streams serve as final broadcast pictures.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart campus, and in particular relates to a campus broadcasting system and a fusion method based on dynamic video fusion. Background Art

[0002] Traditional campus broadcasting systems generally only provide the function of broadcasting sound. The original broadcasting acquisition service only needs to identify the audio device, collect audio data and broadcast it. With the changes in user needs and the evolution of technology, the needs of campus broadcasting systems have changed. In addition to broadcasting sound, it is also necessary to broadcast video. At the same time, it is also required to perform background processing on the video images obtained by the broadcast host, such as multi-screen synthesis, to obtain a better video broadcast image. How to collect audio and video information, synchronize audio and video, and perform local dynamic fusion based on multiple images is an urgent problem that needs to be addressed. Summary of the invention

[0003] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention provides a campus broadcasting system and a fusion method based on dynamic video fusion. Without changing the original operating habits of users, the existing broadcasting acquisition service program is improved and upgraded to identify the names of all audio and video devices on the broadcasting host; collect audio and video data, and keep the audio and video synchronized; at the same time, it supports the dynamic fusion of multiple video screens into one video screen. The campus broadcasting system realizes the function of audio and video broadcasting. Such optimization and upgrading further enriches and expands the functions of the campus broadcasting system.

[0004] Technical solution: A campus broadcasting system based on dynamic video fusion, including a broadcasting background service end, a broadcasting host, a broadcasting terminal and a broadcasting acquisition service; the broadcasting background service end is connected to the broadcasting host through the broadcasting acquisition service; the broadcasting background service end selects the broadcasting terminal for broadcasting; The broadcast background server identifies the audio and video devices on the broadcast host, and uploads the names of the audio and video devices to the broadcast background server at regular intervals; the broadcast playback terminal registers the terminal device to the broadcast background server in real time; after the broadcast background server issues a collection instruction, the broadcast collection service collects audio and video data, performs dynamic video fusion, and pushes the fused audio and video data to the local server; after the broadcast background server issues a broadcast instruction, the broadcast playback terminal obtains the fused audio and video data and plays it.

[0005] Furthermore, the broadcast background server and the broadcast collection service are permanently connected based on the websocket protocol; the broadcast playback terminal device and the broadcast background server perform two-way communication through http and websocket protocols; the broadcast playback terminal device actively requests the broadcast background server to register the terminal device to the campus broadcast system through the http protocol, and the broadcast background server sends a playback instruction with a streaming address through the websocket protocol; the broadcast terminal playback device pulls the audio and video stream according to the corresponding address and plays it.

[0006] Furthermore, the broadcast host includes several microphones for collecting audio data and cameras for collecting video data, and a system desktop; the broadcast collection service identifies the broadcast host, obtains a list of audio and video collection device names, and organizes the obtained audio and video device information according to a defined format. The broadcast collection service periodically reports the organized audio and video device information to the broadcast background server through the websocket protocol.

[0007] A fusion method of a campus broadcasting system based on video dynamic fusion, comprising the following steps: Step S1: The broadcast collection service obtains the audio and video data of the corresponding broadcast host, and uses Laplace pyramid fusion to perform video image fusion; Step S2: stream the fused image, synchronize it with the collected audio data, and push the synchronized video stream to the local RTSP server.

[0008] Furthermore, in step S1, the specific method of using Laplace pyramid fusion to perform video image fusion includes: Step S1.1, spatially align the two images to be fused, and preprocess the images, including image correction, noise reduction, and registration; Step S1.2, extracting features of the image to be fused by SURF feature extraction, extracting dominant features of different images; dynamically generating a mask based on the dominant features of the image; the mask is used to control the weights of different images in the fusion process; Step S1.3, constructing a Gaussian pyramid based on the input image, taking the original image as the basis, using Gaussian blur and downsampling layer by layer to construct a new Gaussian image layer, repeating several times to obtain a Gaussian pyramid with decreasing resolution; Step S1.4, constructing a Laplacian pyramid based on the constructed Gaussian pyramid. Subtracting the upsampled Gaussian image of the next layer from the Gaussian image of the current layer to construct the Laplacian pyramid; Step S1.5, using the mask to perform weighted averaging on the Laplacian pyramids of the two images to generate a mixed Laplacian pyramid; at each level, the degree of fusion of the two images at that level is determined according to the value of the mask; Step S1.6, starting from the top layer of the fused Laplacian pyramid, reconstruct the final fused image by upsampling layer by layer and adding the upsampling result to the next layer.

[0009] Furthermore, in step S2, the fused image is stream processed and converted into a YUV format image, the YUV format image is encoded by FFmpeg audio and video processing software, H264 format video stream data is obtained, and it is synchronized with the PCM or AAC format audio data collected by the broadcaster, and finally a dynamically fused video stream is generated and pushed to the local RTSP server.

[0010] Compared with the prior art solutions, the technical solution adopted by the present invention has the following beneficial effects: The present invention improves and upgrades the existing broadcast collection service program functions without changing the original broadcast collection service process and operation. On the basis of identifying the names of all audio and video devices on the broadcast host, the ffmpeg audio and video processing technology is used to encode the audio and video collected on the broadcast host, and keep the audio and video synchronized; at the same time, the Laplace pyramid image fusion algorithm is used to dynamically fuse multiple video images into one video image. By adding the functions of video device identification, audio and video synchronization and dynamic synthesis of multiple video images, the functions of the campus broadcast system are further expanded, making the campus broadcast system more vital and market competitive. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of the campus broadcasting system structure based on dynamic video fusion provided by the present invention; Figure 2 The working sequence diagram of the campus broadcasting system based on video dynamic fusion provided by the present invention; Figure 3 This is a schematic diagram of the video dynamic fusion method provided by the present invention. DETAILED DESCRIPTION

[0012] The present invention provides a campus broadcasting system and a fusion method based on dynamic video fusion. First, a campus broadcasting system is designed, including a broadcasting background service end, a broadcasting host and a broadcasting playback terminal. At the same time, a broadcasting acquisition service is also provided. A connection is established between the broadcasting background service end and the broadcasting host through the broadcasting acquisition service. The broadcasting background service end issues a broadcasting instruction to collect and broadcast audio and video data. In addition, the present invention also dynamically fuses the collected video images through the broadcasting acquisition service, and synchronizes them with the audio data, pushes the fused audio and video information, and broadcasts the audio and video through the broadcasting playback terminal. The campus broadcasting system and the fusion method provided by the present invention are described in detail below in conjunction with the accompanying drawings.

[0013] like Figure 1-Figure 2 As shown, the campus broadcasting system based on dynamic video fusion provided by the present invention includes a broadcasting background server, a broadcasting host, a broadcasting playback terminal and a broadcasting acquisition service. The broadcasting background server communicates with the broadcasting acquisition service through the websocket protocol, the broadcasting acquisition service acts as a client, the broadcasting background server acts as a server, and the broadcasting acquisition service maintains a websocket long connection with the broadcasting background server.

[0014] The broadcast backend server establishes a connection with the broadcast host through the broadcast collection service, and is also connected to the broadcast playback terminal. Specifically, the broadcast collection service identifies the audio and video collection devices such as microphones, speakers, USB cameras, and system desktops on the broadcast host, obtains a list of audio and video collection device names, and organizes the obtained audio and video device information according to the defined format. The broadcast collection service periodically reports the organized audio and video device information to the broadcast backend server through the websocket protocol.

[0015] The broadcasting terminal device and the broadcasting background server conduct two-way communication through http and websocket protocols. The broadcasting terminal device actively requests the broadcasting background server to register the terminal device to the campus broadcasting system through the http protocol. The broadcasting terminal device maintains websocket communication with the broadcasting background server. The broadcasting background server sends remote control instructions to the broadcasting terminal device through the websocket communication protocol.

[0016] In actual operation, the broadcast background server displays the audio and video equipment information list and broadcast playback terminal equipment list reported by the broadcast collection service in real time. The broadcast background server sends the required audio and video equipment name information to the broadcast collection service. The broadcast collection service collects the audio and video data on the corresponding broadcast host, and performs dynamic audio and video synthesis as needed before streaming. At the same time, the broadcast background server sends the broadcast instruction containing the streaming address to the broadcast terminal playback device, and the broadcast terminal playback device pulls the audio and video stream according to the corresponding address and plays it.

[0017] Based on the design of a campus broadcasting system that can broadcast audio and video data simultaneously, the present invention further proposes a video dynamic fusion method. On the basis of audio and video data encoding synchronization, the video images collected by multiple broadcast host devices at the same time are dynamically fused, and the images after the fusion of multiple video streams are used as the final broadcast images. The content of the video dynamic fusion method described in the present invention is given below in combination with specific embodiments.

[0018] Step S1: The broadcast collection service obtains the audio and video data of the corresponding broadcast host, and uses Laplace pyramid fusion to perform video fusion on the video screen.

[0019] In order to ensure the video details of the video broadcast in campus radio, operating and processing image data at the image pixel level can keep the source image with more original features, richer details and more accurate target spatial position than other fusion levels. Specifically, Step S1.1: spatially align the two images to be fused and preprocess the images, including image correction, noise reduction, and registration.

[0020] Step S1.2: extract features of the image to be fused by SURF feature extraction to extract dominant features of different images. Dynamically generate a mask based on the dominant features of the image. The mask is used to control the weights of different images in the fusion process.

[0021] Step S1.3: construct a Gaussian pyramid based on the input image. Based on the original image, Gaussian blur and downsampling are used layer by layer to construct a new Gaussian image layer. Repeat this process several times to obtain a Gaussian pyramid with decreasing resolution.

[0022] Step S1.4: construct a Laplacian pyramid based on the constructed Gaussian pyramid. Subtract the upsampled Gaussian image of the next layer from the Gaussian image of the current layer to construct a Laplacian pyramid, and each layer of the Laplacian pyramid contains the high-frequency details of the image.

[0023] Step S1.5: Use the mask to perform weighted averaging on the Laplacian pyramids of the two images to generate a mixed Laplacian pyramid. At each level, the degree of fusion of the two images at that level is determined according to the value of the mask.

[0024] Step S1.6, starting from the top layer of the fused Laplacian pyramid, reconstruct the final fused image by upsampling layer by layer and adding the upsampling result to the next layer.

[0025] Step S2: stream the fused image, synchronize it with the collected audio data, and push the synchronized video stream to the local RTSP server.

[0026] like Figure 3 As shown, the fused image is stream processed and converted into a YUV format image. The YUV format image is encoded through FFmpeg audio and video processing software to obtain H264 format video stream data, and synchronized with the PCM or AAC format audio data collected by the broadcaster. Finally, a dynamically fused video stream is generated and pushed to the local RTSP server for subsequent streaming.

[0027] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A campus broadcasting system based on dynamic video fusion, characterized in that: It includes a broadcast background service end, a broadcast host, a broadcast playing terminal and a broadcast collection service; the broadcast background service end is connected to the broadcast host through the broadcast collection service; the broadcast background service end selects the broadcast playing terminal to perform broadcasting; The broadcast backend server identifies the audio and video devices on the broadcast host, and regularly uploads the names of the audio and video devices to the broadcast backend server; the broadcast playback terminal registers the terminal device to the broadcast backend server in real time; after the broadcast backend server issues a collection instruction, the broadcast collection service collects audio and video data, performs video dynamic fusion, and pushes the fused audio and video data to the local server; After the broadcast background server sends the broadcast instruction, the broadcast playback terminal obtains the fused audio and video data and plays it.

2. According to claim 1, a campus broadcasting system based on dynamic video fusion is characterized in that: The broadcast backend server and the broadcast collection service have a long connection based on the websocket protocol; the broadcast playback terminal device and the broadcast backend server have two-way communication through http and websocket protocols; The broadcasting terminal device actively requests the broadcasting background server to register the terminal device to the campus broadcasting system through the http protocol. The broadcasting background server sends a playback instruction with a streaming address through the websocket protocol. The broadcast terminal playback device pulls the audio and video stream according to the corresponding address and plays it.

3. According to the campus broadcasting system based on dynamic video fusion according to claim 1, it is characterized in that: The broadcast host includes several microphones for collecting audio data, cameras for collecting video data, and a system desktop; the broadcast collection service identifies the broadcast host, obtains a list of audio and video collection device names, and organizes the obtained audio and video device information according to a defined format. The broadcast collection service regularly reports the organized audio and video device information to the broadcast background server through the websocket protocol.

4. The fusion method of the campus broadcasting system based on dynamic video fusion according to any one of claims 1 to 3, characterized in that: The following steps are involved: Step S1: The broadcast collection service obtains the audio and video data of the corresponding broadcast host, and uses Laplace pyramid fusion to perform video image fusion; Step S2: stream the fused image, synchronize it with the collected audio data, and push the synchronized video stream to the local RTSP server.

5. The fusion method of campus broadcasting system based on video dynamic fusion according to claim 4 is characterized in that: In step S1, the specific method of using Laplace pyramid fusion to perform video image fusion includes: Step S1.1, spatially align the two images to be fused, and preprocess the images, including image correction, noise reduction, and registration; Step S1.2, extracting features of the image to be fused by SURF feature extraction, extracting dominant features of different images; dynamically generating a mask based on the dominant features of the image; the mask is used to control the weights of different images in the fusion process; Step S1.3, constructing a Gaussian pyramid based on the input image, taking the original image as the basis, using Gaussian blur and downsampling layer by layer to construct a new Gaussian image layer, repeating several times to obtain a Gaussian pyramid with decreasing resolution; Step S1.4, constructing a Laplacian pyramid based on the constructed Gaussian pyramid. Subtracting the upsampled Gaussian image of the next layer from the Gaussian image of the current layer to construct the Laplacian pyramid; Step S1.5, using the mask to perform weighted averaging on the Laplacian pyramids of the two images to generate a mixed Laplacian pyramid; at each level, the degree of fusion of the two images at that level is determined according to the value of the mask; Step S1.6, starting from the top layer of the fused Laplacian pyramid, reconstruct the final fused image by upsampling layer by layer and adding the upsampling result to the next layer.

6. The fusion method of campus broadcasting system based on video dynamic fusion according to claim 4 is characterized in that: In step S2, the fused image is stream processed and converted into a YUV format image, the YUV format image is encoded by FFmpeg audio and video processing software, H264 format video stream data is obtained, and it is synchronized with the PCM or AAC format audio data collected by the broadcaster, and finally a dynamically fused video stream is generated and pushed to the local RTSP server.

Images