Mining audio and video throttling system

Abstract

The invention discloses a mining audio and video throttling system which comprises a throttling detection device and an audio and video protocol processing terminal. The throttling detection device comprises an audio and video data acquisition module, an audio and video data detection module and an audio and video data control transmission module; the audio and video data acquisition module acquires audio and video data; the audio and video data detection module is used for deeply analyzing and processing audio and video data acquired by the audio and video detection module, and determining a strategy for transmitting complete audio and video streams according to a set threshold value; the control transmission module is assembled into a data frame format according to a protocol; the audio and video protocol processing terminal comprises an audio and video receiving module, a data recombination module and a data distribution module. According to the invention, through a flexible real-time data processing and transmission mode, bandwidth and resources are saved, newest data are guaranteed, system integration and expansibility are ensured, and the performance and efficiency of an audio and video data processing system can be effectively improved.

Description

Technical Field

[0001] The invention belongs to the field of coal mine audio and video safety monitoring, and in particular relates to an innovative coal mine audio and video safety monitoring system. Background Art

[0002] In the field of coal mine safety monitoring, audio and video monitoring undoubtedly plays a vital role. It is one of the important measures to ensure the safe operation of coal mines. The current application of audio and video in mines is no longer limited to simple video monitoring and storage functions. With the continuous improvement of the intelligent level of coal mines, a large number of AI visual technologies, audio analysis, etc. are widely used in the field of coal mines, which greatly improves the level of safety monitoring and control, and provides a more solid guarantee for the safe production of coal mines.

[0003] At present, audio and video monitoring in coal mines is indeed facing challenges such as multiple monitoring points, wide application, and large volume, which requires high speed and large bandwidth to transmit large amounts of video data. However, the special environment of coal mines, such as deep burial and long communication links, makes it particularly difficult to build a communication backbone network with large bandwidth and high throughput, and the construction and maintenance difficulty and cost are extremely high. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes an innovative audio and video throttling system for mining, which deploys throttling detection devices at each edge of each coal mine monitoring field, and uses the built-in real-time stream detection module of the device to detect the difference between video frames and the difference between front and back audio in real time. When the difference between video frames or the difference between front and back audio is higher than the set threshold, the throttling device will control the complete transmission of the current video stream. When the difference is lower than the set threshold, the throttling device only sends the agreed heartbeat protocol to maintain the activity of the connection and block the complete transmission of the current video stream, thereby greatly reducing the bandwidth occupancy of the backbone network. This system effectively solves the practical problems faced by coal mine video monitoring, such as the high-speed and large-bandwidth transmission requirements, the high cost of backbone network construction and maintenance, and the difficulty of construction. The application of this system will help improve the efficiency and safety of coal mine monitoring and provide strong support for the safe production and intelligent development of coal mines.

[0005] The purpose of the present invention is to provide an innovative audio and video throttling system in a special and complex environment in the field of coal mine video monitoring by integrating and improving existing technologies, which includes a throttling detection device and an audio and video protocol processing terminal; the throttling detection device includes an audio and video data acquisition module, an audio and video data detection module, and an audio and video data control transmission module;

[0006] The audio and video data acquisition module is connected to the monitoring device to obtain audio and video data in the environment from the monitoring device;

[0007] The audio and video data detection module performs in-depth analysis and processing on the audio and video data collected by the audio and video detection module, identifies the differences in the audio and video data, and determines the strategy for transmitting the complete audio and video stream according to the set threshold; the specific method is:

[0008] A. The audio and video data acquisition module collects and parses the video stream or audio stream address, uses the hash algorithm to calculate the address string, and generates a unique calibration value ID for the video stream or audio stream; SHA-256 is used to calculate the calibration value:

[0009] ID = SHA-256 (Url)

[0010] Url represents the address of this video stream or audio stream, and SHA-256 represents the hash function;

[0011] At the same time, the unique calibration value UID of each frame of audio or video data is calculated through hash operation:

[0012] UID=SHA-256({b1,b1,b3,…,b n})

[0013] {b1,b1,b3,…,b n} represents a byte stream of audio or video data per frame.

[0014] The above calculation results are saved in the format of "ID+UID+audio and video data", and then processed in step B;

[0015] B. The audio and video monitoring module receives the "ID+UID+audio and video data" in step A and performs processing in step C;

[0016] C. Compare the differences of adjacent frames of video data and audio data respectively. If the difference is greater than a preset threshold, the complete data is transmitted, otherwise the heartbeat protocol is transmitted;

[0017] The control transmission module selects control transmission with different strategies according to the output result of the audio and video data detection module, and assembles it into a data frame format according to the protocol;

[0018] The audio and video protocol processing terminal includes an audio and video receiving module, a data reorganization module and a data distribution module. The audio and video data transmitted by the throttling device is received by the audio and video receiving module. After receiving the data, the data reorganization module is used to assemble the data into complete audio and video data. The audio and video data is distributed to the terminal for use by the data distribution module.

[0019] The beneficial effects of the present invention are:

[0020] 1. Saving bandwidth and resources: By adopting a throttling strategy, the present invention can select complete transmission or heartbeat protocol according to the actual changes in audio and video, saving bandwidth and resources while ensuring data integrity. For example, when the video data of the previous and next frames or the previous and next audio samples exceed the threshold, complete transmission is performed to ensure data integrity; when the threshold is not exceeded, the heartbeat protocol is selected to save bandwidth and resources.

[0021] 2. System integration and scalability: The present invention integrates the data distribution module into the system, provides UDP and TCP methods for other systems to call, and saves the unique device identifier and audio and video data. This integrated design makes the system have good scalability and compatibility, and can be easily integrated and interacted with other systems to achieve more functions and application scenarios.

[0022] In summary, the present invention can effectively improve the performance and efficiency of the audio and video data processing system through the advantages of flexible real-time data processing and transmission, saving bandwidth and resources, ensuring the latest data, and system integration and scalability.

[0023] More importantly, in the special field of coal mines, audio and video monitoring covers a wide range and has many underground subsystems. However, traditional audio and video data transmission methods require high-speed, high-capacity bandwidth, and coal mines are deeply buried and the environment is harsh, resulting in extremely high costs and difficulties in building high-speed, high-capacity networks. Through the deployment of the throttling system of the present invention, the bandwidth required for audio and video data transmission can be significantly reduced, thereby having extremely high practical value in harsh environments such as coal mines. The advantage of this system is that it can flexibly select transmission strategies according to demand, save bandwidth and resources, ensure the timeliness of data, and integrate multiple subsystems and provide scalability, so that the construction and maintenance costs of the coal mine audio and video monitoring system are greatly reduced, while improving the performance and efficiency of the system. Therefore, the present invention has extremely high practical value in the field of coal mines and can provide reliable technical support for coal mine safety monitoring and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a structural schematic diagram of a mining audio and video throttling system of the present invention;

[0025] Figure 2 It is a structural schematic diagram of the throttling detection device of the present invention;

[0026] Figure 3 It is a schematic diagram of the deployment of the throttling device of the present invention;

[0027] Figure 4 The schematic diagram of the structure of the audio and video protocol processing terminal of the present invention. DETAILED DESCRIPTION

[0028] ID、UID explanation:

[0029] ID: unique identification value of each audio or video stream;

[0030] UID: The unique identification value of each frame of audio or video data.

[0031] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown, a mining audio and video throttling system of the present invention includes a throttling detection device and an audio and video protocol processing terminal; the throttling detection device includes an audio and video data acquisition module, an audio and video data detection module, and an audio and video data control transmission module, such as Figure 2 as shown; Figure 3 As shown, the throttling device is deployed to each subsystem underground or above the well and the edge of the communication backbone network.

[0033] The audio and video data acquisition module is connected to the monitoring equipment to obtain audio and video data in the environment from each monitoring device; in complex environments such as coal mines, monitoring equipment usually includes monitoring cameras and audio acquisition equipment for real-time monitoring of the mining area. The audio and video data acquisition module is responsible for communicating with these devices and receiving audio and video data from them, supporting the access of multiple audio and video protocols. The audio and video data acquisition module plays a role in the mining video throttling system by connecting to the monitoring equipment to obtain audio and video data of the mining area in real time. The audio and video acquisition module accesses the audio and video of each subsystem according to the protocol. The throttling device supports access to the following protocols including but not limited to RTSP, HLS, MPEG-DASH, RTMP, SIP, etc. through UDP and TCP, and connects the audio and video data to the throttling device through Ethernet. The UDP or TCP protocol is used to obtain the audio and video device data of the throttling device on demand, and real-time data processing and transmission are performed according to the transmission strategy. Compared with the traditional fixed transmission method, data can be flexibly acquired and processed according to demand, which improves the real-time and flexibility of the system.

[0034] The audio and video data detection module deeply analyzes and processes the audio and video data collected by the audio and video detection module, identifies the differences in the audio and video data, determines the audio or video data from the header information of the data stream, and determines the strategy for transmitting the complete audio and video stream according to the set threshold, thereby achieving the purpose of throttling; the specific processing method is:

[0035] A. The audio and video data acquisition module collects and parses the video stream or audio stream address, uses the hash algorithm to calculate the address string, and generates a unique calibration value ID for the video stream or audio stream; SHA-256 is used to calculate the calibration value:

[0036] ID = SHA-256 (Url)

[0037] Url represents the address of this video stream or audio stream, and SHA-256 represents the hash function;

[0038] At the same time, the unique calibration value UID of each frame of audio or video data is calculated through hash operation:

[0039] UID=SHA-256({b1,b1,b3,…,b n})

[0040] {b1,b1,b3,…,b n} represents a byte stream of audio or video data per frame.

[0041] The above calculation results are saved in the format of "ID+UID+audio and video data", and then processed in step B.

[0042] B. The audio and video monitoring module receives the "ID+UID+audio and video data" in step A and performs processing in step C;

[0043] C. Compare the differences of adjacent frames of video data and audio data respectively. If the difference is greater than a preset threshold, the complete data is transmitted, otherwise the heartbeat protocol is transmitted;

[0044] For video data, the difference comparison of adjacent frames includes the following two methods:

[0045] C1-1. Take two consecutive video frame images, I1 and I2, and convert them into grayscale images:

[0046] I 1gray =0.2989×I1[:,:,0]+0.5870×I1[:,:,1]+0.1140×I1[:,:,2]

[0047] I 2gray =0.2989×I2[:,:,0]+0.5870×I2[:,:,1]+0.1140×I2[:,:,2]

[0048] Among them, 0.2989 is the weighting coefficient of the red channel, 0.5870 is the weighting coefficient of the green channel, and 0.1140 is the weighting coefficient of the blue channel; I1[:,:,0], I2[:,:,0] represent the selection of the red channel data of all rows and columns (the red brightness values ​​of all pixels), I1[:,:,1], I2[:,:,1] represent the selection of the green channel data of all rows and columns (the green brightness values ​​of all pixels), I1[:,:,2], I2[:,:,2] represent the selection of the blue channel data of all rows and columns (the blue brightness values ​​of all pixels);

[0049] C1-2. Calculate the difference image between two grayscale images by subtracting the grayscale values ​​of corresponding pixels:

[0050] Difference(x, y) = |I 1gray (x, y)-I 2gray (x, y)|

[0051] |.| represents the absolute value, x and y represent the pixel coordinates in the image, Difference(x, y) represents the grayscale difference between the pixel points (x, y) in the two grayscale images, I 1gray (x, y), I 2gray (x, y) represent the grayscale values ​​of (x, y) in the two images respectively;

[0052] C1-3. Threshold the difference image to filter out smaller differences; thresholding is done using the following formula:

[0053]

[0054] Threshold is the set threshold;

[0055] C1-4. Evaluate the degree of difference between adjacent frames by calculating the proportion of non-zero pixels in the difference image:

[0056]

[0057] Among them, Total PiXels is the total number of pixels in the image;

[0058] C1-5, Difference Value Compared with the preset threshold, if it is greater than the threshold, it is determined that the video content has changed significantly and the complete video needs to be transmitted. The data to be transmitted is "ID+UID+complete video"; if it is less than or equal to the threshold, the heartbeat protocol data needs to be transmitted, and the data to be transmitted is "ID+UID+heartbeat protocol";

[0059] C1-6, the data to be transmitted is transmitted to the data control transmission module; when transmitting data to the data control transmission module for the first time, the data to be transmitted selects "ID+UID+complete video data" to ensure the data integrity of the subsequent audio and video data processing system. When transmitting data to the data control transmission module subsequently, the selection is made according to the judgment result of step C1-5.

[0060] In the coal mine production process, due to the complex and harsh environment, the pixel differences between adjacent frames are only suitable for relatively simple changes (such as brightness, color, etc.), and cannot capture complex content differences (such as movement, background changes). Combined with the actual production data of coal mines, in a complex coal mine production environment, the deep learning detection module can be used to perform difference analysis on two video frame images. The deep learning detection module can more accurately extract inter-frame features and perform difference analysis, which can not only meet the actual safety production needs, but also reduce the consumption of video bandwidth. The steps are as follows:

[0061] C2-1. Image preprocessing and feature extraction: Take two consecutive video frame images, I1 and I2, and input them into the deep learning model to extract features, which are expressed as:

[0062] F1=Model(I1), F2=Model(I2)

[0063] Among them, F1 and F2 are high-dimensional feature vectors output by the deep learning model, which can describe the semantic information of the frame (such as scene, action or target).

[0064] C2-2. Feature difference calculation: Use Euclidean distance to measure the difference between the feature vectors of two frames, expressed as:

[0065] ΔF=||F1-F2||2

[0066] Among them, || ||2 represents the Euclidean norm of the vector, which is used to measure the similarity in the feature space;

[0067] C2-3, difference threshold judgment: set the threshold δ to distinguish the significance of the difference between frames; if ΔF>δ, it is considered that there is a significant difference between the two frames, and it is determined that the video content has changed significantly, and the complete video needs to be transmitted, and the data to be transmitted is "ID+UID+complete video"; if it is less than or equal to the threshold, the heartbeat protocol data needs to be transmitted, and the data to be transmitted is "ID+UID+heartbeat protocol";

[0068] C2-4, transfer the data to be transmitted to the data control transmission module; when transmitting data to the data control transmission module for the first time, the data to be transmitted selects "ID+UID+complete video data" to ensure the data integrity of the subsequent audio and video data processing system. When transmitting data to the data control transmission module subsequently, the selection is made according to the judgment result of step C2-3.

[0069] The above two methods can be selected according to the actual needs of coal mine production. This transmission method can reduce bandwidth occupancy, optimize resource utilization and improve system transmission efficiency.

[0070] For audio data, firstly, the audio changes are detected in real time by comparing the time domain waveform differences between the previous and next audio frame samples. If the changes are large, the complete audio data is transmitted, otherwise the heartbeat protocol is transmitted. The audio features are extracted by using the built-in deep learning model, and the differences between the previous and next audio samples are automatically trained and learned by the model, and the differences between adjacent frames are automatically calculated. The difference comparison of adjacent frames includes the following steps:

[0071] C3-1. Represent two adjacent front and back sampled audio frame signals in vector form:

[0072] a=a1,a2,a3,...,a n

[0073] b=b1,b2,b3,...,b n

[0074] The above a1, a2, a3, ..., a n Represents the audio amplitude values ​​of the first sampled audio frame, b1, b2, b3, ..., b n Represents the audio amplitude values ​​of the second sampled audio frame, n is the dimension of the vector;

[0075] C3-2. Calculate the Euclidean distance between two sampled audio frame signals:

[0076]

[0077] If the Euclidean distance exceeds the preset threshold, the complete audio data is transmitted, and the data to be transmitted is "ID+UID+complete audio"; otherwise, the heartbeat protocol data is transmitted, and the data to be transmitted is "ID+UID+heartbeat protocol";

[0078] C3-3, transfer the data to be transmitted to the data control transmission module; when transmitting data to the data control transmission module for the first time, the data to be transmitted selects "ID+UID+complete audio data" to ensure the data integrity of the subsequent audio and video data processing system. When transmitting data to the data control transmission module subsequently, the selection is made according to the judgment result of step C3-2.

[0079] The data control transmission module receives the data fed back by the audio and video data receiving module in the audio and video data processing system according to "ID+UID", and uses the dictionary method to obtain the above results, with the key being ID and the value being an unordered list composed of UIDs. If the above data detection module transmits "ID+UID+complete video / audio", but the dictionary method is used to query that this "ID" has already transmitted the data of "UID", then "ID+UID+heartbeat protocol" is transmitted.

[0080] The control transmission module is assembled into the following data frame format according to the protocol: "protocol frame header transmission strategy transmission content protocol frame tail";

[0081] (1) Protocol frame header: a custom part used to identify the starting position of the data frame to facilitate protocol parsing;

[0082] (2) Transmission strategy: A Boolean value used to indicate the transmission strategy of the data frame:

[0083] 0 means the heartbeat protocol, which transmits less data and is only used to maintain the connection or status update;

[0084] 1 indicates a complete transmission strategy, which means transmitting complete audio and video data;

[0085] (3) Transmission content, i.e., the data to be transmitted transmitted by the audio and video data detection module; under the heartbeat protocol strategy, the content is "ID+UID+heartbeat protocol"; under the complete transmission strategy, the content includes "ID+UID+complete video / audio";

[0086] (4) Protocol frame tail: A custom part used to identify the end position of the data frame and help the receiving end complete frame parsing.

[0087] The throttling detection device uses a dictionary to save the latest complete audio and video data of each device. In the dictionary, the key is the ID; the new value is the data frame, that is, the latest audio and video data collected by the device; the value is the data assembled according to the protocol ("protocol frame header transmission strategy transmission content protocol frame tail").

[0088] The latest complete data of each device is saved to prevent the data from being unable to be parsed when the data processing system first establishes a connection with the camera.

[0089] The audio and video protocol processing terminal includes an audio and video receiving module, a data reorganization module and a data distribution module. Figure 4 The audio and video data transmitted by the throttling device is received through the audio and video receiving module, and after receiving the data, the data is assembled into complete audio and video data through the data reassembly module to ensure the integrity and availability of the data during the transmission process; the audio and video data is distributed to the terminal for use through the data distribution module.

[0090] Receiving module: receiving data from the control transmission module of the throttling transmission device, and transmitting the data to the data reorganization module at the same time. If the received data is "ID+UID+complete video / audio", after successful reception, the "ID+UID" will be fed back to the data control transmission module of the throttling device;

[0091] Data reorganization module: receives the data from the above receiving module and performs the following processing according to the received data:

[0092] a. If the data transmitted is complete, the module will save the audio or video data for a period of time in a dictionary manner.

[0093] key: ID (ID in A above)

[0094] value: queue. Each data item in the queue is: UID+complete video / audio data+data timestamp.

[0095] b. If the transmission is the heartbeat protocol, the complete video / audio data closest to the current timestamp in the ID queue in a above is taken out and reassembled into the complete data of the ID.

[0096] The data reorganization module completes the reorganization of audio and video data, and transmits the latest complete video / audio of each audio and video to the data distribution module in real time.

[0097] Data distribution module: responsible for receiving processed and reorganized audio and video data and transmitting it to the designated target terminal or system to meet the data acquisition needs of other systems.

[0098] The audio and video data processing terminal uses the UDP or TCP protocol to obtain the data of the audio and video equipment from the throttling device through the specified IP address and port. When the transmission starts, the throttling device will pass the complete data of the device. Once the connection is established, the throttling device will send the data according to the specified throttling strategy. The processing system will parse according to the protocol. If the transmission strategy is complete transmission, it will be submitted to the data distribution module for data distribution, and the latest audio and video data will be saved for subsequent use; if the transmission strategy is the heartbeat protocol, the data reorganization module will reorganize it into complete audio and video data according to step b, and provide it to the data distribution module for other systems to call it in the original way. The data distribution module provides UDP and TCP methods for other systems to call, and internally saves the unique identifier of the device and the corresponding audio and video data to facilitate subsequent data access and use.

[0099] The present invention can also selectively add a video display system, which includes an audio and video data receiving module and a display module. The audio and video data receiving module receives the audio and video data from the audio and video protocol processing system data distribution module, decodes and processes the received video data, and displays it through the display module. Users can conveniently monitor the coal mine working scene, obtain relevant information in a timely manner, and ensure work safety and production efficiency. The video display system calls the data distributed by the audio and video data processing system through UDP or TCP, and displays the video on the interface.

[0100] This system is deployed at the edge of each subsystem in the coal mine monitoring field, and the real-time stream detection module is used to analyze the difference between video and audio data, and decide whether to transmit the complete video stream based on the size of the difference. When the monitoring scene is stable, the transmission of duplicate data in the backbone network is reduced, thereby reducing the cost of coal mine network construction and maintenance.

[0101] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A mining audio and video throttling system, characterized in that: It includes a throttling detection device and an audio and video protocol processing terminal; the throttling detection device includes an audio and video data acquisition module, an audio and video data detection module, and an audio and video data control transmission module; The audio and video data acquisition module is connected to the monitoring device to obtain audio and video data in the environment from the monitoring device; The audio and video data detection module performs in-depth analysis and processing on the audio and video data collected by the audio and video detection module, identifies the differences in the audio and video data, and determines the strategy for transmitting the complete audio and video stream according to the set threshold; the specific method is: A. The audio and video data acquisition module collects and parses the video stream or audio stream address, uses the hash algorithm to calculate the address string, and generates a unique calibration value ID for the video stream or audio stream; SHA-256 is used to calculate the calibration value: ID = SHA-256 (Url) Url represents the address of this video stream or audio stream, and SHA-256 represents the hash function; At the same time, the unique calibration value UID of each frame of audio or video data is calculated through hash operation: UID=SHA-256({b1,b1,b3,…,b n }) {b1, b1, b3, …, b n } represents a byte stream of audio or video data per frame. The above calculation results are saved in the format of "ID+UID+audio and video data", and then processed in step B; B. The audio and video monitoring module receives the "ID+UID+audio and video data" in step A and performs processing in step C; C. Compare the differences of adjacent frames of video data and audio data respectively. If the difference is greater than a preset threshold, the complete data is transmitted, otherwise the heartbeat protocol is transmitted; The control transmission module selects control transmission with different strategies according to the output result of the audio and video data detection module, and assembles it into a data frame format according to the protocol; The audio and video protocol processing terminal includes an audio and video receiving module, a data reorganization module and a data distribution module. The audio and video data transmitted by the throttling device is received by the audio and video receiving module. After receiving the data, the data reorganization module is used to assemble the data into complete audio and video data. The audio and video data is distributed to the terminal for use by the data distribution module.

2. A mining audio and video throttling system according to claim 1, characterized in that: For video data, the difference comparison between adjacent frames includes: C1-1. Take two consecutive video frame images, I1 and I2, and convert them into grayscale images: I 1gray =0.2989×I1[:,:,0]+0.5870×I1[:,:,1]+0.1140×I1[:,:,2] I 2gray =0.2989×I2[:,:,0]+0.5870×I2[:,:,1]+0.1140×I2[:,:,2] Among them, I1[:,:,0] and I2[:,:,0] represent the selection of red channel data of all rows and columns, I1[:,:,1] and I2[:,:,1] represent the selection of green channel data of all rows and columns, and I1[:,:,2] and I2[:,:,2] represent the selection of blue channel data of all rows and columns; C1-2. Calculate the difference image between two grayscale images by subtracting the grayscale values ​​of corresponding pixels: Difference(x,y)=|I 1gray (x,y)-I 2gray (x,y)| |.| represents the absolute value, x and y represent the pixel coordinates in the image, Difference(x, y) represents the grayscale difference between the pixel points (x, y) in the two grayscale images, I 1gray (x, y), I 2gray (x, y) represent the grayscale values ​​of (x, y) in the two images respectively; C1-3. Threshold the difference image to filter out smaller differences; thresholding is done using the following formula: Threshold is the set threshold; C1-4. Evaluate the degree of difference between adjacent frames by calculating the proportion of non-zero pixels in the difference image: Among them, Total Pixels is the total number of pixels in the image; C1-5, Difference Value Compared with the preset threshold, if it is greater than the threshold, it is determined that the video content has changed significantly and the complete video needs to be transmitted. The data to be transmitted is "ID+UID+complete video"; if it is less than or equal to the threshold, the heartbeat protocol data needs to be transmitted, and the data to be transmitted is "ID+UID+heartbeat protocol"; C1-6, transfer the data to be transmitted to the data control transmission module; when transmitting data to the data control transmission module for the first time, the data to be transmitted selects "ID+UID+complete video data"; when transmitting data to the data control transmission module subsequently, the selection is made according to the judgment result of step C1-5.

3. A mining audio and video throttling system according to claim 1, characterized in that: For video data, the difference comparison between adjacent frames includes: C2-1. Image preprocessing and feature extraction: Take two consecutive video frame images, I1 and I2, and input them into the deep learning model to extract features, which are expressed as: F1=Model(I1), F2=Model(I2) Among them, F1 and F2 are high-dimensional feature vectors output by the deep learning model, which can describe the semantic information of the frame (such as scene, action or target). C2-2. Feature difference calculation: Use Euclidean distance to measure the difference between the feature vectors of two frames, expressed as: ΔF=||F1-F2||2 Among them, ||||2 represents the Euclidean norm of the vector, which is used to measure the similarity in the feature space; C2-3, difference threshold judgment: set the threshold δ to distinguish the significance of the difference between frames; if ΔF>δ, it is considered that there is a significant difference between the two frames, and it is determined that the video content has changed significantly, and the complete video needs to be transmitted, and the data to be transmitted is "ID+UID+complete video"; if it is less than or equal to the threshold, the heartbeat protocol data needs to be transmitted, and the data to be transmitted is "ID+UID+heartbeat protocol"; C2-4, the data to be transmitted is transmitted to the data control transmission module; when transmitting data to the data control transmission module for the first time, the data to be transmitted selects "ID+UID+complete video data" to ensure the data integrity of the subsequent audio and video data processing system. When transmitting data to the data control transmission module subsequently, the selection is made according to the judgment result of step C2-3.

4. A mining audio and video throttling system according to claim 1, characterized in that: For audio data, the difference comparison of adjacent frames includes the following steps: C3-1. Represent two adjacent front and back sampled audio frame signals in vector form: <h2 style=";text-align:left;direction:ltr">a=a1,a2,a3,…,a<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr">b, b1, b2, b3,..., b<h2 style=";text-align:left;direction:ltr"> n The above a1, a2, a3, ..., a n Represents the audio amplitude values ​​of the first sampled audio frame, b1, b2, b3, ..., b n Represents the audio amplitude values ​​of the second sampled audio frame, n is the dimension of the vector; C3-2. Calculate the Euclidean distance between two sampled audio frame signals: If the Euclidean distance exceeds the preset threshold, the complete audio data is transmitted, and the data to be transmitted is "ID+UID+complete audio"; Otherwise, the heartbeat protocol data is transmitted, and the data to be transmitted is "ID+UID+heartbeat protocol"; C3-3, transmitting the data to be transmitted to the data control transmission module; when transmitting data to the data control transmission module for the first time, the data to be transmitted selects "ID+UID+complete audio data"; when transmitting data to the data control transmission module subsequently, the selection is made according to the judgment result of step C3-2.

5. A mining audio and video throttling system according to claim 1, characterized in that: The control transmission module is assembled into the following data frame format according to the protocol: "protocol frame header transmission strategy transmission content protocol frame tail"; (1) Protocol frame header: a custom part used to identify the starting position of the data frame; (2) Transmission strategy: a Boolean value indicating the transmission strategy of the data frame: 0 indicates the heartbeat protocol, and 1 indicates the full transmission strategy; (3) Transmission content: under the heartbeat protocol strategy, the content is "ID+UID+heartbeat protocol"; Under the complete transmission strategy, the content includes "ID+UID+complete video / audio"; (4) Protocol frame tail: A custom part used to mark the end position of the data frame.

Images