Audio and video acquisition adaptive control and monitoring system based on scene recognition

Abstract

The application discloses a kind of audio and video acquisition adaptive control and monitoring system based on scene identification.Audio and video acquisition module real-time acquisition audio and video frame sequence signal, respectively transmission to scene analysis, joint trigger module;Scene analysis module is completed by double-flow network scene semantic segmentation and classification, and output segmentation chart and label signal to coding parameter decision module;Joint trigger module does joint decision to audio and video signal, generates key frame trigger signal transmission to audio and video coding module;Coding parameter decision module generates content quantization parameter offset chart, frame rate control signal according to semantic importance audio and video coding parameter and transmission;Audio and video coding module is adjusted coding configuration according to parameter, triggers key frame coding and exports code stream.The present application solves the problem that existing monitoring system cannot adaptively adjust acquisition and coding parameter according to scene semantic, causes the problem of code rate waste or key information loss.

Description

Technical Field

[0001] This invention relates to the field of video surveillance and adaptive coding technology, specifically to an audio and video acquisition adaptive control and monitoring system based on scene recognition. Background Technology

[0002] The widespread application of surveillance systems in security, transportation, and park management has made the real-time acquisition, encoding, and transmission of large-scale video data a core technological challenge. In typical surveillance deployments, cameras and microphones continuously capture audio and video streams, which are then compressed by an encoder and transmitted over a network to a monitoring center or cloud storage platform. Encoders typically employ a hybrid video coding framework, with core parameters including quantization parameters, frame rate, resolution, and image group structure. These parameters directly determine the quality and bitrate of the output stream.

[0003] Existing surveillance coding systems generally employ fixed parameter configurations or adaptive adjustment methods based solely on simple statistics. In fixed parameter configuration schemes, the encoder encodes with preset quantization parameters, frame rate, and resolution regardless of changes in the monitored scene content. The drawback of this approach is that when the scene is in a static state for an extended period, a significant amount of bitrate is wasted on repeatedly encoding unchanging background areas; while when intrusions, gatherings, or abnormal movements occur in the scene, fixed parameters cannot provide sufficient coding accuracy to clearly record key details, leading to difficulties in post-event evidence collection. Furthermore, regarding image group structure, existing systems typically insert intra-frame prediction keyframes at fixed intervals. This fixed-interval strategy is independent of the actual moment of scene change and may still encode inter-frame prediction frames during scene transitions, resulting in blockiness or blurring in several frames after the switch, with clarity only restored upon the arrival of the next predetermined keyframe.

[0004] Some improvement schemes attempt to trigger adaptive adjustment using motion vector statistics or pixel brightness change detection. For example, adjusting quantization parameters based on the average amplitude of motion vectors, or forcibly inserting keyframes when the brightness histogram change exceeds a threshold. However, these methods based on low-level pixel or motion statistics lack a deep understanding of scene semantics and cannot distinguish between genuine scene changes and pseudo-changes caused by lighting fluctuations, swaying leaves, camera shake, etc., easily leading to false triggers and parameter oscillations. In terms of audio, existing monitoring systems typically use fixed sampling rates and bitrates for audio acquisition and encoding, failing to adjust according to differences in the acoustic environment. For example, this wastes audio bandwidth in quiet environments and reduces speech intelligibility in noisy environments due to insufficient bitrate.

[0005] Furthermore, while existing technologies have incorporated scene classification to assist coding control, these methods typically only output coarse-grained scene type labels, lacking pixel-level semantic segmentation information and failing to guide the encoder in differentiated quality allocation at the coding unit granularity. Simultaneously, these schemes do not incorporate joint scene change detection of video and audio into keyframe triggering decisions, nor do they establish a dynamic mapping relationship between scene labels and audio coding parameters, resulting in insufficient overall adaptability of the system in complex multi-scene monitoring environments. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide an adaptive control and monitoring system for audio and video acquisition based on scene recognition, which solves the problem that existing monitoring systems cannot adaptively adjust audio and video acquisition and encoding parameters according to scene semantics, resulting in wasted bitrate or loss of key information. This invention performs scene semantic segmentation and classification on monitoring video frames through a dual-stream network, generating a scene semantic segmentation map and scene labels; simultaneously, it performs joint decision-making on audio abrupt changes and video scene switching to form a keyframe trigger signal; the encoding parameter decision module calculates the importance score of each encoding unit based on the scene semantic segmentation map and maps it to a quantization parameter offset map, generating a frame rate control signal and an audio encoding configuration signal based on the scene labels; the audio and video encoding module adjusts the quantization parameters, frame rate, and audio configuration according to the above signals, and encodes keyframes when a keyframe trigger signal is detected, enabling dynamic matching of the encoding strategy with the scene content.

[0007] This invention provides an audio and video acquisition adaptive control and monitoring system based on scene recognition, comprising: The audio and video acquisition module acquires video and audio frame sequence signals of the monitored scene in real time, and transmits the video frame sequence signals to the scene analysis module and the joint triggering module, and the audio frame sequence signals to the joint triggering module. The scene analysis module receives video frame sequence signals, performs scene semantic segmentation and classification through a dual-stream network, and generates scene semantic segmentation map signals and scene label signals, which are then transmitted to the encoding parameter decision module. The joint triggering module receives video and audio frame sequence signals, performs a joint decision on audio sudden changes and video scene switching, forms a key frame trigger signal, and transmits it to the audio and video encoding module. The encoding parameter decision module receives the scene semantic segmentation map and scene label signal, generates video and audio encoding parameter signals (video content quantization parameter offset map and frame rate control signal) based on semantic importance mapping, and transmits them to the audio and video encoding module; The audio and video encoding module receives various signals, adjusts the quantization parameters and frame rate according to the video encoding parameters, adjusts the audio configuration according to the audio encoding parameters, encodes key frames when a key frame trigger signal is detected, and outputs the encoded audio and video bitstream.

[0008] In one embodiment of the present invention, the dual-stream network in the scene analysis module consists of a spatial stream branch and a temporal stream branch. The spatial stream branch uses a residual network backbone to extract spatial semantic feature maps from the input single-frame video frame sequence signal. The temporal stream branch calculates a dense optical flow field for the continuous video frame sequence signal and extracts temporal motion feature maps through a three-dimensional convolutional layer. After the spatial semantic feature map and the temporal motion feature map are concatenated in the channel dimension, they are input into the hollow spatial pyramid pooling module to generate multi-scale context fusion features. The multi-scale context fusion features are output as scene semantic segmentation map signals by a first classifier. The scene semantic segmentation map signals assign scene category labels to each pixel. At the same time, the multi-scale context fusion features are output as scene label signals by a second classifier. The scene label signals include coarse-grained scene types and fine-grained activity categories. The coarse-grained scene types are indoor scenes or outdoor scenes, and the fine-grained activity categories are static unmanned, normal office, crowd gathering, or abnormal chasing.

[0009] In one embodiment of the present invention, the scene analysis module employs a time-interleaved processing strategy for the video frame sequence signal. The scene analysis module performs a complete dual-stream network inference once every preset number of video frames, where the preset number is an integer greater than one. For the intermediate video frame between two adjacent complete inferences, the scene analysis module performs linear interpolation based on the scene semantic segmentation map signals output from the two most recent complete inferences to generate the scene semantic segmentation map signal corresponding to the intermediate video frame, while keeping the scene label signal output from the most recent complete inference unchanged until the next complete inference updates the scene label signal. The linear interpolation calculation formula is to weight and sum the category confidence vectors of corresponding pixels in the two scene semantic segmentation maps according to the timestamp ratio of the intermediate frame.

[0010] In one embodiment of the present invention, the joint triggering module includes an audio mutation detection unit, a video scene switching detection unit, and a joint decision unit. The audio mutation detection unit receives an audio frame sequence signal, extracts Mel-frequency cepstral coefficient features frame by frame, and calculates the cosine distance between the Mel-frequency cepstral coefficient feature vectors of adjacent audio frames. When the cosine distance exceeds a preset audio mutation threshold, the audio mutation detection unit generates an audio mutation event marker signal and transmits it to the joint decision unit. The video scene switching detection unit receives a video frame sequence signal, converts each video frame into a thumbnail and extracts color histogram features, calculates the chi-square distance between the color histograms of adjacent video frames, and maintains a sliding window to record the chi-square distance sequence of the most recent frames. The video scene switching detection unit calculates the mean and standard deviation of the chi-square distance based on this sequence, and uses the mean plus three times the standard deviation as an adaptive video switching threshold. When the chi-square distance of the current video frame exceeds the adaptive video switching threshold, a video scene switching event marker signal is generated and transmitted to the joint decision unit. The calculation formula for the adaptive video switching threshold is as follows: in, To adapt to video switching thresholds, The mean chi-square distance within the sliding window. Let N be the standard deviation of the chi-square distance within the sliding window, and N be the number of frames in the sliding window. Let be the chi-square distance of the i-th frame. The current video frame rate, This represents the maximum video frame rate.

[0011] In one embodiment of the present invention, the joint decision unit receives the audio mutation event marker signal and the video scene switching event marker signal, and executes joint decision logic to form the keyframe trigger signal. The joint decision logic is defined as follows: the audio mutation event marker signal and the video scene switching event marker signal are respectively assigned preset audio weight coefficients and preset video weight coefficients, and then weighted and summed to obtain a joint change score. When the joint change score is greater than or equal to a preset trigger threshold, the joint decision unit generates a valid keyframe trigger signal. The preset video weight coefficient is greater than the preset audio weight coefficient, and the sum of the two is one. When the audio and video encoding module detects the valid keyframe trigger signal, it forcibly encodes the current encoded frame as an intra-frame predicted keyframe and resets the image group counter. The calculation formula for the joint change score is as follows: in, The score is based on the combined audio and video changes. For audio weighting coefficients, Label signals for audio mutation events. For video weighting coefficients, This is a marker signal for video scene transition events, where e is the natural constant. This is an adaptive video switching threshold.

[0012] In one embodiment of the present invention, the coding parameter decision module receives the scene semantic segmentation map signal and scene label signal transmitted by the scene analysis module, and performs quantization parameter offset calculation at the coding tree unit level based on the scene semantic segmentation map signal. The specific process of the quantization parameter offset calculation is as follows: for each coding tree unit in the current video frame, the foreground pixel ratio of the number of foreground object pixels in the coding tree unit to the total number of pixels is calculated, and the normalized value of the motion vector amplitude in the coding tree unit is calculated. The normalized value of the motion vector amplitude is obtained by dividing the motion vector amplitude by the maximum value of the motion vector amplitude in all coding tree units in the current frame. The foreground pixel ratio and the normalized value of the motion vector amplitude are multiplied by a preset foreground weight and a preset motion weight, respectively, and then summed to obtain the scene importance score of the coding tree unit. The calculation formula of the scene importance score is as follows: in, The scene importance score for the coding tree unit. Forward weighting, Foreground pixel ratio For motion weights, This is the normalized value of the motion vector amplitude. The score is given for the combined audio and video changes.

[0013] In one embodiment of the present invention, the coding parameter decision module linearly maps the quantization parameter offset of each coding tree unit within a preset quantization parameter offset range based on the scene importance score of each coding tree unit. The quantization parameter offset corresponding to the coding tree unit with a higher scene importance score is negative to obtain finer quantization, while the quantization parameter offset corresponding to the coding tree unit with a lower scene importance score is positive to perform coarser quantization. The coding parameter decision module adds the basic quantization parameters of the current frame to the quantization parameter offset of each coding tree unit to obtain the final quantization parameters of each coding tree unit. The basic quantization parameters are determined by the target bitrate through the bitrate and Lagrange multiplier model. The coding parameter decision module transmits the quantization parameter offset map signal containing the final quantization parameters of each coding tree unit as part of the video coding parameter signal to the audio and video coding module.

[0014] In one embodiment of the present invention, the encoding parameter decision module further receives a scene tag signal transmitted by the scene analysis module, and generates a frame rate control signal and a resolution control signal based on the scene tag signal. When the scene tag signal indicates a static unmanned scene and continues to exceed a preset static time threshold, the encoding parameter decision module generates a frame rate decrease control signal to gradually reduce the acquisition frame rate from the base frame rate to the minimum frame rate. When the scene tag signal switches from a static unmanned scene to another active scene, the encoding parameter decision module immediately generates a frame rate recovery control signal to restore the acquisition frame rate to the base frame rate. When the proportion of foreground pixels in the scene indicated by the scene tag signal is continuously lower than a first preset proportion threshold, the encoding parameter decision module generates a resolution reduction control signal to reduce the output resolution by one level. When the proportion of foreground pixels recovers to a level higher than a second preset proportion threshold and continues to exceed a preset duration threshold, a resolution recovery control signal is generated to increase the output resolution by one level. Both the frame rate control signal and the resolution control signal are transmitted to the audio and video encoding module as part of the video encoding parameter signal.

[0015] In one embodiment of the present invention, the encoding parameter decision module generates an audio encoding parameter signal based on the scene label signal. The encoding parameter decision module internally maintains a correspondence table between scene types and audio encoding configurations. In the correspondence table, a static unmanned scene corresponds to a first audio bitrate, a first sampling rate, and a configuration with noise reduction disabled; a normal office scene corresponds to a second audio bitrate, a second sampling rate, and a configuration with mild noise reduction; a scene with a large number of people corresponds to a third audio bitrate, a third sampling rate, and a configuration with moderate noise reduction; and an abnormal chasing scene corresponds to a fourth audio bitrate, a fourth sampling rate, and a configuration with noise reduction disabled. The first audio bitrate is lower than the second audio bitrate, the second audio bitrate is lower than the third audio bitrate, and the third audio bitrate is lower than the fourth audio bitrate. The encoding parameter decision module queries the correspondence table based on the currently received scene label signal to determine the target audio encoding parameter signal and transmits it to the audio and video encoding module. When the audio encoding parameter signal changes, the encoding parameter decision module generates a smooth transition signal to control the audio and video encoding module to gradually change the current audio encoding parameter from its original value to the target value linearly within a preset transition time window.

[0016] In one embodiment of the present invention, the system further includes a transmission and feedback module. The transmission and feedback module is connected to the output end of the audio and video encoding module, receives the encoded audio and video bitstream, and sends the bitstream to the receiving end through a communication network. Simultaneously, the transmission and feedback module monitors the transmission status of the communication network in real time. The transmission status includes at least three network status parameters: available bandwidth, packet loss rate, and round-trip time. The transmission and feedback module assembles the network status parameters into a network status vector signal and transmits the network status vector signal to the encoding parameter decision module through a feedback loop. After receiving the network status vector signal, the encoding parameter decision module adjusts the basic quantization parameter in the video encoding parameters according to the available bandwidth and packet loss rate. When the available bandwidth decreases or the packet loss rate increases, the encoding parameter decision module increases the basic quantization parameter to reduce the output bitrate. When the available bandwidth recovers or the packet loss rate decreases, the basic quantization parameter decreases to improve the encoding quality.

[0017] This invention provides an audio and video acquisition adaptive control and monitoring system based on scene recognition. It performs scene semantic segmentation and classification on monitored video frames through a dual-stream network, generating scene semantic segmentation maps and scene labels. Simultaneously, it performs joint decision-making for audio abrupt changes and video scene switching, forming keyframe trigger signals. An encoding parameter decision module calculates the importance score of each encoding unit based on the scene semantic segmentation map and maps it to a quantization parameter offset map. It also generates frame rate control signals and audio encoding configuration signals based on the scene labels. The audio and video encoding module adjusts the quantization parameters, frame rate, and audio configuration according to the aforementioned signals, and encodes keyframes when a keyframe trigger signal is detected, enabling dynamic matching of the encoding strategy with the scene content. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a system architecture diagram of an audio and video acquisition adaptive control and monitoring system based on scene recognition; Figure 2 This is a schematic diagram of the main flow of the overall system; Figure 3 This is a schematic diagram of the scenario analysis module process; Figure 4 This is a schematic diagram of joint triggering, coding decision-making, and coding closed loop. Detailed Implementation

[0020] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0023] Please see Figure 1-4 This invention provides an audio and video acquisition adaptive control and monitoring system based on scene recognition, including an audio and video acquisition module that acquires video and audio frame sequence signals of the monitored scene in real time, and transmits the video frame sequence signals to a scene analysis module and a joint triggering module, and the audio frame sequence signals to the joint triggering module; the scene analysis module receives the video frame sequence signals, performs scene semantic segmentation and classification through a dual-stream network, forms scene semantic segmentation map signals and scene label signals, and transmits them to an encoding parameter decision module; the joint triggering module receives the video and audio frame sequence signals, performs joint decision-making on audio mutation and video scene switching, forms keyframe trigger signals and transmits them to the audio and video encoding module; the encoding parameter decision module receives the scene semantic segmentation map and scene label signals, generates video and audio encoding parameter signals (video quantization parameter offset map and frame rate control signal) based on semantic importance mapping, and transmits them to the audio and video encoding module; the audio and video encoding module receives various signals, adjusts quantization parameters and frame rate according to video encoding parameters, adjusts audio configuration according to audio encoding parameters, encodes keyframes when keyframe trigger signals are detected, and outputs encoded audio and video bitstreams.

[0024] like Figure 1As shown, this invention proposes an adaptive control and monitoring system for audio and video acquisition based on scene recognition. The system aims to dynamically adjust each stage of audio and video acquisition and encoding by introducing scene semantic understanding. Physically, the system comprises five core modules: an audio and video acquisition module, a scene analysis module, a joint triggering module, an encoding parameter decision module, and an audio and video encoding module. These modules have clearly defined signal transmission relationships and work together to complete the entire process from raw audio and video signal input to adaptive encoded bitstream output.

[0025] like Figure 2 The diagram illustrates the overall architecture of a scene-recognition-based audio / video acquisition adaptive control and monitoring system. It comprises six core components: data acquisition, scene analysis, trigger decision, parameter decision, encoding output, and network feedback. The modules form a non-linear interaction and closed-loop feedback mechanism, fully presenting the system's workflow from raw signal acquisition to final bitstream output. The audio / video acquisition module, as the system's data source, undertakes the core task of real-time acquisition of video and audio frame sequences within the monitored scene. Its acquisition process is real-time and continuous, fully covering the visual and auditory information of the monitored scene. After acquisition, the video frame sequence signal is synchronously transmitted to the scene analysis module and the joint trigger module, while the audio frame sequence signal is transmitted separately to the joint trigger module, achieving signal splitting and multi-module multiplexing, providing raw data support for subsequent scene perception and trigger detection. The scene analysis module performs in-depth processing on the received video frame sequence signal, and completes scene semantic segmentation and scene classification operations based on a dual-stream network. Finally, it generates scene semantic segmentation map signal and scene label signal. This module is the core unit for the system to realize scene perception. It can convert visual signals into digital scene semantic information and provide a precise basis for adaptive adjustment of encoding parameters.

[0026] The joint triggering module synchronously receives video and audio frame sequence signals, jointly determining audio abrupt changes and video scene transitions. Overcoming the limitations of single-modal detection, it generates keyframe trigger signals through dual-modal fusion, accurately identifying scene change nodes and providing instructions for keyframe encoding in the audio and video encoding module. The encoding parameter decision module receives scene semantic segmentation map signals and scene label signals, generating video and audio encoding parameter signals based on semantic importance mapping rules. The video encoding parameters include a quantization parameter offset map and frame rate control signals. This module is the core of the system's adaptive control, dynamically allocating encoding resources according to scene characteristics. The audio and video encoding module integrates all received signals, adjusting the quantization parameters and frame rate of the encoding units based on the video encoding parameters, and adjusting the audio encoding configuration based on the audio encoding parameters. Upon detecting a keyframe trigger signal, it immediately executes keyframe encoding, ultimately outputting the encoded audio and video streams. The transmission and feedback module transmits the encoded bitstream to the receiving end via the network, while simultaneously monitoring network status parameters such as available bandwidth, packet loss rate, and round-trip delay in real time. It also feeds back the network status signal to the encoding parameter decision module, which adjusts the encoding parameters in real time based on the network status, forming a complete closed-loop control system. This ensures that the system can operate stably in different network environments, achieving the dual goals of bitrate optimization and critical information protection.

[0027] The audio and video acquisition module is the signal input for the entire system. It consists of at least one camera and at least one microphone, acquiring visual and auditory information from the monitored scene in real time, forming video frame sequence signals and audio frame sequence signals. The video frame sequence signal is a continuous series of two-dimensional images obtained at a set acquisition frequency, with each frame containing brightness and chromaticity information of multiple pixels. The audio frame sequence signal is a discrete sample sequence obtained by sampling and quantizing analog sound waveforms at fixed time intervals. When outputting signals, this module simultaneously provides the video frame sequence signal to the scene analysis module and the joint triggering module, while providing the audio frame sequence signal separately to the joint triggering module, thus constituting the first-level signal distribution within the system.

[0028] The scene analysis module is dedicated to extracting high-level semantic information from video frame sequences. Internally, this module deploys a pre-trained dual-stream deep convolutional neural network, with two branches handling spatial and temporal information respectively. The spatial flow branch, using a residual network as its backbone, performs deep convolution operations on single video frames, abstracting spatial features such as edges, textures, shapes, and object parts layer by layer, ultimately generating a spatial semantic feature map. The temporal flow branch receives two consecutive video frames, first using a dense optical flow algorithm to calculate the motion vector of each pixel between frames, forming a two-dimensional optical flow field. Then, it performs temporal convolution on the optical flow field using stacked three-dimensional convolutional kernels, extracting a temporal motion feature map reflecting motion patterns, velocity changes, and motion continuity. The spatial semantic feature map and the temporal motion feature map are concatenated along the channel dimension, and after dimensionality reduction via a one-to-one convolution, they are fed into a dilated spatial pyramid pooling structure. This structure uses multiple dilated convolutional layers with different dilation rates in parallel, capturing multi-scale information from local details to global context while maintaining the spatial resolution of the feature map. The multi-scale fusion features output by the void space pyramid pooling are passed to two parallel classifiers respectively.

[0029] The first classifier is a pixel-wise classifier that performs pixel-level category inference for each spatial location on the fused feature map, labeling each pixel with a scene category, such as background, human, vehicle, or object. The set of these labels forms the scene semantic segmentation map signal. The second classifier is a global classifier that performs hierarchical classification of the entire frame's features, outputting both coarse-grained scene types representing the overall environment (e.g., indoor or outdoor) and fine-grained activity categories representing specific activity states (e.g., static and unmanned, normal office work, crowd gathering, or abnormal chasing). This output constitutes the scene label signal. The scene analysis module ultimately passes the scene semantic segmentation map signal and the scene label signal to the encoding parameter decision module. To meet real-time requirements on edge monitoring devices with limited computing power, this module can also employ a time-interleaved processing strategy. This means that a complete two-stream network forward inference is performed only every few frames. The semantic segmentation map of intermediate frames is obtained by linear interpolation of the segmentation maps of the preceding and following keyframes, while the scene label retains the previous inference result, thus significantly reducing the average computational cost.

[0030] The joint triggering module independently receives video frame sequence signals and audio frame sequence signals. Its task is to accurately detect changes in real scene content and issue a command to force keyframe encoding accordingly. This module internally includes an audio abrupt change detection unit, a video scene transition detection unit, and a joint decision unit. The audio abrupt change detection unit continuously analyzes the input audio frame sequence signals, extracting Mel-frequency cepstral coefficient features frame by frame. This feature can effectively characterize the human ear's perception of sound frequencies. It then calculates the cosine distance between the Mel-frequency cepstral coefficient vector of the current audio frame and the vector of the previous frame. This distance measures the similarity of the spectral envelope shape between the two sound frames. When the cosine distance exceeds an audio abrupt change threshold adaptively set according to the noise floor, this unit generates an audio abrupt change event marker signal. The video scene transition detection unit first downsamples each video frame to generate a thumbnail to reduce computation, and then statistically analyzes the color histogram features on the thumbnail, typically using a partitioned statistical method based on hue, saturation, and brightness color spaces.

[0031] For two adjacent frames, the chi-square distance between their color histograms is calculated. This distance can sensitively reflect changes in the overall color distribution of the image. This unit also maintains a sliding window, recording the chi-square distance values ​​of the most recent frames, calculating the mean and standard deviation of these distance values ​​in real time, and using the mean plus three times the standard deviation as the adaptive video switching threshold for the current moment. When the chi-square distance corresponding to a frame exceeds this dynamic threshold, a video scene switching event marker signal is generated. The joint decision unit receives the above two event marker signals and performs a fusion decision based on preset video and audio weight coefficients. Since video signals usually reflect scene changes more directly, the video weight is set higher than the audio weight, and their sum is one. The joint change score is calculated by multiplying the video event marker signal by the video weight coefficient and adding the audio event marker signal multiplied by the audio weight coefficient. The formula for calculating the joint change score is as follows: The formula uses a weighted summation-based scoring logic, through... The sigmoid nonlinear transform normalizes the difference between audio and video event markers. When both modalities are triggered simultaneously, the correction term approaches 1, maximizing the score. When a single modality is triggered, the correction term decreases, suppressing false triggers. Simultaneously, it introduces... The logarithmic compensation term incorporates the dynamic changes in the video switching threshold into the score calculation, deeply binding the joint change score to the magnitude of scene changes. It also forms a data linkage with the adaptive video switching threshold formula, making full use of the complementarity of audio and video dual-modal information, improving the robustness of keyframe triggering, avoiding false triggering or missed triggering caused by audio noise and video light interference, ensuring that the audio and video encoding module encodes intra-frame keyframes in a timely manner when the scene changes abruptly, optimizing the bitstream structure and transmission efficiency, and providing a quantitative basis for the degree of scene change for subsequent encoding parameter decisions.

[0032] like Figure 3The diagram illustrates the internal processing flow of the scene analysis module. Through dual-stream network feature extraction, multi-scale fusion, and temporal interpolation, it achieves efficient semantic segmentation and scene classification of video frame sequences. A non-linear branching and iterative update mechanism is established between modules to balance processing accuracy and operational efficiency. The video frame sequence input serves as the starting point for scene analysis, receiving continuous video frame sequence signals from the audio / video acquisition module. This provides stable raw visual data for subsequent feature extraction, ensuring the continuity and real-time nature of scene analysis. The spatial flow branch, with a residual network as its backbone, extracts spatial semantic features from single-frame video frame sequences, generating a spatial semantic feature map. This branch focuses on capturing static visual information of the scene, including object contours, region distribution, pixel categories, and other static semantic features, making it the core branch for achieving static semantic understanding of the scene. The temporal flow branch calculates a dense optical flow field from continuous video frame sequences and extracts temporal motion feature maps through 3D convolutional layers. It focuses on capturing dynamic information such as target motion and image changes within the scene, compensating for the spatial flow branch's limitation of only processing static information and achieving accurate extraction of dynamic scene features. The channel-dimensional feature concatenation module fuses spatial semantic feature maps and temporal motion feature maps along the channel dimension, integrating static spatial features with dynamic temporal features to form a fused feature containing complete scene information, providing foundational data for subsequent multi-scale analysis. The hollow spatial pyramid pooling module processes the fused feature to generate multi-scale contextual fused features, capable of capturing scene context information at different scales, improving feature representation capabilities in complex scenes, and adapting to the analysis needs of different indoor and outdoor scenes and targets of varying sizes. The semantic segmentation map output module generates pixel-level scene semantic segmentation map signals based on the multi-scale fused features, assigning scene category labels to each pixel to achieve refined semantic segmentation of the scene. The scene label output module simultaneously generates scene label signals containing coarse-grained scene types and fine-grained activity categories, clearly defining the overall attributes and activity status of the scene. The temporally interleaved linear interpolation module adopts a temporally interleaved processing strategy. It performs a complete dual-stream network inference once every preset number of frames. For intermediate frames, it generates the corresponding scene semantic segmentation map signal through linear interpolation and keeps the scene label signal unchanged. This module forms a cyclic update mechanism, which reduces the amount of computation while ensuring the accuracy of analysis, improves the efficiency of module operation, and ensures that scene analysis can adapt to the speed requirements of real-time video processing.

[0033] When the score is greater than or equal to the preset trigger threshold, the joint decision unit determines that a real and meaningful scene switch has occurred, and then generates a valid keyframe trigger signal, which is then transmitted to the audio and video encoding module. This joint audio and video mechanism effectively eliminates individual video or audio anomalies caused solely by lighting changes, leaf swaying, or transient noise, significantly improving trigger accuracy. The formula for calculating the adaptive video switching threshold is as follows: The formula is based on the traditional 3σ criterion for threshold calculation. It enhances the ability to represent the dispersion of data within the sliding window by using the square root of the sum of squares of the standard deviations, and also introduces a frame rate normalization correction term. When the acquisition frame rate approaches the maximum frame rate, the correction term approaches 0, and the threshold is determined solely by statistical characteristics. When the acquisition frame rate decreases, the correction term increases, and the threshold increases accordingly. This avoids false detection of scene switching caused by inter-frame information redundancy at low frame rates. The video switching judgment criteria can be dynamically adjusted according to the real-time acquisition status of the monitoring scene, adapting to different indoor and outdoor monitoring scenes and frame rate changes of different acquisition devices. This provides an accurate threshold basis for the generation of subsequent video scene switching event marker signals, ensuring the sensitivity and stability of video scene change detection. It provides a reliable input signal for audio and video joint decision-making and is the basic algorithm guarantee for the normal operation of the entire joint triggering module.

[0034] The encoding parameter decision module is the core hub of adaptive control in this system. It receives scene semantic segmentation map signals and scene label signals from the scene analysis module, and generates multi-dimensional parameter signals for controlling video encoding and audio encoding respectively. For video encoding control, this module first aligns the scene semantic segmentation map signal with the encoding tree cell grid of the current video frame, and calculates the semantic importance score for each encoding tree cell. The calculation process is as follows: The proportion of pixels labeled as foreground objects within the encoding tree cell is counted out of the total number of pixels in that cell; simultaneously, the motion vector amplitude of the encoding tree cell is obtained from the video encoding preprocessing stage and divided by the maximum value of the motion vector amplitudes of all encoding tree cells in the current frame to obtain the normalized motion amplitude. The foreground pixel proportion is multiplied by a preset foreground weight, and the normalized motion amplitude is multiplied by a preset motion weight. The sum of these two products gives the scene importance score for the encoding tree cell, where the foreground weight is greater than the motion weight to highlight the importance of the semantic object. After calculating the importance scores of all coding tree units, the module performs a linear mapping within a preset quantization parameter offset range. This results in the coding tree unit with the highest importance score receiving a negative maximum offset (i.e., a decrease in the quantization parameter) to provide finer quantization and ensure high quality; while the background region with the lowest importance score receives a positive maximum offset (i.e., an increase in the quantization parameter) to implement coarser quantization and save bitrate. The formula for calculating the scene importance score is as follows: The formula uses the weighted sum of foreground proportion and motion vector as the basis for importance characterization, and introduces... The normalization factor incorporates the joint change score into the calculation, while also... The geometric mean term strengthens the synergistic effect of foreground and motion features, enabling the scene importance score to simultaneously consider static semantic priority and dynamic change degree. Together with the first two formulas, it forms a complete technical link, transmitting the joint audio and video change features to the coding parameter decision-making stage. The higher the scene importance score, the more the coding tree unit contains the core monitoring target and the more drastic the motion change, requiring more refined quantization parameters. The lower the score, the more redundant the background area, and coarse quantization can be used to save bitrate. This algorithm realizes multi-dimensional fusion quantization of monitoring scene semantics, motion information, and audio and video changes, providing core data support for quantization parameter offset map generation, frame rate and resolution control, significantly improving the scene adaptability of audio and video coding, reducing the overall bitrate while ensuring the clarity of the monitoring target, and adapting to the dynamic changes of network transmission.

[0035] like Figure 4As shown, this is a closed-loop execution process for joint triggering, encoding parameter decision-making, and audio / video encoding. Through dual-modal trigger detection, multi-dimensional parameter decision-making, and network adaptive feedback, the entire audio / video encoding process achieves adaptive control. A non-linear working logic of branch decision-making and cyclic adjustment is formed between modules. The audio / video frame input stage synchronously receives video frame sequence signals and audio frame sequence signals output from the audio / video acquisition module, providing synchronous dual-modal raw data for subsequent trigger detection and encoding processing, ensuring signal processing consistency. The audio mutation detection unit extracts the Mel-frequency cepstral coefficient features of the audio signal frame by frame, calculates the cosine distance between the feature vectors of adjacent frames, determines the audio mutation state, and generates an audio mutation event marker signal, accurately capturing scene change information in the audio dimension. The video scene switching detection unit converts video frames into thumbnails and extracts color histogram features, calculates the chi-square distance between adjacent frames, generates an adaptive video switching threshold through sliding window statistics, determines the scene switching state, and generates a video scene switching event marker signal, achieving accurate detection of scene mutations in the video dimension. The joint decision unit receives dual-modal event marker signals, generates a joint change score through weighted calculation, and determines whether to generate a valid keyframe trigger signal based on the score, forming two branches: triggered and non-triggered. This effectively integrates audio and video dual-modal information, improving the accuracy of keyframe triggering. The keyframe trigger signal branch outputs a trigger command when the decision is valid, while the ordinary frame encoding branch executes the regular encoding process when no trigger occurs. Together, they provide encoding mode instructions for the audio and video encoding module. The scene semantic / tag input stage receives the semantic segmentation map and scene tag signals output by the scene analysis module, providing scene semantic basis for encoding parameter decisions. The encoding parameter decision module, as the core control unit, generates multi-dimensional encoding parameters based on scene information. Among them, the quantization parameter offset calculation module generates quantization parameter offset based on the scene importance score of the encoding tree unit, realizing quantization accuracy adjustment at the encoding unit level; the frame rate / resolution control module dynamically adjusts the acquisition frame rate and output resolution according to the scene type, balancing resource consumption and image quality; and the audio encoding parameter generation module matches the corresponding audio bit rate, sampling rate, and noise reduction configuration according to the scene, adapting to the audio encoding requirements of different scenes. The audio and video encoding module integrates all parameters and trigger signals, adjusts the encoding configuration as required, executes keyframe or normal frame encoding, and finally outputs a standard audio and video stream. The transmission module is responsible for stream transmission and network status monitoring, feeding back network status parameters to the encoding parameter decision module. The encoding parameter decision module corrects the encoding parameters in real time based on the network status, forming an adaptive closed loop. This ensures that the system can still stably output a high-quality stream even when the network fluctuates, maximizing the adaptability and reliability of the monitoring system.

[0036] The quantization parameter offsets of each coding tree unit constitute a quantization parameter offset map signal. The module also obtains the basic quantization parameters determined by the target bitrate through a Lagrange multiplier model from the bitrate control stage. These basic quantization parameters are added to the offsets of each coding tree unit to obtain the final quantization parameters for each coding tree unit in that frame. This process enables differentiated coding quality allocation based on semantic content within a single frame. Furthermore, the coding parameter decision module utilizes scene label signals to regulate the frame rate. When the scene label continuously indicates a static, unmanned scene within a preset static time threshold, the module generates a frame rate decrement control signal, gradually reducing the acquisition or coding frame rate from the base frame rate to a preset minimum frame rate according to a certain step size or linear curve, significantly compressing the amount of data during low-activity periods. Once the scene label changes to any other activity category, the module immediately generates a frame rate recovery control signal, instantly returning the frame rate to the base frame rate to ensure complete recording of sudden activities. Similarly, the module can also implement resolution adjustment based on the overall proportion of foreground pixels in the scene label. In terms of audio control, the encoding parameter decision module internally maintains a mapping table between scene types and audio encoding configurations. This table defines various audio parameter combinations for different scenarios: for static, unmanned scenarios, the lowest audio bitrate and sampling rate are used, and noise reduction is disabled to maximize audio bandwidth savings; for normal office scenarios, the audio bitrate and sampling rate are appropriately increased, and mild noise reduction is enabled to ensure clear dialogue; for scenarios with many people, due to background noise, the bitrate and sampling rate need to be further increased, and moderate noise reduction is enabled to highlight human voices; for sudden scenarios such as abnormal chases, the highest audio bitrate and sampling rate are configured, and noise reduction is disabled to preserve complete environmental sound as clues for event analysis. The module looks up the target audio encoding parameter signal based on the currently received scene tag signal and sends this signal to the audio and video encoding module; when audio parameters need to be switched, to avoid auditory pops or encoder malfunctions caused by direct switching, the module also generates a smooth transition control signal, instructing the encoder to gradually change the current parameters to the target parameters using a linear interpolation method within a preset transition time window. In summary, the video encoding parameter signal output by the encoding parameter decision module includes a quantization parameter offset map, a frame rate control signal, and a selectable resolution control signal. The output audio encoding parameter signal includes information such as audio bit rate, sampling rate, noise reduction intensity, and transition control. These signals are all passed to the audio and video encoding module.

[0037] The audio / video encoding module is the system's execution mechanism. It aggregates all upstream signals and performs the actual compression encoding of the raw audio and video data accordingly. Its input port receives video and audio frame sequence signals from the audio / video acquisition module. Simultaneously, it receives video and audio encoding parameter signals from the encoding parameter decision module via a parameter interface, and keyframe trigger signals from the joint trigger module via an interrupt or event interface. In the video encoding path, this module typically employs a block-based hybrid encoding architecture, dividing each frame into several encoding tree units. For each encoding tree unit, the module reads the corresponding quantization parameter value from the quantization parameter offset map signal and applies this value during the transformation and quantization steps. This results in finer quantization for key areas such as people and vehicles in the image, while background areas are quantized more coarsely, thereby maximizing subjective quality within the overall bitrate constraint. Simultaneously, the module adjusts the encoding clock or frame selection logic based on the frame rate control signal: when a frame rate reduction signal is received, it reduces the actual output frame rate by skipping frames or increasing the frame interval; when a recovery signal is received, it cancels frame skipping and restores the base frame rate. If a resolution control signal is received, the module downsamples or upsamples the acquired frames before encoding, thereby changing the resolution of the actual encoding input. In the audio encoding path, the module configures the audio encoder in real time based on audio encoding parameter signals, such as setting the corresponding bitrate level, sampling rate, and the on / off state and intensity level of the noise reduction algorithm. If a smooth transition signal is received, the audio encoder internally buffers several frames and performs a slow parameter update. During normal encoding, the video encoder performs inter-frame or intra-frame predictive encoding according to a preset frame type or one determined by the image group structure. Once the module detects a valid trigger signal at the keyframe trigger signal input port, the module's frame type decision logic is immediately interrupted, forcing the currently processed video frame to be set as an intra-frame predictive frame, i.e., a keyframe for full-frame compression, and simultaneously resetting the image group counter to its initial state. This operation ensures that a complete and clear reference image, independent of previous frames, is output immediately at the moment of a realistic scene transition, avoiding screen tearing or blurring at the decoding end caused by sudden changes in the reference frame content. After undergoing the above adaptive processing, the video and audio elementary streams are finally synchronized and packaged according to timestamps by the system multiplexer to form coded audio and video streams that can be stored or transmitted, and then sent out through the output interface.

[0038] As described above, this system constructs a complete technical chain, from multimodal acquisition, scene semantic understanding, and joint audio-video change detection, to unit-level quantization adjustment of the coding tree, frame rate / resolution adaptation, and dynamic mapping of audio parameters. The signal flow between modules is clear: the audio-video acquisition module simultaneously supplies video frames to the scene analysis and joint triggering modules, and audio frames to the joint triggering module; the scene analysis module sends semantic segmentation maps and scene labels to the coding parameter decision module; the joint triggering module independently generates keyframe trigger signals for the coding module; the coding parameter decision module generates multi-dimensional coding configuration signals by parsing semantic information and transmits them to the coding module; the audio-video coding module, as the final bearer, outputs a content-adaptive bitstream under the comprehensive control of various signals. This architecture enables the monitoring system's bitrate allocation, coding quality, and recording integrity to be adjusted in real time according to the semantic changes in scene content, solving the long-standing shortcomings of existing fixed-parameter monitoring systems in terms of resource utilization and event recording effectiveness.

[0039] In one embodiment of the present invention, a scene-recognition-based audio and video acquisition adaptive control and monitoring system was deployed in the open laboratory area of ​​its R&D building. The laboratory area, approximately 200 square meters, includes four sets of experimental tables, two discussion areas, and a large display screen. Daily usage scenarios include unattended operation at night, normal office work and experiments during the day, occasional project discussion gatherings, and occasional abnormal situations caused by equipment malfunctions or operational errors. This embodiment uses this environment as a background to fully demonstrate the system's workflow and dynamic interactions between modules during actual operation. The system hardware consists of four 2-megapixel network cameras, two omnidirectional microphones, one edge computing node, and one network recording device. Each camera corresponds to one video stream, and each microphone corresponds to one audio stream. The edge computing node carries the core software modules of the system, including a scene analysis module, a joint triggering module, an encoding parameter decision module, and an audio and video encoding module. The functions of the audio and video acquisition module are handled by the front-end firmware of the cameras and microphones. After system startup, four video frame sequences were continuously generated at a sampling frequency of 25 frames per second, a resolution of 1920 x 1080 pixels per frame, and a color depth of 8 bits per pixel. Two audio frame sequences were continuously generated at a sampling rate of 48 kHz, a quantization precision of 16 bits, and a frame length of 20 milliseconds. In a typical workday, the scene underwent a complete cycle from no one at night to employees gradually returning to work, a midday gathering for discussion, a sudden anomaly in the afternoon, and then back to no one at night. The system automatically executed different adaptive control strategies at each of these stages.

[0040] At 3:00 AM, the laboratory was deserted, all lights were off, and only the indicator lights on the equipment emitted faint glows. The video frame sequence signals captured by the cameras were almost entirely against a dark background, with only occasional flickering lights. The scene analysis module, following a preset time-interleaving strategy, performed a complete two-stream network inference on the video frames of one of the main cameras every ten frames. Taking the current frame as an example, the spatial flow branch extracted the spatial semantic feature map from the input frame. The edges of objects such as the lab bench, chair, and display screen in the image were blurred and had sparse textures. The temporal flow branch calculated the dense optical flow between the two most recent frames. Since there were no moving objects in the scene, the magnitude of the optical flow vector was close to zero. After the feature maps from the two branches were concatenated along the channel dimension and subjected to hollow spatial pyramid pooling, in the scene semantic segmentation map output by the first classifier, more than 99% of the pixels were marked as background, and only a very small number of pixels were marked as objects due to the reflection of the indicator lights. The second classifier output a coarse-grained scene type as an indoor scene and a fine-grained activity category as a static, unmanned scene. This information constituted the scene semantic segmentation map signal and the scene label signal, which were transmitted to the encoding parameter decision module. Simultaneously, the audio mutation detection unit of the joint triggering module analyzes the audio frame sequence signals collected by the two microphones, calculating the cosine distance of the Mel frequency cepstral coefficient vector of each frame. Since the ambient noise remains stable, the cosine distance fluctuates between 0.05 and 0.1, far below the preset trigger threshold of 0.35, and the audio mutation event marker signal remains zero. The video scene switching detection unit calculates the chi-square distance of the color histograms of adjacent video frame thumbnails. The distance between the current frame and the previous frame is 0.02, while the mean of the last sixty frames within the sliding window is 0.02, the standard deviation is 0.005, and the adaptive threshold is 0.035. The distance value does not exceed the threshold, and the video scene switching event marker signal is also zero. The joint decision unit calculates a joint change score of zero and does not generate a keyframe trigger signal.

[0041] After receiving a static, unmanned scene label signal, the encoding parameter decision module determines the corresponding control parameters based on its internally maintained mapping table. In the video dimension, this module calculates a semantic importance score for each coding tree unit in the current frame. Taking a background coding tree unit located in the upper left corner of the screen as an example, the foreground pixel ratio within this unit is zero, the motion vector amplitude obtained during preprocessing is 0.3 pixels, the maximum motion vector amplitude among all coding tree units in the current frame is 0.5 pixels, and the normalized motion amplitude is 0.6. With a foreground weight of 0.6 and a motion weight of 0.4, the scene importance score for this unit is 0 multiplied by 0.6 plus 0.6 multiplied by 0.4, which equals 0.24. After linear mapping, this score corresponds to a positive offset of 4.5 within the preset quantization parameter offset range of -6 to +6, meaning the quantization parameter for this coding tree unit is increased by 4.5 based on the basic quantization parameter, implementing a coarser quantization. For the vast majority of coding tree units across the entire frame, the importance score is below 0.3, and the quantization parameter offset is always positive. Meanwhile, since the scene tag signal has continuously indicated static and unmanned for the past ten consecutive seconds, exceeding the preset five-second static time threshold, the encoding parameter decision module generates a frame rate reduction control signal. The target frame rate starts from the baseline of 25 frames per second, decreasing by one frame every two seconds, and finally stabilizing at the minimum frame rate of five frames per second after forty seconds. In the audio dimension, the encoding parameter decision module looks up the target audio encoding parameters as follows: the encoding format uses pulse code modulation, the audio bitrate is 64 kilobits per second, the sampling rate is 8 kilohertz, and noise reduction processing is disabled. The module sends these audio parameter signals to the audio and video encoding module immediately, as it is a switch from a higher bitrate to a lower bitrate, and no smooth transition is required. The encoding parameter decision module transmits the quantization parameter offset map signal containing the quantization parameter offset data of each coding tree unit, the frame rate reduction control signal, and the above-mentioned audio encoding parameter signals to the audio and video encoding module together.

[0042] Upon receiving the aforementioned signal, the audio / video encoding module first divides the four video frames of the current frame into encoding tree units. It then reads the offset of each unit from the quantization parameter offset map signal and adds it to the basic quantization parameter. For background area units, a larger quantization parameter is used, resulting in coarser encoding of image details and a significant reduction in the number of bits allocated in the bitstream. For the only one or two units containing only faint indicator lights, the quantization parameter remains near the basic value. Simultaneously, the frame rate control logic of the encoding module increases the frame interval frame by frame over the next few tens of seconds, gradually increasing from one frame every forty milliseconds to one frame every two hundred milliseconds. The audio encoding channel directly switches the encoder configuration to pulse code modulation mode, a bit rate of 64 kilobits per second, and an 8 kilohertz sampling rate at the boundary of the next audio frame, and disables the noise reduction algorithm module. Since the joint trigger module does not output a keyframe trigger signal at this time, the video encoder continues to encode according to the preset image group structure, i.e., encoding one intra-frame prediction frame every fifty frames, with the remaining frames being inter-frame prediction frames. Ultimately, all encoded primary streams are multiplexed and packaged into a unified encoded audio and video stream, which is then transmitted over the network to the network video recorder at the monitoring center for storage. During this stage, the single-channel video bitrate is maintained at approximately 200 kilobits per second, and the audio bitrate at approximately 64 kilobits per second, resulting in extremely low total system bandwidth usage and effectively saving storage space during nighttime operation.

[0043] At 9:00 AM, employees entered the lab one after another and turned on the lights, quickly switching the scene from a static, unmanned environment to normal office work. In the next round of dual-stream inference, the scene analysis module detected an overall surge in screen brightness and the appearance of multiple human silhouettes. The first classifier labeled multiple human body regions as human bodies, and the second classifier updated the scene label signal to an indoor scene and normal office work. The scene semantic segmentation map signal and scene label signal were immediately transmitted to the encoding parameter decision module after generation. Simultaneously, the audio mutation detection unit of the joint triggering module detected a sudden increase in the cosine distance of the Mel-frequency cepstral coefficient vector of the audio frame caused by door opening sounds, footsteps, and conversations, reaching 0.62, exceeding the threshold of 0.35, and the audio mutation event label signal became 1; the video scene switching detection unit detected a surge in the chi-square distance of the color histogram caused by turning on lights and the appearance of people, reaching 0.53, far exceeding the current adaptive threshold of 0.04, and the video scene switching event label signal became 1. The joint decision unit calculates the joint change score: the video event marker signal 1 multiplied by the video weight 0.7 plus the audio event marker signal 1 multiplied by the audio weight 0.3 equals 1.0, which reaches the preset trigger threshold of 1.0. Therefore, it is determined that a real scene change has occurred, and a valid keyframe trigger signal is generated and sent to the audio and video encoding module.

[0044] After receiving the scene label signal for normal office work, the encoding parameter decision module looks up the corresponding video and audio control strategies in a table. For video, the module calculates the semantic importance score for each encoding tree unit. Taking an encoding tree unit in the center of the screen containing the upper body of an employee as an example, the foreground pixel ratio within this unit is 0.85, the motion vector amplitude is 3.2 pixels, the maximum motion vector amplitude for the entire frame is 8.0 pixels, and the normalized motion amplitude is 0.4. The scene importance score for this unit is 0.85 multiplied by 0.6 plus 0.4 multiplied by 0.4 equals 0.51 plus 0.16 equals 0.67. This score, within the offset range of -6 to +6, maps to a quantization parameter offset of -2.5. The quantization parameter for this unit is reduced by 2.5 from the base quantization parameter, resulting in finer quantization quality. Meanwhile, the background unit located in the corner of the screen, due to its zero foreground ratio and extremely small motion vector, has an importance score of only about 0.1, mapping to a quantization parameter offset of +5. The quantization parameter offset for important regions across the entire frame is negative, while the offset for background regions is positive, forming a quantization parameter distribution map that matches the importance of the content. Furthermore, as the scene label switches from static and unattended to normal office work, the module instantly reverses the frame rate control signal from a decrement instruction to a recovery instruction, immediately restoring the target frame rate to 25 frames per second. Regarding audio, the encoding parameter decision module looks up the target configuration for the normal office scene: a second-generation internet low-bitrate audio encoder, an audio bitrate of 128 kilobits per second, a sampling rate of 16,000 Hz, and light noise reduction enabled. The module generates an audio encoding parameter signal and transmits it to the encoding module. Since the audio bitrate needs to be increased from 64 kilobits per second to 128 kilobits per second, this signal also includes a smooth transition instruction with a duration of 0.2 seconds.

[0045] The audio and video encoding module performs the following operations under the combined effect of all signals. First, upon receiving a keyframe trigger signal, the module interrupts the normal encoding prediction of the current frame, forcibly sets the frame type of the current frame to an intra-frame prediction frame, encodes the current video frame using full intra-frame compression, and then resets the image group counter to zero. This operation ensures that the first key frame after the employee enters the laboratory and turns on the lights is recorded completely and independently of previous frames. When the decoder reviews the image at that point in time, it is immediately clear without any trailing or block blurring. Subsequent frames resume normal inter-frame prediction encoding, but for each coding tree unit, the quantization process strictly follows the offset specified in the quantization parameter offset map signal. The improved quantization precision in the human area results in clear preservation of facial details and clothing, while the coarse quantization in the background area saves bitrate without affecting the subjective viewing quality. The frame rate switches directly from five frames per second to twenty-five frames per second within tens of milliseconds, restoring smooth playback. The audio encoder linearly increases the bitrate from 64 kilobits per second to 128 kilobits per second within a 0.2-second transition window, switching the sampling rate from 8 kHz to 16 kHz, and activates a mild noise reduction module after the transition. This module uses spectral subtraction to remove steady-state background noise such as air conditioner operation while preserving the energy of human voice frequencies, ensuring high intelligibility of conversations between employees in the recorded file. During this stage, the single-channel video bitrate increases to approximately 800 kilobits per second, and the audio bitrate to approximately 128 kilobits per second, with a reasonable increase in system bandwidth utilization to match the recording quality requirements during office hours.

[0046] At 12:15 PM, six employees gathered in the discussion area in the center of the lab to discuss project plans. The scene category shifted from normal office work to a gathering of people. The scene analysis module's dual-stream network captured the spatial distribution of multiple people in close proximity and more frequent interactive gestures compared to normal office work in the next round of inference. The adjacent human pixel regions in the spatial semantic feature map covered more than 40% of the screen, and the temporal stream detected numerous short, repetitive optical flow patterns. The second classifier updated the scene label signal to an indoor scene and a gathering of people. Upon receiving the gathering of people label, the encoding parameter decision module executed the corresponding strategy switch. In the video dimension, the importance scores of each encoding tree unit increased overall due to the increased coverage of multiple foreground human body regions, especially in the discussion group area in the center of the screen, where the importance scores of multiple encoding tree units exceeded 0.75. The quantization parameter offset was mapped to -3 to -5, further improving the encoding quality. Since the gathering of people is an activity-related scene, the frame rate was maintained at the base frame rate of 25 frames per second. The module may also trigger controls such as slightly increasing the intensity of image sharpening preprocessing to enhance the clarity of human outlines, if the camera supports it. In the audio dimension, the target configuration for the scene of people gathering is as follows: advanced audio encoding format, audio bitrate of 192 kilobits per second, sampling rate of 44,100 Hz, and moderate noise reduction enabled. Due to the mixing of background human voices, moderate noise reduction employs a spatial filtering algorithm based on multi-microphone beamforming to enhance the speech in the discussion area directly in front while suppressing reflected sound and irrelevant conversations from the sides and far away. The audio encoding parameter signal is transmitted to the encoding module, and the process of transitioning the bitrate from 128 kilobits per second to 192 kilobits per second is also completed within two seconds with zero clicks according to the smooth transition instruction. At this time, although the scene changes from normal office work to a gathering of people, the change process is relatively gradual. Neither the audio mutation detection nor the video switching detection of the joint trigger module detects a mutation event exceeding the threshold at the single frame boundary, and the joint change score is less than 1.0, so the keyframe trigger signal remains invalid. The video encoder continues to encode according to the image group structure without forced keyframe intervention. Within minutes of this phase, the video bitrate remained at approximately 1.2 megabits per second, and the audio bitrate at approximately 192 kilobits per second. The multi-person discussion audio in the recorded file was clearly distinct and easily identifiable.

[0047] At 3:22 PM, a short circuit occurred in a power supply device during the experiment, producing a loud bang. A startled experimenter knocked over a metal ladder while retreating. Within 20 milliseconds of the bang, the audio mutation detection unit detected a spike in the cosine distance of the Mel frequency cepstral coefficient vector to 0.91, far exceeding the threshold, and the audio mutation event marker signal changed to 1. Almost simultaneously, the video scene transition detection unit, due to flashes of light and rapid movement of people in the scene, saw its color histogram chi-square distance jump to 0.72, exceeding the current adaptive threshold, and the video scene transition event marker signal changed to 1. The joint decision unit calculated the joint change score as 1 multiplied by 0.7 plus 1 multiplied by 0.3, equaling 1.0, triggering a valid keyframe trigger signal. The encoding parameter decision module then received the results of a new round of inference from the scene analysis module in a very short time: the first classifier detected abnormally bright pixels in the short-circuited device previously marked as an object area, and the motion vector direction of the human body area showed a sudden divergence pattern; the second classifier updated the scene label signal to an indoor scene and abnormal pursuit. In the video dimension, the module improves encoding accuracy according to the anomaly handling strategy, temporarily adjusting the foreground weight from 0.6 to 0.75. This significantly increases the importance score of all coding tree units containing human bodies and abnormal light sources, extending the quantization parameter offset negatively to near the maximum quality offset of -6. The background region quantization parameters are correspondingly positively compensated to maintain the overall bitrate within the preset upper limit. The frame rate remains unchanged at 25 frames per second. The audio target configuration corresponding to the anomaly chasing scene label is as follows: advanced audio encoding format, audio bitrate of 256 kilobits per second, sampling rate of 44,100 Hz, and noise reduction processing disabled to preserve the complete ambient sound as the acoustic basis for subsequent accident analysis. The module generates the corresponding audio encoding parameter signal. After receiving the keyframe trigger signal, the audio and video encoding module immediately performs forced intra-frame prediction frame encoding in the current frame where the explosion sound and sparks appear, ensuring that the scene of the accident moment is independently and completely preserved without relying on previous frames. Immediately afterwards, the video encoder encodes multiple subsequent frames continuously with the highest quality quantization parameter offset map settings, recording the entire process of the experimenter falling and the ladder tipping over in high definition. The audio encoder simultaneously increased the bitrate and disabled noise reduction, fully capturing the acoustic characteristics of the short-circuit sound, the metallic impact sound, and the screams of people. The incident lasted approximately fifteen seconds, during which time the system operated at full quality. After fifteen seconds, the scene returned to normal office conditions, and system parameters gradually reverted to the appropriate levels. In post-incident analysis, keyframes from the moment of the incident and subsequent high-definition frame sequences provided the security team with clear video and audio evidence, including the specific color and shape of the electric arc, and the direction and speed at which the ladder fell, all of which could be reconstructed from the recordings.

[0048] At 11 PM, the last employee left the lab and turned off the lights. The scene analysis module, after inference, output a tag signal that returned to a static, unattended state. The encoding parameter decision module then restarted its timing logic, reducing the frame rate to five frames per second. The quantization parameter offset map recovered to a predominantly positive offset across the entire frame range. After a brief transition, the audio parameters were reduced to 64 kilobits per second pulse code modulation, and noise reduction was disabled. The system re-entered a low-bandwidth, low-storage-consumption nighttime monitoring mode.

[0049] As can be seen from the above-described full-time operation process, the system in this embodiment can dynamically and automatically adjust the quantization parameter distribution and encoding frame rate of video encoding, and the bit rate, sampling rate, and noise reduction strategy of audio encoding according to the real-time semantic changes of the monitored scene, and accurately trigger keyframe encoding at the moment of sudden scene changes. From extremely low bitrate monitoring in the early morning, to the quality and bandwidth balance configuration during normal office work and group discussions during the day, to full-quality emergency recording during emergencies, and finally back to the low-power mode at night, the entire closed-loop control requires no manual intervention, achieving efficient utilization of bitrate resources and storage space, while ensuring the integrity and traceability of key event recordings.

[0050] This invention provides an audio and video acquisition adaptive control and monitoring system based on scene recognition. It performs scene semantic segmentation and classification on monitored video frames through a dual-stream network, generating scene semantic segmentation maps and scene labels. Simultaneously, it performs joint decision-making for audio abrupt changes and video scene switching, forming keyframe trigger signals. An encoding parameter decision module calculates the importance score of each encoding unit based on the scene semantic segmentation map and maps it to a quantization parameter offset map. It also generates frame rate control signals and audio encoding configuration signals based on the scene labels. The audio and video encoding module adjusts the quantization parameters, frame rate, and audio configuration according to the aforementioned signals, and encodes keyframes when a keyframe trigger signal is detected, enabling dynamic matching of the encoding strategy with the scene content.

[0051] Therefore, the scene recognition-based audio and video acquisition adaptive control and monitoring system of the present invention solves the problem that existing monitoring systems cannot adaptively adjust audio and video acquisition and encoding parameters according to scene semantics, resulting in wasted bitrate or loss of key information.

[0052] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A scene-recognition-based audio and video acquisition adaptive control and monitoring system, characterized in that, include: The audio and video acquisition module acquires video and audio frame sequence signals of the monitored scene in real time, and transmits the video frame sequence signals to the scene analysis module and the joint triggering module, and the audio frame sequence signals to the joint triggering module. The scene analysis module receives video frame sequence signals, performs scene semantic segmentation and classification through a dual-stream network, and generates scene semantic segmentation map signals and scene label signals, which are then transmitted to the encoding parameter decision module. The joint triggering module receives video and audio frame sequence signals, performs a joint decision on audio sudden changes and video scene switching, forms a key frame trigger signal, and transmits it to the audio and video encoding module. The encoding parameter decision module receives the scene semantic segmentation map and scene label signal, generates video and audio encoding parameter signals (video content quantization parameter offset map and frame rate control signal) based on semantic importance mapping, and transmits them to the audio and video encoding module; The audio and video encoding module receives various signals, adjusts the quantization parameters and frame rate according to the video encoding parameters, adjusts the audio configuration according to the audio encoding parameters, encodes key frames when a key frame trigger signal is detected, and outputs the encoded audio and video bitstream.

2. The system according to claim 1, characterized in that, The dual-stream network in the scene analysis module consists of a spatial stream branch and a temporal stream branch. The spatial stream branch uses a residual network backbone to extract spatial semantic feature maps from the input single-frame video frame sequence signal. The temporal stream branch calculates a dense optical flow field for the continuous video frame sequence signal and extracts temporal motion feature maps through a three-dimensional convolutional layer. The spatial semantic feature map and the temporal motion feature map are concatenated in the channel dimension and then input into the hollow spatial pyramid pooling module to generate multi-scale context fusion features. The multi-scale context fusion features are processed by a first classifier to output a scene semantic segmentation map signal. This scene semantic segmentation map signal assigns a scene category label to each pixel. At the same time, the multi-scale context fusion features are processed by a second classifier to output a scene label signal. The scene label signal includes a coarse-grained scene type and a fine-grained activity category. The coarse-grained scene type is an indoor scene or an outdoor scene, and the fine-grained activity category is a static unmanned scene, a normal office scene, a crowd gathering scene, or an abnormal chase scene.

3. The system according to claim 1, characterized in that, The scene analysis module employs a time-interleaved processing strategy for the video frame sequence signal. The module performs a complete dual-stream network inference once every preset number of video frames, where the preset number is an integer greater than one. For the intermediate video frame between two consecutive complete inferences, the scene analysis module performs linear interpolation based on the scene semantic segmentation map signals output from the two most recent complete inferences to generate the scene semantic segmentation map signal corresponding to the intermediate video frame. Simultaneously, the scene label signal output from the most recent complete inference remains unchanged until the next complete inference updates the scene label signal. The linear interpolation calculation formula is to weight and sum the category confidence vectors of corresponding pixels in the two scene semantic segmentation maps according to the timestamp ratio of the intermediate frame.

4. The system according to claim 1, characterized in that, The joint triggering module includes an audio mutation detection unit, a video scene switching detection unit, and a joint decision unit. The audio mutation detection unit receives an audio frame sequence signal, extracts Mel-frequency cepstral coefficient features frame by frame, and calculates the cosine distance between the Mel-frequency cepstral coefficient feature vectors of adjacent audio frames. When the cosine distance exceeds a preset audio mutation threshold, the audio mutation detection unit generates an audio mutation event marker signal and transmits it to the joint decision unit. The video scene switching detection unit receives a video frame sequence signal, converts each video frame into a thumbnail, extracts color histogram features, calculates the chi-square distance between the color histograms of adjacent video frames, and maintains a sliding window recording the chi-square distance sequence of the most recent frames. The video scene switching detection unit calculates the mean and standard deviation of the chi-square distance based on this sequence, and uses the mean plus three times the standard deviation as an adaptive video switching threshold. When the chi-square distance of the current video frame exceeds the adaptive video switching threshold, a video scene switching event marker signal is generated and transmitted to the joint decision unit. The calculation formula for the adaptive video switching threshold is as follows: in, To adapt to video switching thresholds, The mean chi-square distance within the sliding window. Let N be the standard deviation of the chi-square distance within the sliding window, and N be the number of frames in the sliding window. Let be the chi-square distance of the i-th frame. The current video frame rate, This represents the maximum video frame rate.

5. The system according to claim 4, characterized in that, The joint decision unit receives the audio mutation event marker signal and the video scene switching event marker signal, and executes joint decision logic to form the keyframe trigger signal. The joint decision logic is defined as follows: the audio mutation event marker signal and the video scene switching event marker signal are respectively assigned preset audio weight coefficients and preset video weight coefficients, and then weighted and summed to obtain a joint change score. When the joint change score is greater than or equal to a preset trigger threshold, the joint decision unit generates a valid keyframe trigger signal. The preset video weight coefficient is greater than the preset audio weight coefficient, and the sum of the two is one. When the audio and video encoding module detects the valid keyframe trigger signal, it forcibly encodes the current encoded frame as an intra-frame predicted keyframe and resets the image group counter. The calculation formula for the joint change score is as follows: in, The score is based on the combined audio and video changes. For audio weighting coefficients, Label signals for audio mutation events. For video weighting coefficients, The signal is used to mark video scene transition events, where e is the natural constant. This is an adaptive video switching threshold.

6. The system according to claim 5, characterized in that, The encoding parameter decision module receives the scene semantic segmentation map signal and scene label signal transmitted by the scene analysis module, and performs quantization parameter offset calculation at the encoding tree unit level based on the scene semantic segmentation map signal. The specific process of the quantization parameter offset calculation is as follows: For each encoding tree unit in the current video frame, calculate the proportion of foreground object pixels in the total number of pixels in the encoding tree unit, and calculate the normalized value of the motion vector amplitude in the encoding tree unit. The normalized value of the motion vector amplitude is obtained by dividing the motion vector amplitude by the maximum value of the motion vector amplitude in all encoding tree units in the current frame. Multiply the foreground pixel proportion and the normalized value of the motion vector amplitude by a preset foreground weight and a preset motion weight, respectively, and then sum them to obtain the scene importance score of the encoding tree unit. The calculation formula of the scene importance score is as follows: in, The scene importance score for the coding tree unit. Forward weighting, Foreground pixel ratio For motion weights, This is the normalized value of the motion vector amplitude. The score is calculated based on the combined changes in audio and video.

7. The system according to claim 6, characterized in that, The encoding parameter decision module linearly maps the quantization parameter offset of each encoding tree unit within a preset quantization parameter offset range based on the scene importance score of each unit. The quantization parameter offset corresponding to the encoding tree unit with a higher scene importance score is negative to obtain finer quantization, while the quantization parameter offset corresponding to the encoding tree unit with a lower scene importance score is positive to perform coarser quantization. The encoding parameter decision module adds the basic quantization parameters of the current frame to the quantization parameter offset of each encoding tree unit to obtain the final quantization parameters of each encoding tree unit. The basic quantization parameters are determined by the target bitrate through the bitrate and Lagrange multiplier model. The encoding parameter decision module transmits the quantization parameter offset map signal containing the final quantization parameters of each encoding tree unit as part of the video encoding parameter signal to the audio and video encoding module.

8. The system according to claim 7, characterized in that, The encoding parameter decision module also receives the scene tag signal transmitted by the scene analysis module, and generates a frame rate control signal and a resolution control signal based on the scene tag signal. When the scene tag signal indicates a static unmanned scene and continues to exceed a preset static time threshold, the encoding parameter decision module generates a frame rate decrease control signal to gradually reduce the acquisition frame rate from the basic frame rate to the minimum frame rate. When the scene tag signal switches from a static unmanned scene to other active scenes, the encoding parameter decision module immediately generates a frame rate recovery control signal to restore the acquisition frame rate to the basic frame rate. When the proportion of foreground pixels in the scene indicated by the scene label signal is continuously lower than the first preset proportion threshold, the encoding parameter decision module generates a resolution downgrade control signal to reduce the output resolution by one level. When the proportion of foreground pixels recovers to a level higher than the second preset proportion threshold and continues to exceed the preset duration threshold, a resolution recovery control signal is generated to increase the output resolution by one level. The frame rate control signal and the resolution control signal are both transmitted to the audio and video encoding module as part of the video encoding parameter signal.

9. The system according to claim 1, characterized in that, The encoding parameter decision module generates audio encoding parameter signals based on the scene label signals. Internally, the module maintains a correspondence table between scene types and audio encoding configurations. In this table, static unmanned scenes correspond to a first audio bitrate, a first sampling rate, and a configuration with noise reduction disabled; normal office scenes correspond to a second audio bitrate, a second sampling rate, and a configuration with mild noise reduction; crowd-gathering scenes correspond to a third audio bitrate, a third sampling rate, and a configuration with moderate noise reduction; and abnormal chasing scenes correspond to a fourth audio bitrate, a fourth sampling rate, and a configuration with noise reduction disabled. The first audio bitrate is lower than the second audio bitrate, the second audio bitrate is lower than the third audio bitrate, and the third audio bitrate is lower than the fourth audio bitrate. The encoding parameter decision module queries the correspondence table based on the currently received scene label signals to determine the target audio encoding parameter signal and transmits it to the audio / video encoding module. When the audio encoding parameter signal changes, the encoding parameter decision module generates a smooth transition signal to control the audio / video encoding module to gradually change the current audio encoding parameters from their original values ​​to the target values ​​within a preset transition time window.

10. The system according to claim 1, characterized in that, The system also includes a transmission and feedback module, which is connected to the output of the audio and video encoding module. This module receives the encoded audio and video streams and transmits them to the receiving end via a communication network. Simultaneously, the transmission and feedback module monitors the transmission status of the communication network in real time. This transmission status includes at least three network status parameters: available bandwidth, packet loss rate, and round-trip time. The transmission and feedback module assembles these network status parameters into a network status vector signal and transmits this signal to the encoding parameter decision module via a feedback loop. Upon receiving the network status vector signal, the encoding parameter decision module adjusts the basic quantization parameter in the video encoding parameters based on the available bandwidth and packet loss rate. When the available bandwidth decreases or the packet loss rate increases, the encoding parameter decision module increases the basic quantization parameter to reduce the output bitrate. When the available bandwidth recovers or the packet loss rate decreases, the basic quantization parameter decreases to improve encoding quality.

Images