Ultrahigh-definition video conference system based on AI and intelligent sensor

Abstract

The invention discloses an ultra-high-definition video conference system based on AI and an intelligent sensor, relates to the technical field of communication and video processing, and can acquire multi-dimensional sensing information such as sound pressure change, illumination intensity fluctuation and image structure disturbance in a conference environment in real time by introducing a multi-source sensing parameter acquisition mechanism. Compared with an existing static configuration type conference system, the conference system has higher environment state recognition capability and parameter response basis. By constructing an environment perception dynamic index, the system can dynamically quantify the disturbance degree of three channels of a current acousto-optic image, a high-precision measurable input basis is provided for subsequent audio and video adjustment, and the problems that an existing system is lagged in response to environment change and fixed in strategy are effectively solved. An environment perception dynamic index is generated through nonlinear fusion modeling, so that various disturbance factors participate in environment state modeling under the same framework, and the problems of strong and weak imbalance and dimension offset easily occurring in a conventional linear superposition mode are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication and video processing, in particular to an ultra-high-definition video conference system based on AI and intelligent sensors. BACKGROUND

[0002] With the diversification of conference room scenes, the uncertainty of lighting conditions and the dynamic change of user positions, it is difficult to rely only on camera acquisition and microphone pickup to meet the needs of high-quality and low-delay remote interaction. Therefore, the system-level introduction of multi-source information fusion sensing mechanisms such as sound pressure sensors, ambient light sensors, and image sensors provides support for real-time regulation of image sharpness, audio filtering strength, and video output frame rate, and becomes a key technology direction to improve the experience of remote conference interaction.

[0003] In the current common remote video conference system, the audio and video acquisition end usually runs with default settings, such as fixed frame rate, fixed image sharpening intensity, and static threshold-based separation processing of human voice and environmental noise. This approach ignores key influencing factors such as changes in external meeting environment light intensity, acoustic background noise level fluctuations, and user activity in the picture, resulting in the system being unable to self-adjust parameters according to actual conditions.

[0004] The root cause of the above problem lies in the fact that the existing video conference system lacks the ability to model the state of the use environment, and its audio and video processing process is usually based on a set of initial configuration parameters, without sensing, feedback, or adjustment, ignoring the influence of real-time sensor data and environmental changes on audio and video signal quality. For example, when the meeting is in progress, the outdoor light gradually becomes dim or the user's position on the screen shifts, the image sensor cannot sense and trigger brightness correction, resulting in a decrease in picture clarity; when people around move or equipment operation produces sudden noise, the sound pressure level suddenly rises, and the system does not have an acoustic disturbance recognition mechanism, which easily causes confusion between the speech channel and the background sound, affecting speech transparency. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides an ultra-high-definition video conference system based on AI and intelligent sensors, which solves the problems mentioned in the background art.

[0006] To achieve the above purpose, the present application is implemented by the following technical scheme: an ultra-high-definition video conference system based on AI and intelligent sensors, comprising a multi-source perception parameter acquisition module, an environmental dynamic perception modeling module, an audio intelligent noise reduction regulation module, a video clarity and brightness adaptive regulation module, an audio and video synchronization quality optimization module, and a system self-feedback and enhancement strategy adjustment module. The multi-source perception parameter acquisition module acquires sensing data in the conference space through sensors and performs preprocessing to obtain a sensing data set CW. The environmental dynamic perception modeling module constructs a composite index reflecting the degree of dynamic change of the environment based on the sensing data set CW, and obtains an environmental perception dynamic index Eenv; The audio intelligent noise reduction control module calculates an audio clarity adjustment coefficient Aa based on the acoustic part of the environmental perception dynamic index Eenv; The video clarity and brightness adaptive control module calculates an image enhancement gain factor Lv based on the ambient illuminance in the sensing data set CW, and dynamically adjusts the image enhancement strategy; The audio-video synchronization quality tuning module calculates a synchronization correction index Sync based on the audio clarity adjustment coefficient Aa and the image enhancement gain factor Lv, and judges whether the audio-video parameters are mismatched; The system self-feedback and enhancement strategy adjustment module calculates a final evaluation index Ψtotal based on the obtained environmental perception dynamic index Eenv and the synchronization correction index Sync, and dynamically adjusts the strategy.

[0007] Preferably, the multi-source perception parameter acquisition module includes a perception signal acquisition unit and a heterogeneous data normalization preprocessing unit; The perception signal acquisition unit synchronously acquires multi-source physical signals of the conference space from the laid acoustic pressure sensors, illumination sensors and image sensors, including instantaneous sound pressure value Pw, spatial average illuminance Lavg, continuous frame-to-frame gray difference CG and edge fluctuation rate Ew, and fits them into an original data set YW; The instantaneous sound pressure value Pw is acquired by the acoustic pressure sensor; The spatial average illuminance Lavg is acquired by the illumination sensor; The continuous frame-to-frame gray difference CG and the edge fluctuation rate Ew are acquired by the image sensor; The continuous frame-to-frame gray difference CG is acquired in the following way: first, the image sensor (such as a CMOS camera) continuously acquires video frames to obtain two consecutive frames of gray images, then calculates the pixel difference and averages to obtain the continuous frame-to-frame gray difference CG; The edge fluctuation rate Ew is acquired in the following way: edge detection is applied to the two consecutive frames of images to obtain edge images, then the number of edge pixels is counted, and finally the edge fluctuation rate Ew is calculated and obtained; The heterogeneous data normalization preprocessing unit synchronously cleanses and normalizes different physical dimensions in the original data set YW, eliminates noise points, abrupt values and cross-modal scale differences, and obtains a sensing data set CW; The cleansing includes abnormal point cleansing; the dynamic median offset method is used to process the data in the original data set YW to eliminate noise points and abrupt values; The processing method is: at the current time point, any parameter is taken out from the original data set YW, and the selected historical time point and future time point are selected in front and back respectively, to form a data window sequence containing multiple time points; in the data window, all values of the same parameter at the time point are extracted, and the median of these values is calculated; then the original value at the current time is subtracted from the median to obtain the offset difference value, which is used as the final cleaning result; The formula of the normalization processing is as follows: ; In the formula, CWo represents the oth data in the sensing data set CW, YWo represents the oth data in the original data set YW, minYWo represents the valley value of the oth data in the original data set YW, and maxYWo represents the peak value of the oth data in the original data set YW.

[0008] Preferably, the environmental dynamic perception modeling module comprises a dimensional disturbance deconstruction unit and a multi-parameter mapping fusion modeling unit. The dimensional disturbance deconstruction unit extracts features of parameters in the sensing data set CW, including sound pressure change rate Dpw, average illumination change rate Davg, gray scale change rate Dcg and fluctuation change rate Dew, and fits them into a local disturbance data set DF. The sound pressure change rate Dpw is obtained by taking the absolute value of the difference between the instantaneous sound pressure value Pw(t) at time t and the instantaneous sound pressure value Pw(t-1) at time t-1 divided by the time interval Δt. The average illumination change rate Davg is obtained by taking the absolute value of the difference between the spatial average illumination Lavg(t) at time t and the spatial average illumination Lavg(t-1) at time t-1 divided by the time interval Δt. The gray scale change rate Dcg is obtained by taking the absolute value of the difference between the continuous frame gray scale difference CG(t) at time t and the continuous frame gray scale difference CG(t-1) at time t-1 divided by the time interval Δt. The fluctuation change rate Dew is obtained by taking the absolute value of the difference between the edge fluctuation rate Ew(t) at time t and the edge fluctuation rate Ew(t-1) at time t-1 divided by the time interval Δt.

[0009] Preferably, the multi-parameter mapping fusion modeling unit maps the data in the local disturbance data set DF into a unified disturbance response function, then performs nonlinear integration modeling to obtain the environmental perception dynamic index Eenv. The environmental perception dynamic index Eenv is obtained by the following formula: ; In the formula, Eenv(t) represents the environmental perception dynamic index at time t, v represents the disturbance sensitivity coefficient, DFi represents the i-th data in the local disturbance dataset DF, and e represents a constant; The environmental state of the video is determined by analyzing the environmental perception dynamic index Eenv. Environment status is obtained through matching in the following ways: When 0.75 ≤ the environmental perception dynamic index Eenv, it indicates that the environment is in a state of high dynamics and strong interference; triggering strong noise reduction and image sharpening enhancement. When 0.4 ≤ Environmental Awareness Dynamics Index Eenv < 0.7, it indicates that the environment is under moderate disturbance; maintain the current processing parameters and sample periodically. When the environmental awareness dynamic index Eenv < 0.4, it indicates that the environment is in a stable state; switch to low power mode and minimize processing intervention.

[0010] Preferably, the audio intelligent noise reduction and control module includes an acoustic disturbance extraction unit and a noise reduction control response modeling unit; The acoustic disturbance extraction unit extracts features from the acoustic component of the environmental perception dynamic index Eenv, extracts the sound pressure change rate Dpw, and calculates the acoustic disturbance sub-index Esound. The acoustic perturbation index Esound is obtained using the following formula: ; In the formula, Esound represents the acoustic perturbation index at time t, e represents a constant, vs represents the sound pressure perturbation response coefficient, and Dpw(t) represents the sound pressure change rate at time t.

[0011] Preferably, the noise reduction control response modeling unit constructs an initial audio clarity coefficient cAa based on the acquired acoustic disturbance sub-index Esound and combined with historical stable states; The initial audio clarity coefficient cAa is obtained using the following formula: ; In the formula, cAa(t) represents the initial audio clarity coefficient at time t, exp represents the exponential function, Ts represents the preset acoustic disturbance recognition threshold, and ks represents the adjustment slope control parameter. The initial audio clarity coefficient cAa is corrected to obtain the trend correction factor Hcorr, and the inertial effect caused by sound disturbance changes is removed to obtain the audio clarity adjustment coefficient Aa. The trend correction factor Hcorr is obtained as follows: within a fixed time window Th, the initial audio clarity coefficient cAa from time t-Th+1 to t is extracted; then all the initial audio clarity coefficients cAa are summed and divided by the time window Th to obtain the average value, which is used as the trend correction factor Hcorr. The audio clarity adjustment factor Aa is obtained by summing the initial audio clarity factor cAa and the trend correction factor Hcorr and then dividing by two. The ambient noise level is determined by analyzing the audio clarity adjustment coefficient Aa. The judgment method is as follows: When 0.7 ≤ audio clarity adjustment coefficient Aa < 1, it indicates that the current acoustic environment is quiet, and the filtering intensity is automatically reduced to retain more speech details; When the audio clarity adjustment coefficient Aa < 0.7, it indicates that strong environmental noise has been detected, and the filtering intensity is automatically increased to suppress background noise interference.

[0012] Preferably, the video clarity and brightness adaptive adjustment module includes a multi-parameter image perturbation sensing unit and an image enhancement gain modeling unit; The multi-parameter image perturbation sensing unit analyzes the spatial average illuminance Lavg in the sensing dataset CW and calculates the luminance missing index Iin. The luminance deficiency index Iin is obtained by the difference between 1 and the spatial average illuminance Lavg; By combining the analysis of the continuous inter-frame grayscale difference CG and the edge fluctuation rate Ew in the sensor dataset CW, the image perturbation intensity index Gsta is calculated and obtained. The image perturbation intensity index Gsta is obtained by taking the square root of the sum of the square of the grayscale difference CG between consecutive frames and the square of the edge volatility Ew.

[0013] The higher the image perturbation intensity index Gsta, the more unstable the image is, making it suitable for enhancing sharpness.

[0014] Preferably, the image enhancement gain modeling unit performs nonlinear fusion of the acquired brightness loss index Iin and the image perturbation intensity index Gsta to calculate the image enhancement gain factor Lv. The image enhancement gain factor Lv is obtained using the following formula: ; In the formula, Iin(t) represents the brightness loss index at time t, ln represents the logarithmic function, Gsta(t) represents the image perturbation intensity index at time t, and vo represents the brightness compensation index adjustment factor. By analyzing the image enhancement gain factor Lv and judging the image, control strategy suggestions are obtained. The determination method is obtained by matching in the following way: When the image enhancement gain factor Lv≥2.0, the image is too dark or has strong fluctuations. The recommended strategy is to increase both brightness and sharpness. When 2.0 > image enhancement gain factor Lv ≥ 1.3, the image is moderately disturbed or slightly dark. The recommended strategy is to moderately improve contrast and edge sharpness. When 1.3 > image enhancement gain factor Lv, the image is considered to be in a normal environment. The recommended strategy is to minimize image processing and preserve the natural state.

[0015] Preferably, the audio and video synchronization quality optimization module normalizes the acquired audio clarity adjustment coefficient Aa and image enhancement gain factor Lv to unify the dimensions of the data and calculates the synchronization correction index Sync. The synchronization correction index Sync is obtained using the following formula: ; In the formula, Aa(t) represents the audio clarity adjustment coefficient at time t, Sync represents the synchronization correction exponent at time t, log represents the logarithmic function, and ec represents a non-zero constant; The obtained synchronization correction index Sync is compared with the preset correction threshold Tync to determine whether the audio and video parameters are mismatched. The judgment method is as follows: When the synchronization correction index Sync < correction threshold Tync, it indicates that synchronization is normal; maintain the current audio and video frame rate and buffer settings. When the correction threshold Tync ≤ synchronization correction index Sync < 1.5 × correction threshold Tync, it indicates a moderate deviation; add 2 audio buffer frames and 1 video interpolation frame. A mismatch is indicated when 1.5 × Tync (the correction threshold) is less than the Sync (the synchronization correction index). Audio insertion is dynamically delayed by 128ms, and video frame rate is reduced to keyframes with dynamic gamma compensation enabled.

[0016] Preferably, the system self-feedback and enhancement strategy adjustment module normalizes the acquired environmental perception dynamic index Eenv and combines it with the synchronization correction index Sync to calculate the final evaluation index Ψtotal. The final evaluation metric Ψtotal is obtained using the following formula: ; In the formula, Ψtotal(t) represents the final evaluation index at time t; Analyze the current value and trend sequence of the final evaluation indicator Ψtotal, and execute the corresponding enhancement strategy switching operation; The analysis method is as follows: When the final evaluation metric Ψtotal ≥ 0.75, it indicates a high dynamic disturbance scene. The strategy is adjusted as follows: enable enhancement mode; increase the audio filter level by one level; and increase the weighted image enhancement kernel by 20%. When 0.75 > the final evaluation index Ψtotal ≥ 0.45, it indicates a moderate disturbance and a critical state. The strategy is adjusted as follows: maintain the strategy unchanged and set the current state as the observation state. When 0.45 > the final evaluation metric Ψtotal, it indicates a stable and low-interference state. The adjustment strategy is to enable power-saving mode, reduce filtering intensity, and switch image processing to a low-power strategy.

[0017] This invention provides an ultra-high-definition video conferencing system based on AI and intelligent sensors, which has the following advantages: (1) During system operation, a multi-source information acquisition structure covering three dimensions—sound pressure, illuminance, and image—was constructed by setting up a sensing signal acquisition unit and a heterogeneous data normalization preprocessing unit. Compared with the traditional method that relies solely on single-channel sampling of images or microphones, this scheme can achieve synchronous acquisition in time and multi-point coverage in space, comprehensively reflecting the real-time audio-visual image status of the conference space. In particular, the two parameters extracted by the image sensor—the continuous inter-frame grayscale difference and the edge fluctuation rate—can effectively characterize the image disturbance trend in the context of frequent dynamics in the current conference scenario, providing preliminary support for subsequent image enhancement strategies.

[0018] (2) An environmental perception dynamic index is generated through nonlinear fusion modeling, enabling various disturbance factors to participate in environmental state modeling within the same framework. This avoids the imbalance of strength and dimension bias that easily occurs in conventional linear superposition methods. The index has three characteristics: continuity, sensitivity, and segmentable matching. It can accurately characterize the state classification of the current meeting space in a high-dynamic disturbance, medium-fluctuation, or stable environment, providing a clear and physically meaningful basis for judgment in subsequent modules.

[0019] (3) By refining the audio intelligent noise reduction and control module into an acoustic disturbance extraction unit and a noise reduction control response modeling unit, fine-grained modeling and dynamic control of noise change trends are achieved, breaking through the static judgment mode of instantaneous sound pressure value in traditional audio processing schemes. The system not only calculates the sound pressure change rate, but also maps it to an acoustic disturbance sub-index, thereby accurately reflecting the dynamic fluctuations of background noise in the conference space. At the same time, in order to improve the stability and continuity of the noise reduction response, after generating the initial audio clarity coefficient, the system further introduces a trend correction mechanism. By extracting historical clarity data within a fixed time window to construct a correction factor, the final output adjustment coefficient has both the characteristics of current disturbance reflection and historical state continuation.

[0020] (4) By introducing an image enhancement gain modeling unit, a nonlinear fusion of the brightness loss index and the image disturbance intensity index was achieved for the first time in a conference video processing scenario, constructing a dynamically adjustable image enhancement gain factor. This approach breaks through the traditional image processing technology's reliance on a single brightness threshold or edge contrast enhancement factor, enabling the system to possess comprehensive perception and hierarchical control capabilities for multi-source visual disturbance signals. The system can not only adjust the image clarity and sharpness based on the image brightness but also on the edge structure stability, effectively adapting to the video presentation needs of various complex scenarios such as low illumination, high dynamic range, and frequent switching, thereby improving the stability of image expression and structural recognition capabilities. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the block flow of an ultra-high-definition video conferencing system based on AI and intelligent sensors according to the present invention. Figure 2 This is a schematic diagram illustrating the process of obtaining the final evaluation indicators of the present invention; Figure 3 This is a trend chart of the final evaluation index of the present invention. Detailed Implementation

[0022] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Example 1

[0023] This invention provides an ultra-high-definition video conferencing system based on AI and intelligent sensors. Please refer to [link / reference]. Figures 1 to 3 It includes a multi-source sensing parameter acquisition module, an environmental dynamic sensing modeling module, an audio intelligent noise reduction and control module, a video clarity and brightness adaptive control module, an audio and video synchronization quality optimization module, and a system self-feedback and enhancement strategy adjustment module. The multi-source sensing parameter acquisition module collects sensor data in the conference space through sensors, performs preprocessing, and obtains the sensor dataset CW; The environmental dynamic perception modeling module constructs a composite index reflecting the degree of dynamic change in the environment based on the sensor dataset CW, and obtains the environmental perception dynamic index Eenv. The audio intelligent noise reduction and control module calculates and obtains the audio clarity adjustment coefficient Aa based on the acoustic part of the environmental perception dynamic index Eenv. The video clarity and brightness adaptive adjustment module calculates and obtains the image enhancement gain factor Lv based on the ambient illuminance in the sensor dataset CW, and dynamically adjusts the image enhancement strategy. The audio and video synchronization quality optimization module calculates and obtains the synchronization correction index Sync based on the audio clarity adjustment coefficient Aa and the image enhancement gain factor Lv, and determines whether the audio and video parameters are mismatched. The system self-feedback and enhancement strategy adjustment module calculates the final evaluation index Ψtotal based on the acquired environmental perception dynamic index Eenv and synchronization correction index Sync, and performs dynamic strategy self-adjustment.

[0024] In this embodiment, by introducing a multi-source sensing parameter acquisition mechanism, it can collect multi-dimensional sensing information such as sound pressure changes, light intensity fluctuations, and image structure disturbances in the conference environment in real time. Compared with existing static configuration conference systems, it has stronger environmental state recognition capabilities and parameter response foundation. By constructing an environmental perception dynamic index, the system can dynamically quantify the degree of disturbance in the current audio, light, and image channels, providing a high-precision and measurable input basis for subsequent audio and video adjustments, effectively making up for the problems of lagging response to environmental changes and fixed strategies in existing systems.

[0025] In audio processing, instead of using static thresholding for noise reduction, a sharpness adjustment parameter is constructed based on acoustic disturbance states. This enables real-time identification and matching of noise reduction strategies for sudden noise events, background interference, and voice-dominated scenarios, maintaining a relatively stable and clear voice channel. In image processing, the system can identify low-light conditions, dynamic changes, and structural shifts, and outputs an image enhancement gain factor accordingly to drive brightness compensation and sharpness enhancement operations, avoiding problems such as blurring and structural distortion in dark scenes.

[0026] By introducing an audio-visual synchronization quality optimization mechanism, the system can identify channel processing asynchrony based on the matching status between current audio and image processing parameters. It then reconciles these asynchrony through automatic adjustments to frame rate and latency buffering, ensuring consistent output across the audio-visual channels and eliminating issues such as audio-visual misalignment and audio-visual asynchrony. Finally, the system's self-feedback and enhancement strategy adjustment module aggregates and evaluates multi-parameter state trends, automatically deciding whether to enter high-enhancement mode, energy-saving mode, or observation mode, thus constructing a complete closed-loop control path for audio-visual processing. This system's overall structure boasts advantages such as strong dynamic adaptability, high audio-visual coordination, and robustness to environmental changes, making it particularly suitable for practical applications with stringent audio-visual quality requirements and significant environmental fluctuations in meeting scenarios. Example 2

[0027] This embodiment is an explanation based on Embodiment 1. Please refer to it. Figure 1Specifically: the multi-source sensing parameter acquisition module includes a sensing signal acquisition unit and a heterogeneous data normalization preprocessing unit; The sensing signal acquisition unit synchronously collects multi-source physical signals of the conference space from the deployed sound pressure sensor, light sensor and image sensor, including instantaneous sound pressure value Pw, spatial average illuminance Lavg, continuous frame gray level difference CG and edge fluctuation rate Ew, and fits them into the original dataset YW. The instantaneous sound pressure value Pw is acquired by a sound pressure sensor; Spatial average illuminance Lavg is acquired through a light sensor; The continuous inter-frame grayscale difference CG and edge fluctuation rate Ew are acquired by the image sensor; The heterogeneous data normalization preprocessing unit performs simultaneous cleaning and normalization on different physical dimensions in the original dataset YW to eliminate noise, abrupt values ​​and cross-modal scale differences, and obtain the sensor dataset CW. Cleaning includes outlier removal; noise and abrupt changes in the original dataset YW are eliminated by processing the data using dynamic median offset. The processing method is as follows: at the current time point, take any parameter from the original dataset YW, and then select historical time points and future time points forward and backward to form a data window sequence containing multiple time points; in the data window, extract all values ​​of the same parameter at the time point and calculate the median of these values; then subtract the median from the original value at the current time to obtain the offset difference, which is used as the final cleaning result. The formula for normalization is as follows: ; In the formula, CWo represents the o-th data in the sensor dataset CW, YWo represents the o-th data in the original dataset YW, minYWo represents the valley value of the o-th data in the original dataset YW, and maxYWo represents the peak value of the o-th data in the original dataset YW.

[0028] In this embodiment, by setting up a sensing signal acquisition unit and a heterogeneous data normalization preprocessing unit, a multi-source information acquisition structure covering three dimensions—sound pressure, illuminance, and image—is constructed. Compared to traditional methods that rely solely on single-channel sampling of images or microphones, this scheme can achieve synchronous acquisition in time and multi-point coverage in space, comprehensively reflecting the real-time audio-visual image status of the conference space. In particular, the two parameters extracted by the image sensor—continuous inter-frame grayscale difference and edge fluctuation rate—can effectively characterize the trend of image disturbance in the context of frequent dynamics in the current conference scenario, providing preliminary support for subsequent image enhancement strategies.

[0029] This embodiment introduces a dynamic median offset method to clean up outliers in the original data. Without relying on assumptions about parametric statistical distribution, it effectively removes transient outliers, enhancing the continuity and reliability of the data. Based on a sliding time window, this method constructs an offset using the median as a benchmark, thereby preserving environmental trend characteristics while suppressing the influence of isolated peaks, thus improving resilience to abnormal states.

[0030] A unified normalization mechanism is employed to standardize and transform data from different channels, enabling sound pressure levels, illuminance levels, and image perturbations to participate in subsequent calculations at the same scale. This effectively supports the unified construction of environmental dynamic indices and cross-modal fusion modeling. Through the aforementioned cleaning and normalization process, the system ultimately generates a sensor dataset with a unified structure, smoothed noise, and comparable values. This provides a high-quality input foundation for the adaptive control algorithms of subsequent modules, significantly improving the system's intelligent processing stability and environmental adaptability in complex conference environments. Example 3

[0031] This embodiment is an explanation based on Embodiment 2. Please refer to it. Figure 1 and 2 Specifically: the environmental dynamic perception modeling module includes a dimensional perturbation deconstruction unit and a multi-parameter mapping fusion modeling unit; The dimensional perturbation deconstruction unit extracts features from the parameters in the sensing dataset CW, including the sound pressure change rate Dpw, the illuminance change rate Davg, the grayscale change rate Dcg, and the fluctuation change rate Dew, and fits them into a local perturbation dataset DF. The rate of change of sound pressure, Dpw, is obtained by dividing the difference between the instantaneous sound pressure value Pw(t) at time t and the instantaneous sound pressure value Pw(t-1) at time t-1 by the time interval Δt, and then taking the absolute value. The rate of change of illuminance Davg is obtained by dividing the difference between the spatial average illuminance Lavg(t) at time t and the spatial average illuminance Lavg(t-1) at time t-1 by the time interval Δt and then taking the absolute value. The grayscale change rate Dcg is obtained by dividing the difference between the grayscale difference CG(t) between consecutive frames at time t and the grayscale difference CG(t-1) between consecutive frames at time t-1 by the time interval Δt and then taking the absolute value. The volatility change rate Dew is obtained by dividing the difference between the marginal volatility Ew(t) at time t and the marginal volatility Ew(t-1) at time t-1 by the time interval Δt, and then taking the absolute value.

[0032] The multi-parameter mapping fusion modeling unit maps the data in the local disturbance dataset DF to a unified disturbance response function, and then performs nonlinear integrated modeling to obtain the environmental perception dynamic index Eenv. The environmental perception dynamic index Eenv is obtained using the following formula: ; In the formula, Eenv(t) represents the environmental perception dynamic index at time t, v represents the disturbance sensitivity coefficient, DFi represents the i-th data in the local disturbance dataset DF, and e represents a constant; The environmental state of the video is determined by analyzing the environmental perception dynamic index Eenv. Environment status is obtained through matching in the following ways: When 0.75 ≤ the environmental perception dynamic index Eenv, it indicates that the environment is in a state of high dynamics and strong interference. When 0.4 ≤ Environmental Perception Dynamics Index Eenv < 0.7, it indicates that the environment is under moderate disturbance. When the environmental perception dynamic index Eenv < 0.4, it indicates that the environment is in a stable state.

[0033] In this embodiment, by setting up a dimensional perturbation deconstruction unit and a multi-parameter mapping fusion modeling unit, the system realizes the dynamic fluctuation extraction and state quantification of multi-source sensor data in the temporal dimension, and can simultaneously capture and evaluate the changing trends of three core environmental factors: acoustics, illumination, and image structure. Unlike traditional schemes that rely solely on the current physical quantity values ​​for static judgment, this embodiment introduces a "rate of change" modeling method, constructing four perturbation indicators for sound pressure, illumination, grayscale difference, and image edge information, forming a perturbation dataset with trends as the core, effectively enhancing the system's ability to identify sudden environmental fluctuations and subtle state changes.

[0034] By generating an environmental perception dynamic index through nonlinear fusion modeling, various disturbance factors participate in environmental state modeling within the same framework, thus avoiding the imbalance of strength and dimension bias that easily occurs in conventional linear superposition methods. This index has three characteristics: continuity, sensitivity, and segmentable matching, and can accurately characterize the state classification of the current meeting space in a high-dynamic disturbance, moderate fluctuation, or stable environment, providing a clear and physically meaningful basis for judgment in subsequent modules.

[0035] This embodiment enhances the system's ability to dynamically model complex environmental evolution processes, enabling the system to move beyond passively waiting for users to perceive changes in visuals or sound before responding. Instead, it rapidly adjusts audio and video processing strategies based on exponential responses at the initial stage of environmental disturbances. This "disturbance-response-strategy linkage" mechanism effectively solves problems such as delayed response, misjudgment of states, and abrupt strategy switching in traditional systems. It improves the overall environmental adaptability, control precision, and interaction consistency of the system, making it particularly suitable for remote conferencing scenarios with frequently changing environmental conditions, such as multiple people entering and exiting, uneven lighting, and background noise disturbances. Example 4

[0036] This embodiment is an explanation based on Embodiment 3. Please refer to it. Figure 1 Specifically: the audio intelligent noise reduction and control module includes an acoustic disturbance extraction unit and a noise reduction control response modeling unit; The acoustic disturbance extraction unit extracts features from the acoustic component of the environmental perception dynamic index Eenv, extracts the sound pressure change rate Dpw, and calculates the acoustic disturbance sub-index Esound. The acoustic perturbation index Esound is obtained using the following formula: ; In the formula, Esound represents the acoustic perturbation index at time t, e represents a constant, vs represents the sound pressure perturbation response coefficient, and Dpw(t) represents the sound pressure change rate at time t.

[0037] The noise reduction control response modeling unit constructs the initial audio clarity coefficient cAa based on the acquired acoustic disturbance sub-index Esound and combined with the historical steady state. The initial audio clarity coefficient cAa is obtained using the following formula: ; In the formula, cAa(t) represents the initial audio clarity coefficient at time t, exp represents the exponential function, Ts represents the preset acoustic disturbance recognition threshold, and ks represents the adjustment slope control parameter. The initial audio clarity coefficient cAa is corrected to obtain the trend correction factor Hcorr, and the inertial effect caused by sound disturbance changes is removed to obtain the audio clarity adjustment coefficient Aa. The trend correction factor Hcorr is obtained as follows: within a fixed time window Th, the initial audio clarity coefficient cAa from time t-Th+1 to t is extracted; then all the initial audio clarity coefficients cAa are summed and divided by the time window Th to obtain the average value, which is used as the trend correction factor Hcorr. The audio clarity adjustment factor Aa is obtained by summing the initial audio clarity factor cAa and the trend correction factor Hcorr and then dividing by two. The ambient noise level is determined by analyzing the audio clarity adjustment coefficient Aa. The judgment method is as follows: When 0.7 ≤ audio clarity adjustment coefficient Aa < 1, it indicates that the current acoustic environment is quiet, and the filtering intensity is automatically reduced to retain more speech details; When the audio clarity adjustment coefficient Aa < 0.7, it indicates that strong environmental noise has been detected, and the filtering intensity is automatically increased to suppress background noise interference.

[0038] The video clarity and brightness adaptive adjustment module includes a multi-parameter image perturbation sensing unit and an image enhancement gain modeling unit; The multi-parameter image perturbation sensing unit analyzes the spatial average illuminance Lavg in the sensing dataset CW and calculates the luminance missing index Iin. The luminance deficiency index Iin is obtained by the difference between 1 and the spatial average illuminance Lavg; By combining the analysis of the continuous inter-frame grayscale difference CG and the edge fluctuation rate Ew in the sensor dataset CW, the image perturbation intensity index Gsta is calculated and obtained. The image perturbation intensity index Gsta is obtained by taking the square root of the sum of the square of the grayscale difference CG between consecutive frames and the square of the edge volatility Ew.

[0039] In this embodiment, by refining the intelligent audio noise reduction and control module into an acoustic disturbance extraction unit and a noise reduction control response modeling unit, fine-grained modeling and dynamic control of noise change trends are achieved, breaking through the static judgment mode of instantaneous sound pressure value in traditional audio processing solutions. The system not only calculates the sound pressure change rate but also maps it to an acoustic disturbance sub-index, thereby accurately reflecting the dynamic fluctuations of background noise in the meeting space. Simultaneously, to improve the stability and continuity of the noise reduction response, after generating the initial audio clarity coefficient, the system further introduces a trend correction mechanism. By extracting historical clarity data within a fixed time window to construct a correction factor, the final output adjustment coefficient combines the characteristics of current disturbance reflection with the continuation of historical states.

[0040] In terms of image channels, the system constructs a brightness deficiency index based on spatial average illumination and an image perturbation intensity index based on dynamic image information, achieving accurate identification of dark scenes and structurally complex scenes. Unlike traditional image enhancement strategies that use absolute image brightness as a reference, this embodiment uses relative deficiency to measure the required compensation level and judges the amplitude of image fluctuations through a comprehensive extraction mechanism of grayscale changes and edge transitions. This shifts the image enhancement strategy from static parameter setting to responsive adjustment to dynamic environmental changes. This multi-parameter fusion image perturbation modeling method enhances the system's adaptability to scenes with low illumination, frequent movements, and complex fields of view, ensuring that images still maintain good clarity and structural readability in complex environments. Example 5

[0041] This embodiment is an explanation based on Embodiment 4. Please refer to it. Figure 2 and Figure 3 Specifically: the image enhancement gain modeling unit performs nonlinear fusion of the acquired brightness loss index Iin and image perturbation intensity index Gsta to calculate and obtain the image enhancement gain factor Lv; The image enhancement gain factor Lv is obtained using the following formula: ; In the formula, Iin(t) represents the brightness loss index at time t, ln represents the logarithmic function, Gsta(t) represents the image perturbation intensity index at time t, and vo represents the brightness compensation index adjustment factor. By analyzing the image enhancement gain factor Lv and judging the image, control strategy suggestions are obtained. The determination method is obtained by matching in the following way: When the image enhancement gain factor Lv≥2.0, the image is too dark or has strong fluctuations. The recommended strategy is to increase both brightness and sharpness. When 2.0 > image enhancement gain factor Lv ≥ 1.3, the image is moderately disturbed or slightly dark. The recommended strategy is to moderately improve contrast and edge sharpness. When 1.3 > image enhancement gain factor Lv, the image is considered to be in a normal environment. The recommended strategy is to minimize image processing and preserve the natural state.

[0042] The audio and video synchronization quality optimization module normalizes the acquired audio clarity adjustment coefficient Aa and image enhancement gain factor Lv to unify the dimensions of the data and calculates the synchronization correction index Sync. The synchronization correction index Sync is obtained using the following formula: ; In the formula, Aa(t) represents the audio clarity adjustment coefficient at time t, Sync represents the synchronization correction exponent at time t, log represents the logarithmic function, and ec represents a non-zero constant; The obtained synchronization correction index Sync is compared with the preset correction threshold Tync to determine whether the audio and video parameters are mismatched. The judgment method is as follows: When the synchronization correction index Sync < the correction threshold Tync, it indicates that synchronization is normal. When the correction threshold Tync ≤ synchronization correction index Sync < 1.5 × correction threshold Tync, it indicates a moderate deviation; When 1.5 × correction threshold Tync < synchronization correction index Sync, it indicates a mismatch.

[0043] The system self-feedback and enhancement strategy adjustment module normalizes the acquired environmental perception dynamic index Eenv and combines it with the synchronization correction index Sync to calculate the final evaluation index Ψtotal. The final evaluation metric Ψtotal is obtained using the following formula: ; In the formula, Ψtotal(t) represents the final evaluation index at time t; Analyze the current value and trend sequence of the final evaluation indicator Ψtotal, and execute the corresponding enhancement strategy switching operation; The analysis method is as follows: When the final evaluation metric Ψtotal≥0.75, it indicates a high-dynamic disturbance scenario, and the strategy is adjusted to enable the enhanced mode. When 0.75 > the final evaluation index Ψtotal ≥ 0.45, it indicates a moderate disturbance and a critical state. The strategy is adjusted as follows: maintain the strategy unchanged and set the current state as the observation state. When 0.45 > the final evaluation index Ψtotal, it indicates a stable and low-interference state, and the strategy is adjusted to enable energy-saving mode.

[0044] In this embodiment, by introducing an image enhancement gain modeling unit, a nonlinear fusion of the brightness loss index and the image perturbation intensity index is achieved for the first time in a conference video processing scenario, constructing a dynamically adjustable image enhancement gain factor. This approach breaks through the traditional image processing technology's reliance on a single brightness threshold or edge contrast enhancement factor, enabling the system to possess comprehensive perception and hierarchical control capabilities for multi-source visual perturbation signals. The system can not only adjust image clarity and sharpness based on image brightness but also on edge structure stability, effectively adapting to video presentation needs in various complex scenarios such as low illumination, high dynamic range, and frequent switching, thereby improving the stability of image expression and structural recognition capabilities.

[0045] This embodiment fundamentally solves the mismatch judgment bias problem caused by the inconsistency in the control scale between audio and video by unifying and normalizing the audio clarity adjustment coefficient and the image enhancement gain factor, and by introducing a logarithmic transformation to construct a synchronization correction index. The synchronization correction index not only reflects the consistency of audio and video processing intensity, but also dynamically distinguishes them by combining preset thresholds, realizing hierarchical identification and step-by-step feedback adjustment of synchronization state. Compared with the traditional strategy of coarse comparison based on frame rate or timestamp offset, this method provides a more refined matching mechanism at the processing intensity level, improving the system's collaborative adjustment capability under extreme conditions of video enhancement or voice denoising.

[0046] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An ultra-high-definition video conferencing system based on AI and intelligent sensors, characterized in that: It includes a multi-source sensing parameter acquisition module, an environmental dynamic sensing modeling module, an audio intelligent noise reduction and control module, a video clarity and brightness adaptive control module, an audio and video synchronization quality optimization module, and a system self-feedback and enhancement strategy adjustment module. The multi-source sensing parameter acquisition module collects sensor data in the conference space through sensors, performs preprocessing, and obtains a sensor dataset. The environmental dynamic perception modeling module constructs a composite index reflecting the degree of dynamic change in the environment based on the sensor dataset, and obtains the environmental dynamic perception index. The audio intelligent noise reduction and control module calculates and obtains the audio clarity adjustment coefficient based on the acoustic part of the environmental perception dynamic index. The video clarity and brightness adaptive adjustment module calculates and obtains the image enhancement gain factor based on the ambient illuminance in the sensor dataset, and dynamically adjusts the image enhancement strategy. The audio and video synchronization quality optimization module calculates and obtains the synchronization correction index based on the audio clarity adjustment coefficient and the image enhancement gain factor, and determines whether the audio and video parameters are mismatched. The system self-feedback and enhancement strategy adjustment module calculates the final evaluation index based on the acquired environmental perception dynamic index and synchronous correction index, and performs dynamic strategy self-adjustment.

2. The ultra-high-definition video conferencing system based on AI and intelligent sensors according to claim 1, characterized in that: The multi-source sensing parameter acquisition module includes a sensing signal acquisition unit and a heterogeneous data normalization preprocessing unit; The sensing signal acquisition unit synchronously collects multi-source physical signals of the conference space from the deployed sound pressure sensor, light sensor and image sensor, including instantaneous sound pressure value Pw, spatial average illuminance Lavg, continuous frame gray level difference CG and edge fluctuation rate Ew, and fits them into the original dataset YW. The instantaneous sound pressure value Pw is acquired by a sound pressure sensor; Spatial average illuminance Lavg is acquired through a light sensor; The continuous inter-frame grayscale difference CG and edge fluctuation rate Ew are acquired by the image sensor; The heterogeneous data normalization preprocessing unit performs simultaneous cleaning and normalization on different physical dimensions in the original dataset YW to eliminate noise, abrupt values ​​and cross-modal scale differences, and obtain the sensor dataset CW. Cleaning includes outlier removal; noise and abrupt changes in the original dataset YW are eliminated by processing the data using dynamic median offset. The processing method is as follows: at the current time point, take any parameter from the original dataset YW, and then select historical time points and future time points forward and backward to form a data window sequence containing multiple time points; in the data window, extract all values ​​of the same parameter at the time point and calculate the median of these values; then subtract the median from the original value at the current time to obtain the offset difference, which is used as the final cleaning result. The formula for normalization is as follows: ； In the formula, CWo represents the o-th data in the sensor dataset CW, YWo represents the o-th data in the original dataset YW, minYWo represents the valley value of the o-th data in the original dataset YW, and maxYWo represents the peak value of the o-th data in the original dataset YW.

3. The ultra-high-definition video conferencing system based on AI and intelligent sensors according to claim 2, characterized in that: The environmental dynamic perception modeling module includes a dimensional perturbation deconstruction unit and a multi-parameter mapping fusion modeling unit; The dimensional perturbation deconstruction unit extracts features from the parameters in the sensing dataset CW, including the sound pressure change rate Dpw, the illuminance change rate Davg, the grayscale change rate Dcg, and the fluctuation change rate Dew, and fits them into a local perturbation dataset DF. The rate of change of sound pressure, Dpw, is obtained by dividing the difference between the instantaneous sound pressure value Pw(t) at time t and the instantaneous sound pressure value Pw(t-1) at time t-1 by the time interval Δt, and then taking the absolute value. The rate of change of illuminance Davg is obtained by dividing the difference between the spatial average illuminance Lavg(t) at time t and the spatial average illuminance Lavg(t-1) at time t-1 by the time interval Δt and then taking the absolute value. The grayscale change rate Dcg is obtained by dividing the difference between the grayscale difference CG(t) between consecutive frames at time t and the grayscale difference CG(t-1) between consecutive frames at time t-1 by the time interval Δt and then taking the absolute value. The volatility change rate Dew is obtained by dividing the difference between the marginal volatility Ew(t) at time t and the marginal volatility Ew(t-1) at time t-1 by the time interval Δt, and then taking the absolute value.

4. The ultra-high-definition video conferencing system based on AI and intelligent sensors according to claim 3, characterized in that: The multi-parameter mapping fusion modeling unit maps the data in the local disturbance dataset DF to a unified disturbance response function, and then performs nonlinear integrated modeling to obtain the environmental perception dynamic index Eenv. The environmental perception dynamic index Eenv is obtained using the following formula: ； In the formula, Eenv(t) represents the environmental perception dynamic index at time t, v represents the disturbance sensitivity coefficient, DFi represents the i-th data in the local disturbance dataset DF, and e represents a constant; The environmental state of the video is determined by analyzing the environmental perception dynamic index Eenv. Environment status is obtained through matching in the following ways: When 0.75 ≤ the environmental perception dynamic index Eenv, it indicates that the environment is in a state of high dynamics and strong interference. When 0.4 ≤ Environmental Perception Dynamics Index Eenv < 0.7, it indicates that the environment is under moderate disturbance. When the environmental perception dynamic index Eenv < 0.4, it indicates that the environment is in a stable state.

5. The ultra-high-definition video conferencing system based on AI and intelligent sensors according to claim 3, characterized in that: The audio intelligent noise reduction and control module includes an acoustic disturbance extraction unit and a noise reduction control response modeling unit; The acoustic disturbance extraction unit extracts features from the acoustic component of the environmental perception dynamic index Eenv, extracts the sound pressure change rate Dpw, and calculates the acoustic disturbance sub-index Esound. The acoustic perturbation index Esound is obtained using the following formula: ； In the formula, Esound represents the acoustic perturbation index at time t, e represents a constant, vs represents the sound pressure perturbation response coefficient, and Dpw(t) represents the sound pressure change rate at time t.

6. The ultra-high-definition video conferencing system based on AI and intelligent sensors according to claim 5, characterized in that: The noise reduction control response modeling unit constructs the initial audio clarity coefficient cAa based on the acquired acoustic disturbance sub-index Esound and combined with the historical steady state. The initial audio clarity coefficient cAa is obtained using the following formula: ； In the formula, cAa(t) represents the initial audio clarity coefficient at time t, exp represents the exponential function, Ts represents the preset acoustic disturbance recognition threshold, and ks represents the adjustment slope control parameter. The initial audio clarity coefficient cAa is corrected to obtain the trend correction factor Hcorr, and the inertial effect caused by sound disturbance changes is removed to obtain the audio clarity adjustment coefficient Aa. The trend correction factor Hcorr is obtained as follows: within a fixed time window Th, the initial audio clarity coefficient cAa from time t-Th+1 to t is extracted; then all the initial audio clarity coefficients cAa are summed and divided by the time window Th to obtain the average value, which is used as the trend correction factor Hcorr. The audio clarity adjustment factor Aa is obtained by summing the initial audio clarity factor cAa and the trend correction factor Hcorr and then dividing by two. The ambient noise level is determined by analyzing the audio clarity adjustment coefficient Aa. The judgment method is as follows: When 0.7 ≤ audio clarity adjustment coefficient Aa < 1, it indicates that the current acoustic environment is quiet, and the filtering intensity is automatically reduced to retain more speech details; When the audio clarity adjustment coefficient Aa < 0.7, it indicates that strong environmental noise has been detected, and the filtering intensity is automatically increased to suppress background noise interference.

7. The ultra-high-definition video conferencing system based on AI and intelligent sensors according to claim 6, characterized in that: The video clarity and brightness adaptive adjustment module includes a multi-parameter image perturbation sensing unit and an image enhancement gain modeling unit; The multi-parameter image perturbation sensing unit analyzes the spatial average illuminance Lavg in the sensing dataset CW and calculates the luminance missing index Iin. The luminance deficiency index Iin is obtained by the difference between 1 and the spatial average illuminance Lavg; By combining the analysis of the continuous inter-frame grayscale difference CG and the edge fluctuation rate Ew in the sensor dataset CW, the image perturbation intensity index Gsta is calculated and obtained. The image perturbation intensity index Gsta is obtained by taking the square root of the sum of the square of the grayscale difference CG between consecutive frames and the square of the edge volatility Ew.

8. The ultra-high-definition video conferencing system based on AI and intelligent sensors according to claim 7, characterized in that: The image enhancement gain modeling unit performs nonlinear fusion of the acquired brightness loss index Iin and image perturbation intensity index Gsta to calculate the image enhancement gain factor Lv. The image enhancement gain factor Lv is obtained using the following formula: ； In the formula, Iin(t) represents the brightness loss index at time t, ln represents the logarithmic function, Gsta(t) represents the image perturbation intensity index at time t, and vo represents the brightness compensation index adjustment factor. By analyzing the image enhancement gain factor Lv and judging the image, control strategy suggestions are obtained. The determination method is obtained by matching in the following way: When the image enhancement gain factor Lv≥2.0, the image is too dark or has strong fluctuations. The recommended strategy is to increase both brightness and sharpness. When 2.0 > image enhancement gain factor Lv ≥ 1.3, the image is moderately disturbed or slightly dark. The recommended strategy is to moderately improve contrast and edge sharpness. When 1.3 > image enhancement gain factor Lv, the image is considered to be in a normal environment. The recommended strategy is to minimize image processing and preserve the natural state.

9. The ultra-high-definition video conferencing system based on AI and intelligent sensors according to claim 8, characterized in that: The audio and video synchronization quality optimization module normalizes the acquired audio clarity adjustment coefficient Aa and image enhancement gain factor Lv to unify the dimensions of the data and calculates the synchronization correction index Sync. The synchronization correction index Sync is obtained using the following formula: ； In the formula, Aa(t) represents the audio clarity adjustment coefficient at time t, Sync represents the synchronization correction exponent at time t, log represents the logarithmic function, and ec represents a non-zero constant; The obtained synchronization correction index Sync is compared with the preset correction threshold Tync to determine whether the audio and video parameters are mismatched. The judgment method is as follows: When the synchronization correction index Sync < the correction threshold Tync, it indicates that synchronization is normal. When the correction threshold Tync ≤ synchronization correction index Sync < 1.5 × correction threshold Tync, it indicates a moderate deviation; When 1.5 × correction threshold Tync < synchronization correction index Sync, it indicates a mismatch.

10. The ultra-high-definition video conferencing system based on AI and intelligent sensors according to claim 9, characterized in that: The system self-feedback and enhancement strategy adjustment module normalizes the acquired environmental perception dynamic index Eenv and combines it with the synchronization correction index Sync to calculate the final evaluation index Ψtotal. The final evaluation metric Ψtotal is obtained using the following formula: ； In the formula, Ψtotal(t) represents the final evaluation index at time t; Analyze the current value and trend sequence of the final evaluation indicator Ψtotal, and execute the corresponding enhancement strategy switching operation; The analysis method is as follows: When the final evaluation metric Ψtotal≥0.75, it indicates a high-dynamic disturbance scenario, and the strategy is adjusted to enable the enhanced mode. When 0.75 > the final evaluation index Ψtotal ≥ 0.45, it indicates a moderate disturbance and a critical state. The strategy is adjusted as follows: maintain the strategy unchanged and set the current state as the observation state. When 0.45 > the final evaluation index Ψtotal, it indicates a stable and low-interference state, and the strategy is adjusted to enable energy-saving mode.

Images