Self-adaptive display adjusting system based on multi-mode perception

Abstract

The invention discloses a self-adaptive display adjusting system based on multi-mode perception, belongs to the technical field of communication, and aims to solve the technical problems that existing display equipment is single in adjusting mode, insufficient in self-adaptive capability, high in energy consumption and poor in use experience. Comprising a multi-mode sensing module which collects multi-source environment data and sends the multi-source environment data to an edge computing unit; the touch interaction module is used for collecting a manual adjustment instruction of a user; the communication module is used for collecting timestamps; the edge calculation unit is used for associating a user manual adjustment instruction and the environment characteristics as reference data based on a timestamp, calling a scene recognition engine to judge a current use scene based on the reference data, and calling a user behavior learning model to predict an adjustment parameter of a display module as a user preference parameter; making a decision based on the current use scene and the user preference parameters, and generating an adjustment instruction; and the self-adaptive adjustment module is used for performing parameter adjustment on the display module based on the adjustment instruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to an adaptive display adjustment system based on multi-modal perception. BACKGROUND

[0002] Current various display devices generally adopt fixed parameters or single sensor adjustment mode, which has significant technical limitations. For example, common screen devices often adjust brightness according to the ambient light sensor in one direction, but ignore the distance between the user and the device, the angle of view and the scene difference, resulting in problems of energy waste due to over-brightness or poor viewing due to over-darkness. At the same time, the existing adjustment algorithm lacks the ability to learn user behavior habits, such as low light mode relying only on time triggering, which cannot dynamically adjust the switching threshold according to user habits; in addition, the linkage between the display device and the surrounding environment device is insufficient, and the multi-dimensional environmental data such as temperature, humidity and human activity are not used to optimize the display parameters, resulting in the dual pain points of fragmented use experience and low energy efficiency.

[0003] The problems of single adjustment mode, insufficient adaptability, high energy consumption and poor use experience of the existing display device are technical problems to be solved. SUMMARY

[0004] The technical task of the present application is to solve the above problems, and provide an adaptive display adjustment system based on multi-modal perception, to solve the problems of single adjustment mode, insufficient adaptability, high energy consumption and poor use experience of the existing display device.

[0005] The adaptive display adjustment system based on multi-modal perception comprises a multi-modal sensing module, a communication module, an edge computing unit, a touch interaction module, an adaptive adjustment module and a display module.

[0006] The multi-modal sensing module is used for collecting multi-source environmental data and sending the multi-source data to the edge computing unit, wherein the multi-source data includes ambient light data, distance data, image data and temperature and humidity data.

[0007] The touch interaction module is used for collecting user manual adjustment instructions and sending the user manual adjustment instructions to the edge computing unit.

[0008] The communication module is used for collecting time stamps and sending the time stamps to the edge computing unit.

[0009] The edge computing unit is configured to receive multi-source data, timestamps, and user manual adjustment instructions, perform data preprocessing and data fusion on the multi-source environment data to obtain environment features, associate the user manual adjustment instructions and the environment features based on the timestamps as reference data, determine a current use scenario based on the reference data by invoking a scene recognition engine, predict adjustment parameters for the display module as user preference parameters by invoking a user behavior learning model, make a decision based on the current use scenario and the user preference parameters, generate an adjustment instruction, and send the adjustment instruction to the adaptive adjustment module.

[0010] The adaptive adjustment module is configured to perform parameter adjustment on the display module based on the adjustment instruction.

[0011] Preferably, the multi-modal sensing module is configured to perform data collection as follows:

[0012] The ambient light sensor is configured to collect light intensity at regular intervals to obtain ambient light data.

[0013] The infrared distance sensor is configured to collect the distance between the user and the device at regular intervals as distance data.

[0014] The camera is configured to collect image data of the user at regular intervals, and the image data is used to determine whether the user is present and the user's line of sight.

[0015] Correspondingly, the multi-source environment data is preprocessed and fused as follows:

[0016] The ambient light data is subjected to a moving average filter, the distance data is subjected to outlier rejection, the user's line of sight vector and the number of faces are extracted from the image data using an image data recognition algorithm, and the pixel brightness distribution of the screen area and the glare area are analyzed to obtain preprocessed multi-source data.

[0017] The preprocessed multi-source environment data is subjected to weighted fusion to obtain environment features, wherein the weight of each type of data in the multi-source environment data can be dynamically adjusted.

[0018] Preferably, the user behavior learning model is a neural network model based on an LSTM network, which takes timestamps, environment features, and user manual adjustment instructions as input and predicts adjustment parameters for the display module as output, including adjustment values for brightness and color temperature.

[0019] The user behavior learning model is based on historical parameter data for model optimization and parameter updating.

[0020] Preferably, when making a decision based on the current use scenario and the user preference parameters, a predefined scene recognition and parameter adjustment strategy is used to make the decision, which includes the following content:

[0021] Low light scene: When the light is detected to be less than the low light illumination threshold < 50 lux and the time is between 22:00-6:00, the system determines it to be a low light scene. Adjustment logic: brightness is reduced to below 100 nit, and color temperature is adjusted to 2700K warm light to reduce eye irritation;

[0022] Concentration use scene: When the user is 0.5-1 m away, the line of sight is continuously focused on the display area for more than 3 seconds, and the illumination is 200-500 lux, the concentration use scene is triggered, and the adjustment logic is: the brightness is increased to 800 nit, the color temperature is adjusted to 5000K to simulate natural light to enhance content clarity; at the same time, the refresh rate of the display module is reduced to 60 Hz to balance clarity and energy consumption;

[0023] Long-distance observation scene: When the user is more than 2 m away and the line of sight occasionally stays, it is determined to be a long-distance observation scene, and the adjustment logic is: the brightness is stable at 600 nit, the display content font is automatically enlarged to 120% of the original size, and the interface layout is simplified;

[0024] Strong light environment scene: When the illumination is > 5000 lux and the image acquisition unit identifies that there are ≥ 2 glare areas on the screen surface, the strong light environment scene is activated, and the adjustment logic is: on the hardware level, the brightness is instantaneously increased to 1000 nit maximum value through the PWM dimming module, and the anti-reflection coating circuit is driven; on the software level, it is switched to a high-contrast mode, and the font edge sharpening process is used to offset the glare blur;

[0025] Multi-person sharing scene: When the image acquisition unit detects ≥ 2 users, and the infrared distance sensor shows that the user distribution distance difference is > 0.5 m, the multi-person sharing scene is entered, and the adjustment logic is: the brightness is fixed at 600 nit, the color temperature is adjusted to 4500K neutral light, and the display content enables the sub-area adaptation mode;

[0026] Fast interaction scene: When the user triggers an operation through touch or voice, and the edge computing unit detects that the operation response time is < 100 ms, it is determined to be a fast interaction scene, and the adjustment logic is: the brightness is temporarily increased by 10% based on the current value, the color temperature is temporarily switched to 5500K to strengthen the visual focus, the interface dynamic effect duration is compressed to within 150 ms, unnecessary background animations are turned off, and the operation feedback speed is prioritized;

[0027] Among them, the scene priority from high to low is: fast interaction scene > strong light environment scene > multi-person sharing scene > concentration use scene > long-distance observation scene > low light scene.

[0028] As a preferred, when the adaptive adjustment module is used to adjust the parameters of the display module based on the adjustment instruction, the scene switching adopts a smooth transition algorithm.

[0029] As preferred, the display module adopts a multi-color backlight unit, supports continuous adjustment of brightness and color temperature, and integrates a PWM dimming module in the driving circuit.

[0030] As preferred, the communication unit integrates Wi-Fi and ZigBee protocols, supports edge computing units, and environmental gateways and other associated devices.

[0031] As preferred, the multi-modal sensing module sequentially collects initial data from each sensor and uploads it to the edge computing unit, and automatically enables a redundancy algorithm if a sensor fails.

[0032] The adaptive display adjustment system based on multi-modal sensing has the following advantages:

[0033] 1. Breakthrough single sensor adjustment mode, fusion of multi-dimensional environment and user data, combination of LSTM behavior learning model and fine identification of 6 types of scenes, realization of "scene adaptation + user adaptation" dual adjustment, significant improvement of adjustment accuracy and scene coverage compared with traditional system;

[0034] 2. By dynamically matching display parameters with actual needs, the energy consumption is significantly reduced compared with the fixed parameter mode under the premise of ensuring user experience, which meets the trend of low-carbon development;

[0035] 3. Realize full-scene automatic adaptation and personalized adjustment, reduce more than 90% of manual operation demand, and improve user satisfaction;

[0036] 4. The system hardware module and algorithm interface adopt standardized design, which can adapt to display devices of different sizes and types, has wide industrial application prospect, and is suitable for various products such as terminal with screen, interactive panel and indoor display device. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] The present application will be further described below in conjunction with the drawings.

[0039] Figure 1 The structure block diagram of the adaptive display adjustment system based on multi-modal sensing of embodiment 1. DETAILED DESCRIPTION

[0040] The present application will be further described below in conjunction with the drawings and specific embodiments so that those skilled in the art can better understand and implement the present application, but the embodiments do not limit the present application, and the technical features in the embodiments and the embodiments can be combined with each other without conflict.

[0041] The embodiment of the present application provides a self-adaptive display adjustment system based on multi-modal perception, which is used for solving the technical problems of single adjustment mode, insufficient adaptive capacity, high energy consumption and poor use experience of the existing display device.

[0042] Embodiment:

[0043] The self-adaptive display adjustment system based on multi-modal perception comprises a multi-modal sensing module, a communication module, an edge computing unit, a touch interaction module, a self-adaptive adjustment module and a display module.

[0044] The multi-modal sensing module is used for collecting multi-source environmental data and sending the multi-source data to the edge computing unit, wherein the multi-source data comprises environmental light data, distance data, image data and temperature and humidity data.

[0045] As a specific implementation, the multi-modal sensing module is used for performing the following data collection:

[0046] (1) The environmental light data is obtained by collecting the light intensity through the environmental light sensor at regular time intervals.

[0047] (2) The distance data is obtained by collecting the distance between the user and the device through the infrared distance sensor at regular time intervals.

[0048] (3) The image data of the user is collected through the camera at regular time intervals, and the image data is used to determine whether the user exists and the user's visual direction.

[0049] The sensors in the multi-modal sensing module collect the initial data in sequence and upload the initial data to the edge computing unit, and if a sensor fails, a redundant algorithm is automatically enabled.

[0050] The touch interaction module is used for collecting the user's manual adjustment instruction and sending the user's manual adjustment instruction to the edge computing unit.

[0051] The communication module is used for collecting the time stamp and sending the time stamp to the edge computing unit.

[0052] The edge computing unit is configured to receive multi-source data, timestamps, and user manual adjustment instructions, perform data preprocessing and data fusion on the multi-source environment data to obtain environment features, associate the user manual adjustment instructions and the environment features based on the timestamps as reference data, determine a current use scenario based on the reference data by invoking a scene recognition engine, predict adjustment parameters for the display module as user preference parameters by invoking a user behavior learning model, make a decision based on the current use scenario and the user preference parameters, generate an adjustment instruction, and send the adjustment instruction to the adaptive adjustment module.

[0053] As a specific implementation, the data preprocessing and data fusion on the multi-source environment data include the following operations:

[0054] (1) Perform sliding average filtering on ambient light data, remove outliers from distance data, extract a user gaze vector and a number of faces from image data through an image data recognition algorithm, analyze screen area pixel brightness distribution and monitor glare areas, and obtain preprocessed multi-source data.

[0055] (2) Perform weighted fusion on the preprocessed multi-source environment data to obtain environment features, wherein the weight of each type of data in the multi-source environment data can be dynamically adjusted.

[0056] The user behavior learning model is a neural network model based on an LSTM network, which takes timestamps, environment features, and user manual adjustment instructions as input and predicts adjustment parameters for the display module as output. The adjustment parameters include adjustment values for brightness and color temperature. The user behavior learning model is optimized and updated based on historical parameter data.

[0057] When making a decision based on the current use scenario and the user preference parameters, a predefined scene recognition and parameter adjustment strategy is used to make the decision. The scene recognition and parameter adjustment strategy includes the following contents:

[0058] Low-light scene: When the detected illumination is less than a low-light illumination threshold < 50 lux and the time is between 22:00 and 6:00, the system determines that it is a low-light scene. Adjustment logic: reduce the brightness to below 100 nit and adjust the color temperature to 2700K warm light to reduce eye irritation.

[0059] Concentration use scenario: When the user is 0.5-1m away, the gaze is continuously focused on the display area for more than 3 seconds, and the illumination is 200-500 lux, the concentration use scenario is triggered. Adjustment logic: increase the brightness to 800 nit, adjust the color temperature to 5000K to simulate natural light to enhance content clarity, and reduce the display module refresh rate to 60Hz to balance clarity and energy consumption.

[0060] Far distance observation scene: when the user distance > 2m and the line of sight occasionally stops, it is determined as a far distance observation scene, and the adjustment logic is: the brightness is stabilized at 600nit, the display content font is automatically enlarged to 120% of the original size, and the interface layout is simplified;

[0061] Strong light environment scene: when the illumination > 5000lux and the image acquisition unit identifies that there are > 2 glare areas on the screen surface, the strong light environment scene is activated, and the adjustment logic is: the hardware level instantaneously increases the brightness to 1000nit maximum value through the PWM dimming module, and drives the anti-reflection coating circuit, and the software level switches to the high contrast mode, and the font edge sharpening processing is used to offset the glare blur;

[0062] Multi-person sharing scene: when the image acquisition unit detects > 2 users, and the infrared distance sensor shows that the user distribution distance difference > 0.5m, the multi-person sharing scene is entered, and the adjustment logic is: the brightness is fixed at 600nit, the color temperature is adjusted to 4500K neutral light, and the display content is enabled to use the sub-area adaptive mode;

[0063] Fast interaction scene: when the user triggers the operation through touch or voice, and the edge computing unit detects that the operation response time < 100ms, it is determined as a fast interaction scene, and the adjustment logic is: the brightness is temporarily increased by 10% on the basis of the current value, the color temperature is temporarily switched to 5500K to strengthen the visual focus, the interface dynamic effect time is compressed to within 150ms, unnecessary background animation is closed, and the operation feedback speed is preferentially guaranteed;

[0064] Among them, the scene priority from high to low is: fast interaction scene > strong light environment scene > multi-person sharing scene > focused use scene > far distance observation scene > low light scene.

[0065] The adaptive adjustment module is used for adjusting the parameters of the display module based on the adjustment instruction. Among them, the scene switching uses a smooth transition algorithm.

[0066] Among them, the display module uses a multi-color backlight unit, supports continuous adjustment of brightness and color temperature, and integrates a PWM dimming module in the driving circuit. The communication unit integrates Wi-Fi and ZigBee protocols, and supports the edge computing unit and the environment gateway and other associated devices.

[0067] For the system disclosed in the embodiment, it involves hardware layer, algorithm layer and application layer. In the hardware layer, it involves multi-modal sensing module, display module, edge computing unit and communication module.

[0068] Multi-modal sensor module: composed of ambient light sensor (collection range 0-10000 lux), infrared distance sensor (detection distance 0.3-3m), image acquisition unit (for line of sight tracking and user presence detection), temperature and humidity sensor (accuracy ±0.5℃ / ±3% RH), all connected to the main controller through I2C bus, sampling frequency can be dynamically adjusted (1-10Hz).

[0069] Display module: adopts multi-color backlight unit, supports continuous adjustment of brightness (0-1000 nit) and color temperature (2000K-6500K), drive circuit integrates PWM dimming module, response delay ≤10ms.

[0070] Edge computing unit: equipped with ARM Cortex-A53 processor (clock speed 1.2GHz), equipped with 2GB RAM for local data processing, to avoid privacy leakage and delay caused by uploading raw data to the cloud.

[0071] Communication module: integrates Wi-Fi and ZigBee protocols, supports data interaction with environmental gateway and other associated devices (such as lighting devices, shading devices), communication delay ≤50ms.

[0072] Algorithm layer involves multi-source data fusion algorithm, user behavior learning model scene recognition engine.

[0073] Multi-source data fusion algorithm: uses Kalman filter to filter noise of sensor data, generates comprehensive environmental feature vector through weighted fusion algorithm (weight dynamically adjusted), where ambient light weight is 0.3, distance weight is 0.2, temperature and humidity weight is 0.1, and user state weight is 0.4.

[0074] User behavior learning model: habit prediction model is built based on LSTM neural network, input parameters include timestamp, environmental features, and user manual adjustment records, output predicted brightness / color temperature adjustment value, model training period is 7 days, parameters are automatically updated daily.

[0075] Scene recognition engine: classifies environmental and user data through decision tree algorithm, defines 6 typical scenarios: low light (illumination <50 lux and time 22:00-6:00), focused use (user distance 0.5-1m and line of sight focused on screen), long distance observation (distance >2m), strong light (illumination >5000 lux and screen surface has glare area), multi-person sharing (users ≥2 households and distance difference >0.5m), fast interaction (operation response <100ms), each scenario corresponds to preset adjustment logic.

[0076] Application layer involves adaptive adjustment module, device linkage interface and user interaction interface.

[0077] Adaptive adjustment module: According to the adjustment instruction output by the algorithm layer, the display parameters are adjusted in real time through the driving circuit, and the adjustment step can be dynamically set according to the change rate (the step increases when the environment changes suddenly).

[0078] Device linkage interface: Provide standardized API for environmental system docking, receive state data of other devices (such as shielding device opening degree, lighting device switch state), as adjustment auxiliary basis.

[0079] User interaction interface: Support manual correction of adjustment parameters through touch or voice instructions, and feedback the correction record to the behavior learning model to realize closed-loop optimization.

[0080] The working process of the system of the embodiment involves system initialization, data acquisition and processing, scene recognition and parameter adjustment, user behavior learning process, and special scene processing.

[0081] During system initialization, after power-on, the multi-modal sensing module performs self-checking, each sensor sequentially acquires initial data and uploads it to the edge computing unit, and if a sensor fails, a redundant algorithm is automatically enabled (such as using ambient light + time adjustment logic by default when distance data is missing). The display module enters the standard mode by default (brightness 500 nit, color temperature 4000K), and at the same time the edge computing unit connects to the environmental gateway through Wi-Fi to obtain the basic environmental data of the current space (such as space type, whether there is a human body present).

[0082] During data acquisition and processing, the ambient light sensor acquires light intensity every 2 seconds, the infrared distance sensor detects the distance between the user and the device every 1 second, the image acquisition unit determines whether the user exists and the line of sight direction through frame difference method (analysis every 5 frames), and the temperature and humidity sensor acquires data every 10 minutes. The edge computing unit pre-processes the raw data: performs sliding average filtering on ambient light data (window size 5), removes outliers from distance data (retains valid data within the range of 0.3-3m), extracts user gaze vector (determines whether to gaze at the display area) and face number (used for multi-person scene recognition) from the collected images through image recognition algorithm, and analyzes the pixel brightness distribution of the screen area to monitor the glare area (strong light scene determination basis).

[0083] Scene recognition and parameter adjustment involve the following scenarios:

[0084] Low light scene: When the detected light is <50 lux and the time is between 22:00-6:00, the system determines it as a low light scene. Adjustment logic: brightness is reduced to below 100 nit (default 80 nit), and color temperature is adjusted to 2700K warm light to reduce eye irritation.

[0085] Focus usage scenario: Triggered when the user is within 0.5-1m, gaze continuously focuses on the display area (more than 3 seconds), and the lighting is 200-500 lux. Adjustment logic: Brightness is increased to 800 nit, and color temperature is adjusted to 5000K (simulating natural light) to enhance content clarity. At the same time, the display module refresh rate is reduced to 60Hz (when there is no dynamic content), balancing clarity and energy consumption.

[0086] Far distance observation scenario: Triggered when the user is more than 2m away and the gaze occasionally stays (single gaze <1 second). Adjustment logic: Brightness is stabilized at 600 nit (to ensure visibility at a distance), the display content font is automatically enlarged to 120% of the original size, and the interface layout is simplified (while retaining core information).

[0087] Strong light environment scenario: Triggered when the lighting is >5000 lux and the image acquisition unit identifies that there are ≥2 glare areas (area >10% of the display area) on the screen surface. Adjustment logic: Hardware level uses PWM dimming module to instantly increase brightness to 1000 nit maximum value, and drives anti-reflection coating circuit (if the display module is equipped); software level switches to high contrast mode (contrast ratio increases by 20%), and font edge sharpening processing is used to offset glare blur.

[0088] Multi-person sharing scenario: Triggered when the image acquisition unit detects ≥2 users (face detection confidence >85%) and the infrared distance sensor shows that the user distribution distance difference is >0.5m. Adjustment logic: Brightness is fixed at 600 nit, and color temperature is adjusted to 4500K neutral light (to balance visual perception at different positions); display content enables sub-area adaptation mode (core information is enlarged by 1.2 times in the central area, and auxiliary information remains the original size in the edge area).

[0089] Fast interaction scenario: Triggered when the user triggers an operation through touch or voice, and the edge computing unit detects that the operation response time is <100ms (such as querying the weather or switching functions). Adjustment logic: Brightness is temporarily increased by 10% based on the current value (falls back after 2 seconds), color temperature is temporarily switched to 5500K to strengthen the visual focus; interface dynamic effect duration is compressed to within 150ms, and unnecessary background animations are turned off to prioritize operation feedback speed.

[0090] During the user behavior learning process, when the user manually corrects the parameters in a certain scene (such as adjusting the system recommended 1000 nit to 900 nit in a strong light scene), the edge computing unit records the environmental characteristics (such as illumination 6000 lux, distance 1.5 m, time 14:00) and the correction value of the operation as samples and adds them to the training set. At 3 a.m. every day, the LSTM model uses the newly added samples for incremental training; for the strong light scene, the model will learn the user's tolerance threshold for high brightness (900 nit), and in the future, it will automatically recommend 900 nit in the same environment and reduce the adjustment step from the default 50 nit to 20 nit (improve the adjustment fineness).

[0091] When multiple scene recognition conditions are met at the same time (such as when the user quickly operates in a strong light environment), the system handles them according to priority: fast interaction scene (priority 1) > strong light environment scene (priority 2) > multi-person sharing scene (priority 3) > focused use scene (priority 4) > distant observation scene (priority 5) > low light scene (priority 6). For example, when fast interaction occurs in a strong light environment, the system first executes the transient brightness increase logic of fast interaction, and then automatically restores the strong light scene parameters after the interaction is completed. When the scene is switched, a smooth transition algorithm (1 second fade duration) is used to avoid visual discomfort caused by sudden parameter changes.

[0092] The system of the embodiment achieves the following effects:

[0093] (1) Realize comprehensive adjustment based on multi-modal perception: By fusing environmental light, human distance, temperature and humidity, user gaze and other multi-dimensional data, the system breaks through the adjustment limitations of traditional single sensors, making the display parameters more suitable for actual use scenarios.

[0094] (2) Build a user behavior learning model: Through the edge computing module, the system records the user's manual adjustment habits of display parameters, forms personalized adjustment strategies, and realizes adaptive display effects for different users.

[0095] (3) Improve device linkage and energy efficiency: The system establishes a data interaction mechanism between display devices and associated devices, optimizes the display state using comprehensive environmental data, reduces energy consumption while ensuring user experience, and solves the problem of energy waste in the fixed parameter mode.

[0096] (4) Optimize special scene adaptation capability: For six typical scenes, such as low light, focused use, distant observation, strong light environment, multi-person sharing, and fast interaction, the system automatically switches the preset adjustment logic, avoids frequent manual operations by the user, and improves the overall performance and ease of use of the system.

[0097] The above describes in detail the adaptive display adjustment system based on multi-modal perception provided by the present application, and the principles and implementation manners of the present application are described by using specific examples; the above description of the examples is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed, and the above description should not be understood as a limitation on the present application.

Claims

1. A multi-modal perception based adaptive display adjustment system, characterized in that, The display device comprises a multi-modal sensing module, a communication module, an edge computing unit, a touch interaction module, an adaptive adjustment module, and a display module. The multi-modal sensing module is configured to collect multi-source environmental data and send the multi-source data to the edge computing unit, wherein the multi-source data comprises ambient light data, distance data, image data, and temperature and humidity data. The touch interaction module is configured to collect user manual adjustment instructions and send the user manual adjustment instructions to the edge computing unit. The communication module is configured to collect timestamps and send the timestamps to the edge computing unit. The edge computing unit is configured to receive the multi-source data, the timestamps, and the user manual adjustment instructions, perform data preprocessing and data fusion operations on the multi-source environmental data to obtain environmental features, associate the user manual adjustment instructions and the environmental features based on the timestamps as reference data, determine a current use scenario based on the reference data by invoking a scene recognition engine, predict adjustment parameters for the display module as user preference parameters by invoking a user behavior learning model, make a decision based on the current use scenario and the user preference parameters, generate adjustment instructions, and send the adjustment instructions to the adaptive adjustment module. The adaptive adjustment module is configured to adjust parameters of the display module based on the adjustment instructions.

2. The multimodal perception based adaptive display adjustment system of claim 1, wherein, The multi-modal sensing module is configured to perform data collection as follows: The ambient light sensor is configured to collect light intensity at regular intervals to obtain ambient light data. The infrared distance sensor is configured to collect the distance between the user and the device at regular intervals as distance data. The camera is configured to collect image data of the user at regular intervals, and the image data is used to determine whether the user is present and the user's line of sight direction. Correspondingly, the multi-source environmental data is preprocessed and fused as follows: The ambient light data is subjected to a moving average filter, the distance data is subjected to outlier rejection, the user's line of sight vector and the number of faces are extracted from the image data by an image data recognition algorithm, and the pixel brightness distribution of the screen area and the glare area are analyzed to obtain preprocessed multi-source data. The preprocessed multi-source environmental data is weighted and fused to obtain environmental features, wherein the weight of each type of data in the multi-source environmental data can be dynamically adjusted.

3. The multimodal perception based adaptive display adjustment system of claim 1, wherein, The user behavior learning model is a neural network model based on an LSTM network, which takes timestamps, environmental features, and user manual adjustment instructions as input and predicts adjustment parameters for the display module as output, wherein the adjustment parameters include brightness and color temperature adjustment values. The user behavior learning model is optimized and updated based on historical parameter data.

4. The multimodal perception based adaptive display adjustment system of claim 1, wherein, When making a decision based on the current use scenario and the user preference parameters, the decision is made according to predefined scene recognition and parameter adjustment strategies, which include the following: Low light scenario: when the light is less than the low light threshold of 50 lux and the time is between 22:00 and 6:00, the system determines that it is a low light scenario. Adjustment logic: brightness is reduced to below 100 nit, and color temperature is adjusted to 2700K warm light to reduce eye irritation. Focused use scenario: when the user is 0.5-1 m away, the line of sight is continuously focused on the display area for more than 3 seconds, and the illumination is 200-500 lux, the focused use scenario is triggered, and the adjustment logic is: the brightness is increased to 800 nit, the color temperature is adjusted to 5000K to simulate natural light to enhance the content clarity; at the same time, the refresh rate of the display module is reduced to 60Hz to balance the clarity and energy consumption; Long-distance observation scenario: when the user is more than 2 m away and the line of sight occasionally stays, it is determined as a long-distance observation scenario, and the adjustment logic is: the brightness is stable at 600 nit, the display content font is automatically enlarged to 120% of the original size, and the interface layout is simplified; Strong light environment scenario: when the illumination is >5000 lux and the image acquisition unit identifies that there are >=2 glare areas on the screen surface, the strong light environment scenario is activated, and the adjustment logic is: the hardware level increases the brightness to 1000 nit maximum value through the PWM dimming module, and drives the anti-reflection coating circuit, the software level switches to high contrast mode, and the font edge sharpening processing is used to offset the glare blur; Multi-person sharing scenario: when the image acquisition unit detects >=2 users, and the infrared distance sensor shows that the user distribution distance difference is >0.5 m, the multi-person sharing scenario is entered, and the adjustment logic is: the brightness is fixed at 600 nit, the color temperature is adjusted to 4500K neutral light, and the display content enables the sub-area adaptation mode; Fast interaction scenario: when the user triggers an operation through touch or voice, and the edge computing unit detects that the operation response time is <100 ms, it is determined as a fast interaction scenario, and the adjustment logic is: the brightness is temporarily increased by 10% based on the current value, the color temperature is temporarily switched to 5500K to strengthen the visual focus, the interface dynamic effect time is compressed to within 150 ms, unnecessary background animation is closed, and the operation feedback speed is prioritized; Among them, the scene priority from high to low is: fast interaction scenario > strong light environment scenario > multi-person sharing scenario > focused use scenario > long-distance observation scenario > low light scenario.

5. The multimodal perception based adaptive display adjustment system of claim 1, wherein, When the adaptive adjustment module adjusts the parameters of the display module based on the adjustment instruction, the scene switching uses a smooth transition algorithm.

6. The multimodal perception based adaptive display adjustment system of claim 1, wherein, The display module uses a multi-color backlight unit, supports continuous adjustment of brightness and color temperature, and integrates a PWM dimming module in the driving circuit.

7. The multimodal perception based adaptive display adjustment system of claim 1, wherein, The communication unit integrates Wi-Fi and ZigBee protocols, supports edge computing unit and environment gateway and other associated devices.

8. The multimodal perception based adaptive display adjustment system of claim 1, wherein, The multi-modal sensing module sequentially collects initial data from each sensor and uploads it to the edge computing unit, and automatically enables a redundancy algorithm if a sensor fails.

Images