Self-adaptive visual compensation system and method

Abstract

The invention relates to the technical field of visual compensation, and discloses a self-adaptive visual compensation system and method, and the system comprises a video analysis module, an environment perception module, a scene recognition module, a prediction module, a compensation strategy module, and an algorithm optimization module. Wherein the video analysis module is used for analyzing a video stream input in real time and extracting feature data of the video stream; the environment sensing module is used for collecting environment parameters; the scene recognition module is used for determining a scene type; the prediction module is used for predicting the change trend of the environmental parameters and the optimization target of the user; the compensation strategy module is used for determining a visual compensation strategy of the video stream; and the algorithm optimization module is used for adaptively selecting a visual compensation algorithm and performing visual compensation on the video stream. According to the method and the device, adaptive visual compensation of multiple scenes is realized, and the real-time performance and the accuracy of video stream processing in a complex environment are remarkably improved.

Description

Technical Field

[0001] This application relates to the technical field of visual compensation, and specifically relates to an adaptive visual compensation system and method. Background Technique

[0002] Current visual compensation technologies mainly rely on two methods: fixed-parameter compensation and offline training models. Fixed-parameter compensation processes images based on preset rules (such as illumination thresholds, fixed filter kernels). Typical applications include gamma correction of industrial cameras, electronic image stabilization of security cameras, etc. Such methods have high computational efficiency but are difficult to cope with dynamic environmental changes. Offline training models train dedicated compensation models through deep learning (such as CNN, GAN) and have achieved certain results in fields such as medical endoscopes and autonomous driving. However, the generalization ability of the models is limited by training data, and the computational resource consumption is large.

[0003] There are typical defects in the current mainstream visual compensation technologies in the industry. In fields such as industry, medicine, and security, independent technical routes are adopted, lacking a unified architecture, with fragmented scenarios and high cross-domain migration costs. When the system cannot identify the scene type, a general compensation mode is adopted, ignoring the differences in environmental characteristics and user needs. The compensation parameters of existing visual compensation systems are usually set empirically and cannot be adjusted in real time according to environmental changes. Existing systems do not incorporate the analysis of user operation intentions, resulting in a mismatch between the compensation results and actual needs.

[0004] For example, the patent application with the publication number CN112534467A discloses a system and method for contrast sensitivity compensation, which is used to correct the vision of users with insufficient vision to distinguish high spatial frequencies. This technical solution can use the user's contrast detection as a function of the spatial frequency in the image to correct the image in real time, but there are still problems raised in the background technique of this application: it cannot adjust the compensation strategy in real time according to environmental changes.

[0005] The information disclosed in this background section is only intended to enhance the overall understanding of this application and should not be regarded as an admission or any form of implication that this information constitutes prior art already known to those of ordinary skill in the art. Summary of the Invention

[0006] The technical problem to be solved by this application is to overcome the defects of the prior art and provide an adaptive visual compensation system and method to achieve adaptive visual compensation in multiple scenarios and improve the real-time performance and accuracy of video stream processing in complex environments.

[0007] To solve the above technical problems, this application provides the following technical solutions: On the one hand, this application provides an adaptive visual compensation system, including a video analysis module, an environment perception module, a scene recognition module, a prediction module, a compensation strategy module, and an algorithm optimization module; wherein: The video analysis module is used to analyze the real-time input video stream and extract the feature data of the video stream; The environment perception module is used to collect the environment parameters of the environment corresponding to the video stream; The scene recognition module performs scene recognition based on the environment parameters and the feature data of the video stream to determine the scene type; The prediction module is used to predict the change trend of the environment parameters; the prediction module is also used to predict the optimization goal of the user; The compensation strategy module determines the visual compensation strategy of the video stream based on the scene type, the change trend of the environment parameters, and the optimization goal of the user; The algorithm optimization module adaptively selects a visual compensation algorithm based on the visual compensation strategy and performs visual compensation on the video stream based on the visual compensation algorithm.

[0008] As a preferred solution of the adaptive visual compensation system described in this application, wherein: the feature data of the video stream includes resolution, frame rate, encoding format, geometric distortion degree, and color offset; The video analysis module includes an information extraction unit, a distortion analysis unit, and a color analysis unit; wherein, the information extraction unit is used to extract the resolution, frame rate, and encoding format of the video stream; The distortion analysis unit is used to extract the geometric distortion degree of the video stream; The color analysis unit is used to extract the color offset of the video stream; The environment parameters include environmental light intensity, lens acceleration, and operation trajectory data; The environment perception module includes a sensor unit and an operation recording unit; wherein, the sensor unit is used to collect the environmental light intensity and lens acceleration; The operation recording unit is used to obtain the operation trajectory data of the user; the operation trajectory data includes the screen coordinates of each touch operation of the user within the most recent m seconds. m is a positive integer.

[0009] As a preferred solution of the adaptive visual compensation system described in this application, wherein: the scene types include industrial inspection scenes, medical imaging scenes, and security monitoring scenes; The scene recognition module includes a scene classification unit and a scene verification unit; wherein, the scene classification unit calculates the confidence level of each scene type based on the video frames; The scene verification unit performs cross-verification based on the confidence level of each scene type and the feature data of the video stream to determine the scene type.

[0010] As a preferred solution of the adaptive visual compensation system described in this application, wherein: the prediction module includes an environment prediction unit and a target prediction unit; The environmental prediction unit is used to predict the change trend of the environmental parameters, specifically including: Extract the environmental light intensity and lens acceleration corresponding to the past consecutive frames of video and normalize them respectively; decompose the lens acceleration into the triaxial accelerations in a three-dimensional coordinate system; input the environmental light intensity and the triaxial acceleration of the lens corresponding to the past consecutive frames of video into a trained LSTM network model, and predict and output the environmental light intensity and the triaxial acceleration of the lens corresponding to the future consecutive frames of video; , Both are positive integers; Based on the environmental light intensity corresponding to the future consecutive frames of video, calculate the light intensity gradient; Based on the triaxial acceleration of the lens corresponding to the future consecutive frames of video, determine whether the lens has a sudden movement; The target prediction unit is used to predict the optimization target of the user, specifically including: extracting the screen coordinates of each touch operation of the user within the most recent m seconds based on the operation trajectory data, and generating the touch operation probability of each point in the video; Detect the moving area in the video stream; combine the touch operation probability and the moving area to identify the potential attention area of the user.

[0011] As a preferred solution of the adaptive visual compensation system described in this application, wherein: the compensation strategy module includes a first strategy unit; the first strategy unit is used to determine the visual compensation strategy of the video stream, specifically including: The first strategy unit is configured with a light intensity threshold and an acceleration threshold; if the environmental light intensity is lower than the light intensity threshold, the visual compensation strategy of the video stream includes a brightness compensation algorithm; If the lens acceleration is greater than the acceleration threshold, the visual compensation strategy of the video stream includes a blur correction algorithm; If the potential attention area of the user is recognized, the visual compensation strategy of the video stream further includes a local enhancement algorithm for locally enhancing the potential attention area.

[0012] As a preferred solution of the adaptive visual compensation system described in this application, wherein: the first strategy unit is further configured with a first distortion threshold and a second distortion threshold; if the current scene type is an industrial inspection scene, then when the geometric distortion degree is greater than the first distortion threshold, the visual compensation strategy of the video stream further includes a distortion compensation algorithm; otherwise, when the geometric distortion degree is greater than the second distortion threshold, the visual compensation strategy of the video stream further includes a distortion compensation algorithm; the first distortion threshold is greater than the second distortion threshold; The first policy unit is also configured with a first color difference threshold and a second color difference threshold; if the current scene type is a medical imaging scene, then when the color offset is greater than the first color difference threshold, the visual compensation policy for the video stream further includes a color correction algorithm; otherwise, when the color offset is greater than the second color difference threshold, the visual compensation policy for the video stream further includes a color correction algorithm; the first color difference threshold is less than the second color difference threshold.

[0013] As a preferred solution of the adaptive visual compensation system described in this application, wherein: the compensation policy module further includes a second policy unit; the second policy unit optimizes and adjusts the visual compensation policy based on the change trend of the environmental parameters, specifically including: The second policy unit is configured with a gradient threshold of the illumination intensity; if the illumination intensity gradient is greater than the gradient threshold and the visual compensation policy includes a brightness compensation algorithm, then increase the iteration frequency of the brightness compensation algorithm; If the camera undergoes a sudden movement and the visual compensation policy includes a blur correction algorithm, then add a multi-frame buffering mechanism to the blur suppression algorithm.

[0014] As a preferred solution of the adaptive visual compensation system described in this application, wherein: the algorithm optimization module includes an algorithm optimization unit; the algorithm optimization unit adaptively selects a visual compensation algorithm based on the visual compensation policy, specifically including: Obtain alternative algorithms for each algorithm included in the visual compensation policy; Generate an algorithm combination for the visual compensation policy; any group of the algorithm combinations includes one alternative algorithm for each algorithm included in the visual compensation policy; Set an optimization objective function based on the scene type; Calculate the optimization objective value for each group of algorithm combinations based on the optimization objective function; Select the algorithm combination with the largest optimization objective value and transmit it to the visual compensation unit.

[0015] As a preferred solution of the adaptive visual compensation system described in this application, wherein: the algorithm optimization module further includes a visual compensation unit; the visual compensation unit performs visual compensation on the video stream based on the visual compensation algorithm, specifically including: Preprocess the input video stream; the preprocessing includes frame alignment and noise suppression; The visual compensation unit is configured with alternative algorithms for each algorithm included in the visual compensation policy; based on the algorithm combination selected by the algorithm optimization unit, perform visual compensation on the video stream and output the video stream after visual compensation.

[0016] In a second aspect, this application provides an adaptive visual compensation method, including the following steps: Analyze the real-time input video stream to extract the feature data of the video stream; collect the environmental parameters of the environment corresponding to the video stream; Perform scene recognition based on the environmental parameters and the feature data of the video stream to determine the scene type; Predict the change trend of the environmental parameters and the optimization objectives of the user; Based on the scene type, the change trend of the environmental parameters, and the optimization objectives of the user, determine the visual compensation strategy for the video stream; Adaptively select a visual compensation algorithm based on the visual compensation strategy, and perform visual compensation on the video stream based on the visual compensation algorithm.

[0017] Compared with the prior art, the beneficial effects achieved by this application are as follows: The visual compensation system provided by this application has multi-scene adaptability. Through a unified system architecture, it adapts to the different requirements of different scenarios such as industry, medical care, and security, avoiding the redundant design of one solution for one scene in traditional technologies and achieving cross-domain compatibility.

[0018] This application uses an LSTM network to predict the light intensity gradient and the motion mutation trend, and adjusts the parameters of the visual compensation algorithm in advance to solve the problem of response lag in traditional methods.

[0019] By analyzing the user's operation trajectory and the content features of the screen, the potential attention areas of the user can be identified, improving the practical value of the compensation result. Dynamically configure the weights of the optimization function based on the scene type to achieve an accurate balance between computing resources and visual effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of this application, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. Among them: Figure 1 is a schematic structural diagram of an adaptive visual compensation system provided by this application; Figure 2 is a flowchart of an adaptive visual compensation method provided by this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The technical solutions of this application will be described in detail below through the drawings and specific embodiments. It should be understood that the specific features in the embodiments of this application and the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. Without conflict, the technical features in the embodiments of this application and the embodiments can be combined with each other.

[0022] Example 1 This example introduces an adaptive visual compensation system. Refer to Figure 1 , this system includes a video analysis module, an environmental perception module, a scene recognition module, a prediction module, a compensation strategy module, and an algorithm optimization module; among which: The video analysis module is used to analyze the real-time input video stream and extract the feature data of the video stream; The feature data of the video stream includes resolution, frame rate, encoding format, geometric distortion degree, and color offset; The video analysis module includes an information extraction unit, a distortion analysis unit, and a color analysis unit; among which, the information extraction unit is used to extract the resolution, frame rate, and encoding format of the video stream; the information extraction unit analyzes the metadata of the video stream or analyzes the video frames to obtain the width and height pixel values of the video, and then determines the resolution; by calculating the time interval between adjacent video frames, the frame rate is determined; the file header or metadata of the video stream is parsed to identify the encoding format used, such as H.264, H.265, etc.

[0023] The distortion analysis unit is used to extract the geometric distortion degree of the video stream; the distortion analysis unit finds feature points in the video frame, including corner points, edge points, etc.; uses the feature points to fit geometric distortion models, including but not limited to perspective distortion, radial distortion, etc.; based on the fitted distortion model, calculates the geometric distortion degree.

[0024] The color analysis unit is used to extract the color offset of the video stream. The color analysis unit extracts the color features of the video frame, including mean value, variance; compares the extracted color features with the preset reference color features, and calculates the color offset.

[0025] The environmental perception module is used to collect the environmental parameters of the environment corresponding to the video stream; The environmental parameters include environmental light intensity, lens acceleration, and operation trajectory data; The environmental perception module includes a sensor unit and an operation recording unit; among which, the sensor unit is used to collect the environmental light intensity and lens acceleration; The operation recording unit is used to obtain the operation trajectory data of the user; the operation trajectory data includes the screen coordinates of each touch operation of the user within the last m seconds. m is a positive integer. For example, m is set to 30, that is, the operation trajectory data is constructed by the screen coordinates of the user's touch operations within the last 30 seconds.

[0026] The scene recognition module performs scene recognition based on the environmental parameters and the feature data of the video stream to determine the scene type; The scene types include industrial inspection scenes, medical imaging scenes, and security monitoring scenes; The scene recognition module includes a scene classification unit and a scene verification unit; among them, the scene classification unit calculates the confidence of each scene type based on video frames; the scene classification unit is configured with a pre-trained ResNet-50 scene classifier for scene recognition according to video frames and outputs the confidence of the current scene for each scene type.

[0027] The scene verification unit performs cross-verification based on the confidence of each scene type and the feature data of the video stream to determine the scene type; the preferred methods for determining the scene type in this embodiment include: If the confidence of the medical imaging scene ≥ 0.8, the resolution in the video stream feature data ≥ 3840×2160 and the encoding format conforms to the DICOM standard, and at the same time the synthetic vector amplitude of the lens acceleration < 0.5g (g is the acceleration of gravity), it is determined as a medical imaging scene.

[0028] If the confidence of the industrial inspection scene ≥ 0.7, the frame rate of the video stream ≥ 60fps and there are periodic brightness mutations, such as a brightness peak appears every 0.5 ± 0.1 seconds, and at the same time the user has no touch operation for 10 consecutive seconds, it is determined as an industrial inspection scene; If the confidence of the security monitoring scene ≥ 0.6, and there are multiple moving targets in the video stream, such as ≥ 3 motion vector regions are detected by the optical flow method, and at the same time the encoding parameter is H.264 and the GOP length > 10 frames, it is determined as a security monitoring scene.

[0029] Furthermore, if it fails to confirm that the current scene is any one of the industrial inspection scene, medical imaging scene, and security monitoring scene, the scene type is marked as unknown, and basic adaptive visual compensation is performed on the video stream; the preferred basic adaptive visual compensation in this embodiment specifically includes: dynamically adjusting the global brightness and contrast according to the environmental light intensity, enabling non-linear stretching in low light, and starting highlight suppression in overexposure; judging the jitter degree based on the lens acceleration data, adaptively selecting an anti-shake algorithm, using electronic image stabilization for light jitter, and enabling multi-frame fusion for strong jitter; parsing the high-frequency interaction area in the operation trajectory data and preferentially performing local sharpening and noise reduction on this area; matching the basic enhancement algorithm according to the video stream resolution and frame rate, disabling super-resolution processing in low resolution, and strengthening temporal noise reduction in high frame rate.

[0030] The prediction module is used to predict the change trend of the environmental parameters; the prediction module is also used to predict the optimization target of the user; The prediction module includes an environment prediction unit and a target prediction unit; The environment prediction unit is used to predict the change trend of the environmental parameters, specifically including: Extract the past continuous Normalize the ambient light intensity and the lens acceleration corresponding to the frame video respectively; decompose the lens acceleration into the three-axis accelerations in a three-dimensional coordinate system; for the past consecutive frames of video, input the ambient light intensity and the three-axis accelerations of the lens into the trained LSTM network model, and predict and output the ambient light intensity and the three-axis accelerations of the lens corresponding to the future consecutive frames of video; , Both are positive integers; in this embodiment, preferably takes the value of 32, takes the value of 8. The changes in the ambient light intensity and the three-axis accelerations of the lens are continuous. In this embodiment, 32 frames of data are used to ensure that the input of the model contains sufficient information; the prediction ability of the LSTM network model gradually decreases as the output step length increases. Therefore, in this embodiment, 8 frames of shorter data are output, which can meet the requirements for predicting the change trend of the environmental parameters at the same time.

[0031] Based on the ambient light intensity corresponding to the future consecutive frames of video, calculate the light intensity gradient; Based on the three-axis accelerations of the lens corresponding to the future consecutive frames of video, determine whether the lens has a sudden movement. For example, calculate the combined acceleration of the three-axis accelerations corresponding to each frame in the future consecutive frames; if the combined acceleration of any frame exceeds the preset acceleration threshold, it is determined that the lens has a sudden movement.

[0032] The target prediction unit is used to predict the optimization target of the user, specifically including: extracting the screen coordinates of each touch operation of the user within the most recent m seconds based on the operation trajectory data, and generating the touch operation probability of each point in the video; Detect the moving area in the video stream; combine the touch operation probability and the moving area to identify the potential attention area of the user.

[0033] The method for the target prediction unit of the present application to identify the potential attention area of the user is as follows: use Gaussian kernel convolution to convert the discrete screen coordinates of the touch operation into the probability density distribution of the continuous touch probability, and construct a touch probability density map; detect the moving area from the video stream by using the lightweight optical flow method; increase the touch probability of the coordinate points in the probability density map corresponding to the moving area by 30%; set a touch probability threshold, and mark all coordinate points with a touch probability greater than the touch probability threshold as interest points; use adaptive threshold segmentation to generate the connected domain of the interest points as the potential attention area of the user.

[0034] The compensation strategy module determines the visual compensation strategy of the video stream based on the scene type, the change trend of the environmental parameters, and the optimization target of the user; The compensation strategy module includes a first strategy unit and a second strategy unit; The first strategy unit is used to determine the visual compensation strategy for the video stream, specifically including: The first strategy unit is configured with a light intensity threshold and an acceleration threshold; if the ambient light intensity is lower than the light intensity threshold, the visual compensation strategy for the video stream includes a brightness compensation algorithm; in this embodiment, the light intensity threshold is preferably 50 lux; the brightness compensation algorithm includes histogram equalization algorithm, Retinex algorithm, etc., which are used to enhance the image brightness.

[0035] If the lens acceleration is greater than the acceleration threshold, the visual compensation strategy for the video stream includes a blur correction algorithm; in this embodiment, 2 times the acceleration due to gravity is preferably the acceleration threshold; the blur correction algorithm includes Wiener filtering algorithm, Laplacian sharpening algorithm, etc., which are used to perform blur correction on the video stream; If a potential area of interest of the user is recognized, the visual compensation strategy for the video stream further includes a local enhancement algorithm, which is used to perform local enhancement on the potential area of interest; for example, a Gaussian mask is superimposed in the spatial domain of the video stream to enhance the details of the target area.

[0036] The first strategy unit is also configured with a first distortion threshold and a second distortion threshold; if the current scene type is an industrial inspection scene, when the geometric distortion degree is greater than the first distortion threshold, the visual compensation strategy for the video stream further includes a distortion compensation algorithm; otherwise, when the geometric distortion degree is greater than the second distortion threshold, the visual compensation strategy for the video stream further includes a distortion compensation algorithm; the first distortion threshold is greater than the second distortion threshold. The industrial inspection scene pays more attention to the geometric dimensions and position accuracy of the inspection target, such as the measurement of the assembly gap of parts, rather than the absolute deformation of the picture edge. Moderately retaining the distortion can exchange for a larger field of view coverage and improve the inspection efficiency. The distortion compensation algorithm includes polynomial correction method, bilinear interpolation method, etc., which are used to perform distortion compensation correction on the video stream.

[0037] The first strategy unit is also configured with a first color difference threshold and a second color difference threshold; if the current scene type is a medical imaging scene, when the color offset amount is greater than the first color difference threshold, the visual compensation strategy for the video stream further includes a color correction algorithm; otherwise, when the color offset amount is greater than the second color difference threshold, the visual compensation strategy for the video stream further includes a color correction algorithm; the first color difference threshold is less than the second color difference threshold. The medical imaging scene has a high demand for color discrimination, and a more sensitive color difference threshold is set to trigger color correction. The color correction algorithm includes color space conversion, gray world assumption, etc., which are used to perform color correction on the video stream.

[0038] The second strategy unit optimally adjusts the visual compensation strategy based on the change trend of the environmental parameters, specifically including: The second strategy unit is configured with a gradient threshold of light intensity; if the light intensity gradient is greater than the gradient threshold and the visual compensation strategy includes a brightness compensation algorithm, the iteration frequency of the brightness compensation algorithm is increased; for example, the basic iteration frequency of the brightness compensation algorithm is 30 Hz and it is increased to 60 Hz to match the change speed of the ambient light.

[0039] If a sudden movement occurs in the lens and the visual compensation strategy includes a blur correction algorithm, a multi-frame buffering mechanism is added to the blur suppression algorithm. Through the multi-frame buffering mechanism, it is ensured that the miscompensation rate of the blur suppression is reduced during sudden movement.

[0040] The algorithm optimization module adaptively selects a visual compensation algorithm based on the visual compensation strategy and performs visual compensation on the video stream based on the visual compensation algorithm.

[0041] The algorithm optimization module includes an algorithm optimization unit and a visual compensation unit; The algorithm optimization unit adaptively selects a visual compensation algorithm based on the visual compensation strategy, specifically including: Obtain alternative algorithms for each algorithm included in the visual compensation strategy; for example, the visual compensation strategy includes a distortion compensation algorithm, which describes a type of visual compensation algorithm, and the specific alternative algorithms corresponding to this type include polynomial correction method, bilinear interpolation method, perspective transformation correction method, etc., and these alternative algorithms can all achieve distortion compensation for the video stream.

[0042] Generate an algorithm combination of the visual compensation strategy; any group of the algorithm combinations includes one alternative algorithm for each algorithm included in the visual compensation strategy; Set an optimization objective function based on the scene type; Calculate the optimization objective value of each group of algorithm combinations based on the optimization objective function; Select the algorithm combination with the largest optimization objective value and transmit it to the visual compensation unit.

[0043] In this embodiment, the method of preferably setting the optimization objective function based on the scene type and calculating the optimization objective value of each group of algorithm combinations includes but is not limited to: Set real-time scores, resolution scores, motion sharpness scores, and low-light performance scores for each alternative algorithm of each algorithm included in the visual compensation strategy; among them, the real-time score is the ratio of the processing delay of the algorithm to the frame period; the resolution score is the resolution retention of the image output by the algorithm; the motion sharpness score is the reciprocal of the proportion of artifacts in the motion area after the algorithm processes; the low-light performance score is calculated based on the color difference after the algorithm processes. For example, for the adaptive color correction algorithm based on Retinex, since it has characteristics such as the need to iteratively calculate the illumination component, maintain the original resolution, excellent chromaticity noise suppression ability, and possible halo artifacts in the motion area, its resolution score and low-light performance score calculated according to the above rules are relatively high, while the real-time score and motion sharpness score are relatively low.

[0044] Perform weighted summation on the real-time score, resolution score, motion sharpness score, and low-light performance score of each alternative algorithm of each algorithm combination to obtain the comprehensive score of the alternative algorithm; among them, the weight values for weighted summation are set based on the scene type. For example: for the industrial inspection scene, industrial inspection is usually strongly related to the conveyor belt speed, and the algorithm delay must be much lower than the production line beat time, so the weight of the real-time score is relatively high, taking the value of 0.8; industrial parts usually have fixed geometric features and do not require ultra-high resolution, so the weight of the resolution score is relatively low, taking the value of 0.2; conveyor belt vibration / robot arm movement may cause dynamic blur, so the weight of the motion sharpness score is relatively high, taking the value of 0.5; the industrial environment illumination is relatively controllable and low-light situations are rare, so the weight of the low-light performance score is relatively low, taking the value of 0.2. Those skilled in the art can refer to the ideas provided in this embodiment and set the optimization objective function for the medical imaging scene and the security monitoring scene according to actual needs, which will not be elaborated here.

[0045] Calculate the sum of the comprehensive scores of each alternative algorithm in each algorithm combination to obtain the optimization objective value of the algorithm combination.

[0046] The visual compensation unit performs visual compensation on the video stream based on the visual compensation algorithm, specifically including: Preprocess the input video stream; in this embodiment, it is preferably that the preprocessing includes frame alignment and noise suppression; The visual compensation unit is configured with alternative algorithms of each algorithm included in the visual compensation strategy; based on the algorithm combination selected by the algorithm optimization unit, perform visual compensation on the video stream and output the video stream after visual compensation is completed.

[0047] Embodiment 2 This embodiment is the second embodiment of this application; based on the same inventive concept as Embodiment 1, refer to Figure 2 , this embodiment introduces an adaptive visual compensation method, including the following steps: Analyze the real-time input video stream to extract the feature data of the video stream; collect the environmental parameters of the environment corresponding to the video stream; the feature data includes resolution, frame rate, encoding format, degree of geometric distortion, and color offset; the environmental parameters include environmental light intensity, lens acceleration, and operation trajectory data; Based on the environmental parameters and the feature data of the video stream, perform scene recognition to determine the scene type; the scene types include industrial inspection scenes, medical imaging scenes, and security monitoring scenes; Predict the change trend of the environmental parameters and the optimization goals of the user; the change trends of the environmental parameters include light intensity gradient and whether there is a sudden movement of the lens; the optimization goals of the user include the potential areas of concern of the user.

[0048] Based on the scene type, the change trend of the environmental parameters, and the optimization goals of the user, determine the visual compensation strategy for the video stream; the visual compensation strategy includes brightness compensation algorithms, blur correction algorithms, local enhancement algorithms, distortion compensation algorithms, color correction algorithms, etc.; Based on the visual compensation strategy, adaptively select the visual compensation algorithm, and perform visual compensation on the video stream based on the visual compensation algorithm. Set different optimization objective functions based on the scene type, and select the best algorithm combination for visual compensation.

[0049] For the specific function implementation of the above steps, refer to the relevant content in the adaptive visual compensation system described in Embodiment 1, and details are not elaborated.

[0050] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose and scope of the present application, and all of these fall within the protection scope of the present application.

Claims

1. An adaptive visual compensation system, characterized in that: It includes a video analysis module, an environment perception module, a scene recognition module, a prediction module, a compensation strategy module, and an algorithm optimization module; among which: The video analysis module is used to analyze the real-time input video stream and extract the feature data of the video stream; The environment perception module is used to collect the environmental parameters of the environment corresponding to the video stream; The scene recognition module performs scene recognition based on the environmental parameters and the feature data of the video stream to determine the scene type; The prediction module is used to predict the change trend of the environmental parameters; the prediction module is also used to predict the optimization goal of the user; The compensation strategy module determines the visual compensation strategy of the video stream based on the scene type, the change trend of the environmental parameters, and the optimization goal of the user; the visual compensation strategy is a set of types of visual compensation algorithms; The algorithm optimization module adaptively selects a visual compensation algorithm based on the visual compensation strategy and performs visual compensation on the video stream based on the visual compensation algorithm; Adaptive selection of a visual compensation algorithm based on the visual compensation strategy specifically includes: setting a corresponding alternative algorithm for each visual compensation algorithm in the visual compensation strategy; determining which item in the corresponding alternative algorithms each visual compensation algorithm in the visual compensation strategy specifically is based on the scene type; Performing visual compensation on the video stream based on the visual compensation algorithm specifically includes: sequentially using each visually compensated algorithm selected adaptively to process the video stream.

2. The adaptive vision compensation system according to claim 1, characterized in that: The feature data of the video stream includes resolution, frame rate, encoding format, geometric distortion degree, and color offset; The video analysis module includes an information extraction unit, a distortion analysis unit, and a color analysis unit; among which, the information extraction unit is used to extract the resolution, frame rate, and encoding format of the video stream; The distortion analysis unit is used to extract the geometric distortion degree of the video stream; The color analysis unit is used to extract the color offset of the video stream; The environmental parameters include environmental light intensity, lens acceleration, and operation trajectory data; The environment perception module includes a sensor unit and an operation recording unit; among which, the sensor unit is used to collect the environmental light intensity and lens acceleration; The operation recording unit is used to obtain the operation trajectory data of the user; the operation trajectory data includes the screen coordinates of each touch operation of the user within the most recent m seconds; m is a positive integer.

3. An adaptive visual compensation system according to claim 2, characterized in that: The scene types include industrial inspection scenes, medical imaging scenes, and security monitoring scenes; The scene recognition module includes a scene classification unit and a scene verification unit; among which, the scene classification unit calculates the confidence level of each scene type based on video frames; The scene verification unit performs cross-verification based on the confidence level of each scene type and the feature data of the video stream to determine the scene type.

4. The adaptive vision compensation system according to claim 3, wherein: The prediction module includes an environment prediction unit and a target prediction unit; The environment prediction unit is used to predict the change trend of the environmental parameters, specifically including: Extract the past consecutive frame video corresponding ambient light intensity and camera acceleration and normalize them respectively; Decompose the camera acceleration into three-axis accelerations in a three-dimensional coordinate system; For the past consecutive frame video corresponding ambient light intensity and the three-axis accelerations of the camera, input them into the trained LSTM network model, predict and output the ambient light intensity and the three-axis accelerations of the camera corresponding to the future consecutive frame video; , are all positive integers; Based on the ambient light intensity corresponding to future consecutive frames of video, calculate the light intensity gradient; Based on the three-axis acceleration of the shot corresponding to future consecutive frames of video, determine whether there is a sudden change in the movement of the shot; The target prediction unit is used to predict the optimization goal of the user, specifically including: extracting the screen coordinates of each touch operation of the user within the most recent m seconds based on the operation trajectory data and generating the touch operation probability of each point in the video; Detect the moving area in the video stream; combine the touch operation probability and the moving area to identify the potential attention area of the user.

5. An adaptive visual compensation system according to claim 4, wherein: The compensation strategy module includes a first strategy unit; the first strategy unit is used to determine the visual compensation strategy of the video stream, specifically including: The first strategy unit is configured with a light intensity threshold and an acceleration threshold; if the ambient light intensity is lower than the light intensity threshold, the visual compensation strategy of the video stream includes a brightness compensation algorithm; If the lens acceleration is greater than the acceleration threshold, the visual compensation strategy of the video stream includes a blur correction algorithm; If the potential attention area of the user is recognized, the visual compensation strategy of the video stream further includes a local enhancement algorithm for locally enhancing the potential attention area.

6. An adaptive visual compensation system according to claim 5, characterized in that: The first strategy unit is also configured with a first distortion threshold and a second distortion threshold; if the current scene type is an industrial inspection scene, when the geometric distortion degree is greater than the first distortion threshold, the visual compensation strategy of the video stream further includes a distortion compensation algorithm; otherwise, when the geometric distortion degree is greater than the second distortion threshold, the visual compensation strategy of the video stream further includes a distortion compensation algorithm; the first distortion threshold is greater than the second distortion threshold; The first strategy unit is also configured with a first color difference threshold and a second color difference threshold; if the current scene type is a medical imaging scene, when the color offset is greater than the first color difference threshold, the visual compensation strategy of the video stream further includes a color correction algorithm; otherwise, when the color offset is greater than the second color difference threshold, the visual compensation strategy of the video stream further includes a color correction algorithm; the first color difference threshold is less than the second color difference threshold.

7. An adaptive visual compensation system according to claim 6, characterized in that: The compensation strategy module further includes a second strategy unit; the second strategy unit optimizes and adjusts the visual compensation strategy based on the change trend of the environmental parameters, specifically including: The second strategy unit is configured with a gradient threshold of light intensity; if the light intensity gradient is greater than the gradient threshold and the visual compensation strategy includes a brightness compensation algorithm, the iteration frequency of the brightness compensation algorithm is increased; If the lens undergoes a sudden movement and the visual compensation strategy includes a blur correction algorithm, a multi-frame buffer mechanism is added to the blur suppression algorithm.

8. An adaptive visual compensation system according to claim 7, characterized in that: The algorithm optimization module includes an algorithm optimization unit; The algorithm optimization unit adaptively selects a visual compensation algorithm based on the visual compensation strategy, specifically including: Obtain the alternative algorithms of each algorithm included in the visual compensation strategy; Generate an algorithm combination of the visual compensation strategy; any group of the algorithm combinations includes one alternative algorithm of each algorithm included in the visual compensation strategy; Set an optimization objective function based on the scene type; Calculate the optimization objective value of each group of algorithm combinations based on the optimization objective function; Select the algorithm combination with the largest optimization objective value and transmit it to the visual compensation unit.

9. An adaptive visual compensation system according to claim 8, characterized in that: The algorithm optimization module further includes a visual compensation unit; the visual compensation unit performs visual compensation on the video stream based on the visual compensation algorithm, specifically including: Preprocess the input video stream; the preprocessing includes frame alignment and noise suppression; The visual compensation unit is configured with alternative algorithms for each algorithm included in the visual compensation strategy; based on the algorithm combination selected by the algorithm optimization unit, visual compensation is performed on the video stream, and the video stream after visual compensation is output.

10. An adaptive visual compensation method, which is implemented based on an adaptive visual compensation system as described in any one of claims 1-9, characterized in that: The method includes the following steps: Analyze the video stream input in real time, and extract the feature data of the video stream; Collect the environmental parameters of the environment corresponding to the video stream; Based on the environmental parameters and the feature data of the video stream, perform scene recognition to determine the scene type; Predict the change trend of the environmental parameters and the optimization target of the user; Based on the scene type, the change trend of the environmental parameters, and the optimization target of the user, determine the visual compensation strategy of the video stream; Based on the visual compensation strategy, adaptively select a visual compensation algorithm, and perform visual compensation on the video stream based on the visual compensation algorithm.

Images