An ISP parameter automatic calibration method and system based on multi-modal semantic feedback

Abstract

The application discloses a kind of based on multimodal semantic feedback's image signal processor parameter automatic calibration method and system, by the calibration control terminal of communication connection with image signal processor simulator execution, including the following steps: obtaining the original image sequence and scene label of target scene, according to the scene label determines the tuning gravity center of multiple image quality evaluation dimensions, initializes the image signal processor simulator and loads the image signal processing module algorithm model of target system on chip, establishes the ideal target coordinate V_gold corresponding to the target scene, and initializes the image signal processor parameter vector S to be calibrated and learning rate η;Current image signal processor parameter vector S is input into the image signal processor simulator, the original image sequence is processed using the image signal processor simulator and output rendering image, the rendering image and the semantic prompt word including multiple image quality evaluation dimensions are input into visual language model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an automatic ISP parameter calibration method and system based on multimodal semantic feedback. Background Technology

[0002] An Image Signal Processor (ISP) is a crucial image processing unit in a camera device. It processes the raw image data output from the image sensor, typically involving multiple steps such as exposure, gain adjustment, white balance, noise reduction, sharpening, gamma correction, wide dynamic range processing, color correction, and local contrast adjustment. The parameter configurations of each processing module within the ISP directly affect the image quality of the final output image, including sharpness, noise level, color reproduction, contrast, shadow detail, and highlight suppression.

[0003] In the development and mass production of security surveillance cameras and intelligent video equipment, it is typically necessary to calibrate ISP parameters for different application scenarios, such as courtyard night scenes, indoor corridors, backlit entrances, and parking lot glare. Traditional ISP parameter calibration mainly relies on image engineers manually adjusting parameters based on the target scene and the actual output image. Engineers repeatedly adjust parameters such as exposure, white balance, noise reduction, sharpening, and wide dynamic range processing, and determine the parameter configuration by observing the adjusted image effect. This method relies heavily on the individual experience of engineers, and different engineers may have different understandings of subjective image quality requirements such as "clear image," "natural skin tone," "controllable noise," and "rich detail in dark areas," which can easily lead to differences in calibration results among different personnel. At the same time, when there are many parameters to be adjusted, or when the scene, image sensor, and hardware platform change, multiple rounds of adjustment and verification need to be performed again, resulting in a long ISP parameter calibration cycle.

[0004] To reduce the degree of human intervention in ISP parameter adjustment, existing technologies have proposed parameter search methods based on objective image quality indicators such as peak signal-to-noise ratio (PSNR) and structural similarity. This involves establishing an image quality evaluation function and using grid search, gradient search, or other optimization methods to find the appropriate ISP parameters. However, indicators such as PSNR and structural similarity primarily reflect pixel differences or image structure differences. Their correlation with the subjective image quality perception of human eyes, such as color naturalness, image clarity, highlight perception, and shadow detail, is not always consistent. Therefore, when objective indicators are used solely as optimization targets, the resulting ISP parameter configuration may not fully meet the actual scene-specific image quality requirements.

[0005] For example, Chinese invention patent application CN116091900A discloses an image processing method, apparatus, and storage medium. It processes an original image to obtain a target image based on ISP parameters, calls a pre-trained image quality assessment model to obtain a target assessment result, and adjusts the ISP parameters based on the target assessment result. The target assessment result may also include objective quality information and subjective quality information in dimensions such as brightness, color, contrast, sharpness, and noise. This scheme can utilize a neural network model to evaluate image quality and assist in the automatic adjustment of ISP parameters, but its parameter adjustment still mainly relies on the scoring results obtained from the image quality assessment model and the corresponding preset parameter relationships.

[0006] For example, Chinese invention patent application CN119450239A discloses an image signal processor optimization method and device. This method uses an ISP simulator to process raw sample images according to an ISP configuration parameter set, obtains unsupervised evaluation scores through an unsupervised pre-trained visual task model, and then updates the ISP configuration parameter set based at least on the unsupervised evaluation scores, thus forming an automatic ISP parameter optimization process. This approach reduces reliance on manually labeled data and expert adjustments, and enables iterative updates to the ISP configuration parameters.

[0007] For example, Chinese invention patent application CN120372881A discloses a method, apparatus, electronic device and storage medium for determining ISP parameters. Iteratively obtains multiple ISP parameter sets in the search space, processes the original image according to the ISP parameter sets to obtain multiple RGB images, and performs image quality evaluation on the multiple RGB images according to the reference image. If the evaluation result does not meet the image quality requirements, the parameter search iteration continues until the target ISP parameter set that meets the requirements is obtained.

[0008] Therefore, existing technologies can already achieve a certain degree of automatic search and adjustment of ISP parameters through image quality assessment models, unsupervised visual models, or reference image assessment results. However, in complex scenarios, there is still room for further improvement in ISP parameter calibration.

[0009] On the one hand, image quality typically involves multiple evaluation dimensions, including sharpness, noise level, color reproduction, contrast, shadow detail, highlight suppression, skin tone naturalness, and motion blur. Different application scenarios place varying degrees of emphasis on each evaluation dimension. How to transform scenario-specific subjective image quality requirements into multi-dimensional quantitative feedback that can be used for closed-loop calibration of ISP parameters, and how to establish a unified and calculable deviation between the current image effect and the desired image effect, remains a problem that automatic ISP calibration needs to solve.

[0010] On the other hand, the different parameters in ISP are not independent of each other. For example, changes in noise reduction parameters may simultaneously affect image details and sharpening effects, while changes in wide dynamic range processing parameters and gamma parameters may simultaneously affect highlight and shadow areas. When multiple parameters have coupled effects on the same or different image quality dimensions, if the parameters are adjusted separately based solely on the overall image quality score, it is easy for one parameter to improve a certain image quality metric while another parameter has a reverse effect. This causes the parameters to change repeatedly during multiple iterations, thereby affecting the stability and convergence efficiency of the parameter optimization process.

[0011] Furthermore, the ISP register addresses, parameter bit widths, parameter upper and lower limits, minimum quantization step sizes, and dependencies between parameters may differ across different on-chip systems. If the continuous parameter values ​​obtained by the automatic optimization algorithm cannot be further correlated and verified with the physical constraints of the actual on-chip system's registers, problems may still arise such as the optimized parameters not being directly configurable on the target hardware platform, parameters exceeding the register range, or parameters losing their expected adjustment effect after quantization.

[0012] Therefore, there is a need for an automatic ISP parameter calibration scheme that can combine multidimensional image semantic evaluation with automatic ISP parameter adjustment, quantify the influence relationship between parameters and image perception effect, suppress conflict updates between strongly coupled parameters, detect oscillation states during the iteration process, and adapt to the hardware parameter constraints of the target on-chip system. Summary of the Invention

[0013] To address the problems existing in the prior art, this invention provides an automatic ISP parameter calibration method and system based on multimodal semantic feedback.

[0014] To achieve the above objectives, this invention provides an automatic ISP parameter calibration method based on multimodal semantic feedback, executed by a calibration control terminal communicatively connected to an image signal processor simulator, comprising the following steps: S1. Obtain the original image sequence and scene label of the target scene, determine the optimization focus of multiple image quality evaluation dimensions based on the scene label, initialize the image signal processor simulator and load the image signal processing module algorithm model of the target on-chip system, establish the ideal target coordinate V_gold corresponding to the target scene, and initialize the image signal processor parameter vector S and learning rate η to be calibrated. S2. Input the current image signal processor parameter vector S into the image signal processor simulator, process the original image sequence using the image signal processor simulator and output a rendered image, input the rendered image and semantic prompts including multiple image quality evaluation dimensions into the visual language model, extract the multidimensional semantic features of the rendered image through the visual language model and form a perceptual vector V, and determine the perceptual vector V as the current perceptual coordinate V_curr according to the same dimensional order and normalization rule as the ideal target coordinate V_gold; S3. Calculate the perception deviation ΔΦ based on the current perception coordinate V_curr and the ideal target coordinate V_gold; when the perception deviation ΔΦ is less than the preset convergence threshold ε, execute step S8; when the perception deviation ΔΦ is not less than the preset convergence threshold ε, execute step S4. S4. Apply positive and negative perturbations to each parameter s_i in the image signal processor parameter vector S respectively. Under different perturbation conditions, obtain rendered images through the image signal processor simulator respectively, and extract the corresponding perception vectors according to step S2. The perception vectors obtained under the perturbation conditions are processed according to the same dimensional order and normalization rule as the ideal target coordinates V_gold. Construct the parameter-perception sensitivity Jacobian matrix J_S based on the perturbation amount of each parameter and the corresponding change in the perception vector. S5. Based on the parameter-sensory sensitivity Jacobian matrix J_S and the difference between the current sensing coordinate V_curr and the ideal target coordinate V_gold, calculate the image signal processor parameter adjustment amount ΔS; analyze the correlation between each column vector in the parameter-sensory sensitivity Jacobian matrix J_S, identify strongly coupled parameter pairs, and construct a conflict suppression operator Ω based on the correlation degree of the strongly coupled parameter pairs and the optimization centroid; use the conflict suppression operator Ω to correct the image signal processor parameter adjustment amount ΔS, obtain the corrected parameter adjustment amount ΔS′, and obtain the updated image signal processor parameter vector S_new based on the corrected parameter adjustment amount ΔS′; S6. Record the perceptual deviations corresponding to multiple iterations to form a historical sequence of perceptual deviations, and determine whether oscillations occur during the parameter update process based on the historical sequence of perceptual deviations; when oscillations occur, decay the learning rate η, and re-execute step S4 using the decayed learning rate; when no oscillations occur, execute step S7. S7. Read the register specification description file of the target on-chip system, and perform upper and lower bound truncation, quantization step size alignment, register bit width check and dependency condition verification on the updated image signal processor parameter vector S_new according to the register specification description file. Load the image signal processor parameter vector that passes the verification as the current image signal processor parameter vector S into the image signal processor simulator, and return to step S2. S8. Serialize the image signal processor parameter vector that meets the preset convergence threshold ε into a static preset file adapted to the target on-chip system, and generate the corresponding verification digest file.

[0015] Preferably, in step S1, the ideal target coordinates V_gold are established by any of the following methods: inputting a manually tuned reference image into the visual language model, extracting the perceptual vector of the manually tuned reference image, and establishing the ideal target coordinates V_gold based on the perceptual vector; Select an aesthetic template that matches the scene tag from the scene template library, and establish the ideal target coordinate V_gold based on the aesthetic template; or, obtain the target value and dimension weight of each image quality evaluation dimension set through the interactive interface, and establish the ideal target coordinate V_gold based on the target value; wherein, the dimension weight of the image quality evaluation dimension is determined according to the scene tag, the setting information received through the interactive interface, or the aesthetic template.

[0016] Preferably, in step S2, extracting the multidimensional semantic features and forming the perceptual vector V includes: scaling the rendered image to the input resolution of the visual language model, and cropping or marking the region of interest in the rendered image according to the scene label; The processed rendered image and the semantic prompts are input into the visual language model. The semantic prompts include sharpness, noise level, color reproduction, contrast, shadow detail, highlight suppression, skin tone naturalness, and motion blur. The feature values ​​of each dimension output by the visual language model are normalized to 0 to 1, and the negative indicators corresponding to noise, overexposure and motion blur are reverse-transformed so that the feature values ​​of each dimension satisfy the condition that the larger the value, the better the image quality. The perceptual vectors V_f,r of the global region and at least one region of interest are extracted from the multi-frame rendered images respectively, and then weighted and fused according to the following formula: V=Σ_fΣ_rγ_f·α_r·V_f,r; where γ_f represents the frame weight of the f-th frame rendered image, α_r represents the region weight of the r-th region, and the sum of the frame weights Σ_fγ_f=1, and the sum of the region weights Σ_rα_r=1.

[0017] Preferably, in step S3, the perception deviation ΔΦ is calculated according to the following formula: ΔΦ=‖V_curr-V_gold‖2=sqrt[Σ_i(V_curr,i-V_gold,i)²]; where V_curr,i represents the coordinate value of the current perception coordinate V_curr in the i-th image quality evaluation dimension, and V_gold,i represents the coordinate value of the ideal target coordinate V_gold in the i-th image quality evaluation dimension; When setting the dimensional weights of each image quality evaluation dimension, the weighted perceptual bias ΔΦ_w is calculated according to the following formula: ΔΦ_w=sqrt[Σ_d w_d·(V_curr,d-V_gold,d)² / Σ_d w_d]; where w_d represents the dimensional weight of the d-th image quality evaluation dimension, and the weighted perceptual bias ΔΦ_w is used as the perceptual bias ΔΦ in step S3 for convergence judgment.

[0018] Preferably, in step S4, the perturbation δ_i of parameter s_i is determined according to the following formula: δ_i = max{p_i·(s_i, max-s_i, min), q_i}; where p_i is 0.5%~2%, s_i, max and s_i, min represent the physical upper limit and physical lower limit of parameter s_i, respectively, and q_i represents the minimum quantization step size of parameter s_i on the target on-chip system; The column vector corresponding to parameter s_i in the parameter-sensing sensitivity Jacobian matrix J_S is calculated according to the following formula: J_S(:,i)=[V(s_i+δ_i)-V(s_i-δ_i)] / (2δ_i); where V(s_i+δ_i) and V(s_i-δ_i) represent the sensing vectors obtained after applying positive and negative perturbations to parameter s_i, respectively; the sensing vectors corresponding to the positive and negative perturbations are processed according to the same dimensional order and normalization rule as the ideal target coordinate V_gold; The parameter-sensory sensitivity Jacobian matrix J_S is reconstructed every L iterations or when an oscillation is detected; where L is the preset reconstruction interval of the parameter-sensory sensitivity Jacobian matrix, and L is an integer greater than or equal to 1; in high-noise scenarios, perceptual vectors are extracted from multiple frames of rendered images obtained under the same perturbation conditions, and the median of the components of the same image quality evaluation dimension in each perceptual vector is taken, and the perceptual vector used to calculate the column vector is composed of the median of each dimension component.

[0019] Preferably, in step S5, the image signal processor parameter adjustment amount ΔS is calculated according to the following formula: ΔS=-η·J_S^T·(V_curr-V_gold); The correlation coefficient ρ_ij between the sensitivity column vectors corresponding to parameters s_i and s_j is calculated according to the following formula: Only when the norms of all the sensitivity column vectors corresponding to the parameters are not 0, the correlation coefficient is calculated as ρ_ij = |J_S(:,i)·J_S(:,j)| / [‖J_S(:,i)‖·‖J_S(:,j)‖]; when the norm of any column vector is 0, ρ_ij is set to 0, and the corresponding parameters are not determined as a strongly coupled parameter pair; when the correlation coefficient ρ_ij is large... When the relevant threshold ρ_th is equal to or greater than the relevant threshold ρ_th, the parameters s_i and s_j are determined as a strongly coupled parameter pair, and the attenuation coefficient C_ij is determined according to the following formula: C_ij=clip[1-λ·(ρ_ij-ρ_th) / (1-ρ_th),C_min,1]; where λ represents the suppression intensity, C_min represents the minimum retention ratio, and clip represents the clipping function; the relevant threshold ρ_th and the corresponding included angle threshold θ_th satisfy ρ_th=cosθ_th; The dominant parameter in the strongly coupled parameter pair is determined according to the dimension weight corresponding to the current scene. The update amount of the dominant parameter is retained, and the update amount of the non-dominant parameter or the synchronous update amount of the strongly coupled parameter pair is attenuated by the attenuation coefficient C_ij to form the conflict suppression operator Ω in the form of a diagonal matrix or a block diagonal matrix. The adjustment amount of the correction parameter ΔS′ and the updated image signal processor parameter vector S_new are obtained according to the following formulas: ΔS′=Ω·ΔS; S_new=S+ΔS′.

[0020] Preferably, in step S6, the perceptual deviation of the most recent N_hist iterations is recorded, where N_hist ≥ 5; when there are K consecutive iterations that simultaneously meet the following conditions, it is determined that the parameter update process has oscillated: |ΔΦ_t-ΔΦ_(t-2)|<ε_osc; (ΔΦ_t-ΔΦ_(t-1))·(ΔΦ_(t-1)-ΔΦ_(t-2))<0; where K≥2, ΔΦ_t, ΔΦ_(t-1) and ΔΦ_(t-2) represent the perception deviations corresponding to the t-th, t-1 and t-2 iterations respectively, ε_osc represents the oscillation threshold, which is 0.01~0.05; after detecting oscillation, the decayed learning rate η′ is calculated according to the following formula: η′=η·β; where β is 0.5~0.8; when η′<η_min, η′ is set to η_min, where η_min is 1 / 10~1 / 100 of the initial learning rate; when oscillation is still detected after reducing the learning rate M times consecutively, an alarm message is output and calibration is paused, where M≥3.

[0021] Preferably, in step S7, the register specification description file includes platform identifier, image signal processing module name, parameter name, register address, bit offset, bit width, lower bound of parameter, upper bound of parameter, quantization step size, default value, scaling factor, serialization field name, and dependency condition field. The calibration control terminal reverse-calculates the normalized parameters into register physical parameter values, and then performs lower bound verification, upper bound verification, and quantization alignment on the updated parameter value s_new according to the following formula: Lower bound check: s_new = max(s_new, s_min); Upper bound check: s_new = min(s_new, s_max); Quantization alignment: s_new = round[(s_new - s_min) / Δs] × Δs + s_min; where s_min represents the lower bound of the parameter value, s_max represents the upper bound of the parameter value, and Δs represents the quantization step size of the parameter value; perform register bit width check on the quantized and aligned parameter value, and perform enable status check, working mode check, and mutual exclusion parameter check according to the dependency condition field.

[0022] Preferably, in step S8, according to the serialization protocol corresponding to the target on-chip system, the image signal processor parameter vector that meets the preset convergence threshold is serialized into a static preset file in XML, JSON, CONF, or BIN format; The verification summary file includes scene labels, visual language model version, source of ideal target coordinates V_gold, initial perceptual deviation, final perceptual deviation, number of iterations, number of learning rate decays, oscillation detection records, initial values, final values ​​and changes of each image quality evaluation dimension, platform identifier of the target on-chip system, hash value and version number of the static preset file.

[0023] The present invention also provides an automatic calibration system for image signal processor parameters based on multimodal semantic feedback, including an image signal processor simulator and a calibration control terminal communicatively connected to the image signal processor simulator; The image signal processor simulator is used to load the image signal processing module algorithm model of the target system-on-a-chip, process the original image sequence of the target scene based on the current image signal processor parameter vector, and output the rendered image. The calibration control terminal includes: The semantic feature extraction module is used to input the rendered image and semantic prompts including multiple image quality evaluation dimensions into a visual language model, extract multi-dimensional semantic features of the rendered image through the visual language model, and form a perceptual vector V. The target quality coordinate quantization module is used to establish the ideal target coordinate V_gold of the target scene, determine the perception vector V as the current perception coordinate V_curr according to the same dimensional order and normalization rule as the ideal target coordinate V_gold, and calculate the perception deviation based on the current perception coordinate V_curr and the ideal target coordinate V_gold. The sensitivity analysis module is used to apply positive and negative perturbations to each parameter in the parameter vector of the image signal processor when the perception deviation does not meet the preset convergence condition. The perception vector obtained under the perturbation condition is processed according to the same dimensional order and normalization rule as the ideal target coordinate V_gold. The parameter-perception sensitivity Jacobian matrix J_S is constructed based on the perturbation amount of each parameter and the corresponding change in the perception vector. The conflict suppression and parameter update module is used to calculate the image signal processor parameter adjustment amount based on the parameter-sensory sensitivity Jacobian matrix J_S, analyze the correlation between each column vector in the parameter-sensory sensitivity Jacobian matrix J_S to identify strongly coupled parameter pairs, construct a conflict suppression operator Ω based on the strongly coupled parameter pairs, and use the conflict suppression operator Ω to correct the image signal processor parameter adjustment amount to obtain the updated image signal processor parameter vector. An oscillation detection module is used to form a historical sequence of perception deviations based on the perception deviations corresponding to multiple iterations, detect whether oscillations occur during the parameter update process based on the historical sequence of perception deviations, and reduce the learning rate when oscillations are detected. The hardware constraint verification module is used to read the register specification description file of the target on-chip system, perform upper and lower bound truncation, quantization step size alignment, register bit width check and dependency condition verification on the updated image signal processor parameter vector according to the register specification description file, and load the verified image signal processor parameter vector into the image signal processor simulator. The serialization delivery module is used to serialize the current image signal processor parameter vector into a static preset file adapted to the target on-chip system when the perception deviation meets the preset convergence condition, and generate a corresponding verification digest file.

[0024] The technical solution of this invention has the following beneficial effects: First, this invention inputs the rendered image output by the image signal processor simulator and semantic cue words including multiple image quality evaluation dimensions into the visual language model, extracts the multidimensional semantic features of the rendered image and forms a perceptual vector, establishes the ideal target coordinates corresponding to the target scene, and calculates the perceptual deviation between the current perceptual coordinates and the ideal target coordinates. This allows image quality requirements such as sharpness, noise level, color reproduction, contrast, dark detail, highlight suppression, skin tone naturalness, and motion blur to be uniformly represented as a calculable multidimensional feedback signal, thereby helping to reduce the result differences caused by relying solely on the subjective observation of engineers for ISP parameter calibration.

[0025] Second, the present invention can determine the optimization focus corresponding to different image quality evaluation dimensions based on the scene label of the target scene, and can combine multiple frames of images and regions of interest to perform perceptual feature fusion, so that ISP parameter calibration can be optimized in a targeted manner according to the image quality focus corresponding to different scenes, reducing the interference of non-critical regions or non-critical quality dimensions on the overall calibration results, and improving the targeting of scene-based ISP parameter calibration.

[0026] Third, this invention applies positive and negative perturbations to the ISP parameters respectively, and constructs a parameter-sensory sensitivity Jacobian matrix based on the changes in the perception vector caused by the parameter perturbations. This quantifies the influence of different ISP parameters on each dimension of image quality evaluation, so that the direction of parameter adjustment is no longer determined solely by a single image quality score, but can be calculated in combination with the actual impact of different parameters on multidimensional perception results, which is beneficial to improving the pertinence of parameter update direction.

[0027] Fourth, this invention identifies strongly coupled parameter pairs by analyzing the correlation between the column vectors corresponding to each parameter in the parameter-sensory sensitivity Jacobian matrix, and constructs a conflict suppression operator based on the degree of correlation between the strongly coupled parameters and the tuning focus corresponding to the current scene, thereby correcting the adjustment amount of the ISP parameters. This can suppress the mutual interference caused by the simultaneous large-scale adjustment of multiple strongly coupled parameters, and reduce the adverse effects of the coupling between parameters such as noise reduction, sharpening, gamma, and wide dynamic range processing on the iterative process.

[0028] Fifth, this invention records the perception deviations corresponding to multiple iterations and forms a historical sequence of perception deviations. When oscillations are detected during the parameter update process, the learning rate is reduced, and the reduced learning rate is used to re-analyze the parameter sensitivity and update the parameters. This enables the system to dynamically adjust the parameter update amplitude according to the actual iteration state, reducing the repeated changes of parameters between multiple configuration states, which is beneficial to improving the stability and convergence ability of the automatic calibration process.

[0029] Sixth, by reading the register specification description file corresponding to the target on-chip system, this invention performs upper and lower bound truncation, quantization step size alignment, register bit width check, and dependency condition verification on the updated ISP parameters, so that the parameter optimization process is combined with the actual hardware register constraints, reducing the problems of optimized parameters exceeding the hardware allowable range, not being able to be configured according to the register step size, or conflicting with the working conditions of other parameters, and improving the feasibility of automatic calibration results when actually deployed to the target on-chip system.

[0030] Seventh, after the perception deviation meets the preset convergence condition, the present invention serializes the final ISP parameter vector into a static preset file adapted to the target on-chip system and generates a corresponding verification summary file. This enables the formation of a closed-loop calibration process from scene input, semantic quality evaluation, automatic iteration of ISP parameters, hardware constraint verification to parameter file output. This reduces the workload required for repeated manual adjustment and manual conversion of parameter files in the traditional ISP calibration process, which is conducive to improving the efficiency of ISP scene-based parameter calibration and mass production configuration, and facilitates subsequent parameter version management, verification and traceability. Attached Figure Description

[0031] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2 This is a flowchart of the automatic calibration method for image signal processor parameters based on multimodal semantic feedback according to the present invention; Figure 3 This is a schematic diagram of the target quality coordinate space and the quantization of perceived deviation in this invention; Figure 4 This is a schematic diagram of the parameter-sensing sensitivity Jacobian matrix of the present invention; Figure 5 This is a schematic diagram of the convergence curve of the perception deviation of the present invention; Figure 6 This is a schematic diagram illustrating the mechanism of the conflict suppression operator of the present invention; Figure 7 This is a schematic diagram of the single quality score and multi-dimensional sensing vector output format of the present invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0033] Reference Figures 1 to 7This invention provides an automatic parameter calibration system and method for image signal processors based on multimodal semantic feedback, applicable to scenario-based parameter calibration of image signal processors in products such as security monitoring cameras and intelligent video devices. Image signal processor (ISP); Vision-Language Model (VLM); System on Chip (SoC); Raw Image Format (RAW); Region of Interest (ROI); Red, Green, and Blue (RGB); Wide Dynamic Range (WDR); Peak Signal-to-Noise Ratio (PSNR); Structural Similarity Index Measure (SSIM); Mean Opinion Score (MOS).

[0034] like Figure 1 As shown, the automatic calibration system for image signal processor parameters in this embodiment includes an image signal processor simulator and a calibration control terminal communicatively connected to the image signal processor simulator. The calibration control terminal includes a semantic feature extraction module, a target quality coordinate quantization module, a sensitivity analysis module, a collision suppression and parameter update module, an oscillation detection module, a hardware constraint verification module, and a serialization delivery module.

[0035] The image signal processor simulator is used to load the image signal processing module algorithm model of the target on-chip system, receive the image signal processor parameter vector output by the calibration control terminal, use the parameter vector to process the original image sequence of the target scene, and output an RGB rendered image with red, green and blue channels.

[0036] The semantic feature extraction module is used to load the visual language model, extract multi-dimensional semantic features from the rendered image output by the image signal processor simulator, and form a perceptual vector V.

[0037] The target quality coordinate quantization module is used to establish the ideal target coordinate V_gold corresponding to the target scene. Following the same dimensional order and normalization rules as the ideal target coordinate V_gold, the perception vector V is determined as the current perception coordinate V_curr, and the perception deviation ΔΦ between the current perception coordinate V_curr and the ideal target coordinate V_gold is calculated.

[0038] The sensitivity analysis module applies positive and negative perturbations to each parameter in the image signal processor parameter vector. The perturbation-obtained perception vector is processed according to the same dimensional order and normalization rule as the ideal target coordinate V_gold. Based on the perturbation amount of each parameter and the change in perception vector before and after perturbation, a parameter-perception sensitivity Jacobian matrix J_S is constructed, thereby quantifying the influence of each image signal processor parameter on different semantic evaluation dimensions.

[0039] The conflict suppression and parameter update module is used to calculate the parameter adjustment amount of the image signal processor based on the parameter-sensory sensitivity Jacobian matrix J_S, perform correlation analysis on the column vectors in the Jacobian matrix, identify strongly coupled parameter pairs, and construct a conflict suppression operator Ω based on the correlation degree of the strongly coupled parameter pairs and the tuning focus corresponding to the current scene. The conflict suppression operator Ω is used to correct the parameter adjustment amount of the image signal processor.

[0040] The oscillation detection module is used to record the perceptual deviations corresponding to multiple iterations, forming a historical sequence of perceptual deviations, and to detect whether oscillations occur during the parameter update process based on the historical sequence of perceptual deviations; when oscillations are detected, the current learning rate is decayed.

[0041] The hardware constraint verification module is used to read the register specification description file of the target on-chip system, perform upper and lower bound truncation, quantization step size alignment, register bit width check and dependency condition verification on the updated image signal processor parameters, and load the verified image signal processor parameters into the image signal processor simulator.

[0042] The serialization delivery module is used to serialize the final image signal processor parameters into a static preset file adapted to the target on-chip system when the perception deviation meets the preset convergence conditions, and to generate the corresponding verification digest file.

[0043] like Figure 2 As shown, the automatic calibration method for image signal processor parameters in this embodiment includes the following steps.

[0044] S1. Initialize the scene and establish semantic anchors.

[0045] The calibration control unit loads the RAW image sequence of the target scene offline and parses the scene label corresponding to the target scene. The scene label can be a courtyard night scene, a corridor backlit scene, or a parking lot with strong light, etc. The calibration control unit determines the optimization focus of each image quality evaluation dimension based on the scene label. For example, for a courtyard night scene, priority is given to improving the dimension weights corresponding to noise level, shadow detail, and sharpening; for a corridor backlit scene, priority is given to improving the dimension weights corresponding to highlight suppression, facial detail, and local contrast.

[0046] The calibration control terminal initializes the image signal processor simulator and loads the image signal processing module algorithm model of the target on-chip system.

[0047] The parameters of the image signal processor to be calibrated are represented by a parameter vector S, where S∈R^N, and N represents the number of adjustable parameters. Adjustable parameters include exposure time, analog gain, digital gain, white balance R-channel gain, white balance G-channel gain, white balance B-channel gain, Gamma curve control points, wide dynamic range compression coefficient, 2D noise reduction intensity, 3D noise reduction intensity, sharpening intensity, color correction matrix parameters, and local contrast parameters.

[0048] The ideal target coordinates V_gold corresponding to the target scene can be established in any of the following ways.

[0049] The first method involves reading a manually tuned reference image, inputting the manually tuned reference image into a visual language model, extracting its perceptual vector through the visual language model, and using the extracted perceptual vector as the ideal target coordinates V_gold.

[0050] The second approach involves selecting an aesthetic template from a scene template library that matches the scene label of the target scene, and establishing the ideal target coordinates V_gold based on the target values ​​corresponding to each image quality evaluation dimension in the aesthetic template.

[0051] The third method involves engineers setting target values ​​and weights for each image quality evaluation dimension through an interactive interface, and then establishing ideal target coordinates V_gold based on the set target values.

[0052] The target value and dimension weights set through the interactive interface can also be used to adjust the ideal target coordinates V_gold formed by scene templates or manually tuned reference images.

[0053] S2. Extract multidimensional image quality perception features.

[0054] The calibration control terminal inputs the current image signal processor parameter vector S into the image signal processor simulator. The image signal processor simulator uses the current image signal processor parameter vector S to process the RAW image sequence and outputs an RGB rendered image.

[0055] The semantic feature extraction module performs unified preprocessing on the RGB rendered image, including scaling the RGB rendered image to the input resolution of the visual language model, and cropping or marking the region of interest in the RGB rendered image according to the scene label of the target scene.

[0056] The semantic feature extraction module inputs the processed RGB rendered image and semantic cues into the visual language model. The semantic cues include at least eight image quality evaluation dimensions: sharpness, noise level, color reproduction, contrast, shadow detail, highlight suppression, skin tone naturalness, and motion blur. For the same calibration task, the same visual language model version, the same semantic cues template, and the same output order for the image quality evaluation dimensions are consistently used.

[0057] The visual language model outputs feature values ​​for each image quality evaluation dimension and normalizes these feature values ​​to ensure they range from 0 to 1. For negative indicators such as noise, overexposure, and motion blur, the corresponding feature values ​​are inversely transformed to ensure that all image quality evaluation dimensions satisfy the condition that higher values ​​indicate better image quality.

[0058] The multidimensional semantic features output by the visual language model are arranged into a perceptual vector according to a preset dimensional order: V=[v_1, v_2,..., v_D]; Where D represents the number of semantic evaluation dimensions, and v_1 to v_D represent the normalized feature values ​​corresponding to different image quality evaluation dimensions.

[0059] For multi-frame images or images containing multiple regions of interest, semantic features are extracted from the global region and at least one region of interest in each frame to obtain perceptual vectors V_f,r, which are then fused according to the following formula: V=Σ_fΣ_rγ_f·α_r·V_f, r; Where γ_f represents the frame weight corresponding to the f-th frame image, and α_r represents the region weight corresponding to the r-th region.

[0060] For key areas such as faces, license plates, and entrance highlights, the corresponding region weight α_r can be set higher than the region weight of the background area to avoid the image quality index of the background area masking the image quality changes of the key area.

[0061] S3. Quantify the perceptual bias.

[0062] The target quality coordinate quantization module determines the perception vector V as the current perception coordinate V_curr corresponding to the current image signal processor parameter configuration, according to the same dimensional order and normalization rules as the ideal target coordinate V_gold.

[0063] The ideal target coordinates V_gold and the current perceived coordinates V_curr use the same image quality evaluation dimension order and normalization rules. For example... Figure 3As shown, the distance between the current perceived coordinate V_curr and the ideal target coordinate V_gold is used to represent the perceived deviation between the current image signal processor parameter configuration and the ideal image quality target.

[0064] The perception deviation ΔΦ is calculated using the following formula: ΔΦ=‖V_curr-V_gold‖2=sqrt[Σ_i(V_curr,i-V_gold,i)²]; Where V_curr,i represents the coordinate value of the current perceived coordinate V_curr in the i-th image quality evaluation dimension, and V_gold,i represents the coordinate value of the ideal target coordinate V_gold in the i-th image quality evaluation dimension. The smaller the perceptual deviation ΔΦ, the closer the image quality corresponding to the current image signal processor parameter configuration is to the ideal target.

[0065] When setting the dimensional weights for each image quality evaluation dimension, the weighted perceptual bias ΔΦ_w is calculated using the following formula: ΔΦ_w=sqrt[Σ_d w_d·(V_curr,d-V_gold,d)² / Σ_d w_d]; where w_d represents the dimensional weight corresponding to the d-th image quality evaluation dimension, V_curr,d represents the coordinate value of the current perceptual coordinate V_curr in the d-th image quality evaluation dimension, and V_gold,d represents the coordinate value of the ideal target coordinate V_gold in the d-th image quality evaluation dimension.

[0066] When using weighted sensing bias ΔΦ_w, the weighted sensing bias ΔΦ_w is used as the sensing bias ΔΦ for convergence judgment.

[0067] In this embodiment, the preset convergence threshold ε is set to 0.1. When the perception deviation ΔΦ is less than 0.1, it indicates that there is no significant subjective image quality difference between the current image signal processor parameter configuration and the ideal target, and step S8 is executed; when the perception deviation ΔΦ is not less than 0.1, step S4 is executed.

[0068] S4. Construct the parameter-sensing sensitivity Jacobian matrix.

[0069] The sensitivity analysis module performs perturbation sampling on the current image signal processor parameter vector S. For each parameter s_i in the parameter vector S, a positive perturbation +δ_i and a negative perturbation -δ_i are applied respectively.

[0070] The perturbation δ_i of parameter s_i is determined based on the physical adjustment range of parameter s_i and the minimum quantization step size corresponding to the target on-chip system: δ_i = max{p_i·(s_i, max-s_i, min), q_i}; where p_i is 0.5%~2%, s_i, max and s_i, min represent the physical upper limit and physical lower limit of parameter s_i, respectively, and q_i represents the minimum quantization step size of parameter s_i on the target chip system.

[0071] In one specific implementation, p_i is set to 1%, meaning the perturbation amount is the larger of 1% of the physical adjustment range of the parameter and the minimum quantization step size q_i. This avoids invalid sampling caused by the perturbation amount being less than the register quantization step size.

[0072] Under the two conditions of parameter s_i+δ_i and parameter s_i-δ_i, the same RAW image is processed using an image signal processor simulator, and the corresponding perceptual vectors V(s_i+δ_i) and V(s_i-δ_i) are extracted through a visual language model.

[0073] like Figure 4 As shown, the column vector corresponding to parameter s_i in the parameter-sensory sensitivity Jacobian matrix J_S is calculated according to the following formula: J_S(:,i)=[V(s_i+δ_i)-V(s_i-δ_i)] / (2δ_i).

[0074] Following the above method, the N parameters in the parameter vector S are perturbed sequentially to construct the parameter-sensory sensitivity Jacobian matrix: J_S=[ V / s_1, V / s_2, ..., V / s_N]∈R^(D×N).

[0075] Each column in the parameter-sensory sensitivity Jacobian matrix J_S represents the degree of influence of an image signal processor parameter on each image quality evaluation dimension, and each row represents the degree of influence of different image signal processor parameters on the same image quality evaluation dimension.

[0076] The parameter-sensing sensitivity Jacobian matrix J_S can be reconstructed every L iterations. Here, L is the preset reconstruction interval for the parameter-sensing sensitivity Jacobian matrix, and L is an integer greater than or equal to 1. When the oscillation detection module detects oscillations during the parameter update process, it also reconstructs the parameter-sensing sensitivity Jacobian matrix J_S to prevent the previously constructed sensitivity matrix from becoming invalid in the nonlinear parameter range.

[0077] For high-noise scenes, multiple frames of images are rendered under the same perturbation conditions. The perceptual vectors corresponding to each frame are extracted. The median of the components of the same image quality evaluation dimension in each perceptual vector is taken. The median of each dimension component is used to form the perceptual vector for calculating the parameter-perceptual sensitivity Jacobian matrix J_S, so as to reduce the influence of single-frame random noise on the sensitivity calculation results.

[0078] S5, perform conflict suppression and parameter updates.

[0079] The conflict suppression and parameter update module calculates the image signal processor parameter adjustment amount ΔS based on the parameter-sensory sensitivity Jacobian matrix J_S: ΔS=-η·J_S^T·(V_curr-V_gold); where η represents the current learning rate, and J_S^T represents the transpose of the parameter-sensory sensitivity Jacobian matrix J_S.

[0080] To identify the coupling relationship between parameters, the angle between the sensitivity column vectors corresponding to parameters s_i and s_j is calculated: θ_ij=arccos{|J_S(:,i)·J_S(:,j)| / [‖J_S(:,i)‖·‖J_S(:,j)‖]}.

[0081] When the included angle θ_ij is less than the preset strong correlation threshold θ_th, it is determined that the parameters s_i and s_j have a strong coupling relationship. The preset strong correlation threshold θ_th ranges from 10° to 30°.

[0082] The degree of coupling between parameters can also be determined based on the correlation coefficient between the sensitivity column vectors: ρ_ij=|J_S(:,i)·J_S(:,j)| / [‖J_S(:,i)‖·‖J_S(:,j)‖].

[0083] The correlation coefficient ρ_ij is the absolute cosine of the angle between the sensitivity column vectors corresponding to parameters s_i and s_j. When the correlation coefficient ρ_ij is greater than or equal to the correlation threshold ρ_th, the parameters s_i and s_j are determined as a strongly coupled parameter pair, and the correlation threshold ρ_th and the angle threshold θ_th satisfy ρ_th = cosθ_th.

[0084] For strongly coupled parameter pairs, the attenuation coefficient C_ij is determined according to the following formula: C_ij=clip[1-λ·(ρ_ij-ρ_th) / (1-ρ_th), C_min, 1]; Where λ represents the suppression intensity, C_min represents the minimum retention ratio, and clip represents the clipping function, which is used to limit the calculation results to between C_min and 1.

[0085] When the update directions of two parameters in a strongly coupled parameter pair simultaneously improve the same image quality evaluation dimension but deteriorate another image quality evaluation dimension, the strongly coupled parameter pair is marked as a conflicting coupled parameter pair. The dominant parameter in the conflicting coupled parameter pair is determined based on the dimension weights corresponding to the current scene. The update amount of the dominant parameter is retained, and the update amount of the non-dominant parameter is multiplied by a decay coefficient C_ij.

[0086] When two strongly coupled parameters correspond to different image signal processing modules but affect the same image quality evaluation dimension, the conflict suppression operator Ω is divided into blocks according to the image signal processing module. Priority is given to suppressing the synchronous large-step update of multiple parameters within the same image signal processing module, while a small learning rate update amount is reserved for parameters between different image signal processing modules.

[0087] Based on the above processing results, a conflict suppression operator Ω in the form of a diagonal matrix or a block diagonal matrix is ​​formed, and the adjustment amount of the correction parameter ΔS′ is obtained according to the following formula: ΔS′=Ω·ΔS.

[0088] The updated image signal processor parameter vector S_new is obtained according to the following formula: S_new = S + ΔS′.

[0089] By attenuating the update amount of strongly coupled parameters through the conflict suppression operator Ω, the mutual interference between parameters such as noise reduction and sharpening, wide dynamic range compression and Gamma curve can be reduced.

[0090] S6. Perform oscillation detection and adaptive learning rate adjustment.

[0091] The oscillation detection module records the sensing deviation corresponding to the most recent N_hist iterations, forming a historical sequence of sensing deviations {ΔΦ_t}, where N_hist≥5.

[0092] When there are K consecutive iterations that simultaneously satisfy the following conditions, it is determined that the parameter update process oscillates: |ΔΦ_t-ΔΦ_(t-2)|<ε_osc; (ΔΦ_t-ΔΦ_(t-1))·(ΔΦ_(t-1)-ΔΦ_(t-2))<0; where K≥2, ΔΦ_t, ΔΦ_(t-1) and ΔΦ_(t-2) represent the perceptual deviations corresponding to the t-th, t-1-th and t-2-th iterations, respectively, and ε_osc represents the oscillation threshold, with a value range of 0.01~0.05.

[0093] The first condition indicates that the perception deviation in round t is close to that in round t-2, and the second condition indicates that the changes in perception deviation in two adjacent rounds have opposite signs. When both conditions are true, it means that the parameter update process oscillates between at least two states.

[0094] After detecting oscillation, the decayed learning rate η′ is calculated according to the following formula: η′=η·β; where β is the learning rate decay coefficient, and the value of β ranges from 0.5 to 0.8.

[0095] Set a lower bound for the learning rate η_min, where η_min is 1 / 10 to 1 / 100 of the initial learning rate. When η′ < η_min, set η′ to η_min. Re-execute step S4 using the updated learning rate and reconstruct the parameter-sensory sensitivity Jacobian matrix J_S.

[0096] If oscillation is still detected after reducing the learning rate M times consecutively, an alarm message is output and calibration is paused, waiting for adjustment of the tuning center or manual processing, where M≥3.

[0097] S7. Perform hardware constraint verification.

[0098] The hardware constraint verification module reads the register specification description file of the target on-chip system. The register specification description file includes platform identifier, image signal processing module name, parameter name, register address, bit offset, bit width, lower bound of parameter, upper bound of parameter, quantization step size, default value, scaling factor, serialization field name, and dependency condition field.

[0099] A set of fields in the register specification description file can be represented as: platform = Hi3516; module = NR; name = nr_strength; addr = 0x12043010; bit_offset = 0; bit_width = 8; min = 0; max = 255; step = 1; default = 96; scale = 1.0; serialize_key=isp.nr.strength.

[0100] For parameters with dependencies, the register specification file may also include enable_key, mode_key, or a mutex parameter name to prevent the generation of parameter combinations that will not work on the target on-chip system.

[0101] The hardware constraint verification module first converts the normalized parameters into register physical parameter values ​​based on the proportional conversion coefficient, and then performs lower bound verification, upper bound verification and quantization alignment in sequence.

[0102] The lower bound check is performed according to the following formula: s_new = max(s_new, s_min).

[0103] The upper bound check is performed according to the following formula: s_new = min(s_new, s_max).

[0104] Quantization alignment is performed according to the following formula: s_new = round[(s_new - s_min) / Δs] × Δs + s_min; Where s_min represents the lower bound of the parameters, s_max represents the upper bound of the parameters, and Δs represents the quantization step size.

[0105] After completing the upper and lower bound checks and quantization alignment, the register bit width check is performed according to the bit width field in the register specification description file, and the enable status check, working mode check and mutual exclusion parameter check are performed according to the dependency condition field.

[0106] The validated image signal processor parameter vector is loaded into the image signal processor simulator as the current image signal processor parameter vector S, and the process returns to step S2 to enter the next iteration.

[0107] S8. Perform result serialization and delivery.

[0108] When the perception deviation ΔΦ is less than the preset convergence threshold ε, the serialization delivery module serializes the final image signal processor parameter vector into a static preset file in XML, JSON, CONF, or BIN format according to the serialization protocol corresponding to the target on-chip system. The static preset file is a parameter file generated according to the serialization protocol of the target on-chip system and used to store and deploy the image signal processor parameter configuration.

[0109] A verification summary file is generated synchronously. The verification summary file includes scene labels, visual language model version, source of ideal target coordinates V_gold, initial perceptual deviation ΔΦ_initial, final perceptual deviation ΔΦ_final, number of iterations, number of learning rate decays, oscillation detection records, initial values, final values ​​and changes of each image quality evaluation dimension, platform identifier of the target on-chip system, hash value and version number of static preset files.

[0110] The verification summary file is used for mass production traceability, platform migration verification, and manual review.

[0111] Example 1: Courtyard Night Scene Calibration This embodiment calibrates the image signal processor parameters for the camera's courtyard night scene, requiring the image to meet the quality targets of image clarity, controllable noise, and preservation of details in dark areas.

[0112] Load a sequence of 100 RAW images of a courtyard night scene, with a resolution of 2560×1440. Based on the courtyard night scene label, set the weights for noise reduction, sharpening, wide dynamic range, and color reproduction to 0.4, 0.2, 0.2, and 0.2, respectively.

[0113] Initialize the image signal processor simulator and load the noise reduction, sharpening, gamma and wide dynamic range module algorithm models of the target system-on-chip Hi3516.

[0114] Engineers provide manually optimized reference images, and a visual language model is used to extract the perceptual vectors from these reference images, which are then used as the ideal target coordinates V_gold. Alternatively, a night scene transparency aesthetic template from the scene template library can be selected as the ideal target.

[0115] The first iteration used the manufacturer's default parameters, and the visual language model calculated an initial perceptual deviation ΔΦ of 0.85, with the main source of deviation being the noise dimension.

[0116] After the fifth iteration, the noise reduction gain was improved under the guidance of the parameter-sensory sensitivity Jacobian matrix J_S, and the sensing bias ΔΦ was reduced to 0.42.

[0117] In the twelfth iteration, the noise reduction parameters and sharpening parameters satisfied the following for three consecutive iterations: |ΔΦ_t-ΔΦ_(t-2)|<0.03; Furthermore, the signs of the changes in perception deviation alternate between adjacent rounds, indicating that oscillations occur during the parameter update process when the noise reduction and sharpening parameters are jointly updated. The learning rate decay coefficient β is set to 0.6, the lower limit of the learning rate η_min is set to 1 / 20 of the initial learning rate, and step S4 is re-executed to construct the parameter-perception sensitivity Jacobian matrix J_S.

[0118] After reducing the learning rate, the oscillations disappeared, and the closed-loop iteration continued. In the eighteenth iteration, the perceptual bias ΔΦ decreased to 0.06, which is less than the preset convergence threshold of 0.1, and the calibration process converged.

[0119] The Hi3516 register specification file is read. The noise reduction gain ranges from 0 to 255, and the quantization step size is 1. The sharpening intensity ranges from 0 to 63, and the quantization step size is 1. After the parameters are truncated to their upper and lower bounds, the quantization step size is aligned, and the register bit width is checked, they meet the hardware constraints of the Hi3516.

[0120] Reference Figure 5The final parameters are serialized into a static preset file in BIN format adapted for Hi3516, and a JSON format verification digest file is generated simultaneously. The verification digest file records the final perceptual deviation ΔΦ_final as 0.06, the number of iteration rounds as 18, and the initial, final, and change values ​​of each image quality evaluation dimension.

[0121] Example 2: Corridor Backlighting Scene Calibration This embodiment calibrates the image signal processor parameters for a corridor entrance scene with strong backlighting to improve the overexposure problem in the face area and enhance facial details and local contrast.

[0122] Based on the corridor backlight scene label, the weight of the wide dynamic range dimension is set to 0.5, and the weight of the noise reduction dimension is set to 0.1.

[0123] In the rendered image, three regions of interest are set: the face region, the background region, and the entrance highlight region. The region weight of the face region is set to 0.5, the region weight of the background region is set to 0.3, and the region weight of the entrance highlight region is set to 0.2.

[0124] The visual language model performs semantic evaluation on the three regions of interest respectively, and performs weighted fusion based on the region weights corresponding to each region to avoid the overexposure of the background region from masking the quality changes of the face region.

[0125] During the iteration process, the parameter-sensory sensitivity Jacobian matrix J_S shows that the angle θ_ij between the column vectors corresponding to the wide dynamic range compression curve parameter and the Gamma curve parameter is 15°, which is less than the preset strong correlation threshold of 20°, indicating that the two parameters constitute a strongly coupled parameter pair.

[0126] The conflict suppression and parameter update module sets the corresponding attenuation coefficient C_ij to 0.6 and uses the conflict suppression operator Ω to reduce the mutual interference generated when the parameters of the wide dynamic range compression curve and the Gamma curve are updated synchronously and significantly.

[0127] After 15 rounds of iteration, the perceptual deviation is less than the preset convergence threshold, the detail retention of the face region is improved by about 40%, and the entrance highlight suppression reaches the preset target.

[0128] Example 3: Cross-platform heterogeneous delivery In this embodiment, the calibration results of the same scenario are adapted to the Hi3516 platform and the RV1106 platform respectively.

[0129] For the Hi3516 platform, read the register specification file corresponding to the Hi3516. The noise reduction intensity range for the Hi3516 platform is 0–255, and the sharpening intensity range is 0–63. After parameter verification, serialize the parameters into a static preset file in BIN or XML format corresponding to the Hi3516.

[0130] For the RV1106 platform, read the corresponding register specification file. The noise reduction intensity range for the RV1106 platform is 0–1023, and the sharpening intensity range is 0–255.

[0131] When the corresponding parameter value ranges of the two platforms are different, the parameter range mapping is performed according to the following formula: s_mapped=round[(s_orig-min_old) / (max_old-min_old)×(max_new-min_new)+min_new]; Where s_orig represents the original platform parameter value, min_old and max_old represent the lower and upper bounds of the corresponding parameters on the original platform, min_new and max_new represent the lower and upper bounds of the corresponding parameters on the target platform, and s_mapped represents the parameter value after mapping to the target platform.

[0132] After completing the parameter range mapping, the upper and lower bounds are truncated, the quantization step size is aligned, the register bit width is checked, and the dependency conditions are verified according to the RV1106 register specification description file. The parameters that pass the verification are then serialized into a static preset file in CONF or JSON format.

[0133] The static preset files corresponding to the two platforms were loaded into the respective image signal processor simulators. The evaluation results of the visual language model show that the final perceptual deviation ΔΦ_final of the Hi3516 platform is 0.06, and the final perceptual deviation ΔΦ_final of the RV1106 platform is 0.08, both less than 0.1, indicating that the image quality of the two platforms has a high degree of consistency.

[0134] Example 4: Multi-user collaborative calibration This embodiment uses a combination of automatic calibration and fine-tuning by engineers to calibrate the image signal processor parameters.

[0135] First, the calibration system performs automatic closed-loop calibration to obtain the first version of the calibration result, with a perception deviation ΔΦ of 0.08.

[0136] After reviewing the first version of the calibration results, the engineers determined that there was still room for improvement in the color reproduction dimension. They then increased the weight of the color reproduction dimension from 0.2 to 0.4 via an interactive interface and adjusted the target value corresponding to the color reproduction dimension in the ideal target coordinate V_gold.

[0137] The calibration system inherits the image signal processor parameter vector corresponding to the calibration results of the first version as the initial parameters for incremental iteration. After 5 rounds of incremental iteration, the perceptual deviation ΔΦ is reduced to 0.05, and the color reproduction dimension reaches the preset target.

[0138] The system saves the automatic calibration results as version v1.0 and the calibration results finely adjusted by engineers as version v1.1. It also records the ideal target coordinates, dimension weights, final parameters, perceptual deviations, and static preset files corresponding to the two versions, and supports comparison and rollback between different versions.

[0139] Example 5: Simulation Verification To verify the feasibility of the method of this invention, simulations were conducted on courtyard night scenes, corridor backlighting, parking lot strong light, and cross-platform migration scenarios. The simulation sample settings are shown in Table 1 below.

[0140] Table 1

[0141] The simulation comparison results of different parameter tuning methods are shown in Table 2 below.

[0142] Table 2

[0143] The convergence process of the perception bias is shown in Table 3 below.

[0144] Table 3

[0145] The simulation results of cross-platform migration are shown in Table 4 below.

[0146] Table 4

[0147] Based on the simulation results above, the method of this invention can converge within 15 to 20 iterations, and the final perception bias ΔΦ_final is reduced to 0.06 to 0.08. Compared with manual parameter tuning, the calibration time for a single scene is reduced from approximately 360 to 420 minutes to approximately 28 to 36 minutes.

[0148] Compared with convolutional neural network evaluation methods that only output a single image quality score, this invention uses a visual language model to form a multi-dimensional perceptual vector that includes sharpness, noise level, color reproduction, contrast, dark detail, highlight suppression, skin tone naturalness, and motion blur. This vector can track the changing trends of different image quality evaluation dimensions and set corresponding dimensional weights according to different scenarios.

[0149] By using the parameter-sensory sensitivity Jacobian matrix J_S, the influence of different image signal processor parameters on each dimension of image quality evaluation can be determined; by using the conflict suppression operator Ω and the oscillation detection mechanism, the oscillation caused by the synchronous adjustment of strongly coupled parameters can be reduced; by using the register specification description file, the parameter range, register bit width, quantization step size and serialization format adaptation between different on-chip systems can be completed, thereby realizing the automated, semantic, quantizable and traceable calibration of image signal processor parameters.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Without departing from the technical concept of the present invention, those skilled in the art can make corresponding adjustments or equivalent substitutions to the above embodiments. The scope of protection of the present invention shall be determined by the scope defined in the claims.

Claims

1. A method for automatic ISP parameter calibration based on multimodal semantic feedback, characterized in that, Performed by a calibration control terminal that communicates with the image signal processor simulator, the following steps are included: S1. Obtain the original image sequence and scene label of the target scene, determine the optimization focus of multiple image quality evaluation dimensions based on the scene label, initialize the image signal processor simulator and load the image signal processing module algorithm model of the target on-chip system, establish the ideal target coordinate V_gold corresponding to the target scene, and initialize the image signal processor parameter vector S and learning rate η to be calibrated. S2. Input the current image signal processor parameter vector S into the image signal processor simulator, process the original image sequence using the image signal processor simulator and output a rendered image, input the rendered image and semantic prompts including multiple image quality evaluation dimensions into the visual language model, extract the multidimensional semantic features of the rendered image through the visual language model and form a perceptual vector V, and determine the perceptual vector V as the current perceptual coordinate V_curr according to the same dimensional order and normalization rule as the ideal target coordinate V_gold; S3. Calculate the perception deviation ΔΦ based on the current perception coordinate V_curr and the ideal target coordinate V_gold; when the perception deviation ΔΦ is less than the preset convergence threshold ε, execute step S8; when the perception deviation ΔΦ is not less than the preset convergence threshold ε, execute step S4. S4. Apply positive and negative perturbations to each parameter s_i in the image signal processor parameter vector S respectively. Under different perturbation conditions, obtain rendered images through the image signal processor simulator respectively, and extract the corresponding perception vectors according to step S2. The perception vectors obtained under the perturbation conditions are processed according to the same dimensional order and normalization rule as the ideal target coordinates V_gold. Construct the parameter-perception sensitivity Jacobian matrix J_S based on the perturbation amount of each parameter and the corresponding change in the perception vector. S5. Based on the parameter-sensory sensitivity Jacobian matrix J_S and the difference between the current sensing coordinate V_curr and the ideal target coordinate V_gold, calculate the image signal processor parameter adjustment amount ΔS; analyze the correlation between each column vector in the parameter-sensory sensitivity Jacobian matrix J_S, identify strongly coupled parameter pairs, and construct a conflict suppression operator Ω based on the correlation degree of the strongly coupled parameter pairs and the optimization centroid; use the conflict suppression operator Ω to correct the image signal processor parameter adjustment amount ΔS, obtain the corrected parameter adjustment amount ΔS′, and obtain the updated image signal processor parameter vector S_new based on the corrected parameter adjustment amount ΔS′; S6. Record the perceptual deviations corresponding to multiple iterations to form a historical sequence of perceptual deviations, and determine whether oscillations occur during the parameter update process based on the historical sequence of perceptual deviations; when oscillations occur, decay the learning rate η, and re-execute step S4 using the decayed learning rate; when no oscillations occur, execute step S7. S7. Read the register specification description file of the target on-chip system, and perform upper and lower bound truncation, quantization step size alignment, register bit width check and dependency condition verification on the updated image signal processor parameter vector S_new according to the register specification description file. Load the image signal processor parameter vector that passes the verification as the current image signal processor parameter vector S into the image signal processor simulator, and return to step S2. S8. Serialize the image signal processor parameter vector that meets the preset convergence threshold ε into a static preset file adapted to the target on-chip system, and generate the corresponding verification digest file.

2. The automatic ISP parameter calibration method based on multimodal semantic feedback according to claim 1, characterized in that, In step S1, the ideal target coordinates V_gold are established in any of the following ways: The manually tuned reference image is input into the visual language model to extract the perceptual vector of the manually tuned reference image, and the ideal target coordinates V_gold are established based on the perceptual vector. Select an aesthetic template that matches the scene tag from the scene template library, and establish the ideal target coordinate V_gold based on the aesthetic template; or, obtain the target value and dimension weight of each image quality evaluation dimension set through the interactive interface, and establish the ideal target coordinate V_gold based on the target value; wherein, the dimension weight of the image quality evaluation dimension is determined according to the scene tag, the setting information received through the interactive interface, or the aesthetic template.

3. The automatic ISP parameter calibration method based on multimodal semantic feedback according to claim 1, characterized in that, In step S2, extracting the multidimensional semantic features and forming the perceptual vector V includes: The rendered image is scaled to the input resolution of the visual language model, and the region of interest in the rendered image is cropped or marked according to the scene label; The processed rendered image and the semantic prompts are input into the visual language model. The semantic prompts include sharpness, noise level, color reproduction, contrast, shadow detail, highlight suppression, skin tone naturalness, and motion blur. The feature values ​​of each dimension output by the visual language model are normalized to 0 to 1, and the negative indicators corresponding to noise, overexposure and motion blur are reverse-transformed so that the feature values ​​of each dimension satisfy the condition that the larger the value, the better the image quality. The perceptual vectors V_f,r of the global region and at least one region of interest are extracted from the multi-frame rendered images respectively, and then weighted and fused according to the following formula: V=Σ_fΣ_rγ_f·α_r·V_f,r; where γ_f represents the frame weight of the f-th frame rendered image, α_r represents the region weight of the r-th region, and the sum of the frame weights Σ_fγ_f=1, and the sum of the region weights Σ_rα_r=1.

4. The automatic ISP parameter calibration method based on multimodal semantic feedback according to claim 1, characterized in that, In step S3, the perception deviation ΔΦ is calculated according to the following formula: ΔΦ=‖V_curr-V_gold‖2=sqrt[Σ_i(V_curr,i-V_gold,i)²]; where V_curr,i represents the coordinate value of the current perception coordinate V_curr in the i-th image quality evaluation dimension, and V_gold,i represents the coordinate value of the ideal target coordinate V_gold in the i-th image quality evaluation dimension; When setting the dimensional weights of each image quality evaluation dimension, the weighted perceptual bias ΔΦ_w is calculated according to the following formula: ΔΦ_w=sqrt[Σ_d w_d·(V_curr,d-V_gold,d)² / Σ_d w_d]; where w_d represents the dimensional weight of the d-th image quality evaluation dimension, and the weighted perceptual bias ΔΦ_w is used as the perceptual bias ΔΦ in step S3 for convergence judgment.

5. The automatic ISP parameter calibration method based on multimodal semantic feedback according to claim 1, characterized in that, In step S4, the perturbation δ_i of parameter s_i is determined according to the following formula: δ_i = max{p_i·(s_i, max-s_i, min), q_i}; where p_i is 0.5%~2%, s_i, max and s_i, min represent the physical upper limit and physical lower limit of parameter s_i, respectively, and q_i represents the minimum quantization step size of parameter s_i on the target on-chip system; The column vector corresponding to parameter s_i in the parameter-sensing sensitivity Jacobian matrix J_S is calculated according to the following formula: J_S(:,i)=[V(s_i+δ_i)-V(s_i-δ_i)] / (2δ_i); where V(s_i+δ_i) and V(s_i-δ_i) represent the sensing vectors obtained after applying positive and negative perturbations to parameter s_i, respectively; the sensing vectors corresponding to the positive and negative perturbations are processed according to the same dimensional order and normalization rule as the ideal target coordinate V_gold. The parameter-sensory sensitivity Jacobian matrix J_S is reconstructed every L iterations or when an oscillation is detected; where L is the preset reconstruction interval of the parameter-sensory sensitivity Jacobian matrix, and L is an integer greater than or equal to 1; in high-noise scenarios, perceptual vectors are extracted from multiple frames of rendered images obtained under the same perturbation conditions, and the median of the components of the same image quality evaluation dimension in each perceptual vector is taken, and the perceptual vector used to calculate the column vector is composed of the median of each dimension component.

6. The automatic ISP parameter calibration method based on multimodal semantic feedback according to claim 1, characterized in that, In step S5, the image signal processor parameter adjustment amount ΔS is calculated according to the following formula: ΔS=-η·J_S^T·(V_curr-V_gold); The correlation coefficient ρ_ij between the sensitivity column vectors corresponding to parameters s_i and s_j is calculated according to the following formula: Only when the norms of all the sensitivity column vectors corresponding to the parameters are not 0, the correlation coefficient is calculated as ρ_ij = |J_S(:,i)·J_S(:,j)| / [‖J_S(:,i)‖·‖J_S(:,j)‖]; when the norm of any column vector is 0, ρ_ij is set to 0, and the corresponding parameters are not determined as a strongly coupled parameter pair; when the correlation coefficient ρ_ij is large... When the relevant threshold ρ_th is equal to or greater than the relevant threshold ρ_th, the parameters s_i and s_j are determined as a strongly coupled parameter pair, and the attenuation coefficient C_ij is determined according to the following formula: C_ij=clip[1-λ·(ρ_ij-ρ_th) / (1-ρ_th),C_min,1]; where λ represents the suppression intensity, C_min represents the minimum retention ratio, and clip represents the clipping function; the relevant threshold ρ_th and the corresponding included angle threshold θ_th satisfy ρ_th=cosθ_th; The dominant parameter in the strongly coupled parameter pair is determined according to the dimension weight corresponding to the current scene. The update amount of the dominant parameter is retained, and the update amount of the non-dominant parameter or the synchronous update amount of the strongly coupled parameter pair is attenuated by the attenuation coefficient C_ij to form the conflict suppression operator Ω in the form of a diagonal matrix or a block diagonal matrix. The adjustment amount of the correction parameter ΔS′ and the updated image signal processor parameter vector S_new are obtained according to the following formulas: ΔS′=Ω·ΔS; S_new=S+ΔS′.

7. The automatic ISP parameter calibration method based on multimodal semantic feedback according to claim 1, characterized in that, In step S6, the perceptual deviation of the most recent N_hist iterations is recorded, where N_hist ≥ 5; when there are K consecutive iterations that simultaneously meet the following conditions, it is determined that the parameter update process has oscillated: |ΔΦ_t-ΔΦ_(t-2)|<ε_osc; (ΔΦ_t-ΔΦ_(t-1))·(ΔΦ_(t-1)-ΔΦ_(t-2))<0; where K≥2, ΔΦ_t, ΔΦ_(t-1) and ΔΦ_(t-2) represent the perception deviations corresponding to the t-th, t-1 and t-2 iterations respectively, ε_osc represents the oscillation threshold, which is 0.01~0.05; after detecting oscillation, the decayed learning rate η′ is calculated according to the following formula: η′=η·β; where β is 0.5~0.8; when η′<η_min, η′ is set to η_min, where η_min is 1 / 10~1 / 100 of the initial learning rate; when oscillation is still detected after reducing the learning rate M times consecutively, an alarm message is output and calibration is paused, where M≥3.

8. The automatic ISP parameter calibration method based on multimodal semantic feedback according to claim 1, characterized in that, In step S7, the register specification description file includes platform identifier, image signal processing module name, parameter name, register address, bit offset, bit width, lower bound of parameter, upper bound of parameter, quantization step size, default value, scaling factor, serialization field name, and dependency condition field. The calibration control terminal reverse-calculates the normalized parameters into register physical parameter values, and then performs lower bound verification, upper bound verification, and quantization alignment on the updated parameter value s_new according to the following formula: Lower bound check: s_new = max(s_new, s_min); Upper bound check: s_new = min(s_new, s_max); Quantization alignment: s_new = round[(s_new - s_min) / Δs] × Δs + s_min; where s_min represents the lower bound of the parameter value, s_max represents the upper bound of the parameter value, and Δs represents the quantization step size of the parameter value; perform register bit width check on the quantized and aligned parameter value, and perform enable status check, working mode check, and mutual exclusion parameter check according to the dependency condition field.

9. The automatic ISP parameter calibration method based on multimodal semantic feedback according to claim 1, characterized in that, In step S8, according to the serialization protocol corresponding to the target on-chip system, the image signal processor parameter vector that meets the preset convergence threshold is serialized into a static preset file in XML, JSON, CONF, or BIN format; The verification summary file includes scene labels, visual language model version, source of ideal target coordinates V_gold, initial perceptual deviation, final perceptual deviation, number of iterations, number of learning rate decays, oscillation detection records, initial values, final values ​​and changes of each image quality evaluation dimension, platform identifier of the target on-chip system, hash value and version number of the static preset file.

10. An automatic ISP parameter calibration system based on multimodal semantic feedback, characterized in that, It includes an image signal processor simulator and a calibration control terminal that is communicatively connected to the image signal processor simulator; The image signal processor simulator is used to load the image signal processing module algorithm model of the target system-on-a-chip, process the original image sequence of the target scene based on the current image signal processor parameter vector, and output the rendered image. The calibration control terminal includes: The semantic feature extraction module is used to input the rendered image and semantic prompts including multiple image quality evaluation dimensions into a visual language model, extract multi-dimensional semantic features of the rendered image through the visual language model, and form a perceptual vector V. The target quality coordinate quantization module is used to establish the ideal target coordinate V_gold of the target scene, determine the perception vector V as the current perception coordinate V_curr according to the same dimensional order and normalization rule as the ideal target coordinate V_gold, and calculate the perception deviation based on the current perception coordinate V_curr and the ideal target coordinate V_gold. The sensitivity analysis module is used to apply positive and negative perturbations to each parameter in the parameter vector of the image signal processor when the perception deviation does not meet the preset convergence condition. The perception vector obtained under the perturbation condition is processed according to the same dimensional order and normalization rule as the ideal target coordinate V_gold. The parameter-perception sensitivity Jacobian matrix J_S is constructed based on the perturbation amount of each parameter and the corresponding change in the perception vector. The conflict suppression and parameter update module is used to calculate the image signal processor parameter adjustment amount based on the parameter-sensory sensitivity Jacobian matrix J_S, analyze the correlation between each column vector in the parameter-sensory sensitivity Jacobian matrix J_S to identify strongly coupled parameter pairs, construct a conflict suppression operator Ω based on the strongly coupled parameter pairs, and use the conflict suppression operator Ω to correct the image signal processor parameter adjustment amount to obtain the updated image signal processor parameter vector. An oscillation detection module is used to form a historical sequence of perception deviations based on the perception deviations corresponding to multiple iterations, detect whether oscillations occur during the parameter update process based on the historical sequence of perception deviations, and reduce the learning rate when oscillations are detected. The hardware constraint verification module is used to read the register specification description file of the target on-chip system, perform upper and lower bound truncation, quantization step size alignment, register bit width check and dependency condition verification on the updated image signal processor parameter vector according to the register specification description file, and load the verified image signal processor parameter vector into the image signal processor simulator. The serialization delivery module is used to serialize the current image signal processor parameter vector into a static preset file adapted to the target on-chip system when the perception deviation meets the preset convergence condition, and generate a corresponding verification digest file.

Images