Dashcam glass multi-layer reflection occlusion image restoration method and system combining optical flow guided semantic segmentation

Abstract

The application discloses a kind of multi-layer reflection shelter image restoration method and system of glass of car data recorder combined with optical flow guiding semantic segmentation.The method is by interlayer glass layer thickness,refractive index,wedge angle and camera internal parameter solution equivalent optical thickness and pixel-by-pixel incident angle,generates space-variable ghost kernel and order attenuation coefficient,builds multi-layer reflection imaging model containing soft saturation response operator;Estimate dense optical flow to transmission layer,estimate rigid motion initialized with angular velocity to reflection layer,accordingly construct motion clue map and inject semantic segmentation network by gated attention;With segmentation entropy as weight,merge two kinds of layer attribution probability and be mutually guided iteration with layered motion;Again,layer attribution guided cross-sparse energy functional separates transmission layer,switches time domain memory according to motion effectiveness factor,according to separation quality score arbitration output and double-track archiving.The application suppresses multi-order ghost residual and saturation zone artifact,stable in all working conditions and meets real-time and evidence integrity.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for restoring images of multi-layer reflection occlusion on dashcam glass using optical flow-guided semantic segmentation. Background Technology

[0002] As a standard feature in modern passenger vehicles, dashcams play a crucial role in accident reconstruction, insurance claim liability determination, and forward-looking driver assistance systems. To balance wide field of view coverage with discreet installation within the vehicle, dashcams are typically mounted on the inside of the windshield near the rearview mirror and equipped with a wide field-of-view optical lens. However, due to the significant angle between the dashcam's optical axis and the windshield's normal, bright objects inside the vehicle, such as dashboard trim, the center console display, the steering wheel, and light-colored clothing, are easily reflected off the glass, overlapping with the actual road scenery on the same image sensor plane, resulting in severe multi-layered reflection obstruction. This obstruction is particularly noticeable in complex lighting conditions, such as at night with oncoming headlights, entering or exiting tunnels, strong road glare in rainy weather, and low-angle direct sunlight in the early morning or late afternoon. It often completely obscures crucial visual information such as license plates, lane lines, and pedestrian outlines, severely weakening the effectiveness of video evidence and the reliability of intelligent driving perception.

[0003] Unlike the reflection phenomenon on single-layer glass in ordinary buildings, automotive windshields generally employ a sandwich structure consisting of two layers of glass and an interlayer film. This multi-interface medium causes objects inside the vehicle to form not a single reflected virtual image in the image, but rather a complex ghosting with multiple orders and significant lateral misalignment. Even more challenging is the fact that, to improve the visual effect of head-up display systems, many modern vehicles widely adopt a wedge-shaped interlayer film design that is thicker at the top and thinner at the bottom. This special wedge-shaped structure causes the misalignment of the aforementioned multi-order ghosting to exhibit a continuously varying non-linear characteristic along the vertical height of the image, making it impossible to use a globally uniform fixed displacement for precise mathematical description. Furthermore, due to the significant difference in physical distance between the reflected virtual image and the real scene outside the vehicle, the reflective layer exhibits varying degrees of defocus blur relative to the transmissive layer, and this defocus scale increases with the increase of the reflection order.

[0004] Faced with the aforementioned complex reflection interference, existing solutions often suffer from numerous insurmountable technical bottlenecks. Regarding hardware suppression, while adding a polarizing filter can achieve light reduction within a local field of view, the extreme difference in the incident angle between the edge and center of an ultra-wide-angle lens means that a polarizing mirror with a single transmission direction cannot globally eliminate reflections. More critically, adding a polarizing mirror significantly reduces the amount of light entering the optical system, fundamentally conflicting with the core requirement of in-vehicle equipment to ensure imaging performance in low-light environments at night. In pure algorithmic single-image dereflection approaches, existing technologies heavily rely on idealized prior assumptions such as gradient sparsity or defocus differences. However, in real driving environments, road textures and the edges of surrounding buildings also exhibit dense gradient features. In areas of strong reflection, image sensors often experience overexposure and saturation, completely disrupting the original linear superposition physical relationship. Once the prior conditions do not match the actual scene, these algorithms are highly susceptible to misjudging real road conditions outside the vehicle as reflection artifacts and forcibly erasing them, fabricating false content out of thin air—absolutely unacceptable for in-vehicle videos that must possess rigorous legal evidentiary attributes.

[0005] Furthermore, layering algorithms relying on multi-frame temporal information or optical flow estimation attempt to achieve image separation by utilizing the relative motion differences between the transmission and reflection layers during vehicle movement. However, current methods of this kind are generally based on a simple assumption of two linear superposition layers, completely ignoring the inherent multi-level ghosting of laminated glass and the spatially varying misalignment caused by the wedge angle. The severe mismatch of the underlying physical model directly results in noticeable misalignment and ghosting remaining in the separated clear image. Moreover, in areas where the two images highly overlap, the constant brightness assumption relied upon by traditional optical flow estimation has long since failed. The resulting initial optical flow error is directly propagated to subsequent image layering calculations, and the erroneous layering results, in turn, worsen the accuracy of the next round of optical flow estimation, forming a vicious cycle of positive error feedback within the system that is difficult to escape.

[0006] In recent years, some technologies have attempted to introduce semantic segmentation networks to assist in image layering, that is, to directly generate hard masks of reflective areas by recognizing the semantic categories of in-vehicle components. However, this approach also faces serious challenges. The segmentation network itself performs reasoning and judgment on pixels that have been heavily contaminated by reflections, and the texture features of the input data have deviated significantly from the normal training distribution, making the network prone to systematic misjudgments. At the same time, the black-and-white hard mask strategy completely abandons the probabilistic uncertainty information originally possessed by deep learning networks, and still applies strong mathematical constraints to blurry areas with extremely low confidence, which will further amplify the separation error. In essence, both motion cues based on time sequence and semantic cues based on appearance have their own blind spots in complex driving scenarios, and existing technologies have not yet proposed a collaborative fusion mechanism that can dynamically and adaptively match cues according to their reliability, resulting in cues from different dimensions often contaminating each other.

[0007] Furthermore, the adaptability to various operating conditions and the systematic engineering deployment are also key pain points that have long been overlooked. When a vehicle is stationary, such as waiting at an intersection or crawling in congestion, the relative motion between the inner and outer layers approaches zero simultaneously. Traditional layered algorithms that rely on motion differences will instantly fail, causing the restored image to frequently jump between effective and ineffective states, resulting in severe flickering. When a vehicle passes over a speed bump or makes a sharp turn, the violent shaking of the vehicle's posture will disrupt the assumption of relative stillness of the reflective layer, easily leaving obvious visual ghosting in the image. At the same time, the edge computing resources of the vehicle's main control chip are usually extremely limited, making it difficult to support the real-time operation of a large network model and dense iterative calculations. More importantly, the lack of a sound quality self-checking mechanism, failure rollback protection mechanism, and a dual-track synchronous archiving strategy for raw data and enhanced images makes it difficult for existing restoration technologies to effectively ensure the integrity and compliance of video materials as judicial evidence while improving the perception quality of the front-end image.

[0008] This invention aims to solve the technical problems existing in the prior art. To this end, it proposes a method and system for restoring images of multi-layer reflection occlusion on dashcam glass by combining optical flow-guided semantic segmentation. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for restoring images of multi-layer reflection occlusion in dashcam glass by combining optical flow-guided semantic segmentation, in order to solve the technical problems in the prior art such as missing multi-level ghost image modeling of laminated glass, mutual contamination between motion cues and semantic cues, instability under static and strong disturbance conditions, and the inability to balance vehicle real-time performance and evidence compliance.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: The method includes steps S1 to S10, namely, constructing a multilayer reflection imaging model containing a spatially variable ghost kernel, order attenuation coefficient, and soft saturation response operator based on the layer thickness, refractive index, and interlayer wedge angle of the front windshield laminated glass and camera intrinsic parameters; acquiring the current frame and the N frames preceding it, and simultaneously acquiring the vehicle speed and the angular velocity of the inertial measurement unit; estimating the dense optical flow for the transmission layer, estimating the parameterized rigid motion using the homography matrix constructed with the angular velocity as the initial value for the reflection layer, and constructing a motion cue map and motion layer assignment probability based on the double-layer photometric residual, motion separation degree, and forward / backward consistency confidence degree; and injecting the motion cue map into the semantic segmentation network through a gated attention unit. The network features at each level are used to output semantic layer attribution probabilities; confidence weights are obtained from the information entropy of semantic category probabilities, and geometric weighted fusion is performed on the two types of layer attribution probabilities; layer motion is estimated by inverse weighting of layer attribution probabilities, and encoder features are reused for iterative refresh until convergence; motion cue effectiveness factor is calculated and switched between joint solution and solution of only the transmission layer subproblem is performed accordingly, wherein the joint solution is based on energy functional, which includes saturated gated data fidelity term, layer attribution guided cross sparsity term, gradient mutual exclusion term and temporal consistency term, and is solved alternately by semi-quadratic splitting; the temporal memory of the reflection layer is updated and the separation quality score is calculated; finally, the output is arbitrated according to the separation quality score and original and enhanced dual-track archiving is performed.

[0011] As a further aspect of this invention: the equivalent optical thickness is determined by the sum of the thicknesses of the two glass layers plus the product of the thickness of the interlayer and the ratio of the two refractive indices; the lateral displacement of the k-th order reflection is proportional to the product of the order, the equivalent optical thickness, and the sine and cosine of the incident angle, and inversely proportional to the square root of the difference between the square of the glass refractive index and the square of the sine of the incident angle; the order attenuation coefficient is given by the product of the odd power of the Fresnel reflectivity, the square of its complement, and the power of the transmittance within the interlayer; the scale parameter of the ghost kernel increases with the square root of the order. This scheme ensures that the positive synthesis operator of the reflection components is strictly consistent with the physical imaging process of real interlayer glass, eliminating ghosting residue at its source.

[0012] As a further aspect of this invention: before calculating the bilayer photometric residual, the image is first high-pass filtered to remove low-frequency brightness coupling between the two layers; dense optical flow is estimated from coarse to fine using a four-level pyramid and confidence levels are given based on forward and backward consistency; the initial homography values ​​for the reflection layer motion are obtained by multiplying the sum of the products of the intrinsic parameter matrix, the identity matrix, the antisymmetric matrix of angular velocity, and the sampling interval, as well as the inverse of the intrinsic parameter matrix, and then iteratively reweighted least squares estimation using the Charbonnier robust kernel. This scheme compresses the degrees of freedom of the reflection layer motion from a pixel-by-pixel vector field to eight global parameters, significantly improving robustness under low texture and strong noise conditions and reducing the computational load of this part by two orders of magnitude.

[0013] As a further aspect of this invention: the gating weights are obtained by concatenating the feature map and the downsampled motion cue map, followed by convolution and a logistic function, and then modulating the feature map channel-by-channel in the form of a product of the modulation intensity and the gating weights; the network has a gating attention unit in each of the four downsampling levels, and is configured with a mapping table that categorizes semantic categories into transmission, reflection, and occlusion classes. This approach enables the segmentation network to recover the correct layer attribution judgment for pixels contaminated by reflection using motion cues, avoiding systematic misjudgments in pure appearance segmentation.

[0014] As a further aspect of this invention: a negative exponent of the ratio of the information entropy of the semantic category probability to the entropy baseline value is used as a confidence weight, and this confidence weight is used as an exponent to geometrically weight and normalize the semantic layer attribution probability and the motion layer attribution probability. This scheme enables the two types of clues to be dynamically matched pixel-by-pixel according to their respective reliability, achieving complementarity rather than mutual contamination between their failure intervals.

[0015] As a further aspect of this invention: the layer assignment-guided cross-sparse term uses the transmission layer assignment probability to weight the reflection layer gradient and the reflection layer assignment probability to weight the transmission layer gradient. Combined with gradient mutual exclusion terms and saturation-gated data fidelity terms, the transmission layer is updated in the frequency domain with a closed-form solution through semi-quadratic splitting, and the reflection layer is updated with its conjugate gradient, with the penalty factor doubling successively. This scheme allows the layer assignment probability to be incorporated into the optimization objective in a soft-constraint manner, avoiding boundary jumps in the hard mask and suppressing artifacts and structure erasure in the saturation region.

[0016] As a further aspect of this invention: based on the motion cue effectiveness factor and angular velocity, a state switching with hysteresis is performed between normal driving state, stationary low-speed state, and violently disturbed state. In the stationary low-speed state, the temporal memory of the reflection layer is frozen and reused as a template. In the violently disturbed state, memory propagation is paused and the output threshold is increased. On the output side, arbitration is performed between the enhanced frame and the original frame according to the separation quality score, and the original bitstream and the enhanced bitstream are written to two tracks frame by frame in alignment. The hash digest of the original bitstream is calculated and signed in segments. This scheme enables the invention to output stably under all operating conditions and completely preserve the original evidence while enhancing image quality.

[0017] The present invention also provides an image restoration system, an electronic device, and a computer-readable storage medium for implementing the above methods.

[0018] Compared with the prior art, the beneficial effects of the present invention are:

[0019] Physical model upgrade and ghosting elimination: This invention upgrades the traditional linear superposition model to a multi-layer reflection model that conforms to the real physical process of laminated glass by analyzing and generating the spatially variable ghosting kernel and order attenuation coefficient. This eliminates paired misaligned ghosting from the root and significantly reduces the ghosting residual energy ratio.

[0020] Motion model dimensionality reduction and stable convergence: The motion of the reflective layer is compressed into a parameterized rigid model, and the initial value is provided by angular velocity. This enables the system to maintain stable convergence in complex environments such as low texture and high noise at night, while greatly reducing the time consumption of the motion estimation process.

[0021] Optimization of cue gating injection and discrimination: Abandoning the hard mask post-processing method, the motion cues are innovatively injected directly into the semantic segmentation network using a gating attention mechanism, which significantly improves the accuracy of the sub-level attribution discrimination within the reflection contamination area.

[0022] Dynamic matching of cues and complementary advantages: A geometric weighted fusion strategy with segmentation entropy as the weight is adopted to achieve pixel-by-pixel dynamic matching of motion cues and semantic cues, effectively breaking the predicament of mutual contamination between the two in their respective failure ranges.

[0023] Mutual guided iteration and error blocking: A two-way mutual guided iteration mechanism was established, which successfully broke the positive feedback loop between optical flow estimation error and layering error, enabling the algorithm to converge very quickly.

[0024] Full-condition adaptive and time-domain stable: By using a condition state machine and a time-domain memory strategy, the method can be made stable even under extreme conditions such as when the vehicle is at zero motion or encounters severe disturbances, which greatly improves the problem of video frame flicker.

[0025] Balancing evidence preservation and image quality enhancement: The solution is equipped with a quality self-assessment, failure rollback, and dual-track archiving mechanism, which improves the visual perception quality of front-end images while properly and completely preserving the legal evidentiary value of the original records.

[0026] Superior hardware deployment and real-time capabilities: The overall algorithm solution has extremely low computational overhead after model pruning and quantization optimization, and can be directly deployed on ordinary automotive-grade chips, fully meeting the low-latency and real-time restoration processing requirements of high-resolution driving videos. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the overall process of the image restoration method described in this invention.

[0029] Figure 2 This is a schematic diagram illustrating the geometric principle of multi-layer reflection imaging of the windshield in the dashcam of the present invention.

[0030] Figure 3 This is the first sub-flowchart of the hierarchical motion estimation and motion cue graph construction in this invention.

[0031] Figure 4 This is a block diagram showing the structural composition of the optical flow-guided semantic segmentation network and motion-gated attention injection in this invention.

[0032] Figure 5 This is a timing diagram of the bidirectional mutual guided iteration of optical flow and semantic segmentation in this invention.

[0033] Figure 6 This is the second sub-flow flowchart of the adaptive fusion of layer affiliation probabilities and multi-layer separation and reconstruction in this invention.

[0034] Figure 7 This is the state transition diagram for adaptive operating conditions, time-domain memory, and dual-track output in this invention.

[0035] Figure 8 This is a block diagram of the composition structure of the image restoration system described in this invention. Detailed Implementation

[0036] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0037] Example 1

[0038] Figure 1 This is a flowchart illustrating the overall process of the image restoration method described in this invention. This embodiment uses a dashcam installed inside the windshield, with an imaging resolution of 1920×1080, a frame rate of 30fps, and a horizontal field of view of 140° as an example to illustrate the complete implementation process.

[0039] Step S1: Joint calibration of glass and camera, and construction of multi-layer reflection imaging model. The structural parameters of the windshield laminated glass are read from the vehicle's electronic identification or user configuration. In this embodiment, the outer glass thickness is 2.1 mm, the interlayer film thickness is 0.76 mm, the inner glass thickness is 2.1 mm, the glass refractive index is 1.52, the interlayer film refractive index is 1.48, the interlayer film wedge angle is 0.5 mrad, and the windshield's installation tilt angle relative to the vertical plane is 29°. The camera intrinsic parameters are obtained from factory calibration, with an equivalent focal length of 1120 pixels and the principal point located at the image center. Since the angle of refraction of light within the interlayer film is greater than its angle of refraction within the glass, an equivalent optical thickness is introduced to treat the three layers as a single homogeneous plate. Simultaneously, the wedge angle causes the glass normal direction to continuously deflect along the image height, thus the incident angle is pixel-by-pixel. Figure 2The above geometric relationships are shown, from which the equivalent optical thickness, pixel-by-pixel incident angle, and the lateral displacement and pixel-domain ghost displacement vector of the k-th order reflection can be obtained, as shown in Equation (1):

[0040]

[0041]

[0042] In the formula, , These are the thicknesses of the outer and inner glass layers, respectively. The thickness of the intermediate film. and These are the refractive indices of the glass and the interlayer, respectively. Equivalent optical thickness; The angle of incidence at the center of the image. The intermediate membrane wedge angle is y and These are the ordinates of the current pixel and the image center, respectively. The effective glass height corresponding to the image height; For the first The lateral displacement of the first-order reflection relative to the first-order reflection. For equivalent focal length, For the distance of the reflected virtual image, Let be the unit vector of the projection direction of the glass normal onto the image plane. This is the pixel-domain ghost displacement vector. Where, the distance to the reflected virtual image is... effective height of glass Obtained through a one-time on-vehicle calibration: A gridded calibration plate is placed at a known position on the dashboard along the optical axis. A recorder captures the virtual image formed by the reflection of the plate through glass, and the virtual image distance is calculated using a pinhole model. This yields the virtual image distance in that field of view. In this embodiment, the image center is located at... Approximately 1.35m; The distance is calculated by inversely from the imaging position of the calibration plate at the upper and lower edges of the windshield, which is approximately 0.52m in this embodiment. For the same vehicle model, the above two quantities only need to be calibrated once and written into the read-only configuration area of ​​the device along with the vehicle model parameters. Substituting the parameters of this embodiment, the equivalent optical thickness is approximately 4.98mm, the lateral displacement of the second-order reflection is approximately 3.12mm, and the corresponding pixel domain ghost displacement is approximately 4.7 pixels. Due to the existence of the wedge angle, the amount of this displacement can vary by 0.9 pixels between the upper and lower edges of the image. If a fixed displacement is used for modeling, obvious ghosting will be introduced.

[0043] Based on this, a variable ghost kernel is generated, and the attenuation coefficients of each order of reflection are calculated according to the Fresnel formula, as shown in formula (2):

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] In the formula, For the first The spatially variable ghost kernel of the first-order reflection, where u is the local coordinate of the kernel. This is the defocusing scale of the reflection relative to the transmission layer at this order. As the baseline defocus scale, this embodiment uses 0.8 pixels; denoted as Fresnel reflectivity of unpolarized light at the air-glass interface. For the angle of refraction, The order attenuation coefficient, The value is the square of the single-pass transmittance of the interlayer; in this embodiment, it is taken as 0.96. At a 29-degree incident angle, the Fresnel reflectance is approximately 4.5%, and the energy proportion of the third-order and higher components is less than 0.5%. Therefore, in this embodiment, the upper limit of the order is used. It equals three, balancing accuracy and convolution operator size.

[0050] Finally, a multi-layer reflection imaging model is constructed. Considering that the sensor will enter a nonlinear response until saturation in scenarios such as oncoming headlights at night and backlight at tunnel entrances, if strict linear superposition is still used, significant model deviation will occur in the high-light region. Therefore, a soft saturation response operator is introduced to monotonically compress the superposition result, as shown in formula (3):

[0051]

[0052]

[0053] In the formula, For the first Frame observation image, and These represent the transmission and reflection layers of the frame, respectively, with circles and asterisks indicating convolution. For soft saturation response operators, This is the sensor's full-scale value. This represents noise. The operator approximates an identity mapping in the low-brightness region and tends to saturate smoothly near full scale, ensuring the model remains effective across the entire dynamic range. The summation sign in the equation indicates... All reflection components originate from the same reflective layer, with differences only in the ghost kernel and order attenuation coefficient. This is the physical essence of multi-interface reflection in laminated glass and the fundamental difference between this invention and the two-layer linear superposition model. Thus, the forward synthesis operator for the reflection components is fully determined, with all parameters derived from calibrable physical quantities rather than data fitting. This ensures cross-vehicle transferability and provides an accurate forward model for subsequent inversion.

[0054] Step S2: Simultaneously acquire multiple frames of images and vehicle condition signals. Take the N frames preceding the current frame as the center; in this embodiment, N=2. At the same time, acquire the vehicle speed through the vehicle diagnostic interface or global navigation satellite system, acquire the three-axis angular velocity through the inertial measurement unit built into the recorder, and perform timestamp alignment and linear interpolation on the above signals according to the exposure center time of the image frames.

[0055] Step S3, hierarchical motion estimation and motion cue map construction, its sub-processes are as follows: Figure 3 As shown. Since the two layers are strongly coupled in low-frequency brightness, it is difficult to distinguish the two layers by directly calculating the photometric residual on the original image. Therefore, a high-pass filter in the form of Gaussian difference is first applied to each frame, and then the motion is estimated in two paths on the filtered image. The transmission layer uses a four-level Gaussian pyramid to estimate the dense optical flow from coarse to fine. The upsampling of the result above each level is used as the initial value, which can cover the displacement of more than 64 pixels per frame. The consistency confidence is calculated from the modulus of the sum of the forward and backward optical flows. The reflection layer utilizes the following fact: the dashboard, steering wheel, A-pillar guard plate and other reflective sources are all rigidly connected to the vehicle body. They hardly produce relative motion to the recorder which is also fixed to the vehicle body. The motion of the reflected image is only caused by the slight vibration of the glass posture and the camera shake. Therefore, it can be described by an eight-parameter homography transformation. Its estimation is modeled as formula (4):

[0056]

[0057]

[0058]

[0059]

[0060] In the formula, For the parameterized rigid motion mapping of the reflective layer, Let pi be an eight-dimensional parameter vector, and let pi and its inverse be the projection and back-projection operators, respectively. It is a homography matrix. For the camera intrinsic parameter matrix, The value is angular velocity; square brackets with an X indicate antisymmetric matrix operators. For frame interval, For pixel weights, This is the image after high-pass filtering. The core is a Chabonnieru bar. Using the homography matrix constructed with angular velocity as the initial value can make the iterative reweighted least squares converge within two to three iterations, avoiding getting trapped in local minima in low-texture night scenes; this step reduces the motion degrees of freedom of the reflection layer from more than two million to eight, and the measured time is reduced from 21.6ms to 0.9ms.

[0061] After obtaining the two motion paths, the reference frame is reverse-mapped along the two motion paths respectively, and the double-layer photometric residual, motion separation degree and motion layer attribution probability are calculated, as shown in formula (5):

[0062]

[0063]

[0064]

[0065]

[0066] In the formula, and These represent the photometric residuals after alignment of the optical flow along the transmission layer and the rigid motion along the reflection layer, respectively. and These are two layers of motion vector fields. For motion separation, This represents the motion layer attribution probability. Its physical meaning is: if the temporal trajectory of a pixel more closely matches the rigid motion of the reflective layer, then the residual aligned along the reflective layer is smaller while the residual aligned along the transmissive layer is larger, strengthening the evidence that the pixel belongs to the reflective layer's motion. Motion separation characterizes the distinguishability of motion between two layers; a value close to zero indicates that the motion cue is invalid at this point. Finally, these four quantities are normalized channel by channel and stacked to form a four-channel motion cue map.

[0067] Step S4, optical flow-guided semantic segmentation, its network composition is as follows: Figure 4 As shown. The semantic segmentation network uses a lightweight inverse residual structure as the encoder, with four downsampling stages and output step sizes of 2, 4, 8 and 16, and channel numbers of 16, 24, 48 and 96, respectively. The decoding end is connected to a hole space pyramid pooling module with dilation rates of 1, 3, 6 and 9, and then upsampled level by level and fused with the features of the encoding end. The key improvement of this invention is that instead of using the segmentation result as a post-processing mask, the motion cue map is injected as a guiding signal at each stage of the encoder, so that the network can perceive motion evidence at the same time as forming appearance features, as shown in formula (6):

[0068]

[0069]

[0070]

[0071] In the formula, This is a four-channel motion clue diagram. Indicates downsampling to the th Level resolution, For the first Level feature map, and The convolution kernel and bias of the gated attention unit at this level. For logical stethoscopes, For gating weights, The modulated feature map, This indicates element-wise multiplication. The modulation intensity coefficient is set to 0.5 in this embodiment. The modulation adopts a residual form with a gating term to ensure that the features remain unchanged when the gating weight is zero, thus stabilizing the training. After this modification, the network can identify the temporal behavior of the texture on the pixel contaminated by reflection by using motion evidence, which is consistent with the rigid motion of the vehicle body, thereby giving the correct layer assignment.

[0072] The decoder uses dual output heads. The first head outputs semantic category probabilities, with a category set containing 16 categories: sky, road surface, lane lines, vehicles, pedestrians, traffic signs, buildings, vegetation, dashboard, steering wheel, A-pillar trim, central control display, occupants, rearview mirror, wipers, and glass smudges. The second head outputs semantic layer attribution probabilities. The two are linked by a mapping table: the first eight categories are categorized as transmission, the last six as reflection, and wipers and glass smudges as occlusion. The semantic layer attribution probability can be predicted directly from the second head or obtained by summing the category probabilities according to the mapping table. During training, the consistency between the two probabilities is used as an additional regularization constraint to improve reliability.

[0073] Step S5, adaptive fusion of layer affiliation probabilities, its sub-processes are as follows: Figure 6 As shown in the upper part. Semantic cues are unreliable in areas with severe reflection contamination, and motion cues are unreliable in areas with similar motion between two layers or with scarce texture. If they are linearly added with fixed weights, their failures will contaminate each other. Therefore, this invention uses the uncertainty of the segmentation output itself as the matching basis and adopts entropy adaptive geometric weighted fusion as shown in formula (7):

[0074]

[0075]

[0076]

[0077]

[0078] In the formula, Information entropy for semantic category probabilities For the first The probability of a class For a set of categories, As the entropy baseline value, this embodiment uses approximately 1.39, which is half the natural logarithm of the number of categories. As confidence weights, and These represent the probability of attribution to the reflective and transmissive layers, respectively. The mechanism is as follows: when the segmentation judgment is certain, the entropy is close to zero and the confidence weight is close to one, meaning the fusion result is almost entirely determined by semantic cues; when the segmentation output is close to a uniform distribution, the confidence weight tends to zero, and the fusion result automatically degenerates into being entirely determined by motion cues. Choosing geometric weighting instead of arithmetic weighting lowers the fusion result when any cue gives a probability close to zero, exhibiting a logical AND property that can suppress false alarms from a single cue.

[0079] Step S6 involves bidirectional mutual guided iteration of optical flow and semantic segmentation, with the timing as follows: Figure 5 As shown. After the j-th iteration, the transmission layer assignment probability is sent to the transmission layer optical flow estimation unit as a pixel-wise weight of the photometric term, so that the optical flow fits the constant brightness constraint only on high-confidence transmission pixels, thereby avoiding the error caused by the failure of this assumption in the superimposed region; at the same time, the reflection layer assignment probability is sent to the reflection layer motion estimation unit as the initial weight of the iterative reweighted least squares, so that the homography parameter is driven only by high-confidence reflection pixels. Then the motion cue map is refreshed and the calculated encoder features are reused, and only the gated attention unit, decoder and fusion steps are re-executed. The iteration termination condition is that the mean absolute value of the difference between the reflection layer assignment probabilities of two adjacent rounds is less than the convergence threshold (0.01 in this embodiment) or the maximum number of iteration rounds is reached (2 in this embodiment). In actual tests, more than 90% of the frames converged in two rounds, and because the encoder features were reused, the additional overhead of the second round was only about 23% of that of the first round. This mechanism is the key to breaking the positive feedback between optical flow error and layering error.

[0080] Steps S7 and S8 involve condition discrimination and multi-layer separation and reconstruction. First, the motion cue validity factor is calculated to determine whether the motion cue of the current frame is sufficient to support layering. Its definition and temporal memory update method are given in formula (10) below. When this factor is not less than a set threshold, the vehicle enters a normal driving state, and the two layers are solved jointly; when it is less than the set threshold, it indicates that the vehicle is waiting at a red light or in a low-speed creeping state, and the motion of both layers approaches zero. At this time, the temporal memory of the reflection layer is frozen and used as a known template, and only the sub-problem concerning the transmission layer is solved. The energy functional on which the separation is based is as follows: Figure 6 The lower part and formula (8) are shown below:

[0081]

[0082]

[0083] In the formula, For energy functionals, It is a saturation gate. The saturation threshold is set to 92% of full scale in this embodiment. For gating standards. to The weights of the three regularization terms are 0.08, 0.015, and 0.04, respectively, in this embodiment. For the alignment operator that moves rigidly along the reflector layer, This is the temporal memory of the reflection layer in the previous frame. The functional contains four terms: the first term is the data fidelity term, whose forward operator is the multi-layer reflection imaging model determined by formula (3), and is weighted by saturation gate, so that saturated pixels hardly participate in data fitting, avoiding erroneous driving due to the truncation of linear relationship; the second term is the layer assignment guided cross-sparse term proposed in this invention, which suppresses the gradient of the reflection layer by weighting the transmission layer assignment probability on pixels judged as transmission, and suppresses the gradient of the transmission layer by weighting the reflection layer assignment probability on pixels judged as reflection. Compared with hard mask, this term directly embeds the continuous value of the layer assignment probability into the optimization target in a soft constraint manner, which will not produce jump artifacts at the mask boundary, and also allows the data term to determine itself in the blur region with a probability close to 0.5; the third term is the gradient mutual exclusion term, which suppresses ill-conditioned solutions in which the same edge appears in two layers at the same time; the fourth term is the temporal consistency term, which constrains the reflection layer to keep in line with its rigid movement and historical memory, significantly improving the inter-frame stability of the output video.

[0084] The functional is non-convex in both layers and is minimized by a semi-quadratic splitting method: after introducing an auxiliary variable to replace the gradient of the transmission layer, the subproblem of the auxiliary variable has a closed-form solution with soft threshold shrinkage, and the subproblem of the transmission layer can be given by the fast Fourier transform under periodic boundary conditions, as shown in Equation (9):

[0085]

[0086]

[0087]

[0088] In the formula, Its inverses are the forward Fourier transform and the inverse Fourier transform, respectively. Indicates complex conjugation. As a penalty factor, For the corresponding direction Auxiliary variables, The inverse is the inverse mapping of the soft saturation response operator. The equivalent observation after deducting the current estimated contribution of the reflector layer, This is a symbolic function. It should be noted that... The inverse mapping diverges when the observed values ​​approach full scale. Therefore, in practice, the observed values ​​are first limited to 99.5% of full scale. Furthermore, the saturation gating of such pixels is close to zero and contributes almost nothing to the data terms, thus not affecting the stability of the solution. The subproblem of the reflection layer does not have a closed-form solution in the frequency domain due to the involvement of the spatially variable ghost kernel. The conjugate gradient method is used for iterative solution. In this embodiment, 10 iterations are taken. The penalty factor is doubled after each alternating iteration. In this embodiment, the penalty factor starts from 0.01 and the maximum number of alternations is 6. The measured relative residual drops below 0.5% in the 4th to 5th alternations.

[0089] Steps S9 and S10 involve temporal memory update, quality self-evaluation, and dual-track output. Their state transition relationships are as follows: Figure 7 As shown. Since the reflection source is rigidly connected to the vehicle body, the reflection layer is highly stable on a timescale of several seconds. Therefore, high-quality reflection layer estimates can be continuously accumulated when the motion cues are sufficiently effective and reused when they fail. The definitions of the effectiveness factor, memory update rate, and separation quality score are shown in formula (10):

[0090]

[0091]

[0092]

[0093]

[0094] In the formula, As a factor for the effectiveness of motor cues, For vehicle speed, For speed reference, this embodiment uses 8 km / h. This represents the median. Using 1.5 pixels as the reference for optical flow amplitude, this embodiment takes 1.5 pixels. and These are the memory update rate and the baseline update rate, respectively. In this embodiment, the latter is set to 0.15. For the updated temporal memory of the reflector layer, To separate mass fractions, Indicates the inner product. For data residuals, Total number of pixels The residual scale is used. The validity factor considers both vehicle speed and measured optical flow amplitude, avoiding misjudgments when vehicles are moving slowly with traffic, based solely on vehicle speed. The separation quality score is composed of the product of two factors: the former is the complementary value of the normalized cross-correlation of the two gradient fields, and its larger value indicates that the two structures are more mutually exclusive and the separation is more thorough; the latter is the negative exponent of the data residual energy, and its larger value indicates that the reconstruction result fully interprets the observations; the product of the two can effectively identify failures such as misjudging the real road surface structure as a reflective layer.

[0095] The output side arbitrates based on the separation quality score: if the score is not less than the output threshold (0.62 in this embodiment), the restored transmission layer is output as the enhanced frame; otherwise, the original frame without any processing is output. This allows the method to identify and avoid its own failures, fundamentally preventing the presentation of incorrectly erased content to the user. Finally, a dual-track archiving method is used: the first track stores the completely unprocessed original bitstream, and calculates a hash digest every 2 seconds, then signs it with the device's private key; the second track stores the enhanced bitstream, and writes the enhancement flag, algorithm version number, and separation quality score of each frame into its supplementary enhancement information. The two tracks are aligned frame by frame in terms of timestamps. This provides a clear and readable enhanced image while fully preserving the original record with legal evidentiary value. Furthermore, third parties can verify whether each frame has been processed and its credibility based on the metadata.

[0096] Example 2

[0097] This embodiment illustrates the composition of the image restoration system described in this invention and its deployment on an in-vehicle embedded platform. The system composition structure is as follows: Figure 8As shown. The system 100 includes: an image and vehicle condition acquisition module 110, which includes an image sensor and image signal processing subunit 111, a vehicle speed acquisition subunit 112, and an inertial measurement subunit 113; a multi-layer reflection imaging model construction module 120, which includes a glass parameter storage subunit 121, an incident angle calculation subunit 122, and a ghost kernel and attenuation coefficient generation subunit 123, which performs a one-time calculation based on the vehicle model and caches the results as a lookup table when the device is powered on; a layered motion estimation module 130, which includes a transmission layer dense optical flow subunit 131, a reflection layer parameterized motion subunit 132, and a motion cue graph construction subunit 133; and an optical flow-guided semantic segmentation module 140, which includes a lightweight encoder 141, a motion gating attention unit 142, a dual-head decoder, and a mapping subunit 140. The module includes: a table subunit 143; a layer-based adaptive fusion module 150, which includes a segmentation entropy calculation subunit 151 and a geometric weighted fusion subunit 152; a bidirectional mutual guidance iterative control module 160, which includes a weight distribution subunit 161 and a convergence determination subunit 162; a multi-layer separation and reconstruction module 170, which includes a saturation gating subunit 171, a semi-quadratic splitting solution subunit 172 and a separation quality scoring subunit 173; a working condition adaptive and time-domain memory module 180, which includes an effectiveness factor calculation subunit 181, a reflection layer memory cache subunit 182 and a state machine scheduling subunit 183; and a dual-track output and evidence preservation module 190, which includes an output arbitration subunit 191, an original code stream signature subunit 192 and an enhanced code stream metadata writing subunit 193.

[0098] In terms of hardware deployment, this embodiment uses an automotive-grade system-on-a-chip integrating a quad-core processor, a graphics processing unit, a neural network processing unit, and a hardware video encoder, with a neural network computing power of 2 TOPS. The tasks are divided as follows: the semantic segmentation network and optical flow network are deployed in the neural network processing unit, undergoing channel pruning and 8-bit symmetric quantization. Quantization employs channel-wise asymmetric calibration with 4000 frames of real driving data as the calibration set, resulting in a layer attribution accuracy decrease of no more than 0.8 percentage points after quantization. The convolution, Fourier transform, and conjugate gradient iteration of the spatially variable ghost kernel are deployed in the graphics processing unit. The ghost kernel is divided into 8 segments based on image height, with each segment considered displacement-invariant, thus transforming the spatially variable convolution into an 8-times shifted convolution. State machine scheduling, quality scoring, hash signature, and bitstream encapsulation are deployed in the processor core. The entire process is organized in a three-stage pipeline: the first stage completes acquisition, high-pass filtering, and dense optical flow estimation; the second stage completes layered motion, segmentation, fusion, and mutual guidance iteration; and the third stage completes separation reconstruction, quality scoring, and dual-track encoding. The measured single-frame latency for the three-stage pipeline is 8.1, 10.7, and 7.5 milliseconds, respectively, with an end-to-end latency of 26.3 milliseconds, meeting the 30fps real-time requirement, and the peak power consumption increment is 1.9 watts.

[0099] To further reduce computational overhead, this embodiment employs two engineering optimizations. First, considering that the reflective layer is dominated by low-frequency components and is relatively defocused compared to the transmissive layer, the solution for the reflective layer is performed at 1 / 4 resolution. After the solution is completed, it is upsampled and guided by filtering to restore it to full resolution, while the transmissive layer is always solved at full resolution. This strategy reduces the separation and reconstruction time by approximately 58% while the peak signal-to-noise ratio decreases by only 0.11 dB. Second, the semantic segmentation network does not run frame-by-frame, but rather performs a complete inference every three frames. The intermediate frames are forward-mapped to the layer assignment probability using the estimated transmissive layer optical flow and then fused over time. If the proportion of holes after mapping exceeds 8%, a complete inference is forcibly triggered. This strategy reduces the average computational overhead of the segmentation part by approximately two-thirds.

[0100] In terms of training and data construction, the synthetic data uses real driving videos as the transmission layer and images of the dashboard and central control screen captured in the vehicle as the reflection layer. Multiple ghosting, order decay, and soft saturation responses are applied to the forward model determined by formulas (1) to (3) before being superimposed. Motion is applied to the transmission layer according to the real vehicle speed distribution, and homography perturbation is applied to the reflection layer according to the measured vehicle vibration spectrum, resulting in a total of 120,000 frames of ground truth data. Real data is collected by simultaneously installing two recorders of the same model inside and outside the vehicle, collecting 38,000 paired samples. The external camera is unaffected by reflections, and after geometric registration, it serves as the approximate ground truth for the transmission layer. The loss function includes layer-attribution cross-entropy, semantic category cross-entropy, category and layer-attribution consistency, reconstruction loss, and gradient mutual exclusion loss, with weights of 1, 0.5, 0.2, 1, and 0.1, respectively.

[0101] Example 3

[0102] This embodiment presents the implementation effects and comparative verification of the present invention under three typical difficult working conditions. The test set includes 12,000 frames of data collected from real vehicles in six cities in China, covering scenarios such as daytime, nighttime, tunnels, and rainy days. The evaluation indicators include peak signal-to-noise ratio, structural similarity, ghost residual energy ratio, layer attribution accuracy, and inter-frame flicker index. The ghost residual energy ratio is defined as the ratio of the autocorrelation peak value of the restoration result along the theoretical ghost displacement direction to the autocorrelation peak value at zero displacement.

[0103] The first scenario involves strong reflections at tunnel entrances and exits at night. In this case, the backlight of the central control display and the instrument panel creates high-contrast reflections against a dark background, and the strong light at the tunnel entrance saturates some pixels. Among the comparative methods, the single-frame de-reflection method based on gradient priors misjudges the repetitive textures of the tunnel walls as reflections and erases a large amount of them, resulting in a structural similarity lower than the original image. While the multi-frame method based on a two-layer linear model can remove the main reflections, it leaves clearly visible misaligned ghosting in the central control screen area, with a ghosting energy ratio as high as 0.184. This invention, by explicitly modeling the spatially varying displacement and attenuation of ghosting at each order and applying gating to saturated pixels, reduces this index to 0.031, improving the peak signal-to-noise ratio by 2.74 dB.

[0104] The second scenario is the stationary state while waiting at a red light during the day. In this case, both layers of motion are zero, the motion separation is close to zero across the entire image, and motion cues are completely ineffective. The multi-frame comparison method outputs almost identical images to the original, indicating complete failure, and a sudden image change occurs at the moment of starting due to the abrupt recovery of motion cues. The effectiveness factor of this invention drops below the threshold in the fourth frame after stopping, the state machine switches to a stationary low-speed state, freezes and reuses the accumulated temporal memory of the reflection layer before stopping as a template, and only solves the transmission layer sub-problem, thus maintaining a de-reflection effect comparable to the driving state even in a stationary state; during the starting process, no image change is observed due to continuous memory updates, and the inter-frame flicker index decreases by 72% compared to the comparison method.

[0105] The third scenario involves turning in the rain and passing over speed bumps. During this scenario, the vehicle's attitude changes drastically, causing significant movement of the reflective source on the image plane, while raindrops adhere to the glass, creating additional occlusion. The state machine of this invention switches to a state of severe perturbation based on the angular velocity of the inertial measurement unit, suspends memory propagation, sets the number of mutual guidance iterations to one, and increases the output threshold. This allows the system to actively backtrack and output the original frame during the ten or so frames with the most severe perturbation, avoiding motion blur; the enhanced output resumes within two frames after the perturbation ends. In this scenario, the attached raindrops are classified as occlusions by the segmentation network, and their layer assignment probability does not participate in the weighting of the cross-sparse term, thus avoiding incorrect inclusion in the reflection layer. The table below shows the quantitative comparison and ablation results of key components on the entire test set.

[0106]

[0107] As shown in the table, the present invention improves the peak signal-to-noise ratio by 2.16 dB, structural similarity by 0.064, reduces the ghost residual energy ratio from 0.171 to 0.031, and improves the layer assignment accuracy from 0.803 to 0.915 compared to the optimal comparison method, while the single-frame time is only 27.6% of the optimal method. The ablation results further indicate that the spatially variable ghost kernel contributes most significantly to the ghost residual energy ratio, and its removal degrades this indicator by four times; motion-gated attention and entropy adaptive fusion mainly contribute to the layer assignment accuracy; bidirectional mutual guidance iteration contributes to both types of indicators; the layer assignment cross-sparse term mainly improves reconstruction quality; temporal memory and state machine have limited impact on the mean indicators of the entire test set, but as described in the second case, they play a decisive role in the usability and temporal stability under static conditions. The decrease in the mean indicators is less than the actual gains under specific conditions because static conditions account for approximately 11% of the entire test set.

[0108] It should be noted that the steps in the method described in this invention do not necessarily have to be executed in the above order. Without affecting the technical effect, step S1 can be pre-executed once when the device is powered on and the result can be cached. The working condition judgment in step S7 can also be executed in parallel with step S3. The specific backbone structure of the semantic segmentation network, the specific implementation form of optical flow estimation, and the specific solution algorithm of energy functional can all be replaced with other equivalent schemes known in the art, as long as they maintain the four core features of motion cue gating injection, layer affiliation entropy adaptive fusion, bidirectional mutual guidance iteration, and layer affiliation guidance cross sparsity described in this invention.

[0109] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0110] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for restoring images of multi-layer reflection occlusion on dashcam glass using optical flow-guided semantic segmentation, characterized in that, Includes the following steps: S1. Obtain the structural parameters of the laminated glass in the windshield of the vehicle where the dashcam is located, as well as the camera intrinsic parameters of the dashcam. The structural parameters include the outer glass thickness, the thickness of the polyvinyl butyral interlayer film, the inner glass thickness, the glass refractive index, the interlayer film refractive index, and the interlayer film wedge angle. Calculate the equivalent optical thickness of the laminated glass based on the structural parameters, and calculate the pixel-by-pixel incident angle along the image height direction based on the interlayer film wedge angle. Calculate the lateral displacement of the k-th order reflection and its corresponding value based on the equivalent optical thickness and the pixel-by-pixel incident angle. The pixel-domain ghost displacement vector is used to calculate the order attenuation coefficient of the k-th order reflection based on Fresnel reflectivity, and an spatially variable ghost kernel is generated with the pixel-domain ghost displacement vector as the center and the scale parameter increasing with order as the standard deviation. A multi-layer reflection imaging model is constructed, which is as follows: the same reflection layer is convolved with the 1st to the Kth spatially variable ghost kernels respectively and weighted by the corresponding order attenuation coefficients. The resulting K reflection components are summed with the transmission layer and then mapped by the soft saturation response operator to obtain the current frame image, where K is an integer not less than 2. S2. Obtain the current frame image and the N frames preceding it, where N is an integer not less than 1, and simultaneously obtain the vehicle speed and the angular velocity output by the inertial measurement unit. S3. Perform layered motion estimation on the current frame image and its previous frame image: estimate the dense optical flow of the transmission layer; use the homography matrix constructed from the angular velocity as the initial value, and use iterative reweighted least squares to estimate the parameterized rigid motion of the reflection layer; calculate the first and second photometric residuals after aligning with the dense optical flow and the parameterized rigid motion, respectively, and the motion separation degree defined by the normalized modulus of the difference between the two motion vectors; stack the motion separation degree, the first photometric residual, the second photometric residual, and the forward and backward consistency confidence into a motion cue map, and calculate the motion layer attribution probability by the ratio of the square of the first photometric residual to the sum of the squares of the first and second photometric residuals; S4. The motion cue map is downsampled step by step and then concatenated with the feature map of the corresponding layer of the semantic segmentation network. The gating attention unit generates gating weights and modulates the feature map channel by channel. The semantic segmentation network outputs the semantic category probability and the semantic layer belonging probability. S5. Calculate the information entropy of the semantic category probability, and obtain the confidence weight from the negative exponent of the information entropy; use the confidence weight as the exponent to geometrically weight and normalize the semantic layer belonging probability and the motion layer belonging probability to obtain the reflection layer belonging probability and the transmission layer belonging probability. S6. Re-estimate the dense optical flow using the transmission layer attribution probability as the photometric term weight, and re-estimate the parameterized rigid motion using the reflection layer attribution probability as the weight of the iterative reweighted least squares. After refreshing the motion cue graph, reuse the encoder features of the semantic segmentation network to re-execute steps S4 and S5 until the mean difference between the reflection layer attribution probabilities of two adjacent rounds is less than the convergence threshold or the maximum number of iteration rounds is reached. S7. Calculate the motion cue effectiveness factor by combining the vehicle speed and the median of the dense optical flow amplitude; S8. When the motion cue effectiveness factor is not less than a set threshold, construct and solve the energy functional for the transmission layer and the reflection layer. The energy functional includes: a data fidelity term based on the multilayer reflection imaging model, which is weighted by saturation gating and monotonically decreases as the pixel value of the current frame exceeds the saturation threshold; a layer-attribution guided cross-sparse term, which is jointly formed by the reflection layer gradient sparse term weighted by the transmission layer attribution probability and the transmission layer gradient sparse term weighted by the reflection layer attribution probability; a gradient mutual exclusion term formed by multiplying the transmission layer gradient magnitude and the reflection layer gradient magnitude point by point; and a temporal consistency term that aligns the reflection layer with the temporal memory of the reflection layer along the parameterized rigid motion, where the temporal memory of the reflection layer is the reflection layer image obtained by recursively accumulating the reflection layer estimates of the previous frames; the transmission layer and the reflection layer are alternately updated using a semi-quadratic splitting method; when the motion cue effectiveness factor is less than the set threshold, the temporal memory of the reflection layer is frozen and used as the reflection layer template, and only the sub-problem of the energy functional with respect to the transmission layer is solved. S9. Update the temporal memory of the reflective layer at a rate proportional to the effectiveness factor of the motion cue, and calculate the separation quality score, which is determined by the product of the normalized cross-correlation complement of the gradient field of the transmission layer and the gradient field of the reflective layer and the negative exponent of the data residual energy. S10. When the separation quality score is not less than the output threshold, the transmission layer is used as the enhanced frame output; otherwise, the original current frame image is output back. The original bitstream and the enhanced bitstream are encoded and archived separately, and the enhancement flag, algorithm version number and the separation quality score of each frame are written into the metadata of the enhanced bitstream.

2. The method for restoring images of multi-layer reflection occlusion on dashcam glass combined with optical flow-guided semantic segmentation according to claim 1, characterized in that, In step S1, the equivalent optical thickness is determined by the sum of the outer and inner glass thicknesses plus the product of the interlayer thickness and the ratio of the glass refractive index to the interlayer refractive index; the lateral displacement of the k-th order reflection is proportional to the product of the order k, the equivalent optical thickness, and the sine and cosine of the pixel-by-pixel incident angle, and inversely proportional to the square root of the difference between the square of the glass refractive index and the square of the sine of the pixel-by-pixel incident angle; the magnitude of the pixel domain ghost displacement vector is equal to the product of the lateral displacement, the focal length, and the cosine of the incident angle divided by the virtual image distance, and its direction is along the projection direction of the laminated glass normal onto the image plane; the order attenuation coefficient is taken as the Fresnel reflectance when k equals 1, and as the product of the 2k-1 power of the reflectance, the square of its complement, and the k-1 power of the single-pass transmittance of the laminate when k is greater than 1; the scale parameter of the spatially variable ghost kernel is the product of the reference scale and the square root of the order; K is 2 to 4.

3. The method for restoring images of multi-layer reflection occlusion on dashcam glass combined with optical flow-guided semantic segmentation according to claim 1, characterized in that, In step S3, the current frame image and the previous frame image are first subjected to high-pass filtering to suppress low-frequency brightness coupling between the two layers. Then, the first photometric residual and the second photometric residual are calculated on the filtered image. The dense optical flow is estimated from coarse to fine using a four-level Gaussian pyramid. The upsampling of the previous level's estimation result is used as the initial value for each level, and the forward and backward consistency confidence is obtained by the negative exponent of the magnitude of the sum of the forward and backward optical flows. The initial value of the homography matrix is ​​obtained by multiplying the sum of the camera intrinsic matrix, the identity matrix, the antisymmetric matrix of the angular velocity, and the sampling interval, and the inverse of the camera intrinsic matrix in sequence. The loss of the iterative reweighted least squares uses the Chabonnier rod kernel.

4. The method for restoring images of multi-layer reflection occlusion on dashcam glass combined with optical flow-guided semantic segmentation according to claim 1, characterized in that, In step S4, the gating weights are obtained by convolution and logistic functions after concatenating the feature map and the downsampled motion cue map in the channel dimension. The channel-wise modulation is to multiply the feature map point by point by the result of adding the modulation intensity coefficient and the gating weights. The encoder of the semantic segmentation network includes four downsampling levels, and the gating attention unit is set at the output of each of the four downsampling levels. The semantic segmentation network is also configured with a mapping table of categories and layers, mapping the sky, road surface, lane lines, vehicles, pedestrians and traffic signs to the transmission class, the dashboard, steering wheel, A-pillar guard plate, central control display, occupants and rearview mirror to the reflection class, and the wipers and glass stains to the occlusion class. The semantic layer assignment probability is obtained by accumulating the semantic category probabilities according to the mapping table.

5. The method for restoring multi-layer reflection occlusion images of dashcam glass combined with optical flow-guided semantic segmentation according to claim 1, characterized in that, In step S5, the information entropy is the negative of the sum of the semantic category probability and its natural logarithm over all categories; the confidence weight is equal to the natural exponent of the negative of the ratio of the information entropy to the entropy benchmark; the geometric weighting is: the semantic layer attribution probability is raised to the power of the confidence weight, the motion layer attribution probability is raised to the power of one minus the confidence weight, and the product of the two is the non-normalized score of the reflection layer. The non-normalized score of the transmission layer is calculated in the same way, and the sum of the two is normalized to obtain the reflection layer attribution probability. When the information entropy increases, the confidence weight decreases, and the fusion result is biased towards motion cues. When the information entropy decreases, the fusion result is biased towards semantic cues.

6. The method for restoring images of multi-layer reflection occlusion on dashcam glass combined with optical flow-guided semantic segmentation according to claim 1, characterized in that, In step S8, the saturation gate is equal to the natural exponent of the inverse of the ratio of the square of the portion of the pixel value of the current frame image that exceeds the saturation threshold to twice the square of the gate scale; when the pixel value does not exceed the saturation threshold, the saturation gate is set to one. The semi-quadratic splitting includes: introducing an auxiliary variable and applying a soft threshold shrinkage to the gradient of the transmission layer, the shrinkage amount being the product of the weight of the layer assignment guiding cross-sparse term and the assignment probability of the reflection layer divided by a penalty factor; fixing the auxiliary variable and updating the transmission layer in the frequency domain with a closed-form solution, the denominator being a product of the penalty factor and the sum of the squares of the frequency response modes of the horizontal and vertical difference operators; fixing the transmission layer and updating the reflection layer using the conjugate gradient method; doubling the penalty factor after each alternation iteration until the relative residual is less than a set value or the maximum number of alternations is reached.

7. The method for restoring images of multi-layer reflection occlusion on dashcam glass combined with optical flow-guided semantic segmentation according to claim 1, characterized in that, Steps S7 to S10 further include switching between normal driving state, stationary low-speed state and violent disturbance state based on the motion cue effectiveness factor and the angular velocity, and the switching between adjacent states is subject to a hysteresis condition, that is, the switching criterion must be continuously met within a preset number of consecutive frames before the switching is performed; under the violent disturbance state, the propagation of the temporal memory of the reflection layer is paused, the maximum number of iterations is set to one, and the output threshold is increased; The dual-track archiving includes: calculating hash digests for segments of the original bitstream, signing them, and writing them into the first track; writing the enhanced bitstream, along with the enhancement flag, algorithm version number, and frame-by-frame separation quality score, into the second track; and aligning the two tracks frame-by-frame on the timestamps.

8. A system for restoring images of multi-layer reflection occlusion on dashcam glass combined with optical flow-guided semantic segmentation, characterized in that, include: The image and vehicle condition acquisition module is used to acquire the current frame image and the N frames preceding it, and simultaneously acquire the vehicle speed and the angular velocity output by the inertial measurement unit. The multi-layer reflection imaging model construction module is used to calculate the equivalent optical thickness and pixel-wise incident angle based on the structural parameters of the front windshield laminated glass and the camera intrinsic parameters, generate the spatially variable ghost kernel and the order attenuation coefficient, and construct the multi-layer reflection imaging model. The processing of the multi-layer reflection imaging model is as follows: the same reflection layer is convolved with the first to the Kth spatially variable ghost kernels respectively and weighted by the corresponding order attenuation coefficients. The resulting K reflection components are summed with the transmission layer and then mapped by the soft saturation response operator to obtain the current frame image. The layered motion estimation module is used to estimate the dense optical flow of the transmission layer. It uses the homography matrix constructed from the angular velocity as the initial value and employs iterative reweighted least squares to estimate the parameterized rigid motion of the reflection layer. It also constructs a motion cue map and the motion layer assignment probability from the two-layer photometric residuals, motion separation degree and forward and backward consistency confidence. The optical flow-guided semantic segmentation module is used to inject the motion cue map into the feature maps of each level of the semantic segmentation network through a gated attention unit, and output the semantic category probability and semantic layer attribution probability. The layer attribution adaptive fusion module is used to obtain confidence weights from the information entropy of the semantic category probabilities, and use these weights as an exponent to perform geometric weighted normalization on the semantic layer attribution probability and the motion layer attribution probability to obtain the reflection layer attribution probability and the transmission layer attribution probability. The bidirectional mutual guidance iterative control module is used to send the transmission layer attribution probability and the reflection layer attribution probability to the hierarchical motion estimation module as photometric term weights and iterative reweighted least squares weights, respectively, and trigger the recalculation of the reuse encoder features after refreshing the motion cue map until convergence. The multi-layer separation and reconstruction module is used to solve the energy functional containing saturation-gated data fidelity terms, layer affiliation-guided cross-sparse terms, gradient mutual exclusion terms, and time-domain consistent terms of the reflection layer using semi-quadratic splitting, and outputs the transmission layer, reflection layer, and separation quality scores. The saturation gate is a weight that monotonically decreases as the pixel value exceeds the saturation threshold. The adaptive working condition and time-domain memory module is used to calculate the motion cue effectiveness factor, schedule the solution strategy between normal driving state, stationary low speed state and violent disturbance dynamic state, and maintain the time-domain memory of the reflective layer according to the update rate proportional to the factor. The dual-track output and evidence preservation module is used to arbitrate between the enhanced frame and the original frame based on the comparison result of the separation quality score and the output threshold, and to encode and archive the original bitstream and the enhanced bitstream separately and align them frame by frame.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for restoring multi-layer reflection occlusion images of a dashcam glass combined with optical flow-guided semantic segmentation as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for restoring multi-layer reflection occlusion images of a dashcam glass combined with optical flow-guided semantic segmentation as described in any one of claims 1 to 7.

Images