Real-time visual processing method and system based on esn-cv collaborative processing

Abstract

The application discloses a real-time visual processing method and system based on ESN-CV collaborative processing, relates to the technical field of visual processing, and first synchronously collects camera images and road surface humidity data of physical sensors, extracts an image reflection intensity distribution matrix as a visual feature; then, humidity time series data and the visual feature are input into an echo state network, a dynamic weight coefficient matrix is generated through space-time feature fusion, and the dynamic weight coefficient matrix is injected into a predefined convolution layer of a target detection network to adjust kernel weight parameters in a element-by-element superposition manner, so that the feature extraction capability for high sensitive areas is enhanced. After a detection result is output, the system triggers a closed-loop feedback through a confidence bias value: when the bias exceeds a limit, an actual offset is calculated in combination with a high-precision map, an error correction vector is generated, and Hadamard product is used to update an ESN reserve pool state, so that a next frame weight generation logic is optimized in real time.

Description

Technical Field

[0001] This invention relates to the field of visual processing technology, specifically to a real-time visual processing method and system based on ESN-CV collaborative processing. Background Technology

[0002] With the rapid development of autonomous driving technology, environmental understanding based on visual perception has become the core foundation for real-time vehicle decision-making. In complex road scenarios (such as rain, nighttime, or strong reflective conditions), images captured by onboard cameras are easily affected by environmental interference, leading to misjudgments and missed detections in traditional visual algorithms for target detection (such as road markings and obstacle recognition). Existing technologies mainly face the following challenges: Traditional convolutional neural network (CNN) detection schemes rely on single visual features, which are susceptible to optical interference such as water film reflection and specular refraction in rainy or foggy weather, resulting in distorted feature extraction. Experiments show that when road surface humidity exceeds a certain threshold, the false detection rate of traditional algorithms for road markings increases significantly.

[0003] To address these issues, the industry has begun exploring solutions that combine lightweight networks with dynamic parameter tuning. Among these, Echo State Networks (ESNs), with their fixed-random connectivity in a reservoir, have demonstrated low computational overhead in time-series signal processing. However, current ESN applications are mostly focused on single-modal data processing, primarily targeting single-sensor signal prediction. There are currently no known solutions for combining ESNs with Computer Vision (CV) models for multimodal feature coupling and dynamic network parameter injection, thus failing to effectively solve the cross-modal spatiotemporal coupling problem between visual and physical sensors.

[0004] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a real-time visual processing method and system based on ESN-CV collaborative processing.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention discloses a real-time visual processing method based on ESN-CV collaborative processing, comprising the following steps:

[0008] Acquire raw image data captured by the camera and road surface moisture-related data output by the physical sensor;

[0009] The original image data is processed to extract visual feature data, including the reflectance intensity distribution matrix;

[0010] The road surface humidity-related data and the visual feature data are input into the echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix.

[0011] The dynamic weight coefficient matrix is ​​injected into the predefined convolutional layer of the target detection network to adjust its kernel weight parameters;

[0012] Based on the adjusted target detection network, road markings are detected in the current frame image, and the detection results and confidence deviation values ​​are output.

[0013] The confidence deviation value is compared with a preset deviation threshold. When the confidence deviation value exceeds the preset deviation threshold, the following operations are performed:

[0014] Extract the road marking position coordinates output by the target detection network, spatially align them with the high-precision map data, and calculate the actual offset.

[0015] An error correction vector is generated based on the actual offset, which is consistent with the number of nodes in the reservoir of the echo state network.

[0016] The error correction vector is multiplied by the state matrix of the reservoir, and the dynamic weight coefficient matrix for the next frame is recalculated based on the updated state matrix.

[0017] Secondly, this invention discloses a real-time vision processing system based on ESN-CV collaborative processing, implementing the aforementioned real-time vision processing method based on ESN-CV collaborative processing, including:

[0018] The multi-source data acquisition module is used to acquire raw image data captured by the camera and road surface humidity-related data output by the physical sensor;

[0019] The visual feature extraction module is used to process the original image data and extract visual feature data, including the reflectance intensity distribution matrix.

[0020] The ESN-CV collaborative processing module is used to input the road surface humidity-related data and the visual feature data into the echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix.

[0021] The kernel weight parameter adjustment module is used to inject the dynamic weight coefficient matrix into the predefined convolutional layer of the target detection network and adjust its kernel weight parameters.

[0022] The dynamic target detection network module is used to perform road marking detection on the current frame image based on the adjusted target detection network, and output the detection results and confidence deviation values.

[0023] The closed-loop feedback processing module is used to compare the confidence deviation value with a preset deviation threshold. When the confidence deviation value exceeds the preset deviation threshold, the following operations are performed:

[0024] Extract the road marking position coordinates output by the target detection network, spatially align them with the high-precision map data, and calculate the actual offset.

[0025] An error correction vector is generated based on the actual offset, which is consistent with the number of nodes in the reservoir of the echo state network.

[0026] The error correction vector is multiplied by the state matrix of the reservoir, and the dynamic weight coefficient matrix for the next frame is recalculated based on the updated state matrix.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0028] 1. By using ESN to perform spatiotemporal feature fusion of road surface humidity data and reflectivity intensity distribution matrix, a dynamic weight coefficient matrix is ​​generated and injected into the predefined convolutional layer of the target detection network to achieve region-sensitive parameter adjustment, adapt to environmental factors such as changes in road surface humidity and reflectivity interference, and improve detection robustness.

[0029] 2. Based on the deviation between the detection results and the high-precision map, the model parameters are corrected in real time to ensure the reliability of the autonomous driving system's decision-making in dynamic scenarios;

[0030] 3. By deeply integrating temporal physical data (such as raindrop frequency) with spatial visual features (such as reflectance intensity distribution), the limitations of traditional visual algorithms that rely on a single data source are overcome, effectively reducing the false detection rate and improving the stability of detection. Attached Figure Description

[0031] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein:

[0032] Figure 1 This is a diagram illustrating the method steps of the present invention;

[0033] Figure 2 This is a data flow diagram of the present invention;

[0034] Figure 3 This is a diagram illustrating the calculation process of the actual offset in this invention.

[0035] Figure 4 This is a flowchart of the visual feature data extraction process of the present invention;

[0036] Figure 5This is a flowchart of the spatiotemporal feature fusion of the present invention;

[0037] Figure 6 This is a flowchart of the dynamic weight coefficient matrix failure handling process of the present invention;

[0038] Figure 7 This is a system module diagram of the present invention. Detailed Implementation

[0039] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0040] Application Overview

[0041] In traditional autonomous driving vision processing systems, multimodal data fusion suffers from insufficient spatiotemporal coupling. Traditional convolutional neural networks rely solely on visual feature extraction without integrating environmental parameters from physical sensors. This leads to a non-linear relationship between reflective intensity distribution and road surface humidity in high-humidity scenarios such as rain, causing optical interference that distorts feature maps. For example, when a rain sensor detects road surface humidity exceeding a threshold, the spatial distribution characteristics of the water film reflective area in the camera-captured image are dynamically correlated with humidity parameters. However, traditional methods cannot establish a cross-modal feature coupling mechanism, resulting in the detection network kernel weights failing to adapt to environmental changes in real time.

[0042] For example, in a rainy highway scene, the vehicle camera is affected by water spray from vehicles ahead, causing specular reflection interference in the road marking area of ​​the image. The visual feature extraction module outputs a reflectivity distribution matrix, but it is not spatiotemporally fused with the average raindrop impact frequency output by the rain sensor. When the object detection network uses fixed kernel weight parameters to process high-variance grid areas, the pixel-level deviation between the center coordinates of the detection box and the high-precision map reference coordinates exceeds the image resolution tolerance range. Although the Kalman filter compensates for the vehicle pitch angle error, it does not synchronously inject attitude angle data and humidity parameters into the feature fusion process, resulting in the Hausdorff distance calculation not considering the attenuation effect of ambient humidity on reflectivity.

[0043] If the above issues are not addressed, the asynchronicity of multimodal data will cause the confidence level of detection results to continuously deviate from the safety threshold. The lack of dynamic coupling between visual features and physical parameters will cause kernel weight adjustments to lag behind environmental changes, and the detection box coordinate offset will increase non-linearly with increasing humidity. When the system does not establish a closed-loop feedback between the error correction vector and the reserve pool state matrix, the dynamic weight coefficient matrix cannot suppress the false detection probability in highly reflective areas, ultimately leading to the autonomous driving system triggering emergency braking or lane keeping anomalies, reducing driving safety redundancy.

[0044] To address the aforementioned issues, this application first analyzes the root cause of detection bias due to the asynchronicity of multimodal data, finding that the lack of a dynamic coupling mechanism between visual features and physical parameters is key. Traditional methods do not spatially correlate rain sensor data with reflectivity distribution, causing the detection network to be unable to adapt in real time to the interference of humidity changes on optical features. To resolve this, this application considers introducing a lightweight network structure with temporal processing capabilities to explore the feasibility of dynamically adjusting the detection network parameters after coupling humidity parameters with visual features.

[0045] After evaluating various network architectures, this application found that the reservoir structure of the Echo State Network (ESN) can effectively integrate the temporal humidity variation trend with the spatial reflectivity distribution. Specifically, the fixed-random connectivity of the reservoir reduces computational overhead while preserving the dynamic correlation of cross-modal features through the state matrix. Furthermore, the closed-loop feedback mechanism ensures that the dynamic weight coefficient matrix adjusts promptly according to environmental changes, avoiding the lag problem of traditional fixed-parameter networks.

[0046] like Figure 1 , Figure 2 As shown, this application proposes a real-time visual processing method based on ESN-CV collaborative processing, comprising the following steps:

[0047] The system acquires raw image data from the camera and road surface moisture-related data from physical sensors. Raw image data refers to uncompressed or noise-reduced image information directly captured by the camera, specifically in RGB three-channel pixel matrix or grayscale image format, used to preserve ambient lighting and object texture details. Road surface moisture-related data refers to a quantitative indicator reflecting the current degree of water film coverage on the road surface. This is specifically calculated by fusing rainfall intensity measured by a rain gauge and wheel slip rate obtained from a wheel speed sensor, used to characterize the impact of rainwater on image reflectivity.

[0048] The original image data is processed to extract visual feature data, including the reflectance intensity distribution matrix. The reflectance intensity distribution matrix is ​​a quantitative description of the reflectance intensity changes in local areas of the image. Specifically, it can be implemented by dividing the image into grid blocks and calculating the variance of the brightness of each grid pixel, which is used to identify image feature distortion areas caused by water film reflection.

[0049] Road surface humidity-related data and visual feature data are input into the echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix. Spatiotemporal feature fusion refers to jointly modeling the trend of road surface humidity change in the time dimension and the reflectivity distribution in the spatial dimension. Specifically, it can be achieved through the state update mechanism of the echo state network's reservoir, which is used to capture the dynamic correlation between optical interference and physical humidity parameters in rain and fog environments.

[0050] The dynamic weight coefficient matrix is ​​injected into the predefined convolutional layer of the object detection network to adjust its kernel weight parameters. The dynamic weight coefficient matrix refers to the real-time adjustment parameter used to adjust the kernel parameters of the object detection network. Specifically, it can be generated by the nonlinear transformation result output by the reservoir and is used to enhance the network's ability to assign attention weights to reflective areas.

[0051] The adjusted target detection network is used to detect road markings in the current frame image, and the detection results and confidence deviation values ​​are output. The confidence deviation value refers to the spatial difference between the detection results and the high-precision map reference. Specifically, it can be calculated by normalizing the Euclidean distance between the center coordinates of the detection box and the map coordinates, and is used to evaluate the effectiveness of the current dynamic weight parameters.

[0052] The confidence deviation value is compared with a preset deviation threshold. When the confidence deviation value exceeds the preset deviation threshold, the following operations are performed:

[0053] Extract the road marking coordinates output by the target detection network, spatially align them with high-precision map data, and calculate the actual offset.

[0054] An error correction vector is generated based on the actual offset, which is consistent with the number of nodes in the reservoir of the echo state network.

[0055] Perform a Hadamard product operation between the error correction vector and the state matrix of the reservoir, and recalculate the dynamic weight coefficient matrix for the next frame based on the updated state matrix.

[0056] The error correction vector refers to the reserve pool state adjustment parameters generated based on the actual offset. Specifically, it can be implemented by converting the geometric offset into node activation intensity adjustment values ​​through the back projection algorithm, which is used to correct the prediction bias of the reserve pool on spatiotemporal characteristics.

[0057] The core innovation of this application lies in achieving cross-modal spatiotemporal coupling between visual features and road surface humidity parameters through an echo state network, constructing dynamic weight coefficients to inject parameters into the target detection network in real time, and introducing a closed-loop feedback mechanism based on a high-precision map to form an adaptive environmental interference compensation system. This architecture effectively solves the problem of feature extraction distortion caused by optical interference in rainy and foggy environments in traditional visual algorithms, and improves the robustness of road marking detection.

[0058] The working process and principle of this application are as follows: First, raw image data and road surface moisture-related data are acquired through a camera and physical sensors. The raw image data is processed to extract visual feature data, including the reflectivity intensity distribution matrix. Then, the road surface moisture-related data and the visual feature data are input into an echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix. This matrix is ​​injected into a predefined convolutional layer of the object detection network to adjust its kernel weight parameters.

[0059] The adjusted target detection network is used to detect road markings in the current frame image, outputting the detection results and confidence deviation values. The confidence deviation value is compared with a preset deviation threshold. If the threshold is exceeded, the following steps are performed: extract the road marking position coordinates output by the target detection network, spatially align them with high-precision map data, and calculate the actual offset. An error correction vector, consistent with the number of nodes in the echo state network's reservoir, is generated based on the actual offset. The error correction vector is then used to perform a Hadamard product with the reservoir state matrix, and the dynamic weight coefficient matrix for the next frame is recalculated based on the updated state matrix.

[0060] This method achieves spatiotemporal fusion of visual features and physical sensor data through an echo state network, dynamically adjusts the target detection network parameters, and continuously optimizes the detection results through a closed-loop feedback mechanism. The reservoir structure of the echo state network can effectively fuse the temporal humidity variation trend and the spatial reflectivity distribution. Its fixed-random connectivity reduces computational overhead and preserves the dynamic correlation of cross-modal features through the state matrix.

[0061] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0062] The system acquires raw image data from a camera and road surface moisture data from a rain sensor. It performs grayscale conversion and mesh generation on the raw image data, calculates the variance of pixel brightness within each mesh region, and generates a reflectivity intensity distribution matrix. The average raindrop impact frequency and reflectivity intensity distribution matrix from the rain sensor are then input into the echo state network.

[0063] The echo state network's reservoir contains randomly connected neurons. A leaky integral mechanism within the reservoir is used to perform a nonlinear transformation on the input data, generating a dynamic weight coefficient matrix. This matrix is ​​then element-wise superimposed on the original kernel weights of predefined convolutional layers in the object detection network, adjusting the kernel weight parameters.

[0064] The adjusted object detection network is used to detect road markings in the current frame image. The center coordinates of the detection boxes for the road markings are extracted from the detection results and compared with the reference coordinates of the high-precision map to calculate the confidence deviation value. When the confidence deviation value exceeds a preset threshold, the vehicle GPS coordinates corresponding to the current frame image are mapped to the local coordinate system of the high-precision map. The attitude angle data obtained by the inertial measurement unit is fused to calculate the Hausdorff distance between the road marking position coordinates and the map reference line as the actual offset.

[0065] An error correction vector is generated based on the actual offset, and a Hadamard product is performed with the state matrix of the echo state network's reservoir. The dynamic weight coefficient matrix for the next frame is then recalculated based on the updated state matrix to achieve closed-loop feedback optimization.

[0066] Through the above scheme, this application achieves dynamic coupling of visual features and physical sensor data, solving the problem of misjudgment in target detection by traditional methods in complex road scenarios. The introduction of the echo state network enables the system to effectively integrate the temporal humidity change trend with the spatial reflectivity distribution, improving the adaptability of the detection network to environmental changes. The real-time adjustment mechanism of the dynamic weight coefficient matrix overcomes the lag of traditional fixed-parameter networks, improving the accuracy and stability of road marking detection. The closed-loop feedback mechanism ensures that the system can continuously optimize the detection results, effectively suppressing the false detection probability in highly reflective areas and improving the perception capability and driving safety of the autonomous driving system under complex weather conditions.

[0067] In some of the schemes described above in this application, when adjusting the target detection network parameters based on the dynamic weight coefficient matrix, the reliability of the detection results directly affects the effectiveness of error correction. Traditional methods lack a quantitative evaluation mechanism for the credibility of detection results, resulting in a lack of feedback basis for network parameter adjustment, which can easily lead to model performance degradation due to continuous error input.

[0068] This application further proposes a method for evaluating the reliability of test results, and the process for generating the confidence deviation value is as follows:

[0069] Based on the detection results, extract the detection boxes of road markings in the current frame image, output the center coordinates of the detection boxes, and obtain the corresponding high-precision map reference coordinates;

[0070] The confidence deviation value is calculated as follows: the confidence deviation value is equal to the absolute difference between the coordinates of the detection box center and the high-precision map reference coordinates divided by the image resolution.

[0071] Specifically, the center coordinates of the detection box are calculated using the vertex coordinates of the rectangular box output by the target detection network, employing a geometric center calculation method; the high-precision map reference coordinates are obtained by real-time matching of the vehicle's real-time positioning coordinates with map data, using a geographic coordinate transformation algorithm in the matching process; the image resolution is dynamically obtained based on the camera sensor parameters, with the resolution unit being pixels per meter; and the confidence deviation value is expressed as a dimensionless numerical value, directly reflecting the physical magnitude of the detection box's positional offset.

[0072] Specifically, after coordinate system transformation, the center point coordinates of the road marking detection boxes output by the object detection network are spatially aligned with the coordinates of the preset reference points for the road markings in the high-precision map. The Euclidean distance between the two points in the image plane coordinate system is calculated and then converted into the actual physical distance deviation based on the current image resolution. This deviation value is normalized to generate a confidence deviation value. When the deviation value exceeds a preset threshold, an error correction mechanism is triggered. For example, when the image resolution is 200 pixels / meter, if the center coordinates of the detection box differ from the reference coordinates by 50 pixels, the confidence deviation value is calculated to be 0.25 meters. This quantification provides real-time feedback signals for updating the dynamic weight coefficient matrix, ensuring a linear correlation between model parameter adjustments and real-world spatial errors.

[0073] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0074] Based on the detection results, the detection bounding boxes for road markings in the current frame image are extracted, the center coordinates of the detection bounding boxes are output, and the corresponding high-precision map reference coordinates are obtained. The formula for calculating the confidence bias value is:

[0075] Confidence deviation = |Detection box center coordinates - High-precision map reference coordinates| / Image resolution

[0076] Specifically, the detection bounding box information for road markings is first extracted from the output of the object detection network. For each detected marking, the center coordinates (x, y) of its detection box are calculated. Simultaneously, the coordinates (x', y') of the standard road marking corresponding to the current vehicle position are retrieved from a high-precision map database. Then, the Euclidean distance between the center coordinates of the detection box and the map reference coordinates is calculated. Finally, this distance is divided by the image resolution (e.g., 1920x1080) to obtain the normalized confidence bias value.

[0077] Through the above technical solution, this application can effectively quantify the degree of deviation between road marking detection results and high-precision map reference data. This allows for an objective evaluation of the performance of current detection algorithms in complex environments, providing a basis for subsequent dynamic parameter adjustments. Furthermore, by introducing image resolution for normalization, this indicator has better versatility and can be applied to image inputs of different resolutions.

[0078] In some of the solutions mentioned above in this application, a method is proposed to spatially align the road marking position coordinates output by the target detection network with high-precision map data and calculate the actual offset. However, during vehicle operation, the dynamic changes in pitch angle and fluctuations in camera viewpoint cause dynamic deviations between the vehicle GPS coordinates and the high-precision map coordinate system. At the same time, directly calculating the coordinate difference cannot accurately reflect the spatial distribution differences between the markings and the map reference lines, resulting in distorted actual offset calculation results.

[0079] like Figure 3 The diagram shown illustrates the calculation process for the actual offset. This application further proposes the following calculation process for the actual offset:

[0080] Map the vehicle GPS coordinates corresponding to the current frame image to the local coordinate system of the high-precision map;

[0081] The attitude angle data obtained by the inertial measurement unit is fused by a Kalman filter to compensate for the influence of the vehicle pitch angle on the camera's field of view.

[0082] Calculate the Hausdorff distance between the road marking coordinates and the map reference line, and use it as the actual offset;

[0083] When the actual offset exceeds the preset safety threshold, the dynamic weight coefficient matrix corresponding to the current frame image in the echo state network is frozen.

[0084] When mapping vehicle GPS coordinates to the local coordinate system of the high-precision map, a bilinear interpolation algorithm is used to correct the GPS coordinates at the sub-pixel level, ensuring that the coordinate transformation error is less than 0.1 meters. During the Kalman filter fusion of attitude angle data, the inertial measurement unit outputs pitch angle data at a frequency of 100Hz. The pitch angle change at the next moment is predicted using the state equation, and then the predicted value is weighted and fused with the actual measured value using the observation equation, with a compensation period of 10 milliseconds. The Hausdorff distance is calculated using a point-by-point scanning method, traversing all point pairs between the road marking coordinates and the map reference line, taking the maximum and minimum distances as the actual offset, with a preset safety threshold of 15 centimeters.

[0085] Specifically, in the vehicle-mounted GPS coordinate mapping stage, a local coordinate system range is first defined on a high-precision map based on the vehicle's real-time location. A coordinate transformation matrix is ​​then used to convert the GPS latitude and longitude coordinates into Cartesian coordinates in the local coordinate system. The parameters of the coordinate transformation matrix are dynamically adjusted according to the map projection type. Further, when fusing pitch angle data using a Kalman filter, the process noise covariance is set to 0.01 and the observation noise covariance to 0.05 during filter initialization. The latest data from the inertial measurement unit is acquired synchronously during each frame image processing to ensure that the pitch angle compensation delay is less than 1 millisecond. When calculating the Hausdorff distance, for the point set A formed by the road marking position coordinates and the map reference line point set B, the minimum distance set from all points in A to B and the minimum distance set from all points in B to A are calculated respectively. The maximum value of the two sets is taken as the actual offset. This method has a computational complexity of O(n²) and is suitable for road marking detection scenarios with no more than 500 points. When the actual offset exceeds 15 cm, the freezing operation of the dynamic weight coefficient matrix is ​​implemented through the flag register. The register state is maintained until it is cleared before the next frame of data input to prevent the error from being passed to the subsequent processing flow.

[0086] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0087] The vehicle's GPS coordinates corresponding to the current frame image are mapped to the local coordinate system of the high-precision map. Specifically, the latitude and longitude coordinates of the current vehicle position are obtained through the vehicle's GPS receiver, and then a coordinate transformation algorithm is used to convert them into the local coordinate system used by the high-precision map. For example, the UTM projection method is used to convert the latitude and longitude coordinates into a Cartesian coordinate system.

[0088] Furthermore, the attitude angle data acquired by the inertial measurement unit (IMU) is fused using a Kalman filter to compensate for the impact of the vehicle's pitch angle on the camera's field of view. Specifically, the vehicle's pitch, roll, and yaw angle data are acquired from the onboard IMU. This attitude angle data, along with GPS position data, is then input into the designed Kalman filter for data fusion, resulting in a more accurate vehicle attitude estimate. Then, based on the vehicle attitude and the camera's mounting position, the tilt angle of the camera's actual imaging plane relative to the ground is calculated for subsequent coordinate transformations.

[0089] Therefore, the Hausdorff distance between the road marking coordinates and the map reference line is calculated as the actual offset. Specifically, the detected road marking coordinates are first transformed from the image coordinate system to the vehicle coordinate system, and then transformed to the world coordinate system based on the vehicle's attitude. Next, the road reference line coordinates for the corresponding location are extracted from the high-resolution map. Finally, the Hausdorff distance between the transformed road marking coordinate set and the map reference line coordinate set is calculated; this distance is the actual offset.

[0090] When the actual offset exceeds a preset safety threshold, the dynamic weight coefficient matrix corresponding to the current frame image in the echo state network is frozen. For example, if the safety threshold is set to 0.5 meters, when the calculated actual offset is greater than 0.5 meters, the dynamic weight coefficient matrix corresponding to the current frame remains unchanged and is not updated. This freezing mechanism can prevent drastic fluctuations in network parameters due to sudden large deviations.

[0091] Through the above technical solution, this application can effectively compensate for the impact of vehicle attitude changes on camera imaging, improving the matching degree between road marking detection results and actual road geometry. Simultaneously, by introducing actual offset calculation and safety threshold judgment mechanisms, significant deviations between detection results and high-precision maps can be detected in a timely manner, and corresponding parameter freezing measures can be taken to avoid drastic fluctuations in network weights. This closed-loop feedback mechanism based on multi-source data fusion can significantly improve the robustness and reliability of the system under complex road conditions.

[0092] In some of the schemes described above in this application, during the process of dynamically adjusting the kernel weight parameters, the target detection network may fail to effectively capture the detailed features of road markings in subsequent layers due to insufficient shallow feature extraction or premature resolution compression, thus affecting the stability of the detection results.

[0093] This application further proposes a process for constructing an object detection network, including:

[0094] The preprocessed raw image data is input into a multi-level convolutional module from shallow to deep layers. Each convolutional module performs the following operations:

[0095] Feature extraction is performed on the input data using convolution kernels of a preset size;

[0096] The extracted feature map is subjected to nonlinear activation processing;

[0097] Max pooling operations are inserted at a preset level to reduce the feature map resolution.

[0098] The pre-defined convolutional kernels are 3×3 or 5×5 in size, used to extract image edge and texture features step by step; non-linear activation is implemented through the ReLU function to suppress the influence of negative features on subsequent layers; the pre-defined layers for max pooling are set at the output of each convolutional module, with a pooling window size of 2×2 and a stride of 2, reducing the feature map resolution to 1 / 4 of the original image.

[0099] Specifically, the raw image data, after preprocessing, is input into the first-level convolutional module. A 3×3 convolutional kernel extracts low-level features, generating a primary feature map containing the road marking contours. After ReLU activation, a 2×2 max pooling operation reduces the resolution, preserving key features and reducing computation. In the intermediate-level convolutional modules, a 5×5 convolutional kernel captures sub-pixel changes in the marking edges, combined with non-linear activation to enhance feature representation. Pooling operations are only performed in the 3rd and 5th levels to avoid excessive resolution compression leading to the loss of features in thin markings. The deep convolutional module uses a 3×3 convolutional kernel to extract global contextual information, achieving the fusion of detailed and semantic features through inter-level feature transfer, providing stable feature input for dynamic weight adjustment. For example, when the input image resolution is 1280×720, after processing by the 5-level convolutional module, the final feature map resolution is reduced to 40×22, while preserving the continuity and directional features of the markings.

[0100] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0101] The process of building an object detection network includes the following steps:

[0102] The preprocessed raw image data is input into a multi-level convolutional module, ranging from shallow to deep layers. Each convolutional module performs the following operations:

[0103] Feature extraction is performed on the input data using a convolution kernel of a preset size. Specifically, a 3x3 convolution kernel is used to perform a sliding convolution operation on the input image to extract local features.

[0104] The extracted feature maps are subjected to non-linear activation processing. For example, the ReLU activation function is used to perform a non-linear transformation on the convolutional feature maps to enhance the network's expressive power.

[0105] Max pooling operations are inserted at preset levels to reduce the feature map resolution. Furthermore, 2x2 max pooling layers are inserted after the 2nd, 4th, and 6th convolutional layers to compress the feature map size and retain salient features.

[0106] Therefore, a target detection network consisting of 8 convolutional layers and 3 pooling layers was constructed. The shallow convolutional modules primarily extract low-level features such as edges and textures, while the deep convolutional modules progressively extract higher-level semantic features. Through multi-level feature extraction and downsampling operations, the network can effectively capture multi-scale target information in images.

[0107] Through the above technical solution, this application constructs a hierarchical target detection network structure. This network progressively extracts image features through multi-level convolutional modules and introduces pooling operations at key levels to reduce computational complexity. This structural design can improve the network's computational efficiency while maintaining detection accuracy, which is beneficial for achieving real-time target detection in vehicular environments. Simultaneously, multi-level feature extraction enhances the network's adaptability to targets of different scales and improves its robustness in complex road scenarios.

[0108] In some of the solutions mentioned above in this application, it is necessary to accurately obtain road surface humidity-related data to improve the reliability of target detection. However, the traditional single-sensor measurement method has insufficient accuracy and cannot effectively reflect the actual humidity changes under complex road conditions.

[0109] This application further proposes:

[0110] Physical sensors include rain sensors and wheel speed sensors. Road surface moisture data is acquired through the following methods:

[0111] Analyze the pulse signal output by the rain sensor and calculate the average raindrop impact frequency within a preset time window;

[0112] Based on the wheel slip ratio output by the wheel speed sensor, the current road surface humidity percentage is calculated using a pre-established slip ratio-humidity mapping table.

[0113] The rain sensor counts the number of raindrop impacts by measuring the time interval of pulse signals. A preset time window of 5 seconds is used, and the average raindrop impact frequency is calculated using a moving average method. The wheel speed sensor calculates the wheel slip ratio using wheel speed pulse signals. The slip ratio-humidity mapping table is established through experimental calibration of slip ratio thresholds under different humidity conditions. For example, a slip ratio of 0.15 corresponds to a humidity percentage of 65%, and a slip ratio of 0.25 corresponds to a humidity percentage of 80%.

[0114] Specifically, the square wave pulse signal output by the rain sensor is input to the timer capture unit of the microcontroller, and the rising edge timestamp of the pulse is recorded by edge triggering. The time interval between adjacent pulses is calculated, and the arithmetic mean of all pulse intervals within the time window is calculated. The reciprocal of the average interval is converted into the raindrop impact frequency. The digital signal output by the wheel speed sensor is converted into the wheel rotation speed value. The slip ratio is calculated by comparing the speed difference between the two wheels on the same axle. The specific calculation formula is slip ratio = (driving wheel speed - non-driving wheel speed) / driving wheel speed. The pre-established slip ratio-humidity mapping table is stored as a two-dimensional lookup table structure, containing a linear mapping relationship between the slip ratio range of 0.05 to 0.35 and the corresponding humidity percentage. When the measured slip ratio is 0.18, the humidity percentage value of 68% is obtained by interpolation through the lookup table. This dual-sensor data fusion method complements physical characteristics, and can still accurately estimate the road surface humidity by changing wheel speed in heavy rain when the rain sensor is obscured by water droplets.

[0115] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0116] Physical sensors include rain sensors and wheel speed sensors. Road surface moisture data is acquired through the following methods:

[0117] The pulse signal output from the rain gauge sensor is analyzed to calculate the average raindrop impact frequency within a preset time window. Specifically, the rain gauge sensor is a piezoelectric sensor, which generates a voltage pulse when a raindrop impacts the sensor surface. A 1-minute time window is set, and the number of pulses within this time window is counted. This count is then divided by 60 seconds to obtain the average raindrop impact frequency.

[0118] Based on the wheel slip ratio output by the wheel speed sensor, the current road surface humidity percentage is calculated using a pre-established slip ratio-humidity mapping table. Further, the wheel speed sensor collects the rotational speed data of all four wheels, calculates the difference between the theoretical travel distance and the actual travel distance, and obtains the wheel slip ratio. A pre-established correspondence table between slip ratio and road surface humidity is used; for example, a slip ratio of 0%–5% corresponds to a humidity of 0%–20%, 5%–10% corresponds to a humidity of 20%–40%, and so on. The current road surface humidity percentage is then obtained by looking up the table.

[0119] Through the above technical solution, this application achieves the fusion of multi-sensor data, improving the accuracy of road surface moisture estimation. This avoids the errors that may exist with a single sensor and enhances the system's adaptability to complex road conditions. Furthermore, the use of preset time windows and mapping tables reduces computational complexity, facilitating real-time processing.

[0120] In some of the solutions described above in this application, when extracting visual feature data of the reflectivity intensity distribution matrix, traditional methods usually use global image brightness statistics or fixed threshold segmentation, which are difficult to adapt to the dynamic changes of road surface reflection under different lighting conditions. Especially in rainy and foggy weather, water film reflection exhibits local high brightness and non-uniform distribution characteristics, resulting in insufficient accuracy in the recognition of reflective areas.

[0121] like Figure 4 The diagram shown is a flowchart for visual feature data extraction; this application further proposes to extract visual feature data including a reflectance intensity distribution matrix, including:

[0122] The original image data is converted into a grayscale image and the regions are divided according to a preset grid size.

[0123] Calculate the variance of pixel brightness within each grid region to generate a reflectance intensity distribution matrix;

[0124] Grids with variance values ​​exceeding a preset variance threshold are marked as highly reflective areas.

[0125] The grayscale conversion uses a weighted average method to preserve the brightness contribution of the red, green, and blue channels. The preset grid size is set according to the ratio of the camera's field of view to the reticle width, typically ranging from 32×32 pixels to 64×64 pixels. Variance calculation uses a sliding window algorithm, with the window step size equal to the grid size to avoid missing edge information. The preset variance threshold is obtained through historical data statistics, with different threshold ranges set for sunny and rainy days, and the threshold for rainy days being 20% ​​to 30% higher than that for sunny days.

[0126] Specifically, the original RGB image is converted to grayscale to form a single-channel brightness matrix, which is then divided into several sub-regions according to a preset grid. The pixel brightness variance of each sub-region is calculated independently; the variance value reflects the dispersion of reflective intensity within that region. High-variance regions characterize abrupt brightness changes caused by specular reflection or water film refraction. For example, when the variance value within a grid exceeds a preset threshold, it is identified as a potential reflective interference region. Local reflective features are quantified through spatial grid division, avoiding the deficiency of ignoring locally high reflective points in global statistics. The generated matrix serves as input to the spatiotemporal feature fusion module, influencing the generation of dynamic weighting coefficients along with humidity data. This method effectively reduced the false detection rate by 35.7% in rainy weather experiments, with the optimal balance between computation time and detection accuracy achieved when the preset grid size was optimized to 48×48 pixels.

[0127] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0128] The original image data is first converted to a grayscale image. The grayscale image is then divided into regions using a 10x10 pixel grid. The variance of the pixel brightness within each grid region is calculated to generate a reflectance intensity distribution matrix. The variance is calculated using the squared formula for the standard deviation: Variance = (Σ(X...)) i -μ) 2 ) / N, where Xi is the brightness value of a single pixel, μ is the average brightness of pixels in the grid area, and N is the total number of pixels in the grid.

[0129] Furthermore, a variance threshold of 100 is set. Each element in the reflectivity distribution matrix is ​​compared with this threshold. If the variance value exceeds 100, the corresponding grid is marked as a high reflectivity region. High reflectivity regions are represented by 1 in the matrix, and other regions are represented by 0, ultimately resulting in a binary high reflectivity region distribution matrix.

[0130] Therefore, by using gridding and variance calculation, the reflective features in the original image are extracted into a numerical matrix representation. This representation method preserves the spatial distribution information of reflective intensity while discretizing continuous brightness values ​​into binary data that is easy to process later.

[0131] Through the above technical solution, this application can effectively extract the reflectance intensity distribution features in images. Mesh processing reduces computational complexity, and variance calculation accurately captures brightness changes in local areas. Binarization further simplifies the data structure, facilitating subsequent processing. This feature extraction method provides reliable input for subsequent spatiotemporal feature fusion and dynamic weight adjustment, helping to improve the accuracy and robustness of road marking detection under complex lighting conditions such as strong reflectivity.

[0132] In some of the above-mentioned schemes in this application, the injection method of dynamic weight coefficient matrix may cause abrupt changes in the weight of convolution kernel, resulting in fluctuations in the feature extraction capability of the target detection network. In scenarios with rapid changes in road surface humidity, gradient instability is likely to occur, affecting the continuity of road marking detection.

[0133] This application further proposes that the adjusted kernel weight is achieved by element-wise superposition of the original kernel weight and the dynamic weight coefficient matrix, and the superposition formula is: adjusted kernel weight = original kernel weight × (1 + dynamic weight coefficient).

[0134] In this system, each element of the dynamic weight coefficient matrix corresponds to the independent channel weight adjustment of the convolutional kernel. Multiplicative stacking preserves the basic feature extraction capability of the original kernel weights, avoiding gradient explosion or vanishing caused by abrupt changes in the weight value range. During the stacking process, the absolute values ​​of the dynamic weight coefficients are constrained within a preset range, for example, setting the coefficient range to [-0.1, 0.3], to prevent individual weight adjustments from exceeding the physical meaning range of the kernel parameters. This stacking method works in conjunction with the batch normalization layer of the object detection network, maintaining the numerical stability of the feature map after weight adjustment through the normalization layer.

[0135] Specifically, when the reflectivity distribution of road surfaces changes abruptly in rainy weather, the dynamic weight coefficient matrix carries information about humidity change trends and uses a superposition formula to fine-tune the kernel weights of predefined convolutional layers. For example, in the third-level convolutional module, the dynamic weight coefficients act on the depth-separable channels of the 3×3 convolutional kernel, causing the kernel parameters to be adjusted by ±10% from their original values. This adjustment process preserves the original edge detection characteristics of the convolutional kernel while enhancing the response sensitivity to highly reflective areas. The (1 + dynamic weight coefficient) structure design in the superposition coefficients ensures that the direction of weight adjustment is consistent with the sign of the original kernel parameters, preventing the reversal of positive and negative weights from causing confusion in feature extraction logic. Experimental data shows that this superposition method reduces the false detection rate of road markings by 12.3% compared to direct replacement weight adjustment in a test scenario with 65% road surface humidity.

[0136] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0137] The kernel weight parameter adjustment process for the object detection network is as follows:

[0138] The dynamic weight coefficient matrix is ​​element-wise superimposed with the original kernel weights of the predefined convolutional layers in the object detection network. The superposition formula is as follows:

[0139] Adjusted kernel weight = original kernel weight × (1 + dynamic weight coefficient).

[0140] Specifically, the dimension of the dynamic weight coefficient matrix is ​​consistent with the dimension of the kernel weights of the predefined convolutional layer. For example, for a 3x3 convolutional kernel, the dynamic weight coefficient matrix is ​​also 3x3 in size. During element-wise stacking, each element in the dynamic weight coefficient matrix is ​​multiplied by the original kernel weight at the corresponding position.

[0141] Thus, the original kernel weights are dynamically adjusted, achieving adaptive optimization of the convolutional layer parameters. The element values ​​in the dynamic weight coefficient matrix are typically in the range of [-1, 1]. Positive values ​​strengthen the original weights, while negative values ​​weaken them. This adjustment mechanism enables the network to dynamically adjust its feature extraction capabilities according to current environmental conditions.

[0142] Through the above technical solution, this application achieves dynamic adjustment of the kernel weights of the target detection network. This adjustment mechanism enables the network to adaptively optimize its feature extraction capabilities according to current environmental conditions, thereby improving the accuracy and robustness of target detection. Especially in complex and variable road environments, such as rainy days or strong reflective conditions, this method can effectively suppress the impact of environmental interference on the detection results and ensure the stability of detection performance.

[0143] In some of the solutions described above in this application, when road surface humidity-related data and visual feature data are input into the echo state network for spatiotemporal feature fusion, the lack of a modeling method for the dynamic coupling relationship between humidity change trends and the spatial distribution of highly reflective areas results in the generated dynamic weight coefficient matrix failing to effectively reflect the impact of rainfall-reflection coupling effect on visual detection, thereby affecting the parameter adjustment accuracy of the target detection network.

[0144] like Figure 5 The diagram shown is a flowchart of spatiotemporal feature fusion. This application further proposes the following specific steps for spatiotemporal feature fusion using an echo-state network:

[0145] The average raindrop impact frequency is input into the reservoir of the echo state network in time series to predict the humidity change trend within a preset time period.

[0146] The humidity change trend and the spatial coordinates of the high reflectivity area are modeled using a two-dimensional Gaussian distribution to generate a spatiotemporally coupled feature vector.

[0147] The spatiotemporal coupling feature vector is nonlinearly transformed by the leakage integral mechanism of the reservoir, and a dynamic weight coefficient matrix is ​​output.

[0148] The system receives time-series data on the average raindrop impact frequency from the reservoir and extracts periodic features of rainfall changes using its internal fixed-random connection structure to predict the humidity trend curve within the next 5 seconds. The spatial coordinates of highly reflective areas are determined by the center points of grid regions with variances exceeding 0.25 in the reflectivity distribution matrix. A two-dimensional Gaussian distribution model is used, with the time axis of the humidity trend curve as the first dimension and the spatial coordinates of the highly reflective areas as the second dimension, generating a joint probability distribution model with a mean of the predicted humidity trend and a variance of a preset empirical value of 0.3. A leakage integration mechanism employs a baseline leakage rate of 0.6 to perform a nonlinear transformation on the spatiotemporally coupled feature vector, outputting a dynamic weight coefficient matrix whose dimension matches the number of kernels in the convolutional layers of the target detection network.

[0149] Specifically, time-series data captures the lag effect of rainfall changes through a recurrent connection structure in the reservoir, predicting humidity trend curves that cover future time windows. The spatial coordinates of high-reflectivity areas and the humidity trend curve are jointly probabilistically modeled using a two-dimensional Gaussian distribution, coupling the temporal humidity changes with the spatial reflectivity intensity distribution. When the leakage integral mechanism performs a nonlinear transformation on the spatiotemporally coupled feature vector, it suppresses noise interference and retains effective features by adjusting the activation decay rate of the reservoir neurons. During the generation of the dynamic weight coefficient matrix, the time-series prediction results and spatial distribution data are softly aligned using a Gaussian kernel function, quantifying the positional deviation between humidity changes and reflectivity areas in the weight matrix. The final output dynamic weight coefficient matrix is ​​applied to the third convolutional layer of the target detection network, enhancing the network's ability to suppress high-reflectivity interference areas in rainy and foggy environments by adjusting the weight distribution of the convolutional kernels.

[0150] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0151] After analyzing the pulse signals collected by the rain sensor, the average raindrop impact frequency in the previous 30 seconds was found to be 12 times per second. This data was input into the reservoir of the echo state network in time series form. The reservoir contains 200 dynamic nodes, whose internal connection weights are randomly initialized and then kept fixed. The reservoir predicts the humidity change trend in the next 5 seconds through time expansion calculations and outputs a humidity change trend sequence. Further, the spatial coordinates of the highly reflective areas marked by the visual feature extraction module are normalized into a set of two-dimensional coordinates in the image coordinate system and jointly modeled with the humidity change trend sequence. Specifically, a two-dimensional Gaussian distribution model is constructed using the center coordinates of each highly reflective area as the mean and the humidity change rate as the variance parameter, generating a spatiotemporally coupled feature vector. The spatiotemporally coupled feature vector is processed through the leakage integration mechanism of the reservoir, where the leakage rate is set to 0.8 and the integration time constant is 0.2 seconds. After processing by a nonlinear activation function, a dynamic weight coefficient matrix with a dimension of 64×64 is output.

[0152] Through the above technical solution, this application effectively solves the problem of cross-modal spatiotemporal coupling between visual features and physical sensor data. By establishing a spatial correlation model between humidity change trends and highly reflective areas, the parameter weights of the target detection network are dynamically adjusted, enabling the network to adaptively suppress specular reflection interference in rainy and foggy environments and improve the robustness of road marking detection. Simultaneously, the nonlinear transformation of the leakage integral mechanism enhances the expressive power of temporal features, allowing the dynamic weight coefficient matrix to accurately reflect the modulation requirements of environmental changes on visual features.

[0153] In some of the above-mentioned schemes in this application, the generation of the dynamic weight coefficient matrix depends on the spatiotemporal feature fusion mechanism of the echo state network. However, during continuous multi-frame detection, when environmental interference (such as instantaneous strong reflection or raindrop occlusion) causes short-term high-frequency fluctuations in the confidence deviation value, the dynamic weight coefficient matrix may produce unexpected parameter oscillations due to sudden changes in the state of the reservoir, thereby causing continuous jumps in the detection results.

[0154] like Figure 6 The diagram shown is a flowchart for handling the failure of the dynamic weight coefficient matrix; this application further proposes:

[0155] When the confidence deviation value of the same caliber is detected to fluctuate beyond the preset fluctuation range within N consecutive frames, the current dynamic weight coefficient matrix is ​​deemed invalid.

[0156] Activate the historical weight rollback mechanism: Replace the current dynamic weight coefficient matrix with the mean of the effective dynamic weight coefficient matrix of the previous M frames;

[0157] The leakage rate of the reservoir is adaptively adjusted by synchronously triggering the echo state network, so that the leakage rate parameter satisfies:

[0158] Leakage rate parameter = preset baseline leakage rate × (1 - current confidence deviation value / maximum permissible confidence deviation value), where the maximum permissible confidence deviation value is calculated based on the camera installation height.

[0159] The historical weight rollback mechanism uses a sliding window to select the weight matrix of the previous M frames and perform an arithmetic average. For example, when M=3, the mean matrix of the effective weights of the previous three frames is used to cover the currently invalid weights. The adaptive leakage rate adjustment dynamically adjusts the reserve pool state update rate based on the ratio of the real-time confidence deviation value to the maximum allowable value. The maximum allowable value is calculated inversely proportional to the camera installation height; for example, at a height of 2 meters, the maximum allowable deviation is 5% of the image width. The mean replacement operation maintains weight dimension consistency through matrix element-level mean calculation, avoiding network structure distortion.

[0160] Specifically, when the confidence deviation of the same target line fluctuates within ±10 pixels for five consecutive frames, the system identifies this as a weight matrix failure event. By calculating the mean weight matrix from the previous three frames (those that did not exceed the tolerance), the system quickly recovers to a stable detection state. Simultaneously, based on the ratio of the current deviation of 6 pixels to the maximum allowable value of 10 pixels, the leakage rate is adjusted from the baseline of 0.8 to 0.8 × (1 - 6 / 10) = 0.32, reducing the update speed of the reserve pool state to suppress noise interference. This process uses hardware registers to implement a circular cache of the weight matrix, ensuring that the historical data retrieval latency is less than 2ms. After the mean weight injection, the network resumes steady-state output in the next frame, and the deviation value falls back within the threshold.

[0161] As a preferred embodiment, the solution of this application is implemented as follows: When the confidence deviation value of the same road marking is detected to fluctuate within ±0.15 in five consecutive frames of images, the current dynamic weight coefficient matrix is ​​determined to be invalid. At this time, the historical weight rollback mechanism is activated, the valid dynamic weight coefficient matrix verified in the previous three frames is called, its arithmetic mean is calculated, and it replaces the current invalid matrix. The adaptive adjustment of the leakage rate of the echo state network reserve pool is triggered synchronously, wherein the preset baseline leakage rate is set to 0.8, and the maximum allowable confidence deviation value is calculated to be 0.3 based on the camera installation height of 1.2 meters. The leakage rate parameter is dynamically adjusted to 0.8×(1-current deviation value / 0.3) according to the current actual confidence deviation value. When the deviation value reaches 0.3, the leakage rate drops to 0, and when the deviation value is 0, the baseline leakage rate is maintained.

[0162] Through the above technical solution, this application effectively solves the problem of fluctuation in detection results caused by weight matrix mismatch in traditional detection algorithms under complex environmental interference. By dynamically rolling back historical effective weight matrices, the accumulation of continuous errors caused by single-frame detection anomalies is avoided. At the same time, the leakage rate parameter of the reserve pool is adaptively adjusted, enhancing the network's ability to suppress abnormal states and ensuring the spatiotemporal continuity of road marking detection results. This mechanism significantly reduces the risk of weight matrix mutations caused by water film reflection in rainy and foggy weather, improving the robustness and environmental adaptability of the detection system.

[0163] In some of the solutions mentioned above in this application, visual perception-based environmental understanding is easily affected by environmental interference in complex road scenarios, leading to misjudgments and missed detections in traditional visual algorithms. Traditional convolutional neural networks rely on single visual features and are susceptible to optical interference such as water film reflection and specular refraction in rainy or foggy weather, resulting in distorted feature extraction. Existing echo-state network applications are mostly focused on single-modal data processing and have not yet solved the cross-modal spatiotemporal coupling problem between visual and physical sensors.

[0164] like Figure 7 As shown, this application further proposes a real-time visual processing system based on ESN-CV collaborative processing, which implements a real-time visual processing method based on ESN-CV collaborative processing, including a multi-source data acquisition module, a visual feature extraction module, an ESN-CV collaborative processing module, a kernel weight parameter adjustment module, a dynamic target detection network module, and a closed-loop feedback processing module.

[0165] The multi-source data acquisition module is configured to acquire raw image data collected by the camera and road surface humidity-related data output by physical sensors, including a rain sensor and a wheel speed sensor.

[0166] The visual feature extraction module is configured to convert the original image data into a grayscale image and divide the region according to a preset grid size, calculate the variance value of the pixel brightness in each grid region, generate a reflective intensity distribution matrix, and mark the grid with variance value exceeding the preset variance threshold as a high reflective region.

[0167] The ESN-CV collaborative processing module is configured to input the average raindrop impact frequency into the reservoir of the echo state network in a time series, predict the humidity change trend within a preset time period, model the humidity change trend with the spatial coordinates of the high reflectivity area using a two-dimensional Gaussian distribution, generate a spatiotemporal coupled feature vector, and perform a nonlinear transformation on the spatiotemporal coupled feature vector through the leakage integration mechanism of the reservoir to output a dynamic weight coefficient matrix.

[0168] The kernel weight parameter adjustment module is configured to perform element-wise superposition of the dynamic weight coefficient matrix with the original kernel weights of the predefined convolutional layers in the object detection network. The superposition formula is as follows:

[0169] Adjusted kernel weight = original kernel weight × (1 + dynamic weight coefficient).

[0170] The dynamic target detection network module is configured to perform road marking detection on the current frame image based on the adjusted target detection network, and output the detection results and confidence deviation value. The confidence deviation value is calculated by dividing the absolute difference between the center coordinates of the detection box and the reference coordinates of the high-precision map by the image resolution.

[0171] The closed-loop feedback processing module is configured to compare the confidence deviation value with a preset deviation threshold. When the confidence deviation value exceeds the preset deviation threshold:

[0172] The coordinates of road markings are extracted and spatially aligned with high-precision map data. The actual offset is calculated, and an error correction vector with the same number of nodes as the reserve pool is generated. The error correction vector is then subjected to a Hadamard product operation with the state matrix of the reserve pool. Based on the updated state matrix, the dynamic weight coefficient matrix for the next frame is recalculated.

[0173] Specifically, the multi-source data acquisition module synchronously acquires raw images and road surface humidity data through cameras and physical sensors. The slip rate data from the wheel speed sensor is used to calculate the road surface humidity percentage using a pre-established mapping table. The visual feature extraction module divides the image into grids and calculates the brightness variance, identifying highly reflective areas to eliminate optical interference. The ESN-CV collaborative processing module performs spatiotemporal modeling of humidity trends and highly reflective area coordinates, fusing cross-modal features using the leakage integration mechanism of the reservoir to generate a dynamic weight coefficient matrix. The kernel weight parameter adjustment module dynamically injects convolution kernel parameters through element-wise stacking, enhancing the network's robustness to reflective interference. After the dynamic target detection network module outputs the detection results, the closed-loop feedback processing module calculates the actual offset by aligning the confidence deviation value with the spatial alignment of the high-precision map. When the offset exceeds the limit, an error correction vector is generated, and the reservoir state matrix is ​​updated through the Hadamard product, thereby adjusting the weight coefficients for the next frame. This closed-loop mechanism allows the system to automatically correct model parameters when detection errors accumulate, avoiding the problem of continuous error amplification caused by environmental interference in traditional methods.

[0174] As a preferred embodiment, the solution of this application is implemented as follows: The system includes a multi-source data acquisition module, which is equipped with an automotive-grade global shutter camera (model OV2740) to acquire raw image data at a resolution of 1280×720 at 30fps. Simultaneously, it reads the pulse signal output by the rain sensor and the slip rate signal from the wheel speed sensor in real time via a LIN bus interface. The visual feature extraction module is built based on the OpenCV library. After converting the raw image to grayscale format, it divides the region into 16×16 pixel grids, calculates the brightness variance of each grid, and generates a reflective intensity distribution matrix. The ESN-CV collaborative processing module uses an echo-state network with a reservoir size of 200 nodes to spatiotemporally couple the raindrop impact frequency time series with the reflective intensity matrix, and outputs a dynamic weight coefficient matrix through a leakage integrator. The kernel weight parameter adjustment module element-wise superimposes the dynamic weight coefficients with the original weights of the 12th layer convolutional kernel of the YOLOv5 network to generate adjusted convolutional kernel parameters. The dynamic target detection network module deploys an optimized detection model based on the TensorRT engine, outputting the coordinates of the detection boxes and the confidence deviation value. When the detection deviation exceeds the limit, the closed-loop feedback processing module calls the high-precision map API to perform coordinate space alignment, calculates the Hausdorff distance to generate an error correction vector, and updates the reserve pool state matrix through the Hadamard product.

[0175] Through the above technical solutions, this application achieves cross-modal spatiotemporal coupling of visual features and physical sensing data, effectively suppresses the interference of specular reflection on image features in rainy and high-humidity scenarios, and enhances the environmental adaptability of the detection model by dynamically adjusting the convolution kernel parameters. The closed-loop feedback mechanism corrects the network weight generation process in real time, reduces the risk of cumulative road marking positioning errors, and improves the stability and reliability of road marking detection in complex optical environments.

[0176] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A real-time visual processing method based on ESN-CV collaborative processing, characterized in that: Includes the following steps: Acquire raw image data captured by the camera and road surface moisture-related data output by the physical sensor; The original image data is processed to extract visual feature data, including the reflectance intensity distribution matrix; The road surface humidity-related data and the visual feature data are input into the echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix. The dynamic weight coefficient matrix is ​​injected into the predefined convolutional layer of the target detection network to adjust its kernel weight parameters; Based on the adjusted target detection network, road markings are detected in the current frame image, and the detection results and confidence deviation values ​​are output. The confidence deviation value is compared with a preset deviation threshold. When the confidence deviation value exceeds the preset deviation threshold, the following operations are performed: Extract the road marking position coordinates output by the target detection network, spatially align them with the high-precision map data, and calculate the actual offset. An error correction vector is generated based on the actual offset, which is consistent with the number of nodes in the reservoir of the echo state network. Perform a Hadamard product operation between the error correction vector and the state matrix of the reserve pool, and recalculate the dynamic weight coefficient matrix for the next frame based on the updated state matrix; The physical sensors include a rain sensor and a wheel speed sensor, and the road surface humidity-related data are obtained through the following methods: The pulse signal output by the rain sensor is analyzed, and the average raindrop impact frequency within a preset time window is calculated. Based on the wheel slip ratio output by the wheel speed sensor, the current road surface humidity percentage is calculated using a pre-established slip ratio-humidity mapping table; The extraction of visual feature data, including the reflectance intensity distribution matrix, includes: The original image data is converted into a grayscale image and the regions are divided according to a preset grid size; Calculate the variance of pixel brightness within each grid region to generate the reflective intensity distribution matrix; Grids with variance values ​​exceeding a preset variance threshold are marked as highly reflective areas; The specific steps for spatiotemporal feature fusion in the echo state network are as follows: The average raindrop impact frequency is input into the reservoir of the echo state network in time series to predict the humidity change trend within a preset time period. The humidity change trend and the spatial coordinates of the highly reflective area are modeled using a two-dimensional Gaussian distribution to generate a spatiotemporally coupled feature vector. The spatiotemporal coupling feature vector is nonlinearly transformed using the leakage integral mechanism of the reservoir to output a dynamic weight coefficient matrix.

2. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The process for generating the confidence bias value is as follows: Based on the detection results, extract the detection boxes of road markings in the current frame image, output the center coordinates of the detection boxes, and obtain the corresponding high-precision map reference coordinates; The formula for calculating the confidence level deviation is: Confidence deviation = |Detection box center coordinates - High-precision map reference coordinates| / Image resolution.

3. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The calculation process for the actual offset is as follows: Map the vehicle GPS coordinates corresponding to the current frame image to the local coordinate system of the high-precision map; The attitude angle data acquired by the inertial measurement unit is fused using a Kalman filter. Calculate the Hausdorff distance between the road marking location coordinates and the map reference line, and use it as the actual offset; When the actual offset exceeds a preset safety threshold, the dynamic weight coefficient matrix corresponding to the current frame image in the echo state network is frozen.

4. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The construction process of the target detection network includes: The preprocessed raw image data is input into a multi-level convolutional module from shallow to deep layers. Each convolutional module performs the following operations: Feature extraction is performed on the input data using convolution kernels of a preset size; The extracted feature map is subjected to nonlinear activation processing; Max pooling operations are inserted at a preset level to reduce the feature map resolution.

5. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The adjustment process of the kernel weight parameters of the target detection network is as follows: The dynamic weight coefficient matrix is ​​element-wise superimposed with the original kernel weights of the predefined convolutional layers in the object detection network. The superposition formula is as follows: Adjusted kernel weight = original kernel weight × (1 + dynamic weight coefficient).

6. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: Also includes: When the confidence deviation value of the same caliber is detected to fluctuate beyond the preset fluctuation range within N consecutive frames, the current dynamic weight coefficient matrix is ​​deemed invalid. Activate the historical weight rollback mechanism: Replace the current dynamic weight coefficient matrix with the mean of the effective dynamic weight coefficient matrix of the previous M frames; The echo state network is synchronously triggered to adaptively adjust the leakage rate of the reservoir, so that the leakage rate parameter satisfies: Leakage rate parameter = preset baseline leakage rate × (1 - current confidence level deviation / maximum allowable confidence level deviation); The maximum permissible confidence deviation value is calculated based on the camera's installation height.

7. A real-time vision processing system based on ESN-CV collaborative processing, characterized in that: The real-time visual processing method based on ESN-CV collaborative processing as described in any one of claims 1 to 6 includes: The multi-source data acquisition module is used to acquire raw image data captured by the camera and road surface humidity-related data output by the physical sensor; The visual feature extraction module is used to process the original image data and extract visual feature data, including the reflectance intensity distribution matrix. The ESN-CV collaborative processing module is used to input the road surface humidity-related data and the visual feature data into the echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix. The kernel weight parameter adjustment module is used to inject the dynamic weight coefficient matrix into the predefined convolutional layer of the target detection network and adjust its kernel weight parameters. The dynamic target detection network module is used to perform road marking detection on the current frame image based on the adjusted target detection network, and output the detection results and confidence deviation values. The closed-loop feedback processing module is used to compare the confidence deviation value with a preset deviation threshold. When the confidence deviation value exceeds the preset deviation threshold, the following operations are performed: Extract the road marking position coordinates output by the target detection network, spatially align them with the high-precision map data, and calculate the actual offset. An error correction vector is generated based on the actual offset, which is consistent with the number of nodes in the reservoir of the echo state network. The error correction vector is multiplied by the state matrix of the reservoir, and the dynamic weight coefficient matrix for the next frame is recalculated based on the updated state matrix.

Images