Real-time visual processing method and system based on ESN-CV cooperative processing

Abstract

The invention discloses a real-time visual processing method and system based on ESN-CV coprocessing, and relates to the technical field of visual processing, and the method comprises the steps: firstly, synchronously collecting a camera image and road surface humidity data of a physical sensor, and extracting an image reflection intensity distribution matrix as a visual feature; then humidity time sequence data and visual features are input into an echo state network, a dynamic weight coefficient matrix is generated through spatio-temporal feature fusion, the dynamic weight coefficient matrix is injected into a predefined convolutional layer of a target detection network, kernel weight parameters are adjusted in an element-by-element superposition mode, and the feature extraction capacity of a high-sensitivity area is enhanced. And after a detection result is output, the system triggers closed-loop feedback through a confidence coefficient deviation value: when the deviation exceeds a limit, the actual offset is calculated by combining a high-precision map, an error correction vector is generated, the state of the ESN reserve pool is updated by utilizing a Hadamard product, and the weight generation logic of the next frame is optimized in real time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual processing, in particular to a real-time visual processing method and system based on ESN-CV collaborative processing. BACKGROUND

[0002] With the rapid development of automatic driving technology, environment understanding based on visual perception has become the core foundation of vehicle real-time decision-making. In complex road scenes (such as rainy days, night or strong light conditions), images captured by vehicle-mounted cameras are easily disturbed by the environment, resulting in false positives, missed detections and other problems in target detection (such as road markings, obstacle recognition) of traditional visual algorithms. The existing technology mainly faces the following challenges: the detection scheme of traditional convolutional neural network (CNN) relies on a single visual feature, which is easily disturbed by water film reflection, mirror refraction and other optical interference in rainy and foggy weather, resulting in distortion of feature extraction. Experiments show that when the road surface humidity exceeds a certain threshold, the false detection rate of the traditional algorithm for road markings increases significantly.

[0003] In view of the above problems, the industry has begun to explore the combination scheme of lightweight network and dynamic parameter adjustment. Among them, the echo state network (Echo State Network, ESN) has the advantage of low computational overhead in time series signal processing due to its fixed random connection characteristics of the reservoir. However, existing ESN applications are mainly focused on single modal data processing, mainly for single sensor signal prediction, and there is no technical solution for coupling multiple modal features and dynamically injecting network parameters in collaboration with computer vision (Computer Vision, CV) models. The cross-modal spatio-temporal coupling problem of vision-physical sensors has not been effectively solved.

[0004] In view of the above problems, the existing technology needs to be improved. SUMMARY

[0005] In view of the deficiencies of the existing technology, the present application provides a real-time visual processing method and system based on ESN-CV collaborative processing.

[0006] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0007] In the first aspect, the present application discloses a real-time visual processing method based on ESN-CV collaborative processing, comprising the following steps:

[0008] Obtaining raw image data captured by a camera and road surface humidity related data output by a physical sensor;

[0009] Processing the raw image data to extract visual feature data including a reflection intensity distribution matrix;

[0010] The road surface humidity related data and the visual feature data are input into an echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix;

[0011] The dynamic weight coefficient matrix is injected into a predefined convolutional layer of a target detection network to adjust kernel weight parameters thereof;

[0012] The target detection network is adjusted based on the dynamic weight coefficient matrix, and the target detection network is used to perform road marking detection on a current frame of image to output a detection result and a confidence bias value;

[0013] The confidence bias value is compared with a preset bias threshold value, and when the confidence bias value exceeds the preset bias threshold value, the following operations are performed:

[0014] The road marking position coordinates output by the target detection network are extracted, and are spatially aligned with high-precision map data to calculate an actual offset;

[0015] An error correction vector consistent with the number of nodes of a reservoir of the echo state network is generated based on the actual offset;

[0016] The error correction vector and a state matrix of the reservoir are subjected to Hadamard product operation, and the dynamic weight coefficient matrix of the next frame is recalculated based on the updated state matrix.

[0017] In a second aspect, the present application discloses a real-time visual processing system based on ESN-CV collaborative processing, which implements the real-time visual processing method based on ESN-CV collaborative processing described above, and comprises:

[0018] A multi-source data acquisition module is configured to acquire original image data collected by a camera and road surface humidity related data output by a physical sensor;

[0019] A visual feature extraction module is configured to process the original image data and extract visual feature data including a reflection intensity distribution matrix;

[0020] An ESN-CV collaborative processing module is configured to input the road surface humidity related data and the visual feature data into an echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix;

[0021] A kernel weight parameter adjustment module is configured to inject the dynamic weight coefficient matrix into a predefined convolutional layer of a target detection network to adjust kernel weight parameters thereof;

[0022] A dynamic target detection network module is configured to perform road marking detection on a current frame of image based on the target detection network adjusted based on the dynamic weight coefficient matrix to output a detection result and a confidence bias value;

[0023] a closed-loop feedback processing module configured to compare the confidence bias value with a preset bias threshold value, and perform the following operations when the confidence bias value exceeds the preset bias threshold value:

[0024] extracting road marking position coordinates output by the target detection network, performing spatial alignment with high-precision map data, and calculating an actual offset;

[0025] generating an error correction vector consistent with the number of nodes of the reservoir of the echo state network according to the actual offset;

[0026] performing Hadamard product operation on the error correction vector and the state matrix of the reservoir, and recalculating the dynamic weight coefficient matrix of the next frame based on the updated state matrix.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] 1. The ESN is used to perform spatiotemporal feature fusion on the road humidity data and the reflection intensity distribution matrix to generate a dynamic weight coefficient matrix, which is injected into the predefined convolution layer of the target detection network to realize region-sensitive parameter adjustment, adapt to environmental factors such as road humidity change and reflection interference, and improve the detection robustness;

[0029] 2. The model parameters are corrected in real time based on the deviation of the detection result and the high-precision map to ensure the decision reliability of the autonomous driving system in a dynamic scene;

[0030] 3. The time-series physical data (such as raindrop frequency) and the spatial visual feature (such as reflection intensity distribution) are deeply fused to break through the limitation of traditional visual algorithms relying on a single data source, effectively reduce the false detection rate, and improve the stability of detection. BRIEF DESCRIPTION OF DRAWINGS

[0031] The disclosure of the present application will be described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes, and are not intended to limit the scope of protection of the present application. In the drawings, the same reference numerals are used to refer to the same parts. Among them:

[0032] Figure 1 is a method step diagram of the present application;

[0033] Figure 2 is a data flow diagram of the present application;

[0034] Figure 3 is a calculation process diagram of the actual offset of the present application;

[0035] Figure 4 is a visual feature data extraction flow diagram of the present application;

[0036] Figure 5Flow chart for spatio-temporal feature fusion of the present application;

[0037] Figure 6 Flow chart for dynamic weight coefficient matrix failure processing of the present application;

[0038] Figure 7 System module diagram of the present application. DETAILED DESCRIPTION

[0039] It is easy to understand that, according to the technical solution of the present application, those skilled in the art can propose a plurality of structure modes and implementation modes which can be replaced with each other without changing the essential spirit of the present application. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present application, and should not be regarded as the whole or as a limitation or restriction on the technical solution of the present application.

[0040] SUMMARY

[0041] In the traditional existing automatic driving vision processing system, there is a defect of insufficient spatio-temporal coupling in multi-modal data fusion. The traditional convolutional neural network only relies on visual feature extraction and does not fuse the environmental parameters output by the physical sensor, resulting in a nonlinear correlation between the reflectivity intensity distribution and the road humidity in high-humidity scenes such as rainy days, and the feature map distortion caused by optical interference. For example, when the rain sensor detects that the road humidity exceeds the threshold, the spatial distribution characteristics of the water film reflection area in the image captured by the camera have a dynamic correlation with the humidity parameter, but the traditional method cannot establish a cross-modal feature coupling mechanism, resulting in the inability of the detection network kernel weight to adapt to environmental changes in real time.

[0042] For example, in a rainy highway scene, the vehicle-mounted camera is affected by the splashing water mist of the preceding vehicle, and the road marking area in the image appears mirror reflection interference. The visual feature extraction module outputs the reflectivity intensity distribution matrix, but does not perform spatio-temporal fusion with the average raindrop impact frequency output by the rain sensor. When the target detection network uses fixed kernel weight parameters to process the high-variance grid area, the pixel-level deviation of the detection box center coordinates from the high-precision map reference coordinates exceeds the tolerance range of the image resolution. Although the Kalman filter compensates for the vehicle pitch angle error, it does not synchronize the attitude angle data with the humidity parameter into the feature fusion process, resulting in the Hausdorff distance calculation not considering the attenuation effect of environmental humidity on reflectivity intensity.

[0043] If the above problems are not solved, the multi-modal data asynchrony will cause the detection result confidence to deviate from the safety threshold continuously. The lack of dynamic coupling between visual features and physical parameters will cause the nuclear weight adjustment to lag behind the environmental changes, and the detection box coordinate offset will increase nonlinearly with the increase of humidity. When the system does not establish a closed-loop feedback of the error correction vector and the reserve pool state matrix, the dynamic weight coefficient matrix cannot suppress the false detection probability in the high light reflection area, eventually leading to the automatic driving system triggering emergency braking or lane keeping abnormality, reducing the driving safety redundancy.

[0044] In the face of the above problems, the present application first analyzes the root cause of the detection deviation caused by multi-modal data asynchrony, and finds that the lack of dynamic coupling mechanism between visual features and physical parameters is the key. The traditional method does not perform spatio-temporal correlation between the rainfall sensor data and the light intensity distribution, resulting in the detection network being unable to adapt to the disturbance of humidity changes on optical features in real time. In view of this, the present application considers introducing a lightweight network structure with time series processing capability, and explores the feasibility of dynamically adjusting the detection network parameters after coupling the humidity parameters and visual features.

[0045] After evaluating a variety of network architectures, the present application finds that the reserve pool structure of the echo state network (ESN) can effectively fuse the humidity change trend in the time dimension and the light intensity distribution in the space dimension. Among them, the fixed random connection characteristics of the reserve pool can not only reduce the computational overhead, but also retain the dynamic association of cross-modal features through the state matrix. In addition, the design of the closed-loop feedback mechanism can ensure that the dynamic weight coefficient matrix is adjusted in a timely manner with environmental changes, avoiding the lag problem of traditional fixed parameter networks.

[0046] As shown in Figure 1 , Figure 2 , the present application proposes a real-time visual processing method based on ESN-CV collaborative processing, including the following steps:

[0047] Obtain the original image data captured by the camera and the road surface humidity related data output by the physical sensor; the original image data refers to the image information captured directly by the camera without compression or noise reduction processing, which can be implemented in RGB three-channel pixel matrix or grayscale image format to retain the environmental lighting and object texture details. The road surface humidity related data refers to a quantitative index reflecting the current road surface water film coverage degree, which can be calculated by fusing the rainfall intensity measured by the rainfall sensor and the wheel slip rate obtained by the wheel speed sensor, and is used to represent the influence degree of rainwater on the image light reflection characteristics.

[0048] The original image data is processed to extract visual feature data including a reflection intensity distribution matrix; the reflection intensity distribution matrix is a quantitative description of the change of the reflection intensity of the local area of the image, which can be realized by grid partitioning and calculating the brightness variance of each grid pixel, and is used to identify the image feature distortion area caused by water film reflection.

[0049] The road surface humidity related data and the visual feature data are input into the echo state network for spatio-temporal feature fusion to generate a dynamic weight coefficient matrix; the spatio-temporal feature fusion refers to jointly modeling the time dimension road surface humidity change trend and the space dimension reflection intensity distribution, which can be realized by the reserve pool state updating mechanism of the echo state network, and is used to capture the dynamic correlation between optical interference and physical humidity parameters in a rain and fog environment.

[0050] The dynamic weight coefficient matrix is injected into the predefined convolution layer of the target detection network to adjust the kernel weight parameters; the dynamic weight coefficient matrix is a real-time adjustment parameter for adjusting the convolution kernel parameters of the target detection network, which can be realized by generating a nonlinear transformation result of the reserve pool output, and is used to enhance the attention weight distribution ability of the network to the reflection area.

[0051] The adjusted target detection network is used to detect the road marking of the current frame image, and the detection result and the confidence deviation value are output; the confidence deviation value refers to the spatial position difference between the detection result and the high-precision map benchmark, which can be realized by calculating the Euclidean distance between the detection frame center coordinates and the map coordinates, and is used to evaluate the effectiveness of the current dynamic weight parameter.

[0052] The confidence deviation value is compared with the preset deviation threshold, and when the confidence deviation value exceeds the preset deviation threshold, the following operations are performed:

[0053] The road marking position coordinates output by the target detection network are extracted, and are spatially aligned with the high-precision map data to calculate the actual offset;

[0054] An error correction vector consistent with the number of nodes of the reserve pool of the echo state network is generated according to the actual offset;

[0055] The error correction vector and the state matrix of the reserve pool are subjected to Hadamard product operation, and the dynamic weight coefficient matrix of the next frame is recalculated based on the updated state matrix.

[0056] The error correction vector is a reserve pool state adjustment parameter generated based on the actual offset, which can be realized by converting the geometric offset into a node activation intensity adjustment value through a back projection algorithm, and is used to correct the prediction deviation of the reserve pool to the spatio-temporal features.

[0057] The core innovation of the present application is to realize the cross-modal spatio-temporal coupling of visual features and road surface humidity parameters through an echo state network, construct a dynamic weight coefficient to perform real-time parameter injection on a target detection network, and introduce a closed-loop feedback mechanism based on a high-precision map to form an adaptive environmental interference compensation system. This architecture effectively solves the feature extraction distortion problem caused by optical interference in rain and fog environments for traditional visual algorithms, and improves the robustness of road marking detection.

[0058] The working process and principle of the present application are as follows: first, raw image data and road surface humidity related data are obtained through a camera and a physical sensor. The raw image data is processed to extract visual feature data including a reflection intensity distribution matrix. Then, the road surface humidity related data and the visual feature data are input into an echo state network for spatio-temporal feature fusion to generate a dynamic weight coefficient matrix. This matrix is injected into a predefined convolution layer of the target detection network to adjust its kernel weight parameters.

[0059] Based on the adjusted target detection network, road marking detection is performed on the current frame image to output a detection result and a confidence deviation value. The confidence deviation value is compared with a preset deviation threshold value. When the threshold value is exceeded, the following operations are performed: the road marking position coordinates output by the target detection network are extracted, spatially aligned with the high-precision map data, and the actual offset is calculated. An error correction vector consistent with the number of reservoir nodes of the echo state network is generated based on the actual offset. The error correction vector is subjected to Hadamard product operation with the reservoir state matrix, and the dynamic weight coefficient matrix of the next frame is recalculated based on the updated state matrix.

[0060] This method realizes spatio-temporal fusion of visual features and physical sensor data through an echo state network, dynamically adjusts target detection network parameters, and continuously optimizes detection results through a closed-loop feedback mechanism. The reservoir structure of the echo state network can effectively fuse the humidity change trend in the time dimension and the reflection intensity distribution in the space dimension. Its fixed random connection characteristic not only reduces the computational overhead, but also preserves the dynamic association of cross-modal features through the state matrix.

[0061] As a preferred embodiment, the scheme of the present application is implemented as follows:

[0062] Raw image data captured by a camera and road surface humidity data output by a rain sensor are obtained. The raw image data is subjected to grayscale conversion and grid division, the variance value of pixel brightness in each grid region is calculated, and a reflection intensity distribution matrix is generated. The average raindrop impact frequency output by the rain sensor and the reflection intensity distribution matrix are input into an echo state network.

[0063] The reservoir of the echo state network comprises randomly connected neuron nodes. The input data is nonlinearly transformed by a leaky integrate-and-fire mechanism of the reservoir to generate a dynamic weight coefficient matrix. The matrix is element-wise superimposed with original kernel weights of a predefined convolution layer in the target detection network to adjust the kernel weight parameters.

[0064] The current frame image is subjected to road marking detection using the adjusted target detection network. The center coordinates of the detection frame of the road marking in the detection result are extracted, compared with the high-precision map reference coordinates, and a confidence deviation value is calculated. When the confidence deviation value exceeds a preset threshold, the vehicle-mounted GPS coordinates corresponding to the current frame image are mapped to the local coordinate system of the high-precision map, the attitude angle data obtained by the inertial measurement unit is fused, and the Hausdorff distance between the road marking position coordinates and the map reference line is calculated as the actual offset.

[0065] An error correction vector is generated according to the actual offset, and a Hadamard product operation is performed on the state matrix of the reservoir of the echo state network. The dynamic weight coefficient matrix of the next frame is recalculated based on the updated state matrix to realize closed-loop feedback optimization.

[0066] Through the above scheme, the present application realizes dynamic coupling of visual features and physical sensor data, and solves the misjudgment problem of target detection in a complex road scene in a traditional method. The introduction of the echo state network enables the system to effectively fuse the humidity change trend in the time dimension and the reflection intensity distribution in the space dimension, improving the adaptability of the detection network to environmental changes. The real-time adjustment mechanism of the dynamic weight coefficient matrix overcomes the hysteresis of the traditional fixed parameter network, improving the accuracy and stability of road marking detection. The design of the closed-loop feedback mechanism ensures that the system can continuously optimize the detection result, effectively suppresses the false detection probability in the high-reflectivity area, and improves the perception ability and driving safety of the autonomous driving system in complex weather conditions.

[0067] In some of the above schemes of the present application, when adjusting the parameters of the target detection network based on the dynamic weight coefficient matrix, the reliability of the detection result directly affects the effectiveness of the error correction. Traditional methods lack a quantitative evaluation mechanism for the reliability of the detection result, resulting in a lack of feedback basis for network parameter adjustment, and the model performance is easily degraded by continuous error input.

[0068] The present application further proposes an evaluation method for the reliability of the detection result, and the generation process of the confidence deviation value is as follows:

[0069] According to the detection result, the detection frame of the road marking of the current frame image is extracted, the center coordinates of the detection frame are output, and the corresponding high-precision map reference coordinates are obtained;

[0070] The calculation formula of the confidence deviation value is: the confidence deviation value is equal to the absolute difference between the center coordinates of the detection frame and the reference coordinates of the high-precision map divided by the image resolution.

[0071] The center coordinates of the detection frame are calculated from the rectangular frame vertex coordinates output by the target detection network, and the geometric center calculation method is adopted; the reference coordinates of the high-precision map are obtained by real-time matching of the vehicle real-time positioning coordinates and the map data, and the geographic coordinate conversion algorithm is adopted in the matching process; the image resolution is dynamically obtained according to the camera sensor parameters, and the resolution unit is pixel / meter; the calculation result of the confidence deviation value is expressed as a dimensionless value, which directly reflects the physical order of magnitude of the position deviation of the detection frame.

[0072] Specifically, after the center point coordinates of the road marking detection frame output by the target detection network are converted through the coordinate system, the center point coordinates are spatially aligned with the reference point coordinates of the marking in the high-precision map. The Euclidean distance of the two in the image plane coordinate system is calculated, and then the actual physical distance deviation is converted according to the current image resolution. The deviation value is normalized to generate the confidence deviation value, and the deviation value exceeding the preset threshold triggers the error correction mechanism. For example, when the image resolution is 200 pixels / meter, the center coordinates of the detection frame and the reference coordinates differ by 50 pixels, and the confidence deviation value is calculated as 0.25 meters. This quantitative index provides real-time feedback signals for the update of the dynamic weight coefficient matrix, ensuring the linear correlation between model parameter adjustment and real space error.

[0073] As a preferred embodiment, the scheme of the application is implemented as follows:

[0074] According to the detection result, the detection frame of the road marking of the current frame image is extracted, the center coordinates of the detection frame are output, and the corresponding high-precision map reference coordinates are obtained. The calculation formula of the confidence deviation value is:

[0075] The confidence deviation value = |detection frame center coordinates-high-precision map reference coordinates| / image resolution

[0076] Specifically, first, the detection frame information of the road marking is extracted from the output result of the target detection network. For each detected marking, the center coordinates (x, y) of its detection frame are calculated. At the same time, the standard road marking coordinates (x', y') corresponding to the current vehicle position are queried from the high-precision map database. Then the Euclidean distance between the center coordinates of the detection frame and the map reference coordinates is calculated. Finally, the distance is divided by the resolution of the image (such as 1920x1080) to obtain the normalized confidence deviation value.

[0077] Through the above technical solution, this application can effectively quantify the degree of deviation between road marking detection results and high-precision map benchmark data. This allows for an objective assessment of the performance of the current detection algorithm in complex environments, providing a basis for subsequent dynamic parameter adjustments. Furthermore, by introducing image resolution for normalization, this metric becomes more universal and applicable to image inputs of varying resolutions.

[0078] In some of the above-mentioned schemes in this application, a method is proposed to spatially align the road marking position coordinates output by the target detection network with the high-precision map data and calculate the actual offset. However, due to the dynamic changes in the pitch angle during vehicle driving, the camera viewing angle fluctuates, resulting in a dynamic deviation between the on-board GPS coordinates and the high-precision map coordinate system. At the same time, directly calculating the coordinate difference cannot accurately reflect the spatial distribution difference between the marking and the map reference line, resulting in distortion of the actual offset calculation result.

[0079] like Figure 3 The figure shows the calculation process of the actual offset. The present application further proposes that the calculation process of the actual offset is:

[0080] Map the vehicle-mounted GPS coordinates corresponding to the current frame image to the local coordinate system of the high-precision map;

[0081] The attitude angle data obtained by the inertial measurement unit is fused through the Kalman filter to compensate for the impact of the vehicle's pitch angle on the camera's viewing angle;

[0082] Calculate the Hausdorff distance between the road marking position coordinates and the map reference line as the actual offset;

[0083] When the actual offset exceeds the preset safety threshold, the dynamic weight coefficient matrix corresponding to the current frame image in the echo state network is frozen.

[0084] When the vehicle's GPS coordinates are mapped to the local coordinate system of the high-precision map, a bilinear interpolation algorithm is used to perform sub-pixel corrections on the GPS coordinates to ensure that the coordinate conversion error is less than 0.1 meters. During the Kalman filter's fusion of attitude angle data, the inertial measurement unit outputs pitch angle data at a frequency of 100Hz. The state equation is used to predict the pitch angle change at the next moment, and the predicted value is then weightedly fused with the actual measured value through the observation equation. The compensation period is 10 milliseconds. The Hausdorff distance is calculated using a point-by-point scanning method, traversing all pairs of road marking position coordinates and map reference lines. The maximum and minimum distances are taken as the actual offset, and the preset safety threshold is set to 15 centimeters.

[0085] Specifically, in the vehicle-mounted GPS coordinate mapping stage, first, the local coordinate system range is determined in the high-precision map according to the real-time position of the vehicle, and the GPS longitude and latitude coordinates are converted into the plane rectangular coordinates of the local coordinate system by using the coordinate conversion matrix, wherein the parameters of the coordinate conversion matrix are dynamically adjusted according to the map projection type. Further, when the Kalman filter is used to fuse the pitch angle data, the process noise covariance is set to 0.01 and the observation noise covariance is set to 0.05 in the filter initialization stage, and the latest data of the inertial measurement unit is synchronously acquired during the processing of each frame of image, so as to ensure that the pitch angle compensation delay is lower than 1 millisecond. When the Hausdorff distance is calculated, the minimum distance set of all points in A to B and the minimum distance set of all points in B to A are calculated respectively, and the maximum value in the two sets is taken as the actual offset, and the calculation complexity of this method is O(n2), which is suitable for the line detection scene with the number of points not exceeding 500. When the actual offset exceeds 15 centimeters, the freezing operation of the dynamic weight coefficient matrix is realized by using the flag register, and the register state is maintained to zero before the next frame of data is input, so as to prevent the error from being transmitted to the subsequent processing flow.

[0086] As a preferred embodiment, the scheme of the application is implemented as follows:

[0087] The vehicle-mounted GPS coordinates corresponding to the current frame of image are mapped to the local coordinate system of the high-precision map. Specifically, the longitude and latitude coordinates of the current vehicle position are acquired by using the vehicle-mounted GPS receiver, and then the coordinates are converted into the local coordinate system used by the high-precision map by using the coordinate conversion algorithm. For example, the UTM projection method is used to convert the longitude and latitude coordinates into the plane rectangular coordinate system.

[0088] Further, the attitude angle data acquired by the inertial measurement unit is fused by using the Kalman filter to compensate the influence of the vehicle pitch angle on the camera view angle. In the implementation, the pitch angle, roll angle and yaw angle data of the vehicle are acquired from the vehicle-mounted inertial measurement unit, and the attitude angle data and the GPS position data are input into the designed Kalman filter for data fusion to obtain more accurate vehicle attitude estimation. Then, according to the vehicle attitude and the installation position of the camera, the inclination angle of the actual imaging plane of the camera relative to the ground is calculated, which is used for subsequent coordinate transformation.

[0089] Therefore, the Hausdorff distance between the road marking position coordinates and the map reference line is calculated as the actual offset. Specifically, first, the detected road marking position coordinates are converted from the image coordinate system to the vehicle coordinate system, and then converted to the world coordinate system according to the vehicle attitude. Then, the road reference line coordinates at the corresponding position in the high-precision map are extracted. Finally, the Hausdorff distance between the converted road marking coordinate point set and the map reference line coordinate point set is calculated, and the distance is the actual offset.

[0090] When the actual offset exceeds the preset safety threshold, the dynamic weight coefficient matrix corresponding to the current frame image in the frozen echo state network is frozen. For example, the safety threshold is set to 0.5 meters, and when the calculated actual offset is greater than 0.5 meters, the dynamic weight coefficient matrix corresponding to the current frame is kept unchanged and no update operation is performed. This freezing mechanism can prevent the network parameters from fluctuating sharply due to sudden large deviations.

[0091] Through the above technical solutions, the application can effectively compensate for the influence of vehicle attitude changes on camera imaging, and improve the matching degree of road marking detection results and actual road geometry. At the same time, by introducing the actual offset calculation and safety threshold judgment mechanism, significant deviations between the detection results and the high-precision map can be found in time, and appropriate parameter freezing measures can be taken to avoid sharp fluctuations in network weights. This closed-loop feedback mechanism based on multi-source data fusion can significantly improve the robustness and reliability of the system in complex road conditions.

[0092] In some of the above schemes of the application, during the process of dynamically adjusting the kernel weight parameters, the target detection network is insufficient in shallow feature extraction or resolution compression is too early, which causes the subsequent levels to be unable to effectively capture the detailed features of the road markings, affecting the stability of the detection results.

[0093] The application further proposes a construction process of the target detection network, which includes:

[0094] The preprocessed original image data is input into the multi-level convolution module from shallow to deep, and each level of convolution module performs the following operations:

[0095] Feature extraction is performed on the input data by a convolution kernel of a preset size;

[0096] Nonlinear activation processing is performed on the extracted feature map;

[0097] A max-pooling operation is inserted at a preset level to reduce the resolution of the feature map.

[0098] Among them, the convolution kernel of the preset size adopts a size of 3x3 or 5x5 for step-by-step extraction of image edge and texture features; the nonlinear activation processing is realized by a ReLU function to suppress the influence of negative value features on the subsequent levels; the preset level of the max-pooling operation is set at the output end of each convolution module, and the pooling window size is 2x2 and the step is 2, which reduces the resolution of the feature map to 1 / 4 of the original image.

[0099] Specifically, the original image data is input into the first-level convolution module after preprocessing, and low-level features are extracted through a 3x3 convolution kernel to generate a primary feature map containing the contour of the road marking. After ReLU activation, a 2x2 max-pooling operation is performed to reduce the resolution, retain key features, and reduce the amount of calculation. In the middle-level convolution module, a 5x5 convolution kernel is used to capture sub-pixel level changes of the marking edge, and a nonlinear activation is combined to enhance the feature expression ability. The pooling operation is only performed in the 3rd and 5th modules to avoid excessive compression of the resolution, which can cause the loss of slender marking features. The deep convolution module uses a 3x3 convolution kernel to extract global context information, and through inter-level feature transmission, it realizes the fusion of detailed and semantic features, providing stable feature input for dynamic weight adjustment. For example, when the input image resolution is 1280x720, after processing by the 5-level convolution module, the final feature map resolution is reduced to 40x22, while retaining the continuity and directional features of the marking.

[0100] As a preferred embodiment, the scheme of the present application is implemented as follows:

[0101] The construction process of the target detection network includes the following steps:

[0102] The preprocessed original image data is input into the multi-level convolution module from the shallow layer to the deep layer. Each level of convolution module performs the following operations:

[0103] Feature extraction is performed on the input data by a convolution kernel of a predetermined size. Specifically, a 3x3 convolution kernel is used to perform sliding convolution operation on the input image to extract local features.

[0104] Nonlinear activation processing is performed on the extracted feature map. For example, a ReLU activation function is used to perform nonlinear transformation on the feature map after convolution to enhance the expression ability of the network.

[0105] A max-pooling operation is inserted at a predetermined level to reduce the resolution of the feature map. Further, a 2x2 max-pooling layer is inserted after the 2nd, 4th, and 6th convolution layers to compress the feature map size and retain significant features.

[0106] Thus, a target detection network containing 8 convolution layers and 3 pooling layers is constructed. The shallow convolution module mainly extracts low-level features such as edges and textures, and the deep convolution module gradually extracts higher-level semantic features. Through multi-level feature extraction and downsampling operations, the network can effectively capture multi-scale target information in the image.

[0107] By the technical solution, the application constructs a hierarchical target detection network structure. The network gradually extracts image features through multiple convolution modules, and introduces a pooling operation at a key level to reduce the computational complexity. This structure design can improve the computational efficiency of the network while maintaining the detection accuracy, which is conducive to realizing real-time target detection in a vehicle-mounted environment. At the same time, multi-level feature extraction also enhances the adaptability of the network to different scale targets and improves the detection robustness in complex road scenes.

[0108] In some of the above schemes of the application, it is necessary to accurately obtain road surface humidity related data to improve the reliability of target detection, but the traditional single sensor measurement method has the problem of insufficient accuracy, and cannot effectively reflect the actual humidity change under complex road conditions.

[0109] The application further proposes:

[0110] The physical sensor includes a rain sensor and a wheel speed sensor, and the road surface humidity related data is obtained by the following method:

[0111] The pulse signal output by the rain sensor is analyzed, and the average raindrop impact frequency in a preset time window is calculated.

[0112] Based on the wheel slip rate output by the wheel speed sensor, the current road surface humidity percentage is calculated through a pre-established slip rate-humidity mapping table.

[0113] Among them, the rain sensor counts the number of raindrop impacts through the time interval of the pulse signal, and the preset time window can be set to 5 seconds. The average raindrop impact frequency is calculated by using the moving average method. The wheel speed sensor calculates the wheel slip rate through the wheel speed pulse signal, and the slip rate-humidity mapping table is established by experimentally calibrating the slip rate threshold under different humidity conditions. For example, when the slip rate reaches 0.15, the corresponding humidity percentage is 65%, and the slip rate 0.25 corresponds to the humidity percentage 80%.

[0114] Specifically, the square wave pulse signal output by the rain sensor is captured by the timer input capture unit of the microcontroller, and the rising edge timestamp of the pulse is recorded by edge triggering. The time interval of adjacent pulses is calculated, the arithmetic mean of all pulse intervals within the time window is calculated, and the reciprocal of the average interval is converted into the raindrop impact frequency. The digital signal output by the wheel speed sensor is converted into wheel speed values, and the slip ratio is calculated by comparing the speed difference of the wheels on the same axis. The specific calculation formula is slip ratio=(drive wheel speed-non-drive wheel speed) / drive wheel speed. The pre-established slip ratio-humidity mapping table is stored as a two-dimensional lookup table structure, containing a linear mapping relationship between the slip ratio interval of 0.05 to 0.35 and the corresponding humidity percentage. When the measured slip ratio is 0.18, the humidity percentage value of 68% is obtained by table lookup interpolation calculation. This dual-sensor data fusion method is complementary in physical characteristics, and in heavy rain weather, when the rain sensor is shielded by water droplets, the road surface humidity can still be accurately calculated by the change of wheel speed.

[0115] As a preferred embodiment, the scheme of the application is implemented as follows:

[0116] The physical sensors include a rain sensor and a wheel speed sensor, and the road surface humidity related data is obtained by the following method:

[0117] The pulse signal output by the rain sensor is analyzed, and the average raindrop impact frequency within a preset time window is calculated. Specifically, the rain sensor uses a piezoelectric sensor that generates a voltage pulse when a raindrop hits the sensor surface. A 1-minute time window is set, the number of pulses within the time window is counted, and the average raindrop impact frequency is obtained by dividing by 60 seconds.

[0118] Based on the wheel slip ratio output by the wheel speed sensor, the current road surface humidity percentage is calculated by a pre-established slip ratio-humidity mapping table. Further, the wheel speed sensor collects the speed data of the four wheels, calculates the difference between the theoretical driving distance and the actual driving distance, and obtains the wheel slip ratio. A pre-established slip ratio and road surface humidity correspondence table is established, for example, slip ratio 0%~5% corresponds to humidity 0%~20%, 5%~10% corresponds to 20%~40%, and so on. The current road surface humidity percentage is obtained by table lookup.

[0119] Through the above technical scheme, the application realizes the fusion of multi-sensor data, improves the accuracy of road surface humidity estimation. Thus, the error of a single sensor is avoided, and the adaptability of the system to complex road conditions is enhanced. Further, by pre-setting the time window and the mapping table, the calculation complexity is reduced, which is conducive to real-time processing.

[0120] In some of the above schemes of the present application, when extracting the visual feature data of the reflection intensity distribution matrix, the traditional method usually adopts global image brightness statistics or fixed threshold segmentation, which is difficult to adapt to the dynamic changes of road reflection under different lighting conditions. Especially in rainy and foggy weather, the water film reflection presents the characteristics of local high brightness and non-uniform distribution, which leads to insufficient recognition accuracy of the reflection area.

[0121] As shown in the flow chart of the visual feature data extraction process, the present application further proposes a method for extracting the visual feature data including the reflection intensity distribution matrix, which comprises the following steps: Figure 4

[0122] Converting the original image data into a gray-scale image and dividing the area according to the preset grid size;

[0123] Calculating the variance value of the pixel brightness in each grid area to generate the reflection intensity distribution matrix;

[0124] Marking the grid with variance value exceeding the preset variance threshold as the high reflection area.

[0125] Wherein, the gray-scale conversion adopts the weighted average method to retain the brightness contribution of the red, green and blue channels, and the preset grid size is set according to the proportion of the camera field of view angle and the scale of the mark line, usually set to 32x32 pixels to 64x64 pixels. The variance calculation adopts the sliding window algorithm, and the window step is equal to the grid size, avoiding the omission of edge information. The preset variance threshold is obtained through historical data statistics, and different threshold intervals are set for sunny and rainy days, and the threshold for rainy days is increased by 20% to 30% compared with that for sunny days.

[0126] Specifically, the original RGB image forms a single-channel brightness matrix after gray-scale conversion, and is divided into several sub-areas according to the preset grid. The pixel brightness variance of each sub-area is calculated independently, and the variance value reflects the dispersion degree of the reflection intensity in the area. The high variance area represents the sudden brightness caused by the mirror reflection or water film refraction, for example, when the variance value in the grid exceeds the preset threshold, it is determined as a potential reflection interference area. Through spatial grid division, local reflection feature quantization is realized, avoiding the defect of global statistics ignoring local high reflection points. The generated matrix is input to the space-time feature fusion module, and together with the humidity data, it affects the generation of dynamic weight coefficient. This method effectively reduces the false detection rate by 35.7% in rainy weather experiments, and when the preset grid size is optimized to 48x48 pixels, the calculation time and detection accuracy reach the optimal balance.

[0127] As a preferred embodiment, the scheme of the present application is implemented as follows:

[0128] ​The original image data is first converted into a grayscale image. The grayscale image is regionally divided according to a grid size of 10x10 pixels. The variance value of the brightness of the pixels in each grid region is calculated to generate a specular intensity distribution matrix. The variance calculation uses the formula for the square of the standard deviation: variance = (Σ(X i -μ) 2 ) / N, where Xi is the brightness value of a single pixel, μ is the average brightness of the pixels in the grid region, and N is the total number of pixels in the grid.

[0129] Further, the variance threshold is set to 100. Each element in the specular intensity distribution matrix is compared with the threshold, and if the variance value exceeds 100, the corresponding grid is marked as a high-specular region. The high-specular region is represented by 1 in the matrix, and other regions are represented by 0. Finally, a binary high-specular region distribution matrix is obtained.

[0130] Thus, by grid processing and variance calculation, the specular features in the original image are extracted into a matrix representation of numerical values. This representation method not only retains the spatial distribution information of the specular intensity, but also discretizes the continuous brightness values into binary data for subsequent processing.

[0131] Through the above technical solutions, the present application can effectively extract the specular intensity distribution features in the image. Grid processing reduces the computational complexity, and variance calculation can accurately capture the brightness changes in local regions. Binary labeling further simplifies the data structure and facilitates subsequent processing. This feature extraction method provides reliable input for subsequent spatiotemporal feature fusion and dynamic weight adjustment, which helps to improve the accuracy and robustness of road marking detection under complex lighting conditions such as strong specular reflection.

[0132] In some of the above schemes of the present application, the injection method of the dynamic weight coefficient matrix may cause the convolution kernel weight to mutate, causing fluctuations in the feature extraction capability of the target detection network, and easily causing gradient instability in the scene where the road surface humidity changes rapidly, affecting the continuity of road marking detection.

[0133] The present application further proposes that the adjusted kernel weight is realized by element-wise superposition of the original kernel weight and the dynamic weight coefficient matrix, and the superposition formula is adjusted kernel weight = original kernel weight x (1 + dynamic weight coefficient).

[0134] Each element of the dynamic weight coefficient matrix corresponds to an independent channel weight adjustment amount of the convolution kernel. The basic feature extraction capability of the original kernel weight is retained through multiplicative superposition, avoiding gradient explosion or disappearance caused by weight value domain mutation. The absolute value of the dynamic weight coefficient is constrained within a preset interval during the superposition process, for example, the coefficient value domain is set to [-0.1, 0.3], preventing individual weight adjustment from exceeding the physical meaning range of the kernel parameter. This superposition method cooperates with the batch normalization layer of the target detection network to maintain the numerical stability of the feature map after weight adjustment through the normalization layer.

[0135] Specifically, when the road surface reflection intensity distribution changes abruptly in rainy weather, the dynamic weight coefficient matrix carries humidity change trend information, and the kernel weight of the pre-defined convolution layer is adjusted in fine granularity through the superposition formula. For example, in the third convolution module, the dynamic weight coefficient acts on the depth separable channel of the 3x3 convolution kernel, causing the kernel parameter to produce a ±10% amplitude adjustment based on the original value. This adjustment process retains the original edge detection characteristics of the convolution kernel while enhancing the response sensitivity to high reflection areas. Through the design of the (1+dynamic weight coefficient) structure in the superposition coefficient, the weight adjustment direction is ensured to be consistent with the sign of the original kernel parameter, preventing the positive and negative weights from flipping and causing feature extraction logic confusion. Experimental data show that, in a test scenario with a road surface humidity of 65%, the false detection rate of the target detection network is reduced by 12.3% compared to direct replacement weight adjustment.

[0136] As a preferred embodiment, the scheme of the present application is implemented as follows:

[0137] The kernel weight parameter adjustment process of the target detection network is as follows:

[0138] The dynamic weight coefficient matrix is element-wise superimposed with the original kernel weight of the pre-defined convolution layer in the target detection network, and the superposition formula is:

[0139] The adjusted kernel weight = original kernel weight x (1+dynamic weight coefficient).

[0140] Specifically, the dimension of the dynamic weight coefficient matrix is consistent with the dimension of the kernel weight of the pre-defined convolution layer. For example, for a 3x3 convolution kernel, the dynamic weight coefficient matrix is also 3x3 in size. When performing element-wise superposition, each element in the dynamic weight coefficient matrix is multiplied by the original kernel weight at the corresponding position.

[0141] As a result, the original kernel weight is dynamically adjusted, achieving adaptive optimization of the convolution layer parameters. The element value in the dynamic weight coefficient matrix is usually within the range of [-1, 1], and when the coefficient is positive, the original weight is enhanced, and when the coefficient is negative, the original weight is weakened. This adjustment mechanism enables the network to dynamically adjust the feature extraction capability according to the current environmental conditions.

[0142] By the technical solution, the dynamic adjustment of the kernel weight of the target detection network is realized. The adjustment mechanism enables the network to adaptively optimize the feature extraction capability according to the current environmental conditions, and improves the accuracy and robustness of target detection. Especially in complex and variable road environments such as rainy days or strong light conditions, the method can effectively suppress the influence of environmental interference on the detection results and ensure the stability of the detection performance.

[0143] In some schemes of the application, when the road surface humidity related data and the visual feature data are input into the echo state network for spatio-temporal feature fusion, due to the lack of modeling method for the dynamic coupling relationship between the humidity change trend and the spatial distribution of high light reflection area, the generated dynamic weight coefficient matrix cannot effectively reflect the influence of the rainfall-light coupling effect on visual detection, thereby affecting the parameter adjustment accuracy of the target detection network.

[0144] As shown in the flowchart of spatio-temporal feature fusion, the application further proposes specific steps for spatio-temporal feature fusion by the echo state network: Figure 5

[0145] The average raindrop impact frequency is input into the reservoir pool of the echo state network in time series to predict the humidity change trend in the future preset time length;

[0146] The humidity change trend and the spatial coordinates of the high light reflection area are modeled by two-dimensional Gaussian distribution to generate a spatio-temporal coupling feature vector;

[0147] The spatio-temporal coupling feature vector is nonlinearly transformed by the leaky integral mechanism of the reservoir pool to output a dynamic weight coefficient matrix.

[0148] The reservoir pool receives the time series data of the average raindrop impact frequency, extracts the periodic characteristics of the rainfall change using its internal fixed random connection structure, and predicts the humidity trend curve in the next 5 seconds. The spatial coordinates of the high light reflection area are determined by the center points of the grid regions in the light intensity distribution matrix whose variance values exceed 0.25. The two-dimensional Gaussian distribution modeling takes the time axis of the humidity trend curve as the first dimension and the spatial coordinates of the high light reflection area as the second dimension to generate a joint probability distribution model with a mean value of the humidity trend prediction value and a variance of a preset empirical value of 0.3. The leaky integral mechanism adopts a benchmark leakage rate of 0.6 to nonlinearly transform the spatio-temporal coupling feature vector, and outputs a dynamic weight coefficient matrix with a dimension matching the number of convolution layer kernels of the target detection network.

[0149] ​Specifically, the time series data captures the lag effect of rainfall variation through the recurrent connection structure of the reservoir, and predicts the humidity trend curve covering the future time window. The spatial coordinates of the high-reflective region and the humidity trend curve establish a joint probability model through two-dimensional Gaussian distribution, coupling the humidity change in the time dimension with the reflective intensity distribution in the spatial dimension. When the leaky integral mechanism performs nonlinear transformation on the spatiotemporal coupling feature vector, it adjusts the decay rate of the activation state of the reservoir neurons to suppress noise interference and retain effective features. In the generation process of the dynamic weight coefficient matrix, the time series prediction result and the spatial distribution data are aligned through the Gaussian kernel function, so that the position deviation of the humidity change and the high-reflective region is quantitatively expressed in the weight matrix. The final output dynamic weight coefficient matrix acts on the third convolutional layer of the target detection network, and adjusts the weight distribution of the convolution kernel to enhance the network's ability to suppress high-reflective interference regions in rain and fog environments.

[0150] As a preferred embodiment, the scheme of the application is implemented as follows:

[0151] After analyzing the pulse signal collected by the rain sensor, the average raindrop impact frequency in the last 30 seconds is obtained as 12 times per second. This data is input into the reservoir of the echo state network in the form of a time series, and the reservoir contains 200 dynamic nodes whose internal connection weights are randomly initialized and remain fixed. The reservoir predicts the humidity change trend in the next 5 seconds through time expansion operation and outputs the humidity change trend sequence. Further, the spatial coordinates of the high-reflective region marked by the visual feature extraction module are normalized to a two-dimensional coordinate set in the image coordinate system, and are jointly modeled with the humidity change trend sequence. Specifically, a two-dimensional Gaussian distribution model is constructed with the center coordinates of each high-reflective region as the mean value and the humidity change rate as the variance parameter, generating a spatiotemporal coupling feature vector. The spatiotemporal coupling feature vector is processed through the leaky integral mechanism of the reservoir, with a leak rate of 0.8 and an integration time constant of 0.2 seconds. After processing by the nonlinear activation function, a dynamic weight coefficient matrix of 64x64 dimensions is output.

[0152] Through the above technical scheme, the application effectively solves the cross-modal spatiotemporal coupling problem of visual features and physical sensor data. By establishing a spatial correlation model of humidity change trend and high-reflective region, the parameter weights of the target detection network are dynamically adjusted, so that the network can adaptively suppress specular reflection interference in rain and fog environments and improve the robustness of road marking detection. At the same time, the nonlinear transformation of the leaky integral mechanism enhances the expression ability of the time series features, so that the dynamic weight coefficient matrix can accurately reflect the modulation needs of environmental changes on visual features.

[0153] In some of the above schemes of the present application, the generation of the dynamic weight coefficient matrix depends on the spatio-temporal feature fusion mechanism of the echo state network. However, in the process of continuous multi-frame detection, when environmental interference (such as instantaneous strong reflection or raindrop shielding) causes the confidence deviation value to fluctuate for a short time and at a high frequency, the dynamic weight coefficient matrix may produce unexpected parameter oscillation due to the sudden change of the reservoir state, thereby causing the detection result to jump continuously.

[0154] As shown in FIG. 7, it is a dynamic weight coefficient matrix failure processing flowchart; the present application further proposes: Figure 6

[0155] When it is detected that the confidence deviation value of the same markline fluctuates more than the preset fluctuation range within N consecutive frames, it is determined that the current dynamic weight coefficient matrix is invalid;

[0156] Activate the historical weight rollback mechanism: call the mean value of the effective dynamic weight coefficient matrix of the previous M frames to replace the current dynamic weight coefficient matrix;

[0157] Synchronously trigger the adaptive adjustment of the reservoir leakage rate of the echo state network, so that the leakage rate parameter satisfies:

[0158] Leakage rate parameter = preset reference leakage rate x (1 - current confidence deviation value / maximum allowable confidence deviation value), and the maximum allowable confidence deviation value is calculated according to the camera installation height.

[0159] Among them, the historical weight rollback mechanism selects the weight matrix of the previous M frames through a sliding window for arithmetic mean, for example, when M = 3, the mean value matrix of the previous three effective weights is taken to cover the current invalid weight. The adaptive adjustment of the leakage rate dynamically adjusts the reservoir state update rate according to the ratio of the real-time confidence deviation value to the maximum allowable value, wherein the maximum allowable value is inversely proportional to the camera installation height, for example, when the height is 2 meters, the maximum allowable deviation is 5% of the image width. The mean value replacement operation maintains the weight dimension consistency through matrix element level mean value calculation, avoiding network structure deformation.

[0160] Specifically, when the confidence deviation value of the same markline is detected to fluctuate within ±10 pixels for five consecutive frames, the system identifies it as a weight matrix failure event. By calling the mean value of the weight matrix of the previous three frames without exceeding the limit, it quickly recovers to a stable detection state. At the same time, according to the ratio of the current deviation value 6 pixels to the maximum allowable value 10 pixels, the leakage rate is adjusted from the reference value 0.8 to 0.8 x (1-6 / 10) = 0.32, and the reservoir state update speed is reduced to suppress noise interference. The process realizes the circular buffer of the weight matrix through the hardware register, ensuring that the historical data retrieval delay is less than 2ms. After the mean weight injection, the network recovers to a steady state output in the next frame of detection, and the deviation value falls within the threshold.

[0161] ​As a preferred embodiment, the scheme of the application is implemented as follows: when the fluctuation range of the confidence deviation value of the same road marking in 5 consecutive frames of images exceeds ±0.15, it is determined that the current dynamic weight coefficient matrix is invalid. At this time, the history weight rollback mechanism is activated, the previous 3 frames of verified valid dynamic weight coefficient matrices are called, their arithmetic mean is calculated and replaces the current invalid matrix. The adaptive adjustment of the leak rate of the echo state network reservoir is triggered synchronously, wherein the preset reference leak rate is set to 0.8, and the maximum allowable confidence deviation value is calculated to be 0.3 by the camera installation height of 1.2 meters. The leak rate parameter is dynamically adjusted to 0.8x(1-current deviation value / 0.3) according to the current actual confidence deviation value, and when the deviation value reaches 0.3, the leak rate decreases to 0, and when the deviation value is 0, the reference leak rate is maintained.

[0162] Through the above technical scheme, the application effectively solves the problem of detection result fluctuation caused by weight matrix mismatch under complex environmental interference in the traditional detection algorithm. By dynamically rolling back the history valid weight matrix, the continuous error accumulation caused by single frame detection anomaly is avoided, and the reservoir leak rate parameter is adaptively adjusted, which enhances the inhibition ability of the network to abnormal state, and ensures the spatio-temporal continuity of the road marking detection result. The mechanism significantly reduces the risk of weight matrix mutation caused by water film reflection in rain and fog weather, and improves the robustness and environmental adaptability of the detection system.

[0163] In some of the above schemes of the application, the environment understanding based on visual perception is easily disturbed in complex road scenes, leading to misjudgment and missed detection in target detection of traditional visual algorithms. The traditional convolutional neural network relies on a single visual feature, which is easily affected by water film reflection, mirror refraction and other optical interference in rain and fog weather, and the feature extraction is distorted. The existing echo state network application is mainly concentrated in single modal data processing, and the cross-modal spatio-temporal coupling problem of visual-physical sensors has not been solved.

[0164] As shown in Figure 7 The application further proposes a real-time visual processing system based on ESN-CV collaborative processing, implements a real-time visual processing method based on ESN-CV collaborative processing, and includes a multi-source data acquisition module, a visual feature extraction module, an ESN-CV collaborative processing module, a kernel weight parameter adjustment module, a dynamic target detection network module and a closed-loop feedback processing module.

[0165] The multi-source data acquisition module is configured to acquire original image data collected by a camera and road surface humidity related data output by a physical sensor, wherein the physical sensor includes a rain sensor and a wheel speed sensor.

[0166] The visual feature extraction module is configured to convert the original image data into a grayscale image and divide the image into regions according to a preset grid size, calculate the variance value of the brightness of the pixels in each grid region, generate a reflection intensity distribution matrix, and mark the grid with a variance value exceeding a preset variance threshold as a high reflection region.

[0167] The ESN-CV cooperative processing module is configured to input the average raindrop impact frequency into the reservoir pool of the echo state network in a time sequence, predict the humidity change trend in a future preset time period, model the humidity change trend and the spatial coordinates of the high reflection region in a two-dimensional Gaussian distribution, generate a space-time coupling feature vector, perform nonlinear transformation on the space-time coupling feature vector through the leaky integral mechanism of the reservoir pool, and output a dynamic weight coefficient matrix.

[0168] The kernel weight parameter adjustment module is configured to perform element-wise superposition of the dynamic weight coefficient matrix and the original kernel weight of the predefined convolution layer in the target detection network, and the superposition formula is:

[0169] The adjusted kernel weight = the original kernel weight x (1 + the dynamic weight coefficient).

[0170] The dynamic target detection network module is configured to perform road marking detection on the current frame image based on the adjusted target detection network, and output the detection result and the confidence bias value, wherein the calculation formula of the confidence bias value is the absolute difference between the center coordinates of the detection frame and the reference coordinates of the high-precision map divided by the image resolution.

[0171] The closed-loop feedback processing module is configured to compare the confidence bias value with a preset bias threshold, and when the confidence bias value exceeds the preset bias threshold:

[0172] extract the road marking position coordinates and perform spatial alignment with the high-precision map data, calculate the actual offset, generate an error correction vector consistent with the number of nodes of the reservoir pool, perform Hadamard product operation on the error correction vector and the state matrix of the reservoir pool, and recalculate the dynamic weight coefficient matrix of the next frame based on the updated state matrix.

[0173] Specifically, the multi-source data acquisition module synchronously acquires original images and road surface humidity data through a camera and physical sensors, wherein the slip rate data of the wheel speed sensor is used to calculate the road surface humidity percentage through a pre-established mapping table. The visual feature extraction module divides the image grid and calculates the brightness variance, and identifies the high-reflective area to eliminate optical interference. The ESN-CV collaborative processing module models the humidity trend and the high-reflective area coordinates in space and time, integrates the cross-modal features by using the leaky integrator mechanism of the reservoir, and generates a dynamic weight coefficient matrix. The kernel weight parameter adjustment module realizes dynamic injection of the convolution kernel parameters through element-by-element superposition, thereby enhancing the robustness of the network to reflective interference. After the dynamic target detection network module outputs the detection result, the closed-loop feedback processing module calculates the actual offset by spatial alignment of the confidence bias value and the high-precision map, and generates an error correction vector when the offset is out of limit, and updates the reservoir state matrix by Hadamard product, so as to adjust the weight coefficients of the next frame. This closed-loop mechanism automatically corrects the model parameters when the detection error accumulates, thereby avoiding the problem of continuous error amplification caused by environmental interference in traditional methods.

[0174] As a preferred embodiment, the scheme of the application is implemented as follows: the system includes a multi-source data acquisition module, which carries a vehicle-grade global shutter camera with a model of OV2740, acquires original image data with a resolution of 1280x720 at a rate of 30fps, and simultaneously reads the pulse signals output by the rain sensor and the slip rate signals of the wheel speed sensor in real time through the LIN bus interface. The visual feature extraction module is based on the OpenCV library, converts the original image to grayscale format, divides the area into 16x16 pixel grids, calculates the brightness variance of each grid, and generates a reflective intensity distribution matrix. The ESN-CV collaborative processing module uses an echo state network with a reservoir size of 200 nodes to perform spatiotemporal coupling of the raindrop impact frequency time series and the reflective intensity matrix, and outputs a dynamic weight coefficient matrix through a leaky integrator. The kernel weight parameter adjustment module element-by-element superimposes the dynamic weight coefficients and the original weights of the 12th layer convolution kernel of the YOLOv5 network to generate adjusted convolution kernel parameters. The dynamic target detection network module deploys the optimized detection model based on the TensorRT engine, and outputs the lane detection box coordinates and confidence bias value. When the detection bias is out of limit, the closed-loop feedback processing module calls the high-precision map API for coordinate spatial alignment, calculates the Hausdorff distance to generate an error correction vector, and updates the reservoir state matrix by Hadamard product.

[0175] By the technical scheme, the application realizes cross-modal space-time coupling of visual features and physical sensing data, effectively suppresses the interference of mirror reflection on image features in a rainy and high-humidity scene, and enhances the environmental adaptability of the detection model by dynamically adjusting the convolution kernel parameters. The closed-loop feedback mechanism corrects the network weight generation process in real time, reduces the cumulative risk of the lane marking positioning error, and improves the stability and reliability of the road lane marking detection in a complex optical environment.

[0176] The technical scope of the present application is not limited to the above description, and those skilled in the art can make various modifications and changes to the above embodiments without departing from the technical idea of the present application, and these modifications and changes should be within the protection scope of the present application.

Claims

1. A real-time visual processing method based on ESN-CV collaborative processing, characterized by: The steps include: Obtain the original image data collected by the camera and the road humidity related data output by the physical sensor; Processing the original image data to extract visual feature data including a reflection intensity distribution matrix; Inputting the road surface moisture related data and the visual feature data into an echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix; Injecting the dynamic weight coefficient matrix into a predefined convolutional layer of the target detection network to adjust its kernel weight parameters; Performing road marking detection on the current frame image based on the adjusted target detection network, and outputting the detection result and confidence deviation value; Compare the confidence deviation value with a preset deviation threshold, and when the confidence deviation value exceeds the preset deviation threshold, perform the following operations: Extracting the road marking position coordinates output by the target detection network, spatially aligning them with the high-precision map data, and calculating the actual offset; generating an error correction vector consistent with the number of nodes in the reserve pool of the echo state network according to the actual offset; A Hadamard product operation is performed on the error correction vector and the state matrix of the reserve pool, and the dynamic weight coefficient matrix of the next frame is recalculated based on the updated state matrix.

2. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The generation process of the confidence deviation value is: Extract the detection frame of the road markings in the current frame image based on the detection results, output the center coordinates of the detection frame, and obtain the corresponding high-precision map reference coordinates; The calculation formula of the confidence deviation value is: Confidence deviation value = |detection box center coordinates - high-precision map reference coordinates| / image resolution.

3. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The calculation process of the actual offset is: Mapping the vehicle-mounted GPS coordinates corresponding to the current frame image to the local coordinate system of the high-precision map; The attitude angle data obtained by the inertial measurement unit is fused through the Kalman filter; Calculating the Hausdorff distance between the position coordinates of the road marking and the map reference line as the actual offset; When the actual offset exceeds a preset safety threshold, the dynamic weight coefficient matrix corresponding to the current frame image in the echo state network is frozen.

4. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The construction process of the target detection network includes: The preprocessed original image data is input into the multi-level convolution module from shallow to deep layers. Each level of convolution module performs the following operations: Extract features from input data using convolution kernels of preset sizes; Perform nonlinear activation processing on the extracted feature map; Insert max pooling operations at preset levels to reduce feature map resolution.

5. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1 is characterized in that: The physical sensors include a rain sensor and a wheel speed sensor, and the road surface humidity related data is obtained in the following manner: Analyzing the pulse signal output by the rain sensor and calculating the average raindrop impact frequency within a preset time window; Based on the wheel slip ratio output by the wheel speed sensor, the current road surface humidity percentage is calculated using a pre-established slip ratio-humidity mapping table.

6. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The extraction of visual feature data including the reflection intensity distribution matrix includes: Converting the original image data into a grayscale image and dividing the image into regions according to a preset grid size; Calculating the variance of pixel brightness within each grid area to generate the reflection intensity distribution matrix; The grids whose variance values ​​exceed the preset variance threshold are marked as high reflective areas.

7. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The adjustment process of the kernel weight parameters of the target detection network is: The dynamic weight coefficient matrix is ​​element-by-element superimposed with the original kernel weights of the predefined convolutional layer in the target detection network. The superposition formula is: Adjusted kernel weight = original kernel weight × (1 + dynamic weight coefficient).

8. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: The specific steps of the echo state network for spatiotemporal feature fusion are as follows: The average raindrop impact frequency is input into the storage pool of the echo state network in time series to predict the humidity change trend within a preset time period in the future; Performing a two-dimensional Gaussian distribution model on the humidity change trend and the spatial coordinates of the high-reflective area to generate a spatiotemporal coupling feature vector; The space-time coupling eigenvector is nonlinearly transformed through the leaky integration mechanism of the reservoir to output a dynamic weight coefficient matrix.

9. The real-time visual processing method based on ESN-CV collaborative processing according to claim 1, characterized in that: Also includes: When it is detected that the confidence deviation value fluctuation of the same marking line in N consecutive frames exceeds the preset fluctuation range, it is determined that the current dynamic weight coefficient matrix is ​​invalid; Activate the historical weight rollback mechanism: call the mean of the effective dynamic weight coefficient matrix of the previous M frames to replace the current dynamic weight coefficient matrix; Synchronously trigger the adaptive adjustment of the leakage rate of the reservoir of the echo state network so that the leakage rate parameter satisfies: Leak rate parameter = preset baseline leak rate × (1-current confidence deviation value / maximum allowable confidence deviation value); The maximum allowable confidence deviation value is calculated based on the camera installation height.

10. Real-time visual processing system based on ESN-CV collaborative processing, characterized by: The method for real-time visual processing based on ESN-CV collaborative processing according to any one of claims 1 to 9 is implemented, comprising: Multi-source data acquisition module, used to obtain raw image data collected by the camera and road humidity related data output by the physical sensor; A visual feature extraction module, configured to process the original image data and extract visual feature data including a reflection intensity distribution matrix; An ESN-CV collaborative processing module is used to input the road surface moisture related data and the visual feature data into an echo state network for spatiotemporal feature fusion to generate a dynamic weight coefficient matrix; A kernel weight parameter adjustment module, configured to inject the dynamic weight coefficient matrix into a predefined convolutional layer of a target detection network to adjust its kernel weight parameters; A dynamic target detection network module is used to detect road markings on the current frame image based on the adjusted target detection network, and output the detection result and confidence deviation value; The closed-loop feedback processing module is configured to compare the confidence deviation value with a preset deviation threshold, and when the confidence deviation value exceeds the preset deviation threshold, perform the following operations: Extracting the road marking position coordinates output by the target detection network, spatially aligning them with the high-precision map data, and calculating the actual offset; generating an error correction vector consistent with the number of nodes in the reserve pool of the echo state network according to the actual offset; A Hadamard product operation is performed on the error correction vector and the state matrix of the reserve pool, and the dynamic weight coefficient matrix of the next frame is recalculated based on the updated state matrix.

Images