System and method for precise video capture and analysis of hard shoulder based on multi-modal fusion

Abstract

This application relates to the field of intelligent traffic monitoring technology, and discloses a system and method for accurate capture and analysis of hard shoulder video based on multimodal fusion. The system includes a radar sensing unit, a video acquisition unit, an edge computing unit, an evidence generation unit, and a communication management unit. The edge computing unit performs spatiotemporal registration of radar and video data, establishes target association to generate a fused trajectory, and dynamically determines illegal occupancy based on lane status commands. After triggering, it calculates zoom and pan-tilt control parameters to drive the camera for accurate capture. The evidence generation unit spatiotemporally binds image sequences with radar data, time, and location information to generate a structured evidence chain. The communication management unit uploads the evidence chain and pushes warning information containing the identifier and geographical location of the violating vehicle. The method corresponds to this system. This application can achieve high-precision identification, reliable evidence generation, and closed-loop processing of hard shoulder illegal occupancy under the dual constraints of adaptability to complex environments and dynamic management coordination.

Description

Technical Field

[0001] This application relates to the field of intelligent traffic monitoring technology, specifically a system and method for precise capture and analysis of hard shoulder video based on multimodal fusion. Background Technology

[0002] In the practice of intelligent traffic management on highways, the hard shoulder serves as a crucial passage for emergency rescue and traffic management during special periods. Accurate identification and evidence consolidation of illegal occupation of the hard shoulder are vital for ensuring traffic safety and law enforcement credibility. Current mainstream monitoring solutions mostly employ a single video analysis architecture, relying on fixed cameras to collect images and determining violations based on preset rules (such as fixed dwell time thresholds). Under complex weather and lighting conditions such as rain, fog, low light at night, and strong glare, the video perception quality of this solution deteriorates significantly, leading to decreased confidence in vehicle target detection and interruption of trajectory tracking continuity. Some systems that attempt to introduce millimeter-wave radar as a supplementary sensing source lack a precise spatiotemporal registration mechanism between the radar coordinate system and the image coordinate system, as well as target association logic. Multi-source data is simply superimposed or independently judged, failing to form a stable and reliable fused trajectory. Instead, data heterogeneity leads to erroneous associations or trajectory breaks. Meanwhile, the existing system's judgment logic is disconnected from the traffic management platform, and it cannot adapt the judgment parameters in real time according to the actual working mode of the hard shoulder (such as dynamic instructions such as temporary opening of traffic during holidays and emergency closure due to accidents). This results in a large number of misjudgments or omissions during the switching of control strategies, which weakens the effectiveness of law enforcement evidence and affects the coordination efficiency of traffic management. Summary of the Invention

[0003] Therefore, it is necessary to provide a system and method for accurate capture and analysis of hard shoulder video based on multimodal fusion to address the above-mentioned technical problems. This system and method can achieve high-precision identification, credible evidence generation, and closed-loop handling of illegal occupation of hard shoulders under the dual constraints of adaptability to complex environments and dynamic management synergy.

[0004] On the one hand, this application provides a system for precise capture and analysis of hard shoulder video based on multimodal fusion, including: The radar sensing unit is configured to output radar sensing data of moving targets within the hard shoulder area in real time. The radar sensing data includes target location information and motion status information. The video acquisition unit includes a camera with optical zoom and gimbal tracking capabilities, configured to acquire video streams of the hard shoulder area and perform target tracking and capture in response to control commands; An edge computing unit is communicatively connected to both the radar sensing unit and the video acquisition unit. It is configured to perform spatiotemporal registration of the radar sensing data and the video stream, establish a correlation between radar-detected targets and video-detected targets, and generate a fused target trajectory. Based on the fused target trajectory, it determines whether a target illegally occupies a hard shoulder; if the determination is yes, it generates a capture trigger signal. Furthermore, it calculates camera zoom parameters and gimbal control parameters based on the capture trigger signal, generates control commands, and sends them to the video acquisition unit. The evidence generation unit is configured to spatiotemporally bind the captured image sequence with the corresponding radar sensing data, time information and hard shoulder area location information to generate a structured evidence chain data package. The communication management unit is communicatively connected to both the edge computing unit and the external traffic management platform. It is configured to receive lane status instructions issued by the external traffic management platform and transmit them to the edge computing unit; upload the structured evidence chain data packet to the external traffic management platform; and generate early warning information with the identification and geographical location information of the violating vehicle and push it to the roadside information display device.

[0005] In one embodiment, when the edge computing unit performs spatiotemporal registration, it maps the target position in the radar coordinate system to the image coordinate system based on a preset coordinate transformation relationship, and determines whether to establish a target association based on the spatial distance between the mapped position and the center point of the video detection target.

[0006] In one embodiment, the preset coordinate transformation relationship is achieved through an end-to-end coordinate transformation matrix constructed from radar installation parameters and camera calibration parameters. The end-to-end coordinate transformation matrix is ​​the product of the projection matrix from the world coordinate system to the image coordinate system and the rotation and translation matrix from the radar coordinate system to the world coordinate system.

[0007] In one embodiment, when the edge computing unit generates the fused target trajectory, it uses a weighted fusion algorithm to fuse radar trajectory data sequences and visual trajectory data sequences for successfully associated targets, thereby generating a smooth and continuous fused target trajectory.

[0008] In one embodiment, the weight of radar trajectory data in the weighted fusion algorithm is positively correlated with the radar signal quality index, the weight of visual trajectory data is positively correlated with the image quality index, and the sum of the weights of radar trajectory data and visual trajectory data is 1.

[0009] In one embodiment, when the edge computing unit determines that the target is illegally occupying the hard shoulder, it comprehensively determines whether the target's continuous stay time within the electronic fence of the hard shoulder area exceeds a time threshold and whether the target's instantaneous speed is lower than a speed threshold.

[0010] In one embodiment, the edge computing unit dynamically adjusts the time threshold and speed threshold according to the lane status command transmitted by the communication management unit, wherein the lane status command indicates the current working mode of the hard shoulder.

[0011] In one embodiment, the edge computing unit calculates the overall image quality score of the video stream in real time. When the overall image quality score is lower than a preset quality threshold, the weight ratio of radar trajectory data in the weighted fusion is increased, and the target motion trend is predicted based on the radar trajectory data to generate the gimbal control parameters.

[0012] In one embodiment, when the overall image quality score is lower than a preset quality threshold, the edge computing unit uses a kinematic prediction model to calculate the predicted position of the target at the next moment. The state transition equation of the kinematic prediction model is expressed as: in, The position of the target at the current sampling time. Let the velocity value of the target be at the current sampling time. The target's position at the next predicted time. The velocity value of the target at the next predicted time. To predict the time step, The process noise vector is defined as follows: The edge computing unit generates control commands for the gimbal's horizontal rotation angle and pitch rotation angle based on the target's position at the next prediction time.

[0013] On the other hand, this application provides a method for accurate capture and analysis of hard shoulder video based on multimodal fusion, applied to the system for accurate capture and analysis of hard shoulder video based on multimodal fusion as described above, including the following steps: Acquire radar sensing data of moving targets within the hard shoulder area, wherein the radar sensing data includes target location information and motion status information; Video streams of the hard shoulder area are captured using cameras equipped with optical zoom and gimbal tracking capabilities; Receive lane status instructions from external traffic management platforms; Spatiotemporal registration is performed on the radar sensing data and the video stream to establish the correlation between radar-detected targets and video-detected targets, and a fused target trajectory is generated; Based on the fused target trajectory and the lane status command, determine whether the target is illegally occupying the hard shoulder; If the determination result is yes, a capture trigger signal is generated, and the camera zoom parameters and gimbal control parameters are calculated based on the capture trigger signal, and control commands are sent to the camera; The camera is driven to synchronously perform optical zoom and gimbal tracking operations according to the control command, and to capture continuous frames of the target to obtain an image sequence. The image sequence is spatiotemporally bound with the corresponding radar sensing data, time information, and hard shoulder area location information to generate a structured evidence chain data package; The structured evidence chain data package is uploaded to the external traffic management platform, and a warning message with the identification and geographical location information of the violating vehicle is generated and pushed to the roadside information display device.

[0014] The aforementioned system and method for precise video capture and analysis of hard shoulder based on multimodal fusion, through spatiotemporal registration and target association of fused radar sensing data and video streams, generates a highly robust fused target trajectory, effectively overcoming the perception blind spots and trajectory tracking interruptions of single sensors in adverse environments such as rain, fog, and night. Based on the fused target trajectory and combined with lane status commands issued by an external traffic management platform, the violation judgment logic is dynamically optimized, improving the accuracy of hard shoulder violation identification and its adaptability to dynamic switching of traffic management strategies. Zoom control parameters and gimbal tracking control parameters are precisely calculated based on the capture trigger signal, driving the camera to synchronously execute optical... Zooming and gimbal tracking enable clear and continuous capture of key areas of target vehicles. The captured image sequence is spatiotemporally bound with corresponding radar sensing data, time information, and hard shoulder area location information to form a complete and time-sequential structured evidence chain data package, ensuring the integrity and legal validity of the evidence chain. The evidence chain data package is uploaded to an external traffic management platform, and a warning message with the identification and geographical location information of the violating vehicle is generated and pushed to the roadside information display device. This achieves credible solidification of evidence of violations and closed-loop linkage of handling instructions, comprehensively improving the reliability of hard shoulder violation monitoring, the effectiveness of law enforcement evidence, and the timeliness of traffic incident response. Attached Figure Description

[0015] Figure 1 A structural block diagram of a system for precise capture and analysis of hard shoulder video based on multimodal fusion, provided in an embodiment of this application; Figure 2 A flowchart illustrating the method for precise capture and analysis of hard shoulder video based on multimodal fusion provided in this application embodiment. Detailed Implementation

[0016] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0017] In highway operation and management, the hard shoulder serves the dual functions of emergency rescue channel and traffic management during special periods. The accurate identification and effective handling of illegal occupation of the hard shoulder are directly related to road safety and traffic efficiency.

[0018] Current mainstream monitoring solutions in the industry generally adopt a single video analysis architecture: continuously collecting video streams of the hard shoulder area through fixed-installation high-definition cameras, and combining this with preset static rules (such as "a vehicle stopping for more than 30 seconds is considered a violation") for behavior recognition. This solution can maintain basic functionality under ideal conditions such as clear daytime, but it exposes systemic defects in real-world complex traffic scenarios. When encountering rain or fog that reduces visibility, low light at night that degrades the image signal-to-noise ratio, or strong glare that blurs the target outline, the vehicle detection confidence of the video analysis module drops significantly, the trajectory tracking process is frequently interrupted, and a large number of missed detections or false associations occur.

[0019] Some improvement schemes attempt to introduce millimeter-wave radar as an auxiliary sensing source, but due to the lack of a precise spatiotemporal registration mechanism between the radar coordinate system and the image coordinate system, the target position detected by the radar is only roughly mapped to the video image. The target association logic based on the position deviation threshold is not established, resulting in a state of "physical superposition and logical separation" of multi-source data. That is, radar data cannot correct visual tracking drift, and visual data is also difficult to verify radar false alarms. The continuity and reliability of the fused trajectory have not been substantially improved.

[0020] More importantly, there is an information silo between the existing system's judgment logic and the traffic management platform: the hard shoulder needs to dynamically switch its working mode (such as temporary opening or emergency closure) in scenarios such as holiday tidal traffic and emergency accident response, but the monitoring system still uses fixed thresholds to judge violations. It cannot adjust the dwell time threshold and speed threshold in real time according to the lane status instructions issued by the outside, which causes legal vehicles to be misjudged as violations in the open mode and real violations to be missed in the closed mode due to the lenient thresholds.

[0021] The aforementioned deficiencies collectively make it difficult for existing technologies to achieve high-precision identification, credible evidence generation, and closed-loop linkage of handling of illegal occupation of hard shoulders under the dual constraints of "adaptability to complex environments" and "coordination of dynamic management," which seriously restricts the practical value of intelligent transportation systems in terms of law enforcement credibility and emergency response effectiveness.

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Furthermore, the collection, transmission, storage, and use of all data in the embodiments of this application strictly comply with relevant data security regulations / laws: system deployment requires authorization and registration by the provincial traffic management department; vehicle identification information (including license plate images) recognition and evidence chain generation are only performed in the legally mandated business scenario of hard shoulder violation monitoring; the original video stream is automatically encrypted and deleted within 24 hours after the structured evidence chain data packet is generated, retaining only the de-identified evidence chain data; all communication links are encrypted using the national cryptographic SM4 algorithm, and data stored on the device side is signed and stored using the SM9 identifier cryptosystem; system operation logs are regularly audited by a third-party security authority to ensure compliance and controllability throughout the data lifecycle.

[0023] Firstly, this embodiment provides a system for precise capture and analysis of hard shoulder video based on multimodal fusion, such as... Figure 1 As shown, the system includes: The radar sensing unit is configured to output radar sensing data of moving targets within the hard shoulder area in real time. The radar sensing data includes target location information and motion status information. The video acquisition unit includes a camera with optical zoom and gimbal tracking capabilities, configured to acquire video streams of the hard shoulder area and perform target tracking and capture in response to control commands; The edge computing unit is communicatively connected to the radar sensing unit and the video acquisition unit, respectively. It is configured to perform spatiotemporal registration of radar sensing data and video stream, establish the correlation between radar-detected targets and video-detected targets, and generate fused target trajectories. Based on the fused target trajectory, it determines whether the target illegally occupies the hard shoulder. When the determination result is yes, it generates a capture trigger signal. It also calculates the camera zoom parameters and gimbal control parameters based on the capture trigger signal, generates control commands, and sends them to the video acquisition unit. The evidence generation unit is configured to spatiotemporally bind the captured image sequence with the corresponding radar sensing data, time information and hard shoulder area location information to generate a structured evidence chain data package. The communication management unit is connected to both the edge computing unit and the external traffic management platform. It is configured to receive lane status instructions from the external traffic management platform and transmit them to the edge computing unit; upload structured evidence chain data packets to the external traffic management platform; and generate warning information with the identification and geographical location information of the violating vehicles and push it to the roadside information display device.

[0024] Specifically, the radar sensing unit employs a millimeter-wave radar sensor operating at a frequency of 92 GHz to 94 GHz. It is mounted on an L-shaped bracket atop the outer guardrail post of the hard shoulder, at a height of 6.5 meters ± 0.5 meters, with an elevation angle adjusted to -7 degrees ± 2 degrees, ensuring complete coverage of the entire width of the hard shoulder and a longitudinal range of 500 meters. The radar sensing unit establishes a wired communication connection with the edge computing unit via an industrial-grade gigabit Ethernet interface (RJ45, compliant with IEEE 802.3at standard). The communication protocol uses TCP / IP, and the data frames include timestamps, target IDs, and three-dimensional coordinates. radial velocity In practical applications, the radar sensing unit can be any millimeter-wave radar product that meets the requirements for detection range and accuracy (such as Continental AG ARS548 and Bosch MRR). This application does not limit this.

[0025] The video acquisition unit uses a 4-megapixel high-definition network camera with 40x optical zoom (1 / 1.8-inch sensor size, minimum illumination 0.001 lux). The pan-tilt unit is a heavy-duty omnidirectional pan-tilt unit (horizontal rotation range 0° to 360° continuous, pitch range -45° to +90°). The camera is rigidly fixed to the pan-tilt unit flange with M8 stainless steel bolts. The pan-tilt unit control cable (RS485 interface) and video signal cable (gigabit Ethernet cable) are connected to the edge computing unit via an IP67 waterproof connector. In practical applications, the specific models of the camera and pan-tilt unit can be selected according to the length of the monitored section, installation height, and environmental conditions. This embodiment does not limit this.

[0026] The edge computing unit uses an embedded AI computing device (the main control chip is Rockchip RK3588, with a built-in 6-core ARM processor and 4-TOPS NPU) and runs a customized Linux operating system. The device has a built-in hardware clock module that synchronizes with the provincial traffic time server via the NTP protocol. The time synchronization accuracy is better than ±10 milliseconds, ensuring that the timestamps of radar sensing data and video streams are strictly aligned.

[0027] The evidence generation unit and the communication management unit are deployed as independent software processes on the edge computing unit, exchanging data with the main process of the edge computing unit through shared memory. The communication management unit integrates a 5G communication module (supporting SA / NSA dual mode) and establishes an MQTT communication link with the external traffic management platform based on TLS 1.3 encryption. The message format conforms to JT / T 1078 "Video Communication Protocol for Satellite Positioning System of Road Transport Vehicles".

[0028] During system operation, the radar sensing unit continuously outputs radar sensing data of moving targets in the hard shoulder area, while the video acquisition unit simultaneously acquires video streams of the hard shoulder area and transmits them to the edge computing unit. The edge computing unit performs spatiotemporal registration and target association on the radar sensing data and video streams, generating continuous and stable fused target trajectories, effectively compensating for the blind spots of single sensors in adverse environments. The fused target trajectory, combined with lane status commands, participates in the judgment logic for illegal hard shoulder occupation, improving the adaptability of the judgment results to dynamic switching of traffic management strategies. After triggering capture, the calculated camera zoom parameters and gimbal control parameters drive the video acquisition unit to complete clear and continuous capture of key areas of the target vehicle. The evidence generation unit spatiotemporally binds the captured image sequence, corresponding radar sensing data, precise timestamps, and electronic fence coordinates of the hard shoulder area to generate evidence compliant with GA / T standards. The 1400-2017 standard structured evidence chain data package (the data package is in JSON format and includes fields such as: Base64 encoded captured image, radar trajectory segment, timestamp sequence, electronic fence ID, and license plate recognition result); the communication management unit encrypts and uploads the structured evidence chain data package to the external traffic management platform, and generates warning information with the identification and geographical location information of the violating vehicle. This warning information is then pushed to the roadside information display device through the roadside variable message sign controller, realizing the reliable solidification of evidence of violations and the closed-loop linkage of handling instructions, thereby improving the reliability of hard shoulder violation monitoring, the effectiveness of law enforcement evidence, and the timeliness of traffic incident response.

[0029] To address the issues of target association errors and broken target trajectories caused by the heterogeneity between the radar coordinate system and the image coordinate system, one embodiment refines the target association determination step in spatiotemporal registration. Specifically, when the edge computing unit performs spatiotemporal registration, it maps the target position in the radar coordinate system to the image coordinate system based on a preset coordinate transformation relationship, and determines whether to establish target association based on the spatial distance between the mapped position and the center point of the target detected in the video.

[0030] In practice, the preset coordinate transformation relationship is determined in advance through an offline calibration process: A calibration marker is placed every 15 meters longitudinally along the hard shoulder area (a total of 12 markers), covering the entire width of the hard shoulder laterally. The calibration markers use a high-contrast black and white checkerboard pattern (9×6 squares, each square 20 cm on each side), a well-known calibration tool in the field of computer vision. During calibration, the radar sensing unit and video acquisition unit are simultaneously triggered to collect the calibration marker data (the hardware trigger signal is generated by the calibration controller, ensuring a time synchronization error of less than 5 milliseconds). The three-dimensional coordinates of each calibration marker in the radar coordinate system are then obtained. (i is the marker number, i=1,2,...,12) and its pixel coordinates in the video image. A predefined coordinate transformation relationship is constructed using a second-order polynomial mapping function: in The predicted pixel coordinates of the radar target mapped to the image coordinate system, with coefficients... to , to The solution is obtained by fitting the calibration data using the least squares method (the optimization objective is...). The solution process uses the Levenberg-Marquardt algorithm (a well-known method in the field of nonlinear least squares optimization), and the obtained coefficients are stored in the non-volatile memory of the edge computing unit. In practical applications, the preset coordinate transformation relationship can also be adopted by other well-known mapping models such as perspective transformation matrix, and this application does not limit this.

[0031] During real-time processing, the edge computing unit will calculate the three-dimensional position coordinates of the radar-detected target. Input a preset coordinate transformation relationship, and output its predicted pixel coordinates in the image coordinate system. The video analysis module uses the YOLOv5s object detection algorithm (input image resolution 640×640 pixels, confidence threshold set to 0.5; this algorithm is a well-known object detection method in the field of computer vision, and its open-source implementation can be found in the Ultralytics official repository) to detect vehicles in the video stream of the hard shoulder area, and outputs the center pixel coordinates of the bounding box of the detected targets. ; Calculate the Euclidean distance between the predicted pixel coordinates and the center pixel coordinates of the detected target in the video: in The unit is pixels; preset position association threshold Through on-site calibration, it was determined that: 100 pairs of radar-detected targets and video-detected targets with known correlations were collected during a clear day, and the statistical distribution of their Euclidean distances was calculated. The 95th percentile value was taken as the initial value of the position correlation threshold (a typical value is 5% of the image diagonal length, for example, for a 1920×1080 resolution image). Take 105 pixels), then fine-tune on-site; if If the radar-detected target and the video-detected target are the same physical target, an association is established and a unique association ID is assigned; otherwise, it is marked as an unassociated target and temporarily stored in the association buffer to wait for subsequent frame matching.

[0032] Based on the above, this mechanism effectively combines the high-precision ranging capability of the radar sensing unit with the high-resolution visual information of the video acquisition unit through precise coordinate mapping and spatial distance threshold determination. This eliminates false associations caused by sensor installation deviations or environmental interference, and the generated fused target trajectory remains continuous and smooth in both time and space dimensions. This provides a stable and reliable data foundation for subsequent determination and accurate capture of illegal occupation of hard shoulders, and improves the system's perception robustness and tracking reliability in complex environments.

[0033] To address the problem in existing technologies where the lack of systematic integration of installation and calibration parameters during the coordinate transformation process between radar and video sensors leads to accumulated mapping position deviations and decreased target association accuracy, one embodiment refines the mathematical construction mechanism of the preset coordinate transformation relationship. Specifically, the preset coordinate transformation relationship is realized through an end-to-end coordinate transformation matrix constructed from radar installation parameters and camera calibration parameters. This end-to-end coordinate transformation matrix is ​​the product of the projection matrix from the world coordinate system to the image coordinate system and the rotation and translation matrix from the radar coordinate system to the world coordinate system.

[0034] In practice, radar installation parameters include radar installation height. (Unit: meters, obtained through actual measurement with a laser rangefinder, typical value 6.5 meters), Radar elevation angle (Unit: degrees, calibrated by a digital inclinometer, typical value -7 degrees), Radar azimuth angle (Unit: degrees, measured with a total station, typical value 0 degrees); Camera calibration parameters were obtained using the Zhang Zhengyou calibration method: a 9×6 checkerboard calibration board (30 mm side length per grid) was used to acquire 20 calibration images at different angles within the camera's field of view. The camera intrinsic parameter matrix was calculated using the calibrateCamera function in the OpenCV 4.5.5 library. ( and This is the focal length in pixels. and (Primary point coordinates) and distortion coefficient vector The origin of the world coordinate system is set at the center of the starting point of the hard shoulder. The axis extends along the direction of the road. The axis is horizontal and perpendicular to the road centerline. The axis points vertically upwards; the origin of the radar coordinate system is located at the phase center of the radar antenna. The axis points in the direction of the radar beam's principal axis; construct the rotation and translation matrix from the radar coordinate system to the world coordinate system. (4×4 homogeneous matrix): Rotation part From azimuth With pitch angle Calculated in ZYX Euler angle order ( Translation part Construct the projection matrix from the world coordinate system to the image coordinate system. (3×4 matrix): ,in It is a 3-order identity matrix. As a zero vector, the world coordinate points are corrected using the Brown-Conrady distortion model before projection. End-to-end coordinate transformation matrix. (3×4 matrix) Calculated using matrix multiplication: ,in Indicates taking The first three rows form a 3×4 submatrix. For a point in the radar coordinate system... Its homogeneous coordinates in the image coordinate system The normalized pixel coordinates are The matrix calculation process is executed once during the initialization phase of the edge computing unit, and the result is stored in non-volatile memory. In practical applications, the end-to-end coordinate transformation matrix can also be constructed by the direct calibration method (using the Levenberg-Marquardt algorithm for optimization after collecting calibration marker data), and this application does not limit this.

[0035] Based on the above, the end-to-end coordinate transformation matrix tightly couples the radar installation geometry with the camera imaging model at the mathematical level, eliminating the accumulation of intermediate errors introduced by step-by-step coordinate transformation; the mapping position deviation is controlled within 3% of the image diagonal length, providing a high-precision spatial reference for the target association step, improving the spatial continuity and geometric consistency of the fused target trajectory, and enhancing the system's tolerance to sensor installation tolerances.

[0036] To address the issue that existing technologies using simple superposition or independent discrimination of multi-source trajectory data result in jitter, breaks, or scale distortion in the fused trajectory, affecting the reliability of violation determination, one embodiment refines the algorithm for generating the fused target trajectory. Specifically, when the edge computing unit generates the fused target trajectory, it uses a weighted fusion algorithm to fuse radar trajectory data sequences and visual trajectory data sequences for successfully associated targets, generating a smooth and continuous fused target trajectory.

[0037] In practice, the radar trajectory data sequence is the sequence of world coordinate positions of successfully correlated targets output by the radar sensing unit in N consecutive frames (N=10). ,in For frame number, The current frame; the visual trajectory data sequence is the position sequence of the same target mapped to the world coordinate system after a preset coordinate transformation relationship. The weighted fusion algorithm applies weighted fusion to each frame. Independently calculate the fused position coordinates: in The weight of the radar trajectory data in the k-th frame. The weights of the visual trajectory data in the k-th frame are used; after fusion, a sliding window mid-range filter is used for further smoothing: taking the current frame and the two frames before and after it (a total of 5 frames). Median filtering is performed independently on the x-coordinate sequence and the y-coordinate sequence, and the filtered coordinates are output separately. As the target trajectory points for fusion. The sliding window boundary frames are processed using a mirror-fill strategy; weights and The specific calculation logic is detailed later. In practical applications, the sliding window size can be dynamically adjusted according to the road curvature and vehicle speed (range 3 to 7 frames), and the median filter can also be replaced by well-known temporal smoothing algorithms such as Savitzky-Golay filtering. This application does not limit this.

[0038] Based on the above, the weighted fusion algorithm combines frame-level weight allocation and temporal smoothing to effectively suppress the interference of instantaneous noise and outliers from a single sensor on the trajectory shape. The generated fused target trajectory is geometrically continuous in the spatial dimension and smooth in the temporal dimension, avoiding trajectory jumps or breaks. It provides a stable and reliable kinematic basis for judging illegal occupation of hard shoulders and improves the reliability of trajectory tracking in scenarios with fluctuating sensor data.

[0039] To address the problem that existing fixed-weight fusion strategies cannot adapt to dynamic changes in sensor data quality, leading to a sharp drop in the reliability of fused trajectories in adverse environments such as rain, fog, and nighttime, one embodiment refines the dynamic weight calculation mechanism in the weighted fusion algorithm. Specifically, in the weighted fusion algorithm, the weights of radar trajectory data are positively correlated with radar signal quality indicators, the weights of visual trajectory data are positively correlated with image quality indicators, and the sum of the weights of radar trajectory data and visual trajectory data is 1.

[0040] In practice, the radar signal quality index directly adopts the target detection signal-to-noise ratio (SNR, unit: decibel) output by the radar sensing unit, which is provided in real time by the radar hardware; the image quality index is obtained by applying the Tenengrad gradient algorithm (calculating the sum of squares of the Sobel gradient magnitudes in the region) to the target bounding box outside the video frame by 15%, and then combining it with the target detection confidence output by the video analysis module, and weighted averaged with a weight of 0.7:0.3; the edge computing unit collects continuous monitoring data for 10 minutes during the system initialization phase, and calculates the minimum and maximum values ​​of the radar signal quality index and the image quality index as normalization parameters; during operation, the real-time acquired radar signal quality index and image quality index are mapped to the [0,1] interval through minimum-maximum normalization, and the normalized radar signal quality index is defined as follows. The normalized image quality index is labeled as Weights of radar trajectory data Weights of visual trajectory data Calculate as follows: , ,make sure ;when When (indicating that the quality of both source data is at an extremely low level), the system will force a setting. , This ensures that radar data plays a dominant role in the fusion process. Weight calculation is performed in real time by the fusion algorithm module of the edge computing unit. In practical applications, the calculation of image quality indicators can also adopt well-known image processing algorithms such as Laplacian variance and Brenner gradient, which are not limited in this application.

[0041] Based on the above, this dynamic weight allocation mechanism enables the fusion process to respond in real time to changes in sensor data quality: when the image quality index of the video is affected by rain, fog, or low light at night, the weight of the radar trajectory data is automatically increased to strengthen the support of radar data for the fused trajectory; when the signal-to-noise ratio of the radar is reduced due to interference from metal guardrail reflections, the weight of the visual trajectory data is increased accordingly; the constraint that the sum of the weights is always 1 ensures the spatial scale consistency of the fused trajectory and avoids trajectory scale drift; this mechanism improves the system's environmental adaptability under complex weather and lighting conditions and ensures the continuity and reliability of the fused target trajectory under sensor performance fluctuation scenarios.

[0042] To address the issue that existing systems use fixed thresholds for violation detection, which cannot adapt to the dynamic switching of hard shoulder operating modes, leading to misjudgments of legally passing vehicles during open traffic periods and missed detection of genuine violations during emergency closure periods, one embodiment refines the logic for determining illegal hard shoulder occupation. Specifically, when the edge computing unit determines that a target is illegally occupying the hard shoulder, it comprehensively considers whether the target's continuous stay within the electronic fence of the hard shoulder area exceeds a time threshold and whether the target's instantaneous speed is below a speed threshold.

[0043] In practical implementation, during the system deployment phase, the electronic fence for the hard shoulder area is marked by maintenance personnel using management software on the video screen as a polygonal vertex (at least 4 vertices). This is then converted into a closed polygonal region in the world coordinate system using a preset coordinate transformation relationship and stored in the edge computing unit. Continuous dwell time calculation: When the fused target trajectory point first enters the electronic fence area, the edge computing unit starts an independent timer for that target; if the target trajectory point leaves the electronic fence area, the timer is reset to zero; if the target remains within the electronic fence area, the accumulated time is used as the continuous dwell time. Instantaneous speed calculation: The coordinates of the fused target trajectory point in the current frame and the coordinates in the previous frame are taken, the Euclidean distance between the two points (unit: meters) is calculated, divided by the frame interval time (unit: seconds), and the instantaneous speed (unit: meters / second) is obtained and converted to kilometers per hour. Judgment logic: When the continuous dwell time is greater than the time threshold and the converted instantaneous speed is less than the speed threshold, the edge computing unit determines that the target is illegally occupying the hard shoulder. The initial values ​​of the time threshold and speed threshold are set in the system configuration file, with typical values ​​of 30 seconds and 5 kilometers per hour, respectively. In practical applications, electronic fence calibration can also be performed by measuring coordinates on-site with a laser rangefinder and then importing them into the system. Instantaneous velocity calculation can also be performed by linear fitting of three consecutive frames of trajectory points to suppress noise. This application does not limit this.

[0044] Based on the above, this judgment logic effectively distinguishes between temporary emergency parking and illegal occupation within the hard shoulder area through dual constraints of time and speed; the precise calibration of the electronic fence ensures that the judgment area strictly corresponds to the physical hard shoulder; and the real-time quantitative calculation of continuous dwell time and instantaneous speed provides an objective basis for judgment, avoiding subjective misjudgment. This mechanism lays the logical foundation for subsequent dynamic threshold adjustments, improving the accuracy and robustness of violation judgment in static scenarios.

[0045] To address the issue of a disconnect between the violation judgment logic and the traffic management platform, which prevents real-time adjustment of judgment parameters based on the actual operating mode of the hard shoulder (such as temporary opening during holidays or emergency closure due to accidents), leading to an increased rate of false positives and false negatives during control strategy transitions, one embodiment refines the dynamic adjustment mechanism for judgment thresholds. Specifically, the edge computing unit dynamically adjusts the time and speed thresholds based on lane status instructions transmitted by the communication management unit, with the lane status instructions indicating the current operating mode of the hard shoulder.

[0046] In practice, lane status commands are generated by an external traffic management platform and sent to the edge computing unit in JSON format via the communication management unit. The command content includes a working mode identifier field (a value of "closed" indicates the regular closed mode, and "open" indicates the traffic-borrowing open mode) and a command timestamp. The edge computing unit's internal non-volatile memory contains a pre-set threshold mapping table, which defines the correspondence between working mode identifiers and threshold parameters: in the regular closed mode, the time threshold is 30 seconds and the speed threshold is 5 km / h; in the traffic-borrowing open mode, the time threshold is 120 seconds and the speed threshold is 20 km / h. Upon receiving the lane status command, the edge computing unit parses the working mode identifier, queries the threshold mapping table, updates the currently used time and speed thresholds to the corresponding parameters in the mapping table, and records the update timestamp. The updated thresholds take effect immediately and are used for subsequent violation determination of all targets. The threshold mapping table supports remote configuration and updates via the external traffic management platform to adapt to differences in management strategies for different road segments. In practical applications, the working mode identifier can also be extended to a multi-level mode (such as "emergency" representing emergency mode), and the threshold mapping table will be updated accordingly. This application embodiment does not limit this.

[0047] Based on the above, this dynamic adjustment mechanism synchronizes the system's judgment logic with traffic management strategies in real time: during periods when the hard shoulder is open to traffic, the judgment threshold is relaxed to avoid misjudging legally passing vehicles as violations; during periods when the hard shoulder is closed for control, the judgment threshold is tightened to improve the detection rate of genuine violations; the threshold adjustment process is driven by external commands, requiring no manual on-site intervention, achieving deep collaboration between the monitoring system and the traffic management platform. This mechanism enhances the system's adaptability to dynamic traffic management scenarios, reduces misjudgments and omissions during strategy switching, and strengthens law enforcement credibility and traffic management efficiency.

[0048] To address the issue of significantly reduced reliability of visual trajectory data in low-quality video environments such as rain, fog, and nighttime, leading to distorted fused trajectories and failed gimbal tracking command generation, one embodiment refines the dynamic weight adjustment and motion trend prediction mechanism of the edge computing unit when video quality deteriorates. Specifically, the edge computing unit calculates the comprehensive image quality score of the video stream in real time. When the comprehensive image quality score is lower than a preset quality threshold, the weight ratio of radar trajectory data in the weighted fusion is increased, and the target motion trend is predicted based on the radar trajectory data to generate gimbal control parameters.

[0049] In practice, the comprehensive image quality score is directly based on the aforementioned normalized image quality index. (This metric integrates the sharpness and contrast information of the target area in the video frame.) Preset quality threshold During the system deployment phase, on-site calibration determined that: 100 frames of video should be continuously acquired in typical rain and fog scenarios with visibility below 100 meters, and calculations should be performed. The statistical distribution is used, and its 10th percentile value is taken as... (Typical value is 0.35). When calculated in real time... At that time, the edge computing unit performs the following operations: First, it assigns weights to the radar trajectory data. The weight of the visual trajectory data is forced to be 0.85. The ratio is set to 0.15 (this ratio setting refers to the processing logic of the aforementioned dual-source data quality extremely low scenario, ensuring that radar data plays a dominant role in the fusion); secondly, the use of visual trajectory data in motion trend prediction is suspended, and instead, the predicted position of the target at the next moment is calculated based on the radar trajectory data points (world coordinate sequence) of the most recent 5 consecutive frames, using a linear extrapolation method: first-order linear fitting is performed on the X coordinate sequence and Y coordinate sequence respectively to obtain the motion slope, and the predicted coordinates are calculated in combination with the frame interval time Δt (taking the frame period of the video acquisition unit as 0.033 seconds). The predicted coordinates are mapped to the image coordinate system through a preset coordinate transformation relationship, serving as the input basis for generating gimbal control parameters; the calculation logic of the gimbal control parameters is consistent with the aforementioned description. In practical applications, the comprehensive image quality score can also adopt an independently designed multi-index fusion scoring mechanism, and the linear extrapolation method can also be replaced by short-time prediction methods known in the fields of image processing and target tracking, such as uniform motion models; this application embodiment does not limit this.

[0050] Based on the above, the mechanism proactively reduces its reliance on visual data when video quality deteriorates. By forcibly adjusting weights and predicting motion trends dominated by radar, it maintains the continuity of the fused target trajectory and the stability of the generation of gimbal control commands. It avoids tracking interruptions or gimbal pointing deviations caused by visual data failure, ensuring accurate capture capabilities in harsh environments and improving the overall robustness of the system.

[0051] To address the issue that simple linear extrapolation struggles to accurately predict target position when video quality is severely degraded and target motion states exhibit acceleration changes, leading to lag or deviation in gimbal tracking, one embodiment refines the specific implementation of the kinematic prediction model and the logic for generating gimbal control parameters. Specifically, when the overall image quality score is below a preset quality threshold, the edge computing unit uses the kinematic prediction model to calculate the predicted position of the target at the next moment. The state transition equation of the kinematic prediction model is expressed as: in, The position of the target at the current sampling time. Let the velocity value of the target be at the current sampling time. The target's position at the next predicted time. The velocity value of the target at the next predicted time. To predict the time step, The process noise vector is used; the edge computing unit generates control commands for the gimbal's horizontal and pitch rotation angles based on the target's position at the next prediction time.

[0052] In practical implementation, the kinematic prediction model constructs a discrete state-space model based on the assumption of uniform motion, which is suitable for predicting the short-term (Δt≤0.1 seconds) motion of vehicles in hard shoulder scenarios; position state With speed state Obtained from radar trajectory data after Kalman filtering and smoothing (Kalman filter initialization: state covariance matrix) Process noise covariance matrix The observation noise covariance R = 1.0. After each frame of radar data is input, state prediction and measurement update are performed sequentially. The time interval Δt is taken as the actual frame period of the video acquisition unit (typical value 0.033 seconds). Process noise vector. Modeled as zero-mean Gaussian white noise, its statistical properties are determined by Characterization. During prediction, the edge computing unit will use the current filtering state. Substituting into the state transition equation, the predicted state at the next time step is calculated, and the result is extracted. The predicted position of the target at the next moment (two-dimensional coordinates in the world coordinate system) ). Gimbal control parameter generation: Camera installation world coordinates ( Substitute the installation height into the geometric formula to calculate the horizontal rotation angle of the gimbal. ( (Initial horizontal tilt angle of the camera), pitch angle The calculation results are encoded into PTZ control commands via the Pelco-D protocol and sent to the video acquisition unit. In practical applications, the kinematic prediction model can also be extended to a uniform acceleration model that includes an acceleration term. The Kalman filter parameters can be adjusted according to the on-site calibration data, and the PTZ control protocol can also adopt well-known standard protocols such as ONVIF. This application does not limit this aspect.

[0053] Based on the above, this kinematic prediction model effectively integrates historical motion information of radar trajectories and suppresses measurement noise through state-space modeling and Kalman filtering, thereby improving the temporal continuity and spatial accuracy of the predicted position. By combining the camera installation geometric parameters to accurately calculate the gimbal control parameters, it ensures that the gimbal can still accurately point to the predicted target position when visual information is missing, maintaining the clarity and target integrity of continuous frame captures. This mechanism enhances the tracking reliability of the system and the generation quality of structured evidence chain data packets in extremely harsh environments.

[0054] In summary, the system for precise capture and analysis of hard shoulder video based on multimodal fusion in this implementation constructs a five-order closed-loop technical link of "perception-fusion-determination-execution-solidification": The system synchronously acquires radar perception data and video streams, and achieves precise mapping between the radar coordinate system and the image coordinate system based on a preset coordinate transformation relationship (an end-to-end coordinate transformation matrix constructed from radar installation parameters and camera calibration parameters). The system determines the target association based on the spatial distance between the mapped position and the center point of the video detection target, generating a continuous and smooth fused target trajectory. This trajectory fusion process employs a dynamic weighting mechanism, where the radar trajectory data weight is positively correlated with the radar signal quality index, and the visual trajectory data weight is positively correlated with the image quality index, with the sum of the weights always being 1, ensuring that the trajectory maintains spatial continuity and scale consistency even when sensor data quality fluctuates. The system receives lane status instructions from an external traffic management platform and, according to the hard shoulder working mode indicated by the instructions (such as normal...),... The system dynamically adjusts the time and speed thresholds for violation determination in both closed and open traffic borrowing modes. It combines the target trajectory with dual conditions to determine the target's duration of stay and instantaneous speed within the electronic fence of the hard shoulder area. After determining a violation, the system calculates zoom control parameters and PTZ tracking control parameters to drive the camera to perform optical zoom and PTZ tracking actions. When the overall video quality score is lower than a preset quality threshold, the system forcibly increases the weight ratio of radar trajectory data and uses a kinematic prediction model based on radar trajectory data (including state transition equations and Kalman filter smoothing) to calculate the target's predicted position at the next moment, generating high-precision PTZ control commands. Finally, the captured image sequence is spatiotemporally bound with radar sensing data, time information, and hard shoulder area location information to generate a structured evidence chain data package conforming to industry standards and upload it. Simultaneously, a warning message containing the violating vehicle's identification and geographical location information is generated and pushed to the roadside information display device.

[0055] The system for precise video capture and analysis of hard shoulder based on multimodal fusion, employing this implementation method, overcomes the perception blind spots and trajectory tracking interruptions of single sensors in adverse environments such as rain, fog, and nighttime through precise spatiotemporal registration and dynamic weighted fusion of radar and video, ensuring the robustness of the fused target trajectory. A threshold dynamic adjustment mechanism driven by lane status commands eliminates the disconnect between violation judgment logic and traffic management strategies, enhancing the system's adaptability to dynamic switching of hard shoulder operating modes and avoiding misjudgments during open periods and omissions during closed periods. Radar-led prediction and precise gimbal control ensure the clarity and target integrity of captured images in adverse environments through radar-dominated prediction when video quality deteriorates. Combined with a structured evidence chain generated by spatiotemporal binding of multi-source data, it achieves credible solidification of violation evidence and closed-loop linkage of handling commands. Overall, the system achieves technical synergy in three aspects: adaptability to complex environments, dynamic management collaboration, and the effectiveness of law enforcement evidence, providing stable, reliable, and compliant technical support for intelligent management and control of hard shoulders.

[0056] On the other hand, this embodiment provides a method for accurate capture and analysis of hard shoulder video based on multimodal fusion, which is applied to the system for accurate capture and analysis of hard shoulder video based on multimodal fusion as described above, such as... Figure 2 As shown, the method includes the following steps in sequence: Acquire radar perception data of moving targets within the hard shoulder area. The radar perception data includes target location information and motion status information. Video streams of the hard shoulder area are captured using cameras equipped with optical zoom and gimbal tracking capabilities; Receive lane status instructions from external traffic management platforms; Spatiotemporal registration is performed on radar sensing data and video streams to establish the correlation between radar-detected targets and video-detected targets, and to generate fused target trajectories; Based on the fusion of target trajectory and lane status instructions, determine whether the target is illegally occupying the hard shoulder; If the determination result is yes, a capture trigger signal is generated, and the camera zoom parameters and PTZ control parameters are calculated based on the capture trigger signal, and control commands are sent to the camera; The camera is driven to synchronously perform optical zoom and gimbal tracking actions according to control commands, and captures continuous frames of the target to obtain an image sequence. The image sequence is spatiotemporally bound with the corresponding radar sensing data, time information, and hard shoulder area location information to generate a structured evidence chain data package; The structured evidence chain data package is uploaded to an external traffic management platform, and a warning message with the identification and geographical location information of the offending vehicle is generated and pushed to the roadside information display device.

[0057] Corresponding to the aforementioned system for precise capture and analysis of hard shoulder video based on multimodal fusion, the specific execution process of the method for precise capture and analysis of hard shoulder video based on multimodal fusion in this embodiment is as follows: Radar sensing data is acquired in real time by a 92 GHz to 94 GHz millimeter-wave radar sensor deployed on the top of the guardrail post on the outer side of the hard shoulder. The data frame contains the target's three-dimensional coordinates, radial velocity, and timestamp, with a sampling frequency of 20 Hz. The video stream is acquired at a rate of 25 frames per second by a 40x optical zoom high-definition network camera mounted on the same post. The camera and heavy-duty pan-tilt unit are rigidly connected via a mechanical flange. Lane status commands are issued by an external traffic management platform via a 5G communication link in JSON format. The commands include a working mode identifier ("closed" or "open") and a timestamp. In the spatiotemporal registration step, a pre-stored end-to-end coordinate transformation matrix (constructed from radar installation parameters and camera calibration parameters) is called to map the radar coordinates to the image coordinate system. The Euclidean distance between the mapped point and the center point of the video-detected target is calculated. When the distance is less than a preset position association threshold, an association is established. The target trajectory generation adopts a dynamic weighted fusion algorithm, with the radar trajectory data weights and radar signal quality indicators (signal-to-noise ratio) as factors. The visual trajectory data weights are positively correlated with the image quality index (Tenengrad gradient and detection confidence weighted value), and the sum of the weights is always 1. The data is then smoothed by a 5-frame sliding window mid-range filter. In the violation determination step, a preset threshold mapping table is queried based on the lane status command (normal closed mode: time threshold 30 seconds, speed threshold 5 km / h; traffic borrowing open mode: time threshold 120 seconds, speed threshold 20 km / h), comprehensively determining the target's continuous dwell time and instantaneous speed within the electronic fence. When the overall image quality score is lower than the preset quality threshold (0.35), the radar trajectory data weight is forcibly set to 0.85, and a kinematic prediction model is used based on the radar trajectory data to calculate the target's predicted position at the next moment, generating gimbal control parameters. Three consecutive frames are captured, with each frame spaced 33 milliseconds apart. The spatiotemporal binding step combines the image sequence, radar trajectory fragments, precise timestamps (synchronization accuracy ±10 milliseconds), electronic fence ID, and license plate recognition results according to GA / T. The 1400-2017 standard encapsulates the evidence chain into a JSON-formatted structured data packet. The upload and push steps transmit the data packet to an external traffic management platform via TLS 1.3 encrypted MQTT protocol, generating the text "License plate [recognition result] occupies the hard shoulder at [station number], please leave immediately" and pushing it to the roadside variable message sign. Furthermore, all data processing in this method strictly complies with the Personal Information Protection Law, the Cybersecurity Law, and traffic data security regulations. Vehicle identification information processing is legally authorized, and the original video stream is encrypted and deleted within 24 hours of evidence chain generation.

[0058] Based on the above, this method generates a continuous and stable fused target trajectory through precise spatiotemporal registration and dynamic weighted fusion of radar sensing data and video streams, effectively overcoming the perception blind spots and tracking interruptions of single sensors in adverse environments such as rain, fog, and night. The lane status command-driven threshold dynamic adjustment mechanism ensures real-time synchronization between violation judgment logic and traffic management strategies, avoiding misjudging legally passing vehicles during open periods and missing genuine violations during closed periods. Radar-led prediction and precise gimbal control when video quality deteriorates ensure the clarity and target integrity of captured images in adverse environments. The structured evidence chain data package generated by spatiotemporal binding of multi-source data conforms to industry standards, ensuring the integrity and legal validity of the evidence chain. The closed-loop push of warning information and the evidence chain achieves timeliness and synergy in handling violations. Overall, this method forms a technical closed loop in terms of adaptability to complex environments, dynamic management synergy, and the effectiveness of law enforcement evidence, providing a reproducible, highly reliable, and compliant technical implementation path for intelligent management of hard shoulders, improving traffic management efficiency and law enforcement credibility.

[0059] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0060] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0061] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A system for precise capture and analysis of hard shoulder video based on multimodal fusion, characterized in that, include: The radar sensing unit is configured to output radar sensing data of moving targets within the hard shoulder area in real time. The radar sensing data includes target location information and motion status information. The video acquisition unit includes a camera with optical zoom and gimbal tracking capabilities, configured to acquire video streams of the hard shoulder area and perform target tracking and capture in response to control commands; An edge computing unit is communicatively connected to the radar sensing unit and the video acquisition unit, respectively, and is configured to perform spatiotemporal registration of the radar sensing data and the video stream, establish the correlation between radar-detected targets and video-detected targets, and generate fused target trajectories; Based on the fused target trajectory, it is determined whether the target illegally occupies the hard shoulder. When the determination result is yes, a capture trigger signal is generated. Furthermore, based on the capture trigger signal, the camera zoom parameters and gimbal control parameters are calculated, and control commands are generated and sent to the video acquisition unit; The evidence generation unit is configured to spatiotemporally bind the captured image sequence with the corresponding radar sensing data, time information and hard shoulder area location information to generate a structured evidence chain data package. The communication management unit is communicatively connected to both the edge computing unit and the external traffic management platform, and is configured to receive lane status commands issued by the external traffic management platform and transmit them to the edge computing unit; and to upload the structured evidence chain data packet to the external traffic management platform. It also generates warning information with identification and geographical location information of vehicles violating regulations, and pushes it to roadside information display devices.

2. The system for precise capture and analysis of hard shoulder video based on multimodal fusion according to claim 1, characterized in that, When performing spatiotemporal registration, the edge computing unit maps the target position in the radar coordinate system to the image coordinate system based on a preset coordinate transformation relationship, and determines whether to establish a target association based on the spatial distance between the mapped position and the center point of the video detection target.

3. The system for precise capture and analysis of hard shoulder video based on multimodal fusion according to claim 2, characterized in that, The preset coordinate transformation relationship is achieved through an end-to-end coordinate transformation matrix constructed from radar installation parameters and camera calibration parameters. The end-to-end coordinate transformation matrix is ​​the product of the projection matrix from the world coordinate system to the image coordinate system and the rotation and translation matrix from the radar coordinate system to the world coordinate system.

4. The system for precise capture and analysis of hard shoulder video based on multimodal fusion according to claim 2, characterized in that, When the edge computing unit generates the fused target trajectory, it uses a weighted fusion algorithm to fuse radar trajectory data sequences and visual trajectory data sequences for successfully associated targets, generating a smooth and continuous fused target trajectory.

5. The system for precise capture and analysis of hard shoulder video based on multimodal fusion according to claim 4, characterized in that, In the weighted fusion algorithm, the weights of radar trajectory data are positively correlated with radar signal quality indicators, the weights of visual trajectory data are positively correlated with image quality indicators, and the sum of the weights of radar trajectory data and visual trajectory data is 1.

6. The system for precise capture and analysis of hard shoulder video based on multimodal fusion according to any one of claims 1 to 5, characterized in that, When the edge computing unit determines that a target is illegally occupying the hard shoulder, it comprehensively determines whether the target's continuous stay time within the electronic fence of the hard shoulder area exceeds the time threshold and whether the target's instantaneous speed is lower than the speed threshold.

7. The system for precise capture and analysis of hard shoulder video based on multimodal fusion according to claim 6, characterized in that, The edge computing unit dynamically adjusts the time threshold and speed threshold according to the lane status command transmitted by the communication management unit, and the lane status command indicates the current working mode of the hard shoulder.

8. The system for precise capture and analysis of hard shoulder video based on multimodal fusion according to claim 1 or 4, characterized in that, The edge computing unit calculates the overall image quality score of the video stream in real time. When the overall image quality score is lower than a preset quality threshold, it increases the weight ratio of radar trajectory data in the weighted fusion and predicts the target motion trend based on the radar trajectory data to generate the gimbal control parameters.

9. The system for precise capture and analysis of hard shoulder video based on multimodal fusion according to claim 8, characterized in that, When the overall image quality score is lower than a preset quality threshold, the edge computing unit uses a kinematic prediction model to calculate the predicted position of the target at the next moment. The state transition equation of the kinematic prediction model is expressed as: in, The position of the target at the current sampling time. Let the velocity value of the target be at the current sampling time. The target's position at the next predicted time. The velocity value of the target at the next predicted time. To predict the time step, The process noise vector is defined as follows: The edge computing unit generates control commands for the gimbal's horizontal rotation angle and pitch rotation angle based on the target's position at the next prediction time.

10. A method for accurate capture and analysis of hard shoulder video based on multimodal fusion, characterized in that, Includes the following steps: Acquire radar sensing data of moving targets within the hard shoulder area, wherein the radar sensing data includes target location information and motion status information; Video streams of the hard shoulder area are captured using cameras equipped with optical zoom and gimbal tracking capabilities; Receive lane status instructions from external traffic management platforms; Spatiotemporal registration is performed on the radar sensing data and the video stream to establish the correlation between radar-detected targets and video-detected targets, and a fused target trajectory is generated; Based on the fused target trajectory and the lane status command, determine whether the target is illegally occupying the hard shoulder; If the determination result is yes, a capture trigger signal is generated, and the camera zoom parameters and gimbal control parameters are calculated based on the capture trigger signal, and control commands are sent to the camera; The camera is driven to synchronously perform optical zoom and gimbal tracking operations according to the control command, and to capture continuous frames of the target to obtain an image sequence. The image sequence is spatiotemporally bound with the corresponding radar sensing data, time information, and hard shoulder area location information to generate a structured evidence chain data package; The structured evidence chain data package is uploaded to the external traffic management platform, and a warning message with the identification and geographical location information of the violating vehicle is generated and pushed to the roadside information display device.

Images