Multi-mode monitoring all-in-one machine

Abstract

The invention provides a multi-mode monitoring all-in-one machine, and relates to the technical field of intelligent traffic. The all-in-one machine comprises a machine body; an active protective housing; the vertical array optical subsystem is embedded into the active protective shell and comprises a high-definition camera module and an infrared thermal imaging module; the self-adaptive geometric calibration mechanism is arranged on a base of the vertical array optical subsystem and is used for calibrating the angle change of the vertical array optical subsystem; the self-sensing attitude compensation base is fixedly arranged at the lower end of the active protective shell and is used for adjusting the attitude change of the vertical array optical subsystem; the AI video processing terminal is in communication connection with the vertical array optical subsystem and is used for acquiring environmental parameters; dynamically adjusting a visible light weight and an infrared weight based on the environmental parameters; obtaining a visible light video and an infrared video after adjustment, and extracting visible light features and infrared features; and carrying out interaction and fusion on the visible light features and the infrared features through a cross-modal attention mechanism to generate an enhanced feature map. According to the invention, the adaptability and reliability of monitoring can be improved.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a multimodal monitoring integrated machine. Background Technology

[0002] Existing multimodal monitoring systems have three types of problems during long-term operation: First, the optical axis and pitch angle of the multi-module array drift over time, requiring manual recalibration; second, the equipment attitude is affected by wind vibration or pole deformation, resulting in monitoring coverage errors; and third, the outer casing protection structure is simple and lacks environmental adaptability.

[0003] Traditional solutions rely heavily on manual calibration and fixed supports, which are cumbersome to maintain and have unstable accuracy. Especially in windy and high / low temperature environments, the accumulation of geometric parameter errors will significantly affect the accuracy of image stitching and AI algorithm recognition. Summary of the Invention

[0004] This application provides a multimodal monitoring integrated machine to solve the problem that the monitoring results are poor due to deviations in the operation of existing multimodal monitoring equipment.

[0005] In the first aspect, this application provides a multimodal monitoring all-in-one machine, including a vertical array optical subsystem, an active protective shell, an adaptive geometric calibration mechanism, a self-sensing attitude compensation base, and an AI video processing terminal; The active protection housing is a longitudinally narrow box-shaped structure, and both the adaptive geometric calibration mechanism and the AI ​​video processing terminal are housed within the active protection housing. The vertical array optical subsystem is embedded in the active protective housing. The vertical array optical subsystem includes a high-definition camera module and an infrared thermal imaging module that are connected in communication. The high-definition camera module and the infrared thermal imaging module are arranged in a vertical array, and the infrared thermal imaging module is arranged at the center of the vertical array optical subsystem in the vertical direction. The adaptive geometric calibration mechanism is respectively mounted on the base of the high-definition camera module and the infrared thermal imaging module, and is used to calibrate the angle changes of the high-definition camera module and the infrared thermal imaging module; The self-sensing attitude compensation base is fixedly installed at the lower end of the active protective shell and is used to adjust the attitude changes of the vertical array optical subsystem. The AI ​​video processing terminal is communicatively connected to the vertical array optical subsystem and is used for: Acquire environmental parameters, including light intensity, visibility, weather type, and image signal-to-noise ratio; Based on the environmental parameters, the visible light weight of the high-definition camera module and the infrared weight of the infrared thermal imaging module are dynamically adjusted. The system acquires visible light video captured by the high-definition camera module and infrared video captured by the infrared thermal imaging module with dynamically adjusted visible light and infrared weights, and extracts visible light features from the visible light video and infrared features from the infrared video. The visible light features and the infrared features are interacted and fused through a cross-modal attention mechanism to generate an enhanced feature map for status monitoring and early warning.

[0006] This application provides a multimodal monitoring all-in-one device, which integrates a vertical array optical subsystem, an active protective housing, an adaptive geometric calibration mechanism, a self-sensing attitude compensation base, and an AI video processing terminal into one unit. The adaptive geometric calibration mechanism and the AI ​​video processing terminal are housed within the active protective housing, with the vertical array optical subsystem embedded within it. This compact design significantly saves installation space, facilitating deployment in various confined spaces. Furthermore, the AI ​​video processing terminal can acquire environmental parameters, including light intensity, visibility, weather type, and image signal-to-noise ratio, and dynamically adjust the visible light weight of the high-definition camera module and the infrared weight of the infrared thermal imaging module based on these parameters. This intelligent weight adjustment method... The AI ​​video processing terminal can automatically optimize the contributions of the two imaging modes based on different environmental conditions. It increases the weight of visible light when there is sufficient light and increases the weight of infrared light when there is no light at night or in low visibility environments. This allows it to acquire high-quality video images in different environments, improving the adaptability and reliability of monitoring. At the same time, the AI ​​video processing terminal can extract visible light features from visible light videos captured by high-definition camera modules and infrared features from infrared videos captured by infrared thermal imaging modules. It then interacts and fuses these two types of features through a cross-modal attention mechanism to generate an enhanced feature map. This cross-modal feature fusion method can make full use of the complementary information of visible light and infrared images, improve the expressive power and discriminative power of features, and thus more accurately monitor the status. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a schematic diagram of the external structure of the multimodal monitoring integrated machine provided in the embodiments of this application; Figure 2 This is a schematic diagram of the internal structure of the multimodal monitoring integrated machine provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the vertical array optical subsystem provided in the embodiments of this application; Figure 4 This is a schematic diagram of an application scenario for multiple multimodal monitoring integrated machines provided in the embodiments of this application. Detailed Implementation

[0009] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0011] Figure 1 This is a schematic diagram of the external structure of the multimodal monitoring integrated machine provided in the embodiments of this application. Figure 2 This is a schematic diagram of the internal structure of the multimodal monitoring integrated machine provided in an embodiment of this application. Figure 1 and Figure 2 As shown, the multimodal monitoring integrated machine includes: a vertical array optical subsystem 1, an active protective housing 2, and an adaptive geometric calibration mechanism 3. Figure 1 Not provided in the text, see [link / reference] Figure 2 ), self-sensing posture compensation base 4 and AI video processing terminal 5 ( Figure 1 Not provided in the text, refer to Figure 2 ).

[0012] In this embodiment, the vertical array optical subsystem 1, active protective shell 2, adaptive geometric calibration mechanism 3, self-sensing attitude compensation base 4, and AI video processing terminal 5 are modularly connected via slots, supporting high-speed signal transmission and rapid maintenance.

[0013] The active protective housing 2 has a longitudinal narrow box-shaped structure, and the adaptive geometry calibration mechanism 3 and the AI ​​video processing terminal 5 are both located inside the active protective housing 2.

[0014] The aspect ratio of the active protective housing 2 can be set to... It features waterproof, dustproof, anti-glare, and active heat dissipation functions.

[0015] In this embodiment, the active protective shell is integrally molded from high-strength aluminum alloy, and its wind resistance coefficient can be... It has an IP67 protection rating and excellent shock resistance, and can withstand wind load impact, high and low temperature alternation and vibration interference in complex environments.

[0016] Additionally, refer to Figure 2 As shown, the multimodal monitoring all-in-one unit also includes an internal heat dissipation module 6. This internal heat dissipation module 6 consists of an aluminum heat sink and a silent fan. When the temperature sensor detects the internal temperature... When needed, the heat dissipation mechanism is automatically activated to ensure the multimodal monitoring unit operates within a safe temperature range. It operates stably in a wide temperature range.

[0017] In this embodiment, the adaptive geometry calibration mechanism 3, the AI ​​video processing terminal 5, the internal heat dissipation module 6, and the power supply unit are all integrated inside the active protection housing 2. The active protection housing 2 uses a sealed cavity partition design to achieve physical isolation and protection of each functional module, while optimizing the internal space layout to ensure heat dissipation efficiency and signal transmission stability.

[0018] In this embodiment, the multimodal monitoring all-in-one machine also includes a supplementary light for providing corresponding supplementary lighting support when the vertical array optical subsystem is taking pictures.

[0019] The vertical array optical subsystem 1 is embedded within the active protective housing 2 and secured with precision clips and positioning pins to ensure installation accuracy and structural stability. The vertical array optical subsystem 1 includes a high-definition camera module 11 and an infrared thermal imaging module 12 connected for communication. The high-definition camera module 11 and the infrared thermal imaging module 12 are arranged in a vertical array, with the infrared thermal imaging module 12 positioned at the center of the vertical array optical subsystem 1, enabling all-weather thermal imaging monitoring.

[0020] In one possible implementation, refer to Figure 1 The high-definition camera module 11 includes a short-focus high-definition camera module 111, a medium-focus high-definition camera module 112, and a long-focus high-definition camera module 113. The long-focus high-definition camera module 113, the medium-focus high-definition camera module 112, and the short-focus high-definition camera module 111 are arranged in a vertical array in the vertical direction and are arranged in a vertical fan-shaped pitch. The short-throw high-definition camera module 111 is used to monitor visible light video within the first coverage area; The high-definition video camera module 112 is used to monitor visible light video in the second coverage area; The long-focus high-definition camera module 113 is used to monitor visible light video within a third coverage area; Among them, the first coverage area, the second coverage area and the third coverage area all have overlapping areas, and with the multimodal monitoring integrated machine as the origin, the first coverage area, the second coverage area and the third coverage area are successively away from the origin.

[0021] Optionally, refer to Figure 1As shown, the short-focus high-definition camera module 111, the medium-focus high-definition camera module 112, and the long-focus high-definition camera module 113, together with the infrared thermal imaging module 12, constitute the core imaging unit of the vertical array optical subsystem 1, which is functionally complementary. To achieve full-range, blind-spot-free monitoring coverage from 0 to 500 meters (near-medium-far), in this embodiment, the long-focus high-definition camera module 113, the medium-focus high-definition camera module 112, and the short-focus high-definition camera module 111 are arranged in a precise vertical array in the longitudinal direction. They are fixed to the positioning guide rail inside the active protection shell 2 via a customized mounting bracket (i.e., included in the self-sensing attitude compensation base 4), ensuring verticality deviation during installation. This ensures the stability and consistency of the array results.

[0022] Meanwhile, the three types of high-definition camera modules 11 are arranged in a vertical fan-shaped pitch pattern, and their pitch angles are optimized and matched to form a layered and progressive monitoring coverage pattern, effectively avoiding monitoring blind spots.

[0023] For example, using the horizontal plane as a reference, the telephoto high-definition camera module, the mid-range high-definition camera module, the infrared thermal imaging module, and the short-focus high-definition camera module each have a focal length below the horizontal plane. , , and The system features a fan-shaped pitch angle range, the parameters of which are determined based on extensive scene measurements and optical simulation optimization. This maximizes the coverage efficiency of each module and the rationality of overlapping areas. The pitch angle of each module is locked via a customized angle adjustment bracket made of high-strength engineering plastic, possessing anti-aging and vibration-resistant properties, ensuring long-term stability of the pitch angle and preventing angle shifts due to environmental vibrations and temperature changes. Simultaneously, the customized angle adjustment bracket supports manual fine-tuning, allowing adjustments based on terrain differences and the distribution characteristics of monitored targets in the actual installation scenario. Precise fine-tuning is performed within the specified range to ensure a high degree of match between the coverage area and actual monitoring needs. Specifically, the pitch angle ranges of the three types of high-definition camera modules form a gradient distribution, with the telephoto high-definition camera module 113 positioned below the horizontal plane. The large tilt angle enables long-distance, wide-range coverage, and the mid-range high-definition camera module 112 is positioned below the horizontal plane. The tilt angle connects mid-range scenes, and the short-focus high-definition camera module 111 is below the horizontal plane. The small pitch angle focuses on close-range precise detection, and the three work together to form a vertical fan-shaped coverage array with no blind spots and hierarchical structure, which greatly improves the spatial coverage efficiency and target acquisition capability of the entire multimodal monitoring integrated machine.

[0024] Among them, the short-throw high-definition camera module 111 is equipped with a small-angle short-throw lens, with a wide field of view. It is designed for close-range monitoring scenarios, capturing high-definition visible light video within the initial coverage area (0-150 meters from the multimodal monitoring unit as the origin). It can clearly capture detailed features of targets at close range, such as vehicle license plates, facial features of individuals, and equipment operating status, meeting the requirements for high-precision close-range identification. Its imaging resolution... Frame rate This ensures smooth and clear video footage, and is suitable for rapid target capture in complex scenarios.

[0025] The 112 high-definition video module with medium telephoto range is equipped with a medium-angle lens, with a field of view between Designed for mid-range monitoring scenarios, this module collects high-definition visible light video within a second coverage area (100-300 meters from the multimodal monitoring unit), focusing on tracking and monitoring the behavior of mid-range targets, such as vehicle trajectories, personnel movement paths, and abnormal object movements within the area. The module supports fixed-focus or dynamic zoom adjustment, with varying dynamic zoom magnification. The focal length can be flexibly adjusted according to monitoring needs, balancing coverage and recognition accuracy, while maintaining the same imaging resolution. This ensures the recognizability of target features in mid-range scenarios.

[0026] The 113 telephoto high-definition camera module is equipped with a wide field of view lens, with a wide field of view... This module is designed for long-distance monitoring scenarios, collecting high-definition visible light video over a third coverage area (250-500 meters from the multimodal monitoring unit). It primarily enables the detection and identification of distant targets, such as vehicle traffic on distant roads, rockfall hazards on mountain slopes, and the operational status of remote equipment in industrial parks. The module boasts ultra-high-definition imaging capabilities and a high resolution. Equipped with image stabilization technology and low-light enhancement algorithm, it can effectively suppress image blur and noise even in low-light or long-distance transmission scenarios, ensuring that the target outline and key features are clearly distinguishable.

[0027] Among them, the first coverage area, the second coverage area, and the third coverage area all have reasonable overlapping areas, and the width of the overlapping area is... By cross-validating image data from overlapping areas, the system provides data support for subsequent trajectory stitching and target association by the AI ​​video processing terminal 5, preventing target loss due to distance switching. Furthermore, using the multimodal monitoring all-in-one machine as the origin, the first, second, and third coverage areas are sequentially moved away from the origin, forming a continuous and seamless dynamic monitoring coverage network from 0 to 500 meters. This achieves full-scene coverage from close-range detail capture to long-range large-scale monitoring, meeting the monitoring needs at different distances in complex environments such as highways, mountain roads, and industrial parks.

[0028] In addition, the short-throw high-definition camera module, medium-throw high-definition camera module, long-throw high-definition camera module, and infrared thermal imaging module are connected to the motherboard via standardized slots, supporting high-speed data transmission and plug-and-play replacement, reducing the time required for single-module installation and removal. It takes only minutes, improving disassembly and assembly efficiency.

[0029] In one possible implementation, refer to Figure 3 As shown, the vertical array optical subsystem 1 may also include optical windows 13, which are respectively wrapped around the periphery of the short-focal-length high-definition camera module 111, the medium-focal-length high-definition camera module 112, the long-focal-length high-definition camera module 113 and the infrared thermal imaging module 12.

[0030] Optionally, the optical window 13 provided in this embodiment is used to enclose the periphery of the short-focal-length high-definition camera module 111, the medium-focal-length high-definition camera module 112, the long-focal-length high-definition camera module 113, and the infrared thermal imaging module 12, forming an independent protective structure of "one module, one window". Each optical window is made of high-transmittance optical glass material, with a transmittance of... Furthermore, it undergoes anti-reflective coating treatment, effectively reducing light reflection and refraction loss, ensuring that image quality is not affected. Window glass thickness. It has a certain impact resistance and can withstand the impact of external objects such as wind, sand and gravel, protecting the internal lens from damage.

[0031] In this embodiment, the optical window and each module lens are seamlessly sealed with sealing rings, and the individual sealing level reaches IP67 standard, which can effectively prevent dust, rain, fog and other impurities from entering the lens and ensure the stable operation of each module in harsh environments.

[0032] Meanwhile, the multimodal monitoring all-in-one machine may also include a nano-hydrophobic coating and a defogging heating film. This coating, possessing superhydrophobic properties, is applied to the outer surface of each optical window, resulting in a contact angle... This allows rainwater and dew to slide off quickly, preventing the formation of a water film on the window surface that could affect imaging. An anti-fogging heating film is attached to the inside of each optical window. This film is made of a low-power, flexible material. When the temperature and humidity sensor detects that the surface temperature of the optical window is lower than the dew point temperature or that fog is present, the heating function is automatically activated to quickly remove fog and frost, ensuring clear imaging all day long.

[0033] In addition, the multimodal monitoring unit may also include an anti-glare shielding mechanism, which is fixedly installed on top of the active protection housing.

[0034] The embodiments of this application combine an optical window, an anti-glare shielding mechanism, a defogging heating film, and a nano-hydrophobic coating to form a triple protection system of "anti-fog-hydrophobic-anti-glare", which effectively avoids the interference of rain, snow, fog, and strong light on the imaging effect.

[0035] Furthermore, in this embodiment, a visible light and infrared light anti-crosstalk partition is provided between adjacent optical windows. The anti-crosstalk partition is made of engineering plastic with excellent light-shielding performance and has a matte black finish. This effectively blocks light crosstalk between different modules, preventing interference between the visible light from the short-focus high-definition camera module, the medium-focus high-definition camera module, and the long-focus high-definition camera module, and the infrared light from the infrared thermal imaging module. This ensures the independence and purity of each module's imaging, providing high-quality raw image data for subsequent multimodal feature fusion. The optical window is seamlessly integrated with the front surface of the active protective shell, and the panel uses an integrated molding process, further improving the sealing and protection performance and structural stability of the multimodal monitoring integrated machine.

[0036] The adaptive geometric calibration mechanism 3 is respectively set on the base of the high-definition camera module 11 and the infrared thermal imaging module 12 to form a modular calibration unit, which is used to calibrate the angle changes of the high-definition camera module 11 and the infrared thermal imaging module 12.

[0037] The adaptive geometry calibration mechanism 3 includes a micro-stepping calibration device, possessing three degrees of freedom adjustment capabilities: pitch, yaw, and roll. Its core components integrate a MEMS accelerometer and a high-precision angle sensor, enabling real-time detection of micro-vibrations, parallax drift, and angle changes in the corresponding module. When a module drift angle exceeds [a certain value], [the mechanism will take action]. At the same time, and the power consumption of the entire calibration process. Adjustment precision This technology enables automatic alignment and long-term stable calibration of the optical axes of multiple modules, maintaining array geometric consistency without manual intervention and significantly reducing maintenance costs. Simultaneously, calibration data is fed back to the AI ​​video processing terminal in real time, providing fundamental parameters for subsequent image optimization.

[0038] The self-sensing attitude compensation base 4 is fixedly installed at the lower end of the active protective shell 2 and is used to adjust the attitude changes of the vertical array optical subsystem 1.

[0039] Optionally, when the attitude change of the multimodal monitoring all-in-one machine exceeds At this time, the micro-balancing actuator built into the self-sensing attitude compensation base 4 automatically starts. Through the coordinated work of the pitch compensation adjustment structure and the horizontal compensation adjustment structure, the overall attitude of the multi-modal monitoring all-in-one machine is dynamically adjusted to ensure that the vertical array optical subsystem 1 always maintains the optimal monitoring angle. At the same time, the attitude change data is fed back to the host computer and AI video processing terminal 5 in real time through the high-speed communication interface for image geometric correction and online calibration, ensuring that the monitoring screen is consistent with the ground plane and avoiding monitoring coverage errors or target recognition deviations caused by attitude deviation.

[0040] In one possible implementation, a gyroscope and an electronic level are integrated within the self-sensing attitude compensation base, and the gyroscope and electronic level work together. Gyroscopes are used to monitor the dynamic attitude changes of the multimodal monitoring all-in-one machine in real time. Electronic level is used to monitor the horizontal status and tilt changes of the multimodal monitoring all-in-one machine in real time.

[0041] Optionally, the self-sensing attitude compensation base integrates a high-precision gyroscope and an industrial-grade electronic level. The two establish a real-time communication link through the core control unit to form a collaborative sensing system, jointly realizing comprehensive and high-precision monitoring of the attitude and level status of the multi-modal monitoring all-in-one machine, providing accurate and reliable data support for subsequent attitude compensation adjustments.

[0042] The gyroscope used is a high-precision MEMS gyroscope, characterized by low drift and high response speed, with an angular rate measurement range reaching [missing information]. Measurement accuracy It can capture the dynamic attitude changes of a multimodal monitoring device in three-dimensional space, including instantaneous dynamic shifts in pitch, yaw, and roll directions caused by wind vibration, vehicle traffic vibration, and slight pole swaying. This gyroscope uses high-frequency data sampling (sampling frequency...) It converts dynamic attitude change data into digital signals and transmits them to the core control unit in real time, ensuring zero-delay perception of attitude changes of the multimodal monitoring all-in-one machine. In particular, it can accurately identify minute vibrations and rapid attitude fluctuations in a short period of time, avoiding the problem of untimely compensation due to monitoring lag.

[0043] The electronic level uses a dual-axis high-precision electronic level with a measurement range of [missing information]. Inclination measurement accuracy can reach Featuring strong resistance to electromagnetic interference and high temperature stability, this electronic level is specifically designed for real-time monitoring of the horizontal state and tilt changes of a multimodal monitoring unit. It senses changes in gravity through a built-in capacitive sensor, accurately outputting the tilt angle data of the multimodal monitoring unit in the horizontal direction, including tilt deviations along the X-axis (left-right) and Y-axis (front-back). This effectively identifies slow-change attitude problems such as tilting caused by foundation settlement, long-term pole deformation, and strong winds. Furthermore, the electronic level has an adaptive temperature compensation function. It corrects the effect of temperature on measurement accuracy over a wide temperature range, ensuring measurement accuracy in complex environments.

[0044] During collaborative operation, the gyroscope focuses on capturing the dynamic attitude changes and instantaneous vibrations of the multimodal monitoring unit, while the electronic level focuses on monitoring the static levelness and slow tilt deviations of the multimodal monitoring unit. The data from both are fused and processed by the core control unit. The core control unit performs time synchronization calibration and cross-validation on the monitoring data from both types of sensors, eliminates abnormal interference data, and generates a complete monitoring result of the multimodal monitoring unit's attitude. This result accurately reflects the instantaneous dynamic fluctuations of the multimodal monitoring unit and accurately captures long-term accumulated tilt deviations, providing a comprehensive and reliable basis for subsequent attitude compensation decisions. This ensures that the self-sensing attitude compensation base can take precise and effective compensation measures for different types of attitude changes.

[0045] The AI ​​video processing terminal 5 is communicatively connected to the vertical array optical subsystem 1, and is used for: Acquire environmental parameters, including light intensity, visibility, weather type, and image signal-to-noise ratio; Based on environmental parameters, the visible light weight of the high-definition camera module 11 and the infrared weight of the infrared thermal imaging module 12 are dynamically adjusted. The system acquires visible light video captured by a high-definition camera module 11 with dynamically adjusted visible light and infrared weights and infrared video captured by an infrared thermal imaging module 12, and extracts visible light features from the visible light video and infrared features from the infrared video. Visible light and infrared features are interacted and fused through a cross-modal attention mechanism to generate an enhanced feature map for status monitoring and early warning.

[0046] In this embodiment, the AI ​​video processing terminal 5 establishes a high-speed communication connection with the vertical array optical subsystem 1 through a standardized slot, supporting the real-time transmission and processing of massive video data. Its core functions include: 1) Environmental parameter collection: The multimodal monitoring unit uses built-in temperature and humidity sensors, light sensors, visibility detectors, and image quality analysis modules to acquire environmental parameters in real time. These parameters include light intensity, visibility, weather type (sunny, rainy, snowy, foggy, etc.) and image signal-to-noise ratio, providing data support for subsequent weight adjustments.

[0047] 2) Dynamic weight adjustment: Based on the collected environmental parameters, the system performs intelligent analysis using a built-in adaptive weighting algorithm to dynamically allocate the visible light weight of the high-definition camera module and the infrared weight of the infrared thermal imaging module. For example, the infrared weight is increased in bright light environments to suppress glare, the infrared weight is enhanced at night or in low visibility environments to improve target recognition, and the visible light weight is optimized during clear daytime to preserve image details, ensuring optimal imaging results in different environments.

[0048] 3) Feature extraction: Through a dedicated image acquisition interface, visible light video captured by a high-definition camera module with dynamically adjusted visible light and infrared weights, and infrared video captured by an infrared thermal imaging module are acquired simultaneously. Using a parallel processing architecture composed of visible light feature extraction branches and infrared feature extraction branches, visible light features such as texture, color, and contour are extracted from the visible light video, and infrared features such as temperature distribution and thermal radiation intensity are extracted from the infrared video. The extraction process adopts an optimized convolutional neural network model, which balances feature extraction accuracy and processing speed.

[0049] 4) Feature fusion and application: By employing a cross-modal attention mechanism to deeply interact and fuse extracted visible light and infrared features, the algorithm focuses on enhancing the feature information of the target area, suppressing background interference, and generating a unified enhanced feature map. This enhanced feature map is then further input into a spatiotemporal trajectory stitching algorithm, a target re-identification algorithm, an anomaly detection network, and a disaster identification algorithm to achieve multi-dimensional status monitoring and early warning functions. Specifically, the spatiotemporal trajectory stitching algorithm, based on time synchronization and spatial coordinate transformation, achieves seamless stitching of target trajectories under different focal lengths and modalities; the target re-identification algorithm combines target appearance features and infrared features to generate a global identity identifier, enabling cross-scene target tracking; the anomaly detection network can identify suspicious behaviors, violations, and other abnormal situations; and the disaster identification algorithm is specifically used to detect disasters such as road flooding, icing, fires, and rockfalls, outputting tiered alarm results. Ultimately, this achieves integrated traffic event and disaster monitoring, providing users with comprehensive and accurate monitoring and early warning services.

[0050] In one possible implementation, when multiple monitoring points are set up on the target road segment, and each monitoring point is equipped with a multimodal monitoring device; the AI ​​video processing terminal can also be used for: Multi-dimensional features are extracted from multiple enhanced feature maps, including license plate character features, vehicle model outline features, and infrared heat source shape features. For each enhanced feature map, multi-dimensional features of all vehicles in that enhanced feature map are extracted. These multi-dimensional features include license plate character features, vehicle model outline features, and infrared heat source shape features. Based on the multi-dimensional features of each vehicle in each enhanced feature map, determine whether there is a target vehicle on the target road segment. The target vehicle can be any vehicle on the target road segment. If a target vehicle exists on the target road segment, the trajectory of the target vehicle in multiple enhanced feature maps is linked according to the order of monitoring points to obtain the target vehicle's trajectory chain from far to near the target road segment.

[0051] Among them, reference Figure 4 As shown, when multiple monitoring points are set up on the target road section ( Figure 4 The image shows n monitoring points, each equipped with a multimodal monitoring unit. Each multimodal monitoring unit uses a star topology network (with the road section monitoring center as the central node), supporting wired network or 4G / 5G wireless connectivity (transmission latency...). ).

[0052] Optionally, the AI ​​video processing terminal also has the function of accurate target vehicle identification and trajectory chain construction, specifically: Multi-length enhanced feature maps generated by multimodal feature fusion are deeply analyzed using a dedicated feature extraction network to extract multi-dimensional features from each enhanced feature map. These multi-dimensional features include license plate character features, vehicle model contour features, and infrared heat source shape features. License plate character feature extraction employs a character segmentation and recognition algorithm based on convolutional neural networks to accurately capture the character texture, arrangement order, and font features of the license plate area, exhibiting robustness against tilt, blurring, and changes in lighting conditions. Vehicle model contour feature extraction utilizes edge detection, morphological processing, and feature point matching techniques to obtain key features such as vehicle body size, contour curves, window layout, and wheel style, enabling accurate differentiation between different vehicle models. Infrared heat source shape feature extraction uses thermal radiation data collected by the infrared thermal imaging module to extract the distribution pattern, temperature gradient, and contour boundary features of core heat sources such as the vehicle's engine and exhaust system, ensuring stable vehicle target identification even in low visibility or occluded scenarios.

[0053] After feature extraction, the AI ​​video processing terminal uses a multi-dimensional feature matching algorithm to determine whether the same target vehicle exists on the target road segment (the target vehicle can be any vehicle on the target road segment). During the matching process, a feature similarity calculation model is established to perform accurate cross-modal and cross-frame comparisons of license plate character features, vehicle model outline features, and infrared heat source shape features in different enhanced feature maps. Combined with timestamp information and spatial coordinate association, the risk of mismatch caused by environmental interference and similar targets is eliminated, ensuring the accuracy of target vehicle identification.

[0054] If, after multiple rounds of feature matching verification, it is determined that the same target vehicle exists on the target road segment, the AI ​​video processing terminal calls the spatiotemporal trajectory stitching algorithm, according to the order of monitoring points and GPS timestamps (with an accuracy of up to...). This method seamlessly connects the trajectory of the same target vehicle from multiple enhanced feature maps. During the connection process, based on time synchronization calibration and spatial coordinate transformation technology, it corrects trajectory deviations under different focal length modules and different monitoring distances, generating a continuous and complete target vehicle trajectory chain from far to near distance, clearly presenting the entire driving path of the target vehicle within the 0-500 meter monitoring range.

[0055] In one possible implementation, the AI ​​video processing terminal can also be used for: For the enhanced feature maps collected by the multimodal monitoring integrated machine at each monitoring point, the following steps are performed: If the confidence level of the matching between the license plate character features and the vehicle model outline features within the first distance is not less than the first preset confidence level, then the vehicle within the first distance is determined to be the target vehicle. The direction of the first distance is from the initial position of the monitoring point in the detection area of ​​the target road segment to the target position of the monitoring point in the detection area of ​​the target road segment. If the confidence level of matching the vehicle outline features and infrared heat source shape features within the second distance is not less than the second preset confidence level, then the vehicle within the second distance is determined to be the target vehicle. The direction of the second distance is from the target position of the monitoring point in the detection area of ​​the target road segment to the end position of the monitoring point in the detection area of ​​the target road segment, and the first preset confidence level is greater than the second preset confidence level.

[0056] Optionally, the AI ​​video processing terminal in this embodiment can also adopt a segmented feature matching strategy to further improve the accuracy and adaptability of target vehicle recognition. For the enhanced feature map collected by the multimodal monitoring all-in-one machine at each monitoring point, the following steps are performed: To address the imaging characteristics of different distance ranges within the target road segment, differentiated feature matching standards and confidence thresholds are set. The first distance is defined as the distance from the initial position (0-100 meters) to the midpoint (100-300 meters) within the detection area of ​​the target road segment. Within this distance, vehicles are imaged relatively close, and license plate characters are relatively clear, but vehicle model details may be somewhat lost. Therefore, the focus is on combining license plate character features with vehicle model contour features for matching. A comprehensive matching confidence score is obtained by calculating the similarity of license plate characters (including character consistency and order matching) and the structural similarity of the vehicle model contour between two enhanced feature images. If this confidence score is not less than a first preset confidence score (the first preset confidence score ranges from 0.85 to 0.95, with a default value of 0.9), then vehicles within the first distance on the target road segment are determined to be the target vehicles.

[0057] The second distance is defined as the distance from the midpoint (100-300 meters, i.e., the target location) to the farthest point (300-500 meters, i.e., the endpoint) within the detection area of ​​the target road segment. Within this distance, the vehicle imaging distance is relatively far, and the vehicle outline details are rich, but issues such as license plate obstruction and angle tilt may exist. Therefore, the focus is on combining and matching vehicle outline features with infrared heat source shape features. By comparing key feature points of the vehicle outline, vehicle proportions, and the distribution and temperature characteristics of the infrared heat source, a secondary comprehensive matching confidence level is obtained. If this confidence level is not less than the second preset confidence level (the second preset confidence level ranges from 0.75 to 0.85, with a default value of 0.8), then the vehicle on the target road segment within the second distance is determined to be the target vehicle. The first preset confidence level is greater than the second preset confidence level. This setting conforms to the imaging quality characteristics of different distance ranges, ensuring both the accuracy of long-distance recognition and the flexibility of short-distance recognition.

[0058] In one possible implementation, when there is a first vehicle in front of the target vehicle, the AI ​​video processing terminal can also be used for: The target vehicle's trajectory chain from the far distance to the near distance of the target road segment is input into the constructed vehicle path prediction model, and the predicted path of the target vehicle is output. The vehicle path prediction model is constructed based on a long short-term memory network. If the overlap rate between the predicted path of the target vehicle and the target trajectory chain of the first vehicle is not less than a preset percentage, and the relative speed difference between the target vehicle and the first vehicle is greater than a preset speed, then it is determined that there is a risk of rear-end collision between the target vehicle and the first vehicle, and a level one warning is triggered. If the predicted path of the target vehicle enters the no-parking zone and the stay time is not less than the preset time, the target vehicle is determined to have a risk of illegal parking, triggering a level 2 warning; If the predicted path of the target vehicle deviates from the lane by a distance not less than the preset distance, and the target vehicle has no turn signal, then it is determined that the target vehicle has a risk of lane departure, triggering a level three warning.

[0059] Optionally, the AI ​​video processing terminal can also have the function of predicting and classifying vehicle operation risks, specifically: First, the constructed target trajectory chain is input into a well-trained vehicle path prediction model, which outputs the predicted path of the target vehicle for the next 5-10 seconds. The vehicle path prediction model is built on a Long Short-Term Memory (LSTM) network and is trained and optimized through a large amount of historical vehicle trajectory data. It can learn the speed change patterns, steering habits, and road constraints of the vehicle, and accurately predict the vehicle's subsequent driving path.

[0060] Based on predicted paths and real-time monitoring data, multi-dimensional risk assessment is conducted: 1) Rear-end Collision Risk Warning: Real-time extraction of the target vehicle's trajectory chain (the vehicle adjacent to and ahead of the target vehicle) on the target road segment, calculation of the spatial overlap rate between the target vehicle's predicted path and the first vehicle's trajectory chain; simultaneously, through video frame rate analysis and pixel distance conversion, the real-time relative speed difference between the target vehicle and the first vehicle is obtained. If the overlap rate is not less than a preset percentage (range 60%-80%, default 70%), and the relative speed difference is greater than a preset speed (range 10km / h-20km / h, default 15km / h), it indicates a risk of rear-end collision between the two vehicles. The AI ​​video processing terminal immediately triggers a Level 1 warning, sending warning information to the monitoring center via the 4 / 5G remote communication module, including key data such as the positions, speeds, and trajectories of the two vehicles, while simultaneously activating the device's local working indicator light to flash as a reminder.

[0061] 2) Illegal Parking Risk Warning: The coordinate range of the no-parking zone within the preset target road segment (which can be customized by the user through the host computer) is used to monitor in real time whether the predicted path of the target vehicle enters the no-parking zone. If the predicted path falls into the no-parking zone, and the vehicle stay time is determined to be no less than the preset time (the value range is 30 seconds to 60 seconds, and the default value is 45 seconds) based on historical trajectory data, the target vehicle is determined to have a risk of illegal parking, triggering a level 2 warning. The warning information includes the location of the no-parking zone, the duration of the vehicle stay, and vehicle characteristic information.

[0062] 3) Lane Departure Risk Warning: The lane coordinates of the target road segment are extracted using image segmentation technology, and the offset distance between the predicted path of the target vehicle and the center line of the current driving lane is calculated in real time. If the offset distance is not less than the preset distance (the value range is 0.5 meters to 1 meter, and the default value is 0.8 meters), and the video frame analysis confirms that the target vehicle has not activated the turn signal, it is determined that the target vehicle has an unintentional lane departure risk, triggering a level 3 warning to remind monitoring personnel to pay attention in time.

[0063] In addition, all levels of early warning information in this application embodiment are traceable. The AI ​​video processing terminal automatically stores the enhanced feature map, trajectory data and environmental parameters when the early warning is triggered, providing a complete basis for subsequent event verification.

[0064] In one possible implementation, the AI ​​video processing terminal can also be used for: Extract infrared temperature features and visible light texture features from the enhanced feature map; Infrared temperature features and visible light texture features are input into the constructed disaster type identification model, which outputs the disaster type, location, and disaster level of the target road segment. The disaster type identification model is constructed based on a convolutional neural network combined with a self-attention mechanism. The alarm strategy is triggered based on the type, location, and severity of the disaster on the target road segment.

[0065] Optionally, the AI ​​video processing terminal can also integrate intelligent disaster identification and tiered alarm functions to achieve integrated monitoring of traffic incidents and disasters, specifically: The enhanced feature map generated by multimodal fusion is used to simultaneously extract infrared temperature features and visible light texture features through a dual-branch feature extraction network. Specifically, the infrared temperature feature extraction module analyzes infrared thermal imaging data to obtain the temperature distribution matrix of the monitored area, the area of ​​high-temperature anomalies, temperature peaks, and temperature gradient changes, accurately capturing temperature-related disaster features such as fires and high-temperature hazards. The visible light texture feature extraction module uses edge detection, texture analysis, and region segmentation techniques to extract features such as road surface smoothness, color changes, and foreign object outlines, identifying visually distinguishable disaster features such as water accumulation, icing, rockfalls, and obstacles.

[0066] After unifying and normalizing the extracted infrared temperature features and visible light texture features, they are input into a pre-constructed disaster type identification model. This model is built on a Convolutional Neural Network (CNN) combined with a self-attention mechanism and has been trained and optimized using massive disaster scene samples (including data on water accumulation, icing, fire, rockfall, etc., under different weather conditions, lighting conditions, and distances). It possesses powerful feature learning and pattern recognition capabilities. The model output includes the disaster type of the target road segment (specifically categorized as water accumulation, icing, fire, rockfall, obstacles, etc.), precise location coordinates (calculated based on spatial calibration parameters of monitoring equipment), and disaster level (divided into Level 1, Level 2, and Level 3 according to the scope of impact and degree of danger, with Level 1 being the most severe). For example, large-area water accumulation (scope of impact... square meters), large fire (high temperature area) A disaster affecting an area of ​​3-10 square meters is classified as a Level 1 disaster; small-area icing (affected area 3-10 square meters) and small rockfalls (diameter...) are also considered Level 1 disasters. The disaster was classified as a Level II disaster; sporadic water accumulation (affected area) square meters), small obstacles (height) It was determined to be a level three disaster.

[0067] Based on the type, location, and severity of the disaster on the target road segment, the AI ​​video processing terminal automatically triggers the corresponding alarm strategy: For Level 1 disasters, the highest level alarm is immediately activated, and alarm information, including real-time disaster footage, location coordinates, severity assessment, and response suggestions, is simultaneously sent to the monitoring center, traffic management department, and emergency rescue department via a 4 / 5G remote communication module. This also activates the device's local audible and visual alarm. For Level 2 disasters, alarm information is sent to the monitoring center and traffic management department, and the local indicator light flashes continuously to warn passing vehicles to avoid the area. For Level 3 disasters, a warning message is sent to the monitoring center, the disaster situation is recorded, and continuous monitoring is conducted. If the disaster area expands or the severity escalates, the alarm level is automatically upgraded. Alarm information and disaster image data are automatically stored in the local storage unit and simultaneously uploaded to the cloud server, ensuring no data loss and providing complete data support for disaster response and subsequent analysis.

[0068] This application provides a multimodal monitoring all-in-one device, which integrates a vertical array optical subsystem, an active protective housing, an adaptive geometric calibration mechanism, a self-sensing attitude compensation base, and an AI video processing terminal into one unit. The adaptive geometric calibration mechanism and the AI ​​video processing terminal are housed within the active protective housing, with the vertical array optical subsystem embedded within it. This compact design significantly saves installation space, facilitating deployment in various confined spaces. Furthermore, the AI ​​video processing terminal can acquire environmental parameters, including light intensity, visibility, weather type, and image signal-to-noise ratio, and dynamically adjust the visible light weight of the high-definition camera module and the infrared weight of the infrared thermal imaging module based on these parameters. This intelligent weight adjustment method... The AI ​​video processing terminal can automatically optimize the contributions of the two imaging modes based on different environmental conditions. It increases the weight of visible light when there is sufficient light and increases the weight of infrared light when there is no light at night or in low visibility environments. This allows it to acquire high-quality video images in different environments, improving the adaptability and reliability of monitoring. At the same time, the AI ​​video processing terminal can extract visible light features from visible light videos captured by high-definition camera modules and infrared features from infrared videos captured by infrared thermal imaging modules. It then interacts and fuses these two types of features through a cross-modal attention mechanism to generate an enhanced feature map. This cross-modal feature fusion method can make full use of the complementary information of visible light and infrared images, improve the expressive power and discriminative power of features, and thus more accurately monitor the status.

[0069] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0070] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multimodal monitoring integrated machine, characterized in that, It includes a vertical array optical subsystem, an active protective shell, an adaptive geometric calibration mechanism, a self-sensing attitude compensation base, and an AI video processing terminal; The active protection housing is a longitudinally narrow box-shaped structure, and both the adaptive geometric calibration mechanism and the AI ​​video processing terminal are housed within the active protection housing. The vertical array optical subsystem is embedded in the active protective housing. The vertical array optical subsystem includes a high-definition camera module and an infrared thermal imaging module that are connected in communication. The high-definition camera module and the infrared thermal imaging module are arranged in a vertical array, and the infrared thermal imaging module is arranged at the center of the vertical array optical subsystem in the vertical direction. The adaptive geometric calibration mechanism is respectively mounted on the base of the high-definition camera module and the infrared thermal imaging module, and is used to calibrate the angle changes of the high-definition camera module and the infrared thermal imaging module; The self-sensing attitude compensation base is fixedly installed at the lower end of the active protective shell and is used to adjust the attitude changes of the vertical array optical subsystem. The AI ​​video processing terminal is communicatively connected to the vertical array optical subsystem and is used for: Acquire environmental parameters, including light intensity, visibility, weather type, and image signal-to-noise ratio; Based on the environmental parameters, the visible light weight of the high-definition camera module and the infrared weight of the infrared thermal imaging module are dynamically adjusted. The system acquires visible light video captured by the high-definition camera module and infrared video captured by the infrared thermal imaging module with dynamically adjusted visible light and infrared weights, and extracts visible light features from the visible light video and infrared features from the infrared video. The visible light features and the infrared features are interacted and fused through a cross-modal attention mechanism to generate an enhanced feature map for status monitoring and early warning.

2. The multimodal monitoring integrated machine according to claim 1, characterized in that, The high-definition camera module includes a short-focus high-definition camera module, a medium-focus high-definition camera module, and a long-focus high-definition camera module. The long-focus high-definition camera module, the medium-focus high-definition camera module, and the short-focus high-definition camera module are arranged in a vertical array in the vertical direction and in a vertical fan-shaped pitch arrangement. The short-focus high-definition camera module is used to monitor visible light video within the first coverage area; The high-definition camera module with a central focus is used to monitor visible light video within the second coverage area; The telephoto high-definition camera module is used to monitor visible light video within the third coverage area; The first coverage area, the second coverage area, and the third coverage area all have overlapping regions, and with the multimodal monitoring integrated machine as the origin, the first coverage area, the second coverage area, and the third coverage area are sequentially moved away from the origin.

3. The multimodal monitoring integrated machine according to claim 2, characterized in that, With the horizontal plane as a reference, the telephoto high-definition camera module, the mid-range high-definition camera module, the infrared thermal imaging module, and the short-focus high-definition camera module each operate at a distance lower than the horizontal plane. , , and The pitch angle range is arranged in a fan shape.

4. The multimodal monitoring integrated machine according to claim 2, characterized in that, The vertical array optical subsystem also includes optical windows that surround the short-focal-length high-definition camera module, the medium-focal-length high-definition camera module, the long-focal-length high-definition camera module, and the infrared thermal imaging module, respectively.

5. The multimodal monitoring integrated machine according to claim 1, characterized in that, The self-sensing attitude compensation base integrates a gyroscope and an electronic level, which work together. The gyroscope is used to monitor the dynamic changes in attitude of the multimodal monitoring device in real time. The electronic level is used to monitor the horizontal status and tilt angle changes of the multimodal monitoring integrated machine in real time.

6. The multimodal monitoring integrated machine according to claim 4, characterized in that, The multimodal monitoring integrated machine also includes an anti-glare shielding mechanism, a defogging heating film, and a nano-hydrophobic coating; The anti-glare shielding mechanism is fixedly installed above the active protective shell; The defogging heating film is disposed on the optical windows of the short-focus high-definition camera module, the medium-focus high-definition camera module, the long-focus high-definition camera module, and the infrared thermal imaging module; The nano-hydrophobic coating is disposed on the optical windows of the short-focus high-definition camera module, the medium-focus high-definition camera module, the long-focus high-definition camera module, and the infrared thermal imaging module.

7. The multimodal monitoring integrated machine according to claim 1, characterized in that, When multiple monitoring points are set up on the target road section, and each monitoring point is equipped with a multimodal monitoring integrated machine; the AI ​​video processing terminal is also used for: For each enhanced feature map, multi-dimensional features of all vehicles in the enhanced feature map are extracted. The multi-dimensional features include license plate character features, vehicle model outline features, and infrared heat source shape features. Based on the multi-dimensional features of each vehicle in each enhanced feature map, it is determined whether there is a target vehicle on the target road segment, where the target vehicle can be any vehicle on the target road segment; If the target vehicle exists on the target road segment, the running trajectory of the target vehicle in multiple enhanced feature maps is linked according to the monitoring point order to obtain the target vehicle's running trajectory chain from far to near the target road segment.

8. The multimodal monitoring integrated machine according to claim 7, characterized in that, The AI ​​video processing terminal is also used for: For the enhanced feature maps collected by the multimodal monitoring integrated machine at each monitoring point, the following steps are performed: If the confidence level of the matching between the license plate character features and the vehicle model outline features within the first distance is not less than the first preset confidence level, then the vehicle within the first distance is determined to be the target vehicle, and the direction of the first distance is from the initial position of the monitoring point in the detection area of ​​the target road segment to the target position of the monitoring point in the detection area of ​​the target road segment; If the confidence level of the matching between the vehicle profile feature and the infrared heat source shape feature within the second distance is not less than the second preset confidence level, then the vehicle within the second distance is determined to be the target vehicle. The direction of the second distance is from the target position of the monitoring point in the detection area of ​​the target road segment to the end position of the monitoring point in the detection area of ​​the target road segment, and the first preset confidence level is greater than the second preset confidence level.

9. The multimodal monitoring integrated machine according to claim 8, characterized in that, When a first vehicle is present in front of the target vehicle, the AI ​​video processing terminal is also used for: The target vehicle's trajectory chain from the far distance to the near distance of the target road segment is input into the constructed vehicle path prediction model, and the predicted path of the target vehicle is output. The vehicle path prediction model is constructed based on a long short-term memory network. If the overlap rate between the predicted path of the target vehicle and the target trajectory chain of the first vehicle is not less than a preset percentage, and the relative speed difference between the target vehicle and the first vehicle is greater than a preset speed, then it is determined that there is a risk of rear-end collision between the target vehicle and the first vehicle, and a level one warning is triggered. If the predicted path of the target vehicle enters the no-parking zone and the stay time is not less than the preset time, it is determined that the target vehicle has a risk of illegal parking and a level 2 warning is triggered. If the predicted path of the target vehicle deviates from the lane by a distance not less than a preset distance, and the target vehicle has no turn signal, then it is determined that the target vehicle has a lane departure risk, triggering a level three warning.

10. The multimodal monitoring integrated machine according to claim 1, characterized in that, The AI ​​video processing terminal is also used for: Extract the infrared temperature features and visible light texture features from the enhanced feature map; The infrared temperature features and the visible light texture features are input into the constructed disaster type identification model, which outputs the disaster type, location, and disaster level of the target road segment. The disaster type identification model is constructed based on a convolutional neural network combined with a self-attention mechanism. An alarm strategy is triggered based on the type, location, and severity of the disaster on the target road segment.

Images