IoT smart hardware security vision system
By combining scene structure invariants and image light entropy perception in the IoT security vision system, online self-calibration of the camera and inertial measurement unit was achieved, solving the positioning drift problem of low-cost security equipment in weak texture and low light scenes, reducing operation and maintenance costs and improving positioning robustness.
Patent Information
- Application Number
- CN202611249950.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-18
- Publication Date
- 2026-11-17
AI Technical Summary
When low-cost IoT security vision hardware operates for extended periods in complex scenarios with weak textures and low light conditions where there are no satellite navigation signals, the insufficient feature extraction capability of monocular cameras leads to visual odometry positioning drift. Furthermore, the external parameters and time synchronization between the camera and the inertial measurement unit undergo implicit drift due to factors such as equipment vibration and temperature changes. Existing solutions, by increasing hardware costs or relying on manual calibration, are insufficient to support the operation and maintenance needs of large-scale distributed nodes.
By combining scene structure invariant constraints with image optical entropy perception, online self-calibration of the camera and inertial measurement unit is achieved. Using the static structure line in the security scene as a zero-drift benchmark, a joint cost function is constructed for nonlinear optimization to achieve online self-calibration of the extrinsic parameter matrix and time offset. In the weak texture environment, the feature extraction mode is dynamically adjusted to compensate for positioning drift.
Without increasing hardware costs, online self-calibration of the spatiotemporal parameters of the camera and inertial measurement unit was achieved, reducing operation and maintenance costs, improving positioning robustness and feature availability, and making it suitable for the engineering deployment of low-power IoT security nodes.
Smart Images

Figure CN123123542A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of IoT intelligent visual perception technology, specifically to IoT intelligent hardware security visual systems. Background Technology
[0002] With the popularization of IoT technology, intelligent security hardware with autonomous perception and mobility capabilities (such as indoor inspection robots, security drones, and mobile monitoring vehicles) are being deployed in large numbers in complex security scenarios without satellite navigation signals, such as long corridors, underground garages, and low-light warehouses. In such scenarios, the devices rely heavily on visual real-time positioning and mapping technology and inertial measurement units to achieve their own pose estimation and environmental ranging. Visual inertial odometry (VIO) is one of the mainstream technical routes for low-cost mobile robots and drones to achieve autonomous positioning by fusing image feature observations collected by the camera with acceleration and angular velocity data measured by the inertial measurement unit and using nonlinear optimization methods to jointly estimate the device's pose, velocity, and sensor bias. In this technical route, the extrinsic parameter matrix between the camera coordinate system and the inertial measurement unit coordinate system, as well as the time offset between their data, are prerequisite parameters for ensuring the correct fusion of multi-sensor data. These parameters are usually obtained through offline calibration using a checkerboard calibration board before the device leaves the factory and are used as fixed constants during device operation.
[0003] The existing technical solutions that are closest to this invention mainly include the following two categories;
[0004] One is the vision and LiDAR fusion positioning scheme. This scheme uses LiDAR to obtain high-precision depth information of the environment to make up for the lack of feature sparsity in monocular vision in weak texture scenes. Its hardware structure usually includes LiDAR, monocular or binocular camera, inertial measurement unit and embedded processing unit. When working, LiDAR point cloud and visual features are used for environment mapping and pose estimation, respectively, and then the final positioning result is output through fusion algorithm.
[0005] The second is the traditional monocular visual inertial odometry scheme based on point features. Its typical processing flow includes: acquiring image frames and extracting corner features, performing cross-frame tracking or matching of corner features of adjacent frames, reading inertial measurement unit data and performing pre-integration processing, constructing a joint cost function by combining visual reprojection error and inertial measurement unit pre-integration error, solving the device pose through sliding window nonlinear optimization, and obtaining fixed values for the extrinsic parameter matrix and time offset between the camera and the inertial measurement unit through offline calibration board when the device leaves the factory, which remain unchanged during system operation and are not updated online.
[0006] The two existing technical solutions mentioned above have the following drawbacks. First, the vision-LiDAR fusion positioning scheme relies on LiDAR to acquire depth information to compensate for the lack of visual features in weakly textured scenes. This inevitably leads to a significant increase in hardware costs and power consumption, making it difficult to adapt to cost-sensitive low- to mid-range security IoT nodes that rely on solar or battery power and thus hindering large-scale distributed deployment. Second, the traditional monocular vision-inertial odometry scheme treats the extrinsic parameters and time offset between the camera and the inertial measurement unit as fixed constants after factory calibration. However, IoT security devices inevitably experience physical disturbances such as vibration, impact, and temperature changes during long-term operation. This causes the extrinsic parameters and time offset to drift implicitly over time, thus affecting the consistency between visual observation and inertial measurement. Unit observations cannot be accurately aligned in time and space, ultimately leading to the divergence of accumulated positioning errors and even pose estimation failure. Furthermore, due to the lack of online self-calibration capabilities in existing solutions, once external parameters or time offsets drift, on-site recalibration with a calibration board by professionals is required. This maintenance method inevitably results in high maintenance costs and untimely response when dealing with thousands of distributed security nodes, making it difficult to support large-scale engineering implementation. Finally, the number of corner features that can be extracted by low-cost monocular cameras in weak texture scenes such as white walls and low-light corridors is drastically reduced. As a result, traditional visual front-ends that rely on point feature matching are prone to tracking loss or mismatch, leading to rapid accumulation and divergence of positioning errors. To address these issues, we propose an IoT smart hardware security vision system. Summary of the Invention
[0007] The purpose of this invention is to provide an IoT smart hardware security vision system.
[0008] The technical problem this invention aims to solve is that when low-cost IoT security vision hardware operates for extended periods in complex scenes with weak textures and low light conditions without satellite navigation signals, the visual odometry drifts or even fails due to insufficient feature extraction capabilities of monocular cameras. Furthermore, long-term vibrations and temperature changes cause implicit drift in the external parameters and time synchronization between the camera and the inertial measurement unit. Existing solutions either rely on adding hardware such as LiDAR, increasing cost and power consumption, or on manual on-site calibration boards for recalibration, which is difficult to support the maintenance needs of large-scale distributed nodes. This invention aims to achieve online self-calibration of the spatiotemporal parameters of the camera and the inertial measurement unit without increasing hardware costs, through an algorithm combining scene structure invariant constraints and image optical entropy perception, and to solve the positioning drift problem caused by visual feature degradation in weak texture environments.
[0009] To address the aforementioned technical problems, this invention provides an IoT smart hardware security vision system, comprising a monocular RGB camera, a microelectromechanical inertial measurement unit, and an edge computing processing unit, as well as a sensor data acquisition and synchronization module, a structural invariant extraction module, an optical entropy perception adaptive feature extraction module, and a spatiotemporal online self-calibration backend module.
[0010] The sensor data acquisition and synchronization module is connected to the monocular RGB camera and the microelectromechanical inertial measurement unit. It is used to acquire image frame data and inertial measurement data, and write them into a timestamped synchronization buffer queue to provide a data basis for subsequent feature extraction and state estimation.
[0011] The structural invariant extraction module is connected to the sensor data acquisition and synchronization module. It is used to perform edge detection and line segment extraction on image frame data. According to the vanishing point clustering method, the extracted line segments are divided into multiple orthogonal direction clusters that conform to the Manhattan world hypothesis. Candidate structure lines with a length greater than a preset length threshold and a stable observation number of no less than a preset number threshold across consecutive frames are selected. The candidate structure lines are matched and tracked across frames. The candidate structure lines observed stably across multiple consecutive frames are initialized as three-dimensional structure line markers, which serve as zero-drift references that remain static and have constant geometric relationships in security scenarios.
[0012] The optical entropy perception adaptive feature extraction module is connected to the sensor data acquisition and synchronization module. It is used to calculate the local image information entropy of the image frame data and compare the local image information entropy with the preset information entropy threshold. When the local image information entropy is not less than the preset information entropy threshold, it is determined that the current scene texture conditions are good and corner features are extracted using the point feature extraction method. When the local image information entropy is less than the preset information entropy threshold, it is determined that the current scene is in a weak texture or low light state. While extracting the residual corner features, it extracts the line segment features to obtain the point-line joint features to compensate for the information loss caused by the insufficient number of point features.
[0013] The spatiotemporal online self-calibration backend module is connected to the structural invariant extraction module and the optical entropy perception adaptive feature extraction module, respectively. It is used to construct a joint cost function that includes visual point feature reprojection residuals, inertial measurement unit pre-integration residuals, three-dimensional structural line marker reprojection residuals, and orientation constraint residuals based on the Manhattan world hypothesis. The joint cost function is iteratively solved by sliding window nonlinear optimization to obtain the camera pose, velocity, inertial measurement unit offset, extrinsic parameter matrix between the camera and the inertial measurement unit, and time offset between the camera and the inertial measurement unit for each frame within the sliding window. The solved extrinsic parameter matrix and time offset are fed back to the sensor data acquisition and synchronization module for spatiotemporal alignment compensation of subsequently acquired image frame data and inertial measurement data, forming a closed-loop process of continuous online self-calibration.
[0014] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:
[0015] 1. This invention utilizes the inherent static structural invariants in security scenarios as the absolute benchmark for zero drift, and introduces the structural line reprojection residual and orientation constraint residual into the joint optimization process of the camera and inertial measurement unit. Without adding expensive hardware such as lidar, it achieves continuous online self-calibration of extrinsic parameter matrix and time offset, gets rid of dependence on manual on-site calibration board, and significantly reduces the operation and maintenance cost of distributed security nodes.
[0016] 2. This invention dynamically determines the richness of scene texture by using the local information entropy of the image, adaptively switches between point feature extraction mode and point-line joint feature extraction mode, and dynamically adjusts the weight coefficient of structural line features in joint optimization according to the information entropy, so that the system can still maintain sufficient traceable feature observation in weak texture and low light scene, significantly improve the feature availability and tracking robustness of the visual front end, and reduce the positioning drift rate.
[0017] 3. The structure invariant online self-calibration mechanism and the optical entropy perception adaptive feature extraction mechanism of this invention are coupled and work together. The optical entropy perception module provides a more stable and reliable source of structure line observation and weight adjustment basis for the structure invariant self-calibration module. In turn, the structure invariant self-calibration module provides an accurate spatiotemporal reference for the optical entropy perception module. The two form an organic whole and jointly improve the localization robustness of the system in weak texture and long-term drift scenarios.
[0018] 4. The algorithm provided by this invention has low computational complexity and can run on edge computing chips such as ARM processors or lightweight neural network processors, meeting the deployment requirements of low-power IoT security nodes powered by solar energy or batteries, and is engineering-feasible. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the overall data flow of the system in this embodiment of the invention;
[0020] Figure 2 This is a schematic diagram of the optical entropy adaptive feature extraction sub-process in an embodiment of the present invention;
[0021] Figure 3 This is a schematic diagram of the spatiotemporal online self-calibration backend optimization closed-loop process in an embodiment of the present invention. Detailed Implementation
[0022] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.
[0023] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0024] I. System Overall Architecture Example:
[0025] In this embodiment, the hardware components of the IoT smart hardware security vision system include a low-cost monocular RGB camera, a microelectromechanical inertial measurement unit (MEMS), and an edge computing processing unit. The edge computing processing unit is implemented using an ARM processor and is used to carry out the calculations of the subsequent functional modules. The monocular RGB camera acquires image frame data at a fixed frame rate, and the MEMS acquires linear acceleration and angular velocity data at a sampling frequency higher than the image frame rate. The sensor data acquisition and synchronization module receives the above two data streams, timestamps each frame of image data and each set of inertial measurement data, and writes them into a shared synchronization buffer queue for subsequent modules to read in chronological order.
[0026] At the software level, the system sets up a structure invariant extraction module and an optical entropy perception adaptive feature extraction module in parallel after the synchronization buffer module. The two modules read the current image frame data from the synchronization buffer queue for processing. The feature observation data they output are uniformly merged into the spatiotemporal online self-calibration backend module for joint optimization. The pose estimation results output by the backend module are sent to the upper security application layer to support security functions such as equipment inspection navigation, environmental ranging, and anomaly detection. At the same time, the extrinsic parameter matrix and time offset obtained by the backend module are fed back to the sensor data acquisition and synchronization module in real time to perform spatiotemporal alignment compensation for subsequent newly acquired data, forming a closed-loop feedback relationship.
[0027] II. Example of online spatiotemporal self-calibration of scene structure invariant constraints:
[0028] In this embodiment, during the initial operation phase, the system first sets the initial estimated values of the camera intrinsic parameter matrix, the extrinsic parameter matrix between the camera and the inertial measurement unit, and the time offset, and initializes the sliding window and the state vector. The state variables of the most recent few frames are retained in the sliding window to participate in the optimization, while the earlier states are marginalized.
[0029] After acquiring the current image frame data, the structure invariant extraction module first performs edge detection and line segment extraction on the image frame. In this embodiment, a lightweight line segment detection algorithm is used to implement this process. Then, according to the vanishing point clustering method, each extracted line segment is grouped according to the direction of its convergence vanishing point in the image to obtain multiple orthogonal or parallel direction clusters that conform to the Manhattan world hypothesis. Candidate structure lines with length and cross-frame stability that meet the preset conditions are selected from each direction cluster as candidate observations of scene structure invariants.
[0030] For the above candidate structure lines, the structure invariant extraction module continues to perform cross-frame matching and tracking between consecutive frames. In this embodiment, a matching method based on line segment descriptors is used to establish the correspondence between candidate structure lines in adjacent frames. For candidate structure lines that can be stably observed in multiple consecutive frames, they are initialized as three-dimensional structure line markers, and the three-dimensional structure line markers are parameterized using Plück coordinates. Then, the structure line markers are added to the sliding window state vector.
[0031] Accordingly, the sliding window state vector includes the camera pose, velocity, inertial measurement unit offset, extrinsic parameter matrix between the camera and the inertial measurement unit, time offset between the camera and the inertial measurement unit, and parameters of structural line markers and traditional point feature markers for each frame within the window.
[0032] After completing the feature extraction and state vector construction, the spatiotemporal online self-calibration backend module constructs the visual point feature reprojection residual, the inertial measurement unit pre-integration residual, the 3D structural line marker reprojection residual, and the orientation orthogonality constraint residual based on the Manhattan world hypothesis, respectively. These residual terms are then jointly constructed into a unified nonlinear least squares cost function, which can be expressed as:
[0033] ;
[0034] In the formula, This represents the state vector to be estimated, which includes the camera pose, velocity, inertial measurement unit (IMU) bias, extrinsic parameter matrix between the camera and IMU, time offset, and landmark parameters for each frame within the sliding window. This represents the optimal solution of the state vector corresponding to the minimum value of the cost function. This represents a set of traditional point feature observations. Indicates the first Visual reprojection residuals of point features Let its corresponding covariance matrix be... This represents the pre-integrated observation set of the inertial measurement unit. Indicates the first Pre-integrated residuals of each inertial measurement unit Let its corresponding covariance matrix be... Represents the set of observations of structural line features. Indicates the first Reprojection residuals of structural lines Let its corresponding covariance matrix be... This represents a robust kernel function used to suppress the influence of out-of-point observations on the optimization results. The adaptive weighting coefficients of the residual terms of the structure line are dynamically given by the optical entropy sensing mechanism described later.
[0035] The reprojection residual of the three-dimensional structural line marker includes the point-to-line distance between the projected line segment obtained by projecting the structural line marker onto the image plane and the actual observed line segment, as well as the orthogonal constraint residual between two structural lines belonging to different Manhattan main directions. This orthogonal constraint residual can be expressed as:
[0036] ;
[0037] In the formula, and These represent the direction vectors of two structural lines belonging to different Manhattan main directions. Ideally, they are orthogonal to each other. The residual approaches zero and is used to constrain the optimization results to conform to the prior structure of the scenario, thereby reducing the degrees of freedom to be solved.
[0038] Furthermore, considering the asynchronous time offset between the exposure acquisition timestamp of the camera image frame and the sampling timestamp of the inertial measurement unit data, this embodiment employs a first-order compensation model to compensate for the image feature observation position deviation caused by this time offset. This model can be expressed as:
[0039] ;
[0040] In the formula, Indicates the timestamp in the inertial measurement unit. The location of the feature pixels observed by the camera. This represents the time offset between the camera and the inertial measurement unit to be estimated. The optical flow velocity of the feature point on the image plane is represented by a first-order Taylor expansion to compensate for the positional deviation caused by asynchronous sampling. This compensation term is synchronously introduced into the visual reprojection residual calculation process of the aforementioned joint cost function, so that the time offset is used as a parameter to be estimated to participate in the unified optimization solution.
[0041] After constructing the joint cost function, the spatiotemporal online self-calibration backend module uses the Levenberg-Marquardt nonlinear optimization method to iteratively solve the cost function. It jointly updates the camera pose, velocity, inertial measurement unit offset, structure line and point feature landmark parameters within the sliding window, and simultaneously updates the extrinsic parameter matrix and time offset between the camera and the inertial measurement unit to achieve online compensation for spatiotemporal parameter drift. After completing one iteration, the earliest state within the sliding window is marginalized, and its corresponding historical information is retained in a priori form and transferred to the remaining states within the window. Then, the sliding window is moved forward one frame, and the process of data acquisition and feature extraction is returned to process the next frame of image data, forming a closed-loop process of continuous online self-calibration. At the same time, the extrinsic parameter matrix and time offset obtained in this solution are fed back to the sensor data acquisition and synchronization module in real time for spatiotemporal alignment compensation of the subsequently acquired image frame data and inertial measurement data, ensuring that the data input to the optimization backend is always below the latest corrected spatiotemporal reference.
[0042] III. Example of Weak Texture Adaptive Feature Extraction Based on Optical Entropy Perception:
[0043] In this embodiment, after acquiring the current image frame data, the optical entropy-aware adaptive feature extraction module first performs grayscale preprocessing on the image frame, calculates the image grayscale histogram, obtains the frequency distribution of each grayscale level, and then calculates the local image information entropy of the current frame based on the grayscale histogram. The calculation formula can be expressed as:
[0044] ;
[0045] In the formula, This represents the image information entropy of the current frame. This represents the total number of gray levels in the image. Represents grayscale level The probability of a gray level appearing in the current frame is calculated by the ratio of the number of pixels at that gray level to the total number of pixels in the image.
[0046] The calculated information entropy Compared with the preset information entropy threshold Comparison: When Not less than When the current scene texture conditions are deemed good, the process enters the point feature extraction branch. A corner detection algorithm is used to extract corner features, and optical flow tracing is used to track these features across frames. Less than When the current scene is determined to be in a weak texture or low light state, the point and line joint feature extraction branch is entered. While extracting residual corner features, a lightweight line segment detection algorithm is used to extract edge line segment features in the image, and stable line segments that conform to the Manhattan main direction are retained first. Then, cross-frame matching and tracking of the line segment features are performed to establish cross-frame observation association of line features.
[0047] Furthermore, in this embodiment, the weighting coefficients of the structural line feature residuals in the aforementioned joint cost function are dynamically calculated based on the current frame information entropy magnitude. The weighting coefficient can be expressed as:
[0048] ;
[0049] In the formula, This refers to the weighting coefficients of the structural line residual term in the aforementioned joint cost function. This represents the local image information entropy of the current frame. The theoretical maximum value of image information entropy is shown in the above formula. As can be seen, this weight coefficient increases as the information entropy decreases, reflecting the adaptive enhancement of the importance of structural line features in weak texture scenes, enabling the system to maintain positioning accuracy by relying on structural line feature observation when the number of corner features is insufficient.
[0050] The point feature observations, line feature observations, and their corresponding weight coefficients extracted from the current frame are output to the aforementioned spatiotemporal online self-calibration backend module to participate in the joint nonlinear optimization solution. Simultaneously, this embodiment statistically analyzes the tracking success rate of point and line features and the convergence of the optimization residuals within the most recent few frames. If the tracking success rate is found to be consistently low or the optimization residuals show a diverging trend, the information entropy judgment threshold is adjusted accordingly. This makes the switching of feature extraction modes more closely match the lighting and texture changes in the actual scene, forming a continuous adaptive closed-loop adjustment process.
[0051] IV. Example of Spatiotemporal Calibration Relationship between Camera and Inertial Measurement Unit:
[0052] In this embodiment, the spatial geometric relationship between the camera coordinate system and the inertial measurement unit coordinate system is described by the rotation and translation components in the extrinsic parameter matrix. Together, they constitute the rigid body transformation relationship required for the conversion of camera observation data to the inertial measurement unit coordinate system. During the actual operation of the equipment, due to physical disturbances such as vibration and temperature changes, the rotation and translation components, as well as the time offset between the camera image frame exposure timestamp and the inertial measurement unit sampling timestamp, will all experience slow, implicit drift. This invention utilizes the physical characteristic that structural lines such as corner lines, door frame edges, and load-bearing column edges in security scenarios maintain constant positions before and after equipment vibration or temperature changes, and introduces them as an absolute reference with zero drift into the aforementioned joint optimization process. This allows the extrinsic parameter matrix and time offset to be continuously corrected by structural line observations during the continuous operation of the system, thereby avoiding the decrease in positioning accuracy caused by the fixed calibration parameters gradually deviating from the true value over time.
[0053] V. Alternative Implementation Methods:
[0054] In the structural invariant extraction stage, in addition to using straight line features as constraint benchmarks, planar structures such as walls and floors commonly found in security scenarios can also be used as structural invariants. A graph optimization backend based on planar reprojection residuals can be constructed through planar normal vector constraints to supplement or replace the aforementioned constraint method based on straight line features. For IoT nodes with relatively limited computing resources, the aforementioned graph optimization backend can also be replaced with a filtering framework based on error state Kalman filtering to reduce the computational complexity of the online self-calibration process and adapt to edge processing chips with lower power consumption.
[0055] In the optical entropy-aware adaptive feature extraction stage, in addition to using image information entropy as the criterion for texture richness, texture evaluation index based on image gradient statistics can also be used as the criterion for feature extraction mode switching. For IoT nodes equipped with lightweight neural network processors, a lightweight convolutional neural network can also be trained as a texture richness classifier to output the texture confidence of the current frame in real time, which can replace the criterion based on statistical information entropy and obtain stronger discrimination robustness in scenarios where lighting conditions change abruptly.
[0056] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.
Claims
1. An IoT-based intelligent hardware security vision system, comprising a monocular RGB camera, a microelectromechanical inertial measurement unit, and an edge computing processing unit, characterized in that: Also includes: The sensor data acquisition and synchronization module is connected to the monocular RGB camera and the microelectromechanical inertial measurement unit, and is used to acquire image frame data and inertial measurement data, and write the image frame data and the inertial measurement data into a timestamped synchronization buffer queue; The structure invariant extraction module, connected to the sensor data acquisition and synchronization module, is used to perform edge detection and line segment extraction on the image frame data. Based on the vanishing point clustering method, the extracted line segments are divided into multiple orthogonal direction clusters that conform to the Manhattan world hypothesis. Candidate structure lines with a length greater than a preset length threshold and a stable observation count across consecutive frames of no less than a preset number threshold are selected. The candidate structure lines are then matched and tracked across frames, and the candidate structure lines observed stably across multiple consecutive frames are initialized as three-dimensional structure line markers. An optical entropy-sensing adaptive feature extraction module, connected to the sensor data acquisition and synchronization module, is used to calculate the local image information entropy of the image frame data and compare the local image information entropy with a preset information entropy threshold: when the local image information entropy is not less than the preset information entropy threshold, corner features are extracted using a point feature extraction method; when the local image information entropy is less than the preset information entropy threshold, line segment features are extracted while extracting residual corner features to obtain point-line joint features. The spatiotemporal online self-calibration backend module is connected to the structural invariant extraction module and the optical entropy perception adaptive feature extraction module, respectively. It is used to construct a joint cost function including visual point feature reprojection residuals, inertial measurement unit pre-integration residuals, and reprojection residuals of the three-dimensional structural line markers. The reprojection residuals of the three-dimensional structural line markers include point-to-line distances and orientation constraint residuals based on the Manhattan world hypothesis. The joint cost function is iteratively solved by sliding window nonlinear optimization to obtain the camera pose, velocity, inertial measurement unit offset, extrinsic parameter matrix between the camera and the inertial measurement unit, and time offset between the camera and the inertial measurement unit for each frame within the sliding window. The solved extrinsic parameter matrix and time offset are fed back to the sensor data acquisition and synchronization module for spatiotemporal alignment compensation of the subsequently acquired image frame data and inertial measurement data.
2. The IoT smart hardware security vision system according to claim 1, characterized in that: The structural invariant extraction module is specifically used to parametrically represent the three-dimensional structural line marker using Plück coordinates. The reprojection residual of the three-dimensional structural line marker includes the point-to-line distance between the projected line segment obtained by projecting the three-dimensional structural line marker onto the image plane and the observed straight line segment, as well as the orthogonal constraint residual between the direction vectors of two structural lines belonging to different Manhattan main directions.
3. The IoT smart hardware security vision system according to claim 1, characterized in that: The optical entropy-aware adaptive feature extraction module is further configured to calculate the weighting coefficients of the reprojection residuals of the three-dimensional structural line markers in the joint cost function based on the local image information entropy. The weighting coefficients are determined according to the following relationship: ; In the formula, This represents the weighting coefficient. This represents the local image information entropy of the current image frame. The weighting coefficient represents the theoretical maximum value of image information entropy, and increases as the local image information entropy decreases.
4. The IoT smart hardware security vision system according to claim 3, characterized in that: The optical entropy-aware adaptive feature extraction module is also used to statistically analyze the tracking success rate of a preset number of intra-frame point features and line segment features, as well as the convergence of the optimized residual of the joint cost function, and adjust the preset information entropy threshold based on the statistical results.
5. The IoT smart hardware security vision system according to claim 1, characterized in that: The spatiotemporal online self-calibration backend module is also used to compensate for the image feature observation position deviation caused by the time offset according to the following relationship: ; In the formula, Indicates the timestamp in the inertial measurement unit. The location of the feature pixels observed by the camera. This indicates the time offset. This represents the optical flow velocity of the corresponding feature point on the image plane.
6. The IoT smart hardware security vision system according to claim 1, characterized in that: The spatiotemporal online self-calibration backend module is also used to marginalize the earliest state in the sliding window after completing one iteration of solution, retain the prior information obtained by marginalization and transfer it to the remaining state in the sliding window, and slide the sliding window forward one frame.
7. The IoT smart hardware security vision system according to claim 1, characterized in that: The edge computing processing unit is an ARM processor.
8. A spatiotemporal online self-calibration method for an IoT smart hardware security vision system, applied to an IoT smart hardware security vision system including a monocular RGB camera, a microelectromechanical inertial measurement unit, and an edge computing processing unit, characterized in that, The method includes: Acquire image frame data and inertial measurement data, and write the image frame data and inertial measurement data into a timestamped synchronization buffer queue; Edge detection and line segment extraction are performed on the image frame data. The extracted line segments are divided into multiple orthogonal direction clusters that conform to the Manhattan world hypothesis according to the vanishing point clustering method. Candidate structure lines with a length greater than a preset length threshold and a stable observation number of no less than a preset number threshold across consecutive frames are selected. Cross-frame matching and tracking are performed on the candidate structure lines. The candidate structure lines that are stably observed in multiple consecutive frames are initialized as three-dimensional structure line markers. Calculate the local image information entropy of the image frame data, and compare the local image information entropy with a preset information entropy threshold: when the local image information entropy is not less than the preset information entropy threshold, corner features are extracted using point feature extraction method; when the local image information entropy is less than the preset information entropy threshold, line segment features are extracted while extracting residual corner features to obtain point-line joint features. A joint cost function is constructed, comprising visual point feature reprojection residuals, inertial measurement unit pre-integration residuals, and reprojection residuals of the three-dimensional structural line markers. The reprojection residuals of the three-dimensional structural line markers include point-to-line distances and orientation constraint residuals based on the Manhattan world hypothesis. The joint cost function is iteratively solved by sliding window nonlinear optimization to obtain the camera pose, velocity, inertial measurement unit offset, extrinsic parameter matrix between the camera and the inertial measurement unit, and time offset between the camera and the inertial measurement unit for each frame within the sliding window. The obtained extrinsic parameter matrix and the time offset are fed back to the synchronization buffer queue for spatiotemporal alignment compensation of the subsequently acquired image frame data and inertial measurement data.
9. The spatiotemporal online self-calibration method for an IoT smart hardware security vision system according to claim 8, characterized in that: The method further includes: calculating the weighting coefficients of the reprojection residuals of the three-dimensional structural line targets in the joint cost function based on the local image information entropy, wherein the weighting coefficients are determined according to the following relationship: ; In the formula, This represents the weighting coefficient. This represents the local image information entropy of the current image frame. This represents the theoretical maximum value of image information entropy.
10. The spatiotemporal online self-calibration method for an IoT smart hardware security vision system according to claim 9, characterized in that: The method further includes: statistically analyzing the tracking success rate of a preset number of intra-frame point features and line segment features, as well as the convergence of the optimized residual of the joint cost function, and adjusting the preset information entropy threshold based on the statistical results.
