Integrated asset checking system and method based on automatic driving patrol car

Abstract

The invention discloses an integrated asset checking system and method based on an automatic driving patrol car, the system comprises an edge machine room and the automatic driving patrol car, and an x86 server deployed with T-Rex2 and YOLOV13 is arranged in the edge machine room; a laser radar, a panoramic pan-tilt camera and a WiFi antenna are fixedly installed on the top of the automatic driving patrol car, a differential GPS sensor is fixedly installed on the side face of the automatic driving patrol car, an inertial sensor and an AI reasoning box are fixedly installed in the automatic driving patrol car, and FastLIO2, ROS2 + rviz and an error state Kalman filter are deployed in the AI reasoning box. The inertial sensor, the laser radar, the panoramic holder camera and the differential GPS sensor are all connected with the AI reasoning box, and the AI reasoning box is connected with the x86 server through the WiFi antenna. According to the invention, the application of the automatic driving patrol car in asset checking is expanded, the accuracy, coverage efficiency, real-time performance and data security of asset checking are improved, and the operation cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent patrol technology, and particularly relates to an integrated asset inventory system and method based on an automatic driving patrol vehicle. BACKGROUND

[0002] With the continuous expansion of the scale of inventory building space, the number of various assets is increasing day by day, and the traditional asset inventory method mainly relies on manual inspection, which is low in efficiency and easy to miss. Using an intelligent patrol vehicle to replace security, cleaning and mechanical and electrical maintenance personnel for daily inspection can effectively improve the inspection efficiency. However, the existing intelligent patrol vehicle is mainly applied to the detection of abnormal situations in the management space and the processing of work orders, such as garbage overflow, personnel falling into water and personnel gathering, and has not been applied to the inventory of fixed equipment and facilities on the daily inspection route of security, cleaning and mechanical and electrical maintenance personnel.

[0003] When the existing intelligent patrol vehicle is applied to asset inventory, there are problems such as insufficient recognition accuracy and inability to accurately locate assets. Moreover, due to the inability to autonomously select an angle and point for observing and recording fixed facilities and equipment with an open field of view, when facilities and equipment are blocked by pedestrians and vehicles during public area shooting, the intelligent patrol vehicle cannot capture accurate information about the number of fixed facilities and equipment, resulting in inaccurate asset inventory reports.

[0004] Therefore, it is urgent to provide a complete solution from the aspects of autonomous navigation, path planning, fixed-point inspection, asset inventory report generation and the like of the intelligent patrol vehicle, so as to improve the recognition accuracy, positioning accuracy and accuracy of generating asset inventory reports of the automatic driving patrol vehicle. SUMMARY

[0005] The present application aims to provide an integrated asset inventory system and method based on an automatic driving patrol vehicle, which improves the accuracy, coverage efficiency, real-time performance and data security of the automatic driving patrol vehicle in asset inventory, and reduces the operating cost.

[0006] The present application is achieved by the following technical solutions:

[0007] An integrated asset inventory system based on an automatic driving patrol vehicle, comprising an edge machine room and an automatic driving patrol vehicle, wherein: an x86 server is arranged in the edge machine room, and the x86 server is deployed with a T-Rex2; a laser radar (1), a panoramic pan-tilt camera (2) and a WiFi antenna (4) are fixedly installed on the top of the automatic driving patrol vehicle; a differential GPS sensor (3) is fixedly installed on the side of the automatic driving patrol vehicle; an inertial sensor and an AI inference box (5) are fixedly installed in the interior of the automatic driving patrol vehicle; the AI inference box (5) is deployed with a FastLIO2, a ROS2+rviz and a target recognition algorithm YOLOV13.

[0008] The inertial sensor, laser radar (1), panoramic gimbal camera (2) and differential GPS sensor (3) are connected with the AI inference box (5), and the AI inference box (5) is connected with the x86 server through the WiFi antenna (4).

[0009] Further, the differential GPS sensor (3) is RTK-GPS.

[0010] Further, the AI inference box (5) is an NVIDIA Jetson NX edge inference card with a computing power of 21TOPS INT8.

[0011] An integrated asset inventory method based on an automatic driving patrol vehicle, comprising the following steps:

[0012] Step 1, SLAM mapping

[0013] Step 1.1, automatic driving patrol vehicle mapping: first, remotely control the driving direction of the automatic driving patrol vehicle, perform laser 3D scanning in the space to be inspected, and then tightly couple the point cloud data collected by the laser radar (1) through the error state Kalman filter and the angular velocity ω and acceleration a collected by the inertial sensor;

[0014] Step 1.2, constructing a semantic-topological double graph: first, the laser radar (1) scans the area to be inspected in real time through the mode of the automatic driving patrol vehicle, and the FastLIO2 outputs dense point clouds in ROS2+rviz, sets the object bounding box detected by the YOLOV13 model as B={B1,B2,…B i ,...,B n}, B i contains the category c i and the two-dimensional coordinates (x i ,y i ,w i ,h i ) of the bounding box, the object bounding box B is back projected to the three-dimensional space to generate semantic anchor points A={A1,A2,…,A n}, each semantic anchor point A i includes the three-dimensional coordinates (X i ,Y i ,Z i ) of the position, the category label c i and the normal vector n i =(n i,x ,n i,y ,n i,z ), and then the two-dimensional coordinates (X i ,Yi ) is performed to correct the deviation, and the topological graph G is represented as:

[0015] G = (V, E)

[0016] In the formula: V is a node, each node i∈V represents a parking photograph waypoint, including position (X i , Y i , Z i ) and attitude (q i,w , q i,x , q i,y , q i,z ); E represents an edge, each edge i→j∈E represents a path from node i to node j, and the weight w ij of each edge is defined as:

[0017] w ij = α·d ij + β·cost ij

[0018] In the formula: d ij is the Euclidean distance from i to j; cost ij is a cost function considering slope and obstacles; and α and β are weight coefficients for balancing the contributions of different factors;

[0019] Then the semantic-topological double graph is stored as map.pgm+yaml in ROS Navigation2, modified in ROS2+gimp, and the dynamic obstacles and obvious abnormal points are removed, and the.pgm file is saved by covering in the gimp interface;

[0020] Step 2, deploy T-Rex2 and YOLOV13

[0021] Step 2.1, deploy T-Rex2 framework in the x86 server inside the edge machine room, and deploy YOLOV13 on the AI inference box (5), the panoramic pan-tilt camera (2) uploads the video data of the space to be inspected to the AI inference box (5), extracts key frames, and the key frames are uploaded to the x86 server by the WiFi antenna (4), and T-Rex2 performs pre-labeling on the uploaded key frames to generate pre-labeled data;

[0022] Step 2.2, the CoG inference chain encodes the natural language instruction into a 77-dimensional CLIP token, cross-attention with the visual token to get a spatial mask M∈R H×W , and outputs the category c and 2D target detection box through the lightweight detection head of T-Rex2;

[0023] Step 2.3. Train and deploy the YOLOv3 model using the pre-labeled data generated by T-Rex2;

[0024] Step 3. Check the points, materials, and frequency of daily patrols by cleaning and security personnel, and check each asset anchor point A i Solve the optimal shooting pose T * , which is represented as:

[0025]

[0026] In the formula: T ∈ SE(3) represents the spatial position T, which is described as a Lie group; n is the optical axis of the panoramic pan-tilt camera, is the normal vector perpendicular to the optical axis of the panoramic pan-tilt camera; f(θ sun ) is the inverse light penalty term; α, β, and γ are weight coefficients, respectively;

[0027] Step 4. Joint optimization of inspection path, the specific process is:

[0028] Step 4.1. Abstract the inspection as a Prize-Collecting TSP problem model with time window, set the constraint conditions of the problem model, and the problem model is represented as:

[0029]

[0030] In the formula: ∑ i∈ Vr i x i represents maximizing the total reward, ∑ (i,j)∈E c ij y ij represents minimizing the total cost, and r i represents the importance of assets for valuation;

[0031] Step 4.2. Solve the problem model, the specific process is:

[0032] For coarse granularity, use the SCIP heuristic algorithm of OR-Tools to give the initial path; for fine granularity, use Monte Carlo tree search to optimize the path;

[0033] Step 5. Generate asset inventory report, the specific process is:

[0034] Step 5.1, establish Asset(id, class, install_date, spec) and State(id, timestamp, status_vector, img_hash) two table models, query through the (id, timestamp) joint primary key, realize the second-level historical backtracking, wherein: id represents the number; class represents the asset category; install_date is the pre-input installation time of the facility equipment; spec represents the pre-input note information of the equipment facility; Timestamp represents the specific target detection time; status_vector represents the information of whether the facility equipment still exists, status_vector uses 0 to represent that the facility equipment is missing, and uses 1 to represent that the facility equipment exists; img_hash represents the hash value of the picture;

[0035] When calculating the number of device installations, randomly sampling the data to obtain the average value as the final number of asset inventory results, which is used to generate an asset inventory report;

[0036] Step 5.2, the detected face or license plate is mosaicked, and the original image is AES-256 encrypted and retained, and automatically pulverized for 7 days; at the same time, the coordinates (x, y, z) of the located person or car are added with Laplace noise;

[0037] Step 6, manually review the asset inventory report, update the asset weight, and repeat steps 3-6 for multiple rounds of asset inventory.

[0038] Further, in the step 1.1, the point cloud data collected by the tightly coupled laser radar (1) is filtered by the error state Kalman filter And the specific process of the angular velocity ω and the acceleration a collected by the inertial sensor is as follows:

[0039] Step 1.1.1, first predict the state vector using the state transition function f() and the control input u, which is represented as:

[0040]

[0041] In the formula: u is the control input; w is the process noise, which is set as Gaussian white noise; f() is the state transition function; x is the state vector, including position, attitude, speed and IMU zero offset, wherein: the position refers to the position coordinates (x, y, z) of the autonomous driving patrol vehicle in three-dimensional space; the attitude refers to the rotation attitude of the autonomous driving patrol vehicle, which is represented as a quaternion (q w , q x , q y , q z ); the speed refers to the speed of the autonomous driving patrol vehicle in three-dimensional space, which is represented as (vx ,v y ,v z ); the IMU bias refers to the bias b a and the bias b g of the gyroscope of the inertial sensor;

[0042] The state vector x is expressed as: x = [x, y, z, q w , q x , q y , q z , v x , v y , v z , b a , b g ] T

[0043] The control input u includes the measurement values a and g of the inertial sensor, wherein: a is the measurement value of the accelerometer; g is the gyroscope measurement value; a and g are respectively expressed as:

[0044] a = (a x , a y , a z )

[0045] g = (g x , g y , g z )

[0046] Therefore, the control input u is expressed as u = [a, g] T ;

[0047] Step 1.1.2, first calculate the observation value z using the observation function h(), which is expressed as:

[0048] z = k(x) + v

[0049] In the formula: the observation value z includes the point cloud data collected by the lidar 1 and the measurement value of the inertial sensor; v is the measurement noise, which is set as Gaussian white noise; the observation function h() is used to map the state vector x to the measurement space, h() is the fusion of the point cloud data collected by the lidar 1 and the measurement value of the inertial sensor, which is expressed as:

[0050] h(x) = [h lidar (x), h imu (x)] T

[0051] In the formula: h lidar (x) represents the observation model of the point cloud data collected by the lidar, and h imu (x) represents the observation model of the measurement value of the inertial sensor;

[0052] For the observation model of the lidar point cloud data, it is assumed that the measurement error of the lidar is derived from the position error and the attitude error, h lidar (x) is represented as:

[0053]

[0054] wherein R(q) is the rotation matrix corresponding to the quaternion q = (q w , q x , q y , q z ) of the autonomous patrol vehicle, t = (x, y, z) is the position coordinate of the autonomous patrol vehicle, is the point cloud data collected by the lidar;

[0055] For the observation model of the inertial sensor measurement value, it is assumed that the measurement error of the inertial sensor is derived from the zero offset of the accelerometer and the gyroscope, and is represented as:

[0056] h imu (x) = [a - b a , g - b g ] T

[0057] wherein a is the measurement value of the accelerometer, g is the measurement value of the gyroscope, b a is the zero offset of the accelerometer, and b g is the zero offset of the gyroscope;

[0058] Step 1.1.3, in order to describe the linear change of the observation function h(x) to the state vector x, the Jacobian matrix of the observation function h(x) is defined as the observation matrix H in the form of partial derivative, and is represented as:

[0059]

[0060] Step 1.1.4, the Kalman gain K is calculated, and is represented as:

[0061] K = PH T (HPH T + Q) -1

[0062] wherein Q is the covariance matrix of the observation noise v;

[0063] Step 1.1.5, the state vector x is updated by using the Kalman gain, and is represented as:

[0064] x←x+K(z-h(x))

[0065] Step 1.1.6, the state covariance matrix P is updated, and is represented as:

[0066] P←(I-KH)P

[0067] In the formula, I represents a diagonal matrix, the elements on the diagonal are 1, and the remaining elements are 0.

[0068] Further, the specific process of step 2.3 is:

[0069] Step 2.3.1, screening the required samples from the pre-labeled data to obtain the screened data;

[0070] Step 2.3.2, data augmentation is performed on the screened data to obtain the augmented data;

[0071] Step 2.3.3, input the augmented data into the YOLOV13 model, and use the pre-training network of YOLOV13 to perform incremental fine-tuning training on the YOLOV13 model;

[0072] Step 2.3.4, lightening YOLOV13 to speed up inference.

[0073] Further, the data augmentation of step 2.3.2 includes random cropping, flipping and color adjustment.

[0074] Further, the lightening of YOLOV13 in step 2.3.4 is through AWQ4-bit and KV-Cache8-bit to lighten YOLOV13, which is represented as:

[0075]

[0076] In the formula, round() represents an integer operator not greater than the number ; W represents the original weight; W quant represents the weight of the quantized YOLOV13 neural network; k represents the quantization bit width.

[0077] Further, the constraint conditions of the problem model in step 4.1 include: the number of times each node i is visited x i ∈{0,1}, each node belongs to the entire anchor candidate point range V; each node pair (i,j) is in the adjacent node range E; if node i is visited, it must be reached from node j, and y ij ≤x i ; the time t i of visiting node i must be within the time window [a i ,b i ]; if the path from node i to node j is selected, t j =t i +c ij , t i represents the time of the patrol vehicle visiting node i; tj c represents the time it takes for the patrol car to visit node j. ij This indicates the time it takes for the patrol car to travel from point i to point j; the path must start from the starting point and end at the destination.

[0078] Furthermore, in step 5.2, Laplace noise is added to the coordinates (x, y, z) of the located person or vehicle, as shown below:

[0079] (x′,y′,z′)=(x,y,z)+(η x ,η y ,η z )

[0080] In the formula: η i ~Lap(0,b); b=0.3m, satisfying the differential privacy budget of ε∈[0.1,1], making the probability of a resident being located by reverse coordinates of a single point ≤0.05.

[0081] The present invention has the following beneficial technical effects:

[0082] 1) High recognition accuracy: On the one hand, T-Rex2 is used as the basic framework. As an advanced visual-language model task adaptive spatial representation extraction framework, T-Rex2 has powerful visual recognition and spatial understanding capabilities. It can quickly learn and locate similar fixed equipment and facilities in images with few or zero-sample prompts. This not only provides the target candidate box data format for the YOLOv13 target recognition algorithm, but also allows for fine-tuning and training of YOLOv13, shortening training time and saving annotation costs while reducing the probability of missed detections. On the other hand, considering the impact of lighting, shadows, and moving objects on the captured images, random sampling is introduced to appropriately estimate the number of fixed equipment and facilities, presenting a complete report on fixed equipment and facilities to space asset managers. Furthermore, angles and points with wide field of vision and easy observation and recording of fixed facilities and equipment are selected on the fixed-point driving route of the autonomous patrol vehicle to optimize the inspection path. This facilitates the autonomous patrol vehicle to obtain accurate information on the number of fixed equipment and facilities, thereby generating an accurate asset inventory report and laying the foundation for subsequent asset management.

[0083] 2) High coverage efficiency: Based on semantic-topological dual graphs, the same autonomous patrol vehicle can be used for multiple tasks such as monthly fire inspection and cleaning patrol, and achieves 100% coverage of asset inventory within the inspection space without blind spots.

[0084] 3) High real-time performance: YOLOV13's edge-end single inference time is <20ms, and millimeter-level inference meets the requirements for real-time high-definition video stream detection. In addition, the process delay from video capture to upload is <1s, which further improves the efficiency of asset inventory.

[0085] 4) High data security: The detected faces and license plates are masked, the original images are encrypted and set to self-destruct after 7 days, and noise is added to meet the differential privacy standard, which makes the security higher. In the field of asset inventory, the data is usable but not visible, and can be directly deployed in public sensitive scenarios such as universities and parks.

[0086] 5) Low operating costs: It can replace several full-time inspectors, reducing labor costs. Attached Figure Description

[0087] Figure 1 A schematic diagram of the structure of the invented autonomous patrol vehicle;

[0088] Figure 2 For the invention Figure 1 The flowchart for the asset inventory;

[0089] In the image: 1. LiDAR; 2. Panoramic gimbal camera; 3. Differential GPS sensor; 4. WiFi antenna; 5. AI inference box. Detailed Implementation

[0090] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0091] refer to Figure 1 As shown, an integrated asset inventory system based on an autonomous patrol vehicle includes an edge data center and an autonomous patrol vehicle. The edge data center houses an x86 server, which is equipped with T-Rex2 and the target recognition algorithm YOLOv13. The autonomous patrol vehicle is equipped with a LiDAR 1, a panoramic gimbal camera 2, and a WiFi antenna 4 fixedly mounted on its top. A differential GPS sensor 3 is fixedly mounted on the side of the autonomous patrol vehicle. An inertial measurement unit (IMU) and an AI inference box 5 are fixedly mounted inside the autonomous patrol vehicle. The AI ​​inference box 5 is equipped with FastLIO2, ROS2+rviz, and the target recognition algorithm YOLOv13.

[0092] The inertial sensor, lidar 1, panoramic gimbal camera 2, and differential GPS sensor 3 are all connected to the AI ​​inference box 5, which is connected to the x86 server via WiFi antenna 4.

[0093] Preferably, the WiFi antenna 4 is a 5G+Wi-Fi dual-link antenna, which can realize AP and mixed network connection, and supports local caching and interrupted transmission after network disconnection.

[0094] Preferably, the differential GPS sensor 3 is an RTK-GPS.

[0095] Preferably, the laser radar 1 is a 16-line 360° LIDAR.

[0096] Preferably, the AI inference box 5 is a NVIDIA Jetson NX edge inference card with a computing power of 21 TOPS INT8.

[0097] Preferably, the T-Rex2 is an open-source visual-language task adaptive spatial representation extraction framework released by the THU-MIG team of Tsinghua University, which is composed of a ViT visual encoder, a CLIP text encoder, and a lightweight detection head, and supports expanding new categories through natural language prompts with zero or a small number of samples.

[0098] Preferably, the FastLIO2 is a "tightly coupled laser radar-inertial odometry and mapping algorithm FastLIO2" proposed by the MaRS laboratory of the University of Hong Kong, which fuses 3D LiDAR point cloud and IMU data using error state Kalman filter (ESKF) to realize centimeter-level real-time positioning and dense point cloud mapping.

[0099] As shown in Figure 2 , an integrated asset inventory method based on an autonomous driving patrol vehicle includes the following steps:

[0100] Step 1, SLAM mapping

[0101] Step 1.1, autonomous driving patrol vehicle mapping: control the driving direction of the autonomous driving patrol vehicle through a remote control handle, perform laser 3D scanning and mapping in the space to be inspected, and the map range includes the ground space of asset-related facilities and equipment, multi-story or high-story inspection space. The autonomous driving patrol vehicle takes the elevator to reach the multi-story or high-story inspection space, and the point cloud data collected by the tightly coupled laser radar 1 and the angular velocity ω and acceleration a collected by the IMU are fused through error state Kalman filter, so as to accurately estimate the position, attitude and speed of the autonomous driving patrol vehicle, wherein: R 3 is a three-dimensional real number space representing the three-dimensional coordinates of the laser point cloud;

[0102] The specific process of fusing the point cloud data collected by the tightly coupled laser radar 1 and the angular velocity ω and acceleration a collected by the IMU is as follows:

[0103] Step 1.1.1, first use the state transition function f() and the control input u to predict the state vector, denoted as:

[0104]

[0105] wherein: u is the control input; w is the process noise, which is set as Gaussian white noise; f() is the state transition function; x is the state vector, including the position, attitude, velocity and IMU bias, wherein: the position refers to the position coordinates (x, y, z) of the autonomous patrol vehicle in three-dimensional space; the attitude refers to the rotational attitude of the autonomous patrol vehicle, which is expressed as a quaternion (q w , x , y , z ); the velocity refers to the velocity of the autonomous patrol vehicle in three-dimensional space, which is expressed as (v x , y , z ); the IMU bias refers to the zero bias b a of the accelerometer of the IMU and the zero bias b g of the gyroscope of the IMU;

[0106] The state vector x is expressed as: x = [x, y, z, q w , x , y , z , v x , y , z , b a , b g ] T

[0107] The control input u includes the measurement values a and g of the IMU, wherein: a is the measurement value of the accelerometer; g is the gyroscope measurement value; a and g are respectively expressed as:

[0108] a = (a x , a y , a z )

[0109] g = (g x , g y , g z )

[0110] Therefore, the control input u is expressed as u = [a, g] T ;

[0111] Step 1.1.2, first calculate the observation value z using the observation function h(), which is expressed as:

[0112] z = h(x) + v

[0113] In the formula: the observation value z includes the point cloud data collected by the lidar 1 and the measurement value of the IMU; v is the measurement noise, which is set as Gaussian white noise; the observation function h() is used to map the state vector x to the measurement space, and h() is the fusion of the point cloud data collected by the lidar 1 and the measurement value of the IMU, which is expressed as:

[0114] h(x) = [h lidar (x), h imu (x)] T

[0115] In the formula: h lidar (x) represents the observation model of the point cloud data collected by the lidar 1, and h imu (x) represents the observation model of the measurement value of the IMU;

[0116] 1) For the observation model of the point cloud data collected by the lidar 1, it is assumed that the measurement error of the lidar is derived from the position error and the attitude error, and h lidar (x) is expressed as:

[0117]

[0118] In the formula: R(q) is the rotation matrix corresponding to the quaternion q = (q w , q x , q y , q z ), t = (x, y, z) is the position coordinate of the autonomous patrol vehicle, is the point cloud data collected by the lidar 1;

[0119] 2) For the observation model of the measurement value of the IMU, it is assumed that the measurement error of the IMU is derived from the zero offset of the accelerometer and the gyroscope, and h imu (x) is expressed as:

[0120] h imu (x) = [a - b a , g - b g ] T

[0121] In the formula: a is the measurement value of the accelerometer, g is the measurement value of the gyroscope, b a is the zero offset of the accelerometer, and b g is the zero offset of the gyroscope;

[0122] Step 1.1.3, in order to describe the linear change of the observation function h(x) to the state vector x, the Jacobian matrix of the observation function h(x) is defined as the observation matrix H in the form of partial derivative, which is expressed as:

[0123]

[0124] Step 1.1.4, calculate the Kalman gain K, expressed as:

[0125] K = PH T (HPH T +Q) -1

[0126] In the formula: Q is the covariance matrix of observation noise v;

[0127] Step 1.1.5, update the state vector x using the Kalman gain K, expressed as:

[0128] x←x+K(z-h(x))

[0129] Step 1.1.6, update the state covariance matrix P, expressed as:

[0130] P←(I-KH)P

[0131] In the formula: I represents a diagonal matrix, the elements on the diagonal are 1, and the rest are 0;

[0132] Through steps 1.1.1-1.1.6, the tight coupling of the point cloud data collected by the laser radar 1 and the measurement values of the inertial sensor IMU is realized, providing high-precision position and attitude information for the asset inventory of the autonomous driving patrol vehicle;

[0133] Step 1.2, construct a semantic-topological double graph: through the mode of the autonomous driving patrol vehicle, the laser radar 1 scans the area to be checked by security and cleaning in real time, and FastLIO2 outputs dense point clouds in ROS2+rviz, sets the object bounding box detected by the YOLOV13 model as B={B1,B2,…B i ,...,B n}, B i contains the category c i and the two-dimensional coordinates (x i ,y i ,w i ,h i ) of the bounding box, and the object bounding box B is back projected to the three-dimensional space to generate semantic anchor points A={A1,A2,…,A n}, each semantic anchor point A i includes the three-dimensional coordinates (X i ,Y i ,Z i ) of the position, the category label c i and the normal vector n i =(n i,x ,n i,y ,n i,z );

[0134] Since the precision of outdoor mapping is difficult to control due to the influence of environmental temperature, light intensity and obstacles, the two-dimensional coordinates (X i ,Y i ) of the semantic anchor point A are corrected by the GPS signal collected by the differential GPS sensor 3 installed on the autonomous driving patrol vehicle, and the topological graph G is represented as:

[0135] G=(V,E)

[0136] In the formula, V is a node, each node i∈V represents a parking photograph flight point, including position (X i ,Y i ,Z i ) and attitude (q i,w ,q i,x ,q i,y ,q i,z ); E represents an edge, each edge i→j∈E represents a path from node i to node j, and the weight w ij of each edge is defined as:

[0137] w ij =α·d ij +β·cost ij

[0138] In the formula, d ij is the Euclidean distance from i to j; cost ij is a cost function considering slope and obstacles; and α and β are weight coefficients for balancing the contribution of different factors.

[0139] The semantic-topological double graph is stored as "map.pgm+yaml" in ROSNavigation2, and after preliminary mapping, it is modified in ROS2+gimp to remove dynamic obstacles and obvious abnormal points, and the.pgm file is saved in the gimp interface for subsequent patrol line setting.

[0140] The semantic-topological double graph refers to the semantic layer and the topological layer superimposed on the geometric map generated by FastLIO2, wherein the semantic layer is a set of 3D semantic anchor points obtained by T-Rex2 detection frame back projection; the topological layer is a directed graph with parking points as nodes and passable paths as edges; the semantic layer and the topological layer are aligned through the same coordinate system, and together constitute a hybrid graph structure for path planning and asset indexing.

[0141] Step 2, deploy T-Rex2 and YOLOV13

[0142] Step 2.1, deployment architecture: deploy T-Rex2 framework on x86 server inside edge machine room, deploy YOLOV13 on AI inference box 5, T-Rex2 framework provides data pre-labeling service through Gradio interactive webpage and API, administrator only needs to input asset marker recognition label information (for example: fire extinguisher red cylinder, underground garage parking space), T-Rex2 provides initialized labeling data for YOLOV13; in the initialization stage, i.e. data cold start stage, panoramic PTZ camera 2 uploads video data of the space to be inspected to AI inference box 5, extracts key frames, key frames are uploaded to x86 server by WiFi antenna 4, T-Rex2 pre-labels the uploaded key frames to generate pre-labeling data;

[0143] Step 2.2, prompt-word-task mapping: CoG inference chain encodes natural language instructions into 77-dimensional CLIP tokens, and performs cross-attention with visual tokens to obtain spatial mask M ∈ R H×W , and outputs class c and 2D target detection box through the lightweight detection head of T-Rex2, for new classes, only need to append asset marker recognition labels to the prompt word library;

[0144] The CoG inference chain is a chain-like spatial positioning inference mechanism in T-Rex2 framework for parsing natural language instructions, the input is prompt word token, and the output is 2D / 3D spatial mask, which drives the detection head to complete class-position joint prediction, this embodiment extends it to a "prompt word-asset class-3D anchor point" triple mapping table, realizes zero-code class expansion, without adding training data, only by updating the prompt word library can T-Rex2 recognize new asset classes;

[0145] Step 2.3, YOLOv13 model training and inference deployment: transmit the pre-labeling data generated by T-Rex2 to AI inference box 5 through WiFi antenna 4, train YOLOv13 model using pre-labeling data generated by T-Rex2, the specific process is as follows:

[0146] Step 2.3.1, data screening: screen high-quality samples from pre-labeled data, remove duplicate, inaccurate or incomplete data, get screened data;

[0147] Step 2.3.2, data augmentation: augment the screened data, including random cropping, flipping and color adjustment, to improve the generalization ability of the model, get augmented data;

[0148] Step 2.3.3, fine-tuning training: input the enhanced data into the YOLOV13 model, and use the pre-trained network of YOLOV13 to perform incremental fine-tuning training on the YOLOV13 model;

[0149] Step 2.3.4, inference acceleration: pruning or lightening YOLOV13 through TensorFlow Lite, ONNX Runtime, AWQ 4-bit (Activation-aware Weight Quantization) or KV-Cache 8-bit to reduce the size and computational complexity of the model, so that it can run efficiently on the edge inference card NVIDIA Jetson NX;

[0150] In this embodiment, AWQ 4-bit and KV-Cache 8-bit are used to lighten YOLOV13, which is represented as:

[0151]

[0152] In the formula, round() represents an integer operator that is not greater than the number ; W represents the original weight; W quant represents the weight of the quantized YOLOV13 neural network; k represents the quantization bit width, such as 4-bit quantization, which means that each weight value is represented by 4 bits, i.e. k = 4;

[0153] Step 3, image fixed-point acquisition protocol: check the points, materials and frequency of daily patrol of cleaners and security guards, and check each asset anchor point A i to solve the optimal shooting pose T * , so as to obtain the position and direction that need to be fixed-point inspected by the automatic driving patrol vehicle every day and every hour, T * is represented as:

[0154]

[0155] In the formula, T∈SE(3) represents the spatial position T, which is described as a Lie group; n is the optical axis of the panoramic pan-tilt camera, is the normal vector perpendicular to the optical axis of the panoramic pan-tilt camera; f(θ sun ) is the inverse light penalty term; α, β and γ represent weight coefficients, respectively;

[0156] The image fixed-point acquisition protocol is a vehicle-end operation specification composed of an optimal shooting pose solving algorithm and a centimeter-level parking control, so that each acquisition has: target asset pixel ratio ≥ 25%; camera optical axis angle ≤ 15°; inverse light penalty term is minimum; horizontal / vertical parking error ≤ 3 cm, and heading error ≤ 0.5°;

[0157] Step 4, joint optimization of inspection path, the specific process is:

[0158] Step 4.1, problem modeling: abstract inspection as a Prize-Collecting TSP problem model with time window, represented as:

[0159]

[0160] In the formula: ∑ i∈V r i x i represents maximizing the total reward, ∑( i,j)∈E c ij y ij represents minimizing the total cost, r i represents considering asset importance for assignment, for example: fire extinguisher, r i =10; for garbage can, r i =2;

[0161] The constraint conditions of the Prize-Collecting TSP problem model with time window are: 1), the number of times each node i is visited x i ∈{0,1}, each node belongs to the entire anchor candidate point range V; 2), each node pair (i,j) is in the adjacent node range E; 3), if node i is visited, it must be reached from node j, and y ij ≤x i ; 4), the time t i of visiting node i must be within the time window [a i ,b i ]; 5), if the path from node i to node j is selected, t j =t i +c ij , t i represents the time when the patrol car visits node i, t j represents the time when the patrol car visits node j, c ij represents the time when the patrol car travels from node i to node j; 6), the path must start from the starting point and end at the ending point;

[0162] The Prize-Collecting TSP problem model with time window is a path optimization model with time window based on the classic Traveling Salesman Problem, which introduces "reward-punishment" term, and realizes multi-task combined inspection through OR-Tools MCTS two-level solution.

[0163] Step 4.2, problem model solution, the specific process is:

[0164] For coarse granularity, use the SCIP heuristic algorithm of OR-Tools to give an initial path, so as to give a feasible solution in a shorter time;

[0165] For fine granularity, use Monte Carlo Tree Search (MCTS) to optimize the path. MCTS finds a better path by simulating multiple path choices;

[0166] Step 5, generate asset inventory report, the specific process is:

[0167] Step 5.1, two table model: establish Asset(id, class, install_date, spec) and State(id, timestamp, status_vector, img_hash) two table model, query through the joint primary key of (id, timestamp), realize the second-level historical backtracking, wherein: id represents the number; Class represents the asset category; install_date is a pre-entered parameter, which is the installation time of the facility equipment; Spec represents the note information of the equipment and facility, which is also the pre-entered information; Timestamp represents the specific target detection time; status_vector represents the information of whether the facility equipment still exists, 0 represents missing, 1 represents existing; img_hash represents the hash value of the picture; Considering the environmental influence factors (such as light shadow, people and car daily activities), when calculating the number of equipment facilities, random sampling is carried out on the whole data, and the average value is taken as the final number of asset inventory results, which is used to generate the asset inventory report;

[0168] The Asset(id, class, install_date, spec) and State(id, timestamp, status_vector, img_hash) two table model is a two-class data table of asset-state, wherein: Asset(id, class, install_date, spec) is a static attribute, State(id, timestamp, status_vector, img_hash) is a real-time state, and the joint primary key of asset id, timestamp and space hash value realizes second-level traceability;

[0169] Step 5.2, privacy calculation: 64x64 pixel mosaic for real-time detected face or license plate, and AES-256 encryption for original image retention, 7 days automatic crushing; At the same time, add Laplace noise to the coordinates (x, y, z) of the located person or car Indicated as:

[0170] (x',y',z') = (x, y, z) + (η x ,η y ,η z )

[0171] wherein: η i ~ Lap(0, b); b = 0.3m, satisfies the difference privacy budget ε ∈ [0.1, 1], the probability of household reverse positioning of single point coordinates ≤0.05;

[0172] Step 6, end-to-end work: manual review of asset inventory report, update asset weight, repeat steps 3-6, and conduct multiple rounds of asset inventory.

[0173] In this embodiment, under the condition of 20T edge computing power, the asset inventory system constructed with centimeter-level positioning, zero-sample category expansion, route optimization and full-link privacy compliance has the following advantages in the asset inventory process:

[0174] 1) Zero-sample expansion mechanism of edge computing inference model: for the first time, the visual-language model T-Rex2 framework is combined with the target detection algorithm YOLOV13 to form a complete closed loop from "prompt word engineering + CoG inference chain + small model fine-tuning training + edge model inference + asset report", which supports administrators to describe a new asset category in <2 minutes, without the need to retrain models and label, thus getting rid of the problem of high cost of traditional model training;

[0175] 2) Dynamic map and path optimization based on "semantic-topology" dual graph fusion: on the three-dimensional geometric map generated by FastLIO2 laser-inertial SLAM, 3D semantic anchor points converted from T-Rex2 detection boxes are superimposed to form a "asset-road" hybrid map, which realizes real-time joint optimization of inspection route and asset importance weight, so that the same patrol vehicle can dynamically combine different tasks such as fire inspection and cleaning inspection, thereby reducing labor cost and process;

[0176] 3) Privacy compliance framework: the edge-center collaborative architecture of "asset-state" two-table integration is created, the edge terminal identifies asset state in real time, automatically codes sensitive faces or license plates, and the original image is encrypted and retained for 7 days before being automatically destroyed, thus meeting the privacy requirements;

[0177] 4) "recognition-inventory" closed loop: the autonomous driving patrol vehicle is responsible for real-time detection, encryption and caching; the edge machine room is responsible for map optimization, global path re-planning, AI model training and inference, through the two-level architecture, the end-to-end closed loop of "visual perception-data governance-SLAM mapping-automated pre-annotation training-edge inference-generation of asset inventory report and manual review" is realized, and a complete solution for asset inventory is given.

Claims

1. An integrated asset inventory system based on an autonomous patrol vehicle, characterized in that, The system includes an edge data center and an autonomous patrol vehicle. The edge data center is equipped with an x86 server and a T-Rex2 is deployed on the x86 server. The autonomous patrol vehicle is equipped with a lidar (1), a panoramic gimbal camera (2), and a WiFi antenna (4) on its top. The autonomous patrol vehicle is equipped with a differential GPS sensor (3) on its side. The autonomous patrol vehicle is equipped with an inertial sensor and an AI inference box (5) inside its interior. The AI ​​inference box (5) is equipped with FastLIO2, ROS2+rviz, and the target recognition algorithm YOLOV13. The inertial sensor, lidar (1), panoramic gimbal camera (2) and differential GPS sensor (3) are all connected to the AI ​​inference box (5), which is connected to the x86 server via WiFi antenna (4).

2. The integrated asset inventory system based on an autonomous patrol vehicle according to claim 1, characterized in that, The differential GPS sensor (3) is an RTK-GPS.

3. The integrated asset inventory system based on an autonomous patrol vehicle according to claim 1, characterized in that, The AI ​​inference box (5) is an NVIDIA Jetson NX edge inference card with a computing power of 21 TOPS INT8.

4. An integrated asset inventory method based on an autonomous patrol vehicle according to any one of claims 1 to 3, characterized in that, Includes the following steps: Step 1: SLAM mapping Step 1.1, Autonomous Patrol Vehicle Mapping: First, use the remote control to control the autonomous patrol vehicle's direction of travel, perform laser 3D scanning in the area to be inspected, and then use the point cloud data collected by the error state Kalman filter tightly coupled to the lidar (1). The angular velocity ω and acceleration a are collected by the inertial sensor; Step 1.2, Constructing a semantic-topological dual graph: First, using the autonomous patrol vehicle's route mode, the LiDAR (1) scans the area to be inspected in real time, and FastLIO2 outputs a dense point cloud in ROS2+rviz. The bounding boxes of objects detected by the YOLOV13 model are set as B={B1,B2,…B i ,...,B n }, B i Category c of objects i and the two-dimensional coordinates (x) of the bounding box i ,y i ,w i ,h i The bounding box B of the object is back-projected into 3D space to generate semantic anchor points A = {A1, A2, ..., A...}. n }, each semantic anchor point A i Includes the three-dimensional coordinates (X) of the position i ,Y i Z i Category label c i and normal vector n i =(n i,x ,n i,y ,n i,z Then, the GPS signal collected by the differential GPS sensor (3) is used to determine the two-dimensional coordinates (X, Y, X) of the semantic anchor point A. i ,Y i To correct the deviation, the topology graph G is represented as: G = (V, E) In the formula: V represents a node, and each node i∈V represents a waypoint for parking and taking photos, including the location (X). i ,Y i Z i ) and attitude (q) i,w ,q i,x ,q i,y ,q i,z E represents an edge, and each edge i→j∈E represents a path from node i to node j. The weight w of each edge is... ij Defined as: w ij =α·d ij +β·cost ij In the formula: d ij Cost is the Euclidean distance from i to j. ij It is a cost function that takes into account slope and obstacles; α and β are weighting coefficients used to balance the contributions of different factors; Then, the semantic-topological dual graph is stored as map.pgm+yaml in ROS Navigation2, modified in ROS2+gimp to remove dynamic obstacles and obvious outliers, and the .pgm file is overwritten and saved in the gimp interface. Step 2: Deploy T-Rex2 and YOLOv13 Step 2.1 Deploy the T-Rex2 framework on the x86 server inside the edge data center, deploy YOLOV13 on the AI ​​inference box (5), and upload the video data of the space to be inspected collected by the panoramic gimbal camera (2) to the AI ​​inference box (5), extract key frames, and upload the key frames to the x86 server by the WiFi antenna (4). T-Rex2 pre-annotates the uploaded key frames and generates pre-annotated data. Step 2.2: The CoG inference chain processes natural language instructions. Encoded as a 77-dimensional CLIP token, cross-attention is performed with the visual token to obtain a spatial mask M∈R. H×W The lightweight detection head of T-Rex2 outputs category c and 2D object detection boxes; Step 2.3: Use the pre-labeled data generated by T-Rex2 to train and deploy the YOLOv13 model for inference; Step 3: Inventory the locations, materials, and frequency of daily patrols by cleaning and security personnel, and conduct an inventory of each asset anchor point A. i Solving for the optimal shooting pose T * , is represented as: In the formula: T∈SE(3) represents the spatial location T, which is represented by a Lie group description; n is the optical axis of the panoramic gimbal camera, n T f(θ) represents the normal vector perpendicular to the optical axis of the panoramic gimbal camera. sun ) represents the backlight penalty term; α, β, and γ represent the weighting coefficients, respectively; Step 4: Joint optimization of inspection paths, the specific process is as follows: Step 4.1: Abstract the inspection process into a Prize-CollectingTSP problem model with a time window, and set the constraints of the problem model. The problem model is represented as follows: In the formula: ∑ i∈V r i x i To maximize the total reward, ∑ (i,j)∈E c ij y ij R represents minimizing the total cost. i This indicates that values ​​are assigned based on the importance of the assets. Step 4.2, Solving the problem model, the specific process is as follows: For coarse-grained approaches, the SCIP heuristic algorithm from OR-Tools is used to provide an initial path; for fine-grained approaches, Monte Carlo tree search is used to optimize the path. Step 5: Generate an asset inventory report. The specific process is as follows: Step 5.1: Establish a two-table model of Asset(id, class, install_date, spec) and State(id, timestamp, status_vector, img_hash). A second-level historical backtracking can be achieved through a combined primary key query using (id, timestamp). Here, id represents the number; class represents the asset category; install_date is the pre-entered installation time of the facility / equipment; spec represents the pre-entered remarks information of the facility / equipment; timestamp represents the time of specific target detection; status_vector represents whether the facility / equipment is still in existence (0 indicates missing, 1 indicates present); and img_hash represents the hash value of the image. When calculating the quantity of equipment and facilities, the data is randomly sampled, and the average value is used as the final asset inventory result to generate an asset inventory report. Step 5.2: Blur the faces or license plates detected in real time, and encrypt the original images with AES-256 and store them for 7 days; at the same time, add Laplace noise to the coordinates (x,y,z) of the located people or vehicles. Step 6: Manually review the asset inventory report, update the asset weights, and repeat steps 3-6 to conduct multiple rounds of asset inventory.

5. The integrated asset inventory method based on an autonomous patrol vehicle according to claim 4, characterized in that, In step 1.1, the point cloud data acquired by the error-state Kalman filter tightly coupled lidar (1) The specific process of acquiring angular velocity ω and acceleration a from the inertial sensor is as follows: Step 1.1.1: First, using the state transition function f() and the control input u, predict the state vector, which is expressed as: In the formula: u is the control input; w is the process noise, set as Gaussian white noise; f() is the state transition function; x is the state vector, including position, attitude, velocity, and IMU bias, where: position refers to the position coordinates (x, y, z) of the autonomous patrol vehicle in three-dimensional space; attitude refers to the rotational attitude of the autonomous patrol vehicle, represented by quaternions (q w ,q x ,q y ,q z Speed ​​refers to the speed of the autonomous patrol vehicle in three-dimensional space, denoted as (v). x ,v y ,v z IMU zero bias refers to the zero bias of the accelerometer in the inertial sensor. a The zero bias b of the gyroscope and the inertial sensor g ; The state vector x is represented as: x = [x, y, z, q] w ,q x ,q y ,q z ,v x ,v y ,v z ,b a ,b g ] T The control input u includes the measured values ​​a and g from the inertial sensor, where: a is the measured value from the accelerometer; g is the measured value from the gyroscope; and a and g are respectively represented as: a=(a x ,a y ,a z ) g=(g x ,g y ,g z ) Therefore, the control input u is represented as u = [a, g]. T ; Step 1.1.2: First, calculate the observed value z using the observation function h(), expressed as: z = h(x) + v In the formula: the observed value z includes the point cloud data collected by the lidar (1) and the measured value of the inertial sensor; v is the measurement noise, set as Gaussian white noise; the observation function h() is used to map the state vector x to the measurement space, and h() is the fusion of lidar point cloud data and inertial sensor measured value, expressed as: h(x)=[h lidar (x),h imu (x)] T Where: h lidar (x) represents the observation model of the point cloud data collected by lidar (1), h imu (x) represents the observation model of the inertial sensor's measurement values; The measurement error of the lidar (1) is assumed to originate from position error and attitude error, h lidar (x) is represented as: In the formula: R(q) is a quaternion q = (q w ,q x ,q y ,q z The corresponding rotation matrix, t=(x,y,z), represents the position coordinates of the autonomous patrol vehicle. It is point cloud data collected by lidar (1); The measurement error of the inertial sensor is assumed to originate from the zero bias of the accelerometer and gyroscope, h imu (x) is represented as: h imu (x)=[a-b a ,g-b g ] T In the formula: a is the measurement value of the accelerometer, g is the measurement value of the gyroscope, and b is the measurement value of the gyroscope. a It is the zero bias of the accelerometer, b g It is the zero bias of the gyroscope; Step 1.1.3: To describe the linear change of the observation function h(x) with respect to the state vector x, the Jacobian matrix of the observation function h(x) is defined as the observation matrix H in the form of partial derivatives, expressed as: Step 1.1.4: Calculate the Kalman gain K, expressed as: K=PH T (HPH T +Q) -1 In the formula: Q is the covariance matrix of the observation noise v; Step 1.1.5: Update the state vector x using Kalman gain, expressed as: x←x+K(zh(x)) Step 1.1.6: Update the state covariance matrix P, expressed as: P←(I-KH)P In the formula: I represents a diagonal matrix, with the elements on the diagonal being 1 and the rest being 0.

6. The integrated asset inventory method based on autonomous patrol vehicles according to claim 4, characterized in that, The specific process of step 2.3 is as follows: Step 2.3.1: Select the required samples from the pre-labeled data to obtain the filtered data; Step 2.3.2: Perform data augmentation on the filtered data to obtain augmented data; Step 2.3.3: Input the enhanced data into the YOLOV13 model and use the pre-trained network built into YOLOV13 to perform incremental fine-tuning training on the YOLOV13 model; Step 2.3.4: Perform lightweight processing on YOLOV13 to accelerate inference.

7. The integrated asset inventory method based on an autonomous patrol vehicle according to claim 6, characterized in that, The data augmentation in step 2.3.2 includes random cropping, flipping, and color adjustment.

8. The integrated asset inventory method based on an autonomous patrol vehicle according to claim 6, characterized in that, Step 2.3.4, which involves lightweighting YOLOV13, uses AWQ 4-bit and KV-Cache 8-bit to perform lightweighting, as shown below: In the formula: round() means not greater than a number Integer operators; W represents the original weight; W quant This represents the weights of the quantized YOLOv13 neural network; k represents the quantization bit width.

9. The integrated asset inventory method based on an autonomous patrol vehicle according to claim 4, characterized in that, The constraints of the problem model in step 4.1 include: the number of times each node i is visited x. i ∈{0,1}, each node belongs to the entire range of anchor candidate points V; each node pair (i,j) is within the range of neighboring nodes E; if node i is visited, it must be reached from node j, and y ij ≤x i Time t for accessing node i i Must be within the time window [a i ,b i [Within]; if the path from node i to node j is selected, then t j =t i +c ij , t i t represents the time t takes for the patrol car to visit node i. j c represents the time it takes for the patrol car to visit node j. ij This indicates the time it takes for the patrol car to travel from point i to point j; the path must start from the starting point and end at the destination.

10. The integrated asset inventory method based on an autonomous patrol vehicle according to claim 4, characterized in that, In step 5.2, Laplace noise is added to the coordinates (x, y, z) of the located person or vehicle, as shown below: (x′,y′,z′)=(x,y,z)+(η x ,η y ,η z ) In the formula: η i ~Lap(0,b); b=0.3m, satisfying the differential privacy budget of ε∈[0.1,1], making the probability of a resident being located by reverse coordinates of a single point ≤0.05.

Images