Automatic Control System for Mobile Charging Robot Based on Spatial Modeling and Visual Recognition Technology

Abstract

The present invention relates to the technical field of automatic control of mobile charging robots. Specifically, it relates to an automatic control system for mobile charging robots based on spatial modeling and visual recognition technologies. It includes an environmental space modeling unit that uses a dynamic spatio-temporal fusion environmental modeling algorithm to utilize multiple sensors to continuously sense and update the three-dimensional space map model of the parking lot; a vehicle recognition and positioning unit that accurately locates the target vehicle in the three-dimensional space map model through a non-linear multi-feature fusion positioning algorithm; a path planning and scheduling unit that plans the optimal path and schedules tasks based on the three-dimensional space map model and the vehicle positioning result; and an interaction and maintenance unit that is responsible for the communication between the system and users and staff, and monitors and manages the robots. This system can accurately sense and update dynamic environmental space information through the dynamic spatio-temporal fusion environmental modeling algorithm and the non-linear multi-feature fusion positioning algorithm, and can achieve accurate recognition and positioning of target vehicles in complex vehicle scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic control of mobile charging robots, and more specifically, to an automatic control system for mobile charging robots based on spatial modeling and visual recognition technology. Background Art

[0002] The automatic control system for mobile charging robots based on spatial modeling and visual recognition technology aims to perform real-time modeling of the environment, accurately identify the vehicle position, and optimize the execution of charging tasks. By using a dynamic spatio-temporal fusion environment modeling algorithm and a non-linear multi-feature fusion positioning algorithm, it controls the path planning and task scheduling of the mobile charging robot to achieve efficient and accurate charging operations for target vehicles in complex and dynamic environments.

[0003] Existing automatic control systems for mobile charging robots usually have difficulty in real-time updating the three-dimensional positions of vehicles and obstacles in a dynamically changing parking lot environment. Moreover, due to the narrow, dynamic and changeable parking lot environment and the possible occlusion of the license plate of the target vehicle, problems such as inaccurate vehicle positioning recognition, inability to quickly avoid obstacles in path planning, and unreasonable charging task scheduling may occur in a multi-vehicle dense scenario. Therefore, an automatic control system for mobile charging robots based on spatial modeling and visual recognition technology is designed. Summary of the Invention

[0004] The purpose of the present invention is to provide an automatic control system for mobile charging robots based on spatial modeling and visual recognition technology to solve the problems of inaccurate vehicle positioning recognition, inability to quickly avoid obstacles in path planning, and unreasonable charging task scheduling in a multi-vehicle dense scenario as proposed in the above background art due to the narrow, dynamic and changeable parking lot environment and the possible occlusion of the license plate of the target vehicle.

[0005] To achieve the above purpose, the present invention provides an automatic control system for mobile charging robots based on spatial modeling and visual recognition technology, including:

[0006] An environmental space modeling unit, which is used to sense the parking lot environment in real time, and construct and update a three-dimensional space map model in real time by using a dynamic spatio-temporal fusion environment modeling algorithm and the data captured by lidar, cameras, and ultrasonic sensors;

[0007] It also includes a vehicle recognition and positioning unit, which is used to identify and position the target vehicle, and accurately position the vehicle in the three-dimensional space map model by integrating a non-linear multi-feature fusion positioning algorithm;

[0008] It also includes a path planning and scheduling unit, which is used to plan the optimal driving path and perform task scheduling according to the three-dimensional space map model provided by the environmental space modeling unit and the three-dimensional vehicle positioning of the vehicle recognition and positioning unit;

[0009] It further includes an interaction and maintenance unit, which is used for the system to directly communicate with users and staff, and to monitor and manage the charging process and the self-maintenance of the robot when it is idle.

[0010] As a further improvement of this technical solution, the environmental space modeling unit includes a data acquisition and fusion module and a space modeling and dynamic update module;

[0011] The data acquisition and fusion module captures three-dimensional point cloud data of the environment from a lidar, captures two-dimensional image data from a camera, and detects the obstacle distance through an ultrasonic sensor, and uses data fusion technology to fuse the three-dimensional point cloud data, two-dimensional image data, and obstacle distance into a global three-dimensional coordinate system to obtain three-dimensional data;

[0012] The space modeling and dynamic update module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional space map model, and uses a dynamic spatio-temporal fusion environment modeling algorithm to update the three-dimensional map model in real time.

[0013] As a further improvement of this technical solution, the data acquisition and fusion module captures three-dimensional point cloud data of the environment from a lidar, captures two-dimensional image data from a camera, and detects the obstacle distance through an ultrasonic sensor, and uses data fusion technology to fuse the three-dimensional point cloud data, two-dimensional image data, and obstacle distance into a global three-dimensional coordinate system to obtain three-dimensional data. The specific steps are as follows:

[0014] S1.1.1. Capture the three-dimensional point cloud data P = {P 1 , P 2 , …, P i , … P n} of the environment from the lidar, where P i = (x i , y i , z i );

[0015] S1.1.2. Capture the two-dimensional image data I = {i 1 , i 2 , …, i j , … i m} from the camera. Each pixel point i j includes two-dimensional coordinates (u j , v j ) and color information (r j , g j , b j );

[0016] S1.1.3. Detect the obstacle distance D = {d 1 , d 2 , …, d l , … d k} through an ultrasonic sensor, where d l represents the distance from each measurement point to the ultrasonic sensor;

[0017] S1.1.4. Use the camera intrinsic matrix K and depth information to convert each pixel point i of the two-dimensional image data I captured by the camera j into a point i in three-dimensional coordinates j = (x j , y j , z j ). According to the distance d l measured by the ultrasonic sensor and the angle information θ l , convert it into a point u in three-dimensional space l = (x l , y l , z l );

[0018] S1.1.5. According to the three-dimensional point cloud data P = {P 1 , P 2 , …, P i , … P n} and the transformed three-dimensional coordinate point i j = (x j , y j , z j ), and use the ICP algorithm to solve the transformation matrix T to unify the point i j and the point u l into the coordinate system of the lidar through the transformation matrix T, obtaining the new three-dimensional coordinates i' j = (x' j , y' j , z' j ) and u' l = (x' l , y' l , z' l ). Finally, obtain the transformed camera data I' = {i' 1 , i' 2 , …, i' j , … i' m} and the ultrasonic sensor data U' = {u' 1 , u' 2 , … u' k};

[0019] S1.1.6. Integrate the transformed camera data I', ultrasonic sensor data U', and the three-dimensional point cloud data P of the environment captured by the lidar to obtain a fused point cloud data set F in the global three-dimensional coordinate system. final .

[0020] As a further improvement of this technical solution, in S1.1.4, using the camera intrinsic matrix K and depth information, each pixel point i of the two-dimensional image data I captured by the camera j is converted into a point i in three-dimensional coordinates j =(x j , y j , z j ). According to the distance d l measured by the ultrasonic sensor and the angle information θ l , it is converted into a point u in three-dimensional space l =(x l , y l , z l ). The specific method is as follows:

[0021] Camera intrinsic matrix K:

[0022]

[0023] where f x and f y are the focal lengths of the camera; c x and c y are the optical centers;

[0024]

[0025] where h is the height of the ultrasonic sensor from the ground.

[0026] As a further improvement of this technical solution, in S1.1.5, according to the three-dimensional point cloud data P={P 1 , P 2 , …, P i , …P n} and the transformed three-dimensional coordinate point i j =(x j , y j , z j ), and using the ICP algorithm to solve the transformation matrix T, the point i j and the point u l are unified into the coordinate system of the lidar through the transformation matrix T to obtain the new transformed three-dimensional coordinates i' j =(x' j , y' j , z' j ) and u' l=(x', l ,y', l ,z') l ), and finally obtain the transformed camera data I' = {i' 1 , i' 2 , …, i' j , … i' m} and ultrasonic sensor data U' = {u' 1 , u' 2 , … u' k}, and the specific method is as follows:

[0027] Transformation matrix T:

[0028]

[0029] where R is the rotation matrix; t is the translation vector;

[0030]

[0031]

[0032] I' = {i' 1 , i' 2 , …, i' j , … i' m};

[0033] U' = {u' 1 , u' 2 , … u' l , … u' k};

[0034] In step S1.1.6, the transformed camera data I', ultrasonic sensor data U', and the three-dimensional point cloud data P of the environment captured by the lidar are fused to obtain a fused point cloud data set F final in the global three-dimensional coordinate system, and the specific method is as follows:

[0035] F final = P ∪ I' ∪ U';

[0036] where F final is the fused point cloud data set in the global three-dimensional coordinate system, containing the three-dimensional point data of all sensors.

[0037] As a further improvement of this technical solution, the spatial modeling and dynamic update module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional spatial graph model, and uses the dynamic spatio-temporal fusion environment modeling algorithm to update the three-dimensional graph model in real time. The specific steps are as follows:

[0038] S1.2.1. Define the nodes V of the three-dimensional spatial graph model, where each node corresponds to the fused point cloud data set F generated by the data acquisition and fusion module final ={(x 1 ,y 1 ,z 1 ),(x 2 ,y 2 ,z 2 ),…,(x n ,y n ,z n )}, define the edge set E of the three-dimensional spatial graph model. In the fused point cloud data set F final , the Euclidean distance between two points p i and p j is d(p i ,p j );

[0039] S1.2.2. According to the nodes V and the edge set E, construct the initial three-dimensional spatial graph model G(C, E):

[0040] G(V, E) = {V = F, E = {(p i ,p j )∣d(p i ,p j ) < ∈}};

[0041] Among them, d(p i ,p j ) < ∈ means that the Euclidean distance between two points p i and p j is d(p i ,p j ) is less than a threshold ∈, then these two points are considered adjacent points, that is, adjacent nodes, and an edge is established between these two points, where ∈ is a manually preset threshold;

[0042] S1.2.3. The initial feature vector of each point p i is its three-dimensional coordinates Through the dynamic graph convolutional network, aggregate the feature information of neighboring nodes and update the feature vector of the nodes in the three-dimensional spatial graph model to

[0043] S1.2.4. Add time series modeling to the three-dimensional spatial graph model. Through the spatio-temporal graph neural network combined with convolutional operations in the time dimension, capture the spatio-temporal evolution of the features of point p i and update the feature vector of the nodes in the three-dimensional spatial graph model to

[0044] S1.2.5. Calculate the point p iDisplacement Δp in consecutive time steps t and t + 1 i =(Δx i , Δy i , Δz i ). Use the Kalman filter to predict the trajectory of the moving object, incorporate the predicted future position into the three - dimensional spatial graph model, and update the three - dimensional coordinates p i of point p i (t + 1)=p i (t)+Δp i to obtain the real - time updated three - dimensional spatial graph model G'(V', E').

[0045] As a further improvement of this technical solution, in S1.2.3, the initial feature vector of each point p i is its three - dimensional coordinates Aggregate the feature information of neighboring nodes through the dynamic graph convolutional network, and update the feature vector of the nodes in the three - dimensional spatial graph model to The specific method is as follows:

[0046]

[0047] Among them, is the set of adjacent nodes of node i; deg(i) is the degree of node i; deg(j) is the degree of node j; is the time - dependent weight; is the time - dependent bias; σ(·) is the non - linear activation function; is the feature vector of node j at the k - th layer;

[0048] In S1.2.4, add time - series modeling to the three - dimensional spatial graph model. Through the spatio - temporal graph neural network combined with convolutional operations in the time dimension, capture the spatio - temporal evolution of the features of point p i and update the feature vector of the nodes in the three - dimensional spatial graph model to The specific method is as follows:

[0049]

[0050] Among them, T is the time window length; W t,k is the weight at the t - th time step and the k - th layer; b t,k is the bias at the t - th time step and the k - th layer; is the attention weight of node j to node i at time t; is the feature vector of node j at time t and the k - th layer.

[0051] As a further improvement of this technical solution, the vehicle identification and positioning unit includes a license plate recognition module and a three - dimensional positioning module;

[0052] Among them, the license plate recognition module uses a multi-directional camera to recognize the license plate of the target vehicle, and uses optical character recognition technology to recognize the license plate number and extract the visual features of the vehicle, including vehicle color, shape and size. The specific method is as follows:

[0053] C plate = OCR(I plate );

[0054] Among them, I plate is the license plate image captured by the camera; C plate is the license plate number processed by OCR;

[0055] f vehicle = [f color , f shape , f size ;

[0056] Among them, f color is the vehicle color feature; f shape is the shape feature; f size is the size feature; f vehicle is the vehicle visual feature vector;

[0057] Among them, the three-dimensional positioning module integrates the license plate information from the license plate recognition module and the visual features of the vehicle's color, shape and size through a non-linear multi-feature fusion positioning algorithm, and accurately locates the position of the vehicle in the three-dimensional space map model and binds it to the parking space in real time. The specific method is as follows:

[0058] S2.1. Use a non-linear multi-feature fusion network to fuse the license plate information C plate with the vehicle visual feature vector f vehicle into a comprehensive feature vector f fused :

[0059] f fused = σ(W plate ·C plate + W other ·f vehicle + b);

[0060] Among them, W plate is the weight matrix for processing license plate information; W other is the weight matrix for processing visual features; b is the bias vector;

[0061] S2.2. Use the three-dimensional data captured by the camera to obtain the initial position p init of the vehicle in the three-dimensional space map model = (x init , y init , z init);

[0062] Among them, x init is the initial X-axis coordinate of the vehicle in three-dimensional space; y init is the initial Y-axis coordinate of the vehicle in three-dimensional space; z init is the initial Z-axis coordinate of the vehicle in three-dimensional space;

[0063] S2.3. In the three-dimensional space graph model G'(V', E'), based on the comprehensive feature vector f fused , perform feature matching to find the optimal vehicle matching position p opt .

[0064] As a further improvement of this technical solution, the path planning and scheduling unit includes a path planning module and a task scheduling module;

[0065] Among them, the path planning module calculates the optimal path for the mobile charging robot to reach the target vehicle from the current starting position based on the three-dimensional space graph model G'(V', E') and the vehicle matching position p opt . The specific method is as follows:

[0066] S3.1. Let the starting point be p start , and the target position be p opt . Use the A* algorithm to minimize the cost function f(p) = g(p) + h(p);

[0067] Among them, g(p) is the actual path cost from the starting point p start to the current node p; h(p) is the heuristic estimated cost from the current node p to the target position p opt .

[0068] Among them, the task scheduling module allocates and schedules tasks according to the status of multiple mobile charging robots in the system.

[0069] As a further improvement of this technical solution, the interaction and maintenance unit includes a human-computer interaction module, a charging process monitoring module, and a self-maintenance management module;

[0070] Among them, the human-computer interaction module displays the status information of the charging robot and the vehicle in real time, including the current position, charging status, and charging progress; receives various instructions issued by the user, including reservation charging, emergency stop, and adjustment of charging parameters;

[0071] The charging process monitoring module is used to monitor whether the charging current, voltage, and temperature are within a reasonable range. Once an abnormal situation is detected, it immediately takes measures to interrupt the charging, sends an alarm to the user and the staff, and records the detailed data of each charging; it monitors the charging working condition in real time and notifies the automatic plugging and unplugging robot, the car owner, and the service staff in advance to return for gun unplugging operation;

[0072] The self-maintenance management module is used to maintain and manage the charging robot itself when it is idle, and regularly conduct self-detection on each component and system of the charging robot; it monitors the power status of the robot and automatically finds the nearest charging pile for power replenishment operation when it is idle.

[0073] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0074] 1. In the automatic control system of the mobile charging robot based on spatial modeling and visual recognition technology, based on the dynamic spatio-temporal fusion environment modeling algorithm, using the data of lidar, camera, and ultrasonic sensor, a three-dimensional space map model is constructed and updated in real time, which can perceive and model vehicles and obstacles in a complex dynamic environment in real time, maintain the accurate update of the three-dimensional space map model, and ensure the efficient navigation and obstacle avoidance of the robot in a complex environment.

[0075] 2. In the automatic control system of the mobile charging robot based on spatial modeling and visual recognition technology, through the non-linear multi-feature fusion positioning algorithm, accurate identification and three-dimensional positioning of the target vehicle can be realized in a multi-vehicle dense and complex scene. Even when the license plate is partially blocked or the appearance features are similar, the system can still ensure the accurate identification and positioning of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 is the overall flow block diagram of the present invention;

[0077] The meanings of each label in the figure are as follows:

[0078] 1. Environmental space modeling unit; 2. Vehicle identification and positioning unit; 3. Path planning and scheduling unit; 4. Interaction and maintenance unit. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0079] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0080] Embodiment

[0081] Please refer toFigure 1 As shown in the figure, an automatic control system for a mobile charging robot based on spatial modeling and visual recognition technology is provided, including:

[0082] An environmental space modeling unit 1, which is used to perceive the parking lot environment in real time. Through a dynamic spatio-temporal fusion environmental modeling algorithm, using the data captured by lidar, cameras, and ultrasonic sensors, it constructs and updates a three-dimensional space map model in real time. The environmental space modeling unit 1 includes a data acquisition and fusion module and a space modeling and dynamic update module;

[0083] In this embodiment, the data acquisition and fusion module captures three-dimensional point cloud data of the environment from the lidar, captures two-dimensional image data from the cameras, and detects the obstacle distance through the ultrasonic sensors, and uses data fusion technology to fuse the three-dimensional point cloud data, two-dimensional image data, and obstacle distance into a global three-dimensional coordinate system to obtain three-dimensional data;

[0084] The space modeling and dynamic update module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional space map model, and uses a dynamic spatio-temporal fusion environmental modeling algorithm to update the three-dimensional map model in real time;

[0085] The dynamic spatio-temporal fusion environmental modeling algorithm combines a dynamic graph convolutional network and a spatio-temporal graph neural network in the automatic control system of the mobile charging robot based on spatial modeling and visual recognition technology. It can update the three-dimensional space map model in real time when the environment changes, not only dealing with the spatial relationships of static objects, but also being able to capture the spatio-temporal evolution of dynamic objects, ensuring that the environmental model is always consistent with the actual situation;

[0086] When the mobile charging robot is performing tasks, the surrounding environment may change at any time, such as the movement of vehicles, the appearance or disappearance of obstacles. Through the dynamic spatio-temporal fusion environmental modeling algorithm, the system can perceive these changes in real time and immediately update the three-dimensional environmental model, ensuring that path planning and task scheduling are based on the latest environmental information, and improving the adaptability and reaction speed of the system;

[0087] This dynamic spatio-temporal fusion environmental modeling algorithm can efficiently fuse multi-modal data from lidar, cameras, and ultrasonic sensors to construct a unified three-dimensional space map model. It is not just a simple superposition of data, but through a deep learning network to uniformly express and associate the features of different modal data, making the model more robust and accurate;

[0088] In a complex environment, different sensors may provide different information. For example, lidar provides accurate distance information, cameras provide rich visual features, and ultrasonic sensors provide short-range obstacle detection. Through a dynamic spatio-temporal fusion environment modeling algorithm, these multi-modal data can be uniformly processed to generate a more comprehensive and accurate three-dimensional environment model, thereby improving the system's performance in diverse scenarios.

[0089] The dynamic spatio-temporal fusion environment modeling algorithm can predict the future trajectories of dynamic objects and incorporate these prediction results into the three-dimensional spatial map model. By calculating the displacement of objects using the optical flow method and combining it with a Kalman filter for trajectory prediction, the algorithm can pre-sense possible environmental changes and update the model in a timely manner.

[0090] In this embodiment, the data acquisition and fusion module captures three-dimensional point cloud data of the environment from a lidar, captures two-dimensional image data from a camera, and detects the obstacle distances through ultrasonic sensors, and uses data fusion technology to fuse the three-dimensional point cloud data, two-dimensional image data, and obstacle distances into a global three-dimensional coordinate system to obtain three-dimensional data. The specific steps are as follows:

[0091] S1.1.1. Capture three-dimensional point cloud data P = {P 1 , P 2 , …, P i , … P n} of the environment from the lidar, where P i = (x i , y i , z i );

[0092] S1.1.2. Capture two-dimensional image data I = {i 1 , i 2 , …, i j , … i m} from the camera. Each pixel point i j includes two-dimensional coordinates (u j , v j ) and color information (r j , g j , b j );

[0093] S1.1.3. Detect obstacle distances D = {d 1 , d 2 , …, d l , … d k} through ultrasonic sensors, where d l represents the distance of each measurement point from the ultrasonic sensor d l ;

[0094] S1.1.4. Use the camera intrinsic matrix K and depth information to convert each pixel point i of the two-dimensional image data I captured by the camera j into a point i in three-dimensional coordinates j =(x j , y j , z j ). According to the distance d l measured by the ultrasonic sensor and the angle information θ l , convert it into a point u l =(x l , y l , z l ) in three-dimensional space;

[0095] S1.1.5. According to the three-dimensional point cloud data P = {P 1 , P 2 , …, P i , … P n} and the transformed three-dimensional coordinate point i j =(x j , y j , z j ), and use the ICP algorithm to solve the transformation matrix T to unify the point i j and the point u l into the coordinate system of the lidar through the transformation matrix T, obtaining the new transformed three-dimensional coordinates i' j =(x' j , y' j , z' j ) and u' l =(x' l , y' l , z' l ). Finally, obtain the transformed camera data I' = {i' 1 , i' 2 , …, i' j , … i' m} and the ultrasonic sensor data U' = {u' 1 , u' 2 , … u' k};

[0096] S1.1.6. Fuse the transformed camera data I', ultrasonic sensor data U', and the three-dimensional point cloud data P of the environment captured by the lidar to obtain the fused point cloud dataset F in the global three-dimensional coordinate system final .

[0097] In this embodiment S1.1.4, use the camera intrinsic matrix K and depth information to convert each pixel point i of the two-dimensional image data I captured by the camera jThe point i converted to three-dimensional coordinates j =(x j , y j , z j ), according to the distance d l measured by the ultrasonic sensor and the angle information θ l , convert it to the point u l =(x l , y l , z l ) in three-dimensional space. The specific method is as follows:

[0098] The internal camera matrix K:

[0099]

[0100] where f x and f y are the focal lengths of the camera; c x and c y are the optical centers;

[0101]

[0102]

[0103] where h is the height of the ultrasonic sensor from the ground.

[0104] In this embodiment S1.1.5, according to the three-dimensional point cloud data P = {P 1 , P 2 , …, P i , … P n} and the converted three-dimensional coordinate point i j =(x j , y j , z j ), and use the ICP algorithm to solve the transformation matrix T, and unify the point i j and the point u l to the coordinate system of the lidar through the transformation matrix T, and obtain the new three-dimensional coordinates i' j =(x' j , y j ', z' j ) and u' l =(x' l , y' l , z' l ). Finally, obtain the transformed camera data I' = {i' 1 , i' 2 , …, i' j , … i' m} and the ultrasonic sensor data U' = {u' 1,u' 2 ,…u' k}, The specific method is as follows:

[0105] Transformation matrix T:

[0106]

[0107] where R is the rotation matrix; t is the translation vector;

[0108]

[0109] I' = {i' 1 ,i' 2 ,…,i' j ,…i' m};

[0110] U' = {u' 1 ,u' 2 ,…u' l ,…u' k};

[0111] In this embodiment S1.1.6, the transformed camera data I', ultrasonic sensor data U', and the three-dimensional point cloud data P of the environment captured by the lidar are fused to obtain the fused point cloud data set F in the global three-dimensional coordinate system final , The specific method is as follows:

[0112] F final = p ∪ I' ∪ U';

[0113] where F final is the fused point cloud data set in the global three-dimensional coordinate system, containing the three-dimensional point data of all sensors.

[0114] In this embodiment, the spatial modeling and dynamic update module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional space map model, and uses the dynamic spatio-temporal fusion environment modeling algorithm to update the three-dimensional map model in real time. The specific steps are as follows:

[0115] S1.2.1. Define the nodes V of the three-dimensional space map model, and each node corresponds to the fused point cloud data set F generated by the data acquisition and fusion module final = {(x 1 ,y 1 ,z 1 ),(x 2 ,y 2 ,z 2 ),…,(x n ,y n ,z n)}, define the edge set E of the three-dimensional spatial graph model. In the fused point cloud dataset F final For two points p i and p j in it, the Euclidean distance between them is d(p i , p j );

[0116] S1.2.2. Construct the initial three-dimensional spatial graph model G(V, E) according to the nodes V and the edge set E:

[0117] G(V, E) = {V = F, E = {(p i , p j ) | d(p i , p j ) < ∈}};

[0118] Among them, d(p i , p j ) < p means that the Euclidean distance d(p i and p j between two points p i , p j ) is less than a threshold ∈, then these two points are considered adjacent points, that is, adjacent nodes, and an edge is established between these two points, where ∈ is a manually preset threshold;

[0119] The threshold ∈ is used to define whether two points p i and p j are adjacent. If the Euclidean distance between two nodes is less than the threshold ∈, they are considered adjacent, and thus an edge is established between these two nodes in the graph model; by setting the value of ∈, the sparsity of the graph model can be controlled. A smaller threshold ∈ will result in fewer edges in the graph, and the graph model is sparser; while a larger threshold ∈ will increase the number of edges in the graph, and the graph model is denser;

[0120] S1.2.3. The initial feature vector of each point p i is its three-dimensional coordinates Through the dynamic graph convolutional network, aggregate the feature information of neighboring nodes and update the feature vector of the nodes in the three-dimensional spatial graph model to

[0121] S1.2.4. Add time series modeling to the three-dimensional spatial graph model. Through the spatio-temporal graph neural network and the convolutional operation in the time dimension, capture the spatio-temporal evolution of the features of point p i and update the feature vector of the nodes in the three-dimensional spatial graph model to

[0122] S1.2.5. Calculate the displacement Δp of point p i at consecutive time steps t and t + 1 by the optical flow methodi =(Δx i , Δy i , Δz i ), and use the Kalman filter to predict the trajectory of the moving object, incorporate the predicted future position into the three-dimensional spatial graph model, and update point p i 's three-dimensional coordinates p i (t + 1)= p i (t)+Δp i , and obtain the real-time updated three-dimensional spatial graph model G'(V', E').

[0123] Optical flow method is a technique used to estimate the movement of pixel points in an image. By comparing the changes in image brightness in consecutive frames, it estimates the motion vector of each pixel point in the time series. Optical flow can describe the moving speed and direction of objects in the image and is a commonly used method for dynamic object detection in computer vision;

[0124] In the construction and update of the three-dimensional spatial graph model, the optical flow method is used to calculate the displacement of each point in the environment; through the optical flow method, the system can determine the movement of points between two time steps, thereby capturing the dynamic changes of objects in three-dimensional space;

[0125] The Kalman filter is a recursive algorithm used to estimate the state in a dynamic system with noise. Through two steps of prediction and update, using the estimated value and observed value of the current state, it continuously corrects the prediction of the system state and finally obtains the optimal estimate. The Kalman filter performs excellently in tracking dynamic changing systems and is particularly suitable for real-time and online application scenarios;

[0126] The Kalman filter is used to predict the trajectory of the moving object, and based on the displacement of points calculated by the optical flow method in consecutive time steps, the Kalman filter further predicts the future motion trajectory of the object and estimates its position in the next time step. This prediction not only considers the current motion state but also combines the influence of noise and error, providing a more accurate future position estimate. By incorporating these predicted positions into the three-dimensional spatial graph model, the system can achieve real-time dynamic update of the environment, enabling the model to better reflect the actual motion of objects;

[0127] By calculating the actual displacement using the optical flow method and combining it with the trajectory prediction of the Kalman filter, the system can real-time update the three-dimensional spatial graph model and accurately reflect the changes in the dynamic environment;

[0128] In this embodiment S1.2.3, for each point p i 's initial feature vector is its three-dimensional coordinates Aggregate the feature information of neighboring nodes through a dynamic graph convolutional network, and update the feature vector of the nodes in the three-dimensional spatial graph model to The specific method is as follows:

[0129]

[0130] Among them, is the set of adjacent nodes of node i; deg(i) is the degree of node i; deg(j) is the degree of node j; is the time-dependent weight; is the time-dependent bias; σ(·) is the non-linear activation function; is the feature vector of node j at the k-th layer;

[0131] At the beginning of network training, set According to the number of layers and input dimension of the network, assign appropriate initial values to the weight matrices;

[0132] is initialized to a small constant value, so that the network does not have obvious biases at the initial stage of training, and can be flexibly adjusted during the training process;

[0133] In this embodiment S1.2.4, time series modeling is added to the three-dimensional spatial graph model. Through spatio-temporal graph neural networks combined with convolutional operations in the time dimension, capture the spatio-temporal evolution of the features of point p i and update the feature vector of the nodes in the three-dimensional spatial graph model to The specific method is as follows:

[0134]

[0135] Among them, T is the time window length; W t,k is the weight at the t-th time step and the k-th layer; b t,k is the bias at the t-th time step and the k-th layer; is the attention weight of node j to node i at time t; is the feature vector of node j at time t and the k-th layer.

[0136] W tk is set using the Xavier initialization method; b t,k is also initialized to a small constant value;

[0137] Calculate through the self-attention mechanism. First, calculate the feature similarity between node i and node j, and then normalize it to the attention weight through the softmax function.

[0138] It also includes a vehicle identification and positioning unit 2, which is used to identify and position the target vehicle. By integrating a non-linear multi-feature fusion positioning algorithm, the vehicle is accurately positioned in the three-dimensional space map model. The vehicle identification and positioning unit 2 includes a license plate recognition module and a three-dimensional positioning module;

[0139] The integrated non-linear multi-feature fusion positioning algorithm is a technology that realizes accurate positioning by fusing multiple vehicle features (such as license plate information and vehicle visual features). In the automatic control system of mobile charging robots based on spatial modeling and visual recognition technology, in the case of a dense vehicle scene or a large change in environmental light, the algorithm combines license plate information and visual features to achieve accurate positioning of the vehicle. By taking advantage of the different features in different situations and non-linearly fusing the license plate information and vehicle visual features, the recognition and positioning ability of the system in complex environments is effectively improved;

[0140] In this embodiment, the license plate recognition module uses a multi-directional camera to recognize the license plate of the target vehicle, and uses optical character recognition technology to recognize the license plate number and extract the visual features of the vehicle, including vehicle color, shape, and size. The specific method is as follows:

[0141] C plate =OCR(I plate );

[0142] Wherein, I plate is the license plate image captured by the camera; C plate is the license plate number after OCR processing;

[0143] f vehicle =[f color ,f shape ,f size ;

[0144] Wherein, f color is the vehicle color feature; f shape is the shape feature; f size is the size feature; f vehicle is the vehicle visual feature vector;

[0145] In this embodiment, the three-dimensional positioning module integrates the license plate information from the license plate recognition module and the visual features of the vehicle's color, shape, and size through the non-linear multi-feature fusion positioning algorithm, and accurately locates the position of the vehicle in the three-dimensional space map model and binds it to the parking space in real time. The specific method is as follows:

[0146] S2.1. Use the non-linear multi-feature fusion network to fuse the license plate information C plate with the vehicle visual feature vector f vehcle into a comprehensive feature vector f fused:

[0147] f fused = σ(W plate ·C plate + W other ·f vehicle + b);

[0148] Among them, W plate is the weight matrix for processing license plate information; W other is the weight matrix for processing visual features; b is the bias vector;

[0149] W plate and W other are set using the Xavier initialization method; b is also initialized to a small constant value;

[0150] S2.2. Use the initial position p of the vehicle in the three-dimensional space graph model captured by the camera init = (x init , y init , z init );

[0151] Among them, x init is the initial X-axis coordinate of the vehicle in the three-dimensional space; y init is the initial Y-axis coordinate of the vehicle in the three-dimensional space; z init is the initial Z-axis coordinate of the vehicle in the three-dimensional space;

[0152] S2.3. In the three-dimensional space graph model G'(V', E'), based on the comprehensive feature vector f fused , perform feature matching to find the best vehicle matching position p opt .

[0153] It further includes a path planning and scheduling unit 3, and the path planning and scheduling unit 3 is used to plan the optimal driving path according to the three-dimensional space graph model provided by the environmental space modeling unit 1 and the three-dimensional vehicle positioning of the vehicle identification and positioning unit 2, and perform task scheduling. The path planning and scheduling unit 3 includes a path planning module and a task scheduling module;

[0154] Among them, in this embodiment, the path planning module is based on the three-dimensional space graph model G'(V', E') and the vehicle matching position p opt , and calculates the optimal path for the mobile charging robot to reach the target vehicle from the current starting position. The specific method is as follows:

[0155] S3.1. Let the starting point be p start , and the target position be p opt , and use the A* algorithm to minimize the cost function f(p) = g(p) + h(p);

[0156] Among them, g(p) is the actual path cost from the starting point p start to the current node p; h(p) is from the current node p to the target position p opt heuristic estimated cost;

[0157] The A* algorithm is a heuristic algorithm for path planning and graph search, which combines the advantages of the Dijkstra algorithm and heuristic search, and can efficiently find the optimal path from the starting point to the target point. The A algorithm evaluates the comprehensive cost function of each node during the search process, selects the path with the minimum cost for expansion, and finally finds the path with the minimum cost;

[0158] The path planning module combines the real-time updated three-dimensional space map model provided by the environment perception and space modeling unit, dynamically perceives and adjusts the planned path. If new obstacles are detected during driving or the path is blocked, the system will re-plan the path to ensure that the robot reaches the target position smoothly;

[0159] For each step in the planned path, the module will detect the feasibility of the path. If the path is no longer feasible, it will immediately re-plan;

[0160] Among them, the task scheduling module allocates and schedules tasks according to the status of multiple mobile charging robots in the system.

[0161] It also includes an interaction and maintenance unit 4. The interaction and maintenance unit 4 is used for direct communication between the system and users and staff, and monitors and manages the charging process and the self-maintenance of the robot when it is idle. The interaction and maintenance unit 4 includes a human-computer interaction module, a charging process monitoring module, and a self-maintenance management module;

[0162] Among them, in this embodiment, the human-computer interaction module displays the status information of the charging robot and the vehicle in real time, including the current position, charging status, and charging progress; receives various instructions issued by the user, including reservation charging, emergency stop, and adjustment of charging parameters;

[0163] In this embodiment, the charging process monitoring module is used to monitor whether the charging current, voltage, and temperature are within a reasonable range. Once an abnormal situation is detected, it immediately takes measures to interrupt the charging, sends an alarm to the user and staff, and records the detailed data of each charging; monitors the charging working condition in real time, and notifies the automatic plugging and unplugging robot, the car owner, and the service staff to return for unplugging operation in advance;

[0164] In this embodiment, the self-maintenance management module is used to maintain and manage the charging robot itself when the charging robot is idle, and regularly self-checks each component and system of the charging robot; monitors the power status of the robot, and automatically finds the nearest charging pile for power replenishment operation when it is idle.

[0165] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and the descriptions in the specification are only preferred examples of the present invention, and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and all these changes and improvements fall within the scope of the present invention claimed.

Claims

1. The automatic control system of mobile charging robot based on space modeling and visual recognition technology is characterized by: include: An environmental space modeling unit (1), the environmental space modeling unit (1) is used to perceive the parking environment in real time, and to construct and update a three-dimensional space graph model in real time by using data captured by a laser radar, a camera and an ultrasonic sensor through a dynamic spatiotemporal fusion environmental modeling algorithm; A vehicle identification and positioning unit (2), the vehicle identification and positioning unit (2) being used to identify and locate a target vehicle, and to accurately locate the vehicle in a three-dimensional spatial graph model by integrating a nonlinear multi-feature fusion positioning algorithm; A path planning and scheduling unit (3), the path planning and scheduling unit (3) being used to plan an optimal driving path and perform task scheduling according to the three-dimensional space graph model provided by the environment space modeling unit (1) and the three-dimensional vehicle positioning of the vehicle identification and positioning unit (2); An interaction and maintenance unit (4), wherein the interaction and maintenance unit (4) is used for the system to communicate directly with users and staff, and to monitor and manage the charging process and the self-maintenance of the robot when it is idle; The environmental space modeling unit (1) comprises a data acquisition and fusion module and a space modeling and dynamic update module; The data acquisition and fusion module captures the three-dimensional point cloud data of the environment from the laser radar, captures the two-dimensional image data from the camera, and detects the obstacle distance through the ultrasonic sensor, and uses the data fusion technology to fuse the three-dimensional point cloud data, the two-dimensional image data and the obstacle distance into a global three-dimensional coordinate system to obtain three-dimensional data; The spatial modeling and dynamic updating module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional spatial graph model, and uses the dynamic space-time fusion environment modeling algorithm to update the three-dimensional graph model in real time. The specific steps are as follows: S1.2.

1. Define nodes of the three-dimensional spatial graph model , each node corresponds to the fused point cloud dataset generated by the data acquisition fusion module , defines the edge set of the three-dimensional spatial graph model , in the fusion point cloud dataset Two points and The Euclidean distance between ; S1.2.2, according to the node and edge set , construct the initial three-dimensional space graph model : ； in, Indicates two points and The Euclidean distance between Less than a threshold , then the two points are considered to be adjacent points, that is, adjacent nodes, and an edge is established between the two points, where It is a threshold preset by humans; S1.2.

3. For each point The initial eigenvector of is its three-dimensional coordinates , through the dynamic graph convolutional network, the feature information of the neighborhood nodes is aggregated, and the feature vector of the three-dimensional spatial graph model node is updated as ; S1.2.

4. Add time series modeling to the three-dimensional spatial graph model, and capture points by combining the convolution operation of the time dimension with the spatiotemporal graph neural network. The spatiotemporal evolution of features, the feature vector of the node of the three-dimensional spatial graph model is updated as ; S1.2.

5. Calculate points by optical flow method In continuous time steps and Displacement in , use the Kalman filter to predict the trajectory of the moving object, incorporate the predicted future position into the three-dimensional space graph model, and update the point according to the motion trajectory prediction results The three-dimensional coordinates of , get a real-time updated three-dimensional spatial graph model ; The vehicle identification and positioning unit (2) comprises a license plate recognition module and a three-dimensional positioning module; The license plate recognition module uses a multi-directional camera to recognize the license plate of the target vehicle, recognizes the license plate number through optical character recognition technology, and extracts the visual features of the vehicle, including the vehicle color, shape and size; The three-dimensional positioning module integrates the license plate information from the license plate recognition module and the visual features of the vehicle's color, shape and size through a nonlinear multi-feature fusion positioning algorithm, and accurately locates the vehicle's position in the three-dimensional space model, and binds it to the parking space in real time.

2. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 1 is characterized by: The data acquisition and fusion module captures the 3D point cloud data of the environment from the laser radar, captures the 2D image data from the camera, and detects the obstacle distance through the ultrasonic sensor, and uses the data fusion technology to fuse the 3D point cloud data, 2D image data and obstacle distance into a global 3D coordinate system to obtain 3D data. The specific steps are as follows: S1.1.

1. Capturing 3D point cloud data of the environment from LiDAR , ; S1.1.

2. Capturing 2D image data from the camera , each pixel Including 2D coordinates and color information ; S1.1.

3. Detect obstacle distance through ultrasonic sensor ,in Indicates the distance of each measuring point from the ultrasonic sensor; S1.1.

4. Using the camera intrinsic matrix and depth information, converting the two-dimensional image data captured by the camera Each pixel Point converted to 3D coordinates , according to the distance measured by the ultrasonic sensor and angle information , converting it into a point in three-dimensional space ; S1.1.

5. Based on 3D point cloud data And the transformed three-dimensional coordinate points , and use the ICP algorithm to solve the transformation matrix , point and Point Through the transformation matrix Unify to the laser radar coordinate system to obtain the transformed new 3D coordinates and , and finally get the transformed camera data and ultrasonic sensor data ; S1.1.

6. Transformed camera data , ultrasonic sensor data And the 3D point cloud data of the environment captured by LiDAR Fusion is performed to obtain a fused point cloud dataset in the global three-dimensional coordinate system .

3. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 2 is characterized by: In S1.1.4, the camera internal parameter matrix is ​​used and depth information, converting the two-dimensional image data captured by the camera Each pixel Point converted to 3D coordinates , according to the distance measured by the ultrasonic sensor and angle information , converting it into a point in three-dimensional space , the specific method is as follows: Camera intrinsic matrix : ； in, and is the focal length of the camera; and It is the optical center; ； ； in, is the height of the ultrasonic sensor from the ground.

4. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 3 is characterized by: In S1.1.5, based on the three-dimensional point cloud data And the transformed three-dimensional coordinate points , and use the ICP algorithm to solve the transformation matrix , point and Point Through the transformation matrix Unify to the laser radar coordinate system to obtain the transformed new 3D coordinates and , and finally get the transformed camera data and ultrasonic sensor data , the specific method is as follows: Transformation Matrix : ； in, is the rotation matrix; is the translation vector; ； ； ； ； ； The camera data after transformation in S1.1.6 , ultrasonic sensor data And the 3D point cloud data of the environment captured by LiDAR Fusion is performed to obtain a fused point cloud dataset in the global three-dimensional coordinate system , the specific method is as follows: ； in, It is a fused point cloud dataset in a global 3D coordinate system, containing 3D point data from all sensors.

5. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 4 is characterized by: For each point in S1.2.3 The initial eigenvector of is its three-dimensional coordinates , through the dynamic graph convolutional network, the feature information of the neighborhood nodes is aggregated, and the feature vector of the three-dimensional spatial graph model node is updated as , the specific method is as follows: ； in, Is a node The set of adjacent nodes of ; Is a node degree; Is a node degree; is the time-dependent weight; is the time-dependent bias; is a nonlinear activation function; For Node In the The feature vector of the layer; In S1.2.4, time series modeling is added to the three-dimensional spatial graph model, and the convolution operation of the spatiotemporal graph neural network combined with the time dimension is used to capture the points The spatiotemporal evolution of features, the feature vector of the node of the three-dimensional spatial graph model is updated as , the specific method is as follows: ； in, is the time window length; It is time step and The weight of the layer; It is time step and Bias of the layer; Is a node For Node In time The attention weight on ; For Node In time and The feature vector of the layer.

6. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 5 is characterized by: The license plate recognition module uses a multi-directional camera to recognize the license plate of the target vehicle, recognizes the license plate number through optical character recognition technology, and extracts the visual features of the vehicle, including the color, shape and size of the vehicle. The specific method is as follows: ； in, The license plate image captured by the camera; The license plate number after OCR processing; ； in, is the vehicle color feature; is the shape feature; is the size feature; is the vehicle visual feature vector; The three-dimensional positioning module integrates the license plate information from the license plate recognition module and the visual features of the vehicle's color, shape and size through a nonlinear multi-feature fusion positioning algorithm, and accurately locates the position of the vehicle in the three-dimensional space model, and binds it to the parking space in real time. The specific method is as follows: S2.1, using nonlinear multi-feature fusion network, the license plate information and the vehicle visual feature vector Fusion into comprehensive feature vector : ； in, is the weight matrix for processing license plate information; is the weight matrix for processing visual features; is the bias vector; S2.

2. The initial position of the vehicle in the 3D space model using the 3D data captured by the camera ; in, is the initial X-axis coordinate of the vehicle in three-dimensional space; is the initial Y-axis coordinate of the vehicle in three-dimensional space; is the initial Z-axis coordinate of the vehicle in three-dimensional space; S2.

3. Three-dimensional spatial graph model In the paper, based on the comprehensive feature vector , perform feature matching and find the best vehicle matching position .

7. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 1 is characterized by: The path planning and scheduling unit (3) comprises a path planning module and a task scheduling module; The path planning module is based on a three-dimensional spatial graph model. Matching position with vehicle , calculate the optimal path for the mobile charging robot to reach the target vehicle from the current starting position. The specific method is as follows: S3.1, set the starting point to , the target location is , use the A* algorithm to minimize the cost function ; in, From the starting point To the current node The actual path cost; From the current node To the target location Heuristic estimation cost of ; Among them, the task scheduling module allocates and schedules tasks according to the status of multiple mobile charging robots in the system.

8. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 1 is characterized by: The interaction and maintenance unit (4) comprises a human-computer interaction module, a charging process monitoring module and a self-maintenance management module; The human-computer interaction module displays the status information of the charging robot and the vehicle in real time, including the current location, charging status and charging progress; receives various instructions from the user, including charging reservation, emergency stop, and adjustment of charging parameters; The charging process monitoring module is used to monitor whether the charging current, voltage and temperature are within a reasonable range. Once an abnormal situation is detected, measures are taken immediately to interrupt charging, send alarms to users and staff, and record detailed data of each charging; monitor the charging condition in real time, and notify the automatic plug-in robot, the owner and the service personnel in advance to return to perform the gun extraction operation; The self-maintenance management module is used to maintain and manage the charging robot itself when it is idle, and regularly perform self-inspections on the various components and systems of the charging robot; monitor the battery status of the robot, and automatically search for the nearest charging pile for charging when idle.

Images