Automatic management and control system for mobile charging robot based on spatial modeling and visual identification technology

Abstract

The invention relates to the technical field of mobile charging robot automatic management and control, in particular to a mobile charging robot automatic management and control system based on spatial modeling and visual identification technologies. The system comprises an environment space modeling unit which senses and updates a three-dimensional space graph model of a parking lot in real time by using various sensors through a dynamic space-time fusion environment modeling algorithm; the vehicle identification and positioning unit accurately positions a target vehicle in the three-dimensional space map model through a nonlinear multi-feature fusion positioning algorithm; the path planning and scheduling unit plans an optimal path and schedules tasks based on the three-dimensional space map model and the vehicle positioning result; and the interaction and maintenance unit is responsible for communication between the system and the user as well as the staff and monitoring and managing the robot. According to the system, through a dynamic space-time fusion environment modeling algorithm and a nonlinear multi-feature fusion positioning algorithm, dynamic environment space information is accurately sensed and updated, and accurate identification and positioning of a target vehicle in a complex vehicle scene can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic control of mobile charging robots, and in particular to an automatic control system of mobile charging robots based on space modeling and visual recognition technologies. Background Art

[0002] The automatic control system of the mobile charging robot based on spatial modeling and visual recognition technology is designed to model the environment in real time, accurately identify the vehicle position and optimize the execution of charging tasks. Through the dynamic spatiotemporal fusion environment modeling algorithm and the nonlinear multi-feature fusion positioning algorithm, the path planning and task scheduling of the mobile charging robot are controlled to achieve efficient and accurate charging operations on the target vehicle in a complex and dynamic environment.

[0003] The existing mobile charging robot automatic control system usually finds it difficult to update the three-dimensional position of vehicles and obstacles in real time in a dynamically changing parking environment. In addition, due to the narrow and dynamic parking environment and the possibility that the license plate of the target vehicle may be obscured, it will lead to inaccurate vehicle positioning in multi-vehicle dense scenes, inability to quickly avoid obstacles in path planning, and unreasonable charging task scheduling. Therefore, an automatic control system for a mobile charging robot based on spatial modeling and visual recognition technology is designed. Summary of the invention

[0004] The purpose of the present invention is to provide an automatic control system for a mobile charging robot based on spatial modeling and visual recognition technology, so as to solve the problems raised in the above background technology that the parking lot environment is narrow, dynamic and changeable, and the license plate of the target vehicle may be blocked, which may lead to inaccurate vehicle positioning in multi-vehicle dense scenes, inability to quickly avoid obstacles in path planning, and unreasonable charging task scheduling.

[0005] To achieve the above objectives, the present invention provides an automatic control system for a mobile charging robot based on space modeling and visual recognition technology, comprising:

[0006] An environmental space modeling unit, which is used to perceive the parking environment in real time, and to construct and update a three-dimensional space graph model in real time by using data captured by a laser radar, a camera, and an ultrasonic sensor through a dynamic spatiotemporal fusion environmental modeling algorithm;

[0007] It also includes a vehicle identification and positioning unit, which is used to identify and locate the target vehicle, and accurately locate the vehicle in the three-dimensional space graph model by integrating a nonlinear multi-feature fusion positioning algorithm;

[0008] It also includes a path planning and scheduling unit, which is used to plan the optimal driving path and perform task scheduling according to the three-dimensional space graph model provided by the environment space modeling unit and the three-dimensional vehicle positioning of the vehicle identification and positioning unit;

[0009] It also includes an interaction and maintenance unit, which is used for the system to communicate directly with users and staff, and to monitor and manage the charging process and the self-maintenance of the robot when it is idle.

[0010] As a further improvement of the technical solution, the environmental space modeling unit includes a data acquisition and fusion module and a space modeling and dynamic update module;

[0011] The data acquisition and fusion module captures the three-dimensional point cloud data of the environment from the laser radar, captures the two-dimensional image data from the camera, and detects the obstacle distance through the ultrasonic sensor, and uses the data fusion technology to fuse the three-dimensional point cloud data, the two-dimensional image data and the obstacle distance into a global three-dimensional coordinate system to obtain three-dimensional data;

[0012] The spatial modeling and dynamic updating module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional spatial graph model, and uses a dynamic space-time fusion environment modeling algorithm to update the three-dimensional graph model in real time.

[0013] As a further improvement of the technical solution, the data acquisition and fusion module captures the three-dimensional point cloud data of the environment from the laser radar, captures the two-dimensional image data from the camera, and detects the obstacle distance through the ultrasonic sensor, and uses the data fusion technology to fuse the three-dimensional point cloud data, the two-dimensional image data and the obstacle distance into a global three-dimensional coordinate system to obtain three-dimensional data. The specific steps are as follows:

[0014] S1.1.1. The three-dimensional point cloud data P of the environment captured by the laser radar is P = {P 1 ,P 2 ,…,P i ,…P n}, P i =(x i ,y i ,z i );

[0015] S1.1.2, capture two-dimensional image data from the camera I = {i 1 ,i 2 ,…,i j ,…i m}, each pixel i j Including two-dimensional coordinates (u j ,v j ) and color information (r j ,g j ,b j );

[0016] S1.1.3, using ultrasonic sensors to detect obstacle distance D = {d 1 ,d 2 ,…,d l ,…d k}, where d l Indicates the distance of each measuring point from the ultrasonic sensor;

[0017] S1.1.4, using the camera internal parameter matrix K and depth information, each pixel i of the two-dimensional image data I captured by the camera j Point i converted to three-dimensional coordinates j =(x j ,y j ,z j ), according to the distance d measured by the ultrasonic sensor l and angle information θ l , converting it into a point u in three-dimensional space l =(x l ,y l ,z l );

[0018] S1.1.5, according to the three-dimensional point cloud data P = {P 1 ,P 2 ,…,P i ,…P n} and the transformed three-dimensional coordinate point i j =(x j ,y j ,z j ), and use the ICP algorithm to solve the transformation matrix T, and transform point i j and dot u l The transformed matrix T is unified into the laser radar coordinate system to obtain the transformed new three-dimensional coordinate i' j =(x' j ,y' j ,z' j ) and u' l =(x' l ,y' l ,z' l ), and finally obtain the transformed camera data I'={i' 1 ,i' 2 ,…,i' j ,…i' m} and ultrasonic sensor data U'={u' 1 ,u' 2 ,…u' k};

[0019] S1.1.6. Fuse the transformed camera data I', ultrasonic sensor data U' and the three-dimensional point cloud data P of the environment captured by the lidar to obtain a fused point cloud data set F in the global three-dimensional coordinate system. final .

[0020] As a further improvement of the present technical solution, in S1.1.4, the camera internal parameter matrix K and the depth information are used to convert each pixel i of the two-dimensional image data I captured by the camera into j Point i converted to three-dimensional coordinates j =(x j ,y j ,z j ), according to the distance d measured by the ultrasonic sensor l and angle information θ l , converting it into a point u in three-dimensional space l =(x l ,y l ,z l ), the specific method is as follows:

[0021] Camera intrinsic parameter matrix K:

[0022]

[0023] Among them, f x and f y is the focal length of the camera; c x and c y It is the optical center;

[0024]

[0025] Where h is the height of the ultrasonic sensor from the ground.

[0026] As a further improvement of the technical solution, in S1.1.5, according to the three-dimensional point cloud data P={P 1 ,P 2 ,…,P i ,…P n} and the transformed three-dimensional coordinate point i j =(x j ,y j ,z j ), and use the ICP algorithm to solve the transformation matrix T, and transform point i j and dot u l The transformed matrix T is unified into the laser radar coordinate system to obtain the transformed new three-dimensional coordinate i' j =(x' j ,y' j ,z' j ) and u' l=(x' l ,y' l ,z' l ), and finally obtain the transformed camera data I'={i' 1 ,i' 2 ,…,i' j ,…i' m} and ultrasonic sensor data U'={u' 1 ,u' 2 ,…u' k}, the specific method is as follows:

[0027] Transformation matrix T:

[0028]

[0029] Where R is the rotation matrix; t is the translation vector;

[0030]

[0031]

[0032] I'={i' 1 ,i' 2 ,…,i' j ,…i' m};

[0033] U'={u' 1 ,u' 2 ,…u' l ,…u' k};

[0034] In S1.1.6, the transformed camera data I', ultrasonic sensor data U' and the three-dimensional point cloud data P of the environment captured by the laser radar are fused to obtain a fused point cloud data set F in the global three-dimensional coordinate system. final , the specific method is as follows:

[0035] F final =P∪I'∪U';

[0036] Among them, F final It is a fused point cloud dataset in a global 3D coordinate system, containing 3D point data from all sensors.

[0037] As a further improvement of the technical solution, the spatial modeling and dynamic updating module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional spatial graph model, and uses the dynamic spatiotemporal fusion environment modeling algorithm to update the three-dimensional graph model in real time. The specific steps are as follows:

[0038] S1.2.1. Define the nodes V of the three-dimensional spatial graph model. Each node corresponds to the fused point cloud dataset F generated by the data acquisition and fusion module. final ={(x 1 ,y 1 ,z 1 ),(x 2 ,y 2 ,z 2 ),…,(x n ,y n ,z n )}, define the edge set E of the 3D spatial graph model, and fuse the point cloud dataset F final Two points p i and p j The Euclidean distance between them is d(p i ,p j );

[0039] S1.2.2. Based on the node V and edge set E, construct the initial three-dimensional space graph model G(C,E):

[0040] G(V,E)={V=F,E={(p i ,p j )|d(p i ,p j )<∈}};

[0041] Among them, d(p i ,p j )<∈ represents two points p i and p j The Euclidean distance between them is d(p i ,p j ) is less than a threshold ∈, then the two points are considered to be adjacent points, that is, adjacent nodes, and an edge is established between the two points, where ∈ is a manually preset threshold;

[0042] S1.2.3. For each point p i The initial eigenvector of is its three-dimensional coordinates Through the dynamic graph convolutional network, the feature information of the neighborhood nodes is aggregated, and the feature vector of the three-dimensional spatial graph model node is updated as follows:

[0043] S1.2.4. Add time series modeling to the three-dimensional spatial graph model, and capture point p by combining the convolution operation of the spatiotemporal graph neural network with the time dimension. i The spatiotemporal evolution of features, the feature vector of the node of the three-dimensional spatial graph model is updated as

[0044] S1.2.5. Calculate point p by optical flow method iThe displacement Δp at consecutive time steps t and t+1 i =(Δx i ,Δy i ,Δz i ), use the Kalman filter to predict the trajectory of the moving object, incorporate the predicted future position into the three-dimensional space graph model, and update the point p according to the motion trajectory prediction result i The three-dimensional coordinates p i (t+1)=p i (t)+Δp i , and obtain the real-time updated three-dimensional space graph model G'(V',E').

[0045] As a further improvement of this technical solution, for each point p in S1.2.3 i The initial eigenvector of is its three-dimensional coordinates Through the dynamic graph convolutional network, the feature information of the neighborhood nodes is aggregated, and the feature vector of the three-dimensional spatial graph model node is updated as follows: The specific method is as follows:

[0046]

[0047] in, is the set of adjacent nodes of node i; deg(i) is the degree of node i; deg(j) is the degree of node j; is the time-dependent weight; is a time-dependent bias; σ(·) is a nonlinear activation function; is the feature vector of node j at layer k;

[0048] In S1.2.4, time series modeling is added to the three-dimensional spatial graph model, and the point p is captured by combining the convolution operation of the time dimension with the spatiotemporal graph neural network. i The spatiotemporal evolution of features, the feature vector of the node of the three-dimensional spatial graph model is updated as The specific method is as follows:

[0049]

[0050] Where T is the time window length; W t,k is the weight of the t-th time step and the k-th layer; b t,k is the bias of the t-th time step and the k-th layer; is the attention weight of node j to node i at time t; is the feature vector of node j at time t and layer k.

[0051] As a further improvement of the technical solution, the vehicle identification and positioning unit includes a license plate recognition module and a three-dimensional positioning module;

[0052] The license plate recognition module uses a multi-directional camera to recognize the license plate of the target vehicle, recognizes the license plate number through optical character recognition technology, and extracts the visual features of the vehicle, including the color, shape and size of the vehicle. The specific method is as follows:

[0053] C plate =OCR(I plate );

[0054] Among them, I plate is the license plate image captured by the camera; C plate The license plate number after OCR processing;

[0055] f vehicle =[f color ,f shape ,f size ];

[0056] Among them, f color is the vehicle color feature; f shape is the shape feature; f size is the size feature; f vehicle is the vehicle visual feature vector;

[0057] The three-dimensional positioning module integrates the license plate information from the license plate recognition module and the visual features of the vehicle's color, shape and size through a nonlinear multi-feature fusion positioning algorithm, and accurately locates the position of the vehicle in the three-dimensional space model, and binds it to the parking space in real time. The specific method is as follows:

[0058] S2.1, using nonlinear multi-feature fusion network, the license plate information C plate and the vehicle visual feature vector f vehicle Fusion into comprehensive feature vector f fused :

[0059] f fused =σ(W plate ·C plate +W other ·f vehicle + b);

[0060] Among them, W plate is the weight matrix for processing license plate information; W other is the weight matrix for processing visual features; b is the bias vector;

[0061] S2.2. The initial position p of the vehicle in the three-dimensional space model using the three-dimensional data captured by the camera init =(x init ,y init ,z init);

[0062] Among them, x init is the initial X-axis coordinate of the vehicle in three-dimensional space; init is the initial Y-axis coordinate of the vehicle in three-dimensional space; z init is the initial Z-axis coordinate of the vehicle in three-dimensional space;

[0063] S2.3, in the three-dimensional spatial graph model G'(V',E'), based on the comprehensive feature vector f fused , perform feature matching and find the best vehicle matching position p opt .

[0064] As a further improvement of the present technical solution, the path planning and scheduling unit includes a path planning module and a task scheduling module;

[0065] The path planning module is based on the three-dimensional spatial graph model G'(V',E') and the vehicle matching position p opt , calculate the optimal path for the mobile charging robot to reach the target vehicle from the current starting position. The specific method is as follows:

[0066] S3.1, let the starting point be p start , the target position is p opt , use the A* algorithm to minimize the cost function f(p)=g(p)+h(p);

[0067] Among them, g(p) is the start The actual path cost to the current node p; h(p) is the cost from the current node p to the target location p opt Heuristic estimation cost of ;

[0068] Among them, the task scheduling module allocates and schedules tasks according to the status of multiple mobile charging robots in the system.

[0069] As a further improvement of the technical solution, the interaction and maintenance unit includes a human-computer interaction module, a charging process monitoring module and a self-maintenance management module;

[0070] The human-computer interaction module displays the status information of the charging robot and the vehicle in real time, including the current location, charging status and charging progress; receives various instructions from the user, including charging reservation, emergency stop, and adjustment of charging parameters;

[0071] The charging process monitoring module is used to monitor whether the charging current, voltage and temperature are within a reasonable range. Once an abnormal situation is detected, measures are taken immediately to interrupt charging, send alarms to users and staff, and record detailed data of each charging; monitor the charging condition in real time, and notify the automatic plug-in robot, the owner and the service personnel in advance to return to perform the gun extraction operation;

[0072] The self-maintenance management module is used to maintain and manage the charging robot itself when it is idle, and regularly perform self-inspections on the various components and systems of the charging robot; monitor the battery status of the robot, and automatically search for the nearest charging pile for charging when idle.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] 1. In the automatic control system of the mobile charging robot based on spatial modeling and visual recognition technology, based on the dynamic space-time fusion environment modeling algorithm, the data of lidar, camera and ultrasonic sensor are used to build and update the three-dimensional space map model in real time. It can perceive and model vehicles and obstacles in complex dynamic environments in real time, keep the three-dimensional space map model accurately updated, and ensure the robot's efficient navigation and obstacle avoidance in complex environments.

[0075] 2. In the mobile charging robot automatic control system based on spatial modeling and visual recognition technology, a nonlinear multi-feature fusion positioning algorithm is used to achieve accurate recognition and three-dimensional positioning of the target vehicle in dense and complex scenes with multiple vehicles. Even when the license plate is partially blocked or the appearance features are similar, the system can still ensure accurate recognition and positioning of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is the overall flow chart of the present invention;

[0077] The meaning of each number in the figure is:

[0078] 1. Environmental space modeling unit; 2. Vehicle identification and positioning unit; 3. Path planning and scheduling unit; 4. Interaction and maintenance unit. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] Example

[0081] See also Figure 1 As shown, a mobile charging robot automatic control system based on space modeling and visual recognition technology is provided, including:

[0082] An environmental space modeling unit 1, which is used to perceive the parking environment in real time, and to construct and update a three-dimensional space graph model in real time by using data captured by a laser radar, a camera and an ultrasonic sensor through a dynamic spatiotemporal fusion environmental modeling algorithm. The environmental space modeling unit 1 includes a data acquisition and fusion module and a space modeling and dynamic update module;

[0083] In this embodiment, the data acquisition and fusion module captures the three-dimensional point cloud data of the environment from the laser radar, captures the two-dimensional image data from the camera, and detects the obstacle distance through the ultrasonic sensor, and uses the data fusion technology to fuse the three-dimensional point cloud data, the two-dimensional image data and the obstacle distance into a global three-dimensional coordinate system to obtain three-dimensional data;

[0084] The spatial modeling and dynamic updating module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional spatial graph model, and uses a dynamic spatiotemporal fusion environment modeling algorithm to update the three-dimensional graph model in real time;

[0085] The dynamic spatiotemporal fusion environment modeling algorithm combines the dynamic graph convolutional network and the spatiotemporal graph neural network in the mobile charging robot automatic control system based on spatial modeling and visual recognition technology. It can update the three-dimensional spatial graph model in real time when the environment changes. It can not only process the spatial relationship of static objects, but also capture the spatiotemporal evolution of dynamic objects, ensuring that the environmental model is always consistent with the actual situation.

[0086] When the mobile charging robot is performing tasks, the surrounding environment may change at any time, such as the movement of vehicles, the appearance or disappearance of obstacles. Through the dynamic spatiotemporal fusion environment modeling algorithm, the system can perceive these changes in real time and immediately update the three-dimensional environment model to ensure that path planning and task scheduling are based on the latest environmental information, improving the system's adaptability and response speed;

[0087] This dynamic spatiotemporal fusion environment modeling algorithm can efficiently fuse multimodal data from lidar, cameras, and ultrasonic sensors to build a unified three-dimensional spatial graph model. It is not just a simple superposition of data, but a unified expression and association of the features of different modal data through a deep learning network, making the model more robust and accurate.

[0088] In complex environments, different sensors may provide different information, such as lidar providing precise distance information, cameras providing rich visual features, and ultrasonic sensors providing close-range obstacle detection. Through the dynamic spatiotemporal fusion environment modeling algorithm, these multimodal data can be processed uniformly to generate a more comprehensive and accurate three-dimensional environment model, thereby improving the system's performance in diverse scenarios.

[0089] The dynamic spatiotemporal fusion environment modeling algorithm can predict the future trajectory of dynamic objects and integrate these prediction results into the three-dimensional space graph model. The displacement of the object is calculated by the optical flow method, and then the trajectory is predicted by combining the Kalman filter. The algorithm can perceive possible environmental changes in advance and update the model in time.

[0090] In this embodiment, the data acquisition and fusion module captures the 3D point cloud data of the environment from the laser radar, captures the 2D image data from the camera, and detects the obstacle distance through the ultrasonic sensor, and uses the data fusion technology to fuse the 3D point cloud data, 2D image data and obstacle distance into a global 3D coordinate system to obtain 3D data. The specific steps are as follows:

[0091] S1.1.1. The three-dimensional point cloud data P of the environment captured by the laser radar is P = {P 1 ,P 2 ,…,P i ,…P n}, P i =(x i ,y i ,z i );

[0092] S1.1.2, capture two-dimensional image data from the camera I = {i 1 ,i 2 ,…,i j ,…i m}, each pixel i j Including two-dimensional coordinates (u j ,v j ) and color information (r j ,g j ,b j );

[0093] S1.1.3, using ultrasonic sensors to detect obstacle distance D = {d 1 ,d 2 ,…,d l ,…d k}, where d l Indicates the distance d from each measuring point to the ultrasonic sensor l ;

[0094] S1.1.4, using the camera internal parameter matrix K and depth information, each pixel i of the two-dimensional image data I captured by the camera j Point i converted to three-dimensional coordinates j =(x j ,y j ,z j ), according to the distance d measured by the ultrasonic sensor l and angle information θ l , converting it into a point u in three-dimensional space l =(x l ,y l ,z l );

[0095] S1.1.5, according to the three-dimensional point cloud data P = {P 1 ,P 2 ,…,P i ,…P n} and the transformed three-dimensional coordinate point i j =(x j ,y j ,z j ), and use the ICP algorithm to solve the transformation matrix T, and transform point i j and dot u l The transformed matrix T is unified into the laser radar coordinate system to obtain the transformed new three-dimensional coordinate i' j =(x' j ,y' j ,z' j ) and u' l =(x' l ,y' l ,z' l ), and finally obtain the transformed camera data I'={i' 1 ,i' 2 ,…,i' j ,…i' m} and ultrasonic sensor data U'={u' 1 ,u' 2 ,…u' k};

[0096] S1.1.6. Fuse the transformed camera data I', ultrasonic sensor data U' and the three-dimensional point cloud data P of the environment captured by the lidar to obtain a fused point cloud data set F in the global three-dimensional coordinate system. final .

[0097] In this embodiment S1.1.4, the camera intrinsic parameter matrix K and the depth information are used to convert each pixel i of the two-dimensional image data I captured by the camera into jPoint i converted to three-dimensional coordinates j =(x j ,y j ,z j ), according to the distance d measured by the ultrasonic sensor l and angle information θ l , converting it into a point u in three-dimensional space l =(x l ,y l ,z l ), the specific method is as follows:

[0098] Camera intrinsic parameter matrix K:

[0099]

[0100] Among them, f x and f y is the focal length of the camera; c x and c y It is the optical center;

[0101]

[0102]

[0103] Where h is the height of the ultrasonic sensor from the ground.

[0104] In this embodiment S1.1.5, according to the three-dimensional point cloud data P={P 1 ,P 2 ,…,P i ,…P n} and the transformed three-dimensional coordinate point i j =(x j ,y j ,z j ), and use the ICP algorithm to solve the transformation matrix T, and transform point i j and dot u l The transformed matrix T is unified into the laser radar coordinate system to obtain the transformed new three-dimensional coordinate i' j =(x' j ,y j ',z' j ) and u' l =(x' l ,y' l ,z' l ), and finally obtain the transformed camera data I'={i' 1 ,i' 2 ,…,i' j ,…i' m} and ultrasonic sensor data U'={u' 1,u' 2 ,…u' k}, the specific method is as follows:

[0105] Transformation matrix T:

[0106]

[0107] Where R is the rotation matrix; t is the translation vector;

[0108]

[0109] I'={i' 1 ,i' 2 ,…,i' j ,…i' m};

[0110] U'={u' 1 ,u' 2 ,…u' l ,…u' k};

[0111] In this embodiment S1.1.6, the transformed camera data I', ultrasonic sensor data U' and the three-dimensional point cloud data P of the laser radar captured environment are fused to obtain a fused point cloud data set F in the global three-dimensional coordinate system. final , the specific method is as follows:

[0112] F final = p∪I'∪U';

[0113] Among them, F final It is a fused point cloud dataset in a global 3D coordinate system, containing 3D point data from all sensors.

[0114] In this embodiment, the spatial modeling and dynamic updating module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional spatial graph model, and uses the dynamic spatiotemporal fusion environment modeling algorithm to update the three-dimensional graph model in real time. The specific steps are as follows:

[0115] S1.2.1. Define the nodes V of the three-dimensional spatial graph model. Each node corresponds to the fused point cloud dataset F generated by the data acquisition and fusion module. final ={(x 1 ,y 1 ,z 1 ),(x 2 ,y 2 ,z 2 ),…,(x n ,y n ,z n)}, define the edge set E of the three-dimensional spatial graph model. In the fused point cloud dataset F final For two points p i and p j in it, the Euclidean distance between them is d(p i , p j );

[0116] S1.2.2. Construct the initial three-dimensional spatial graph model G(V, E) according to the nodes V and the edge set E:

[0117] G(V, E) = {V = F, E = {(p i , p j ) | d(p i , p j ) < ∈}};

[0118] Among them, d(p i , p j ) < p means that the Euclidean distance d(p i and p j between two points p i , p j ) is less than a threshold ∈, then these two points are considered adjacent points, i.e., adjacent nodes, and an edge is established between these two points, where ∈ is a manually preset threshold;

[0119] The threshold ∈ is used to define whether two points p i and p j are adjacent. If the Euclidean distance between two nodes is less than the threshold ∈, they are considered adjacent, and thus an edge is established between these two nodes in the graph model; by setting the value of ∈, the sparsity of the graph model can be controlled. A smaller threshold ∈ will result in fewer edges in the graph and a sparser graph model; while a larger threshold ∈ will increase the number of edges in the graph and make the graph model denser;

[0120] S1.2.3. The initial feature vector of each point p i is its three-dimensional coordinates Through the dynamic graph convolutional network, aggregate the feature information of neighboring nodes and update the feature vector of the nodes in the three-dimensional spatial graph model to

[0121] S1.2.4. Add time series modeling to the three-dimensional spatial graph model. Through the spatio-temporal graph neural network combined with convolutional operations in the time dimension, capture the spatio-temporal evolution of the features of point p i and update the feature vector of the nodes in the three-dimensional spatial graph model to

[0122] S1.2.5. Calculate the displacement Δp of point p i at consecutive time steps t and t + 1 by the optical flow methodi =(Δx i ,Δy i ,Δz i ), use the Kalman filter to predict the trajectory of the moving object, incorporate the predicted future position into the three-dimensional space graph model, and update the point p according to the motion trajectory prediction result i The three-dimensional coordinates p i (t+1)=p i (t)+Δp i , and obtain the real-time updated three-dimensional space graph model G'(V',E').

[0123] Optical flow is a technique used to estimate the movement of pixels in an image. It estimates the motion vector of each pixel in a time series by comparing the changes in image brightness in consecutive frames. Optical flow can describe the movement speed and direction of objects in an image and is a commonly used dynamic object detection method in computer vision.

[0124] In the construction and updating of the 3D spatial graph model, the optical flow method is used to calculate the displacement of each point in the environment; through the optical flow method, the system can determine the movement of the point between two time steps, thereby capturing the dynamic changes of the object in the 3D space;

[0125] The Kalman filter is a recursive algorithm used to estimate the state in a dynamic system with noise. Through the two steps of prediction and update, the estimated value and observed value of the current state are used to continuously correct the prediction of the system state and finally obtain the optimal estimate. The Kalman filter performs well in tracking dynamically changing systems and is particularly suitable for real-time and online application scenarios.

[0126] The Kalman filter is used to predict the trajectory of a moving object. Based on the displacement of a point in a continuous time step calculated by the optical flow method, the Kalman filter further predicts the future trajectory of the object and estimates its position in the next time step. This prediction not only takes into account the current state of motion, but also combines the effects of noise and error to provide a more accurate estimate of the future position. By incorporating these predicted positions into the three-dimensional spatial graph model, the system can achieve real-time dynamic updates of the environment, allowing the model to better reflect the actual movement of the object.

[0127] By calculating the actual displacement using the optical flow method and combining it with the trajectory prediction of the Kalman filter, the system can update the three-dimensional spatial graph model in real time to accurately reflect changes in the dynamic environment;

[0128] In this embodiment S1.2.3, for each point p i The initial eigenvector of is its three-dimensional coordinates Through the dynamic graph convolutional network, the feature information of the neighborhood nodes is aggregated, and the feature vector of the three-dimensional spatial graph model node is updated as follows: The specific method is as follows:

[0129]

[0130] in, is the set of adjacent nodes of node i; deg(i) is the degree of node i; deg(j) is the degree of node j; is the time-dependent weight; is a time-dependent bias; σ(·) is a nonlinear activation function; is the feature vector of node j at layer k;

[0131] At the beginning of network training, the Xavier initialization method is used to set According to the number of layers and input dimensions of the network, give appropriate initial values ​​to the weight matrix;

[0132] Initialize to a small constant value so that the network has no obvious deviation in the early stage of training, while being able to adjust flexibly during training;

[0133] In this embodiment S1.2.4, time series modeling is added to the three-dimensional spatial graph model, and the point p is captured by combining the convolution operation of the spatiotemporal graph neural network with the time dimension. i The spatiotemporal evolution of features, the feature vector of the node of the three-dimensional spatial graph model is updated as The specific method is as follows:

[0134]

[0135] Where T is the time window length; W t,k is the weight of the t-th time step and the k-th layer; b t,k is the bias of the t-th time step and the k-th layer; is the attention weight of node j to node i at time t; is the feature vector of node j at time t and layer k.

[0136] W tk Use Xavier initialization method for setting; b t,k Also initialized to a small constant value;

[0137] Through the self-attention mechanism, the feature similarity between node i and node j is calculated first, and then normalized to the attention weight through the softmax function.

[0138] It also includes a vehicle identification and positioning unit 2, which is used to identify and locate the target vehicle, and accurately locate the vehicle in the three-dimensional space graph model by integrating a nonlinear multi-feature fusion positioning algorithm. The vehicle identification and positioning unit 2 includes a license plate recognition module and a three-dimensional positioning module;

[0139] The integrated nonlinear multi-feature fusion positioning algorithm is a technology that achieves accurate positioning by integrating multiple vehicle features (such as license plate information and vehicle visual features). In the mobile charging robot automatic control system based on spatial modeling and visual recognition technology, in dense vehicle scenes or when the ambient light changes greatly, the algorithm combines license plate information and visual features to achieve accurate positioning of the vehicle. By taking advantage of the advantages of different features in different situations and nonlinearly fusing license plate information and vehicle visual features, the system's recognition and positioning capabilities in complex environments are effectively improved.

[0140] In this embodiment, the license plate recognition module uses a multi-directional camera to recognize the license plate of the target vehicle, recognizes the license plate number through optical character recognition technology, and extracts the visual features of the vehicle, including the color, shape and size of the vehicle. The specific method is as follows:

[0141] C plate =OCR(I plate );

[0142] Among them, I plate is the license plate image captured by the camera; C plate The license plate number after OCR processing;

[0143] f vehicle =[f color ,f shape ,f size ];

[0144] Among them, f color is the vehicle color feature; f shape is the shape feature; f size is the size feature; f vehicle is the vehicle visual feature vector;

[0145] In this embodiment, the three-dimensional positioning module integrates the license plate information from the license plate recognition module and the visual features of the color, shape and size of the vehicle through a nonlinear multi-feature fusion positioning algorithm, and accurately locates the position of the vehicle in the three-dimensional space model, and binds it to the parking space in real time. The specific method is as follows:

[0146] S2.1, using nonlinear multi-feature fusion network, the license plate information C plate and the vehicle visual feature vector f vehcle Fusion into comprehensive feature vector f fused:

[0147] f fused =σ(W plate ·C plate +W other ·f vehicle + b);

[0148] Among them, W plate is the weight matrix for processing license plate information; W other is the weight matrix for processing visual features; b is the bias vector;

[0149] W plate and W other Use the Xavier initialization method to set; b is also initialized to a small constant value;

[0150] S2.2. The initial position p of the vehicle in the three-dimensional space model using the three-dimensional data captured by the camera init =(x init ,y init ,z init );

[0151] Among them, x init is the initial X-axis coordinate of the vehicle in three-dimensional space; init is the initial Y-axis coordinate of the vehicle in three-dimensional space; z init is the initial Z-axis coordinate of the vehicle in three-dimensional space;

[0152] S2.3, in the three-dimensional spatial graph model G'(V',E'), based on the comprehensive feature vector f fused , perform feature matching and find the best vehicle matching position p opt .

[0153] It also includes a path planning and scheduling unit 3, which is used to plan the optimal driving path and perform task scheduling according to the three-dimensional space graph model provided by the environment space modeling unit 1 and the three-dimensional vehicle positioning of the vehicle identification and positioning unit 2. The path planning and scheduling unit 3 includes a path planning module and a task scheduling module;

[0154] In this embodiment, the path planning module is based on the three-dimensional spatial graph model G'(V', E') and the vehicle matching position p opt , calculate the optimal path for the mobile charging robot to reach the target vehicle from the current starting position. The specific method is as follows:

[0155] S3.1, let the starting point be p start , the target position is p opt , use the A* algorithm to minimize the cost function f(p)=g(p)+h(p);

[0156] Among them, g(p) is the start The actual path cost to the current node p; h(p) is the cost from the current node p to the target location p opt Heuristic estimation cost of ;

[0157] A* algorithm is a heuristic algorithm for path planning and graph search. It combines the advantages of Dijkstra algorithm and heuristic search, and can efficiently find the optimal path from the starting point to the target point. A algorithm evaluates the comprehensive cost function of each node during the search process, selects the path with the minimum cost for expansion, and finally finds the path with the minimum cost.

[0158] The path planning module combines the real-time updated three-dimensional spatial graph model provided by the environmental perception and spatial modeling unit to dynamically perceive and adjust the planned path. If new obstacles are detected or the path is blocked during driving, the system will re-plan the path to ensure that the robot reaches the target location smoothly.

[0159] For each step in the planned path, the module checks the feasibility of the path and immediately replans if the path is no longer feasible;

[0160] Among them, the task scheduling module allocates and schedules tasks according to the status of multiple mobile charging robots in the system.

[0161] It also includes an interaction and maintenance unit 4, which is used for the system to communicate directly with users and staff, and monitor and manage the charging process and the self-maintenance of the robot when it is idle. The interaction and maintenance unit 4 includes a human-computer interaction module, a charging process monitoring module and a self-maintenance management module;

[0162] Among them, in this embodiment, the human-computer interaction module displays the status information of the charging robot and the vehicle in real time, including the current location, charging status and charging progress; receives various instructions from the user, including scheduled charging, emergency stop, and adjustment of charging parameters;

[0163] In this embodiment, the charging process monitoring module is used to monitor whether the charging current, voltage and temperature are within a reasonable range. Once an abnormal situation is detected, measures are taken immediately to interrupt charging, and alarms are issued to users and staff, and detailed data of each charging is recorded; the charging condition is monitored in real time, and the automatic plug-in robot, the owner and the service personnel are notified in advance to return to perform the gun extraction operation;

[0164] In this embodiment, the self-maintenance management module is used to maintain and manage the charging robot itself when it is idle, and to regularly perform self-inspections of the various components and systems of the charging robot; monitor the battery status of the robot, and automatically search for the nearest charging station for charging when idle.

[0165] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. The automatic control system of mobile charging robot based on space modeling and visual recognition technology is characterized by: include: An environmental space modeling unit (1), the environmental space modeling unit (1) is used to perceive the parking environment in real time, and to construct and update a three-dimensional space graph model in real time by using data captured by a laser radar, a camera and an ultrasonic sensor through a dynamic spatiotemporal fusion environmental modeling algorithm; A vehicle identification and positioning unit (2), the vehicle identification and positioning unit (2) is used to identify and locate the target vehicle, and accurately locate the vehicle in a three-dimensional space graph model by integrating a nonlinear multi-feature fusion positioning algorithm; A path planning and scheduling unit (3), wherein the path planning and scheduling unit (3) is used to plan an optimal driving path and perform task scheduling based on the three-dimensional space graph model provided by the environment space modeling unit (1) and the three-dimensional vehicle positioning of the vehicle identification and positioning unit (2); An interaction and maintenance unit (4), wherein the interaction and maintenance unit (4) is used for the system to communicate directly with users and staff, and to monitor and manage the charging process and the self-maintenance of the robot when it is idle.

2. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 1 is characterized by: The environmental space modeling unit (1) comprises a data acquisition and fusion module and a space modeling and dynamic updating module; The data acquisition and fusion module captures the three-dimensional point cloud data of the environment from the laser radar, captures the two-dimensional image data from the camera, and detects the obstacle distance through the ultrasonic sensor, and uses the data fusion technology to fuse the three-dimensional point cloud data, the two-dimensional image data and the obstacle distance into a global three-dimensional coordinate system to obtain three-dimensional data; The spatial modeling and dynamic updating module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional spatial graph model, and uses a dynamic space-time fusion environment modeling algorithm to update the three-dimensional graph model in real time.

3. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 2 is characterized by: The data acquisition and fusion module captures the 3D point cloud data of the environment from the laser radar, captures the 2D image data from the camera, and detects the obstacle distance through the ultrasonic sensor, and uses the data fusion technology to fuse the 3D point cloud data, 2D image data and obstacle distance into a global 3D coordinate system to obtain 3D data. The specific steps are as follows: S1.1.

1. The three-dimensional point cloud data P of the environment captured by the laser radar is P = {P1, P2, ..., P i ,…P n },i∈[1,n],P i =(x i ,y i ,z i ); S1.1.2, capture two-dimensional image data from the camera I = {i1, i2, ..., i j ,…i m }, j∈[1,m], each pixel i j Including two-dimensional coordinates (u j ,v j ) and color information (r j ,g j ,b j ); S1.1.3, using ultrasonic sensors to detect obstacle distance D = {d1, d2, ..., d l ,…d k },l∈[1,k], where d l Indicates the distance of each measuring point from the ultrasonic sensor; S1.1.4, using the camera internal parameter matrix K and depth information, transform each pixel i of the two-dimensional image data I captured by the camera into j Point i converted to three-dimensional coordinates j =(x j ,y j ,z j ), according to the distance d measured by the ultrasonic sensor l and angle information θ l , converting it into a point u in three-dimensional space l =(x l ,y l ,z l ); S1.1.5, according to the three-dimensional point cloud data P = {P1, P2, ..., P i ,…P n } and the transformed three-dimensional coordinate point i j =(x j ,y j ,z j ), and use the ICP algorithm to solve the transformation matrix T, and transform point i j and dot u l The new three-dimensional coordinate i' is obtained by unifying it into the laser radar coordinate system through the transformation matrix T j =(x' j ,y' j ,z' j ) and u' l =(x' l ,y' l ,z' l ), and finally obtain the transformed camera data I'={i'1,i'2,…,i' j ,…i' m } and ultrasonic sensor data U'={u'1,u'2,...,u' l ,…u' k }; S1.1.

6. Fuse the transformed camera data I', ultrasonic sensor data U' and the three-dimensional point cloud data P of the environment captured by the lidar to obtain a fused point cloud data set F in the global three-dimensional coordinate system. final .

4. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 3 is characterized by: In S1.1.4, the camera internal parameter matrix K and depth information are used to convert each pixel i of the two-dimensional image data I captured by the camera into j Point i converted to three-dimensional coordinates j =(x j ,y j ,z j ), according to the distance d measured by the ultrasonic sensor l and angle information θ l , converting it into a point u in three-dimensional space l =(x l ,y l ,z l ), the specific method is as follows: Camera intrinsic parameter matrix K: Among them, f x and f y is the focal length of the camera; c x and c y It is the optical center; Where h is the height of the ultrasonic sensor from the ground.

5. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 3 is characterized by: In S1.1.5, according to the three-dimensional point cloud data P = {P1, P2, ..., P i ,…P n } and the transformed three-dimensional coordinate point i j =(x j ,y j ,z j ), and use the ICP algorithm to solve the transformation matrix T, and transform point i j and dot u l The new three-dimensional coordinate i' is obtained by unifying it into the laser radar coordinate system through the transformation matrix T j =(x' j ,y' j ,z' j ) and u' l =(x' l ,y' l ,z' l ), and finally obtain the transformed camera data I'={i'1,i'2,…,i' j ,…i' m } and ultrasonic sensor data U'={u'1,u'2,...,u' l ,…u' k }, the specific method is as follows: Transformation matrix T: Where R is the rotation matrix; t is the translation vector; I’={i’1,i’2,…,i’ j ,…i’ m }; U'={u'1,u'2,…u' l ,…in' k }; In S1.1.6, the transformed camera data I', ultrasonic sensor data U' and the three-dimensional point cloud data P of the environment captured by the laser radar are fused to obtain a fused point cloud data set F in the global three-dimensional coordinate system. final , the specific method is as follows: F final =P∪I’∪U’; Among them, F final It is a fused point cloud dataset in a global 3D coordinate system, containing 3D point data from all sensors.

6. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 2 is characterized by: The spatial modeling and dynamic updating module uses the three-dimensional data generated by the data acquisition and fusion module to construct a dynamically updated three-dimensional spatial graph model, and uses the dynamic space-time fusion environment modeling algorithm to update the three-dimensional graph model in real time. The specific steps are as follows: S1.2.

1. Define the nodes V of the three-dimensional spatial graph model. Each node corresponds to the fused point cloud dataset F generated by the data acquisition and fusion module. final ={(x1,y1,z1),(x2,y2,z2),…,(x n ,y n ,z n )}, define the edge set E of the 3D spatial graph model, and fuse the point cloud dataset F final Two points p i and p j The Euclidean distance between them is d(p i ,p j ); S1.2.

2. Based on the node V and edge set E, construct the initial three-dimensional space graph model G(V,E): G(V,E)={V=F,E={(p i ,p j )∣d(p i ,p j )<∈}}; Among them, d(p i ,p j )<∈ represents two points p i and p j The Euclidean distance between them is d(p i ,p j ) is less than a threshold ∈, then the two points are considered to be adjacent points, that is, adjacent nodes, and an edge is established between the two points, where ∈ is a manually preset threshold; S1.2.

3. For each point p i The initial eigenvector of is its three-dimensional coordinates Through the dynamic graph convolutional network, the feature information of the neighborhood nodes is aggregated, and the feature vector of the three-dimensional spatial graph model node is updated as follows: S1.2.

4. Add time series modeling to the three-dimensional spatial graph model, and capture point p by combining the convolution operation of the spatiotemporal graph neural network with the time dimension. i The spatiotemporal evolution of features, the feature vector of the node of the three-dimensional spatial graph model is updated as S1.2.

5. Calculate point p by optical flow method i The displacement Δp at consecutive time steps t and t+1 i =(Δx i ,Δy i ,Δz i ), use the Kalman filter to predict the trajectory of the moving object, incorporate the predicted future position into the three-dimensional space graph model, and update the point p according to the motion trajectory prediction result i The three-dimensional coordinates p i (t+1)=p i (t)+Δp i , and obtain the real-time updated three-dimensional space graph model G'(V',E').

7. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 6 is characterized by: For each point p in S1.2.3 i The initial eigenvector of is its three-dimensional coordinates Through the dynamic graph convolutional network, the feature information of the neighborhood nodes is aggregated, and the feature vector of the three-dimensional spatial graph model node is updated as follows: The specific method is as follows: in, is the set of adjacent nodes of node i; deg(i) is the degree of node i; deg(j) is the degree of node j; is the time-dependent weight; is a time-dependent bias; σ(·) is a nonlinear activation function; is the feature vector of node j at layer k; In S1.2.4, time series modeling is added to the three-dimensional spatial graph model, and the point p is captured by combining the convolution operation of the time dimension with the spatiotemporal graph neural network. i The spatiotemporal evolution of features, the feature vector of the node of the three-dimensional spatial graph model is updated as The specific method is as follows: Where T is the time window length; W t,k is the weight of the t-th time step and the k-th layer; b t,k is the bias of the t-th time step and the k-th layer; is the attention weight of node j to node i at time t; is the feature vector of node j at time t and layer k.

8. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 6 is characterized by: The vehicle identification and positioning unit (2) comprises a license plate recognition module and a three-dimensional positioning module; The license plate recognition module uses a multi-directional camera to recognize the license plate of the target vehicle, recognizes the license plate number through optical character recognition technology, and extracts the visual features of the vehicle, including the color, shape and size of the vehicle. The specific method is as follows: C plate =OCR(I plate ); Among them, I plate is the license plate image captured by the camera; C plate The license plate number after OCR processing; f vehicle =[f color ,f shape ,f size ]; Among them, f color is the vehicle color feature; f shape is the shape feature; f size is the size feature; f vehicle is the vehicle visual feature vector; The three-dimensional positioning module integrates the license plate information from the license plate recognition module and the visual features of the vehicle's color, shape and size through a nonlinear multi-feature fusion positioning algorithm, and accurately locates the position of the vehicle in the three-dimensional space model, and binds it to the parking space in real time. The specific method is as follows: S2.1, using nonlinear multi-feature fusion network, the license plate information C plate and the vehicle visual feature vector f vehicle Fusion into comprehensive feature vector f fused : f fused =σ(W plate ·C plate +W other ·f vehicle +b); Among them, W plate is the weight matrix for processing license plate information; W other is the weight matrix for processing visual features; b is the bias vector; S2.

2. The initial position p of the vehicle in the three-dimensional space model using the three-dimensional data captured by the camera init =(x init ,y init ,z init ); Among them, x init is the initial X-axis coordinate of the vehicle in three-dimensional space; init is the initial Y-axis coordinate of the vehicle in three-dimensional space; z init is the initial Z-axis coordinate of the vehicle in three-dimensional space; S2.3, in the three-dimensional spatial graph model G'(V',E'), based on the comprehensive feature vector f fused , perform feature matching and find the best vehicle matching position p opt .

9. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 1 is characterized by: The path planning and scheduling unit (3) comprises a path planning module and a task scheduling module; The path planning module is based on the three-dimensional spatial graph model G'(V',E') and the vehicle matching position p opt , calculate the optimal path for the mobile charging robot to reach the target vehicle from the current starting position. The specific method is as follows: S3.1, let the starting point be p start , the target position is p opt , use the A* algorithm to minimize the cost function f(p)=g(p)+h(p); Among them, g(p) is the start The actual path cost to the current node p; h(p) is the cost from the current node p to the target location p opt Heuristic estimation cost of ; Among them, the task scheduling module allocates and schedules tasks according to the status of multiple mobile charging robots in the system.

10. The mobile charging robot automatic control system based on space modeling and visual recognition technology according to claim 1 is characterized by: The interaction and maintenance unit (4) comprises a human-computer interaction module, a charging process monitoring module and a self-maintenance management module; The human-computer interaction module displays the status information of the charging robot and the vehicle in real time, including the current location, charging status and charging progress; receives various instructions from the user, including charging reservation, emergency stop, and adjustment of charging parameters; The charging process monitoring module is used to monitor whether the charging current, voltage and temperature are within a reasonable range. Once an abnormal situation is detected, measures are taken immediately to interrupt charging, send alarms to users and staff, and record detailed data of each charging; monitor the charging condition in real time, and notify the automatic plug-in robot, the owner and the service personnel in advance to return to perform the gun extraction operation; The self-maintenance management module is used to maintain and manage the charging robot itself when it is idle, and regularly perform self-inspections on the various components and systems of the charging robot; monitor the battery status of the robot, and automatically search for the nearest charging pile for charging when idle.

Images