System and method for detecting abnormity of robot in battery swap station based on visual identification

Abstract

The invention discloses a power swap station robot anomaly detection system and method based on visual identification, and belongs to the technical field of artificial intelligence. The method comprises the following steps: continuously acquiring video data in a battery replacement process, preprocessing, and transmitting to a local database and a cloud; recognizing the robot and components thereof in the video data, and marking key parts of the robot; determining the posture of the robot, and obtaining the joint position and the motion trail of the robot; extracting appearance features of the robot by using a convolutional neural network, and extracting key point descriptors SIFT for matching and tracking; a behavior model based on deep learning is trained, and time sequence features of normal operation are learned; an anomaly detection model is constructed, and low-dimensional representation of normal operation of the robot is learned; during operation, abnormality is detected by comparing reconstruction errors of actual operation and normal operation; and constructing a correlation model between the charging pile of the battery swap station and the robot, and adjusting the parameters of the charging pile and the operation of the robot.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a system and method for detecting abnormalities in a battery swap station robot based on visual recognition. Background Art

[0002] With the popularity of electric vehicles, battery swap stations, as important facilities for rapid energy replenishment of electric vehicles, have attracted more and more attention. In battery swap stations, robots are usually responsible for battery replacement operations, and their efficiency and reliability directly affect the smooth progress of the battery swap process. However, various abnormal situations may occur during the battery swap process, such as mechanical failures, operational errors, electrical abnormalities, etc., which pose challenges to the safety and efficiency of battery swap stations. Therefore, how to achieve real-time monitoring and abnormality detection of the operation process of battery swap station robots has become an urgent problem to be solved.

[0003] Existing anomaly detection methods are mostly based on fixed thresholds or simple rule judgments, which cannot adapt to complex and changeable actual working conditions and are prone to false alarms or missed alarms. Existing systems often lack intelligent analysis capabilities and cannot fully utilize a large amount of historical data and real-time data for in-depth analysis and prediction, resulting in insufficient accuracy and timeliness of anomaly detection. After detecting an anomaly, the existing system has a slow response speed and often requires manual intervention, which is difficult to adjust and correct in time, affecting the efficiency of battery replacement. Summary of the invention

[0004] The purpose of the present invention is to provide a battery swap station robot anomaly detection system and method based on visual recognition to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A method for detecting abnormality of a battery swap station robot based on visual recognition, the method comprising:

[0007] S100, continuously collecting video data during the battery replacement process through a camera, performing preprocessing, and transmitting the preprocessed video data to a local database and the cloud;

[0008] S200, using the YOLO target detection algorithm to identify the robot and its components in the video data and mark the key parts of the robot; using OpenPose to determine the robot's posture, obtain the robot's joint position and motion trajectory; using a convolutional neural network to extract the robot's appearance features, and extract the key point descriptor SIFT for matching and tracking;

[0009] S300, obtain the video data of the normal operation of the historical robot, combine it with the charging pile parameters of the corresponding battery swap station, train the LSTM behavior model based on deep learning, and learn the time series characteristics of normal operation; build an anomaly detection model based on Autoencoder to learn the low-dimensional representation of the normal operation of the robot; at runtime, detect anomalies by comparing the reconstruction error between the actual operation and the normal operation;

[0010] S400. When an abnormality is detected, a random forest algorithm is used to construct an association model between the charging piles of the battery swap station and the robot. The charging pile parameters of the battery swap station and the operation of the robot are adjusted according to the abnormality detection results and the association model until the abnormality in the battery swap process is eliminated.

[0011] According to step S100, the camera continuously collects video data during the operation of the battery swap station, records all operation processes of the battery swap robot, and transmits the collected video data to a local storage device in real time;

[0012] Extract each frame of image from the continuous video stream and perform single-frame image processing; use Gaussian filtering to denoise the image; use histogram equalization to improve the image contrast; use adaptive histogram equalization to prevent distortion caused by excessive image enhancement; correct camera distortion and perform color correction;

[0013] The preprocessed image data is stored in the local database, and the locally stored data is backed up regularly; the preprocessed video data is transmitted to the cloud storage platform via the network, and the transmission protocol HTTPS is used to ensure the security of data transmission.

[0014] According to step S200, the YOLO target detection algorithm is used for detection, the v5 model version of YOLO is selected, and the pre-trained model weights and configuration files are downloaded; the training data is annotated using an annotation tool, and the annotation content includes various key parts of the robot, and the key parts include a robotic arm, a battery interface, a robot camera, a sensor, a display screen, and an indicator light; the key parts of the robot are annotated on the image, and different colored borders and labels are used to distinguish them; the annotated image is stored in a local database and uploaded to the cloud.

[0015] YOLOv5 is a widely used object detection model. The core idea of ​​YOLOv5 is to predict multiple bounding boxes and category probabilities in a single forward pass, thereby achieving real-time object detection.

[0016] According to step S200, OpenPose is downloaded and installed, and a pre-trained OpenPose model is loaded; the pre-processed image is input into the OpenPose model, and OpenPose outputs the joint positions of the robot in each frame of the image;

[0017] OpenPose is an open source library for pose estimation that can extract the joint positions of a human or robot from an image and generate a skeleton model.

[0018] For each frame, OpenPose outputs the joint position (x i ,y i ), where i represents different joints; the posture is determined by the joint position. The robot arm has n joints, and the joint position is represented by P =

[0019] {(x1,y1),(x2,y2),...,(x n ,y n )}; For each joint i, the position at time t is (x i (t),y i (t)), the motion trajectory is expressed as the trajectory of joint position changes over time; the mathematical representation of the motion trajectory is T i =

[0020] {(x i (t0),y i (t0)),(x i (t1),y i (t1)),...,(x i (t m ),y i (t m ))}, where t0, t1, ..., t m are different time points, m is the total number of time points; the Kalman filter is used to smooth the joint motion trajectory.

[0021] According to step S200, the pre-trained convolutional neural network model VGG16 is selected to extract image features; the key points are detected in the image using the SIFT algorithm, and for each key point k i , calculate its position (x i ,y i ) and scale s i ;

[0022] For each key point, calculate its feature descriptor d i , the descriptor contains the local feature information of the area around the key point, and constructs the descriptor set D of all key points: D = {d1, d2, ..., d n}, where d i is the key point k i feature descriptors;

[0023] By calculating the Euclidean distance (d i ,d j), find the matching key point pairs between the current frame and the previous frame: distance(d i ,d j )=||d i -d j ||, where d j is the key point k i The feature descriptor of the previous frame; using the nearest neighbor matching algorithm, find the matching point of each key point;

[0024] For the matched key point pairs, calculate the motion vector:

[0025] Δx i = x′ i -x i , Δy i =y′ i -y i ;

[0026] Where Δx i and Δy i are the horizontal and vertical coordinates of the key point's motion vector between images, respectively, x′ i and y′ i They are respectively the key points k in the previous frame i The horizontal and vertical coordinates of the matched key points; update the position and descriptor of the key points, track the key points according to the motion vector, and form the operation data of the robot.

[0027] According to step S300, the video data of the normal operation of the historical robot and the corresponding parameter data of the charging pile are obtained, the video data is extracted frame by frame, and each frame of the image is timestamped to correspond to the charging pile parameter data; the pre-trained convolutional neural network model VGG16 is selected to extract image features to obtain image feature vectors, the charging pile parameter data is normalized, and the normalized charging pile parameter data is combined with the image feature vector of the corresponding timestamp to form a time series feature; the image feature vectors of the continuous timestamps and the charging pile parameter data are combined to construct a time series feature set, and the time series feature set is marked as a sample of normal operation;

[0028] The time series feature set is divided into a training set and a validation set. The LSTM model is selected as the learning model for the time series features. The LSTM network is constructed, with the input being the time series features and the output being the predicted value of whether the operation is normal. The training set is used for model training and the parameters of the LSTM model are optimized. The mean square error is selected as the loss function and Adam is selected as the optimization algorithm. The validation set is used to evaluate the model performance and the model parameters and hyperparameters are adjusted to ensure that the model can identify the time series features of the normal operation of the robot.

[0029] LSTM is a special recurrent neural network suitable for processing and predicting time series data. LSTM can effectively solve the problems of gradient disappearance and gradient explosion in long sequences.

[0030] During the battery replacement process, the image feature vector and charging pile parameter data are extracted in real time to form the feature sequence of the current time; the feature sequence of the current time is input into the trained LSTM model to predict normal operation.

[0031] According to step S300, an Autoencoder model is selected, wherein the Autoencoder model is composed of an encoder and a decoder, and is used to learn a low-dimensional representation of input data; the input high-dimensional time series features are compressed into a low-dimensional representation, and the original high-dimensional features are reconstructed from the low-dimensional representation, with the goal of making the reconstructed features similar to the original features; the mean square error is used to measure the difference between the reconstructed features and the original features; the training set is used to train the model, and the parameters of the Autoencoder are optimized so that it can learn a low-dimensional representation of normal operation; the validation set is used to evaluate the model performance to ensure that the model can reconstruct the time series features of normal operation;

[0032] Autoencoder is a neural network model for unsupervised learning that achieves feature learning and data reconstruction by compressing and decompressing data;

[0033] The feature sequence of the current time is input into the trained Autoencoder model for encoding and reconstruction to obtain the reconstructed feature sequence, and the reconstruction error between the reconstructed feature sequence and the original feature sequence is calculated; the reconstruction error is used to determine whether the current operation is normal, and a threshold is set. If the reconstruction error exceeds the threshold, it is marked as an abnormal operation, and the time point when the abnormality occurred and the corresponding feature data are recorded; when an abnormality occurs, the maintenance personnel are notified, and the data of the abnormal operation is stored in the local database and the cloud.

[0034] According to step S400, charging pile parameters and robot operation data are collected, and the correlation between the charging pile parameters and the robot operation is determined using a correlation analysis method. A correlation model between the charging pile parameters and the robot operation is constructed using a random forest algorithm. The input of the correlation model is the charging pile parameter P, and the output is the robot operation parameter R: R=f(P), where f is the correlation model;

[0035] Extract the charging pile parameters and robot operation parameters during the abnormality, input the charging pile parameters during the abnormality into the association model to predict the corresponding robot operation parameters; compare the predicted robot operation parameters with the robot operation parameters during the abnormality to find the differences; formulate the charging pile parameter adjustment strategy and the robot operation adjustment strategy according to the prediction results of the association model; monitor the adjusted operation and detect whether there is an abnormality in real time. If a new anomaly is detected, repeat the adjustment process until the reconstruction error is lower than the threshold.

[0036] When adjusting the charging pile parameters, it is necessary to match the parameter range required for normal operation. The charging pile parameter adjustment algorithm is as follows:

[0037] P adjusted =P abnormal +ΔP;

[0038] Among them, P adjusted is the charging pile parameter after adjustment, P abnormal is the charging pile parameter when abnormal, ΔP is the charging pile parameter adjustment increment, and the correlation model f is derived by reverse deduction. -1 Sure;

[0039] When adjusting the robot operating parameters, the predicted robot operating parameters are used as the adjusted charging pile parameters; during the abnormality detection and parameter adjustment process, maintenance personnel are notified in real time to provide abnormal information and adjustment suggestions.

[0040] A battery swap station robot anomaly detection system based on visual recognition, comprising:

[0041] Data acquisition module: including: video acquisition unit, video preprocessing unit and data transmission unit; the video acquisition unit is responsible for continuously collecting video data during the battery replacement process through the camera, the video preprocessing unit is responsible for preprocessing the collected video, and the data transmission unit is responsible for transmitting the preprocessed video data to the local database and the cloud;

[0042] Feature extraction module: including: target detection unit, posture estimation unit, appearance feature extraction unit and key point tracking unit; the target detection unit uses the YOLO target detection algorithm to identify the robot and its components in the video data and mark the key parts of the robot. The posture estimation unit uses OpenPose to determine the posture of the robot, obtain the joint position and motion trajectory of the robot. The appearance feature extraction unit uses a convolutional neural network to extract the appearance features of the robot. The key point tracking unit extracts the key point descriptor SIFT for matching and tracking.

[0043] Anomaly detection module: including: behavior model training unit, anomaly detection model construction unit and real-time anomaly detection unit; wherein the behavior model training unit obtains the video data of the historical normal operation of the robot, combines the charging pile parameters of the corresponding battery swap station, trains the behavior model LSTM based on deep learning, and learns the time series characteristics of normal operation; the anomaly detection model construction unit constructs an anomaly detection model based on Autoencoder, learns the low-dimensional representation of the normal operation of the robot, and the real-time anomaly detection unit detects anomalies by comparing the reconstruction error between the actual operation and the normal operation during operation;

[0044] Abnormality handling module: including: an association model construction unit and a parameter adjustment unit; wherein, when the association model construction unit detects an abnormality, it uses a random forest algorithm to construct an association model between the charging pile of the battery swap station and the robot, and the parameter adjustment unit adjusts the charging pile parameters of the battery swap station and the operation of the robot according to the abnormality detection result and the association model until the abnormal situation in the battery swap process is eliminated.

[0045] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0046] The present invention adopts advanced deep learning algorithms, such as YOLO target detection, OpenPose posture estimation, convolutional neural network, LSTM behavior model and Autoencoder anomaly detection model, to realize intelligent detection and real-time analysis of the robot operation process, greatly improving the accuracy and timeliness of anomaly detection.

[0047] After detecting an abnormality, the present invention constructs an association model between the charging pile and the robot through a random forest algorithm, automatically adjusts the charging pile parameters and the robot operation, quickly eliminates the abnormality, and ensures the smooth progress of the battery replacement process.

[0048] The whole process of the present invention is highly automated, from abnormality detection to parameter adjustment and response, without the need for human intervention, thus greatly improving the efficiency and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 It is a schematic diagram of the steps of a method for detecting abnormality of a battery swap station robot based on visual recognition according to the present invention;

[0051] Figure 2 It is a system structure diagram of a battery swap station robot anomaly detection system based on visual recognition in the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] See also Figure 1 and Figure 2 , the present invention provides a technical solution:

[0054] According to one embodiment of the present invention, Figure 1 A method for detecting abnormality of a battery swap station robot based on visual recognition is shown in a schematic diagram of steps. A method for detecting abnormality of a battery swap station robot based on visual recognition is shown in a schematic diagram of steps. The method comprises:

[0055] S100, continuously collecting video data during the battery replacement process through a camera, performing preprocessing, and transmitting the preprocessed video data to a local database and the cloud;

[0056] S200, using the YOLO target detection algorithm to identify the robot and its components in the video data and mark the key parts of the robot; using OpenPose to determine the robot's posture, obtain the robot's joint position and motion trajectory; using a convolutional neural network to extract the robot's appearance features, and extract the key point descriptor SIFT for matching and tracking;

[0057] S300, obtain the video data of the normal operation of the historical robot, combine it with the charging pile parameters of the corresponding battery swap station, train the LSTM behavior model based on deep learning, and learn the time series characteristics of normal operation; build an anomaly detection model based on Autoencoder to learn the low-dimensional representation of the normal operation of the robot; at runtime, detect anomalies by comparing the reconstruction error between the actual operation and the normal operation;

[0058] S400. When an abnormality is detected, a random forest algorithm is used to construct an association model between the charging piles of the battery swap station and the robot. The charging pile parameters of the battery swap station and the operation of the robot are adjusted according to the abnormality detection results and the association model until the abnormality in the battery swap process is eliminated.

[0059] According to step S100, the camera continuously collects video data during the operation of the battery swap station, records all operating processes of the battery swap robot, and the collected video data is transmitted to the local storage device in real time; in this embodiment, a high-definition camera is used to continuously collect video data during the operation of the battery swap station, record all operating processes of the battery swap robot, and the video resolution is 1920x1080, and the frame rate is 60 frames per second.

[0060] Extract each frame of image from the continuous video stream and perform single-frame image processing; use Gaussian filtering to denoise the image, with parameter σ = 1.5; use histogram equalization to improve the contrast of the image, and perform global histogram equalization on the image; use adaptive histogram equalization to prevent distortion caused by excessive image enhancement, using CLAHE, ClipLimit = 2.0, TileGridSize = (8,8); correct the distortion of the camera, perform color correction, use the camera's intrinsic parameter matrix and distortion coefficient to correct the image, and adjust the white balance and color of the image;

[0061] The preprocessed image data is stored in the local database, and the locally stored data is backed up regularly; the preprocessed video data is transmitted to the cloud storage platform via the network, and the transmission protocol HTTPS is used to ensure the security of data transmission.

[0062] According to step S200, use the YOLO target detection algorithm for detection, select the v5 model version of YOLO, and download the pre-trained model weights and configuration files; use the labeling tool LabelImg to label the training data, and the labeling content includes various key parts of the robot, and the key parts include the robotic arm, battery interface, robot camera, sensor, display screen and indicator light; mark the key parts of the robot on the image, and use different colored borders and labels to distinguish them; store the labeled image in the local database and upload it to the cloud.

[0063] The YOLOv5 model was trained and evaluated on the test set, with a mean average precision (mAP) of 0.85. Detection accuracy: robotic arm: 0.88, battery interface: 0.83, robot camera: 0.84, sensor: 0.82, display: 0.85, indicator light: 0.86.

[0064] According to step S200, OpenPose is downloaded and installed, and a pre-trained OpenPose model is loaded; the pre-processed image is input into the OpenPose model, and OpenPose outputs the joint positions of the robot in each frame of the image;

[0065] OpenPose is an open source library for pose estimation that can extract the joint positions of a human or robot from an image and generate a skeleton model.

[0066] For each frame, OpenPose outputs the joint position (x i ,y i ), where i represents different joints; the posture is determined by the joint position. The robot arm has n joints, and the joint position is represented by P =

[0067] {(x1,y1),(x2,y2),...,(x n ,y n )}; For each joint i, the position at time t is (x i (t),y i (t)), the motion trajectory is expressed as the trajectory of joint position changes over time; the mathematical representation of the motion trajectory is T i =

[0068] {(x i (t0),y i (t0)),(x i (t1),y i (t1)),...,(x i (t m ),y i (t m ))}, where t0, t1, ..., t m are different time points, and m is the total number of time points; the joint motion trajectory is smoothed using a Kalman filter with filter parameters R=1, Q=0.1.

[0069] According to step S200, the pre-trained convolutional neural network model VGG16 is selected to extract image features; the key points are detected in the image using the SIFT algorithm, and for each key point k i , calculate its position (x i ,y i ) and scale s i ;

[0070] For each key point, calculate its feature descriptor d i , the descriptor contains the local feature information of the area around the key point, and constructs the descriptor set D of all key points: D = {d1, d2, ..., d n}, where d i is the key point k i feature descriptors;

[0071] By calculating the Euclidean distance (d i ,d j ), find the matching key point pairs between the current frame and the previous frame: distance(d i ,d j )=||d i -d j ||, where d j is the key point k i The feature descriptor of the previous frame; using the nearest neighbor matching algorithm, find the matching point of each key point;

[0072] For the matched key point pairs, calculate the motion vector:

[0073] Δx i = x′ i -x i , Δy i =y′ i -y i ;

[0074] Where Δx i and Δy i are the horizontal and vertical coordinates of the key point's motion vector between images, respectively, x′ i and y′ i They are respectively the key points k in the previous frame i The horizontal and vertical coordinates of the matched key points; update the position and descriptor of the key points, track the key points according to the motion vector, and form the operation data of the robot.

[0075] According to step S300, the video data of the normal operation of the historical robot and the corresponding parameter data of the charging pile are obtained, the video data is extracted frame by frame, and each frame of the image is timestamped to correspond to the charging pile parameter data; the pre-trained convolutional neural network model VGG16 is selected to extract image features to obtain image feature vectors, the charging pile parameter data is normalized, and the normalized charging pile parameter data is combined with the image feature vector of the corresponding timestamp to form a time series feature; the image feature vectors of the continuous timestamps and the charging pile parameter data are combined to construct a time series feature set, and the time series feature set is marked as a sample of normal operation;

[0076] The time series feature set is divided into a training set and a validation set. The LSTM model is selected as the learning model for the time series features. The LSTM network is constructed, with the input being the time series features and the output being the predicted value of whether the operation is normal. The training set is used for model training and the parameters of the LSTM model are optimized. The mean square error is selected as the loss function and Adam is selected as the optimization algorithm. The validation set is used to evaluate the model performance and the model parameters and hyperparameters are adjusted to ensure that the model can identify the time series features of the normal operation of the robot.

[0077] On the validation set, the LSTM model achieved an accuracy of 0.92, a precision of 0.91, a recall of 0.89, and an F1 value of 0.90. Experiments show that the LSTM model can effectively learn the time series characteristics of the normal operation of the battery swap robot and has good detection capabilities for abnormal operations. By integrating image features and charging pile parameter data, the detection accuracy and robustness of the model have been significantly improved.

[0078] During the battery replacement process, the image feature vector and charging pile parameter data are extracted in real time to form the feature sequence of the current time; the feature sequence of the current time is input into the trained LSTM model to predict normal operation.

[0079] According to step S300, an Autoencoder model is selected, wherein the Autoencoder model consists of an encoder and a decoder, and is used to learn a low-dimensional representation of input data; the input high-dimensional time series features are compressed to a low-dimensional representation, and the original high-dimensional features are reconstructed from the low-dimensional representation, with the goal of making the reconstructed features similar to the original features; the mean square error is used to measure the difference between the reconstructed features and the original features; the training set is used to train the model and optimize the parameters of the Autoencoder so that it can learn a low-dimensional representation of normal operation; the validation set is used to evaluate the model performance to ensure that the model can reconstruct the time series features of normal operation; VGG16 is used to extract image features, and the dimension of each feature vector is 4096. After the charging pile parameters are normalized, the dimension of each data point is 3, forming a time series feature, and the dimension of each feature vector is 4099.

[0080] Encoder: The input dimension is 4099, compressed to 256 dimensions, decoder: reconstruct the 256-dimensional representation into 4099 dimensions, use the training set for model training, and the training data ratio is 80%.

[0081] Input the feature sequence of the current time into the trained Autoencoder model, encode and reconstruct it, obtain the reconstructed feature sequence, and calculate the reconstruction error between the reconstructed feature sequence and the original feature sequence; judge whether the current operation is normal by the reconstruction error, set a threshold, and if the reconstruction error exceeds the threshold, mark it as an abnormal operation, and record the time point when the abnormality occurred and the corresponding feature data; when an abnormality occurs, notify the maintenance personnel, and store the data of the abnormal operation in the local database and the cloud. In the validation set, the mean of the reconstruction error is 0.05 and the standard deviation is 0.01. Set the threshold to 0.08, and mark it as abnormal if it exceeds this value.

[0082] In the test set, the number of abnormal operations detected was 15, all of which were accurately marked; the data of abnormal operations was stored in the local database and the cloud, and maintenance personnel were notified.

[0083] According to step S400, charging pile parameters and robot operation data are collected, and the correlation between the charging pile parameters and the robot operation is determined using a correlation analysis method. A correlation model between the charging pile parameters and the robot operation is constructed using a random forest algorithm. The input of the correlation model is the charging pile parameter P, and the output is the robot operation parameter R: R=f(P), where f is the correlation model;

[0084] Extract the charging pile parameters and robot operation parameters during the abnormality, input the charging pile parameters during the abnormality into the association model to predict the corresponding robot operation parameters; compare the predicted robot operation parameters with the robot operation parameters during the abnormality to find the differences; formulate the charging pile parameter adjustment strategy and the robot operation adjustment strategy according to the prediction results of the association model; monitor the adjusted operation and detect whether there is an abnormality in real time. If a new anomaly is detected, repeat the adjustment process until the reconstruction error is lower than the threshold.

[0085] When adjusting the charging pile parameters, it is necessary to match the parameter range required for normal operation. The charging pile parameter adjustment algorithm is as follows:

[0086] P adjusted =P abnormal +ΔP;

[0087] Among them, P adjusted is the charging pile parameter after adjustment, P abnormal is the charging pile parameter when abnormal, ΔP is the charging pile parameter adjustment increment, and the correlation model f is derived by reverse deduction. -1 Sure;

[0088] When adjusting the robot operating parameters, the predicted robot operating parameters are used as the adjusted charging pile parameters; during the abnormality detection and parameter adjustment process, maintenance personnel are notified in real time to provide abnormal information and adjustment suggestions.

[0089] A battery swap station robot anomaly detection system based on visual recognition, comprising:

[0090] Data acquisition module: including: video acquisition unit, video preprocessing unit and data transmission unit; the video acquisition unit is responsible for continuously collecting video data during the battery replacement process through the camera, the video preprocessing unit is responsible for preprocessing the collected video, and the data transmission unit is responsible for transmitting the preprocessed video data to the local database and the cloud;

[0091] Feature extraction module: including: target detection unit, posture estimation unit, appearance feature extraction unit and key point tracking unit; the target detection unit uses the YOLO target detection algorithm to identify the robot and its components in the video data and mark the key parts of the robot. The posture estimation unit uses OpenPose to determine the posture of the robot, obtain the joint position and motion trajectory of the robot. The appearance feature extraction unit uses a convolutional neural network to extract the appearance features of the robot. The key point tracking unit extracts the key point descriptor SIFT for matching and tracking.

[0092] Anomaly detection module: including: behavior model training unit, anomaly detection model construction unit and real-time anomaly detection unit; wherein the behavior model training unit obtains the video data of the historical normal operation of the robot, combines the charging pile parameters of the corresponding battery swap station, trains the behavior model LSTM based on deep learning, and learns the time series characteristics of normal operation; the anomaly detection model construction unit constructs an anomaly detection model based on Autoencoder, learns the low-dimensional representation of the normal operation of the robot, and the real-time anomaly detection unit detects anomalies by comparing the reconstruction error between the actual operation and the normal operation during operation;

[0093] Abnormality handling module: including: an association model construction unit and a parameter adjustment unit; wherein, when the association model construction unit detects an abnormality, it uses a random forest algorithm to construct an association model between the charging pile of the battery swap station and the robot, and the parameter adjustment unit adjusts the charging pile parameters of the battery swap station and the operation of the robot according to the abnormality detection result and the association model until the abnormal situation in the battery swap process is eliminated.

[0094] In this embodiment, a total of 15 abnormal operations were detected, the reconstruction error threshold was set to 0.08, and the reconstruction errors of all abnormal operations exceeded 0.1. The accuracy of the association model constructed by the random forest algorithm is 95%. The charging pile parameters are adjusted according to the association model, specifically: charging voltage: adjusted from 220V to 230V, charging current: adjusted from 5A to 4.5A. Adjust the robot operation: operation sequence: optimize the battery interface connection sequence, operation speed: reduce the operation speed of the robot arm to avoid errors caused by rapid operation. After the adjustment, all abnormal operations were eliminated, and no abnormal operations occurred during the retest, indicating that the adjustment is effective.

[0095] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0096] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormality of a battery swap station robot based on visual recognition, characterized in that: The method comprises: S100, continuously collecting video data during the battery replacement process through a camera, performing preprocessing, and transmitting the preprocessed video data to a local database and the cloud; S200, using the YOLO target detection algorithm to identify the robot and its components in the video data and mark the key parts of the robot; using OpenPose to determine the robot's posture, obtain the robot's joint position and motion trajectory; using a convolutional neural network to extract the robot's appearance features, and extract the key point descriptor SIFT for matching and tracking; S300, obtain the video data of the normal operation of the historical robot, combine it with the charging pile parameters of the corresponding battery swap station, train the LSTM behavior model based on deep learning, and learn the time series characteristics of normal operation; build an anomaly detection model based on Autoencoder to learn the low-dimensional representation of the normal operation of the robot; at runtime, detect anomalies by comparing the reconstruction error between the actual operation and the normal operation; S400. When an abnormality is detected, a random forest algorithm is used to construct an association model between the charging piles of the battery swap station and the robot. The charging pile parameters of the battery swap station and the operation of the robot are adjusted according to the abnormality detection results and the association model until the abnormality in the battery swap process is eliminated.

2. According to claim 1, a method for detecting abnormalities of a battery swap station robot based on visual recognition is characterized in that: According to step S100, the camera continuously collects video data during the operation of the battery swap station, records all operation processes of the battery swap robot, and transmits the collected video data to a local storage device in real time; Extract each frame of image from the continuous video stream and perform single frame image processing; Use Gaussian filtering to denoise the image; use histogram equalization to improve the contrast of the image; Use adaptive histogram equalization to prevent distortion caused by over-enhancement of the image; correct camera distortion and perform color correction; The pre-processed image data is stored in the local database, and the locally stored data is backed up regularly; The preprocessed video data is transmitted to the cloud storage platform via the network, and the transmission protocol HTTPS is used to ensure the security of data transmission.

3. The method for detecting abnormality of a battery swap station robot based on visual recognition according to claim 1 is characterized in that: According to step S200, the YOLO target detection algorithm is used for detection, the v5 model version of YOLO is selected, and the pre-trained model weights and configuration files are downloaded; the training data is annotated using an annotation tool, and the annotation content includes various key parts of the robot, and the key parts include a robotic arm, a battery interface, a robot camera, a sensor, a display screen, and an indicator light; the key parts of the robot are annotated on the image, and different colored borders and labels are used to distinguish them; the annotated image is stored in a local database and uploaded to the cloud.

4. The method for detecting abnormality of a battery swap station robot based on visual recognition according to claim 1 is characterized in that: According to step S200, OpenPose is downloaded and installed, and a pre-trained OpenPose model is loaded; the pre-processed image is input into the OpenPose model, and OpenPose outputs the joint positions of the robot in each frame of the image; For each frame, OpenPose outputs the joint position (x i ,y i ), where i represents different joints; the posture is determined by the joint position. The robot arm has n joints, and the joint position is represented by P = {(x1,y1),(x2,y2),...,(x n ,y n )}; For each joint i, the position at time t is (x i (t),y i (t)), the motion trajectory is expressed as the trajectory of joint position changes over time; the mathematical representation of the motion trajectory is T i ={(x i (t0),y i (t0)),(x i (t1),y i (t1)),...,(x i (t m ),y i (t m ))}, where t0, t1, ..., t m are different time points, m is the total number of time points; the Kalman filter is used to smooth the joint motion trajectory.

5. The method for detecting abnormality of a battery swap station robot based on visual recognition according to claim 4 is characterized in that: According to step S200, a pre-trained convolutional neural network model VGG16 is selected to extract image features; Use the SIFT algorithm to detect key points in the image. For each key point k i , calculate its position (x i ,y i ) and scale s i ; For each key point, calculate its feature descriptor d i , the descriptor contains the local feature information of the area around the key point, and constructs the descriptor set D of all key points: D = {d1, d2, ..., d n }, where d i is the key point k i feature descriptors; By calculating the Euclidean distance (d i ,d j ), find the matching key point pairs between the current frame and the previous frame: distance(d i ,d j )=||d i -d j ||, where d j is the key point k i The feature descriptor of the previous frame; using the nearest neighbor matching algorithm, find the matching point of each key point; For the matched key point pairs, calculate the motion vector: Δx i =x′ i -x i ,Δy i =y′ i -y i ; Where Δx i and Δy i are the horizontal and vertical coordinates of the key point's motion vector between images, respectively, x′ i and y′ i They are respectively the key points k in the previous frame i The horizontal and vertical coordinates of the matched key points; update the position and descriptor of the key points, track the key points according to the motion vector, and form the operation data of the robot.

6. The method for detecting abnormality of a battery swap station robot based on visual recognition according to claim 1 is characterized in that: According to step S300, the video data of the normal operation of the historical robot and the corresponding parameter data of the charging pile are obtained, the video data is extracted frame by frame, and each frame of the image is timestamped to correspond to the charging pile parameter data; the pre-trained convolutional neural network model VGG16 is selected to extract image features to obtain image feature vectors, the charging pile parameter data is normalized, and the normalized charging pile parameter data is combined with the image feature vector of the corresponding timestamp to form a time series feature; Combine the image feature vectors of continuous timestamps and the charging pile parameter data to construct a time series feature set, and mark the time series feature set as samples of normal operation; Divide the time series feature set into a training set and a validation set, select the LSTM model as the learning model for the time series features, build an LSTM network, input the time series features, and output the predicted value of whether the operation is normal; use the training set to train the model and optimize the parameters of the LSTM model; select the mean square error as the loss function and Adam as the optimization algorithm; Use the validation set to evaluate model performance and adjust model parameters and hyperparameters to ensure that the model can identify the time series characteristics of normal robot operation; During the battery replacement process, the image feature vector and charging pile parameter data are extracted in real time to form the feature sequence of the current time; the feature sequence of the current time is input into the trained LSTM model to predict normal operation.

7. The method for detecting abnormality of a battery swap station robot based on visual recognition according to claim 6 is characterized in that: According to step S300, an Autoencoder model is selected, wherein the Autoencoder model is composed of an encoder and a decoder, and is used to learn a low-dimensional representation of input data; the input high-dimensional time series features are compressed into a low-dimensional representation, and the original high-dimensional features are reconstructed from the low-dimensional representation, with the goal of making the reconstructed features similar to the original features; the mean square error is used to measure the difference between the reconstructed features and the original features; the training set is used to train the model, and the parameters of the Autoencoder are optimized so that it can learn a low-dimensional representation of normal operation; the validation set is used to evaluate the model performance to ensure that the model can reconstruct the time series features of normal operation; The feature sequence of the current time is input into the trained Autoencoder model for encoding and reconstruction to obtain the reconstructed feature sequence, and the reconstruction error between the reconstructed feature sequence and the original feature sequence is calculated; the reconstruction error is used to determine whether the current operation is normal, and a threshold is set. If the reconstruction error exceeds the threshold, it is marked as an abnormal operation, and the time point when the abnormality occurred and the corresponding feature data are recorded; when an abnormality occurs, the maintenance personnel are notified, and the data of the abnormal operation is stored in the local database and the cloud.

8. The method for detecting abnormality of a battery swap station robot based on visual recognition according to claim 1 is characterized in that: According to step S400, charging pile parameters and robot operation data are collected, and the correlation between the charging pile parameters and the robot operation is determined using a correlation analysis method. A correlation model between the charging pile parameters and the robot operation is constructed using a random forest algorithm. The input of the correlation model is the charging pile parameter P, and the output is the robot operation parameter R: R=f(P), where f is the correlation model; Extract the charging pile parameters and robot operation parameters during the abnormality, input the charging pile parameters during the abnormality into the association model to predict the corresponding robot operation parameters; compare the predicted robot operation parameters with the robot operation parameters during the abnormality to find the differences; formulate the charging pile parameter adjustment strategy and the robot operation adjustment strategy according to the prediction results of the association model; monitor the adjusted operation and detect whether there is an abnormality in real time. If a new anomaly is detected, repeat the adjustment process until the reconstruction error is lower than the threshold.

9. The method for detecting abnormality of a battery swap station robot based on visual recognition according to claim 8 is characterized in that: When adjusting the charging pile parameters, it is necessary to match the parameter range required for normal operation. The charging pile parameter adjustment algorithm is as follows: P adjusted =P abnormal +ΔP; Among them, P adjusted is the charging pile parameter after adjustment, P abnormal is the charging pile parameter when abnormal, ΔP is the charging pile parameter adjustment increment, and the correlation model f is derived by reverse deduction. -1 Sure; When adjusting the robot operating parameters, the predicted robot operating parameters are used as the adjusted charging pile parameters; during the abnormality detection and parameter adjustment process, maintenance personnel are notified in real time to provide abnormal information and adjustment suggestions.

10. A battery swap station robot anomaly detection system based on visual recognition, using a battery swap station robot anomaly detection method based on visual recognition as claimed in any one of claims 1 to 9, characterized in that: include: Data acquisition module: including: video acquisition unit, video preprocessing unit and data transmission unit; the video acquisition unit is responsible for continuously collecting video data during the battery replacement process through the camera, the video preprocessing unit is responsible for preprocessing the collected video, and the data transmission unit is responsible for transmitting the preprocessed video data to the local database and the cloud; Feature extraction module: including: target detection unit, posture estimation unit, appearance feature extraction unit and key point tracking unit; the target detection unit uses the YOLO target detection algorithm to identify the robot and its components in the video data and mark the key parts of the robot. The posture estimation unit uses OpenPose to determine the posture of the robot, obtain the joint position and motion trajectory of the robot. The appearance feature extraction unit uses a convolutional neural network to extract the appearance features of the robot. The key point tracking unit extracts the key point descriptor SIFT for matching and tracking. Anomaly detection module: including: behavior model training unit, anomaly detection model construction unit and real-time anomaly detection unit; wherein the behavior model training unit obtains the video data of the historical normal operation of the robot, combines the charging pile parameters of the corresponding battery swap station, trains the behavior model LSTM based on deep learning, and learns the time series characteristics of normal operation; the anomaly detection model construction unit constructs an anomaly detection model based on Autoencoder, learns the low-dimensional representation of the normal operation of the robot, and the real-time anomaly detection unit detects anomalies by comparing the reconstruction error between the actual operation and the normal operation during operation; Abnormality handling module: including: an association model construction unit and a parameter adjustment unit; wherein, when the association model construction unit detects an abnormality, it uses a random forest algorithm to construct an association model between the charging pile of the battery swap station and the robot, and the parameter adjustment unit adjusts the charging pile parameters of the battery swap station and the operation of the robot according to the abnormality detection result and the association model until the abnormal situation in the battery swap process is eliminated.

Images