Following equipment cooperative control method and device, storage medium and electronic equipment

Abstract

The invention discloses a following equipment cooperative control method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring following state information of following equipment and target state information of target equipment; based on the following state information and the target state information, a target model is adopted for prediction, a first distance estimation value between the following device and the target device is determined, and the target model is obtained based on historical data training of following of the target device by the following device; based on the following state information and the target state information, processing in a pose change mode, and determining a second distance estimation value between the following device and the target device; and determining a target distance estimation value between the following device and the target device based on the first distance estimation value and the second distance estimation value. The technical problem that the relative positioning accuracy of the following device and the target device is not ideal is solved.

Description

Technical Field

[0001] The present invention relates to the field of positioning technology, and in particular to a method, device, storage medium and electronic device for collaborative control of a follower device. Background Art

[0002] As the core components of the power grid, power cables and tunnels undertake the important tasks of power transmission and power supply guarantee. However, some tunnels are facing the problems of aging and disrepair. In addition, the special characteristics of the underground environment, such as humidity, narrowness, and complex terrain, make manual inspection difficult and risky. Traditional inspection methods are not only inefficient, but also require a lot of manpower and material investment, and can no longer meet the high requirements of modern power grids for safety, efficiency, and intelligence. Generally speaking, in the narrow, complex and highly similar environment of underground tunnels, the application of robots is restricted by multiple factors such as their applicable scenarios, cruising capabilities, and passability. It is difficult to independently complete a comprehensive inspection of the entire tunnel, and multiple devices are required to follow each other and work together to complete the overall inspection work.

[0003] Related technologies rely on traditional positioning systems such as GPS (Global Positioning System) to determine the relative distance between unmanned systems that follow each other. They need to rely on a network environment for data exchange, but they are not able to adapt to the underground networkless environment, resulting in an unsatisfactory relative distance and difficulty in maintaining a stable following state.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method, an apparatus, a storage medium and an electronic device for cooperatively controlling a following device, so as to at least solve the technical problem that the relative positioning accuracy between a following device and a target device is not ideal.

[0006] According to one aspect of an embodiment of the present invention, a method for collaborative control of a following device is provided, comprising: obtaining following state information of a following device and target state information of a target device; based on the following state information and the target state information, using a target model for prediction to determine a first distance estimate between the following device and the target device, wherein the target model is trained based on historical data of the following of the target device by the following device; based on the following state information and the target state information, using a posture change method for processing to determine a second distance estimate between the following device and the target device; based on the first distance estimate and the second distance estimate, determining a target distance estimate between the following device and the target device.

[0007] Optionally, based on the following state information and the target state information, a target model is used for prediction to determine a first distance estimate between the following device and the target device, including: based on the following state information and the target state information, a convolutional neural network is used for feature extraction to obtain a motion feature vector; based on the motion feature vector, a long short-term memory network is used for processing to obtain the first distance estimate, wherein the long short-term memory network is provided with a state memory unit and a hidden layer, the state memory unit is used to store the operating state information of the previous moment, and the hidden layer is used to record the estimated values ​​of the center of mass sideslip angles corresponding to the following device and the target device at the previous moment.

[0008] Optionally, based on the following state information and the target state information, a posture change method is used for processing to determine a second distance estimate between the following device and the target device, including: determining dynamic position change information between the target device and the following device; based on the dynamic position change information, determining observation information between the target device and the following device, wherein the observation information is used to characterize the association between the ranging data and the relative position between the target device and the following device; based on the observation information, an extended Kalman filter is used for posture change processing to obtain the second distance estimate.

[0009] Optionally, the following state information includes following motion data, the target state information includes target motion data, and the determining of the dynamic position change information between the target device and the following device includes: determining the target motion state of the target device in the corresponding target coordinate system based on the target motion data; determining the following motion state of the following device in the corresponding following coordinate system based on the following motion data; and obtaining the dynamic position change information by using a coordinate system transformation matrix based on the target motion state and the following motion state.

[0010] Optionally, the following state information includes following position data, the target state information includes target position data, and determining the observation information between the target device and the following device based on the dynamic position change information includes: determining the ranging data between the target device and the following device based on the following position data and the target position data; and determining the observation information based on the ranging data, the dynamic position change information, and the noise matrix of the target device and the following device.

[0011] Optionally, determining the target distance estimate between the following device and the target device based on the first distance estimate and the second distance estimate includes: determining the mean absolute error and the root mean square error of the first distance estimate and the second distance estimate; determining the absolute value of the deviation between the mean absolute error and the root mean square error; when the absolute value of the deviation is equal to a predetermined threshold, determining the average value between the first distance estimate and the second distance estimate; using the average value as the target distance estimate; or, when the absolute value of the deviation is greater than a predetermined threshold, determining the arithmetic square root of the sum of the squares of the first distance estimate and the second distance estimate; and using the arithmetic square root as the target distance estimate.

[0012] Optionally, the following device and the target device are respectively provided with a position sensor and an inertial measurement unit, the position sensor is an ultra-wideband UWB sensor, the UWB sensor has data transmission capability, and the following device and the target device are respectively a drone or a ground robot.

[0013] According to another aspect of an embodiment of the present invention, a following device cooperative control device is provided, including: an acquisition module, used to acquire following state information of a following device, and target state information of a target device; a first determination module, used to determine a first distance estimate between the following device and the target device by using a target model for prediction based on the following state information and the target state information, wherein the target model is obtained by training based on historical data of the following device following the target device; a second determination module, used to determine a second distance estimate between the following device and the target device by using a posture change method for processing based on the following state information and the target state information; and a distance determination module, used to determine a target distance estimate between the following device and the target device based on the first distance estimate and the second distance estimate.

[0014] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, wherein the non-volatile storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing any one of the following device cooperative control methods.

[0015] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the following device collaborative control methods.

[0016] In an embodiment of the present invention, by obtaining the following state information of the following device and the target state information of the target device; based on the following state information and the target state information, using the target model for prediction, determining the first distance estimation value between the following device and the target device, wherein the target model is obtained by training based on the historical data of the following device following the target device; based on the following state information and the target state information, using the posture change method for processing, determining the second distance estimation value between the following device and the target device; based on the first distance estimation value and the second distance estimation value, determining the target distance estimation value between the following device and the target device. The purpose of improving the accuracy of the target distance estimation value through two distance estimation values ​​is achieved, and the technical effect of improving the accuracy of the relative position positioning between the following device and the target device is achieved, thereby solving the technical problem of the unsatisfactory relative positioning accuracy of the following device and the target device. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of an optional follow-up device cooperative control method provided by an embodiment of the present invention;

[0019] Figure 2 is a first schematic diagram of an optional follower device cooperative control method provided by an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of an algorithm of an optional follower device cooperative control method provided by an embodiment of the present invention;

[0021] Figure 4 is a schematic flow chart of an optional follow-up device cooperative control method provided by an embodiment of the present invention;

[0022] Figure 5 It is a schematic diagram of an optional follow-up device cooperative control apparatus provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0026] CNN (Convolutional Neural Network) is a deep learning model mainly used for image and video recognition and natural language processing. It can automatically extract features from images by simulating the working principle of the human visual system. CNN is composed of multiple layers of convolutional layers and pooling layers. The convolutional layer uses the pixel values ​​of a small area (convolution kernel) to extract features and introduces nonlinearity through activation functions. The pooling layer is used to reduce the spatial dimension of the features while increasing the invariance to image displacement.

[0027] LSTM (Long Short-Term Memory) is a special RNN (Recurrent Neural Network) that can learn long-term dependencies of data. It solves the gradient vanishing problem of traditional RNN by introducing a gating mechanism (forget gate, input gate, output gate). The LSTM network updates its internal state at each time step, and decides which information should be forgotten through the forget gate, which information should be added to the cell state through the input gate, and finally decides what information to output through the output gate.

[0028] UWB (Ultra-Wideband) is a wireless communication technology with a wide frequency band that can provide high data transmission rate and high-precision positioning. UWB uses extremely short pulse signals to transmit data and achieves positioning by measuring the arrival time of signals between different devices.

[0029] EFK (Extend Kalman Filter) is a filter used to estimate the state of a nonlinear system. It is an extension of the Kalman filter and is suitable for situations where the system dynamics and observation models are nonlinear.

[0030] IMU (Interial Measurement Unit) is a device that measures and reports the specific force, angular velocity, and direction of an object around a magnet, helping to calculate the position, velocity, and direction of an object.

[0031] Z-Score standardization is a method of data standardization that converts data into a distribution with a mean of 0 and a standard deviation of 1. It helps to eliminate the impact of different dimensions between data and allows data to be compared under the same standard.

[0032] MAE (Mean Absolute Error) is an indicator to measure the accuracy of the prediction model. It calculates the average absolute difference between the actual value and the predicted value. It is a non-negative value. The closer it is to 0, the more accurate the model prediction is.

[0033] RMSE (Root Mean Square Error): Root mean square error is an indicator to measure the accuracy of the prediction model. It calculates the square root of the mean of the squared difference between the actual value and the predicted value. It is a non-negative value. The closer it is to 0, the more accurate the model prediction is.

[0034] According to an embodiment of the present invention, a method embodiment of collaborative control of a following device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] Figure 1 is a flow chart of an optional follower device cooperative control method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0036] Step S102, obtaining the following state information of the following device and the target state information of the target device;

[0037] It can be understood that it is necessary to obtain the real-time motion state information of the following device and the target device. The above following state information and target state information are the basis for subsequent relative position positioning and prediction.

[0038] Optionally, the execution subject may be any one of the following device and the target device, and preferably the one with higher computing power and higher power is used to calculate the target distance estimation value.

[0039] Optionally, the distance data between the follower device and the target device is converted into a data set of n-dimensional vectors, and then the data is normalized using Z-Score to remove the dimension before the next step of processing.

[0040] In an optional embodiment, the following device and the target device are respectively provided with a position sensor and an inertial measurement unit, the position sensor is an ultra-wideband UWB sensor, the UWB sensor has data transmission capability, and the following device and the target device are respectively a drone or a ground robot.

[0041] It can be understood that the following device and the target device are respectively provided with a position sensor and an inertial measurement unit. The position sensor is a UWB (ultra-wideband) sensor. The UWB sensor provides high-precision ranging capability and high-speed data transmission capability, and can monitor the position change of the device in real time. The UWB sensor can measure the distance by the flight distance, and the IMU sensor can provide the motion data of the device, including speed, angular velocity, etc. The following device and the target device can be a drone or a ground robot respectively. Through the UWB sensor and the IMU sensor, the movement trajectory and posture changes of the device can be detected in real time.

[0042] Optionally, the following device and the target device may be devices of various models and functions, such as a drone, a quadruped robot, an underwater robot, a snake-like robot, a tracked robot, etc.

[0043] Optionally, when the following device is a drone, denoted as drone A, and the target device is a quadruped robot (ie, a ground robot), denoted as quadruped robot B, the drone and the quadruped robot are respectively equipped with a UWB sensor and an IMU sensor. Figure 2 : is a schematic diagram of an optional follow-up device cooperative control system provided by an embodiment of the present invention, wherein three UWB sensors are installed on drone A, denoted as UWB1, UWB2, and UWB3. A UWB4 sensor is installed on quadruped robot B, and multiple sets of ranging data can be obtained by performing ranging processing of UWB signal flight time between UWB1, UWB2, and UWB3 and UWB4 respectively, and denoted as d1, d2, and d3.

[0044] Through UWB communication, the status information of the follower device and the target device can be obtained, and the relative position of the drone can be estimated. The IMU sensor can obtain the three-axis acceleration and three-axis angular velocity of the drone's motion. They are respectively recorded as the three-axis acceleration (ax, ay, az) and three-axis angular velocity (ωax, ωay, ωaz) of drone A, and the three-axis acceleration (bx, by, bz) and three-axis angular velocity (ωbx, ωby, ωbz) of ground robot B. The Z-Score normalization is used to scale the numerical features to eliminate the dimensional differences and value range differences between different features. UWB sensors can not only provide accurate distance measurement between devices, but also have the ability of high-speed data transmission. It allows drones and ground robots to exchange information in real time when performing tasks, and transmit their respective motion states back through UWB, thereby achieving accurate relative positioning.

[0045] Step S104, based on the following state information and the target state information, a target model is used to make a prediction to determine a first distance estimation value between the following device and the target device, wherein the target model is trained based on historical data of the following device following the target device;

[0046] It can be understood that by using the historical following data of the following device and the target device to train a target model (such as a deep learning model), the target model can predict the relative distance between the following device and the target device, that is, the first distance estimation value.

[0047] In an optional embodiment, based on the following state information and the target state information, a target model is used for prediction to determine a first distance estimate between the following device and the target device, including: based on the data information acquired by the UWB sensors in the following device and the target device, a convolutional neural network is used for feature extraction to obtain a motion feature vector; based on the motion feature vector, a long short-term memory network is used for processing to obtain a first distance estimate, wherein the long short-term memory network is provided with a state memory unit and a hidden layer, the state memory unit is used to store the operating state information of the previous moment, and the hidden layer is used to record the estimated values ​​of the center of mass sideslip angles corresponding to the following device and the target device at the previous moment.

[0048] It can be understood that the target model extracts motion feature vectors from UWB sensor data through a convolutional neural network (CNN), while the long short-term memory network (LSTM) uses the temporal dependencies in time series data to predict future motion states. Through training, the model learns the dynamic relationship between the follower device and the target device, so that it can predict the first distance estimate between them.

[0049] Optionally, Figure 3is an algorithm diagram of an optional follower device cooperative control method provided by an embodiment of the present invention, such as Figure 3 As shown, the data acquired by the UWB sensor and the IMU sensor respectively (i.e., the following state information and the target state information) are first subjected to feature data extraction by a convolutional neural network, and then predicted by a long short-term memory network according to the time series to obtain a first distance estimation value.

[0050] In convolutional neural networks, one-dimensional convolution (Conv1D) is often used to process sequence data, such as time series, text data, etc. In the convolution operation, the convolution kernel slides along the input sequence and performs calculations to obtain a new feature representation. In order to reduce the complexity of the parameter learning process and improve the algorithm's operation speed, the Z-Score standardization method is used to convert the original data into data that conforms to the standard normal distribution with a mean of 0 and a variance of 1.

[0051] Long Short-Term Memory Network (LSTM) is an improved time-recurrent neural network (RNN). The LSTM can effectively solve the gradient vanishing problem in the traditional model training process and is suitable for the analysis and prediction of long time series. Compared with the standard RNN network, LSTM adds a memory state unit in the hidden layer neural node to store past information, and uses three gate structures (input gate, forget gate, output gate) to control the forgetting and updating of historical information.

[0052] The above-mentioned following state information and target state information are used as sensor measurement data, and the sensor measurement data is organized into a form suitable for convolutional neural network input. The feature vector is extracted by inputting the convolutional neural network through the measurement data input layer to obtain the motion feature vector.

[0053] The extracted feature vector (i.e., motion feature vector) is used as the input of LSTM, denoted as x t , state memory unit C t-1 Responsible for storing the motion state information of the previous moment, the hidden unit h t-1 Record the estimated value of the center of mass sideslip angle at the last moment, and use the above input x t , C t-1 、h t-1 They work together on the forget gate to decide which information needs to be forgotten. t and h t-1 The two inputs are transformed by the Sigmoid function and the tanh function respectively, and act together on the state memory unit to determine which information needs to be retained. Output gate h t is the updated state memory unit C t-1 With output o t Joint decision.

[0054] It is preferred to set LSTM to four layers. The first, second, and third layers are output in sequence form. All hidden layer output information is input to the next layer. In the fourth layer, only the output information of the last hidden layer is output. This can prevent the network from overfitting. The output layer takes the predicted value output by the last time step of LSTM as input, and outputs the estimated relative position through the fully connected layer to obtain the final estimated value. This is the first distance estimate.

[0055] Optionally, the following state information and the target state information are sensor measurement data, which is a time series data. For the time series data, the measurement value at each time point is used as an input sample, and the entire time series data is divided into a training set, a validation set, and a test set. The above convolutional neural network is trained based on yi=f(ki*x+bi), where x is the data of the target device and the following device collected by the input layer at a certain moment, where ki is the i-th convolution kernel, bi is the deviation of the i-th convolution kernel, * represents the convolution operation, f(.) is the activation function, and ReLU is selected as the activation function.

[0056] Step S106, based on the following state information and the target state information, a second distance estimation value between the following device and the target device is determined by processing in a posture change manner;

[0057] It can be understood that in addition to using the target model for prediction, it is also necessary to determine the second distance estimate between the follower device and the target device through posture change processing. The IMU sensor is used to detect the spatial position and posture changes of the device in real time, and the extended Kalman filter algorithm is used to obtain the second distance estimate, which is conducive to improving the accuracy of the target distance estimate.

[0058] It should be noted that the relative distance between devices is calculated by using IMU sensors in combination with consistent physical models and environmental information. The distance estimate (i.e., the second distance estimate) is obtained by using the extended Kalman filter (EKF) algorithm based on the data collected by the UWB sensor and the IMU sensor respectively, which can effectively handle nonlinear problems and perform appropriate approximation and optimization during the calculation process, thereby improving the real-time performance and accuracy of the algorithm.

[0059] In an optional embodiment, based on the following state information and the target state information, a posture change method is used for processing to determine a second distance estimate between the following device and the target device, including: determining dynamic position change information between the target device and the following device; based on the dynamic position change information, determining observation information between the target device and the following device, wherein the observation information is used to characterize the association between the ranging data and the relative position between the target device and the following device; based on the observation information, an extended Kalman filter is used for posture change processing to obtain a second distance estimate.

[0060] It can be understood that the dynamic position change information between the target device and the follower device refers to the real-time change of the position and posture of the device during movement, which is obtained by the IMU sensor. Based on the dynamic position change information, the observation information of the follower device and the target device can be determined. The observation information refers to the information that can characterize the association between the distance measurement data and the relative position between the follower device and the target device. The distance measurement data is obtained by the UWB sensor. Using the above observation information, the extended Kalman filter (EKF) is used to process the posture change. In EFK, the state of the target device and the follower device is predicted by differential equations, and then the prediction value is updated using the observation information to finally obtain the second distance estimate.

[0061] In an optional embodiment, the following state information includes following motion data, and the target state information includes target motion data, and the dynamic position change information between the target device and the following device is determined, including: based on the target motion data, determining the target motion state of the target device in the corresponding target coordinate system; based on the following motion data, determining the following motion state of the following device in the corresponding following coordinate system; based on the target motion state and the following motion state, using the coordinate system transformation matrix to obtain the dynamic position change information.

[0062] It can be understood that the target coordinate system and the following coordinate system are used to describe the position and movement of the target device and the following device in their environment, respectively. The coordinate system conversion matrix is ​​needed to convert their position information from their respective coordinate systems to a common coordinate system in order to understand their relative positions to each other. Through the coordinate system conversion, combined with the state information of the following device and the target device, the dynamic position change information of the following device and the target device can be obtained by establishing differential equations.

[0063] Optionally, the following device and the target device obtain their motion states, including acceleration and angular velocity, through an IMU sensor. Based on the target motion data and the following motion data, the state information X AB According to the state information and coordinate system transformation matrix, the differential equation is established based on Newton's laws of motion and the law of conservation of angular momentum. That is the dynamic position change information mentioned above.

[0064] The coordinate system conversion matrix (denoted as C) is used to convert information in one coordinate system to another coordinate system. It consists of a rotation matrix R and a translation vector t. The rotation matrix is ​​used to describe the rotation from one coordinate system to another, which can be calculated by Euler angles (such as pitch, yaw, and roll). The rotation matrix R is a 3×3 orthogonal matrix. The translation vector t is used to describe the distance between the origin of the coordinate system of the following device and the target device. The coordinate system conversion matrix can be expressed as follows:

[0065] In an optional embodiment, the following state information includes following position data, the target state information includes target position data, and based on the dynamic position change information, the observation information between the target device and the following device is determined, including: determining the ranging data between the target device and the following device based on the following position data and the target position data; determining the observation information based on the ranging data, the dynamic position change information, and the noise matrix of the target device and the following device.

[0066] It can be understood that based on the dynamic position change information, the distance measurement data can be obtained by using the acquired following position data and target position data. The dynamic position change information refers to the position changes of the following device and the target device during the observation process, which affects the accurate prediction and update of the observation information. If the target device is moving quickly, the following device needs to adjust its position faster to maintain the following state. In practical applications, the position data of both the target device and the following device will be affected by various noises, such as sensor errors, environmental interference, etc. By introducing a noise matrix, this error can be described, which helps to take these uncertain factors into account when calculating the observation information.

[0067] Optionally, the noise matrix includes error estimates of parameters such as position and velocity.

[0068] Optionally, in the EKF filter, based on the differential equation (i.e. the above dynamic position change information), the state of the target device and the follower device is predicted one step further, and then the state prediction value of the target device and the follower device is obtained. Multiple sets of distance measurement data are recorded as d1, d2, d3, and the above dynamic position change information is further predicted to construct the observation information Z k , which can be expressed as follows:

[0069]

[0070] Among them, V k is the measurement noise matrix of the target device and the follower device (i.e. the above noise matrix), whose mean is 0. is the distance value calculated based on multiple sets of distance information d1, d2, and d3 obtained by the UWB sensor at time k.

[0071] In an optional embodiment, based on the observation information, an extended Kalman filter is used to perform posture change processing to obtain a second distance estimation value, including: when multiple position sensors are provided on the following device and the multiple position sensors respectively correspond to installation position coordinates, based on the observation information and the installation position coordinates respectively corresponding to the multiple position sensors, an observation matrix representing the mapping relationship between the observation information and the dynamic position change information is obtained; based on the observation matrix, an extended Kalman filter is used to perform posture change processing to obtain a second distance estimation value.

[0072] It can be understood that the observation information is combined with the installation position coordinates of the sensor to construct an observation matrix. The above observation matrix is ​​a mathematical model that describes the relationship between the observation information and the dynamic position change information. It combines the ranging data with the actual position change of the device to provide a mapping from the observation data to the state of the target device and the follower device. Using this observation matrix, the posture change processing is performed through the extended Kalman filter. The extended Kalman filter is a powerful tool that considers the nonlinear characteristics of the state transition and prediction model of the target device and the follower device, and linearly processes these nonlinear models at each moment. The extended Kalman filter first predicts the state of the target device and the follower device, updates this prediction based on the observation data, and provides a second distance estimate of the relative position between the follower device and the target device.

[0073] Optionally, the extended Kalman filter includes two stages: prediction and update. In the prediction stage, the state and error covariance of the target device and the follower device are first predicted based on the state estimate and control input of the previous time step. After the observation model is linearized, the predicted state estimate is used to predict the observation value. The state estimate and error covariance are updated by the calculated observation error covariance and Kalman gain. Through the above steps, the extended Kalman filter can approximate the true state of the target device and the follower device and provide accurate state estimate and error covariance.

[0074] Alternatively, according to the standard EKF calculation formula: Z k =h k +V k Among them, h k is the observation matrix, V k As the noise matrix, the following expression can be obtained:

[0075]

[0076] Among them, q i (i=1,2,3) is UWBi (i=1,2,3) The installation position coordinates of the sensor on the following device, is the predicted value of the relative position of the following device and the target device at the kth moment, that is, the second distance estimation value obtained at the kth moment.

[0077] Step S108: determining a target distance estimation value between the follower device and the target device based on the first distance estimation value and the second distance estimation value.

[0078] It can be understood that the accuracy of the first distance estimate predicted by the target model and the second distance estimate obtained by the posture change processing are analyzed. The first distance estimate and the second distance estimate are analyzed using the mean absolute error (MAE) and the root mean square error (RMSE). The closer the MAE and RMSE results are to 0, the higher the prediction accuracy.

[0079] In an optional embodiment, based on the first distance estimate and the second distance estimate, determining the target distance estimate between the following device and the target device includes: determining the average absolute error and the root mean square error of the first distance estimate and the second distance estimate; determining the absolute value of the deviation between the average absolute error and the root mean square error; when the absolute value of the deviation is equal to a predetermined threshold, determining the average value between the first distance estimate and the second distance estimate; using the average value as the target distance estimate; or, when the absolute value of the deviation is greater than the predetermined threshold, determining the arithmetic square root of the sum of the squares of the first distance estimate and the second distance estimate; and using the arithmetic square root as the target distance estimate.

[0080] It can be understood that the mean absolute error (MAE) and the root mean square error (RMSE) of the first distance estimate and the second distance estimate are calculated respectively to quantify the degree of difference between the two indicators RMSE and MAE. The absolute value of the deviation between MAE and RMSE is used to evaluate the degree of difference between the two error metrics, and the absolute value of the deviation reflects the consistency or difference between the two error evaluation methods.

[0081] If the absolute value of the deviation is equal to the predetermined threshold, it means that the two error measurement methods, RMSE and MAE, are statistically very close or similar, so the first distance estimate and the second distance estimate are comparable in accuracy. Calculating the average of the first distance estimate and the second distance estimate and using this average as the target distance estimate reduces the impact of random errors through averaging, providing a relatively stable estimation method.

[0082] If the absolute value of the deviation is greater than a predetermined threshold, it indicates that there is a significant difference between the two error measurement methods, RMSE and MAE, which means that one of the distance estimates is more reliable than the other. It is chosen to calculate the sum of the squares of the first distance estimate and the second distance estimate, and then take the arithmetic square root of the sum of squares (that is, the square root of the Euclidean distance) as the target distance estimate. This is conducive to obtaining a more robust estimate when the difference between the first distance estimate and the second distance estimate is large.

[0083] Optionally, RMSE and MAE are both used to describe the error between the predicted value and the true value. The difference between RMSE and MAE is that RMSE first squares the deviation. If the error is large, RMSE can amplify the error. The closer the RMSE and MAE results are to 0, the higher the prediction accuracy of the model. The formulas are as follows:

[0084]

[0085] in, represents the first distance estimate at the kth moment, represents the second distance estimate at the kth moment, and n represents a total of n moments.

[0086] Optionally, the relative position determination formula is as follows:

[0087]

[0088] in, is the estimated value of the target distance at time k.

[0089] Through the above step S102, the following state information of the following device and the target state information of the target device are obtained; in step S104, based on the following state information and the target state information, a target model is used for prediction to determine a first distance estimation value between the following device and the target device, wherein the target model is obtained based on the historical data training of the following device for the target device; in step S106, based on the following state information and the target state information, a posture change method is used for processing to determine a second distance estimation value between the following device and the target device; in step S108, based on the first distance estimation value and the second distance estimation value, a target distance estimation value between the following device and the target device is determined. The purpose of completing the distance prediction between the following device and the target device through the relationship between the two distance estimation values ​​can be achieved, and the technical effect of greatly improving the accuracy and precision of the relative position positioning between the following device and the target device is achieved, thereby solving the technical problem of unsatisfactory relative positioning accuracy between the following device and the target device.

[0090] Based on the above embodiments and optional embodiments, the present invention proposes an optional implementation mode, in which the application and following device is a drone, and the target device is a quadruped robot, which conduct inspections in a cable tunnel and follow each other. Figure 4 This is a schematic flow chart of an optional follower device cooperative control method provided by an embodiment of the present invention. The UWB sensor and the IMU sensor are respectively provided on the drone and the quadruped robot. The UWB sensor can collect ranging data, and the IMU sensor can collect motion data. The UWB between the drone and the quadruped robot can realize data exchange in a networkless environment at a short distance.

[0091] In order to improve the accuracy of the calculated target distance estimate, on the one hand, the target model can be used for processing, and the convolutional neural network can be used to collect the distance measurement data of the UWB sensor, and the IMU sensor can be used to collect the motion data for feature extraction to obtain the motion feature vector. The above motion feature vector is input into the long short-term memory network for processing to obtain the first distance estimate. On the other hand, the extended Kalman filter can be used for posture transformation processing to obtain the second distance estimate.

[0092] The mean absolute error (MAE) and root mean square error (RMSE) of the first distance estimate and the second distance estimate are calculated respectively to quantify the degree of difference between the two indicators RMSE and MAE. If the absolute value of the deviation is equal to a predetermined threshold, the average value of the first distance estimate and the second distance estimate is selected to be calculated, and this average value is used as the target distance estimate. If the absolute value of the deviation is greater than the predetermined threshold, the square sum of the first distance estimate and the second distance estimate is selected to be calculated, and then the arithmetic square root of this square sum (i.e., the square root of the Euclidean distance) is taken as the target distance estimate.

[0093] The above optional implementation method at least achieves the following effects: a method for collaborative control of a following device is proposed, which combines the data feature extraction capability of CNN (convolutional neural network) with the short-term memory characteristics of LSTM (long short-term memory) network, constructs a comprehensive target model and adds UWB sensors and IMU sensors to improve positioning accuracy. In the filtering method, the EKF (extended Kalman filter) algorithm used has better real-time performance under the condition of equal estimation accuracy. The EKF algorithm can effectively handle nonlinear problems, and appropriate approximation and optimization are performed during the calculation process, thereby improving the real-time performance of the algorithm. Not only does it overcome the excessive dependence of traditional estimation algorithms on model estimation accuracy, but it also optimizes the inherent time lag of unmanned systems. By comparing the estimated value of the target model with the observed value of the EKF (extended Kalman filter), the model parameters are adjusted in real time, thereby reducing the dependence on a specific training data set, and more accurate and stable estimation effects can be achieved in various environments, providing strong support for the intelligence and autonomy of unmanned systems.

[0094] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0095] In this embodiment, a follow-up device cooperative control device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module" and "device" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0096] According to an embodiment of the present invention, there is also provided an apparatus embodiment for implementing the method for cooperatively controlling a following device. Figure 5 A schematic diagram of a follower device cooperative control apparatus according to an embodiment of the present invention is shown in FIG. Figure 5 As shown, the above-mentioned following device cooperative control device includes an acquisition module 502, a first determination module 504, a second module 506, and a distance determination module 508. The device is described below.

[0097] An acquisition module 502 is used to acquire the following state information of the following device and the target state information of the target device;

[0098] A first determining module 504 is connected to the acquiring module 502 and is used to determine a first distance estimation value between the following device and the target device by using a target model for prediction based on the following state information and the target state information, wherein the target model is obtained by training based on historical data of the following device following the target device;

[0099] A second determination module 506, connected to the first determination module 504, is used to determine a second distance estimation value between the following device and the target device by processing in a posture change manner based on the following state information and the target state information;

[0100] The distance determination module 508 is connected to the second determination module 506 and is used to determine a target distance estimation value between the following device and the target device based on the first distance estimation value and the second distance estimation value.

[0101] In a following device cooperative control device provided by an embodiment of the present invention, an acquisition module 502 is used to acquire the following state information of the following device and the target state information of the target device; a first determination module 504 is connected to the acquisition module 502 and is used to predict based on the following state information and the target state information, using a target model to determine a first distance estimation value between the following device and the target device, wherein the target model is obtained by training based on historical data of the following device following the target device; a second determination module 506 is connected to the first determination module 504 and is used to process based on the following state information and the target state information, using a posture change method to determine a second distance estimation value between the following device and the target device; a distance determination module 508 is connected to the second determination module 506 and is used to determine a target distance estimation value between the following device and the target device based on the first distance estimation value and the second distance estimation value. The purpose of completing the distance prediction of the following device and the target device through the relationship between the two distance estimation values ​​is achieved, and the technical effect of greatly improving the accuracy and precision of the relative position positioning between the following device and the target device is achieved, thereby solving the technical problem of unsatisfactory relative positioning accuracy of the following device and the target device.

[0102] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0103] It should be noted that the acquisition module 502, the first determination module 504, the second determination module 506, and the distance determination module 508 correspond to steps S102 to S108 in the embodiment, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the modules as part of the device can be run in a computer terminal.

[0104] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0105] The above-mentioned following device collaborative control device may also include a processor and a memory. The acquisition module 502, the first determination module 504, the second determination module 506, the distance determination module 508, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0106] The processor includes a kernel, which retrieves the corresponding program unit from the memory. There can be one or more kernels. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.

[0107] An embodiment of the present invention provides a non-volatile storage medium on which a program is stored. When the program is executed by a processor, a method for cooperative control of a follower device is implemented.

[0108] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: obtaining the following state information of the following device and the target state information of the target device; based on the following state information and the target state information, using a target model for prediction to determine a first distance estimation value between the following device and the target device, wherein the target model is obtained by training based on historical data of the following device following the target device; based on the following state information and the target state information, using a posture change method for processing to determine a second distance estimation value between the following device and the target device; based on the first distance estimation value and the second distance estimation value, determining a target distance estimation value between the following device and the target device. The device in this article may be a server, a PC, etc.

[0109] The present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: obtaining following state information of a following device and target state information of a target device; based on the following state information and the target state information, using a target model for prediction to determine a first distance estimate between the following device and the target device, wherein the target model is obtained by training based on historical data of the following device following the target device; based on the following state information and the target state information, using a posture change method for processing to determine a second distance estimate between the following device and the target device; based on the first distance estimate and the second distance estimate, determining a target distance estimate between the following device and the target device.

[0110] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0114] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0115] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0116] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0117] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0118] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0119] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for cooperative control of a following device, characterized in that: include: Obtain the following status information of the following device and the target status information of the target device; Based on the following state information and the target state information, a target model is used for prediction to determine a first distance estimation value between the following device and the target device, wherein the target model is trained based on historical data of the following device following the target device; Based on the following state information and the target state information, a second distance estimation value between the following device and the target device is determined by processing in a posture change manner; Based on the first distance estimate and the second distance estimate, a target distance estimate between the following device and the target device is determined.

2. The method according to claim 1, characterized in that The step of using a target model to predict based on the following state information and the target state information to determine a first distance estimation value between the following device and the target device includes: Based on the following state information and the target state information, a convolutional neural network is used to extract features to obtain a motion feature vector; Based on the motion feature vector, a long short-term memory network is used for processing to obtain the first distance estimation value, wherein the long short-term memory network is provided with a state memory unit and a hidden layer, the state memory unit is used to store the operation state information of the previous moment, and the hidden layer is used to record the estimated values ​​of the center of mass sideslip angles corresponding to the following device and the target device respectively at the previous moment.

3. The method according to claim 1, characterized in that The determining of a second distance estimation value between the following device and the target device by processing in a posture change manner based on the following state information and the target state information includes: Determine dynamic position change information between the target device and the follower device; Based on the dynamic position change information, determining observation information between the target device and the following device, wherein the observation information is used to characterize the association between the ranging data and the relative position between the target device and the following device; Based on the observation information, an extended Kalman filter is used to perform posture change processing to obtain the second distance estimation value.

4. The method according to claim 3, characterized in that The following state information includes following motion data, the target state information includes target motion data, and the determining of dynamic position change information between the target device and the following device includes: Based on the target motion data, determining a target motion state of the target device in a corresponding target coordinate system; Based on the following motion data, determining a following motion state of the following device in a corresponding following coordinate system; Based on the target motion state and the follow-up motion state, the dynamic position change information is obtained by using a coordinate system conversion matrix.

5. The method according to claim 3, characterized in that: The following state information includes following position data, the target state information includes target position data, and the determining of observation information between the target device and the following device based on the dynamic position change information includes: Determining distance measurement data between the target device and the following device based on the following position data and the target position data; The observation information is determined based on the ranging data, the dynamic position change information, and the noise matrix of the target device and the follower device.

6. The method according to claim 1, characterized in that The determining, based on the first distance estimation value and the second distance estimation value, a target distance estimation value between the following device and the target device comprises: determining a mean absolute error and a root mean square error of the first distance estimate and the second distance estimate; determining an absolute value of a deviation between the mean absolute error and the root mean square error; When the absolute value of the deviation is equal to a predetermined threshold, determining an average value between the first distance estimation value and the second distance estimation value; using the average value as the target distance estimation value; or, When the absolute value of the deviation is greater than a predetermined threshold, an arithmetic square root of the sum of the squares of the first distance estimation value and the second distance estimation value is determined; and the arithmetic square root is used as the target distance estimation value.

7. The method according to any one of claims 1 to 6, characterized in that: The following device and the target device are respectively provided with a position sensor and an inertial measurement unit. The position sensor is an ultra-wideband UWB sensor. The UWB sensor has data transmission capability. The following device and the target device are respectively a drone or a ground robot.

8. A collaborative control device for a following device, characterized in that: include: An acquisition module, used to acquire the following state information of the following device and the target state information of the target device; a first determination module, configured to determine a first distance estimation value between the following device and the target device by using a target model for prediction based on the following state information and the target state information, wherein the target model is trained based on historical data of the following device following the target device; A second determination module is used to determine a second distance estimation value between the following device and the target device by processing in a posture change manner based on the following state information and the target state information; The distance determination module is configured to determine a target distance estimation value between the following device and the target device based on the first distance estimation value and the second distance estimation value.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the following device cooperative control method described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the following device collaborative control method described in any one of claims 1 to 7.

Images