Mobile equipment control method and device, mobile equipment and readable storage medium

Abstract

The invention relates to a mobile equipment control method and device, electronic equipment, a computer readable storage medium and a computer program product, and the method comprises the steps: obtaining weight matrix adjustment data under the condition that a data collection device is determined to be in an abnormal state based on motion state data; according to the weight matrix adjustment data, correcting process parameters of the motion state data-based predicted pose data to obtain target pose data; and according to the target pose data and the target position, the mobile device is controlled to act, and the target position is the position to which the mobile device needs to move currently. The operation state of each sensor is monitored in real time based on sensor data in the operation process of the mobile equipment, and the weight of an abnormal sensor is adaptively adjusted under the condition that the abnormal sensor is recognized, and is corrected to the process of predicting pose data based on motion state data, so that the accuracy of a fusion positioning result is ensured, and the accuracy of the fusion positioning result is improved. And thus, accurate movement of the mobile equipment is ensured.

Description

Technical Field

[0001] The present application relates to the field of mobile robot positioning technology, and in particular to a mobile device control method, device, mobile device, computer-readable storage medium and computer program product. Background Art

[0002] With the development of mobile robot technology, robots with mobility, autonomous positioning and navigation have received extensive attention and strong support both in industry and academia. In the field of mobile robot positioning and navigation, the Extended Kalman Filter (EKF) is the most common and widely used multi-sensor fusion algorithm. It is suitable for nonlinear systems and can integrate multiple sensors such as wheel odometry, inertial measurement unit (IMU), visual odometry (VO), global positioning system (GPS / GNSS), etc. to improve positioning accuracy.

[0003] However, in the current EKF fusion solution, if one or more sensors are unstable, such as wheel odometer slippage, visual odometer drift, and GPS signal distortion, it will interfere with the overall fusion positioning result and cause positioning errors in mobile robots in practical applications. Summary of the invention

[0004] Based on this, it is necessary to provide a mobile device control method, device, electronic device, computer-readable storage medium and computer program product to address the technical problem of the above-mentioned mobile robot positioning error.

[0005] In a first aspect, the present application provides a method for controlling a mobile device, the method comprising:

[0006] Acquire motion state data collected by at least one data collection device;

[0007] In the case where it is determined based on the motion state data that the data acquisition device is in an abnormal state, obtaining weight matrix adjustment data;

[0008] According to the weight matrix adjustment data, a process parameter of predicting the posture data based on the motion state data is corrected to obtain target posture data;

[0009] The movement of the mobile device is controlled according to the target posture data and the target position, wherein the target position is the position to which the mobile device currently needs to move.

[0010] In one embodiment, the method further comprises:

[0011] Inputting the motion state data into a device state monitoring model to obtain a device state monitoring result;

[0012] It is determined whether there is a data acquisition device in an abnormal state according to the device state monitoring result.

[0013] In one embodiment, the step of obtaining the device status monitoring model includes:

[0014] The motion state data is used as a state set, the trust weight data is used as an action set, and a reward function is designed according to the state judgment result of each data acquisition device; the trust weight data is used to characterize whether the state of each data acquisition device is a normal state or an abnormal state;

[0015] Build and initialize the policy network, value network, and target network;

[0016] The experience sample data set is collected by combining the current state set, action set, reward function and next state set;

[0017] The strategy network, the value network and the target network are updated based on the experience sample data set to obtain a device status monitoring model.

[0018] In one of the embodiments, the device status monitoring result includes the trust weight data;

[0019] Judging whether a data acquisition device is in an abnormal state according to the device state monitoring result includes:

[0020] outputting the status of each data acquisition device based on the trust weight data;

[0021] In the case that the state of any data acquisition device is an abnormal state, determining that there is a data acquisition device in an abnormal state;

[0022] When the states of all data acquisition devices are normal, it is determined that no data acquisition device is in an abnormal state.

[0023] In one embodiment, obtaining weight matrix adjustment data includes:

[0024] Calculation is performed based on the trust weight data and the original weight matrix data to obtain weight matrix adjustment data.

[0025] In one embodiment, predicting the posture data based on the motion state data includes:

[0026] The motion state data collected by each data acquisition device are fused through the extended Kalman filter algorithm to predict the posture data.

[0027] In one embodiment, the step of modifying the process parameters of predicting the posture data based on the motion state data according to the weight matrix adjustment data includes:

[0028] The covariance matrix of the extended Kalman filter algorithm in the fusion process is corrected according to the weight matrix adjustment data.

[0029] In a second aspect, the present application provides a mobile device control device, the device comprising:

[0030] A data acquisition module, used to acquire motion state data acquired by at least one data acquisition device;

[0031] An abnormality processing module, used for obtaining weight matrix adjustment data when it is determined based on the motion state data that a data acquisition device is in an abnormal state;

[0032] A parameter correction module, used for correcting the process parameters of the posture data predicted based on the motion state data according to the weight matrix adjustment data to obtain the target posture data;

[0033] The mobile control module is used to control the movement of the mobile device according to the target posture data and the target position, where the target position is the position to which the mobile device currently needs to move.

[0034] In a third aspect, the present application provides a mobile device, including a controller and at least one data acquisition device connected to the controller, wherein the controller is used to control the action of the mobile device according to the method described in any of the above embodiments.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0036] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0037] The above-mentioned mobile device control method, device, electronic device, computer-readable storage medium and computer program product, after acquiring the motion state data collected by at least one data acquisition device, obtains weight matrix adjustment data when it is determined based on the motion state data that the data acquisition device is in an abnormal state, and corrects the process parameters of the posture data predicted based on the motion state data according to the weight matrix adjustment data to obtain the target posture data; controls the movement of the mobile device according to the target posture data and the target position, and the target position is the position to which the mobile device currently needs to move. By monitoring the operating status of each sensor in real time based on the sensor data during the operation of the mobile device, and adaptively adjusting the weight of the abnormal sensor when an abnormal sensor is identified, and correcting it to the process of predicting the posture data based on the motion state data, the accuracy of the fusion positioning result is guaranteed, thereby ensuring the precise movement of the mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A diagram of an application environment of a mobile device control method in an embodiment;

[0040] Figure 2 A schematic diagram of a flow chart of a method for controlling a mobile device in one embodiment;

[0041] Figure 3 A schematic diagram of a flow chart of a method for controlling a mobile device in another embodiment;

[0042] Figure 4 A schematic flow chart of a step of acquiring a device status monitoring model in one embodiment;

[0043] Figure 5 is a flowchart of a method for controlling a mobile device in yet another embodiment;

[0044] Figure 6 A schematic diagram of a flow chart of a method for controlling a mobile device in another embodiment;

[0045] Figure 7 is a structural block diagram of a mobile device control device in one embodiment;

[0046] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The mobile device control method provided in the embodiment of the present application can be applied to Figure 1 In the mobile device shown. The mobile device includes a controller 102 and at least one data acquisition device 104 connected to the controller 102, and the controller is used to control the action of the mobile device according to the motion state data collected by each data acquisition device 104. It can be understood that the mobile device targeted by the present application is a device that can automatically control the movement of the position, which can be a mobile robot or an autonomous driving vehicle. The data storage system can store the data that the controller 102 needs to process. The data storage system can be integrated on the controller 102, or it can be placed on the cloud or other network servers.

[0049] Specifically, the controller 102 obtains motion state data collected by at least one data acquisition device 104, and when it is determined based on the motion state data that the data acquisition device 104 is in an abnormal state, obtains weight matrix adjustment data, and corrects process parameters of posture data predicted based on the motion state data according to the weight matrix adjustment data to obtain target posture data, and finally controls the movement of the mobile device according to the target posture data and the target position, where the target position is the position to which the mobile device currently needs to move.

[0050] Among them, the controller 102 can be a control chip or a control circuit board set inside the mobile device, or it can be an external control system based on wireless communication. The external control system can be implemented through devices such as terminals or servers. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0051] In an exemplary embodiment, Figure 2 As shown, a mobile device control method is provided, and the method is applied to Figure 1The controller 102 in FIG. 1 is taken as an example to illustrate the method, which includes the following steps 201 to 208. Among them:

[0052] Step 201: Acquire motion state data collected by at least one data collection device.

[0053] Among them, the data acquisition device is a device that measures the current motion information of the mobile device and the state information of the environment in which the mobile device is located that affects the movement of the mobile device, and is used to assist the mobile device in achieving high-precision positioning and navigation. It can be understood that for mobile devices that move in different scenarios, the type of data acquisition device they are equipped with is not fixed. For example, for indoor mobile devices, an inertial measurement unit (IMU) and an encoder combination are generally used for positioning and navigation, while a global positioning system (GPS / GNSS) is used for positioning outdoors. Other sensors such as wheel odometers (odom) and visual odometers (VO) can also be used as data acquisition devices for mobile devices to improve the positioning accuracy of mobile devices.

[0054] Specifically, based on the different data acquisition devices carried on the mobile device, the collected motion state data is also different, which can be used to characterize the current different motion states of the mobile device, and can also be used to characterize the different state information of the environment in which the mobile device is currently located. Taking the data acquisition device as an inertial measurement unit (IMU) as an example, the collected motion state data can be the acceleration and angular velocity information of the mobile device. Taking the data acquisition device as a global positioning system (GPS / GNSS) as an example, the collected motion state data can be the location information of the mobile device. Taking the data acquisition device as a wheel odometer (odom) or a visual odometer (VO) as an example, the collected motion state data can be the displacement information of the mobile device.

[0055] Furthermore, after the mobile device is started, the controller can continuously obtain the motion state data collected by at least one data acquisition device carried by the mobile device as a data basis for positioning and navigating the mobile device. It can be understood that based on the different working principles of the data acquisition device, the acquired motion state data can be data at the current time point (such as acceleration, angular velocity and position information), or data corresponding to the current time period (such as displacement information).

[0056] Step 204: when it is determined based on the motion state data that the data acquisition device is in an abnormal state, obtain weight matrix adjustment data.

[0057] It can be understood that for current mobile devices, after the data acquisition device mounted thereon is started to assist in positioning and navigation, the motion state data collected by the fixed data acquisition device is fused and calculated to predict the posture data. If the data acquisition device is unstable, such as the wheel odometer slipping, the visual odometer drifting, or the GPS signal distortion, the result will still be included in the fusion calculation to obtain the posture data, which will interfere with the overall fusion positioning result and cause positioning errors of the mobile device in actual applications.

[0058] For example, after the positioning error of the mobile device is found based on the predicted posture data, the data acquisition device in the abnormal state can be replaced to ensure the normal operation of the mobile device. For example, the data acquisition device can be replaced with a data acquisition device with higher accuracy and higher robustness. However, it can be understood that the above solution requires stopping the operation of the mobile device and replacing the data acquisition device on it, which not only delays the normal operation of the mobile device, but also may cost more.

[0059] Furthermore, in an embodiment of the present application, it is possible to first determine whether there is a data acquisition device in an abnormal state based on the motion state data, and when it is determined based on the motion state data that there is a data acquisition device in an abnormal state, obtain weight matrix adjustment data to correct the process parameters of the posture data obtained by the fusion calculation, thereby preventing the motion state data collected by the data acquisition device in an abnormal state from interfering with the overall fusion positioning result without affecting the operation of the mobile device, thereby ensuring the precise movement of the mobile device.

[0060] Specifically, the method of determining whether the data acquisition device is in an abnormal state based on the motion state data may be to preset a range of motion state data under a normal state for the data acquisition device, and then make a judgment based on whether the collected motion state data exceeds the range of motion state data under a normal state. Alternatively, a state monitoring model may be obtained by training a machine learning algorithm based on historical motion state data collected by the data acquisition device under normal and abnormal states, and then a judgment may be made based on the trained state monitoring model to determine whether the data acquisition device is in an abnormal state. Exemplarily, the state monitoring model may be trained by a reinforcement learning algorithm.

[0061] Step 206, modifying the process parameters of predicting the posture data based on the motion state data according to the weight matrix adjustment data to obtain the target posture data.

[0062] Among them, the weight matrix adjustment data is used to characterize which data acquisition device is in an abnormal state, and how to reduce the impact of the motion state data collected by the data acquisition device in the abnormal state on the overall fusion positioning result.

[0063] Specifically, the method of predicting posture data based on motion state data can be to first predict the posture data of the next moving state of the mobile device based on each motion state data collected, and then perform fusion calculation on the predicted posture data obtained based on each motion state data to obtain the target posture data.

[0064] Among them, the process of predicting the posture data of the next moving state of the mobile device based on the collected motion state data can be to first estimate the posture of the mobile device based on each motion state data to obtain the estimated posture data of the current state of the mobile device, and then predict the next moving state of the mobile device through the corresponding dynamic model based on the estimated posture data corresponding to each motion state data to obtain multiple predicted posture data.

[0065] Secondly, the method of performing fusion calculation based on each predicted posture data to obtain the target posture data can be achieved through various types of multi-sensor fusion algorithms, such as weighted average method, Kalman filter algorithm and DS evidence reasoning method, etc., or artificial intelligence algorithms such as fuzzy logic reasoning and neural network algorithm. In this embodiment, the posture data predicted by each motion state data is fused and calculated by the extended Kalman filter algorithm (EKF) to obtain the target posture data.

[0066] Furthermore, the fusion calculation of each predicted posture data by the multi-sensor fusion algorithm is essentially to design the fusion confidence of each motion state data (that is, each data acquisition device) in the algorithm process, and then combine the fusion confidence to perform fusion calculation on each predicted posture data. Furthermore, when it is determined based on the motion state data that the data acquisition device is in an abnormal state, it is necessary to adjust the fusion confidence of the data acquisition device in the abnormal state so that it participates less in the fusion calculation and reduces its impact on the overall fusion positioning result.

[0067] Furthermore, the fusion confidence needs to be corrected according to the weight matrix adjustment data to obtain the target posture data that is not disturbed by the abnormal state of the data acquisition device.

[0068] Step 208, controlling the movement of the mobile device according to the target posture data and the target position, where the target position is the position to which the mobile device currently needs to move.

[0069] Specifically, the target position is the position to which the mobile device is currently preparing to move. After obtaining the target posture data that is not disturbed by the abnormal state of the data acquisition device, the target position can be used as the end point, and the mobile device can be controlled according to the target posture data until it reaches the target position.

[0070] Exemplarily, a method of controlling the movement of a mobile device according to target posture data with the target position as the end point may be to obtain lidar point cloud data based on a lidar, and then build a high-precision point cloud map for the mobile device based on the lidar point cloud data and the target posture data through a SLAM (Simultaneous Localization and Mapping) algorithm, and then control the movement of the mobile device to reach the target position based on the relative position of the target position on the high-precision point cloud map.

[0071] The above-mentioned mobile device control method monitors the operating status of each sensor in real time based on sensor data during the operation of the mobile device, and when an abnormal sensor is identified, adaptively adjusts the weight of the abnormal sensor, and corrects the process of predicting the posture data based on the motion state data to ensure the accuracy of the fused positioning result, thereby ensuring the precise movement of the mobile device.

[0072] In an exemplary embodiment, Figure 3 As shown, the mobile device control method further includes the following steps 202 to 203, wherein:

[0073] Step 202: input the motion state data into the device state monitoring model to obtain the device state monitoring result.

[0074] Among them, the device status monitoring model is a prediction model with good recognition effect obtained by inputting a large amount of historical motion state data collected by the data acquisition device in normal and abnormal states into the reinforcement learning network model for training, adjustment and verification. Among them, the reinforcement learning network model that can be used is not unique, including but not limited to Deep Q-Networks (DQN), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO) and Deterministic Policy Gradient (DPG) etc.

[0075] Specifically, after obtaining the motion state data collected by at least one data acquisition device, each motion state data can be input into the device state monitoring model to obtain the device state monitoring result. It can be understood that the device state monitoring result can be used to characterize whether there is an abnormal state of the data acquisition device.

[0076] Furthermore, for the above-mentioned types of reinforcement learning algorithms, DQN is the first method to combine deep learning with reinforcement learning. By using large neural networks as function approximators, control strategies can be learned directly from high-dimensional video pixels. However, since DQN has no way to output the behavior value function for each action, it cannot solve continuous action space problems such as mobile device control. The deterministic policy gradient method (DPG) can solve the problem of continuous action space by expressing the strategy as a policy function and mapping the state to a deterministic action, but it cannot learn strategies directly from high-dimensional inputs. The Deep Deterministic Policy Gradient (DDPG) algorithm combines the two, introduces the successful experience of the DQN algorithm on the basis of the DPG algorithm, and realizes end-to-end learning directly from raw data.

[0077] Exemplarily, the present application can obtain a device state monitoring model by training a deep deterministic policy gradient algorithm. The deep deterministic policy gradient algorithm combines deep learning and deterministic policy gradient methods, and is particularly suitable for reinforcement learning tasks in continuous action spaces.

[0078] Specifically, a deterministic policy means that under a given state, the policy directly outputs a specific action rather than a probability distribution. Formally, a deterministic policy can be expressed as μ(s), where s is the state and μ is the policy function. Policy gradient refers to a method of optimizing a policy by gradient ascent. For a deterministic policy, the goal is to maximize the expected return, which can be expressed as:

[0079]

[0080] Among them, θ is the strategy parameter, τ is the trajectory, and R(τ) is the total return of the trajectory.

[0081] Furthermore, the deterministic policy gradient can be calculated by the following formula:

[0082]

[0083] in, For behavioral strategies, is the generated state distribution, is the action value function.

[0084] Taking the device state monitoring model obtained by training the deep determination policy gradient algorithm as an example, in an exemplary embodiment, Figure 4 As shown, the step of obtaining the device state monitoring model in step 202 includes the following steps 302 to 308, wherein:

[0085] Step 302, taking the motion state data as a state set, taking the trust weight data as an action set, and designing a reward function according to the state judgment result of each data acquisition device; the trust weight data is used to characterize whether the state of each data acquisition device is a normal state or an abnormal state.

[0086] Specifically, the motion state data collected by each data collection device is represented by a state to form a state set s t The trust weight data corresponding to each data collection device is used as an action representation to form an action set a t It can be understood that the trust weight data corresponding to the data acquisition device includes a binary decision on whether the data acquisition device is trusted, that is, whether the motion state data collected by the data acquisition device can participate in the fusion calculation of the target posture data.

[0087] Among them, the trust weight data can be used to characterize the state of each data acquisition device as normal or abnormal. Then, in the model training stage, the motion state data and the trust weight data can be determined to have a one-to-one correspondence through data annotation, and this relationship is used to characterize each motion state data as data collected by the data acquisition device in a normal state or an abnormal state. Then, in the stage of model prediction, the trust weight data can be obtained based on the motion state data collected in real time to determine whether the data acquisition device is in a normal state or an abnormal state at this time.

[0088] Furthermore, the reward function r can be designed according to the state judgment results of each data acquisition device. t That is, if the trust weight data output by the model based on the motion state data is correctly judged for the state of each data acquisition device, a positive reward is given, otherwise, a negative reward is given.

[0089] Step 304, construct and initialize the policy network, value network and target network.

[0090] Specifically, the policy function μ(s,θ) is used to construct a policy network (Actor network), which maps the state s to a deterministic action. The value network (Critic network) is constructed for the action-value function.

[0091] Furthermore, the value network is approximated by a neural network In the training process of the policy network μ(s,θ), end-to-end learning can be achieved directly from the original data. However, if the same neural network is used to represent the final target network and the network currently being updated, the learning process will be unstable because the same network parameters need to be used to calculate the gradient of the network while frequently updating the gradient. Therefore, it is possible to use a value network based on the value network. And the strategy network μ(s,θ) is copied to obtain the target network and .

[0092] Furthermore, after obtaining the strategy network, value network and target network, they need to be initialized respectively.

[0093] Step 306, collect the experience sample data set by combining the current state set, action set, reward function and next state set.

[0094] It is understandable that because the data of reinforcement learning has Markov properties, it does not meet the premise that the samples need to be independent and identically distributed for training neural networks. Therefore, when using neural networks for reinforcement learning, the training process is very unstable. In order to ensure the learning effect, it is necessary to break the correlation of training data, so experience replay can be used to solve this problem. Experience replay specifically refers to combining the experience (state, action, reward, etc.) obtained from model interaction, storing it in the "experience replay buffer", and then randomly sampling it during the model training process for model update. Such a design can not only improve the efficiency of data utilization, break the time correlation between data, and promote training to obtain a more comprehensive and stable model.

[0095] Specifically, actions can be performed in the initialized network environment based on the current state set to collect a state transition sequence (s ) composed of the current state set, action set, reward function and next state set. t ,a t ,r t ,s t+1 ), and store them to form an experience sample data set.

[0096] Exemplarily, in order to ensure effective exploration of the algorithm during the learning process, noise, such as Gaussian noise or Ornstein-Uhlenbeck process, may be added when selecting actions under a deterministic strategy.

[0097] Step 308 , updating the strategy network, the value network, and the target network based on the experience sample data set to obtain a device status monitoring model.

[0098] Specifically, samples can be randomly drawn in batches from the empirical sample data set (s t ,a t ,r t ,s t+1 ) to update the value network. Among them, the mean square error loss function can be used to update the network parameters in the value network , which is specifically expressed as the following formula:

[0099]

[0100] in, To obtain the target value through the target network calculation, , is the discount factor.

[0101] Furthermore, the policy gradient formula is used to update the parameters of the policy network , as follows:

[0102]

[0103] Furthermore, in order to improve the stability of learning, a soft update method is used to gradually update the target network parameters, for example, a sliding average method can be used:

[0104]

[0105]

[0106] in, It is a positive constant, usually set between 0.001 and 0.01.

[0107] It can be understood that a trained device status monitoring model can be obtained by repeating the above network update process until the strategy converges or the maximum number of iterations is reached.

[0108] Step 203: determine whether any data acquisition device is in an abnormal state according to the device state monitoring result.

[0109] Specifically, the device state monitoring result is output by the device state monitoring model, which can be used to characterize whether there is a data acquisition device in an abnormal state. The device state monitoring result may include the corresponding state monitoring result of each data acquisition device, and then it can be determined whether each data acquisition device is in an abnormal state. It can be understood that as long as there is a data acquisition device in an abnormal state, it can be determined that there is a data acquisition device in an abnormal state.

[0110] In an exemplary embodiment, the device status monitoring result includes trust weight data. Figure 5 As shown, step 203 includes the following steps 402 to 406, wherein:

[0111] Step 402: output the status of each data collection device based on the trust weight data.

[0112] Step 404: when the state of any data acquisition device is abnormal, determine that there is a data acquisition device in abnormal state.

[0113] Step 406: When all data acquisition devices are in normal state, determine that no data acquisition device is in abnormal state.

[0114] Specifically, the trust weight data is represented by a binary decision of whether to trust the data acquisition device. For example, the closer the decision value is to 1, the closer the data acquisition device is to a normal state, and vice versa, the closer it is to 0, the closer it is to an abnormal state. Furthermore, the trust value thresholds corresponding to the normal state and the abnormal state can be preset, and then the trust value thresholds are compared with the trust weight data output by the data acquisition device to determine the state of each data acquisition device. It can be understood that the state of the output data acquisition device can be a normal state or an abnormal state.

[0115] Furthermore, after obtaining the status of all data acquisition devices, it can be determined whether it is necessary to correct the process parameters for predicting the posture data based on the motion state data according to whether there is a data acquisition device in an abnormal state. It can be understood that when the status of all data acquisition devices is normal, the overall fusion positioning result will not be disturbed, and the prediction of posture data based on the motion state data can be performed normally without any adjustment of the process parameters. In the case where the status of any data acquisition device is abnormal, it may include that there is one data acquisition device in an abnormal state, or multiple data acquisition devices in an abnormal state, and then it is necessary to determine whether there is a data acquisition device in an abnormal state, and the process parameters need to be corrected to avoid the motion state data collected by the data acquisition device in an abnormal state from interfering with the overall fusion positioning result, so as to ensure the accurate movement of the mobile device.

[0116] Exemplarily, the method for obtaining weight matrix adjustment data in the present application can be to directly use the trust weight data output by the model based on the motion state data as a process parameter to participate in the calculation. It can be understood that the weight matrix adjustment data is essentially used to correct the fusion confidence of the fusion calculation based on the predicted posture data, and the trust weight data is represented by a binary decision on whether to trust the data acquisition device, which is actually the weighted confidence set for the data acquisition device, and can actually directly replace the process parameters to participate in the fusion calculation, so that it participates less in the fusion calculation and reduces its impact on the overall fusion positioning result.

[0117] In an exemplary embodiment, obtaining weight matrix adjustment data in step 204 includes: performing calculations based on the trust weight data and the original weight matrix data to obtain the weight matrix adjustment data.

[0118] Specifically, the fusion confidence calculated based on the predicted posture data can be corrected by combining the trust weight data with the original weight matrix data to obtain the weight matrix adjustment data. With such a design, the fusion confidence of the data acquisition device in a normal state in the original weight matrix data can be used, so that the impact on the overall fusion positioning result is smaller.

[0119] In an exemplary embodiment, predicting the posture data based on the motion state data in step 206 includes: fusing the motion state data collected by each data collection device through an extended Kalman filter algorithm to predict the posture data.

[0120] Specifically, the extended Kalman filter algorithm can achieve data fusion through the following steps, including:

[0121] 1. Predict the predicted state at the next moment based on the optimal estimated state and state transfer matrix at the current moment;

[0122] 2. Predict the predicted covariance matrix at the next moment based on the optimal estimated covariance matrix at the current moment and the process noise matrix;

[0123] 3. Calculate the Kalman gain at the next moment based on the prediction covariance matrix of the next moment predicted by the current moment and the covariance matrix of the measured value at the next moment;

[0124] 4. Calculate the optimal estimated state at the next moment based on the predicted state at the current moment, the measured state at the next moment, the Kalman gain, and the observation matrix;

[0125] 5. Based on the predicted covariance matrix and Kalman gain of the next moment predicted at the current moment, predict the optimal estimated covariance matrix of the next moment.

[0126] The above iterative steps are repeated in this way, and the motion state data collected by each data acquisition device are fused and processed to obtain the optimal estimated state, that is, the target posture data is predicted.

[0127] It can be understood that for the above-mentioned positioning error problem caused by the instability of the data acquisition device, the inventors have found through research that an adaptive fusion EKF algorithm can be used to optimize the extended Kalman filter algorithm. Specifically, it can be achieved by changing the parameters of the covariance matrix in the original algorithm, thereby adjusting the weights of each data acquisition device to improve the positioning error problem. However, the adaptive fusion EKF algorithm requires additional calculation steps to estimate and adjust the noise covariance matrices R and Q, which will greatly increase the amount of calculation. In particular, when it is applied to actual mobile devices, serious processing delays occur.

[0128] In an exemplary embodiment, in step 206, the process parameters of predicting posture data based on motion state data are corrected according to the weight matrix adjustment data, including: correcting the covariance matrix in the fusion process of the extended Kalman filter algorithm according to the weight matrix adjustment data.

[0129] Specifically, based on the above findings, this implementation intends to use a device state monitoring model trained with a deterministic policy gradient to perform real-time detection on each data acquisition device to determine whether it is in an abnormal state. Then, in the case where a data acquisition device is in an abnormal state, weight matrix adjustment data is obtained to correct the covariance matrix of the extended Kalman filter algorithm.

[0130] It can be understood that the weight matrix adjustment data can be calculated based on the trust weight data and the original weight matrix data and used to adjust the covariance matrix, or it can be directly used to replace the covariance matrix data.

[0131] In this embodiment, the device status monitoring model obtained by the above training does not directly participate in data fusion. Instead, when an abnormality occurs in the data acquisition device, the covariance matrix used for fusion by the EKF algorithm is changed to achieve adaptive adjustment of the fusion confidence of each data acquisition device during the EKF fusion process to ensure the accuracy of the fusion positioning result.

[0132] In a specific embodiment, the mobile device is a mobile robot, and the data acquisition device selects three sensors including a wheel odometer (odom), an inertial measurement unit (IMU) and a global positioning system (GPS / GNSS). The mobile device control method provided in this application is explained by taking the case where the wheel odometer of the mobile robot is disturbed (such as slipping) during operation as an example.

[0133] Among them, the inertial measurement unit (IMU) is a device used to measure the motion state of a target in three-dimensional space. It is based on Newton's laws of mechanics. By measuring the acceleration of the target in the inertial reference system, integrating it over time, and transforming it into the navigation coordinate system, it is possible to obtain information such as speed, yaw angle, and position in the navigation coordinate system, thereby achieving precise positioning. It usually includes an accelerometer to measure the acceleration of the target in all directions, so that the speed and displacement of the target can be calculated. It may also include a gyroscope to measure the angular velocity of the target around each axis to determine the rotational motion of the target.

[0134] The Global Positioning System (GPS / GNSS) is a high-precision radio navigation positioning system based on artificial satellites. It mainly relies on radio signals sent by satellites for positioning and navigation. The radio signal includes the satellite's position, speed and time information. After the mobile device receives the radio signal, it determines its own position, speed and time by measuring the transmission time of the signal.

[0135] A wheel odometer (odom) is a device that uses wheel speed to measure the travel of a robot or vehicle. By monitoring the rotation angle and travel distance of each wheel, the position change of the mobile device during movement can be derived. Specifically, a photoelectric encoder is usually used to measure the wheel speed. When the motor rotates, it drives the code disk (magnetic code disk or photoelectric code disk, etc.) connected to the wheel to rotate. The encoder receives a pulse signal, and the number of pulses multiplied by the coefficient can be used to determine how many times the wheel has rotated, and then the displacement information of the mobile device can be calculated.

[0136] Specifically, we can first collect a large amount of historical motion state data under normal and abnormal conditions based on three sensors, namely, wheel odometer (odom), inertial measurement unit (IMU) and global positioning system (GPS / GNSS). Then, we can form a state set based on the historical motion state data collected above. ,in, Represents location information, Represents angle information, Represents line speed information, Represents angular velocity information, Represents acceleration information. And the historical state of the data acquisition device is labeled to obtain an action set.

[0137] Furthermore, the above state set and action set are formed into a training set and a test set, and the device state monitoring model is obtained through training through the process from step 302 to step 308. Then the trained model is deployed to the mobile robot. The specific training process of the device state monitoring model can also be referred to Figure 6 As shown. After the device status monitoring model is deployed, the mobile robot can monitor the status of each sensor in real time to determine whether it is in an abnormal state. If one or more sensor data are detected to be abnormal, the data is adjusted according to the weight matrix obtained by monitoring, and the covariance matrix of the EKF algorithm is adaptively and dynamically adjusted.

[0138] Two examples of covariance matrices obtained by adjusting the model are given below to illustrate the change of the adaptively changed covariance when the wheel odometer is disturbed.

[0139] The noise covariance matrix of each sensor when it is in normal state is as follows:

[0140]

[0141] The estimated covariance matrix when each sensor is in a normal state is as follows:

[0142]

[0143] The noise covariance matrix of the wheel odometer when it is disturbed is as follows:

[0144]

[0145] The estimated covariance matrix of the wheel odometer when it is disturbed is as follows:

[0146]

[0147] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0148] Based on the same inventive concept, the embodiment of the present application also provides a mobile device control device for implementing the mobile device control method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more mobile device control device embodiments provided below can refer to the limitations of the mobile device control method above, and will not be repeated here.

[0149] In an exemplary embodiment, Figure 7 As shown, a mobile device control device is provided, including: a data acquisition module 710, an exception handling module 720, a parameter correction module 730 and a mobile control module 740, wherein:

[0150] A data acquisition module 710 is used to acquire motion state data acquired by at least one data acquisition device;

[0151] The abnormality processing module 720 is used to obtain weight matrix adjustment data when it is determined based on the motion state data that the data acquisition device is in an abnormal state;

[0152] A parameter correction module 730 is used to correct the process parameters of predicting the posture data based on the motion state data according to the weight matrix adjustment data to obtain the target posture data;

[0153] The mobile control module 740 is used to control the movement of the mobile device according to the target posture data and the target position, where the target position is the position to which the mobile device currently needs to move.

[0154] In an exemplary embodiment, the mobile device control device further includes:

[0155] The state monitoring module is used to input the motion state data into the device state monitoring model to obtain the device state monitoring results; and determine whether the data acquisition device is in an abnormal state according to the device state monitoring results.

[0156] In an exemplary embodiment, the mobile device control device further includes:

[0157] The model training module is used to use the motion state data as a state set, the trust weight data as an action set, and design a reward function according to the state judgment result of each data acquisition device; the trust weight data is used to characterize the state of each data acquisition device as a normal state or an abnormal state; the current state set, action set, reward function and next state set are combined to collect an experience sample data set; based on the experience sample data set, the policy network, value network and target network are trained to obtain a trained device state monitoring model.

[0158] In an exemplary embodiment, the device status monitoring result includes trust weight data;

[0159] The status monitoring module is also used to output the status of each data acquisition device based on the trust weight data; when the status of any data acquisition device is abnormal, it is determined that there is a data acquisition device in an abnormal state; when the status of all data acquisition devices is normal, it is determined that there is no data acquisition device in an abnormal state.

[0160] In an exemplary embodiment, the exception handling module 720 is further configured to perform calculations based on the trust weight data and the original weight matrix data to obtain weight matrix adjustment data.

[0161] In an exemplary embodiment, the mobile device control device further includes:

[0162] The posture prediction module is used to fuse the motion state data collected by each data acquisition device through the extended Kalman filter algorithm to predict the posture data.

[0163] In an exemplary embodiment, the parameter correction module 730 is further used to correct the covariance matrix in the fusion process of the extended Kalman filter algorithm according to the weight matrix adjustment data.

[0164] Each module in the above mobile device control device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0165] In an exemplary embodiment, Figure 1 As shown, a mobile device is provided, including a controller 102 and at least one data acquisition device 104 connected to the controller 102 , and the controller 102 is used to control the movement of the mobile device according to the motion state data collected by each data acquisition device 104 .

[0166] The implementation solution provided by the mobile device to solve the problem is similar to the implementation solution recorded in the above-mentioned mobile device control method. Therefore, the specific limitations in the one or more mobile device embodiments provided can refer to the limitations on the mobile device control method above and will not be repeated here.

[0167] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as motion state data, weight matrix adjustment data and target posture data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a mobile device control method is implemented.

[0168] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0169] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0171] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0172] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0173] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for controlling a mobile device, characterized in that: The method comprises: Acquire motion state data collected by at least one data collection device; In the case where it is determined based on the motion state data that the data acquisition device is in an abnormal state, obtaining weight matrix adjustment data; According to the weight matrix adjustment data, a process parameter of predicting the posture data based on the motion state data is corrected to obtain target posture data; The movement of the mobile device is controlled according to the target posture data and the target position, wherein the target position is the position to which the mobile device currently needs to move.

2. The method according to claim 1, characterized in that The method further comprises: Inputting the motion state data into a device state monitoring model to obtain a device state monitoring result; It is determined whether a data acquisition device is in an abnormal state according to the device state monitoring result.

3. The method according to claim 2, characterized in that The step of obtaining the device status monitoring model comprises: The motion state data is used as a state set, the trust weight data is used as an action set, and a reward function is designed according to the state judgment result of each data acquisition device; the trust weight data is used to characterize whether the state of each data acquisition device is a normal state or an abnormal state; Build and initialize the policy network, value network, and target network; The experience sample data set is collected by combining the current state set, action set, reward function and next state set; The strategy network, the value network and the target network are updated based on the experience sample data set to obtain a device status monitoring model.

4. The method according to claim 3, characterized in that The device status monitoring result includes the trust weight data; Judging whether a data acquisition device is in an abnormal state according to the device state monitoring result includes: outputting the status of each data acquisition device based on the trust weight data; In the case that the state of any data acquisition device is an abnormal state, determining that there is a data acquisition device in an abnormal state; When the states of all data acquisition devices are normal, it is determined that no data acquisition device is in an abnormal state.

5. The method according to claim 3, characterized in that: Get weight matrix adjustment data, including: Calculation is performed based on the trust weight data and the original weight matrix data to obtain weight matrix adjustment data.

6. The method according to any one of claims 1 to 5, characterized in that: Predicting the posture data based on the motion state data includes: The motion state data collected by each data acquisition device are fused through the extended Kalman filter algorithm to predict the posture data.

7. The method according to claim 6, characterized in that The step of modifying the process parameters of predicting the posture data based on the motion state data according to the weight matrix adjustment data includes: The covariance matrix of the extended Kalman filter algorithm in the fusion process is corrected according to the weight matrix adjustment data.

8. A mobile device control device, characterized in that: The device comprises: A data acquisition module, used to acquire motion state data acquired by at least one data acquisition device; An abnormality processing module, used for obtaining weight matrix adjustment data when it is determined based on the motion state data that a data acquisition device is in an abnormal state; A parameter correction module, used for correcting the process parameters of the posture data predicted based on the motion state data according to the weight matrix adjustment data to obtain the target posture data; The mobile control module is used to control the movement of the mobile device according to the target posture data and the target position, where the target position is the position to which the mobile device currently needs to move.

9. A mobile device, characterized in that: It comprises a controller and at least one data acquisition device connected to the controller, wherein the controller is used to control the action of the mobile device according to the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images