Mobile device control method, apparatus, mobile device, and readable storage medium

Abstract

This application relates to a mobile device control method, apparatus, electronic device, computer-readable storage medium, and computer program product. When an abnormal state of the data acquisition device is determined based on motion state data, weight matrix adjustment data is obtained. The process parameters for predicting pose data based on motion state data are corrected according to the weight matrix adjustment data to obtain target pose data. Based on the target pose data and the target position, the mobile device's actions are controlled, where the target position is the position the mobile device needs to move to. By monitoring the operating status of each sensor in real time based on sensor data during the mobile device's operation, and adaptively adjusting the weights of abnormal sensors when abnormal sensors are detected, and correcting this adjustment in the motion state data prediction process, the accuracy of the fused positioning results is ensured, thereby guaranteeing precise movement of the mobile device.

Description

Technical Field

[0001] This application relates to the field of mobile robot positioning technology, and in particular to a mobile device control method, apparatus, mobile device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of mobile robot technology, robots with mobility, autonomous localization, and navigation capabilities have received widespread attention and strong support in both industry and academia. In the field of mobile robot localization and navigation, the Extended Kalman Filter (EKF) is one of the most common and widely used multi-sensor fusion algorithms. It is suitable for nonlinear systems and can fuse multiple sensors such as wheel odometry, inertial measurement units (IMUs), visual odometry (VO), and global positioning systems (GPS / GNSS) to improve positioning accuracy.

[0003] However, in the current EKF fusion scheme, if one or more sensors become unstable, such as wheel odometry slippage, visual odometry drift, or GPS signal distortion, it will interfere with the overall fusion positioning results, causing positioning errors of mobile robots in practical applications. Summary of the Invention

[0004] Therefore, it is necessary to provide a mobile device control method, apparatus, electronic device, computer-readable storage medium, and computer program product to address the aforementioned technical problem of mobile robot positioning errors.

[0005] In a first aspect, this application provides a mobile device control method, the method comprising:

[0006] Acquire motion state data collected by at least one data acquisition device;

[0007] If it is determined based on the motion state data that the data acquisition device is in an abnormal state, the weight matrix adjustment data is obtained;

[0008] The process parameters for predicting pose data based on motion state data are corrected according to the weight matrix adjustment data to obtain target pose data.

[0009] Based on the target pose data and target position, the mobile device is controlled to move to the desired location.

[0010] In one embodiment, the method further includes:

[0011] The motion state data is input into the device state monitoring model to obtain the device state monitoring results;

[0012] The device status monitoring results are used to determine whether the data acquisition device is in an abnormal state.

[0013] In one embodiment, the step of obtaining the device condition monitoring model includes:

[0014] Motion state data is used as a state set, trust weight data is used as an action set, and a reward function is designed according to the state judgment results of each data acquisition device; the trust weight data is used to characterize whether the state of each data acquisition device is normal or abnormal.

[0015] Construct and initialize the policy network, value network, and target network;

[0016] An empirical sample dataset is obtained by combining the current set of states, the set of actions, the reward function, and the set of the next state.

[0017] The policy network, the value network, and the target network are updated based on the empirical sample dataset to obtain the device status monitoring model.

[0018] In one embodiment, the device status monitoring result includes the trust weight data;

[0019] Determining whether a data acquisition device is in an abnormal state based on the device status monitoring results includes:

[0020] The status of each data acquisition device is output based on the trust weight data.

[0021] If any data acquisition device is in an abnormal state, it is determined that there is a data acquisition device in an abnormal state.

[0022] If all data acquisition devices are in normal condition, it is determined that no data acquisition device is in an abnormal state.

[0023] In one embodiment, obtaining the weight matrix adjustment data includes:

[0024] The weight matrix adjustment data is obtained by calculating the weight weight data and the original weight matrix data.

[0025] In one embodiment, predicting pose data based on the motion state data includes:

[0026] The motion state data collected by each data acquisition device is fused using the extended Kalman filter algorithm to predict the pose data.

[0027] In one embodiment, the process of adjusting the parameters of the pose prediction data based on the motion state data according to the weight matrix includes:

[0028] The covariance matrix of the extended Kalman filter algorithm is corrected during the fusion process based on the weight matrix adjustment data.

[0029] Secondly, this application provides a mobile device control apparatus, the apparatus comprising:

[0030] The data acquisition module is used to acquire motion state data collected by at least one data acquisition device;

[0031] An anomaly handling module is used to obtain weight matrix adjustment data when it is determined, based on the motion state data, that the data acquisition device is in an abnormal state.

[0032] The parameter correction module is used to correct the process parameters of the pose prediction data based on the motion state data according to the weight matrix adjustment data to obtain the target pose data.

[0033] The motion control module is used to control the movement of the mobile device based on the target pose data and the target position, wherein the target position is the position that the mobile device needs to move to.

[0034] Thirdly, this application provides a mobile device, including a controller and at least one data acquisition device connected to the controller, the controller being used to control the operation of the mobile device according to the method described in any of the above embodiments.

[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0036] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0037] The aforementioned mobile device control method, apparatus, electronic device, computer-readable storage medium, and computer program product, after acquiring motion state data collected by at least one data acquisition device, and upon determining that a data acquisition device is in an abnormal state based on the motion state data, obtain weight matrix adjustment data. Based on the weight matrix adjustment data, the process parameters for predicting pose data based on motion state data are corrected to obtain target pose data. Based on the target pose data and the target position, the mobile device is controlled to move to the desired location. By monitoring the operating status of each sensor in real time based on sensor data during the mobile device's operation, and adaptively adjusting the weights of abnormal sensors when abnormal sensors are detected, and correcting this adjustment in the process of predicting pose data based on motion state data, the accuracy of the fused positioning results is ensured, thereby guaranteeing precise movement of the mobile device. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is an application environment diagram of a mobile device control method in one embodiment;

[0040] Figure 2 This is a flowchart illustrating a mobile device control method in one embodiment;

[0041] Figure 3 This is a flowchart illustrating a mobile device control method in another embodiment;

[0042] Figure 4 This is a flowchart illustrating the steps for acquiring the device status monitoring model in one embodiment;

[0043] Figure 5 This is a flowchart illustrating the mobile device control method in yet another embodiment;

[0044] Figure 6 Here is a flowchart illustrating a mobile device control method in one embodiment;

[0045] Figure 7 This is a structural block diagram of a mobile device control device in one embodiment;

[0046] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The mobile device control method provided in this application embodiment can be applied to, for example, Figure 1 The illustrated mobile device includes a controller 102 and at least one data acquisition device 104 connected to the controller 102. The controller controls the movement of the mobile device based on motion state data acquired by each data acquisition device 104. It is understood that the mobile device addressed in this application is a device capable of automatically controlling positional movement, such as a mobile robot or an autonomous vehicle. The data storage system can store the data that the controller 102 needs to process. The data storage system can be integrated into the controller 102 or located in the cloud or on other network servers.

[0049] Specifically, the controller 102 acquires motion state data collected by at least one data acquisition device 104. If it is determined that the data acquisition device 104 is in an abnormal state based on the motion state data, it obtains weight matrix adjustment data. Based on the weight matrix adjustment data, it corrects the process parameters of the pose data prediction based on the motion state data to obtain the target pose data. Finally, based on the target pose data and the target position, it controls the mobile device to move. The target position is the position that the mobile device needs to move to.

[0050] The controller 102 can be a control chip or control circuit board installed inside a mobile device, or it can be an external control system based on wireless communication. The external control system can be implemented through devices such as terminals or servers. Terminals can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0051] In one exemplary embodiment, such as Figure 2 As shown, a mobile device control method is provided, which is applied to... Figure 1Taking controller 102 as an example, the explanation includes the following steps 201 to 208. Wherein:

[0052] Step 201: Obtain motion state data collected by at least one data acquisition device.

[0053] The data acquisition device measures the current motion information of the mobile device and the state information of the environment affecting its movement, assisting the mobile device in achieving high-precision positioning and navigation. It's understandable that the type of data acquisition device equipped on a mobile device varies depending on the scenario. For example, indoor mobile devices commonly use a combination of inertial measurement unit (IMU) and encoder for positioning and navigation, while outdoor devices use Global Positioning System (GPS / GNSS). Other sensors, such as wheel odometry (ODOM) and visual odometry (VO), can also serve as data acquisition devices for mobile devices to improve their positioning accuracy.

[0054] Specifically, depending on the data acquisition device mounted on the mobile device, the acquired motion state data will also differ. This data can be used to characterize different current motion states of the mobile device, as well as different states of the environment in which the mobile device is currently located. For example, if the data acquisition device is an inertial measurement unit (IMU), the acquired motion state data can be the acceleration and angular velocity information of the mobile device. If the data acquisition device is a global positioning system (GPS / GNSS), the acquired motion state data can be the location information of the mobile device. If the data acquisition device is a wheeled odometer (ODM) or visual odometer (VO), the acquired motion state data can be the displacement information of the mobile device.

[0055] Furthermore, after the mobile device starts running, the controller can continuously acquire motion state data collected by at least one data acquisition device on the mobile device, as the data basis for positioning and navigation of the mobile device. It can be understood that, based on the different working principles of the data acquisition devices, the acquired motion state data can be data at the current point in time (e.g., acceleration, angular velocity, and position information), or data corresponding to the current time period (e.g., displacement information).

[0056] Step 204: If it is determined from the motion state data that the data acquisition device is in an abnormal state, obtain the weight matrix adjustment data.

[0057] It is understandable that for current mobile devices, after initiating positioning and navigation assisted by their onboard data acquisition devices, the motion state data collected by these devices will be used for fusion calculations to predict pose data. If the data acquisition devices are unstable, such as wheel odometer slippage, visual odometer drift, or GPS signal distortion, the results will still be included in the fusion calculation to obtain pose data, interfering with the overall fusion positioning result and causing positioning errors in practical applications.

[0058] For example, after detecting a positioning error in the mobile device based on the predicted pose data, the data acquisition device in an abnormal state can be replaced to ensure the normal operation of the mobile device. For instance, the data acquisition device can be replaced with a more accurate and robust one. However, it is understood that the above solution requires stopping the operation of the mobile device and replacing its onboard data acquisition device, which not only delays the normal operation of the mobile device but may also incur greater costs.

[0059] Therefore, in the embodiments of this application, it can first be determined whether there is a data acquisition device in an abnormal state based on motion state data, and if it is determined that there is a data acquisition device in an abnormal state based on motion state data, weight matrix adjustment data can be obtained to correct the process parameters of the pose data obtained by fusion calculation. In this way, without affecting the operation of the mobile device, the motion state data collected by the data acquisition device in an abnormal state can be avoided from interfering with the overall fusion positioning result, thus ensuring the accurate movement of the mobile device.

[0060] Specifically, determining whether a data acquisition device is in an abnormal state based on motion state data can be achieved by pre-setting a range of motion state data for the data acquisition device under normal conditions, and then judging whether the collected motion state data exceeds this range. Alternatively, it can be based on historical motion state data collected by the data acquisition device under both normal and abnormal conditions, training a state monitoring model using machine learning algorithms, and then using this trained model to determine whether the data acquisition device is in an abnormal state. For example, a reinforcement learning algorithm can be used to train the state monitoring model.

[0061] Step 206: Adjust the process parameters of the pose prediction data based on motion state data according to the weight matrix adjustment data to obtain the target pose data.

[0062] Among them, the weight matrix adjustment data is used to characterize which data acquisition device is in an abnormal state, and how to reduce the impact of motion state data acquired by the data acquisition device in an abnormal state on the overall fusion positioning result.

[0063] Specifically, the method of predicting pose data based on motion state data can be to first predict the pose data of the next movement state of the mobile device based on the collected motion state data, and then perform fusion calculation based on the predicted pose data obtained from the motion state data to obtain the target pose data.

[0064] The process of predicting the pose data of the next moving state of the mobile device based on the collected motion state data can be as follows: First, the pose of the mobile device can be estimated based on the motion state data to obtain the estimated pose data of the current state of the mobile device. Then, based on the estimated pose data corresponding to each motion state data, the next moving state of the mobile device can be predicted through its corresponding dynamic model to obtain multiple predicted pose data.

[0065] Secondly, the method of fusing and calculating the target pose data based on the various predicted pose data can be achieved through various types of multi-sensor fusion algorithms, such as weighted averaging, Kalman filtering, and DS evidence reasoning, or artificial intelligence algorithms such as fuzzy logic reasoning and neural network algorithms. In this embodiment, the extended Kalman filter (EKF) algorithm is used to fuse and calculate the pose data predicted from the various motion state data to obtain the target pose data.

[0066] Furthermore, the multi-sensor fusion algorithm is used to fuse and calculate the predicted pose data. Essentially, this requires designing a fusion confidence level for each motion state data point (i.e., each data acquisition device) during the algorithm process, and then combining this fusion confidence level with the fusion calculation of the predicted pose data. Consequently, if the motion state data indicates that a data acquisition device is in an abnormal state, the fusion confidence level of the abnormal data acquisition device needs to be adjusted to reduce its participation in the fusion calculation and minimize its impact on the overall fusion positioning result.

[0067] Furthermore, the fusion confidence needs to be corrected based on the weight matrix to obtain target pose data that is not affected by the abnormal state of the data acquisition device.

[0068] Step 208: Based on the target pose data and target position, control the mobile device's actions. The target position is the position that the mobile device needs to move to.

[0069] Specifically, the target location is the position that the mobile device is currently preparing to move to. After obtaining the target pose data that is not affected by abnormal conditions from the data acquisition device, the target location can be used as the endpoint, and the mobile device can be controlled to move until it reaches the target location.

[0070] For example, the method of controlling the mobile device's actions based on the target pose data with the target location as the endpoint can be based on acquiring LiDAR point cloud data, and then using the LiDAR point cloud data and the target pose data to construct a high-precision point cloud map for the mobile device through the SLAM (Simultaneous Localization and Mapping) algorithm. Finally, based on the relative position of the target location on the high-precision point cloud map, the mobile device is controlled to move to the target location.

[0071] The aforementioned mobile device control method monitors the operating status of each sensor in real time based on sensor data during the operation of the mobile device. When an abnormal sensor is detected, it adaptively adjusts the weight of the abnormal sensor and corrects it in the process of predicting pose data based on motion state data, so as to ensure the accuracy of the fused positioning results and thus ensure the precise movement of the mobile device.

[0072] In one exemplary embodiment, such as Figure 3 As shown, the above-mentioned mobile device control method further includes steps 202 to 203, wherein:

[0073] Step 202: Input the motion state data into the device state monitoring model to obtain the device state monitoring results.

[0074] The device status monitoring model is a predictive model with good recognition performance, obtained by inputting a large amount of historical motion state data collected by the data acquisition device under normal and abnormal conditions into a reinforcement learning network model for training, tuning, and verification. The reinforcement learning network model that can be used is not unique, and includes, but is not limited to, Deep Q-Networks (DQN), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), and Deterministic Policy Gradient (DPG).

[0075] Specifically, after acquiring motion state data from at least one data acquisition device, the various motion state data can be input into the device status monitoring model to obtain device status monitoring results. It can be understood that the device status monitoring results can be used to characterize whether the data acquisition device is in an abnormal state.

[0076] Furthermore, among the aforementioned reinforcement learning algorithms, DQN was the first method to combine deep learning with reinforcement learning. By using a large neural network as a function approximator, it can directly learn control policies from high-dimensional video pixels. However, because DQN cannot output the behavior value function for each action, it cannot solve the problem of continuous action space, such as mobile device control. Deterministic Policy Gradient (DPG), by representing the policy as a policy function and mapping the state to a deterministic action, can solve the problem of continuous action space, but it cannot directly learn the policy from high-dimensional input. The Deep Deterministic Policy Gradient (DDPG) algorithm combines the two, incorporating the successful experience of DQN into the DPG algorithm, achieving end-to-end learning directly from the raw data.

[0077] For example, this application can train a device state monitoring model using a deep deterministic policy gradient algorithm. The deep deterministic policy gradient algorithm combines deep learning and deterministic policy gradient methods, and is particularly suitable for reinforcement learning tasks in continuous action spaces.

[0078] Specifically, a deterministic policy refers to a policy that, given a state, directly outputs a specific action rather than a probability distribution. Formally, a deterministic policy can be represented as μ(s), where s is the state and μ is the policy function. Policy gradient refers to the method of optimizing the policy through gradient ascent. For a deterministic policy, the goal is to maximize the expected reward, which can be expressed as:

[0079]

[0080] Where θ is the policy parameter, τ is the trajectory, and R(τ) is the total reward of the trajectory.

[0081] Furthermore, the deterministic policy gradient can be calculated using the following formula:

[0082]

[0083] in, For behavioral strategies, For the generated state distribution, This is the action value function.

[0084] Taking the device state monitoring model trained by the deep deterministic policy gradient algorithm as an example, in an exemplary embodiment, such as... Figure 4 As shown, the step of obtaining the device condition monitoring model in step 202 includes the following steps 302 to 308, wherein:

[0085] Step 302: The motion state data is used as a state set, the trust weight data is used as an action set, and a reward function is designed according to the state judgment results of each data acquisition device; the trust weight data is used to characterize whether the state of each data acquisition device is normal or abnormal.

[0086] Specifically, the motion state data collected by each data acquisition device is represented by states to form a state set s. t The trust weight data corresponding to each data acquisition device is used as an action representation to form an action set a. t It can be understood that the trust weight data corresponding to the data acquisition device includes a binary decision on whether to trust the data acquisition device, that is, whether the motion state data acquired by the data acquisition device can participate in the fusion calculation of the target pose data.

[0087] Trust weight data can be used to characterize whether each data acquisition device is in a normal or abnormal state. Furthermore, during model training, data annotation can establish a one-to-one correspondence between motion state data and trust weight data, indicating whether each motion state data point was acquired by the data acquisition device in a normal or abnormal state. Then, during model prediction, trust weight data can be obtained based on the real-time acquired motion state data to determine whether the data acquisition device is currently in a normal or abnormal state.

[0088] Furthermore, a reward function r can be designed based on the status judgment results of each data acquisition device. t In other words, if the trust weight data output by the model based on the motion state data correctly determines the state of each data acquisition device, a positive reward is given; otherwise, a negative reward is given.

[0089] Step 304: Construct and initialize the policy network, value network, and target network.

[0090] Specifically, a policy network (Actor network) is constructed by representing the policy as a policy function μ(s,θ), mapping the state s to a deterministic action. Once the policy is deterministic, it is then... A value network (Critic network) is obtained by constructing the action value function.

[0091] Furthermore, we approximate the value network using neural networks. During the training of the policy network μ(s,θ), end-to-end learning can be achieved directly from the raw data. However, if the same neural network is used to represent both the final target network and the currently updated network, the learning process becomes unstable because the same network parameters are frequently used for gradient updates and gradient calculations. Therefore, a value network-based approach can be adopted. A copy of the policy network μ(s,θ) is performed to obtain the target network. and .

[0092] Furthermore, after obtaining the policy network, value network, and target network, they need to be initialized separately.

[0093] Step 306: Collect an experience sample dataset by combining the current state set, action set, reward function, and next state set.

[0094] This is understandable because reinforcement learning data exhibits the Markov property, which violates the assumption that training neural networks requires samples to be independent and identically distributed. Consequently, the training process is highly unstable when using neural networks for reinforcement learning. To ensure learning effectiveness, it's necessary to break the correlation between training data. Therefore, experience replay can be used to address this issue. Specifically, experience replay refers to combining experiences (states, actions, rewards, etc.) gained from model interactions and storing them in an "experience replay buffer." These buffers are then randomly sampled during model training for model updates. This design not only improves data utilization efficiency and breaks the temporal correlation between data points, but also leads to a more comprehensive and stable model.

[0095] Specifically, actions can be performed in the initialized network environment based on the current set of states to collect a state transition sequence (s) that combines the current set of states, the set of actions, the reward function, and the next set of states. t ,a t ,r t ,s t+1 ), and store them to form an empirical sample dataset.

[0096] For example, in order to ensure the effective exploration of the algorithm during the learning process, noise, such as Gaussian noise or the Ornstein-Uhlenbeck process, can be added when selecting actions under a deterministic policy.

[0097] Step 308: Update the policy network, value network, and target network based on the empirical sample dataset to obtain the device status monitoring model.

[0098] Specifically, samples can be randomly drawn in batches from the empirical sample dataset (s t ,a t ,r t ,s t+1 This can be done by updating the value network. Specifically, the mean squared error loss function can be used to update the network parameters in the value network. Specifically, it is expressed as the following formula:

[0099]

[0100] in, To obtain the target value through the target network, , This is the discount factor.

[0101] Furthermore, the parameters of the policy network are updated using the policy gradient formula. The details are as follows:

[0102]

[0103] Furthermore, to improve the stability of learning, a soft update method is used to gradually update the target network parameters, such as by using a moving average.

[0104]

[0105]

[0106] in, It is a positive number, usually set between 0.001 and 0.01.

[0107] It is understandable that the trained device state monitoring model can be obtained by repeating the above network update process until the policy converges or the maximum number of iterations is reached.

[0108] Step 203: Determine whether the data acquisition device is in an abnormal state based on the device status monitoring results.

[0109] Specifically, the device status monitoring results are output by the device status monitoring model and can be used to characterize whether any data acquisition devices are in an abnormal state. The device status monitoring results may include the status monitoring results for each data acquisition device, thus determining whether each data acquisition device is in an abnormal state. It can be understood that if even one data acquisition device is in an abnormal state, it can be determined that all data acquisition devices are in an abnormal state.

[0110] In one exemplary embodiment, device status monitoring results include trust weight data. For example, as... Figure 5 As shown, step 203 includes steps 402 to 406, wherein:

[0111] Step 402: Output the status of each data acquisition device based on the trust weight data.

[0112] Step 404: If any data acquisition device is in an abnormal state, determine that there is a data acquisition device in an abnormal state.

[0113] Step 406: If all data acquisition devices are in normal status, determine that no data acquisition device is in an abnormal state.

[0114] Specifically, the trust weight data represents a binary decision on whether to trust the data acquisition device. For example, the closer the decision value is to 1, the closer the data acquisition device is to a normal state; conversely, the closer it is to 0, the closer it is to an abnormal state. Furthermore, trust threshold values ​​corresponding to normal and abnormal states can be preset, and these threshold values ​​are then compared with the trust weight data output by the data acquisition devices to determine the state of each device. It can be understood that the output state of the data acquisition device can be either normal or abnormal.

[0115] Furthermore, after obtaining the status of all data acquisition devices, it can be determined whether the process parameters for predicting pose data based on motion state data need to be corrected based on whether any data acquisition device is in an abnormal state. It can be understood that if all data acquisition devices are in a normal state, the overall fusion positioning result will not be interfered with, and the prediction of pose data based on motion state data can be executed normally without any adjustment to the process parameters. However, if any data acquisition device is in an abnormal state (this could include one or multiple abnormal data acquisition devices), it is necessary to determine whether a data acquisition device is in an abnormal state and correct the process parameters to prevent the motion state data acquired by the abnormal data acquisition device from interfering with the overall fusion positioning result and ensuring the accurate movement of the mobile device.

[0116] For example, this application obtains the weight matrix adjustment data by directly using the trust weight data output by the model based on motion state data as a process parameter in the calculation. It can be understood that the weight matrix adjustment data is essentially used to correct the fusion confidence level of the fusion calculation based on predicted pose data, while the trust weight data represents a binary decision on whether to trust the data acquisition device. In fact, it is also the weight reset confidence level set for the data acquisition device. Therefore, it can directly replace the process parameters in the fusion calculation, thus reducing its participation in the fusion calculation and minimizing its impact on the overall fusion positioning result.

[0117] In an exemplary embodiment, obtaining the weight matrix adjustment data in step 204 includes: calculating the weight matrix adjustment data based on the trust weight data and the original weight matrix data.

[0118] Specifically, it is also possible to combine the trusted weight data with the original weight matrix data to obtain weight matrix adjustment data, which is then used to correct the fusion confidence score calculated based on the predicted pose data. This design can utilize the fusion confidence score of the data acquisition device in a normal state within the original weight matrix data, thus minimizing the impact on the overall fusion positioning result.

[0119] In an exemplary embodiment, step 206, predicting pose data based on motion state data, includes: fusing the motion state data acquired by each data acquisition device using an extended Kalman filter algorithm to predict pose data.

[0120] Specifically, the extended Kalman filter algorithm can achieve data fusion through the following steps:

[0121] 1. Predict the state at the next moment based on the optimal estimated state and state transition matrix at the current moment;

[0122] 2. Predict the predicted covariance matrix for the next time step based on the optimal estimated covariance matrix and process noise matrix at the current time step;

[0123] 3. Calculate the Kalman gain for the next time step based on the prediction covariance matrix of the current time step and the measurement covariance matrix of the next time step.

[0124] 4. Based on the predicted state at the current time, the measured state at the next time, the Kalman gain, and the observation matrix, calculate the optimal estimated state at the next time.

[0125] 5. Based on the predicted covariance matrix and Kalman gain of the current time step, predict the optimal estimated covariance matrix of the next time step.

[0126] By repeating the above iterative steps, the motion state data collected by each data acquisition device is fused and processed to obtain the optimal estimated state, that is, the predicted target pose data.

[0127] Understandably, regarding the aforementioned problem of positioning errors caused by unstable data acquisition devices, the inventors discovered that an adaptive fusion EKF algorithm can be used to optimize the extended Kalman filter algorithm. Specifically, this can be achieved by changing the parameters of the covariance matrix in the original algorithm, thereby adjusting the weights of each data acquisition device to improve the positioning error problem. However, the adaptive fusion EKF algorithm requires additional computational steps to estimate and adjust the noise covariance matrices R and Q, which significantly increases the computational load. This results in severe processing delays, especially when applied to actual mobile devices.

[0128] In an exemplary embodiment, step 206, which involves adjusting the data based on the weight matrix to correct the process parameters for predicting pose data based on motion state data, includes: adjusting the covariance matrix during the fusion process of the extended Kalman filter algorithm based on the weight matrix.

[0129] Specifically, based on the above findings, this implementation proposes to employ a device state monitoring model trained with a deterministic strategy gradient to perform real-time monitoring of each data acquisition device to determine whether it is in an abnormal state. Furthermore, if any data acquisition device is in an abnormal state, weight matrix adjustment data will be obtained to correct the covariance matrix fused by the extended Kalman filter algorithm.

[0130] It is understandable that the weight matrix adjustment data can be calculated based on the trusted weight data and the original weight matrix data, and used to adjust the covariance matrix, or it can be data directly used to replace the covariance matrix.

[0131] In this embodiment, the device status monitoring model obtained through training does not directly participate in data fusion. Instead, after an anomaly occurs in the data acquisition device, the covariance matrix used for fusion by the EKF algorithm is changed to adaptively adjust the fusion confidence of each data acquisition device during the EKF fusion process, thereby ensuring the accuracy of the fusion positioning results.

[0132] In one specific embodiment, a mobile device is used as a mobile robot, and the data acquisition device uses three types of sensors: wheeled odometer (odom), inertial measurement unit (IMU), and global positioning system (GPS / GNSS). Taking the case where the wheeled odometer is interfered with (such as slipping) during the operation of the mobile robot as an example, the mobile device control method provided in this application is explained.

[0133] An inertial measurement unit (IMU) is a device used to measure the motion of a target object in three-dimensional space. Specifically, based on Newton's laws of motion, it measures the acceleration of the target object in an inertial reference frame, integrates this acceleration over time, and transforms it into a navigation coordinate system. This allows for the acquisition of information such as velocity, yaw angle, and position in the navigation coordinate system, thus achieving precise positioning. It typically includes an accelerometer to measure the acceleration of the target object in various directions, thereby calculating the target object's velocity and displacement. It may also include a gyroscope to measure the angular velocity of the target object around various axes to determine its rotational motion.

[0134] The Global Positioning System (GPS / GNSS) is a high-precision radio navigation positioning system based on artificial Earth satellites. It primarily relies on radio signals transmitted by satellites for positioning and navigation. These radio signals include information about the satellite's position, velocity, and time. After receiving the radio signals, mobile devices determine their own position, velocity, and time by measuring the signal's transmission time.

[0135] An odometry device (ODM) is a device that measures the travel distance of a robot or vehicle by monitoring the rotational speed of its wheels. By monitoring the rotational angle and distance traveled by each wheel, the device's pose changes during movement can be deduced. Specifically, an optical encoder is typically used to measure wheel speed. When the motor rotates, it drives a code disk (magnetic or optical) connected to the wheel to rotate. The encoder receives pulse signals, and the number of pulses multiplied by a coefficient gives the number of wheel rotations, thus allowing the calculation of the device's displacement.

[0136] Specifically, a large amount of historical motion state data can be collected under normal and abnormal conditions using three types of sensors: odometry, inertial measurement unit (IMU), and global positioning system (GPS / GNSS). A state set is then formed based on this collected historical motion state data. ,in, Represents location information. Representative angle information, Represents linear velocity information. Represents angular velocity information. This represents acceleration information. The action set is obtained by labeling the historical states of the data acquisition device.

[0137] Furthermore, the aforementioned set of states and sets of actions are combined to form a training set and a test set, respectively. The device state monitoring model is then trained through steps 302 to 308. The trained model is then deployed to the mobile robot. The specific training process for the device state monitoring model can be referenced from [reference needed]. Figure 6 As shown, after deploying the device status monitoring model, the mobile robot can monitor the status of each sensor in real time and determine whether it is in an abnormal state. If abnormal data from one or more sensors is detected, the data is adjusted according to the weight matrix obtained from the monitoring, and the covariance matrix of the EKF algorithm is adaptively and dynamically adjusted.

[0138] The following are two examples of covariance matrices obtained from model adjustments to illustrate the adaptive changes in covariance when a wheel odometer is disturbed.

[0139] The noise covariance matrices of each sensor when it is in normal operating condition are as follows:

[0140]

[0141] The estimated covariance matrices of each sensor when it is in normal condition are as follows:

[0142]

[0143] The noise covariance matrix of a wheeled odometer when subjected to interference is as follows:

[0144]

[0145] The estimated covariance matrix of the wheel odometer when subjected to disturbance is as follows:

[0146]

[0147] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0148] Based on the same inventive concept, this application also provides a mobile device control apparatus for implementing the aforementioned mobile device control method. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the mobile device control apparatus provided below can be found in the limitations of the mobile device control method described above, and will not be repeated here.

[0149] In one exemplary embodiment, such as Figure 7 As shown, a mobile device control device is provided, including: a data acquisition module 710, an exception handling module 720, a parameter correction module 730, and a motion control module 740, wherein:

[0150] The data acquisition module 710 is used to acquire motion state data collected by at least one data acquisition device;

[0151] The anomaly handling module 720 is used to obtain weight matrix adjustment data when it is determined from motion state data that the data acquisition device is in an abnormal state.

[0152] The parameter correction module 730 is used to correct the process parameters of the pose prediction data based on motion state data according to the weight matrix adjustment data to obtain the target pose data.

[0153] The motion control module 740 is used to control the movement of the mobile device based on the target pose data and the target position, where the target position is the position that the mobile device needs to move to.

[0154] In one exemplary embodiment, the mobile device control device further includes:

[0155] The status monitoring module is used to input motion status data into the device status monitoring model to obtain device status monitoring results; and to determine whether the data acquisition device is in an abnormal state based on the device status monitoring results.

[0156] In one exemplary embodiment, the mobile device control device further includes:

[0157] The model training module is used to take motion state data as a state set, trust weight data as an action set, and design a reward function based on the state judgment results of each data acquisition device. The trust weight data is used to characterize whether the state of each data acquisition device is normal or abnormal. An empirical sample dataset is collected by combining the current state set, action set, reward function, and the next state set. The policy network, value network, and target network are trained based on the empirical sample dataset to obtain the trained device state monitoring model.

[0158] In one exemplary embodiment, the device status monitoring results include trust weight data;

[0159] The status monitoring module is also used to output the status of each data acquisition device based on trust weight data; if the status of any data acquisition device is abnormal, it determines that there is a data acquisition device in an abnormal state; if the status of all data acquisition devices is normal, it determines that there is no data acquisition device in an abnormal state.

[0160] In an exemplary embodiment, the exception handling module 720 is further configured to calculate, based on the trust weight data and the original weight matrix data, to obtain weight matrix adjustment data.

[0161] In one exemplary embodiment, the mobile device control device further includes:

[0162] The pose prediction module is used to fuse motion state data collected by various data acquisition devices using an extended Kalman filter algorithm to predict pose data.

[0163] In an exemplary embodiment, the parameter correction module 730 is further configured to correct the covariance matrix in the fusion process of the extended Kalman filter algorithm based on the weight matrix.

[0164] Each module in the aforementioned mobile device control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0165] In one exemplary embodiment, such as Figure 1 As shown, a mobile device is provided, including a controller 102 and at least one data acquisition device 104 connected to the controller 102. The controller 102 is used to control the operation of the mobile device according to the motion state data collected by each data acquisition device 104.

[0166] The solution provided by this mobile device is similar to the solution described in the above-mentioned mobile device control method. Therefore, the specific limitations of the one or more mobile device embodiments provided can be found in the limitations of the mobile device control method above, and will not be repeated here.

[0167] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores motion state data, weight matrix adjustment data, and target pose data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a mobile device control method.

[0168] Those skilled in the art will understand that Figure 8The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0169] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0172] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0173] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A mobile device control method, characterized in that, The method includes: Acquire motion state data collected by at least one data acquisition device; The motion state data is input into the device state monitoring model to obtain the device state monitoring result. The steps for obtaining the device state monitoring model include: using the motion state data as a state set, the trust weight data as an action set, and designing a reward function based on the state judgment results of each data acquisition device; the trust weight data is used to characterize whether the state of each data acquisition device is normal or abnormal; constructing and initializing a policy network, a value network, and a target network; collecting an experience sample dataset by combining the current state set, action set, reward function, and the next state set; updating the policy network, the value network, and the target network based on the experience sample dataset to obtain the device state monitoring model; the device state monitoring result includes the trust weight data. If it is determined from the device status monitoring results that a data acquisition device is in an abnormal state, the weight matrix adjustment data is obtained by calculating based on the trust weight data and the original weight matrix data. The process parameters for predicting pose data based on motion state data are corrected according to the weight matrix adjustment data to obtain target pose data. Based on the target pose data and target position, the mobile device is controlled to move to the desired location.

2. The method according to claim 1, characterized in that, The data acquisition device includes a wheeled odometer, a visual odometer, an inertial measurement unit, and a global positioning system. The motion state data includes the acceleration and angular velocity information, position information, and displacement information of the mobile device.

3. The method according to claim 1, characterized in that, Determining whether a data acquisition device is in an abnormal state based on the device status monitoring results includes: The status of each data acquisition device is output based on the trust weight data. If any data acquisition device is in an abnormal state, it is determined that there is a data acquisition device in an abnormal state. If all data acquisition devices are in normal condition, it is determined that no data acquisition device is in an abnormal state.

4. The method according to claim 3, characterized in that, The method further includes: Preset trust threshold values ​​for normal and abnormal states respectively; The trust value threshold is compared with the trust weight data corresponding to the data acquisition device to determine the status of each data acquisition device.

5. The method according to any one of claims 1 to 4, characterized in that, Predicting pose data based on the motion state data includes: The motion state data collected by each data acquisition device is fused using the extended Kalman filter algorithm to predict the pose data.

6. The method according to claim 5, characterized in that, The process of adjusting the parameters of the pose prediction data based on the motion state data according to the weight matrix includes: The covariance matrix of the extended Kalman filter algorithm is corrected during the fusion process based on the weight matrix adjustment data.

7. A mobile device control device, characterized in that, The device includes: The data acquisition module is used to acquire motion state data collected by at least one data acquisition device; A state monitoring module is used to input the motion state data into a device state monitoring model to obtain device state monitoring results. The steps for obtaining the device state monitoring model include: using the motion state data as a state set, using trust weight data as an action set, and designing a reward function based on the state judgment results of each data acquisition device; the trust weight data is used to characterize whether the state of each data acquisition device is normal or abnormal; constructing and initializing a policy network, a value network, and a target network; collecting an experience sample dataset by combining the current state set, action set, reward function, and the next state set; updating the policy network, the value network, and the target network based on the experience sample dataset to obtain the device state monitoring model; the device state monitoring results include the trust weight data. An anomaly handling module is used to calculate, based on the trust weight data and the original weight matrix data, to obtain weight matrix adjustment data when it is determined that a data acquisition device is in an abnormal state based on the device status monitoring results. The parameter correction module is used to correct the process parameters of the pose prediction data based on the motion state data according to the weight matrix adjustment data to obtain the target pose data. The motion control module is used to control the movement of the mobile device based on the target pose data and the target position, wherein the target position is the position that the mobile device needs to move to.

8. A mobile device, characterized in that, The device includes a controller and at least one data acquisition device connected to the controller, the controller being used to control the operation of a mobile device according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images