Unmanned vehicle control method, device and equipment and readable storage medium

Abstract

The invention relates to an unmanned vehicle control method, device and equipment and a readable storage medium. The method comprises the steps that in the process that the unmanned vehicle executes grabbing operation for a target object, object image data of the target object collected by information collection equipment deployed on the unmanned vehicle in a current time window and environment data of the environment where the target object is located are obtained; processing the obtained object image data and environment data to obtain a current state vector; selecting a target action from the candidate actions according to the value function and the current state vector; and controlling the unmanned vehicle to grab the target object based on the object coordinate of the target object and the current position coordinate of the unmanned vehicle under the condition that the target action is the grabbing action. By adopting the method, the operation reliability of the unmanned vehicle can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an unmanned vehicle control method, apparatus, device, and readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and automation technologies, unmanned operating systems are playing an increasingly important role in industries such as manufacturing, logistics, and warehousing. In particular, intelligent grasping and control systems based on unmanned vehicles have become a research hotspot in material handling, cargo grasping, and transportation tasks.

[0003] Existing autonomous vehicle control methods typically select a preset driving path and action sequence based on the autonomous vehicle's state and actions, generate a grasping operation strategy, and control the autonomous vehicle to perform the grasping operation according to the grasping operation strategy.

[0004] However, existing unmanned vehicle control methods rely on preset path planning and fixed action sequences, which are difficult to adapt to the real-time decision-making requirements in complex dynamic environments, thus reducing the reliability of unmanned vehicle operations. Summary of the Invention

[0005] Therefore, it is necessary to provide an unmanned vehicle control method, device, equipment, and readable storage medium that can improve the reliability of unmanned vehicle operation in response to the above-mentioned technical problems.

[0006] Firstly, this application provides an unmanned vehicle control method, including:

[0007] During the process of the unmanned vehicle performing the grasping operation of the target object, the information collection device deployed on the unmanned vehicle acquires the object image data of the target object and the environmental data of the environment in which the target object is located within the current time window.

[0008] The acquired object image data and environmental data are processed to obtain the current state vector;

[0009] Based on the value function and the current state vector, a target action is selected from each candidate action; whereby the value function describes the contribution of each candidate action performed by the autonomous vehicle in each candidate state to the operation process of the autonomous vehicle; the value function is updated according to the actual action performance of the autonomous vehicle in the previous time window;

[0010] When the target action is a grasping action, the autonomous vehicle is controlled to grasp the target object based on the object coordinates and the current position coordinates of the autonomous vehicle.

[0011] In some alternative implementations, the method further includes:

[0012] The current execution reward value is determined based on the result of the autonomous vehicle's action on the target.

[0013] The value function is updated based on the current execution reward value and the action value between the next state vector and each candidate action in the value function; where the next state vector is predicted based on the current predicted state vector.

[0014] In some alternative implementations, the value function is updated based on the current execution reward value and the action value between the next state vector and each candidate action in the value function, including:

[0015] The value function is updated for the first time based on the current execution reward value and the action value between the next state vector and each candidate action in the value function;

[0016] Based on the preset adjustment parameters, the current execution reward value, and the action value between the next state vector and each candidate action in the value function after the first update, the value function after the first update is updated a second time.

[0017] Each candidate action pair is sampled, and the value function after the second update is updated a third time based on the current execution reward value, the sampling result, and the value function after the second update. Each candidate action pair contains a candidate state vector and a candidate action associated with the candidate state vector.

[0018] In some optional implementations, the value function after the second update is updated a third time based on the current execution reward value, the sampling processing result, and the value function after the second update, including:

[0019] Based on each sampled action pair in the sampling process results and the value function after the second update, determine the action value of each sampled action pair;

[0020] Based on the action value of each sampled action pair and the current execution reward value, the value function after the second update is updated a third time.

[0021] In some alternative implementations, the autonomous vehicle is controlled to grasp the target object based on the object coordinates and the current position coordinates of the autonomous vehicle, including:

[0022] Based on the object coordinates of the target object and the current position coordinates of the autonomous vehicle, each candidate path node is determined;

[0023] Based on the distance information between each candidate path node and the obstacle, select the target path node from each candidate path node;

[0024] Based on each target path node, determine the movement path from the current position coordinates to the object coordinates;

[0025] The driverless vehicle is controlled to move to the object's coordinates according to the movement path and perform a grasping action to grab the target object.

[0026] In some alternative implementations, the target path node is selected from the candidate path nodes based on the distance information between each candidate path node and the obstacle, including:

[0027] Based on the distance information between each candidate path node and the obstacle, determine the collision risk value of each candidate path node;

[0028] The target path node is selected from the candidate path nodes based on their collision risk values.

[0029] In some alternative implementations, the movement path from the current position coordinates to the object coordinates is determined based on each target path node, including:

[0030] For each target path node, the turning angle associated with the target path node is determined based on the node coordinates of the target path node and the node coordinates of other path nodes, and the turning cost of the target path node is determined based on the turning angle; where other path nodes are other target path nodes adjacent to the target path node.

[0031] Based on the turning cost and collision risk value of each target path node, as well as the positional relationship between each target path node, determine the movement path from the current position coordinates to the object coordinates.

[0032] Secondly, this application also provides an unmanned vehicle control device, comprising:

[0033] The data acquisition module is used to acquire object image data of the target object and environmental data of the target object's environment within the current time window during the unmanned vehicle's grasping operation of the target object.

[0034] The data processing module is used to process the acquired object image data and environmental data to obtain the current state vector;

[0035] The action determination module is used to select the target action from each candidate action based on the value function and the current state vector. The value function describes the contribution of each candidate action performed by the autonomous vehicle in each candidate state to the operation process of the autonomous vehicle. The value function is updated based on the actual action execution of the autonomous vehicle in the previous time window.

[0036] The action execution module is used to control the unmanned vehicle to grasp the target object based on the object coordinates of the target object and the current position coordinates of the unmanned vehicle when the target action is a grasping action.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0038] During the process of the unmanned vehicle performing the grasping operation of the target object, the information collection device deployed on the unmanned vehicle acquires the object image data of the target object and the environmental data of the environment in which the target object is located within the current time window.

[0039] The acquired object image data and environmental data are processed to obtain the current state vector;

[0040] Based on the value function and the current state vector, a target action is selected from each candidate action; whereby the value function describes the contribution of each candidate action performed by the autonomous vehicle in each candidate state to the operation process of the autonomous vehicle; the value function is updated according to the actual action performance of the autonomous vehicle in the previous time window;

[0041] When the target action is a grasping action, the autonomous vehicle is controlled to grasp the target object based on the object coordinates and the current position coordinates of the autonomous vehicle.

[0042] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0043] During the process of the unmanned vehicle performing the grasping operation of the target object, the information collection device deployed on the unmanned vehicle acquires the object image data of the target object and the environmental data of the environment in which the target object is located within the current time window.

[0044] The acquired object image data and environmental data are processed to obtain the current state vector;

[0045] Based on the value function and the current state vector, a target action is selected from each candidate action; whereby the value function describes the contribution of each candidate action performed by the autonomous vehicle in each candidate state to the operation process of the autonomous vehicle; the value function is updated according to the actual action performance of the autonomous vehicle in the previous time window;

[0046] When the target action is a grasping action, the autonomous vehicle is controlled to grasp the target object based on the object coordinates and the current position coordinates of the autonomous vehicle.

[0047] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0048] During the process of the unmanned vehicle performing the grasping operation of the target object, the information collection device deployed on the unmanned vehicle acquires the object image data of the target object and the environmental data of the environment in which the target object is located within the current time window.

[0049] The acquired object image data and environmental data are processed to obtain the current state vector;

[0050] Based on the value function and the current state vector, a target action is selected from each candidate action; whereby the value function describes the contribution of each candidate action performed by the autonomous vehicle in each candidate state to the operation process of the autonomous vehicle; the value function is updated according to the actual action performance of the autonomous vehicle in the previous time window;

[0051] When the target action is a grasping action, the autonomous vehicle is controlled to grasp the target object based on the object coordinates and the current position coordinates of the autonomous vehicle.

[0052] The aforementioned unmanned vehicle control method, device, equipment, and readable storage medium, during the unmanned vehicle's grasping operation of a target object, processes the object image data of the target object and the environmental data of the target object's environment collected within the current time window to obtain a current state vector. Based on a value function and the current state vector, a target action is selected from various candidate actions. Then, when the target action is a grasping action, the unmanned vehicle is controlled to grasp the target object based on the object's coordinates and the unmanned vehicle's current position coordinates. The value function describes the contribution of each candidate action executed by the unmanned vehicle in each candidate state to the unmanned vehicle's operation. The value function is updated based on the unmanned vehicle's actual action execution in the previous time window. Compared to related technologies that execute grasping tasks based on a preset driving path and action sequence, the above method, on the one hand, ensures the rationality of action selection by selecting a suitable target action based on the current state vector through a real-time updated value function; on the other hand, when the target action is a grasping action, controlling the unmanned vehicle to grasp the target object by combining the object coordinates and the unmanned vehicle's current position coordinates ensures the reliability of the grasping action execution. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1This is a flowchart illustrating an unmanned vehicle control method in one embodiment;

[0055] Figure 2 This is a schematic diagram of the value function update process in one embodiment;

[0056] Figure 3 This is a schematic diagram of the value function update process in another embodiment;

[0057] Figure 4 This is a schematic diagram of the process of controlling an unmanned vehicle to grasp a target object in one embodiment;

[0058] Figure 5 This is a flowchart illustrating the process of determining a movement path in one embodiment;

[0059] Figure 6 This is a flowchart illustrating the unmanned vehicle control method in another embodiment;

[0060] Figure 7 This is a structural block diagram of an unmanned vehicle control device in one embodiment;

[0061] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] With the rapid development of artificial intelligence and automation technologies, unmanned operating systems are playing an increasingly important role in industries such as manufacturing, logistics, and warehousing. In particular, intelligent grasping and control systems based on unmanned vehicles have become a research hotspot in material handling, cargo grasping, and transportation tasks.

[0064] Existing autonomous vehicle control methods typically select a preset driving path and action sequence based on the autonomous vehicle's state and actions, generate a grasping operation strategy, and control the autonomous vehicle to perform the grasping operation according to the grasping operation strategy.

[0065] However, existing unmanned vehicle control methods rely on preset path planning and fixed action sequences, which are difficult to adapt to the real-time decision-making requirements in complex dynamic environments, thus reducing the reliability of unmanned vehicle operations.

[0066] Based on this, in an exemplary embodiment, an unmanned vehicle control method is provided, which will be described using an example of the method being applied to a terminal device, such as... Figure 1 As shown, the specific steps include:

[0067] S101, during the process of the unmanned vehicle performing the grasping operation of the target object, the object image data of the target object and the environmental data of the environment where the target object is located are acquired by the information acquisition device deployed on the unmanned vehicle within the current time window.

[0068] The target object is the object that the autonomous vehicle needs to grasp. The grasping operation is the process of grasping and processing the object, which may include three steps: grasping, adjusting, and releasing. The information acquisition equipment refers to the relevant equipment for collecting information, such as visual sensors and smart sensors.

[0069] The current time window refers to the data acquisition period in which the current moment occurs. Object image data refers to the image data of the target object, which may include, but is not limited to, the target object's position, orientation, size, and shape. Environmental data refers to relevant data about the external environment in which the target object is located, which may include, but is not limited to, temperature, humidity, and light intensity.

[0070] Optionally, while the autonomous vehicle is performing a grasping operation on a target object, visual sensors deployed on the vehicle can acquire image data of the target object within the vehicle's operating area during the current time window, and intelligent sensors can acquire environmental data of the target object during the current time period. Furthermore, the acquired image data and environmental data can be processed through cleaning, noise reduction, and normalization.

[0071] It is worth noting that by working together with visual sensors and smart sensors, comprehensive image information of the target object and external environmental conditions can be acquired. This comprehensive acquisition of multi-dimensional data provides rich operational information. Furthermore, by cleaning, denoising, and normalizing, the accuracy of image data and environmental variable data is improved, and a clearer and more accurate data foundation is provided for subsequent data analysis and decision-making.

[0072] S102, process the acquired object image data and environmental data to obtain the current state vector.

[0073] The current state vector represents the state information of the target object within the current time window. It's worth noting that during the autonomous vehicle's movement, the state vector associated with the target object can be used to infer the state of the autonomous vehicle itself.

[0074] Optionally, a convolutional neural network (CNN) can be constructed first and trained using a training dataset. A loss function and an Adaptive Moment Estimation (Adam) optimizer can be defined to iteratively optimize the CNN parameters until the loss converges. Afterward, object image data can be input into the trained CNN to obtain object feature vectors. These feature vectors include the center coordinates of the target object in 3D space, the object's spatial pose angle, and the object's width and height.

[0075] Furthermore, the object feature vector and environmental data can be fused to obtain the current state vector associated with the target object. This current state vector can include environmental variables corresponding to the object feature vector and environmental data, as well as state information from adjacent historical time steps.

[0076] S103: Select the target action from each candidate action based on the value function and the current state vector.

[0077] The value function describes the contribution of each candidate action performed by the autonomous vehicle in each candidate state to the vehicle's operation. The contribution is the action value; the greater the contribution, the higher the action value, and the better the action is for the overall operation. That is, the value function can include the action value of each state vector-action pair. For example, an action value function algorithm (Q-learning algorithm) can be used to generate the value function by combining the state vectors and the action set. Furthermore, to ensure the reliability of the value function, it can be updated in real time based on the actual action execution of the autonomous vehicle within the previous time window.

[0078] The so-called candidate actions are the actions that the unmanned vehicle may take during the grasping operation, such as grasping, moving and releasing (grabbing is used to grasp the target object, moving is used to adjust the position, angle or direction of movement of the unmanned vehicle to approach the target object according to the current position and posture of the object, and releasing is used to perform the object release operation to complete the grasping task).

[0079] Optionally, a candidate action set A' can be constructed first based on empirical rules for autonomous vehicle operation, where the candidate action set A' can include all candidate actions. Then, based on the value function (Q-learning algorithm), the action value between the current state vector and each candidate action can be determined according to the current state vector, and the candidate action with the highest action value can be selected as the target action.

[0080] Action value characterizes the contribution of any candidate action to the autonomous vehicle's operation under different candidate states. An ideal action is the optimal action at each time step. Action execution result characterizes whether the ideal action was executed at each time step. Execution reward value is the reward value associated with the action execution result.

[0081] For example, a value function can be used to determine the action value of each candidate action based on the current state vector, and the candidate action with the highest action value among all candidate actions can be taken as the target action under the current time window. For example, the state vector S under time step t can be solved using the Q-learning algorithm with reference to the following formula (1). t Action value Q(S) between candidate actions t A t ). Among them, Q(S) t A t Let S represent the current state vector S at time step t, and Q value when action A is performed at time step t. Then, the maximum Q(S) is... t A t A in ) t As the target action Where, arg max A' The operator for finding the maximum value (i.e., finding the action that maximizes the Q value from the candidate action set A').

[0082] (1)

[0083] Understandably, dividing continuous data into smaller time steps using the sliding time window technique and calculating the target action at each time step helps the agent more accurately capture changes in the environmental state. The action set can cover the possible behaviors of the autonomous vehicle in the grasping task, thus ensuring that there are appropriate strategies to choose from in different situations. For example, the grasping action is suitable for capturing target objects, the moving action is suitable for navigation needs in complex environments, and the releasing action is the sign of task completion. By integrating multiple actions, various grasping tasks can be handled efficiently, and by using the Q-learning algorithm to solve for the state and action at each time step, optimal decision support can be provided to the agent.

[0084] Furthermore, by learning the Q-value function, the agent can gradually understand which actions bring higher long-term rewards in specific states, thereby optimizing its execution efficiency and task success rate. During execution, the greedy operation ensures that the action with the maximum Q-value is selected, enabling the agent to find the optimal path in the environment. Moreover, the greedy operation allows the present invention to select the optimal action at each decision, quickly guiding the autonomous vehicle to execute the task. Secondly, at each time step, the action with the maximum Q-value is selected to perform the task execution, ensuring the optimal effect of task execution.

[0085] S104: When the target action is a grasping action, control the unmanned vehicle to grasp the target object based on the object coordinates of the target object and the current position coordinates of the unmanned vehicle.

[0086] Here, "object coordinates" refers to the position coordinates of the target object. "Current position coordinates" refers to the position coordinates of the autonomous vehicle at the current moment.

[0087] Optionally, if the target action is a grasping action, a movement path from the object's coordinates to the current position coordinates can be determined based on obstacle information in the area where the target object is located, and the autonomous vehicle can be controlled to grasp the target object based on the movement path. If the target action is not a grasping action (movement / release), control parameters can be recorded and the corresponding movement or release operation can be executed.

[0088] In the aforementioned unmanned vehicle control method, during the unmanned vehicle's grasping operation of a target object, the current state vector is obtained by processing the object image data and environmental data of the target object's environment collected within the current time window. Based on the value function and the current state vector, a target action is selected from among the candidate actions. Then, if the target action is a grasping action, the unmanned vehicle is controlled to grasp the target object based on the object's coordinates and the unmanned vehicle's current position coordinates. The value function describes the contribution of each candidate action performed by the unmanned vehicle in each candidate state to the vehicle's operation. The value function is updated based on the actual actions performed by the unmanned vehicle in the previous time window. Compared to related technologies that execute grasping tasks based on preset driving paths and action sequences, this method, on the one hand, ensures the rationality of action selection by selecting a suitable target action based on the current state vector through a real-time updated value function; on the other hand, when the target action is a grasping action, controlling the unmanned vehicle to grasp the target object by combining the object coordinates and the unmanned vehicle's current position coordinates ensures the reliability of the grasping action execution.

[0089] Based on the above embodiments, this application provides an optional method for updating the value function, such as... Figure 2As shown, the specific steps include:

[0090] S201, determine the current execution reward value based on the result of the unmanned vehicle's action on the target.

[0091] Here, the so-called action execution result refers to the execution result of the target action. For example, in executing the target action... When the corresponding grab / move / release is successful, the reward value is positive; otherwise, it is negative; and when there is no action, it is zero. The so-called current execution reward value is the reward value obtained by the autonomous vehicle when performing the target action within the current time window.

[0092] Optionally, after obtaining the action execution result of the autonomous vehicle on the target action, the standard reward value associated with the action execution result can be used as the current execution reward value.

[0093] It is worth noting that the reward mechanism is crucial for reinforcement learning systems. In this embodiment, a positive reward is given for successful grasping, moving, or releasing, a negative reward is given for failure, and zero reward is given for no action. This mechanism helps the agent distinguish between effective and ineffective actions, further optimizing the decision-making process. Through this reward feedback mechanism, the present invention can self-adjust and gradually improve the task execution strategy.

[0094] S202, update the value function based on the current execution reward value and the action value between the next state vector and each candidate action in the value function.

[0095] The next state vector is determined based on the current predicted state vector.

[0096] Optionally, the next state vector can be predicted based on the autonomous vehicle's historical operating experience and the current state vector. Then, the action value between the next state vector and each candidate action can be determined by combining the current value function. Furthermore, the current value function is updated by combining the current execution reward value and the action value between the next state vector and each candidate action in the value function.

[0097] In this embodiment, by updating the value function based on the current execution reward value and the action value between the next state vector and each candidate action in the value function, the autonomous vehicle can gradually learn and optimize its action strategy as it continuously receives reward values. This enables the autonomous vehicle to dynamically adjust its strategy based on feedback, thereby performing tasks more efficiently in complex environments.

[0098] Based on the above embodiments, this application provides another optional method for updating the value function, such as... Figure 3 As shown, it specifically includes the following:

[0099] S301, based on the current execution reward value and the action value between the next state vector and each candidate action in the value function, perform the first update of the value function.

[0100] Optionally, the current value function can be updated by combining the current execution reward value with the action value between the next state vector and each candidate action in the value function.

[0101] For example, the following formula (2) can be used to evaluate the vector S of the driverless car entering the next state. t+1 The value of each candidate action Q(S) t+1 After that, based on the highest value candidate action and the current execution reward value R, t The current value function is updated to obtain the first value function.

[0102] (2)

[0103] Among them, Q 1 (S t A t () represents the Q value updated by the state vector S at time step t when it performs action A at time step t in the first value function; This represents the learning rate (a fixed value of 0.1 or 0.01). This represents the discount factor (a constant value of 0.9 or 0.99). In formula (2), Q(S) t+1 ,A') is determined based on the unupdated value function.

[0104] S302, based on the preset adjustment parameters, the current execution reward value, and the action value between the next state vector and each candidate action in the value function after the first update, perform a second update on the value function after the first update.

[0105] Among them, the so-called preset adjustment parameter can be a relaxation parameter, which is used to control the weight of the execution reward value.

[0106] Optionally, preset adjustment parameters can be used to adjust the current execution reward value and the action value between the next state vector and each candidate action in the value function after the first update, so as to perform a second update on the value function after the first update.

[0107] For example, you can refer to the following formula (3) based on the relaxation parameters. For the current execution reward value R t The second value function is obtained by processing the action values ​​between the next state vector and each candidate action in the first value function. Here, Q... 2 (S t A t() represents the Q value updated when the state vector S at time step t performs action A at time step t in the second value function. In formula (3), Q(S) t+1 ,A') is determined based on the first-valued function.

[0108] (3)

[0109] S303, sample each candidate action pair in the value function after the second update, and update the value function for the third time based on the sampling results and the current execution reward value.

[0110] Each candidate action pair includes a candidate state vector and a candidate action associated with the candidate state vector.

[0111] In one alternative implementation, random sampling can be performed from each candidate action pair to obtain sampled action pairs. Then, the value function after the second update is updated a third time by combining the action value of each sampled action pair in the value function after the second update with the current execution reward.

[0112] In another alternative implementation, the action value of each sampled action pair is determined based on the sampled action pairs in the sampling processing results and the value function after the second update; the value function after the second update is then updated a third time based on the action value of each sampled action pair and the current execution reward value.

[0113] Specifically, refer to the following formula (4). Using the Monte Carlo method, for each sampled action pair, the action value of that sampled action pair can be determined in the value function after the second update. Then, the action values ​​of each sampled action pair are averaged to obtain the average action value. Further, the average action value is updated using the current execution reward value to obtain the third value function.

[0114] (4)

[0115] Among them, Q 3 (S t A t ) represents the Q-value updated by the state vector S at time step t when it performs action A at time step t in the third value function; m represents the total number of samples, V n (Q i ) represents the Q value of the i-th state vector randomly sampled in the n-th iteration. In this embodiment, it can be the Q value in the second iteration, i.e., the second value function.

[0116] Furthermore, the value function that has been updated three times can be used as the value function associated with the next time window. That is, the value function associated with the current time window is the value function obtained after three updates based on the action execution of the unmanned vehicle in the previous time window.

[0117] Understandably, to ensure the reliability of action value determination, in addition to the three updates, a confidence upper limit formula from multi-armed slot machines can be introduced. That is, the third value function, combined with the confidence upper limit formula from multi-armed slot machines, determines the target action for the next time window. The confidence upper limit formula from multi-armed slot machines adds an "uncertainty reward" or "exploration bonus" to the Q-value of each candidate action.

[0118] For example, the target action for the next time window can be determined by referring to the following formula (5), which combines the third value function with the number of visits between each state vector and the action, using the upper confidence bound (UCB) formula in multi-armed slot machines.

[0119] Among them, A j ' represents the j-th action in the action set, Represents a constant factor (used to adjust the exploration intensity); log(∙) represents the logarithmic function; n represents the current iteration number; N(S t A t ) represents the number of visits between the state vector and actions at time step t, that is, in state S t Next, execute action A t The number of times in history.

[0120] (5)

[0121] Understandably, the UCB formula in multi-armed slot machines dynamically adjusts the balance between exploration and exploitation based on the number of visits to each state-action pair. This formula considers the number of visits for each action and adjusts the degree of exploration through a logarithmic function, which effectively avoids getting trapped in local optima. For grasping tasks, the UCB strategy helps the agent adapt to different environments, thereby ensuring the success rate and efficiency of the grasping task. If the required action is grasping, the center coordinates of the target object in three-dimensional space are extracted from the object feature vector. If the current action is not grasping, the control parameters are recorded and the corresponding move or release operation is executed. The key to this step is that it ensures the accurate execution of the grasping task by precisely locating the target object, thereby improving the overall operation efficiency.

[0122] In this embodiment, the value function is updated three times. Specifically, a preset adjustment parameter (relaxation parameter) is introduced during the second update based on the initial update. By adjusting the update magnitude of the Q-value, the action selection strategy is further optimized, thereby refining the Q-value update process and avoiding training instability caused by over-updates. This ensures a smoother and more efficient learning process in complex scenarios. Subsequently, Monte Carlo sampling is used to obtain more potential optimization schemes from the updated Q-values. The random sampling method can extract multiple possible action values ​​from the second updated Q-values ​​and combine these values ​​for three updates. This improves the accuracy of the Q-values ​​and allows for better adaptation to environmental changes through multiple updates, enhancing the diversity and flexibility of action selection.

[0123] Based on the above embodiments, this application provides an optional method for controlling an unmanned vehicle to grasp a target object, such as... Figure 4 As shown, it specifically includes the following content:

[0124] S401, based on the object coordinates of the target object and the current position coordinates of the unmanned vehicle, determine each candidate path node.

[0125] Candidate path nodes are the path nodes that exist between the object's coordinates and the current position coordinates.

[0126] Optionally, a mesh generation method can be used to mesh the environmental information of the target object, so that each mesh point is represented as a candidate path node.

[0127] S402, Select the target path node from each candidate path node based on the distance information between each candidate path node and the obstacle.

[0128] The so-called obstacle information refers to information that obstructs the operation of the autonomous vehicle. The target path node is the path node that is not affected by obstacles.

[0129] In one alternative approach, the location of obstacles can be identified and the obstacle area can be marked using LiDAR deployed on the autonomous vehicle. Then, for each candidate path node, the distance between that candidate path node and each obstacle can be determined, and the minimum distance is taken as the collision distance associated with that candidate path node.

[0130] Furthermore, candidate path nodes whose collision distance is less than a distance threshold can be used as target path nodes.

[0131] S403, determine the movement path from the current position coordinates to the object coordinates based on each target path node.

[0132] Optionally, the angle information between two connected target path nodes can be combined to construct a movement path from the current position coordinates to the object coordinates. Alternatively, each target path node can be input into a trained path generation model, which will then generate a movement path from the current position coordinates to the object coordinates based on the target path nodes.

[0133] S404 controls the unmanned vehicle to move to the object coordinates according to the movement path and perform a grasping action to grasp the target object.

[0134] Optionally, the required actions and final movement path can be converted into code information based on programming language technology and input into the controller to control the unmanned vehicle's grasping operation. The control code generated by the programming language enables automated and precise control command output, allowing the intelligent unmanned vehicle to perform grasping actions with higher accuracy and efficiency, reducing errors and improving operation quality.

[0135] Furthermore, a visual interface can be used to display the control results during the control process. Successful capture is indicated in green, and failure in red. In addition, the visual interface provides real-time feedback during task execution, especially in capture tasks. Successful capture is indicated in green, and failure in red; this intuitive display method instantly reflects the task's execution status.

[0136] To ensure the reliability of operational data storage, data can be stored in a database. This involves storing the required actions, the final crawling path, and the crawling results as files, adding corresponding timestamps to the files, and then storing them in the database. By storing the action, path, and result information of the crawling task as files and adding timestamps to the files before storing them in the database, the standardization, traceability, and security of the task data can be ensured.

[0137] In this embodiment, environmental information is acquired by lidar, and the environmental data is transformed into a gridded set of path points based on the grid generation method. Each path point represents a candidate path node, which ensures efficient computation of path planning in discrete space and guarantees the reliability of unmanned vehicle control.

[0138] Based on the above embodiments, this application provides an optional method for determining target path nodes, specifically, determining the collision risk value of each candidate path node based on the distance information between each candidate path node and the obstacle; and selecting the target path node from each candidate path node based on the collision risk value of each candidate path node.

[0139] The collision risk value is used to characterize the probability of an autonomous vehicle colliding with an obstacle when it is located at a candidate path node.

[0140] Optionally, a gridded set of path points can be generated based on each candidate path node, and the coordinates of the points in the grid corresponding to the candidate path nodes can be used as the coordinates of the candidate path nodes. Then, the object coordinates of the target object are added to the path point set as the final target point, and the path node set is initialized. During path node initialization, it is ensured that all generated path nodes are not located within obstacle areas.

[0141] Furthermore, you can refer to the following formula (6) to traverse the set of path nodes, use Euclidean distance to calculate the distance from each path node to the obstacle area, and select the minimum distance value to calculate the collision risk value.

[0142] (6)

[0143] Among them, Risk(P) e ) represents the collision risk value of the e-th path node, min Oh ∈O indicates that the minimum distance value is selected from the set of path nodes O using the minimization operation, ∈ indicates belonging to, d(P e O h ) represents the distance between the e-th path node and the h-th obstacle, dius(O h ) represents the size of the h-th obstacle (the radius or size of the obstacle can be obtained from the calibrated obstacle area).

[0144] Optionally, a decision threshold can be set using the domain method, and the candidate path node can be determined to have a collision risk by comparing the collision risk value with the decision threshold. Candidate path nodes that do not have a collision risk can be selected as target path nodes.

[0145] In this embodiment of the application, by calculating the distance from each path node to the obstacle area and evaluating its collision risk value, potential collision paths can be effectively eliminated, thereby preventing the autonomous vehicle from entering the dangerous area. The benefit of this step is to ensure the safety of path planning, avoid collisions with obstacles, and ensure the smooth completion of the task.

[0146] Based on the above embodiments, this application provides an optional method for determining the movement path, such as... Figure 5 As shown, it specifically includes the following:

[0147] S501, for each target path node, determine the turning angle associated with the target path node based on the node coordinates of the target path node and the node coordinates of other path nodes, and determine the turning cost of the target path node based on the turning angle.

[0148] Other path nodes are those adjacent to the target path node. Turning angle refers to the turning angle of the autonomous vehicle at the target path node. Turning cost represents the cost incurred by the autonomous vehicle when turning.

[0149] Optionally, you can refer to the following formula (7) to use the polar angle function to calculate the direction angle between two adjacent target path nodes, and obtain the direction angle between the target path node and the adjacent target path node; then, you can refer to the following formula (8) to calculate the turning angle between adjacent target path nodes based on the direction angle.

[0150] (7)

[0151] (8)

[0152] Where, ∆θ e θ represents the turning angle between the e-th target path node and e+1, and between e+1 and e+2. e+1,e+2 θ represents the directional angle from the (e+1)th target path node to the (e+2)th target path node. e,e+1 This represents the direction angle from the e-th target path node to e+1. For turning angle; This indicates otherwise.

[0153] Furthermore, the turning angle can be weighted by referring to the following formula (9) to obtain the turning cost of the target path node.

[0154] (9)

[0155] Among them, Turn(P) e (This is) the value of turning around; For coefficient weights.

[0156] S502, based on the turning cost and collision risk value of each target path node, and the positional relationship between each target path node, determine the movement path from the current position coordinates to the object coordinates.

[0157] Optionally, the turning costs of each target path node can be summed to obtain the total turning cost. Then, the total turning cost and the distance from the autonomous vehicle to the target path node with no collision risk are normalized, and the results are summed using addition to obtain the comprehensive cost.

[0158] Furthermore, target path nodes without collision risk can be integrated into a path sequence, and the path sequence can be used as individuals in a genetic algorithm. A population can be randomly generated for initialization, and an objective function can be defined based on the collision risk value and the overall cost. The objective function is referred to in the following formula (10).

[0159] (10)

[0160] Among them, Fess(P e ) represents the objective function of the e-th target path node, Cost(P) e ) represents the total cost of the e-th target path node, and H represents the path sequence.

[0161] Referring to the following formula (11), the objective function value is used as the individual fitness value. A random variable is generated using a random number generator to calculate the fitness value. Then, individuals with the two largest fitness values ​​are selected sequentially for crossover to generate crossover individuals. Next, crossover individuals are randomly selected from the crossover individuals, and random variables are applied to the path node positions in the crossover individuals to generate mutated individuals. Wherein, P ted P represents a variant individual. k φ represents the position of the path node in the kth intersecting individual, and φ represents the random variation.

[0162] (11)

[0163] After iterating to the maximum number of times, the individual with the maximum fitness value is output, and the path sequence is extracted as the final movement path.

[0164] In this embodiment, after selecting path nodes with no collision risk, the accuracy of path planning is further optimized by calculating the directional angle and turning angle between adjacent path nodes. Large-angle turns often imply higher control costs; therefore, by adjusting the turning angle and calculating the turning cost, the smoothness and cost of the path can be effectively controlled. Adjusting the turning angle using a turning constant helps balance the economy and efficiency of path planning. The adjusted turning cost more realistically reflects the actual execution difficulty of the path, providing more suitable path selection for autonomous vehicles. This step improves the smoothness of the path, reduces energy consumption and time consumption, and thus improves the practical feasibility of path planning. Secondly, by... By combining turning costs and path distances, a comprehensive cost assessment is obtained. This comprehensive cost considers both turning costs and path safety, enabling autonomous vehicles to simultaneously optimize paths while maintaining smoothness, safety, and efficiency. This method provides multi-dimensional optimization criteria for path planning by comprehensively considering different factors, and can generate optimal or near-optimal paths in complex environments, further improving the task execution efficiency and safety of autonomous vehicles. As the environment dynamically changes, the autonomous vehicle can perceive environmental changes in real time and adjust its path planning strategy accordingly. The path planning method of this invention can adapt to real-time environmental changes, such as the appearance of new obstacles or changes in environmental conditions, ensuring the stability and efficiency of autonomous vehicles in different environments.

[0165] Figure 6 This is a flowchart illustrating an autonomous vehicle control method in another embodiment. Based on the above embodiments, this embodiment provides an optional example of an autonomous vehicle control method. (Combined with...) Figure 6 The specific implementation process is as follows:

[0166] S601, during the process of the unmanned vehicle performing the grasping operation of the target object, acquires the object image data of the target object and the environmental data of the environment in which the target object is located, which are collected by the information collection device deployed on the unmanned vehicle within the current time window.

[0167] S602, the acquired object image data and environmental data are processed to obtain the current state vector, and the target action is selected from each candidate action according to the value function and the current state vector.

[0168] The value function describes the contribution of each candidate action performed by the autonomous vehicle in each candidate state to the operation of the autonomous vehicle; the value function is updated based on the actual actions performed by the autonomous vehicle in the previous time window.

[0169] S603, determine the current execution reward value based on the result of the autonomous vehicle's action on the target.

[0170] S604, based on the current execution reward value and the action value between the next state vector and each candidate action in the value function, perform the first update of the value function.

[0171] The next state vector is predicted based on the current predicted state vector.

[0172] S605, based on the preset adjustment parameters, the current execution reward value, and the action value between the next state vector and each candidate action in the value function after the first update, perform a second update on the value function after the first update.

[0173] S606, sample each candidate action pair, and update the value function a third time based on the current execution reward value, the sampling result, and the value function after the second update.

[0174] Each candidate action pair includes a candidate state vector and a candidate action associated with the candidate state vector.

[0175] Optionally, the action value of each sampled action pair is determined based on the sampled action pairs in the sampling processing results and the value function after the second update; the value function after the second update is then updated for the third time based on the action value of each sampled action pair and the current execution reward value.

[0176] S607: When the target action is a grasping action, the driver is controlled to grasp the target object based on the object coordinates and the current position coordinates of the driver.

[0177] Optionally, each candidate path node is determined based on the object coordinates of the target object and the current position coordinates of the autonomous vehicle; the collision risk value of each candidate path node is determined based on the distance information between each candidate path node and the obstacle; and the target path node is selected from each candidate path node based on the collision risk value of each candidate path node.

[0178] Furthermore, for each target path node, based on its coordinates and those of other path nodes, the associated turning angle is determined, and the turning cost of the target path node is determined based on the turning angle; where other path nodes are those adjacent to the target path node. Based on the turning cost and collision risk value of each target path node, as well as the positional relationships between them, a movement path from the current location coordinates to the object coordinates is determined. The autonomous vehicle is then controlled to move to the object coordinates according to the movement path and execute a grasping action to grab the target object.

[0179] The specific processes of S601-S607 described above can be found in the description of the above method embodiments. Their implementation principles and technical effects are similar, and will not be repeated here.

[0180] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0181] Based on the same inventive concept, this application also provides an unmanned vehicle control device for implementing the unmanned vehicle control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more unmanned vehicle control device embodiments provided below can be found in the limitations of the unmanned vehicle control method described above, and will not be repeated here.

[0182] In one exemplary embodiment, such as Figure 7 As shown, an unmanned vehicle control device 1 is provided, including: a data acquisition module 10, a data processing module 20, an action determination module 30, and an action execution module 40, wherein:

[0183] The data acquisition module 10 is used to acquire object image data of the target object and environmental data of the environment where the target object is located, which are collected by the information collection device deployed on the unmanned vehicle within the current time window during the process of the unmanned vehicle performing the grasping operation of the target object.

[0184] The data processing module 20 is used to process the acquired object image data and environmental data to obtain the current state vector;

[0185] The action determination module 30 is used to select the target action from each candidate action based on the value function and the current state vector; wherein, the value function is used to describe the contribution of each candidate action executed by the autonomous vehicle in each candidate state to the operation process of the autonomous vehicle; the value function is updated according to the actual action execution of the autonomous vehicle in the previous time window;

[0186] The action execution module 40 is used to control the unmanned vehicle to grasp the target object based on the object coordinates of the target object and the current position coordinates of the unmanned vehicle when the target action is a grasping action.

[0187] In an exemplary embodiment, the unmanned vehicle control device 1 further includes a function update module, wherein the function update module is specifically used for:

[0188] Based on the autonomous vehicle's execution result of the target action, the current execution reward value is determined; based on the current execution reward value and the action value between the next state vector and each candidate action in the value function, the value function is updated; wherein, the next state vector is predicted based on the current predicted state vector.

[0189] In one exemplary embodiment, the function update module is further configured to:

[0190] The value function is updated for the first time based on the current execution reward value and the action value between the next state vector and each candidate action in the value function. The value function is then updated a second time based on preset adjustment parameters, the current execution reward value, and the action value between the next state vector and each candidate action in the value function after the first update. Each candidate action pair is sampled, and the value function is updated a third time based on the current execution reward value, the sampling results, and the value function after the second update. Each candidate action pair includes a candidate state vector and the candidate action associated with that state vector.

[0191] In one exemplary embodiment, the function update module is further configured to:

[0192] Based on the sampled action pairs in the sampling process results and the value function after the second update, determine the action value of each sampled action pair; based on the action value of each sampled action pair and the current execution reward value, perform a third update on the value function after the second update.

[0193] In one exemplary embodiment, the action execution module 40 is specifically used for:

[0194] Based on the object coordinates of the target object and the current position coordinates of the unmanned vehicle, candidate path nodes are determined; based on the distance information between each candidate path node and the obstacle, the target path node is selected from the candidate path nodes; based on each target path node, the movement path from the current position coordinates to the object coordinates is determined; the unmanned vehicle is controlled to move to the object coordinates according to the movement path and perform a grasping action to grasp the target object.

[0195] In one exemplary embodiment, the action execution module 40 is further configured to:

[0196] Based on the distance information between each candidate path node and the obstacle, determine the collision risk value of each candidate path node; based on the collision risk value of each candidate path node, select the target path node from among the candidate path nodes.

[0197] In one exemplary embodiment, the action execution module 40 is further configured to:

[0198] For each target path node, the turning angle associated with the target path node is determined based on the node coordinates of the target path node and the node coordinates of other path nodes. The turning cost of the target path node is then determined based on the turning angle. Other path nodes are those adjacent to the target path node. Based on the turning cost and collision risk value of each target path node, as well as the positional relationship between the target path nodes, the movement path from the current position coordinates to the object coordinates is determined.

[0199] The modules in the aforementioned unmanned vehicle control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0200] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an unmanned vehicle control method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0201] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0202] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0203] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0204] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0205] It should be noted that the data involved in this application (including autonomous vehicle operation data, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0206] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0207] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0208] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for controlling an unmanned vehicle, the method comprising: The method comprises: In the process of the unmanned vehicle performing a grabbing operation on a target object, obtaining object image data of the target object and environment data of an environment where the target object is located, which are collected by an information collection device arranged on the unmanned vehicle within a current time window; Processing the obtained object image data and environment data to obtain a current state vector; According to a value function and the current state vector, selecting a target action from candidate actions; wherein the value function is used to describe the contribution of the unmanned vehicle performing each candidate action in each candidate state to the operation process of the unmanned vehicle; the value function is updated according to the actual action execution of the unmanned vehicle in a previous time window; In the case that the target action is a grabbing action, based on object coordinates of the target object and current position coordinates of the unmanned vehicle, controlling the unmanned vehicle to grab the target object.

2. The method of claim 1, wherein, The method further comprises: According to an action execution result of the unmanned vehicle on the target action, determining a current execution reward value; According to the current execution reward value and an action value between a next state vector and each candidate action in the value function, updating the value function; wherein the next state vector is predicted according to the current predicted state vector.

3. The method of claim 2, wherein, The updating of the value function according to the current execution reward value and the action value between the next state vector and each candidate action in the value function comprises: According to the current execution reward value and the action value between the next state vector and each candidate action in the value function, first updating the value function; According to a preset adjustment parameter, the current execution reward value, and the action value between the next state vector and each candidate action in the value function after the first updating, second updating the value function after the first updating; Sampling each candidate action pair and, according to the current execution reward value, the sampling result, and the value function after the second updating, third updating the value function after the second updating; wherein each candidate action pair comprises a candidate state vector and a candidate action associated with the candidate state vector.

4. The method of claim 3, wherein, The third updating of the value function after the second updating according to the current execution reward value, the sampling result, and the value function after the second updating comprises: According to each sampling action pair in the sampling result and the value function after the second updating, determining an action value of each sampling action pair; According to the action value of each sampling action pair and the current execution reward value, third updating the value function after the second updating.

5. The method of claim 1, wherein, The controlling of the unmanned vehicle to grab the target object based on the object coordinates of the target object and the current position coordinates of the unmanned vehicle comprises: Based on the object coordinates of the target object and the current position coordinates of the unmanned vehicle, determining each candidate path node; According to distance information between each candidate path node and an obstacle, selecting a target path node from each candidate path node; determine, according to each target path node, a movement path from the current position coordinate to the object coordinate; control the unmanned vehicle to move to the object coordinate according to the movement path and perform a grabbing action to grab the target object.

6. The method of claim 5, wherein, The selecting the target path node from the candidate path nodes according to the distance information between each candidate path node and the obstacle comprises: determining, according to the distance information between each candidate path node and the obstacle, a collision risk value of each candidate path node; selecting the target path node from the candidate path nodes according to the collision risk value of each candidate path node.

7. The method of claim 5, wherein, The determining, according to each target path node, a movement path from the current position coordinate to the object coordinate comprises: for each target path node, determining a turning angle associated with the target path node according to the node coordinate of the target path node and the node coordinates of other path nodes, wherein the other path nodes are other target path nodes adjacent to the target path node, and determining a turning generation value of the target path node according to the turning angle; determining, according to the turning generation value and the collision risk value of each target path node and the positional relationship between the target path nodes, a movement path from the current position coordinate to the object coordinate.

8. An unmanned vehicle control device characterized by comprising: The device comprises: a data acquisition module configured to acquire, in a process in which the unmanned vehicle performs a grabbing operation on a target object, object image data of the target object and environment data of an environment in which the target object is located, which are collected by an information collection device arranged on the unmanned vehicle within a current time window; a data processing module configured to process the acquired object image data and environment data to obtain a current state vector; an action determination module configured to select a target action from candidate actions according to a value function and the current state vector, wherein the value function is used to describe a contribution degree of the unmanned vehicle performing each candidate action in each candidate state to a running process of the unmanned vehicle, and the value function is updated according to an actual action execution condition of the unmanned vehicle within a previous time window; an action execution module configured to, in a case where the target action is a grabbing action, control the unmanned vehicle to grab the target object based on an object coordinate of the target object and a current position coordinate of the unmanned vehicle. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images