A Method for Searching Objects by a Mobile Manipulator in an Occluded Environment Based on Association Probability

Abstract

The present invention discloses a method for searching for objects by a mobile manipulator in an occluded environment based on association probability. The present invention utilizes the association relationship between objects and imitates the memory mechanism of humans to construct a long-duration object association data set. Secondly, a target algorithm is used to identify objects in the environment, and the visible area of the object is used as known prior information to infer the association relationship between objects. By combining the recognition result with the knowledge in the data set, the association relationship between objects is updated. Finally, corresponding operations are performed according to the operation primitive library to execute the object search task and reduce the expected time required for the search. The present invention combines the human-like memory mechanism and the association relationship between objects to construct a long-duration object association data set. The association relationship between objects is updated through real-time sensor information, and at the same time, the human operation mode for objects is imitated to achieve precise operation of the occluder and improve the efficiency of object search by the mobile manipulator in the occluded environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile robot object search, and relates to a method for constructing an object association probability based on prior knowledge, and a method for searching for an object by a mobile manipulator in an occluded environment based on the association probability. Background Art

[0002] Recently, the progress of key robot technologies, such as power supply, positioning, mapping, navigation, and human-machine interaction, has brought hope for mobile robots to perform daily tasks in a home environment. In this context, there are some successful examples of using mobile robots, for example, making pancakes and pouring liquids. The common feature of these tasks is that the objects related to the tasks are clear or already present in the robot's field of view. However, in most real home scenarios, the environment required to complete daily tasks is often cluttered, and the objects are very likely to be occluded. Therefore, a mobile robot must be able to search for target objects in a cluttered environment, which is a prerequisite for the robot to perform daily tasks.

[0003] In existing object search methods, such as a method for searching for an object by a mobile robot that mimics human memory with the publication number CN114397894A and a method for searching for a target object by a robot with the shortest expected time based on a probability map with the publication number CN111427341A, although the probability method has played a positive role in improving the search efficiency, they have some limitations in considering the impact of environmental dynamics on the search efficiency. A method for searching for a target object by a robot based on an association probability semantic map with the publication number CN115979272A, although considering the probability of the historical distribution of objects, does not consider the unpredictable dynamic nature of objects. Therefore, the present invention aims to solve the problem of efficient object search by a mobile manipulator in an occluded environment. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of object search by a mobile manipulator in an occluded environment, and a method for searching for an object by a mobile manipulator in an occluded environment based on an association probability is proposed, and a probability graph model is used to help the mobile manipulator perform efficient object search.

[0005] The present invention proposes a method for searching for an object by a mobile manipulator in an occluded environment based on an association probability, so that the expected time for the mobile manipulator to search for an object in an occluded environment is small. The specific process is as follows. First, the association relationship and the humanoid memory mechanism are combined to construct a long-endurance object association data set. Secondly, the mobile manipulator uses the target recognition algorithm to take the area of the recognized object as the prior object association relationship, and updates their association probabilities according to the prior knowledge in the data set. Finally, the mobile manipulator performs corresponding operations on the object through the operation primitive library.

[0006] First aspect, a method for searching for an object by a mobile manipulator in an occluded environment based on association probability, comprising the following steps:

[0007] Step (1), construction of an object association probability data set;

[0008] Step (2), construction of a prior object association probability in an occluded environment;

[0009] Step (3), inference of object association probability based on the Bayesian method;

[0010] Step (4), searching for a target by a mobile manipulator in an occluded environment based on association probability

[0011] 4-1 Construction of a posterior object association probability:

[0012] 4-2 Construction of a mobile manipulator operation primitive library;

[0013] 4-3 Construction of a target search probability graph model for a mobile manipulator in an occluded environment;

[0014] 4-4 Target search for a mobile manipulator in an occluded environment:

[0015] According to the target search probability graph model of the mobile manipulator in the occluded environment in step 4-3, combined with the mobile manipulator operation primitive library in step 4-2, the mobile manipulator operates on the occluders in order from the largest to the smallest posterior object association probability in the target search probability graph model of the mobile manipulator in the occluded environment, so as to realize the search for the target object.

[0016] Second aspect, the present invention provides a target search system for a mobile manipulator in an occluded environment, comprising:

[0017] An observation vision module, configured to obtain a long-duration data set related to an object; and in an occluded environment, obtain a prior object association probability;

[0018] A state update module, configured to perform inference of object association probability based on the Bayesian method;

[0019] A motion planning module, configured to use a target search probability graph model of a mobile manipulator in an occluded environment to realize target search of the mobile manipulator in the occluded environment.

[0020] Third aspect, the present invention provides a computing device, comprising a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the method described above is implemented.

[0021] Fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed on a computer, the computer is made to execute the method described above.

[0022] The beneficial effects of the present invention are as follows:

[0023] The present invention combines the association relationship between objects and the human-like memory mechanism to construct a long-duration object association data set. The mobile manipulator uses a target recognition algorithm to identify objects in an occluded environment, and takes the visible area of the occluder as known prior information to infer the association relationship between the occluder and the target object. Then, by combining the prior information with the experience of the object association data set, the association relationship between the occluder and the target object is updated. Finally, by imitating the way humans operate objects, an operation primitive library is constructed, and corresponding operations are performed according to the attributes of the occluder to execute the object search task, thereby reducing the expected time required for the search and significantly improving the efficiency of the mobile manipulator in searching for objects in an occluded environment. Brief Description of the Drawings

[0024] Figure 1 It is a flowchart of the method of the present invention.

[0025] Figure 2 It is the construction process of the object association probability data set. Detailed Embodiments

[0026] The following further analyzes the present invention in conjunction with the drawings. The technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0027] The mobile manipulator platform in this example is Gibbon_Plus, equipped with an Intel Realsense D435 depth camera and a Ruichi Zhiguang LakiBeam1 lidar sensing system, a 6-DOF collaborative manipulator and a pneumatic soft finger actuator, and a NUC11 and a manipulator controller control system.

[0028] A method for a mobile manipulator to search for objects in an occluded environment based on association probability, as Figure 1 includes the following steps:

[0029] Step (1), construction of an object association probability data set, as Figure 2 ;

[0030] The construction of the object association dataset is single-blind. Those involved in dataset construction are unaware of the mechanism and purpose of the dataset, so their habits will not change due to subjective influence, thus ensuring the objectivity of the experiment as much as possible. In an indoor environment with N people, without disturbing normal life, object association statistics are conducted on the living environment of N people, with no less than 3 statistics per day for a duration of 720 hours, and filtering out randomly occurring and unrepresentative object associations. Since the indoor environment is dynamic, the information on object associations changes in real time;

[0031] 1-1 Target recognition:

[0032] Use the YOLOv5 target recognition algorithm adapted to Ubuntu 18.04 to detect the RGB color images obtained by the depth camera, obtain the rectangular frames enclosing the detected objects, and obtain the central pixel coordinates (u, v) of the objects through the rectangular frames; the YOLOv5 target recognition algorithm adapted to Ubuntu 18.04 can recognize the objects in the images and obtain their central coordinates in the pixel coordinate system;

[0033] 1-2 Conversion of two-dimensional pixel coordinates to three-dimensional global coordinates:

[0034] Combine the depth image of the depth camera to obtain the depth value corresponding to the central pixel, and count the object appearance frequency, attributes, and spatial position information; the spatial position conversion is shown in formula (1), which converts the position coordinates in the pixel coordinate system into the position coordinates in the global coordinate system:

[0035]

[0036] In the formula, P uv =[u, v, 1] T represents the homogeneous coordinates of the object in the pixel coordinate system, P w =[X w , Y w , Z w , 1] T represents the homogeneous coordinates of the object in the global coordinate system, Z c represents the object depth value measured by the depth camera, f x and f y are the focal lengths of the depth camera in the x and y directions respectively, c x and c y are the translation relationships between the origin of the pixel coordinate system and the optical axis respectively, and R and t are the rotation and translation transformation relationships of the depth camera coordinate system relative to the global coordinate system;

[0037] 1-3 Conduct statistics on the objects appearing in the living environment and construct the association probability of the objects:

[0038] For the associated probability of objects i and j in the environment, it is affected by the frequency of object appearance, spatial location, and object attributes; the frequency of object appearance affects the object association probability. There is a natural pattern between the object and the spatial location, which has a close association in daily life, and the object attributes can increase the object association probability. Let D assn denote the associated probability of two objects, as shown in Equation (2):

[0039]

[0040] where c(i,j) represents the frequency of appearance of two objects, d(i,j) represents the distance between two objects, α represents the attribute correlation coefficient of two objects. For example, a mouse and a keyboard belong to electronic products and have a high co-occurrence frequency, which is set to 9, and k represents the degree of memory of two objects.

[0041] After normalizing D assn (i,j), the associated probability P norm (i,j) of two objects is obtained, as shown in Equation (3):

[0042]

[0043] Construction of the associated probability distribution of objects 1-4:

[0044] Summarize and statistically analyze all the object association probabilities within 720 hours to calculate the distribution of all object association probabilities;

[0045] Step (2), construction of the prior object association probability in an occluded environment;

[0046] 2-1 Object recognition:

[0047] Use the YOLOv5 object recognition algorithm adapted to Ubuntu 18.04 to detect the RGB image and depth image obtained by the depth camera, obtain the rectangular frame enclosing the detected object, and use the rectangular frame to statistically analyze the area information and spatial location information of the object. The spatial location information is shown in Equation (1);

[0048] 2-2 The area of the occluder at the same spatial location distance; specifically:

[0049] Considering the working space of the mobile manipulator and the accuracy of the depth camera, the two-dimensional visible area of each object should be calculated at the same spatial location distance. The area of each occluder at the same spatial location distance is shown in Equation (4):

[0050]

[0051] where d SRepresents the unified spatial position distance from all occluders to the depth camera, d i Represents the spatial position distance from occluder i to the depth camera, S(i) represents the two-dimensional visible area of occluder i, S s (i) is the two-dimensional visible area of occluder i at the same scale, S s Represents the set of areas of all occluders at the same scale, o N Represents the Nth occluder;

[0052] 2-3 Calculate the prior object association probability:

[0053] Statistically calculate the total area sum ∑S of all occluders at the same scale in step 2-2 s (i), and calculate the ratio of the area of each occluder to the total area as the prior object association probability, as shown in formula (5):

[0054]

[0055] In the formula, P prior (i,j) represents the prior association probability between occluder i and target object j in the occlusion environment, o N Represents the Nth occluder;

[0056] Step (3), Object association probability reasoning based on the Bayesian method;

[0057] 3-1 Construction of the likelihood object association probability in the occlusion environment:

[0058] From step 2-3, it is known that the prior object association probability P of occluder i and target object j in the occlusion environment prior (i,j); According to the object association probability dataset constructed in step (1), extract the object association probability from the dataset and construct the likelihood association probability P of occluder i and target object j in the occlusion environment likelihood (i,j), as shown in formula (6):

[0059] P likelihood ={P likelihood (i,j)|i=o1,...,o N} (6)

[0060] In the formula, P likelihood is the set of likelihood association probabilities between occluder i and target object j;

[0061] 3-2 Update of the observation state of the occluder:

[0062] The mobile manipulator is at z tPerform target recognition in step 2-1 at all times. If the occluder is detected by the target recognition algorithm, set the observation state of the occluder to 1. If the occluder is not detected by the target recognition algorithm, set the observation state of the occluder to 0. Its formula is expressed as:

[0063]

[0064] In the formula, obs_stat(i|z t ) represents the observation state of occluder i at time z t , and o N represents the Nth occluder;

[0065] 3-3 Posterior object association probability update based on the Bayesian method:

[0066] Combined with step 2-3, the prior object association probability P t (i,j|z prior ) of the occluder and the target object in the occluded environment at time z t is known. Combined with step 3-1, the likelihood object association probability P likelihood (i,j) is known. Use Bayes to update the association probability. The posterior object association probability P t+1 (i,j|z posterior ) at time z t+1 is expressed as:

[0067] P posterior (i,j|z t+1 ) = P likelihood (i,j)·P prior (i,j|z t ) (8)

[0068] In the formula, P posterior (i,j|z t+1 ) represents the posterior association probability of occluder i and target object j in the occluded environment at time z t+1 ;

[0069] Step (4), Target search of the mobile manipulator in the occluded environment based on the association probability

[0070] 4-1 Construct the posterior object association probability:

[0071] Combined with step 3-3, the posterior object association probability P posterior (i,j) of occluder i and target object j is known. Construct the posterior object association probability set P posterior , which is expressed as:

[0072] P posterior = {P posterior (i,j)|i = o1,…,o N} (9)

[0073] 4-2 Construct the operation primitive library of the mobile manipulator; specifically:

[0074] The actions of humans to manipulate objects are not limited to grasping, but also include non-grasping actions such as pushing, pulling, and dumping. Inspired by the operations performed by humans to execute tasks, combined with step 2-1, the mobile manipulator constructs operation primitives for the mobile manipulator based on the object attribute information obtained by the target recognition algorithm, as shown in formula (10):

[0075] manipulation_primitives(i) = {grasp, push, pull, dump}, i = o1,..., o N (10)

[0076] In the formula, {grasp, push, pull, dump} are respectively the operations of the end effector of the mobile manipulator: grasping, pushing, pulling, and dumping, and manipulation_primitives(i) represents the operations performed by the mobile manipulator according to the attributes of the occluder i;

[0077] 4-3 Construct the target search probability graph model of the mobile manipulator in an occluded environment; the specific operation is:

[0078] The target search of the mobile manipulator in the occluded environment based on the association probability is regarded as a path-finding problem of a directed graph. Therefore, the target search probability graph model of the mobile manipulator in the occluded environment is represented by a directed graph Node, where the directed graph Node node contains the homogeneous coordinates P of the object in the global coordinate system w , the observation state obs_stat(i) of the occluder, and the posterior object association probability P posterior (i, j), as shown in formula (11):

[0079] Node = {obs_stat(i), P w , P posterior (i, j)|i = o1,..., o N ,} (11)

[0080] In the formula, o N represents the Nth occluder;

[0081] 4-4 Target search of the mobile manipulator in an occluded environment:

[0082] According to the target search probability graph model of the mobile manipulator in the occluded environment in step 4-3, combined with the operation primitive library of the mobile manipulator in step 4-2, the mobile manipulator operates on the occluders in order from largest to smallest according to the posterior object association probability in the target search probability graph model of the mobile manipulator in the occluded environment, so as to realize the search for the target object.

[0083] The posterior object association probability reflects the degree of association between the target object and the known information. Therefore, by sorting the posterior object association probabilities, nodes with a high degree of association with the target object are preferentially selected. For each selected graph node, according to the attributes of the objects in the node, the corresponding mobile manipulator operation primitive is executed to complete the search for the target object.

[0084] The goal of the mobile manipulator is to search for a sequence of graph nodes in an occluded environment. This sequence is sorted based on the posterior association probability, and it is desired to minimize the expected time E(T) by executing this sequence, which is expressed as:

[0085]

[0086] When the mobile manipulator searches for objects in an indoor environment, it continuously updates the association probabilities between objects using the information obtained from the depth camera. This makes the probability distribution model of the objects in the indoor environment gradually conform to the actual environment, enabling the mobile manipulator to search for objects more efficiently in this occluded environment.

[0087] The present invention combines the human-like memory mechanism with the association relationship between objects, and proposes a method for searching objects by a mobile manipulator in an occluded environment based on association probability, which improves the efficiency of the mobile manipulator when searching for objects in an occluded environment.

Claims

1. A target search method for a mobile manipulator in an occluded environment based on association probability, characterized in that The method comprises the following steps: Step (1), construction of object association probability dataset; Step (2), construction of prior object association probability under occlusion environment; Step (3), object association probability reasoning based on Bayesian method; Step (4): Target search of mobile manipulator in occluded environment based on association probability 4-1 Construct the posterior object association probability set P posterior : P posterior = {P posterior (i, j) | i = o1,..., o N} (1) Among which P posterior (i, j) represents the posterior object association probability between the occluder i and the target object j; 4-2 Build a mobile robot operation primitive library; 4-3 Construct a probability graph model for target search of mobile manipulator in occluded environment; 4-4 Mobile robot target search in an obstructed environment: According to the mobile robotic arm target search probability graph model under occluded environment in step 4-3, combined with the mobile robotic arm operation primitive library in step 4-2, the mobile robotic arm operates the occluders in sequence from large to small according to the posterior object association probability in the mobile robotic arm target search probability graph model under occluded environment, thereby realizing the search of the target object.

2. The method according to claim 1, wherein Step (1) specifically includes: 1-1 Target Identification: Use the target recognition algorithm to detect the RGB image obtained by the depth camera, obtain the rectangular frame surrounding the detected object, and obtain the central pixel coordinates (u, v) of the object through the rectangular frame; 1-2 Convert two-dimensional pixel coordinates to three-dimensional global coordinates: Combined with the depth image of the depth camera, the depth value corresponding to the central pixel is obtained, and the frequency, attributes and spatial position information of the object are counted; the spatial position coordinates in the pixel coordinate system are converted into the position coordinates in the global coordinate system: The spatial position conversion is shown in formula (2): where represents the homogeneous coordinates of the object in the pixel coordinate system, represents the homogeneous coordinates of the object in the global coordinate system, Z c represents the depth value of the object measured by the depth camera, f x and f y are the focal lengths of the depth camera in the x and y directions respectively, c x and c y are the translation relationships between the origin of the pixel coordinate system and the optical axis respectively, and R and t are the rotation and translation transformation relationships of the depth camera coordinate system relative to the global coordinate system; 1-3 Count the objects that appear in the living environment and construct the association probability of the objects; 1-4 Construction of object association probability distribution: The association probabilities of all objects within 720 hours are summarized and counted, and the distribution of the association probabilities of all objects is calculated.

3. The method according to claim 2, wherein Steps 1-3 are: Using D assn (i, j) represents the association probability of two objects, as shown in formula (3): Where c(i,j) represents the frequency of two objects, d(i,j) represents the distance between the two objects, α represents the attribute correlation coefficient between the two objects, and k represents the depth of memory of the two objects. Put D assn After normalizing (i,j), two object association probabilities P norm (i,j) are obtained, as shown in formula (4):

4. The method according to claim 2, wherein Step (2) is specifically: 2-1 Target Identification: Use the target recognition algorithm to detect the RGB image and depth image obtained by the depth camera, obtain a rectangular frame surrounding the detected object, and use the rectangular frame to count the area information and spatial position information of the object. The spatial position information is shown in formula (1); 2-2 The area of the obstruction at the same spatial position and distance; 2-3 Calculate the prior object association probability: Statistical step 2-2: Calculate the total sum ΣS of the areas of all occluders at the same scale s (i), Calculate the ratio of the area of each occluder to the total area as the prior object association probability, as shown in formula (5): Wherein, P prior (i, j) represents the prior association probability between the occluder i and the target object j in the occluded environment, and o N represents the Nth occluder.

5. The method according to claim 4, characterized in that Step 2-2 is specifically: Taking into account the working space of the mobile robot and the accuracy of the depth camera, the two-dimensional visible area of each object should be calculated at the same spatial position distance. The area of each occluder at the same spatial position distance is shown by formula (6): where d S represents the unified spatial position distance from all occluders to the depth camera, d i represents the spatial position distance from the i-th occluder to the depth camera, S(i) represents the two-dimensional visible area of the i-th occluder, S s (i) is the two-dimensional visible area of the i-th occluder at the same scale, S s represents the set of areas of all occluders at the same scale, o N represents the N-th occluder.

6. The method according to claim 5, characterized in that Step (3) is specifically: 3-1 Construction of likely object association probability under occlusion environment: Based on the object association probability dataset constructed in step (1), extract the object association probability from the dataset, and construct the likelihood association probability P likelihood (i, j) of the occluder i and the target object j in the occlusion environment, as shown in Equation (7): P likelihood = {P likelihood (i, j) | i = o1,..., o N} (7) where P likelihood is the set of likelihood association probabilities between the occluder i and the target object j; 3-2 Observation status update of obstructions: The mobile manipulator performs target recognition in step 2-1 at time z t If the occluder is detected by the target recognition algorithm, the observation state of the occluder is set to 1; if the occluder is not detected by the target recognition algorithm, the observation state of the occluder is set to 0. 3-3 Update of posterior object association probability based on Bayesian method: Combined with Step 2-3, we know z t The prior object association probability P of the occluder and the target object in the occluded environment at a certain moment prior (i, j|z t ), combined with Step 3-1, we know the likelihood object association probability P likelihood (i, j), use Bayes to update the association probability, and the posterior object association probability P t+1 at time z posterior (i, j|z t+1 ) is expressed as: P posterior (i,j|z t+1 ) = P likelihood (i,j)·P prior (i,j|z t ) (8) where P posterior (i, j|z t+1 ) represents the posterior association probability of the occluder i and the target object j in the occluded environment at time z t+1 ; Step 3-2 The observation state of the occluder is expressed as: where obs_stat(i|z t ) represents the observation state of the occluder i at time z t , and o N represents the Nth occluder.

7. The method according to claim 1, wherein Step 4-2 is specifically: The mobile robot arm obtains the object attribute information through the target recognition algorithm and constructs the mobile robot arm operation primitive library, as shown in formula (10): manipulation_primitives(i) = {grasp, push, pull, dump}, i = o1,..., o N (10) Where {grasp, push, pull, dump} are respectively the grasping, pushing, pulling, and dumping operations of the end effector of the mobile manipulator, and manipulation_primitives(i) represents the operations performed by the mobile manipulator according to the attributes of the occluder i.

8. The method according to claim 1 or 7, characterized in that Step 4-3 is specifically as follows: The target search of a mobile manipulator in an occluded environment based on the association probability is regarded as a path - finding problem in a directed graph. Therefore, the probability graph model for the target search of a mobile manipulator in the occluded environment is represented by a directed - graph Node. The directed - graph Node contains the homogeneous coordinates P of the object in the global coordinate system w , the observation state obs_stat(i) of the occluder, and the posterior object association probability P posterior (i,j), as shown in formula (11): Node={obs_stat(i),P w ,P posterior (i,j)|i=o1,...,o N ,} (11) Where o N represents the Nth occluder.

9. A target search system for a mobile manipulator in an occluded environment implementing the method according to any one of claims 1-8, characterized in that It includes: An observation vision module for obtaining a long-duration dataset associated with an object; And obtaining a prior object association probability in an occluded environment; A state update module for object association probability inference based on the Bayesian method; A motion planning module that uses a target search probability map model of the mobile manipulator in an occluded environment to implement target search of the mobile manipulator in an occluded environment.

10. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1-8 is implemented.

Images