Robot indoor person searching method and system based on active perception

Abstract

The invention discloses a robot indoor person searching method and system based on active perception. According to the method, the distribution and orientation information of the characters in the scene is analyzed in real time based on an active perception strategy, the minimum observation view is dynamically planned, repeated invalid search is effectively avoided, and more efficient character search is realized by filtering irrelevant characters and similar characters in advance for combined observation. According to the method, a multi-feature fusion recognition mechanism is realized through character color features and facial features. The invention provides a multi-person head posture estimation method based on fuzzy reasoning, and deployment and real-time reasoning can be carried out on a mobile robot platform.

Description

Technical Field

[0001] The present invention relates to the field of indoor robot search, and in particular to a robot indoor person search method and system based on active perception. Background Art

[0002] Current robotic person search solutions, such as the Robotic Person Search Method and System (CN115346256A), primarily achieve target location through the following process: A global path is generated based on a preset fixed patrol route or user-specified historical location information. Human detection, facial recognition, height matching, and clothing color comparison are used to extract features and match identities of people within the scene. A successful match triggers a backend notification mechanism and plans a guidance path to assist the searcher in reaching the target location. This approach has a certain path coverage capability in structured scenarios and supports target tracking and real-time location sharing, reducing the risk of repeated losses.

[0003] However, these existing technologies rely on fixed routes or historical locations to generate paths, failing to dynamically adjust observation strategies based on real-time person distribution and orientation. This results in the robot frequently performing ineffective movements, such as repeatedly scanning empty areas or verified non-target people. This significantly reduces search efficiency, particularly in complex indoor environments. Furthermore, these technologies rely heavily on lighting conditions and unobstructed facial information, making missed detections or misjudgments more likely when the robot is backlit, obscured, or facing sideways or away from the target. Furthermore, the lack of a feature weighting mechanism prevents the robot from prioritizing the verification of highly suspected targets. Summary of the Invention

[0004] The present invention aims to overcome the shortcomings and deficiencies of the prior art and provide a method for indoor specific person search by a robot based on active perception. By actively exploring the scene and selecting observation points, a specific target person can be efficiently searched in a home scene.

[0005] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0006] The robot indoor person search method based on active perception includes the following steps:

[0007] S1. Image Information Analysis: Divide the pre-collected scene grid map into blocks of a set size, using the midpoint of the block edge as the block observation point; control the robot to move to the block observation point closest to it, obtain the RGBD environment image observed by the robot in the current block, extract the detection frames and key points of all characters in the RGBD environment image, and detect the position of the characters in the scene based on the robot's depth camera;

[0008] S2. Irrelevant person filtering: Based on the extracted human key points, the upper body and leg areas are segmented to obtain the character feature vector of each person. Based on the Euclidean distance of the character feature vector, non-target people in the RGBD environment image are eliminated to obtain the target to be identified;

[0009] S3, Multi-person Head Pose Estimation: Based on fuzzy inference method, the head orientation angles of all targets to be identified in the robot's field of view are estimated from the RGB environment image;

[0010] S4. Minimum View Planning: Based on the positions and head orientations of all objects to be identified in the scene, a set of sensor views is determined. This constructs a legal set that includes all combinations of people to be observed. The minimum observation set, consisting of multiple legal sets with similar distances and head orientation angles between people, is obtained, thereby finding the target person with the fewest possible observations.

[0011] S5. Optimal observation point evaluation: Calculate the optimal observation point for each legal set in the minimum observation set through weight evaluation, and obtain the optimal observation point sequence for observing the person in the current block;

[0012] S6. Control the robot to move to the nearest observation point in the optimal observation point sequence to observe the face of the person in the field of view. If the target person is found, the task ends. Otherwise, continue to select and move to the next nearest observation point until the optimal observation point sequence is completed. At this time, move to the next block for observation and return to step S1 for image information analysis until all blocks have been observed.

[0013] Furthermore, in step S1, all people in the RGBD environment image are detected using a human pose estimation method;

[0014] The key points of the character include the nose tip key point, the ear key points, the shoulder key points, the torso key points, the waist key points and the toe key points.

[0015] Furthermore, in step S2, the area enclosed by the trunk key points is defined as the upper body area, and the area connected by the waist key points and the toe key points is defined as the leg area;

[0016] The upper body region and the leg region are color histogram encoded respectively, and corresponding RGB feature vectors and HSV feature vectors are generated for the upper body region and the leg region respectively; the RGB feature vectors and HSV feature vectors corresponding to the upper body region and the leg region are concatenated respectively to obtain the regional feature vectors corresponding to the upper body region and the leg region, and the regional feature vectors corresponding to the upper body region and the leg region are connected to form a character feature vector, and the character feature vector Fcandidate of the character to be matched and the character feature vector Ftarget of the target character are obtained.

[0017] Furthermore, in step S2, the Euclidean distance d between the character feature vector Fcandidate of the person to be matched and the character feature vector Ftarget of the target person is calculated;

[0018] Set the adaptive threshold T = μ + ασ, μ is the average distance of the training set, σ is the standard deviation, and α is an adjustable parameter. If d ≤ T, it is determined to be a successful match, and a matching list including all successfully matched persons to be observed is obtained.

[0019] Furthermore, the S3 includes the following steps:

[0020] S3.1. Parameter calculation: Calculate the following three fuzzy input parameters based on the coordinates of the nose tip keypoint, the ear keypoints, and the shoulder keypoints:

[0021] Nose-to-ear distance ratio (NE): the ratio of the relative distance between the nose tip abscissa and the abscissa of the center of both ears;

[0022] Nose-to-shoulder distance ratio NS: the relative offset ratio between the horizontal coordinate of the nose tip and the horizontal coordinate of the center of the shoulders;

[0023] Ear position relationship RP: the vertical position relationship between the key points of both ears in the image;

[0024] S3.2, Fuzzification and Rule Matching: Map the three parameters of nose-to-ear distance ratio NE, nose-to-shoulder distance ratio NS, and ear position relationship RP to the preset fuzzy sets:

[0025] The fuzzy set of the nose-to-ear distance ratio NE is {inside (I), outside (O)};

[0026] The fuzzy set of nose-shoulder distance ratio NS is {left outer (LO), left (L), middle (M), right (R), right outer (RO)};

[0027] The fuzzy set of ear position relation RP is {upper (U), middle (M), lower (D)};

[0028] Based on a preset fuzzy inference rule base, the head orientation fuzzy mark corresponding to the current parameter combination is matched. The rule base includes 30 rules, each of which defines the mapping relationship between the fuzzy set of nose-to-ear distance ratio (NE), nose-to-shoulder distance ratio (NS), and ear position relationship (RP) and the head orientation fuzzy mark;

[0029] S3.3. Defuzzification and orientation output: The matched fuzzy mark of the head orientation is defuzzified and decomposed into a fuzzy set of X-axis and Y-axis coordinates. The output value of each rule is weighted and averaged according to its membership to obtain the precise coordinate values ​​in the X and Y directions. Finally, the horizontal orientation angle of the person's head is synthesized based on the X and Y coordinate values.

[0030] Furthermore, the S4 includes the following steps:

[0031] S4.1. Candidate set initialization: Model all the people to be observed in the scene as person points with orientation attributes, and construct a candidate set including all combinations of person points;

[0032] S4.2. Legal set screening: Screen the legal sets that meet the following constraints from the candidate sets:

[0033] Single element coverage: Each individual character point can constitute a candidate set;

[0034] Subset closure: Any legal set A in the legal set family S that includes all legal sets is composed of elements in the set U of the person to be observed, that is, for any A∈S, it satisfies

[0035] Angle constraint: The angle between any two character points in the same set does not exceed the preset maximum angle deviation θ_max;

[0036] Distance constraint: The Euclidean distance between any two character points in the same set does not exceed the preset maximum distance d_max;

[0037] S4.3. Greedy strategy set selection: Based on the greedy algorithm, iteratively select the legal set that covers the most unobserved characters in the current legal set until all character points are covered, and generate the minimum observation set with the least number of legal sets.

[0038] Furthermore, the S5 includes the following steps:

[0039] S5.1. Definition of visibility parameters: The omnidirectional angles of all accessible locations in the scene grid map are used as candidate observation poses. For each candidate observation pose, the following visibility judgment parameters are defined:

[0040] Angle matching condition: whether the angle between the robot's orientation and the target person's orientation meets the preset angle range;

[0041] Field of view coverage condition: whether the target person is within the observation field of view of the robot's current posture;

[0042] Distance matching condition: whether the physical distance between the target person and the robot is within the effective ranging range of the sensor;

[0043] Occlusion status: whether there are obstacles in the line of sight from the robot to the target person;

[0044] S5.2. Visibility Determination: Determine whether the target person is visible in the candidate observation pose based on the angle matching conditions, field of view coverage conditions, distance matching conditions, and occlusion status. If the person is visible in the candidate observation pose, the validity weight is increased; if not, the validity weight is not increased.

[0045] S5.3. Calculation of validity weight: For each legal set in the minimum observation set, calculate the corresponding candidate observation pose; for each candidate observation pose, calculate the angle between the orientation vector of each character in the corresponding legal set and the vector from the robot to the character; subtract the Euclidean distance d between the character feature vector Fcandidate of the character to be observed and the character feature vector Ftarget of the target character from the adaptive threshold T to obtain the character weight w; accumulate the product of the character weight w and the angle, and obtain the validity score of the candidate observation pose based on visibility judgment, and select the candidate observation pose with the highest validity score as the optimal observation point.

[0046] The robot indoor person search system based on active perception includes the following modules:

[0047] Image Information Analysis Module: This module divides the pre-collected scene grid map into blocks of a set size, using the midpoint of each block's edge as the block observation point. The robot is controlled to move to the nearest block observation point, acquires the RGBD environment image observed by the robot in the current block, extracts the detection frames and key points of all characters in the RGBD environment image, and detects the position of the characters in the scene using the robot's depth camera.

[0048] Irrelevant person filtering module: Based on the extracted key points of the human body, the upper body and leg areas are segmented to obtain the character feature vector of each person. Based on the Euclidean distance of the character feature vector, non-target people in the RGBD environment image are eliminated to obtain the target to be identified.

[0049] Multi-person head pose estimation module: Based on fuzzy inference, it estimates the head orientation angles of all targets to be identified within the robot's field of view from the RGB environment image;

[0050] Minimum View Planning Module: Based on the positions and head orientations of all objects to be identified in the scene, a set of sensor views is determined, and a legal set of all combinations of people to be observed is constructed. The minimum observation set consisting of multiple legal sets with similar distances and head orientation angles between people is obtained, thereby finding the target person with the fewest possible observations.

[0051] Optimal observation point evaluation module: Calculates the optimal observation point for each legal set in the minimum observation set through weight evaluation, and obtains the optimal observation point sequence for observing the characters in the current block;

[0052] Robot control module: controls the robot to move to the nearest observation point in the optimal observation point sequence to observe the face of the person in the field of view. If the target person is found, the task ends. Otherwise, it continues to select and move to the next nearest observation point until the optimal observation point sequence is completed. At this time, it moves to the next block for observation and calls the image information analysis module until all blocks are observed.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] This invention aims to enable robots to search for specific people in a single room. Based on an active perception strategy, this method analyzes the distribution and orientation of people in the scene in real time, dynamically planning the minimum observation view, effectively avoiding repeated and ineffective searches. This method achieves more efficient person search by preemptively filtering out irrelevant people and jointly observing similar people. This method utilizes color and facial features to implement a multi-feature fusion recognition mechanism. This method also proposes a multi-person head pose estimation method based on fuzzy reasoning, which can be deployed and performed in real time on a mobile robot platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 2 is a schematic diagram of the overall framework of a robot indoor person search system based on active perception in an embodiment of the present invention.

[0056] Figure 2 1 is a membership function diagram of the fuzzy inference method used for multi-person head posture estimation in an embodiment of the present invention.

[0057] Figure 3 This is a flowchart of the steps of the robot indoor person search method based on active perception in an embodiment of the present invention.

[0058] Figure 4 It is a scene map that the robot searches for in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto. In one embodiment, a robot indoor person search method based on active perception is as follows: Figure 1 and Figure 3 As shown, the following steps are included:

[0060] S1. Image information analysis: Figure 4As shown in the figure, the pre-collected scene grid map is divided into 3m*3m blocks, and the midpoints of the four sides of the block are used as block observation points; the robot is controlled to move to the block observation point closest to it, and the RGBD environment image observed by the robot in the current block is obtained. The detection frames and key points of all characters in the RGBD environment image are extracted, and the position of the characters in the scene is detected based on the robot's depth camera;

[0061] In one embodiment, a lightweight human pose estimation model (RTMPose) is used to detect all people in the RGBD environment image using a human pose estimation method;

[0062] In one embodiment, the character key points include nose tip key points, ear key points, shoulder key points, torso key points, waist key points and toe key points.

[0063] In one embodiment, the three-dimensional position of each detected person in the scene grid map is calculated to form a preliminary person coordinate list.

[0064] S2. Irrelevant person filtering: Based on the extracted human key points, the upper body and leg areas are segmented to obtain the character feature vector of each person. Based on the Euclidean distance of the character feature vector, non-target people in the RGBD environment image are eliminated to obtain the target to be identified;

[0065] The area enclosed by the torso key points is defined as the upper body area, and the area connected by the waist key points and the toe key points is defined as the leg area;

[0066] Perform color histogram encoding on the upper body region and the leg region, and generate corresponding RGB feature vectors and HSV feature vectors for the upper body region and the leg region respectively;

[0067] In one embodiment, in RGB space, the regional image is divided into a 6×6×6 feature matrix. In HSV space, a 15×15 feature matrix is ​​constructed. To reduce the influence of light intensity on color feature extraction, the present invention excludes the lightness V channel in the HSV color feature and only uses the HS channel. The formula for constructing the feature matrix is ​​as follows:

[0068]

[0069]

[0070] The RGB feature vectors corresponding to the upper body area and the leg area are concatenated with the HSV feature vectors respectively to obtain the regional feature vectors corresponding to the upper body area and the leg area, and the regional feature vectors corresponding to the upper body area and the leg area are connected to form a character feature vector, and the character feature vector Fcandidate of the character to be matched and the character feature vector Ftarget of the target character are obtained.

[0071] Calculate the Euclidean distance d between the character feature vector Fcandidate of the person to be matched and the character feature vector Ftarget of the target person;

[0072] An adaptive threshold T = μ + ασ is set, where μ is the average distance of the training set, σ is the standard deviation, and α is an adjustable parameter. In one embodiment, μ is calculated using the average Euclidean distance of the person's features calculated from images of the same person at different angles in a public person ReID dataset. σ represents the degree of dispersion of these Euclidean distances. By setting α, the strictness of the match can be adjusted. Increasing it can lower the standard for successful matching, while decreasing it can lower it. If d ≤ T, the match is determined to be successful, and a matching list including all successfully matched persons to be observed is obtained.

[0073] S3. Multi-person head pose estimation:

[0074] The method for head pose estimation based on fuzzy reasoning primarily involves fuzzification, inference rules, and defuzzification. Fuzzification first extracts specific features from the coordinates of key points of a person and converts them into input calculation values. Three membership functions are then set, and the calculated values ​​are fuzzified using these membership functions. The three fuzzy inputs activate corresponding inference rules, which are constructed based on the correspondence between fuzzy inputs and fuzzy outputs. The inference engine then performs a composite operation to generate a fuzzy conclusion. Finally, the defuzzification module generates a linear output of head orientation.

[0075] Based on the fuzzy inference method, the head orientation angles of all the targets to be identified in the robot's field of view are estimated from the RGB environment image, including the following steps:

[0076] S3.1. Parameter calculation: Calculate the following three fuzzy input parameters based on the coordinates of the nose tip keypoint, the ear keypoints, and the shoulder keypoints:

[0077] Nose-to-ear distance ratio (NE): the ratio of the relative distance between the nose tip abscissa and the abscissa of the center of both ears;

[0078] Nose-to-shoulder distance ratio NS: the relative offset ratio between the horizontal coordinate of the nose tip and the horizontal coordinate of the center of the shoulders;

[0079] Ear position relationship RP: the vertical position relationship between the key points of both ears in the image;

[0080] S3.2, Fuzzification and Rule Matching: Map the three parameters of nose-to-ear distance ratio NE, nose-to-shoulder distance ratio NS, and ear position relationship RP to the preset fuzzy sets:

[0081] The fuzzy set of the nose-to-ear distance ratio NE is {inside (I), outside (O)};

[0082] The fuzzy set of nose-shoulder distance ratio NS is {left outer (LO), left (L), middle (M), right (R), right outer (RO)};

[0083] The fuzzy set of ear position relation RP is {upper (U), middle (M), lower (D)};

[0084] Based on a preset fuzzy inference rule base, the head orientation fuzzy mark corresponding to the current parameter combination is matched. The rule base includes 30 rules, each of which defines the mapping relationship between the fuzzy set of nose-to-ear distance ratio (NE), nose-to-shoulder distance ratio (NS), and ear position relationship (RP) and the head orientation fuzzy mark;

[0085] S3.3. Defuzzification and orientation output: The matched fuzzy mark of the head orientation is defuzzified and decomposed into a fuzzy set of X-axis and Y-axis coordinates. The output value of each rule is weighted and averaged according to its membership to obtain the precise coordinate values ​​in the X and Y directions. Finally, the horizontal orientation angle of the person's head is synthesized based on the X and Y coordinate values.

[0086] In one embodiment, in order to convert the key point coordinates into fuzzy input, a fuzzification operation is required. ne ,i ns and i rp are the values ​​of the three fuzzy input parameters: nose-to-ear distance ratio NE, nose-to-shoulder distance ratio NS and ear position relationship RP, respectively. center is the distance from the center of the ears to the center of the shoulders, d xcenter and d ycenter are the horizontal and vertical distances from the center of both ears to the center of both shoulders, (x le ,y le ) is the coordinate of the left ear key point, (x re ,y re ) is the coordinate of the right ear key point, (x ls ,y ls ) is the coordinate of the left shoulder key point, (x rs ,y rs ) is the coordinate of the right shoulder key point, x nose is the horizontal coordinate of the nose tip key point, (x me ,y me ) are the coordinates of the centers of both ears, and are calculated as shown in the following formula:

[0087]

[0088]

[0089] The choice of membership function and inference rules has a significant impact on the performance and complexity of the fuzzy system. Different membership functions will result in different fuzzy values. Figure 2 The function graphs of various membership functions are shown. Considering i ne 、i ns and i rp are continuous and non-jumpy. When they are fuzzified, the trapezoidal function is selected as the membership function, and the trapezoidal distribution function is shown below.

[0090]

[0091] i ne The corresponding membership function is defined by the four nodes of the trapezoidal function, as shown in the following two formulas:

[0092] AI=trapezoid(-1.5,-0.5,0.5,1.5)

[0093] AO=~AI

[0094] i ns The input values ​​are mapped to five fuzzy sets through five membership functions, including left outer and right outer decreasing and increasing functions, and three trapezoidal functions, as shown in the following formulas:

[0095] B.LO=S(-7 / 6,-2 / 3)

[0096] BL=trapezoid(-4 / 3,-5 / 6,-1 / 2,0)

[0097] BM=trapezoid(-2 / 3,-1 / 6,1 / 6,2 / 3)

[0098] BR=trapezoid(0,1 / 2,5 / 6,4 / 3)

[0099] B.RO=R(2 / 3,7 / 6)

[0100] i rp The corresponding membership function is as follows:

[0101] relative_position.D=S(-1,-0.1)

[0102] relative_position.M=trapezoid(-1,-0.1,0.1,1)

[0103] relative_position.U=R(0.1,1)

[0104] Through the membership function, NE is mapped to the fuzzy set {inside, outside}, NS is mapped to {left outside, left, center, right, right outside}, and RP is mapped to {upper, center, lower}.

[0105] According to the 30 fuzzy inference rules constructed in the following table, the input parameters of each candidate target are matched to obtain the corresponding fuzzy output labeling strength.

[0106]

[0107]

[0108] The fuzzy output Orient is decomposed into X and Y using the following decomposition rules. This means that the fuzzy sets on the X and Y axes are defuzzified separately, and the head orientation angle is finally calculated using the inverse tangent function. Furthermore, for circular quantities with different weights, the effects of weighted averaging and vector averaging are different. In the vector method, weights affect not only the scale of the scalar values ​​but also the modulus of the vector. For example, if two vectors with a 180-degree angle between them have the same weight, their average value will be a zero vector with a modulus of 0, indicating that there is no average vector for these two vectors. If a vector has a higher weight, the average vector will shift toward it.

[0109]

[0110] Each rule output is multiplied by its rule activation strength (determined by the minimum value of the input parameter membership), and the X component and Y component of all rules are weighted and summed separately;

[0111] X fuzzy mark Coordinate value Y fuzzy mark Coordinate value L -1.000 D -1.000 ML -0.707 MD -0.707 M 0.000 M 0.000 MR 0.707 MU 0.707 R 1.000 U 1.000

[0112] The inverse tangent function (arctan2) is used to convert the weighted X and Y values ​​into an accurate horizontal head orientation angle. Ultimately, the head orientation information of each candidate target is obtained, providing a basis for subsequent observation point evaluation.

[0113] After determining the fuzzy inference rule to be applied, since the two conditions in the rule are connected by "and", the minimum value method is used to determine the strength of the rule, that is, the smaller membership degree in the condition is selected as the rule output strength. Finally, weighted mean defuzzification is used to obtain the accurate output of the horizontal facial heading angle. x is the output of X fuzzy set, o y is the output of the Y fuzzy set, w i is the output strength of the i-th inference rule, x i is the horizontal coordinate value of the fuzzy mark corresponding to the output, yi is the corresponding vertical coordinate value, and o r The final output head orientation angle is calculated as follows:

[0114]

[0115] S4. Minimum View Planning: Based on the positions and head orientations of all objects to be identified in the scene, a set of sensor views is determined. A legal set is constructed that includes all combinations of objects to be observed. The minimum observation set that includes multiple legal sets with similar distances and head orientation angles between objects is obtained, thereby finding the target object with the fewest possible observations. This process includes the following steps:

[0116] S4.1. Candidate set initialization: Model all the people to be observed in the scene as person points with orientation attributes, and construct a candidate set including all combinations of person points;

[0117] S4.2. Legal set screening: Screen the legal sets that meet the following constraints from the candidate sets:

[0118] Single element coverage: Each individual character point can constitute a candidate set;

[0119] Subset closure: Any legal set A in the legal set family S that includes all legal sets is composed of elements in the set U of the person to be observed, that is, for any A∈S, it satisfies

[0120] Angle constraint: The angle between any two character points in the same set does not exceed the preset maximum angle deviation θ_max;

[0121] Distance constraint: The Euclidean distance between any two character points in the same set does not exceed the preset maximum distance d_max;

[0122] S4.3. Greedy strategy set selection: Based on the greedy algorithm, iteratively select the legal set that covers the most unobserved characters in the current legal set until all character points are covered, and generate the minimum observation set with the least number of legal sets.

[0123] S5. Optimal observation point evaluation: Calculate the optimal observation point for each legal set in the minimum observation set through weight evaluation, and obtain the optimal observation point sequence for observing the person in the current block, including the following steps:

[0124] S5.1. Definition of visibility parameters: The omnidirectional angles of all accessible locations in the scene grid map are used as candidate observation poses. For each candidate observation pose, the following visibility judgment parameters are defined:

[0125] Angle matching condition: whether the angle between the robot's orientation and the target person's orientation meets the preset angle range;

[0126] Field of view coverage condition: whether the target person is within the observation field of view of the robot's current posture;

[0127] Distance matching condition: whether the physical distance between the target person and the robot is within the effective ranging range of the sensor;

[0128] Occlusion status: whether there are obstacles in the line of sight from the robot to the target person;

[0129] S5.2. Visibility Determination: Determine whether the target person is visible in the candidate observation pose based on the angle matching conditions, field of view coverage conditions, distance matching conditions, and occlusion status. If the person is visible in the candidate observation pose, the validity weight is increased; if not, the validity weight is not increased.

[0130] S5.3. Calculation of validity weight: For each legal set in the minimum observation set, calculate the corresponding candidate observation pose; for each candidate observation pose, calculate the angle between the orientation vector of each character in the corresponding legal set and the vector from the robot to the character; subtract the Euclidean distance d between the character feature vector Fcandidate of the character to be observed and the character feature vector Ftarget of the target character from the adaptive threshold T to obtain the character weight w; accumulate the product of the character weight w and the angle, and obtain the validity score of the candidate observation pose based on visibility judgment, and select the candidate observation pose with the highest validity score as the optimal observation point.

[0131] S6. Control the robot to move to the nearest observation point in the optimal observation point sequence to observe the face of the person in the field of view. If the target person is found, the task ends. Otherwise, continue to select and move to the next nearest observation point until the optimal observation point sequence is completed. At this time, move to the next block for observation and return to step S1 for image information analysis until all blocks have been observed.

[0132] The robot indoor person search system based on active perception includes the following modules:

[0133] Image Information Analysis Module: This module divides the pre-collected scene grid map into blocks of a set size, using the midpoint of each block's edge as the block observation point. The robot is controlled to move to the nearest block observation point, acquires the RGBD environment image observed by the robot in the current block, extracts the detection frames and key points of all characters in the RGBD environment image, and detects the position of the characters in the scene using the robot's depth camera.

[0134] Irrelevant person filtering module: Based on the extracted key points of the human body, the upper body and leg areas are segmented to obtain the character feature vector of each person. Based on the Euclidean distance of the character feature vector, non-target people in the RGBD environment image are eliminated to obtain the target to be identified.

[0135] Multi-person head pose estimation module: Based on fuzzy inference, it estimates the head orientation angles of all targets to be identified within the robot's field of view from the RGB environment image;

[0136] Minimum View Planning Module: Based on the positions and head orientations of all objects to be identified in the scene, a set of sensor views is determined, and a legal set of all combinations of people to be observed is constructed. The minimum observation set consisting of multiple legal sets with similar distances and head orientation angles between people is obtained, thereby finding the target person with the fewest possible observations.

[0137] Optimal observation point evaluation module: calculates the optimal observation point for each legal set in the minimum observation set through weight evaluation, and obtains the optimal observation point sequence for observing the characters in the current block;

[0138] Robot control module: controls the robot to move to the nearest observation point in the optimal observation point sequence to observe the face of the person in the field of view. If the target person is found, the task ends. Otherwise, it continues to select and move to the next nearest observation point until the optimal observation point sequence is completed. At this time, it moves to the next block for observation and calls the image information analysis module until all blocks are observed.

[0139] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A robot indoor person search method based on active perception, characterized in that: The following steps are involved: S1. Image Information Analysis: Divide the pre-collected scene grid map into blocks of a set size, using the midpoint of the block edge as the block observation point; control the robot to move to the block observation point closest to it, obtain the RGBD environment image observed by the robot in the current block, extract the detection frames and key points of all characters in the RGBD environment image, and detect the position of the characters in the scene based on the robot's depth camera; S2. Irrelevant person filtering: Based on the extracted human key points, the upper body and leg areas are segmented to obtain the character feature vector of each person. Based on the Euclidean distance of the character feature vector, non-target people in the RGBD environment image are eliminated to obtain the target to be identified; S3, Multi-person Head Pose Estimation: Based on fuzzy inference method, the head orientation angles of all targets to be identified in the robot's field of view are estimated from the RGB environment image; S4. Minimum View Planning: Based on the positions and head orientations of all objects to be identified in the scene, a set of sensor views is determined. This constructs a legal set that includes all combinations of people to be observed. The minimum observation set, consisting of multiple legal sets with similar distances and head orientation angles between people, is obtained, thereby finding the target person with the fewest possible observations. S5. Optimal observation point evaluation: Calculate the optimal observation point for each legal set in the minimum observation set through weight evaluation, and obtain the optimal observation point sequence for observing the person in the current block; S6. Control the robot to move to the nearest observation point in the optimal observation point sequence to observe the face of the person in the field of view. If the target person is found, the task ends. Otherwise, continue to select and move to the next nearest observation point until the optimal observation point sequence is completed. At this time, move to the next block for observation and return to step S1 for image information analysis until all blocks have been observed.

2. The method for indoor person search based on active perception by a robot according to claim 1, characterized in that: In step S1, all people in the RGBD environment image are detected using a human pose estimation method; The key points of the character include the nose tip key point, the ear key points, the shoulder key points, the torso key points, the waist key points and the toe key points.

3. The method for indoor person search based on active perception by a robot according to claim 2, characterized in that: In step S2, the area enclosed by the trunk key points is defined as the upper body area, and the area connected by the waist key points and the toe key points is defined as the leg area.

4. The method for indoor person search based on active perception by a robot according to claim 3, characterized in that: In step S2, color histogram encoding is performed on the upper body area and the leg area respectively, and corresponding RGB feature vectors and HSV feature vectors are generated for the upper body area and the leg area respectively; the RGB feature vectors and HSV feature vectors corresponding to the upper body area and the leg area are respectively concatenated to obtain regional feature vectors corresponding to the upper body area and the leg area, and the regional feature vectors corresponding to the upper body area and the leg area are connected to form a character feature vector, and the character feature vector Fcandidate of the character to be matched and the character feature vector Ftarget of the target character are obtained.

5. The method for indoor person search based on active perception by a robot according to claim 4, characterized in that: In step S2, the Euclidean distance d between the character feature vector Fcandidate of the person to be matched and the character feature vector Ftarget of the target person is calculated; Set the adaptive threshold T = μ + ασ, μ is the average distance of the training set, σ is the standard deviation, and α is an adjustable parameter. If d ≤ T, it is determined to be a successful match, and a matching list including all successfully matched persons to be observed is obtained.

6. The method for indoor person search based on active perception by a robot according to claim 2, characterized in that: The S3 comprises the following steps: S3.

1. Parameter calculation: Calculate the following three fuzzy input parameters based on the coordinates of the nose tip keypoint, the ear keypoints, and the shoulder keypoints: Nose-to-ear distance ratio (NE): the ratio of the relative distance between the nose tip abscissa and the abscissa of the center of both ears; Nose-to-shoulder distance ratio NS: the relative offset ratio between the horizontal coordinate of the nose tip and the horizontal coordinate of the center of the shoulders; Ear position relationship RP: the vertical position relationship between the key points of both ears in the image; S3.2, Fuzzification and Rule Matching: Map the three parameters of nose-to-ear distance ratio NE, nose-to-shoulder distance ratio NS, and ear position relationship RP to the preset fuzzy sets: The fuzzy set of the nose-to-ear distance ratio NE is {inside (I), outside (O)}; The fuzzy set of nose-shoulder distance ratio NS is {left outer (LO), left (L), middle (M), right (R), right outer (RO)}; The fuzzy set of ear position relation RP is {upper (U), middle (M), lower (D)}; Based on a preset fuzzy inference rule base, the head orientation fuzzy mark corresponding to the current parameter combination is matched. The rule base includes 30 rules, each of which defines the mapping relationship between the fuzzy set of nose-to-ear distance ratio (NE), nose-to-shoulder distance ratio (NS), and ear position relationship (RP) and the head orientation fuzzy mark; S3.

3. Defuzzification and orientation output: The matched fuzzy mark of the head orientation is defuzzified and decomposed into a fuzzy set of X-axis and Y-axis coordinates. The output value of each rule is weighted and averaged according to its membership to obtain the precise coordinate values ​​in the X and Y directions. Finally, the horizontal orientation angle of the person's head is synthesized based on the X and Y coordinate values.

7. The robot indoor person search method based on active perception according to claim 1 is characterized in that: Step S4 includes the following steps: S4.

1. Candidate set initialization: Model all the people to be observed in the scene as person points with orientation attributes, and construct a candidate set including all combinations of person points; S4.2, Legal set screening: Screening legal sets that meet the constraints from the candidate sets; S4.

3. Greedy strategy set selection: Based on the greedy algorithm, iteratively select the legal set that covers the most unobserved characters in the current legal set until all character points are covered, and generate the minimum observation set with the least number of legal sets.

8. The method for indoor person search based on active perception by a robot according to claim 7, characterized in that: In step S4.2, the constraints are as follows: Single element coverage: Each individual character point can constitute a candidate set; Subset closure: Any legal set A in the legal set family S that includes all legal sets is composed of elements in the set U of the person to be observed, that is, for any A∈S, it satisfies Angle constraint: The angle between any two character points in the same set does not exceed the preset maximum angle deviation θ_max; Distance constraint: The Euclidean distance between any two character points in the same set does not exceed the preset maximum distance d_max.

9. The robot indoor person search method based on active perception according to claim 1 is characterized in that: The S5 comprises the following steps: S5.

1. Definition of visibility parameters: The omnidirectional angles of all accessible locations in the scene grid map are used as candidate observation poses. For each candidate observation pose, the following visibility judgment parameters are defined: Angle matching condition: whether the angle between the robot's orientation and the target person's orientation meets the preset angle range; Field of view coverage condition: whether the target person is within the observation field of view of the robot's current posture; Distance matching condition: whether the physical distance between the target person and the robot is within the effective ranging range of the sensor; Occlusion status: whether there are obstacles in the line of sight from the robot to the target person; S5.

2. Visibility Determination: Determine whether the target person is visible in the candidate observation pose based on the angle matching conditions, field of view coverage conditions, distance matching conditions, and occlusion status. If the person is visible in the candidate observation pose, the validity weight is increased; if not, the validity weight is not increased. S5.

3. Calculation of validity weight: For each legal set in the minimum observation set, calculate the corresponding candidate observation pose; for each candidate observation pose, calculate the angle between the orientation vector of each character in the corresponding legal set and the vector from the robot to the character, and obtain the character weight w by subtracting the Euclidean distance d between the character feature vector Fcandidate of the character to be observed and the character feature vector Ftarget of the target character from the adaptive threshold T. Accumulate the product of the character weight w and the angle, and obtain the validity score of the candidate observation pose based on visibility judgment. Select the candidate observation pose with the highest validity score as the optimal observation point.

10. The robot indoor person search system based on active perception is characterized by: Includes the following modules: Image Information Analysis Module: This module divides the pre-collected scene grid map into blocks of a set size, using the midpoint of each block's edge as the block observation point. The robot is controlled to move to the nearest block observation point, obtain the RGBD environment image observed by the robot in the current block, extract the detection frames and key points of all characters in the RGBD environment image, and detect the position of the characters in the scene using the robot's depth camera. Irrelevant person filtering module: Based on the extracted key points of the human body, the upper body and leg areas are segmented to obtain the character feature vector of each person. Based on the Euclidean distance of the character feature vector, non-target people in the RGBD environment image are eliminated to obtain the target to be identified. Multi-person head pose estimation module: Based on fuzzy inference, it estimates the head orientation angles of all targets to be identified within the robot's field of view from the RGB environment image; Minimum View Planning Module: Based on the positions and head orientations of all objects to be identified in the scene, a set of sensor views is determined, and a legal set of all combinations of people to be observed is constructed. The minimum observation set consisting of multiple legal sets with similar distances and head orientation angles between people is obtained, thereby finding the target person with the fewest possible observations. Optimal observation point evaluation module: calculates the optimal observation point for each legal set in the minimum observation set through weight evaluation, and obtains the optimal observation point sequence for observing the characters in the current block; Robot control module: controls the robot to move to the nearest observation point in the optimal observation point sequence to observe the face of the person in the field of view. If the target person is found, the task ends. Otherwise, it continues to select and move to the next nearest observation point until the optimal observation point sequence is completed. At this time, it moves to the next block for observation and calls the image information analysis module until all blocks are observed.

Images