Path planning method and device based on face gaze recognition, and storage medium

Abstract

The application relates to the field of path planning, and discloses a path planning method and device based on face gaze recognition and a storage medium. The method comprises the following steps: collecting the human posture and face features of a target object by using a preset camera; performing gaze recognition processing on the human posture and face features according to a preset gaze judgment algorithm to generate a gazing person; performing coordinate labeling processing on the gazing person based on a preset camera coordinate system to generate an interest heat map; selecting a nearest interest point and a farthest interest point in the interest heat map; generating a temporary departure path according to the nearest interest point and current position data; performing extension processing on the end point of the temporary departure path to generate a temporary cruising path; and combining the temporary departure path, the temporary cruising path and a temporary return path to obtain a temporary path planning. In the embodiment of the application, the cruising adaptability of a robot to user interest is improved, and the ability of the robot to autonomously re-plan a path is improved.

Description

Technical Field

[0001] This invention relates to the field of path planning, and more particularly to a path planning method, device, and storage medium based on facial gaze recognition. Background Technology

[0002] With the rise of smart retail and unmanned supermarkets, service robots, such as intelligent shopping guide robots, automatic inventory robots, and logistics delivery robots, are increasingly being used in retail scenarios. These robots need to have the ability to navigate and move autonomously in complex indoor environments, and path planning, as the core technology of autonomous mobile robots, directly determines their work efficiency, safety, and the customer interaction experience.

[0003] Existing retail robot path planning technologies primarily rely on LiDAR or visual sensors to construct environmental maps and employ hierarchical planning strategies. Global planning is typically based on static grid maps, using A* or Dijkstra's algorithms to find the theoretical shortest path; local planning often uses Dynamic Window (DWA) or artificial potential field methods for real-time obstacle avoidance. However, existing technologies suffer from path fixation, meaning robots can only patrol along pre-set fixed paths and cannot adjust their path movement according to users' actual needs and interests. Therefore, a new technology is needed to address the current limitations of fixed paths in retail robots, which prevent them from adjusting patrol paths based on user interests. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem that current retail robots have fixed paths and cannot adjust their patrols based on user interests.

[0005] The first aspect of this invention provides a path planning method based on face gaze recognition, comprising the following steps:

[0006] Use a preset camera to capture the human posture and facial features of the target object;

[0007] According to a preset gaze detection algorithm, the human posture and facial features will be processed for gaze recognition to generate a gazer;

[0008] Based on a preset camera coordinate system, coordinate annotation is performed on the gazer to generate an interest heatmap;

[0009] Based on the current location data and the maximum temporary planning radius, the nearest and farthest points of interest are selected in the interest heatmap.

[0010] Based on the nearest point of interest and the current location data, a temporary departure path is generated;

[0011] Based on the direction of the preset cruise path and the farthest point of interest, the endpoint of the temporary departure path is extended to generate a temporary cruise path.

[0012] Based on the endpoint of the temporary cruise path and the cruise path, a temporary return path is generated;

[0013] The temporary departure path, the temporary cruise path, and the temporary return path are combined and processed to obtain a temporary path plan.

[0014] Optionally, in a first implementation of the first aspect of the present invention, the step of performing gaze recognition processing on the human posture and the facial features according to a preset gaze judgment algorithm to generate a gazer includes:

[0015] The relative pose between the face and head / shoulders of the target object is detected to obtain the gaze ray confidence score;

[0016] When the confidence level of the gaze ray is greater than the preset gaze threshold, the eye orientation corresponding to the facial features in the target object is detected.

[0017] Calculate the angle between the eye orientation and the line of sight of the current position data;

[0018] When the angle of gaze is less than a preset threshold, the target object is identified as the person looking at it.

[0019] Optionally, in a second implementation of the first aspect of the present invention, the step of generating an interest heatmap by performing coordinate annotation processing on the person gazing based on a preset camera coordinate system includes:

[0020] An image grid is constructed based on a preset camera coordinate system;

[0021] Analyze the intersection of the gazer's points on the image grid to generate the gazer's point coordinates;

[0022] An interest heatmap is generated based on all the landing point coordinates.

[0023] Optionally, in a third implementation of the first aspect of the present invention, the step of selecting the nearest and farthest points of interest in the heatmap based on the current location data and the maximum temporary planning radius includes:

[0024] Using the current location data as the origin and the maximum temporary planning radius as the radius, data points are filtered out from the interest heatmap to obtain a set of interest points;

[0025] Calculate the distance between each point of interest in the point of interest set and the current location data, and generate a distance set;

[0026] The point of interest corresponding to the maximum value in the distance set is determined as the farthest point of interest, and the point of interest corresponding to the minimum value in the distance set is determined as the nearest point of interest.

[0027] Optionally, in a fourth implementation of the first aspect of the present invention, the step of generating a temporary departure path based on the nearest point of interest and the current location data includes:

[0028] A first annular circle is generated with a radius of a preset safe distance and the nearest point of interest as the origin;

[0029] A temporary departure path is generated by smoothly connecting the current location data to the first annular circle.

[0030] Optionally, in a fifth implementation of the first aspect of the present invention, the step of extending the endpoint of the temporary departure path based on the direction of the preset cruise path and the farthest point of interest to generate a temporary cruise path includes:

[0031] A second annular circle is generated with a radius of a preset safe distance and the farthest point of interest as the origin;

[0032] A temporary cruise path is generated by smoothly connecting the end of the temporary departure path to the second annular circle in the direction of the preset cruise path.

[0033] Optionally, in a sixth implementation of the first aspect of the present invention, the step of generating a temporary return path based on the endpoint of the temporary cruise path and the cruise path further includes:

[0034] Calculate the shortest path between the end point of the temporary cruise path and the cruise path to obtain the return point in the cruise path;

[0035] A temporary return route is generated by smoothly connecting the end point of the temporary cruise route to the return point.

[0036] Optionally, in the seventh implementation of the first aspect of the present invention, after the step of performing coordinate annotation processing on the person gazing based on a preset camera coordinate system to generate an interest heatmap, the following is included:

[0037] Based on a preset decay time, the interest points in the interest heatmap are removed to generate a new interest heatmap.

[0038] A second aspect of the present invention provides a path planning device based on face gaze recognition, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the path planning device based on face gaze recognition to execute the aforementioned path planning method based on face gaze recognition.

[0039] A third aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described path planning method based on face gaze recognition.

[0040] In this embodiment of the invention, the robot captures the human posture and facial features of the target object through a camera, analyzes the gaze of the target object, identifies the person gazing, and marks the person gazing on an interest heatmap. Based on the markings on the interest heatmap, the robot temporarily plans its patrol path based on the points of interest in the interest heatmap, generating a temporary path plan. This allows the robot to meet the user's gaze interest needs through the temporary path plan, improving the robot's patrol adaptability to user interests and enhancing the robot's ability to autonomously replan its path. This solves the technical problem of current retail robots having fixed paths and being unable to adjust their patrols based on user interests. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of an embodiment of the path planning method based on face gaze recognition in this invention.

[0042] Figure 2 This is a schematic diagram of a specific embodiment of the 102 steps of the path planning method based on face gaze recognition in this invention.

[0043] Figure 3 This is a schematic diagram of a specific embodiment of the path planning method based on face gaze recognition in this invention.

[0044] Figure 4 This is a schematic diagram of a path planning method based on face gaze recognition in an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of an embodiment of the path planning device based on face gaze recognition in this invention. Detailed Implementation

[0046] This invention provides a path planning method, device, and storage medium based on face gaze recognition.

[0047] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0048] In the description of the embodiments disclosed in this invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0049] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the path planning method based on face gaze recognition in this invention includes:

[0050] 101. Use a preset camera to capture the human posture and facial features of the target object;

[0051] In this embodiment, a camera is used to monitor facial and human features in real time. A neural network algorithm can be used to analyze the skeletal points of the human posture and the YOLO algorithm can be used to frame the face, thereby extracting human posture data and facial features respectively.

[0052] 102. Based on a preset gaze judgment algorithm, gaze recognition processing is performed on the human posture and facial features to generate a gazer;

[0053] In this embodiment, a gaze detection algorithm is used to analyze the gaze ray based on human posture and facial features. The angle between the eye orientation and the robot's position direction is analyzed, and the angle analysis determines whether the target object is the person who is looking at the robot, thus identifying the person who is looking at the robot.

[0054] For details, please refer to Figure 2 , Figure 2 This is a schematic diagram of a specific embodiment of step 102 of the path planning method based on face gaze recognition in this invention. Step 102 includes the following specific implementation methods:

[0055] 1021. Detect the relative pose between the face and head / shoulders of the target object to obtain the gaze ray confidence score;

[0056] 1022. When the confidence level of the gaze ray is greater than the preset gaze threshold, the eye orientation corresponding to the facial features in the target object is detected.

[0057] 1023. Calculate the angle between the eye orientation and the line of sight of the current position data;

[0058] 1024. When the angle of gaze is less than a preset threshold, the target object is identified as the person looking at it.

[0059] In steps 1021-1024, the relative posture between the face and head and shoulders of the target object is first analyzed, and the gaze direction is estimated by fusing eye region features. The gaze event is filtered by a time window (e.g., 0.5–2s) and its stability is determined. The gaze ray and confidence score are output, and the gaze ray confidence score is obtained.

[0060] When the confidence level of the gaze ray is greater than a preset gaze threshold, the eye orientation corresponding to the facial features of the target object is detected based on facial key points, and the gaze angle between the gaze direction and the robot's position direction is calculated. When the gaze angle is less than a preset recognition threshold, the target object is determined to be gazing at the robot.

[0061] 103. Based on a preset camera coordinate system, perform coordinate annotation processing on the person gazing and generate an interest heatmap;

[0062] In this embodiment, the relative position and orientation of the camera are fixed with the vehicle body. The image captured by the camera is divided into a planar grid. The position information of the gazer's landing point is calculated on the planar grid. The landing point information is updated to the structured data of the corresponding point in the interest intensity map and marked on the environmental map to generate an interest heatmap.

[0063] Specifically, step 103 includes the following specific implementation methods:

[0064] 1031. Construct an image grid based on a preset camera coordinate system;

[0065] 1032. Analyze the intersection of the gazer's points on the image grid, and generate the gazer's point coordinates;

[0066] 1033. Generate an interest heatmap based on all the landing point coordinates.

[0067] In steps 1031-1034, an image grid is first constructed in the camera coordinate system. The intersections of the gazer's points on the image grid are analyzed to generate the gazer's point coordinates. An interest heatmap is then generated on a pre-set environmental map for all analyzed point coordinates. When no valid gaze is observed, the system cruises along a preset path. When a new gaze observation appears, the system smoothly switches to a more rewarding target or returns to the fixed path to continue cruise.

[0068] Furthermore, the following specific implementation method is included after step 103:

[0069] 1034. Based on the preset decay time, remove the interest points from the interest heatmap to generate a new interest heatmap.

[0070] In step 1034, a timer is set for the point of interest of each person looking at the object of interest, with a decay time of 5 minutes. Points of interest that have been decayed for more than 5 minutes are removed from the heatmap of interest and a new heatmap of interest is generated.

[0071] 104. Based on the current location data and the maximum temporary planning radius, select the nearest and farthest points of interest in the interest heatmap;

[0072] In this embodiment, the current machine's location data and the set maximum radius of distance from the robot are used to select the nearest and farthest points of interest from the heatmap of interest.

[0073] Specifically, step 104 includes the following specific implementation methods:

[0074] 1041. Using the current location data as the origin and the maximum temporary planning radius as the radius, select data points from the interest heatmap to obtain a set of interest points;

[0075] 1042. Calculate the distance between each point of interest in the point of interest set and the current location data, and generate a distance set;

[0076] 1043. The point of interest corresponding to the maximum value in the distance set is determined as the farthest point of interest, and the point of interest corresponding to the minimum value in the distance set is determined as the nearest point of interest.

[0077] In steps 1041-1043, the current location data is used as the origin and the maximum temporary planning radius is used as the radius. Data points falling within this range are calculated from the interest heatmap to generate an interest point set.

[0078] Calculate the Euclidean distance between the coordinates of each point of interest in the point of interest set and the current location data in turn to obtain a distance set. Compare the values ​​in the distance set and select the maximum and minimum values. The data point corresponding to the minimum value is determined as the nearest point of interest, and the data point corresponding to the maximum value is determined as the farthest point of interest.

[0079] 105. Generate a temporary departure path based on the nearest point of interest and the current location data;

[0080] In this embodiment, a Bezier curve can be used to smoothly connect the current location data to the nearest point of interest, generating a temporary departure path.

[0081] For details, please refer to Figure 3 , Figure 3 This is a schematic diagram of a specific embodiment of the path planning method based on face gaze recognition in this invention, which includes the following specific implementation methods in steps 105:

[0082] 1051. Generate a first annular circle with a radius of a preset safety distance and the nearest point of interest as the origin;

[0083] 1052. Smoothly connect the current location data to the first annular circle to generate a temporary departure path.

[0084] In steps 1051-1052, the radius of the safety distance is set to 1.5m, and a first annular circle is drawn with the nearest point of interest as the origin. Based on Bézier curves, the current position data is smoothly connected to the first annular circle to generate a temporary departure path.

[0085] 106. Based on the direction of the preset cruise path and the farthest point of interest, extend and expand the endpoint of the temporary departure path to generate a temporary cruise path.

[0086] In this embodiment, the end point of the temporary departure path is used as the starting point, and the direction of the cruise path is set to connect parallel to the farthest point of interest to generate a temporary cruise path.

[0087] You can refer to this. Figure 4 , Figure 4 This is a schematic diagram of a path planning method based on face gaze recognition in an embodiment of the present invention. The cruising path is represented by blue, the green nodes represent the current position data of the robot body, and the red nodes represent the gaze object. The farthest point of interest and the farthest point of interest are analyzed. The robot deviates from the cruising path, and yellow paths (temporary departure path, temporary cruising path, and temporary return path) are calculated to generate a temporary path plan.

[0088] Specifically, the 106 steps include the following specific implementation methods:

[0089] 1061. Using a radius of a preset safe distance and the farthest point of interest as the origin, generate a second annular circle;

[0090] 1062. A temporary cruise path is generated by smoothly connecting the end point of the temporary departure path to the second annular circle in the direction of the preset cruise path.

[0091] In steps 1061-1062, a second annular circle is drawn with a radius of 1.5m for the safety distance and the nearest point of interest as the origin. A temporary cruise path is then generated by smoothly connecting the end of the temporary departure path to the second annular circle using a Bézier curve.

[0092] 107. Based on the endpoint of the temporary cruise path and the cruise path, generate a temporary return path;

[0093] In this embodiment, the endpoint of the temporary cruise path is smoothly connected to the nodes on the cruise path to generate a temporary return path, thus completing the final calculation of the temporary cruise.

[0094] Specifically, step 107 includes the following specific implementation methods:

[0095] 1071. Calculate the shortest path between the end point of the temporary cruise path and the cruise path to obtain the return point in the cruise path;

[0096] 1072. Smoothly connect the return point from the end of the temporary cruise path to generate a temporary return path.

[0097] In steps 1071-1072, the return point in the cruise path is obtained based on the node whose nearest path to the end of the temporary cruise path is perpendicular to the cruise path. The return points in the cruise path are then smoothly connected to the end of the temporary cruise path to generate the temporary return path.

[0098] 108. Combine the temporary departure path, the temporary cruise path, and the temporary return path to obtain a temporary path plan.

[0099] In this embodiment, a temporary path is dynamically replanned based on the interest map. A combination of a temporary departure path, a temporary cruise path, and a temporary return path is established on the original fixed path to obtain a temporary path plan, which can cover and expose potential pedestrians within a safe distance.

[0100] The robot cruises based on temporary path planning. If it encounters an obstacle, it avoids it. If a user stops to watch or operate the robot, it pauses its movement to allow the user to make a purchase. After the purchase is completed, the robot continues to run along the temporary path until it returns to the original cruise path.

[0101] In this embodiment of the invention, the robot captures the human posture and facial features of the target object through a camera, analyzes the gaze of the target object, identifies the person gazing, and marks the person gazing on an interest heatmap. Based on the markings on the interest heatmap, the robot temporarily plans its patrol path based on the points of interest in the interest heatmap, generating a temporary path plan. This allows the robot to meet the user's gaze interest needs through the temporary path plan, improving the robot's patrol adaptability to user interests and enhancing the robot's ability to autonomously replan its path. This solves the technical problem of current retail robots having fixed paths and being unable to adjust their patrols based on user interests.

[0102] Figure 5This is a schematic diagram of a path planning device based on face gaze recognition provided in an embodiment of the present invention. The path planning device 500 based on face gaze recognition can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 510 and memory 520, and one or more storage media 530 for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the path planning device 500 based on face gaze recognition. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the path planning device 500 based on face gaze recognition.

[0103] The path planning device 500 based on facial gaze recognition may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, Free BSD, etc. Those skilled in the art will understand that... Figure 5 The illustrated path planning device structure based on face gaze recognition does not constitute a limitation on path planning devices based on face gaze recognition. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0104] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the path planning method based on face gaze recognition.

[0105] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0107] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A path planning method based on face gaze recognition, characterized in that, Including the following steps: Use a preset camera to capture the human posture and facial features of the target object; According to a preset gaze detection algorithm, the human posture and facial features will be processed for gaze recognition to generate a gazer; Based on a preset camera coordinate system, coordinate annotation is performed on the gazer to generate an interest heatmap; Based on the current location data and the maximum temporary planning radius, the nearest and farthest points of interest are selected in the interest heatmap. Based on the nearest point of interest and the current location data, a temporary departure path is generated; Based on the direction of the preset cruise path and the farthest point of interest, the endpoint of the temporary departure path is extended to generate a temporary cruise path. Based on the endpoint of the temporary cruise path and the cruise path, a temporary return path is generated; The temporary departure path, the temporary cruise path, and the temporary return path are combined and processed to obtain a temporary path plan; The step of selecting the nearest and farthest points of interest in the heatmap based on current location data and the maximum temporary planning radius includes: Using the current location data as the origin and the maximum temporary planning radius as the radius, data points are filtered out from the interest heatmap to obtain a set of interest points; Calculate the distance between each point of interest in the point of interest set and the current location data, and generate a distance set; The point of interest corresponding to the maximum value in the distance set is determined as the farthest point of interest, and the point of interest corresponding to the minimum value in the distance set is determined as the nearest point of interest.

2. The path planning method based on face gaze recognition according to claim 1, characterized in that, The step of performing gaze recognition processing on the human posture and facial features according to a preset gaze judgment algorithm to generate a gazer includes: The relative pose between the face and head / shoulders of the target object is detected to obtain the gaze ray confidence score; When the confidence level of the gaze ray is greater than the preset gaze threshold, the eye orientation corresponding to the facial features in the target object is detected. Calculate the angle between the eye orientation and the line of sight of the current position data; When the angle of gaze is less than a preset threshold, the target object is identified as the person looking at it.

3. The path planning method based on face gaze recognition according to claim 1, characterized in that, The step of generating an interest heatmap by performing coordinate annotation on the person gazing based on a preset camera coordinate system includes: An image grid is constructed based on a preset camera coordinate system; Analyze the intersection of the gazer's points on the image grid to generate the gazer's point coordinates; An interest heatmap is generated based on all the landing point coordinates.

4. The path planning method based on face gaze recognition according to claim 1, characterized in that, The step of generating a temporary departure path based on the nearest point of interest and the current location data includes: A first annular circle is generated with a radius of a preset safe distance and the nearest point of interest as the origin; A temporary departure path is generated by smoothly connecting the current location data to the first annular circle.

5. The path planning method based on face gaze recognition according to claim 1, characterized in that, The step of extending and expanding the endpoint of the temporary departure path based on the direction of the preset cruise path and the farthest point of interest to generate a temporary cruise path includes: A second annular circle is generated with a radius of a preset safe distance and the farthest point of interest as the origin; A temporary cruise path is generated by smoothly connecting the end of the temporary departure path to the second annular circle in the direction of the preset cruise path.

6. The path planning method based on face gaze recognition according to claim 1, characterized in that, The step of generating a temporary return route based on the destination of the temporary cruise route and the cruise route further includes: Calculate the shortest path between the end point of the temporary cruise path and the cruise path to obtain the return point in the cruise path; A temporary return route is generated by smoothly connecting the end point of the temporary cruise route to the return point.

7. The path planning method based on face gaze recognition according to claim 1, characterized in that, After the step of generating an interest heatmap by performing coordinate annotation on the person gazing based on a preset camera coordinate system, the following steps are included: Based on a preset decay time, the interest points in the interest heatmap are removed to generate a new interest heatmap.

8. A path planning device based on face gaze recognition, characterized in that, The path planning device based on face gaze recognition includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the path planning device based on face gaze recognition to perform the path planning method based on face gaze recognition as described in any one of claims 1-7.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the path planning method based on face gaze recognition as described in any one of claims 1-7.

Images