Method and system for guiding robot to automatically generate explanation route and explanation word
By acquiring panoramic photos of exhibits using depth cameras, recognizing and vectorizing them, and using large models to generate explanation routes and words, the problem of the tour guide robot's inability to adapt and adjust is solved, achieving efficient and flexible exhibit explanation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-26
- Publication Date
- 2026-03-31
AI Technical Summary
Existing tour guide robots cannot adapt to dynamic changes in the layout of exhibits, resulting in high manual configuration costs, low efficiency, and cumbersome updates.
By acquiring panoramic photos based on depth cameras, exhibits are identified and their 3D coordinates are obtained. Exhibit descriptions are vectorized using multimodal large models and text embedding models. Exhibits are categorized by cosine similarity or Euclidean distance, clustered to form explanation points, and explanation routes and words are generated.
The system enables the tour guide robot to automatically generate explanation routes and descriptions, and can adaptively adjust according to dynamic changes in the layout of exhibits, reducing human intervention and improving efficiency and flexibility.
Smart Images
Figure CN121756338A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot path planning technology, and in particular to a method and system for automatically generating explanation routes and explanation texts for tour guide robots. Background Technology
[0002] Guide robots have been widely used in various settings such as museums, exhibition halls, science and technology centers, and corporate showrooms. Existing guide robot technologies mainly focus on the following aspects: Pre-set routes and narration content: Traditional guided tour robots rely on manual pre-setting of routes and narration points. The narration content is usually pre-recorded in text or audio format and stored in the robot's knowledge base. When the robot reaches a designated point, the corresponding narration content is played. Its content construction and route planning need to be done manually, which is costly, inefficient, and lacks flexibility.
[0003] Navigation based on simple perception: Guided tour robots generally use SLAM (Simultaneous Localization and Mapping) technology for autonomous navigation and obstacle avoidance. They use multi-sensor fusion, such as LiDAR, depth cameras, and ultrasonic sensors, to perceive and locate the environment, and plan the optimal path from the starting point to the destination. However, this navigation function only solves the problem of "how to get there," and does not solve the core problems of "what to say" and "why say it here."
[0004] Semi-automated tour guide systems: Some semi-automated tour guide solutions exist, such as RFID-based automatic sensing explanation systems or QR code scanning explanation systems. While these systems reduce the workload of tour guides, they require the pre-deployment of RFID transmitters or QR code tags in the environment, necessitate venue modifications, and the explanation content and routes remain preset, unable to adapt to dynamic changes in the exhibit layout.
[0005] In summary, existing technical solutions generally share a common drawback: the generation of guided tour routes and explanatory content relies heavily on manual pre-configuration. This not only consumes a significant amount of manpower and time, but also results in a cumbersome and slow-responding update process when changes occur in the exhibition hall layout or exhibits. Summary of the Invention
[0006] The technical problem solved by this invention: This invention provides a method and system for a tour guide robot to automatically generate explanation routes and explanation texts, solving the problem that existing tour guide robots cannot adaptively adjust according to dynamic changes in the layout of exhibits.
[0007] The technical solution adopted by this invention to solve the above-mentioned technical problems is a method for a tour guide robot to automatically generate explanation routes and explanation texts, comprising the following steps: S1. Generate the scanning points of the tour guide robot based on the scene map; S2. The guide robot moves to each scanning point in sequence and obtains a panoramic photo of each scanning point through the depth camera. S3. Segment the image of each exhibit from the panoramic photo and obtain the three-dimensional coordinates of the representative point of each exhibit; S4. Recognize the image of each exhibit, obtain the descriptive text of each exhibit, and quantize it to obtain the descriptive text vector of each exhibit; S5. Classify the descriptive text vectors of all exhibits by using cosine similarity or Euclidean distance to obtain multiple exhibit categories; S6. Cluster the three-dimensional coordinates of all representative points of exhibits in the same exhibit category to obtain the clusters corresponding to each exhibit category; S7. Calculate the centroid of the three-dimensional coordinates of all representative points of exhibits in each cluster, and use them as explanation points; S8. Connect the explanation points using a preset optimization strategy to form an explanation route. The preset optimization strategy includes the shortest explanation route or explanation of similar exhibits. Generate explanation words corresponding to the explanation points based on the description text of all exhibits in the cluster.
[0008] Furthermore, the scanning points are located along the boundaries of the exhibition hall and the perimeter of the display cases, and the distance between adjacent scanning points is a predetermined distance.
[0009] Furthermore, obtaining the three-dimensional coordinates of each exhibit representative point includes: converting the position of the exhibit representative point in the panoramic photo into the three-dimensional coordinates of the exhibit representative point in the scene coordinate system, wherein the exhibit representative point is the center point of the exhibit.
[0010] Furthermore, the image recognition of each exhibit employs a multimodal large model, including GPT-4V or LLaVA, and the vectorization uses a text embedding model, including Sentence-BERT.
[0011] Furthermore, the explanatory words corresponding to the explanatory points are generated based on the descriptive text of all exhibits in the cluster using a large language model, namely GPT-4.
[0012] This invention also provides a system for a tour guide robot to automatically generate explanation routes and explanation texts, realizing the method described above for a tour guide robot to automatically generate explanation routes and explanation texts. The system includes an environment scanning module, an exhibit recognition and positioning module, a semantic understanding and vectorization module, an explanation point planning module, a route integration module, and an explanation text generation module. The environment scanning module is used to generate scanning points for the tour guide robot based on a scene map, and control the tour guide robot to move to each scanning point sequentially, obtaining panoramic photos of each scanning point through a depth camera. The exhibit recognition and positioning module is used to segment the image of each exhibit from the panoramic photos and obtain the three-dimensional coordinates of the representative point of each exhibit. The semantic understanding and vectorization module is used to process the image of each exhibit... The system identifies and quantifies each exhibit to obtain its descriptive text vector. The explanation point planning module categorizes all exhibit descriptive text vectors using cosine similarity or Euclidean distance to obtain multiple exhibit categories. It then clusters the 3D coordinates of representative points within each exhibit category to obtain clusters for each category. The centroid of the 3D coordinates of all representative points in each cluster is calculated as the explanation point location. The route integration module connects the explanation points using a preset optimization strategy to form an explanation route. This preset optimization strategy includes minimizing the explanation route or explaining similar exhibits. The explanation text generation module generates explanation text corresponding to each explanation point based on the descriptive text of all exhibits in the cluster.
[0013] Furthermore, the scanning points are located along the boundaries of the exhibition hall and the perimeter of the display cases, and the distance between adjacent scanning points is a predetermined distance.
[0014] Furthermore, obtaining the three-dimensional coordinates of each exhibit representative point includes: converting the position of the exhibit representative point in the panoramic photo into the three-dimensional coordinates of the exhibit representative point in the scene coordinate system, wherein the exhibit representative point is the center point of the exhibit.
[0015] Furthermore, the image recognition of each exhibit employs a multimodal large model, including GPT-4V or LLaVA; vectorization uses a text embedding model, including Sentence-BERT.
[0016] Furthermore, based on the descriptive text of all exhibits in the cluster, explanatory words corresponding to the explanatory points are generated, using a large language model, namely GPT-4.
[0017] The beneficial effects of this invention are as follows: This invention provides a method and system for automatically generating explanation routes and explanation texts for a tour guide robot. It generates scanning points for the tour guide robot through a scene map, obtains panoramic photos of each scanning point using a depth camera, segments the image of each exhibit from the panoramic photos, and acquires the 3D coordinates of representative points for each exhibit to obtain the current scene layout. It then identifies the image of each exhibit to obtain a descriptive text vector for each exhibit, categorizes all exhibit descriptive text vectors using cosine similarity or Euclidean distance to obtain multiple exhibit categories, clusters the 3D coordinates of representative points for all exhibits within the same category to obtain clusters corresponding to each exhibit category, calculates the centroid of the 3D coordinates of all representative points in each cluster as explanation points, connects these explanation points using a preset optimization strategy to form an explanation route, and generates explanation texts corresponding to the explanation points based on the descriptive texts of all exhibits in the cluster. Thus, by obtaining the explanation route and explanation texts corresponding to the explanation points through the scene layout, this invention solves the problem that existing tour guide robots cannot adaptively adjust to dynamic changes in exhibit layout. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating a method for a tour guide robot to automatically generate tour routes and explanation text, provided by the present invention. Figure 2 This is a schematic diagram of the system structure of a tour guide robot that automatically generates tour routes and explanations, provided by the present invention. Detailed Implementation
[0019] This invention addresses the problem that existing guided tour robots cannot adaptively adjust to dynamic changes in exhibit layout by providing a method for automatically generating explanation routes and narration text for guided tour robots, such as... Figure 1 As shown, it includes the following steps: S1. Generate the scanning points of the tour guide robot based on the scene map.
[0020] Specifically, the scanning points are located along the boundaries of the exhibition hall and the outer perimeter of the display cases, and the distance between adjacent scanning points is a set distance, which can be 3 meters. The scene map has an independent scene coordinate system, and the scanning points are marked by coordinate points in the scene coordinate system.
[0021] S2. The guide robot moves sequentially to each scanning point and obtains a panoramic photo of each scanning point through a depth camera.
[0022] Specifically, the intrinsic and extrinsic parameters of the depth camera, as well as the coordinates of the depth camera during shooting, are all known parameters. Therefore, for the pixels in the panoramic photos taken by the depth camera, they can be converted to scene coordinates.
[0023] S3. Segment the image of each exhibit from the panoramic photo and obtain the three-dimensional coordinates of the representative point of each exhibit.
[0024] Specifically, obtaining the three-dimensional coordinates of each exhibit representative point includes: converting the position of the exhibit representative point in the panoramic photo into the three-dimensional coordinates of the exhibit representative point in the scene coordinate system. The exhibit representative point can be the center point of the exhibit or any point in the specified exhibit.
[0025] S4. Recognize the image of each exhibit, obtain the descriptive text of each exhibit, and quantize it to obtain the descriptive text vector of each exhibit.
[0026] Specifically, image recognition for each exhibit utilizes a multimodal large model, including GPT-4V or LLaVA, while vectorization employs a text embedding model, including Sentence-BERT. The descriptive text vector for each exhibit is stored in a vector database, which also records the corresponding item ID, image data, and 3D coordinates. This vector database is either Milvus or Pinecone. For example, if GPT-4V or LLaVA is used to recognize an exhibit's image, the resulting descriptive text would be "a Qing Dynasty blue-and-white porcelain vase with fish and seaweed patterns." This text would then be converted into a 1024-dimensional descriptive text vector using Sentence-BERT and stored in the vector database.
[0027] S5. Classify the descriptive text vectors of all exhibits using cosine similarity or Euclidean distance to obtain multiple exhibit categories.
[0028] Specifically, it iterates through all the description text vectors of exhibits in the vector database, calculates the cosine similarity or Euclidean distance between all description text vectors, and classifies the exhibits corresponding to description text vectors with cosine similarity or Euclidean distance greater than the corresponding threshold into the same exhibit category.
[0029] S6. Cluster the three-dimensional coordinates of all representative points of exhibits in the same exhibit category to obtain the clusters corresponding to each exhibit category.
[0030] Specifically, the clustering uses the DBSCAN algorithm to obtain one or more spatially neighboring clusters, that is, one or more spatially neighboring clusters after clustering for each exhibit category.
[0031] S7. Calculate the centroid of the three-dimensional coordinates of all representative points of exhibits in each cluster, and use them as explanation points.
[0032] S8. Connect the explanation points using a preset optimization strategy to form an explanation route. The preset optimization strategy includes the shortest explanation route or explanation of similar exhibits. Generate explanation words corresponding to the explanation points based on the description text of all exhibits in the cluster.
[0033] Specifically, the system can obtain the shortest tour route encompassing all tour points and the tour route corresponding to each exhibit category. The generation of tour routes can be transformed into the Traveling Salesman Problem (TSP), which can be solved to obtain the corresponding tour routes. The narration is obtained through a large-scale speech model. For example, if there are two exhibits in a cluster, with corresponding descriptions of "a Qing Dynasty blue and white porcelain vase with fish and seaweed patterns" and "a Qing Dynasty famille rose plate with a floral and bird pattern in the center," the input to the language model (such as GPT-4) would be "Please generate a coherent narration of approximately 60 seconds based on the exhibit information 'a Qing Dynasty blue and white porcelain vase with fish and seaweed patterns' and 'a Qing Dynasty famille rose plate with a floral and bird pattern in the center.'" The narration might read: "In the display case before us are two exquisite Qing Dynasty porcelain pieces. One is a blue-and-white porcelain vase, its body painted with vivid fish and seaweed patterns, symbolizing wealth and abundance; the other is a famille rose plate, its center decorated with soft colors and delicate brushstrokes of flowers and birds. Together, they reflect the superb porcelain-making craftsmanship of the Qing Dynasty." When the guided tour robot reaches a specific point, it pauses for a period of time, during which the corresponding narration plays. Notably, it can also point to the exhibit using its arm, based on the 3D coordinates of the representative point.
[0034] This invention also provides a system for a tour guide robot to automatically generate tour routes and narration texts, realizing the method described above for a tour guide robot to automatically generate tour routes and narration texts, as shown in the system. Figure 2As shown, the system includes an environment scanning module, an exhibit recognition and localization module, a semantic understanding and vectorization module, a narration point planning module, a route integration module, and a narration text generation module. The environment scanning module generates scanning points for the tour guide robot based on a scene map and controls the robot to move sequentially to each scanning point, obtaining panoramic photos of each scanning point through a depth camera. The exhibit recognition and localization module segments the image of each exhibit from the panoramic photos and obtains the 3D coordinates of a representative point for each exhibit. The semantic understanding and vectorization module recognizes the image of each exhibit, obtains the descriptive text for each exhibit, and quantizes it to obtain the descriptive text for each exhibit. The descriptive text vectors are used to classify all exhibit descriptive text vectors using cosine similarity or Euclidean distance to obtain multiple exhibit categories. The 3D coordinates of representative points of all exhibits within the same category are clustered to obtain clusters corresponding to each exhibit category. The centroid of the 3D coordinates of all representative points in each cluster is calculated as the descriptive point location. The route integration module connects the descriptive points using a preset optimization strategy to form a descriptive route. The preset optimization strategy includes minimizing the descriptive route or focusing on similar exhibits. The descriptive text generation module generates descriptive text corresponding to each descriptive point based on the descriptive text of all exhibits in the cluster.
[0035] Specifically, the scanning points are located along the boundaries of the exhibition hall and the outer perimeter of the display cases, with a predetermined distance between adjacent scanning points. Obtaining the 3D coordinates of each exhibit's representative point includes converting the position of the exhibit's representative point in the panoramic photo into its 3D coordinates within the scene coordinate system; the representative point is the exhibit's center point. Image recognition for each exhibit utilizes a multimodal large model, including GPT-4V or LLaVA, and vectorization employs a text embedding model, including Sentence-BERT. Based on the descriptive text of all exhibits in the cluster, explanatory words corresponding to the explanatory points are generated using a language large model, specifically GPT-4.
Claims
1. A method for automatically generating a tour route and a tour word for a tour robot, characterized by, The method comprises the following steps: S1, generating scanning points of the tour guide robot based on a scene map; S2, the tour guide robot moves to each scanning point in turn, and obtains a panoramic photo of each scanning point through a depth camera; S3, an image of each exhibit is segmented from the panoramic photo, and a three-dimensional coordinate of each exhibit representative point is obtained; S4, the image of each exhibit is identified to obtain a description text of each exhibit, and vectorization is performed to obtain a description text vector of each exhibit; S5, the description text vectors of all exhibits are classified by cosine similarity or Euclidean distance to obtain multiple exhibit categories; S6, the three-dimensional coordinates of all exhibit representative points in the same exhibit category are clustered to obtain a clustered cluster corresponding to each exhibit category; S7, the centroid of the three-dimensional coordinates of all exhibit representative points in each cluster is calculated as a guide point; S8, the guide points are connected by a preset optimization strategy to form a guide route, and the preset optimization strategy includes a shortest guide route or a guide of the same type of exhibits. The guide words corresponding to the guide points are generated based on the description texts of all exhibits in the cluster.
2. The method of automatically generating a guide route and guide words for a guide robot according to claim 1, wherein, The scanning points are along the exhibition hall boundary and the exhibition cabinet periphery, and the distance between adjacent scanning points is a set distance.
3. The method of automatically generating a guide route and guide words for a guide robot according to claim 1, wherein, Obtaining the three-dimensional coordinates of each exhibit representative point includes converting the position of the exhibit representative point in the panoramic photo to a three-dimensional coordinate of the exhibit representative point in the scene coordinate system, and the exhibit representative point is the exhibit center point.
4. The method of automatically generating a guide route and guide words for a guide robot according to claim 1, wherein, The image of each exhibit is identified by a multi-modal large model, the multi-modal large model includes GPT-4V or LLaVA, and the vectorization is performed by a text embedding model, and the text embedding model includes Sentence-BERT.
5. The method of automatically generating a guide route and guide words for a guide robot according to claim 1, wherein, The language large model GPT-4 is used to generate the guide words corresponding to the guide points based on the description texts of all exhibits in the cluster.
6. A system for automatically generating a tour route and a tour word for a tour robot, characterized by, The system for automatically generating a guide route and guide words of the tour guide robot according to claim 1 comprises an environment scanning module, an exhibit identification and positioning module, a semantic understanding and vectorization module, a guide point planning module, a route integration module, and a guide word generation module; the environment scanning module is used to generate scanning points of the tour guide robot based on a scene map, and control the tour guide robot to move to each scanning point in turn to obtain a panoramic photo of each scanning point through a depth camera; the exhibit identification and positioning module is used to segment an image of each exhibit from the panoramic photo, and obtain a three-dimensional coordinate of each exhibit representative point; the semantic understanding and vectorization module is used to identify the image of each exhibit to obtain a description text of each exhibit, and perform vectorization to obtain a description text vector of each exhibit; the guide point planning module is used to classify the description text vectors of all exhibits by cosine similarity or Euclidean distance to obtain multiple exhibit categories, cluster the three-dimensional coordinates of all exhibit representative points in the same exhibit category to obtain a clustered cluster corresponding to each exhibit category, and calculate the centroid of the three-dimensional coordinates of all exhibit representative points in each cluster as a guide point; The route integration module is configured to connect the explanation point positions according to a preset optimization strategy to form an explanation route, and the preset optimization strategy includes a shortest explanation route or an explanation of the same type of exhibits; and the explanation word generation module is configured to generate an explanation word corresponding to the explanation point based on the description text of all the exhibits in the cluster.
7. The system for automatically generating a guide route and guide words of a guide robot according to claim 6, wherein, The scanning point positions are along the exhibition hall boundary and the exhibition cabinet periphery, and the distance between adjacent scanning point positions is a set distance.
8. The system for automatically generating a guide route and guide words of a guide robot according to claim 6, wherein, The three-dimensional coordinates of each exhibit representative point are obtained by converting the position of the exhibit representative point in the panoramic photo into the three-dimensional coordinates of the exhibit representative point in the scene coordinate system, and the exhibit representative point is an exhibit center point.
9. The system for automatically generating a guide route and guide words of a guide robot according to claim 6, wherein, The image of each exhibit is identified by using a multi-modal large model, the multi-modal large model includes GPT-4V or LLaVA, and vectorization is performed by using a text embedding model, and the text embedding model includes Sentence-BERT.
10. The system for automatically generating a guide route and guide words for a guide robot according to claim 6, wherein, The language large model GPT-4 is used to generate an explanation word corresponding to the explanation point based on the description text of all the exhibits in the cluster.