Robot control method and device, electronic equipment and storage medium

Abstract

The application provides a robot control method and device, electronic equipment and storage medium; comprising: in response to a control instruction for a robot, acquiring a local V-Map of a current position of the robot, a panoramic map of the current position and a pre-constructed local semantic map of the current position; based on the local V-Map and the panoramic map, determining a feasible area of the robot at the current position and an area description of the feasible area; based on the local semantic map and the area description, determining a feasible path of the robot at the current position and a path description of the feasible path; based on the control instruction, the path description and the local V-Map, determining a path score of the feasible path; based on the path score, determining a target navigation point of the robot, and controlling the robot to move to the target navigation point. Through the application, the intelligence level and control efficiency of the robot can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to a robot control method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence technology, robot technology has become mature, and the types of robots are increasing, including industrial robots, household robots, military robots, etc. Various industries are constantly developing robots suitable for their own industries, hoping that robots can replace manual work to complete repetitive, tedious and dangerous operations. However, during the movement of the robot, the navigation point of the robot needs to be accurately determined to achieve control of the robot.

[0003] In related technologies, the navigation point of the robot is usually set by the user to a specific coordinate point, or the robot needs to pre-scan the entire environment to construct a map of the entire environment to determine the navigation point and movement path of the robot. However, pre-construction of a map of the entire environment by the robot has a low degree of intelligence, which reduces the control efficiency of the robot. SUMMARY

[0004] The embodiments of the present application provide a robot control method, device, electronic equipment and storage medium, which determine the target navigation point of the robot by obtaining the local map of the current position of the robot, thereby improving the intelligence and control efficiency of the robot.

[0005] The technical solutions of the embodiments of the present application are as follows:

[0006] The embodiments of the present application provide a robot control method, which comprises: in response to a control instruction for a robot, obtaining a local Vinho map of a current position of the robot, a panoramic map of the current position, and a pre-constructed local semantic map of the current position; based on the local Vinho map and the panoramic map, determining a feasible region of the robot at the current position and a region description of the feasible region; based on the local semantic map and the region description, determining a feasible path of the robot at the current position and a path description of the feasible path; based on the control instruction, the path description and the local Vinho map, determining a path score of the feasible path; based on the path score, determining a target navigation point of the robot, and controlling the robot to move to the target navigation point.

[0007] The embodiment of the present application provides a robot control device, comprising: an acquisition module configured to acquire a local Vino map of a current position of a robot, a panoramic map of the current position of the robot and a pre-constructed local semantic map of the current position of the robot in response to a control instruction for the robot; a first determination module configured to determine a feasible area of the robot at the current position and an area description of the feasible area based on the local Vino map and the panoramic map; a second determination module configured to determine a feasible path of the robot at the current position and a path description of the feasible path based on the local semantic map and the area description; a third determination module configured to determine a path score of the feasible path based on the control instruction, the path description and the local Vino map; and a control module configured to determine a target navigation point of the robot based on the path score and control the robot to move to the target navigation point.

[0008] In the above scheme, the first determination module is further configured to: construct a coordinate system with the current position of the robot as a coordinate origin; perform map matching on the local Vino map and the panoramic map based on the coordinate system to obtain the feasible area of the robot at the current position; and generate the area description of the feasible area through a pre-constructed first language model.

[0009] In the above scheme, the first determination module is further configured to: determine coordinate values of each point in the panoramic map and the local Vino map based on the coordinate system; perform point-by-point matching on the panoramic map and the local Vino map based on the coordinate values to obtain a matching area of the panoramic map and the local Vino map; and determine the feasible area of the robot at the current position from the matching area.

[0010] In the above scheme, the local Vino map comprises a plurality of paths; the second determination module is further configured to: determine a first description text of each path in the local Vino map through the local semantic map; the first description text comprises objects included in each path and coordinate values of each object; perform text splicing on the first description text and the area description to obtain a first prompt word; and determine the feasible path of the robot at the current position and the path description of the feasible path from the plurality of paths based on the first prompt word through a pre-constructed second language model.

[0011] In the above scheme, the third determining module is further configured to: perform text conversion on the control instruction to obtain a second description text; determine a second prompt word based on the path description and the second description text; determine a text similarity between the path description and the second description text based on the second prompt word through a third language model constructed in advance; and determine a path score of the feasible path based on the local Veno graph and the text similarity.

[0012] In the above scheme, the feasible path includes N nodes; the third determining module is further configured to: determine an access state of each node in the feasible path based on the local Veno graph; the access state includes a node that has been accessed and a node that has not been accessed; score the corresponding node based on the access state of each node to obtain a score value of each node; perform weighted summation on the score values of all nodes in the feasible path to obtain a comprehensive score of the feasible path; and determine a product of the text similarity and the comprehensive score as the path score of the feasible path.

[0013] In the above scheme, the third determining module is further configured to: determine a feasible path with the highest path score as a target path; and determine a node in the target path that is closest to the current position as the target navigation point.

[0014] In the above scheme, the device further includes an updating module configured to: update an access state of the target navigation point when the robot moves to the target navigation point; and update the local Veno graph based on the updated access state of the target navigation point.

[0015] In the above scheme, the obtaining module is further configured to: obtain position data of the current position of the robot and environment perception data of the current position; determine a local grid map of the current position of the robot through a grid map generation tool based on the position data and the environment perception data; and convert the local grid map into a local Veno graph of the current position of the robot.

[0016] In the above scheme, the obtaining module is further configured to: pre-process the local grid map to obtain a pre-processed grid map; perform skeletonization processing on the pre-processed grid map to obtain an initial Veno graph; determine a coordinate value of each path intersection point in the initial Veno graph and a coordinate value of the current position of the robot; and determine the local Veno graph of the current position of the robot based on the initial Veno graph, the coordinate value of the path intersection point, and the coordinate value of the current position.

[0017] In the scheme, the acquisition module is further configured to acquire scene images collected by the robot after rotating at different rotation angles at the current position; perform image stitching on all the scene images according to the rotation angle corresponding to each scene image by using a preset image stitching algorithm to obtain a stitched image; and perform image fusion on the image overlapping area in the stitched image to obtain a panoramic map of the current position of the robot.

[0018] The electronic device provided in the embodiments of the present application includes a memory configured to store computer executable instructions, and a processor configured to execute the computer executable instructions stored in the memory to implement the robot control method provided in the embodiments of the present application.

[0019] The computer readable storage medium provided in the embodiments of the present application stores computer programs or computer executable instructions, and the computer programs or computer executable instructions are executed by a processor to implement the robot control method provided in the embodiments of the present application.

[0020] The computer program product provided in the embodiments of the present application includes computer executable instructions stored in a computer readable storage medium, and when a processor of an electronic device reads the computer executable instructions from the computer readable storage medium and executes the computer executable instructions, the robot control method provided in the embodiments of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] When the robot is controlled, the local Vino map, the panoramic map and the local semantic map of the current position of the robot are acquired, so that the robot can accurately understand the environment around the robot, the accuracy of the path planning of the robot is improved through multi-dimensional information fusion, the robot can determine the feasible area in the current environment through the local Vino map and the panoramic map, the robot can quickly respond when performing a task, invalid paths are avoided, and the navigation efficiency is improved, the robot can select the best walking path through the calculation of the path score of each path, so that the energy consumption is reduced and the efficiency of task completion is improved, and in addition, the local map of the current position of the robot is only used to determine the target navigation point of the robot, the entire environment does not need to be scanned in advance by the robot, and the intelligent degree and the control efficiency of the robot are improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is an optional flowchart of the robot control method provided in the embodiments of the present application;

[0024] Figure 2 is a flowchart of acquiring the local Vino map of the current position of the robot provided in the embodiments of the present application;

[0025] Figure 3 FIG. 1 is a flowchart of a process for determining a feasible region of a robot at a current location and a region description of the feasible region according to an embodiment of the present application;

[0026] Figure 4 FIG. 2 is a flowchart of a process for determining a feasible path of a robot at a current location and a path description of the feasible path based on a local semantic map and a region description according to an embodiment of the present application;

[0027] Figure 5 FIG. 3 is a system architecture diagram of a robot control method according to an embodiment of the present application;

[0028] Figure 6 FIG. 4 is a structural block diagram of a robot control device according to an embodiment of the present application;

[0029] Figure 7 FIG. 5 is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without making any creative labor fall within the scope of protection of the present application.

[0031] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0032] If the similar description of "first / second" appears in the application file, the following description is added, in the following description, the terms "first\second\third" involved only distinguish similar objects, and do not represent the specific order of the objects, and it can be understood that "first\second\third" can be interchanged with the specific order or sequence as allowed, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0033] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0034] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as commonly understood by one of ordinary skill in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] In determining the target navigation point of the robot, first, in response to the control instruction for the robot, the local Vinoview of the current position of the robot, the panoramic map of the current position and the pre-constructed local semantic map of the current position are acquired; then, based on the local Vinoview and the panoramic map, the feasible region of the robot at the current position and the region description of the feasible region are determined; based on the local semantic map and the region description, the feasible path of the robot at the current position and the path description of the feasible path are determined; based on the control instruction, the path description and the local Vinoview, the path score of the feasible path is determined; based on the path score, the target navigation point of the robot is determined, and the robot is controlled to move to the target navigation point. In this way, by acquiring the local Vinoview, the panoramic map and the local semantic map of the current position of the robot, the robot can accurately understand the environment around the robot, and the accuracy of the robot path planning is improved through multi-dimensional information fusion; moreover, by acquiring the local Vinoview and the panoramic map, the robot can determine the feasible region in the current environment, so that the robot can quickly react when performing a task, avoid invalid paths and improve navigation efficiency; and by calculating the path score of each path, the robot can select the best walking path, thereby reducing energy consumption and improving the efficiency of task completion; in addition, the present application only uses the local map of the current position of the robot to determine the target navigation point of the robot, without the need for the robot to pre-scan the entire environment, thereby improving the intelligent degree and control efficiency of the robot.

[0036] The robot control method provided by the embodiments of the present application can be applied to electronic devices such as robots, notebook computers, tablet computers, desktop computers, smart home appliances and smart vehicle devices, and the embodiments of the present application do not make any limitation on the specific type of electronic device.

[0037] The robot control method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0038] Figure 1 is an optional flowchart of a robot control method provided by an embodiment of the present application, which can be applied to an electronic device. In the following, an electronic device will be taken as an example for illustrative description. As shown in Figure 1 , the method comprises the following steps S101 to S105:

[0039] Step S101, in response to a control instruction for the robot, acquiring a local Vigno map of a current position of the robot, a panoramic map of the current position, and a pre-constructed local semantic map of the current position.

[0040] The control instruction for the robot refers to a command or signal used to guide the behavior and action of the robot, wherein the control instruction for the robot can be derived from user input, including voice information or textual description information input by the user; or can be derived from a preset program or intelligent algorithm; the control instruction for the robot usually includes motion instructions (such as forward, backward, turn, etc.), task instructions (such as grabbing, object recognition, cleaning, photographing, etc.), navigation instructions (such as navigating to a specific coordinate, avoiding obstacles, etc.), and state instructions (such as battery level, sensor state, task completion status, etc.), etc., and the specific control instruction is not limited herein.

[0041] Vigno map is a set of continuous polygonal structures composed of perpendicular bisectors connecting two adjacent points, which divides the plane into multiple polygonal regions, each region contains a generator, and the distance from any point in the region to the generator is shorter than the distance to other generators; the local Vigno map refers to the Vigno map constructed within a limited range of the current position of the robot; the panoramic map of the current position refers to a panoramic image generated by wide-angle shooting or image stitching; the semantic map refers to a map generated by adding semantic information of objects in the environment, such as the category, position and attribute of the object, to the traditional map; the local semantic map refers to the semantic map constructed within a limited range of the current position of the robot.

[0042] In some embodiments, referring to Figure 2 , Figure 2 is a flowchart of acquiring a local Vigno map of a current position of a robot provided by an embodiment of the present application, Figure 2 illustrates that in step S101, acquiring the local Vigno map of the current position of the robot can be implemented through the following steps S1011 to S1013:

[0043] Step S1011, acquiring position data of the current position of the robot and environment perception data of the current position.

[0044] The position data refers to coordinate information of a current position of the robot, and the coordinate information of the current position of the robot is acquired through a positioning system (a global positioning system (GPS), an inertial navigation system, etc.) installed in the robot. The environment perception data refers to information about an environment around the robot acquired by the robot through a sensor, and the environment perception data can include obstacle information (such as a position, a size, a shape, etc. of an obstacle), environment features (including a terrain (such as flat or slope), and static elements (such as a wall or a door)), etc., and specific environment perception data is not limited herein. The environment perception data is acquired through various sensors installed in the robot, and a specific process can include: emitting a laser beam through a laser radar and measuring a time of reflection back to generate point cloud data of the environment; or performing distance measurement through an ultrasonic sensor to detect a near-distance obstacle.

[0045] In step S1012, based on the position data and the environment perception data, a local grid map of a current position of the robot is determined through a grid map generation tool.

[0046] The grid map is a grid-based map representation method, each grid represents a small area in the environment, and contains state information about the area, such as an occupancy state of the area; the grid map generation tool refers to an algorithm or a software tool for creating a grid map, and the grid map generation tool divides the environment into grids using data collected by sensors, and assigns state information to each grid.

[0047] In some embodiments, the position data and the environment perception data of the current position of the robot are acquired, and then the acquired data is processed, including filtering, denoising, etc. Then, according to a preset map resolution and a map size, the environment of the current position of the robot is divided into uniform grids, and the state information of each grid is determined according to the perception data, and the state information can be whether the grid is occupied (i.e. whether there is an object). Finally, according to the divided grids and the state information of each grid, a local grid map is constructed through a grid map generation tool.

[0048] In step S1013, the local grid map is converted into a local Voronoi diagram of the current position of the robot.

[0049] Through processing of the constructed local grid map, a local Voronoi diagram reflecting spatial relationships around the current position of the robot is generated, thereby helping the robot better understand the environment around the current position.

[0050] In some embodiments, step S1013 can be implemented by the following method: first, preprocessing the local grid map to obtain a preprocessed grid map; then, performing skeletonization processing on the preprocessed grid map to obtain an initial vinov graph; next, determining the coordinate value of each path intersection point in the initial vinov graph and the coordinate value of the current position of the robot; finally, determining the local vinov graph of the current position of the robot based on the initial vinov graph, the coordinate value of the path intersection point and the coordinate value of the current position.

[0051] Here, preprocessing the local grid map refers to the process of optimizing the original local grid map, including noise removal, data smoothing, etc. Skeletonization processing is an image processing technique used to extract the main structural features of an object, forming a slender "skeleton" representation. The main features of skeletonization processing include structure simplification, i.e. simplifying the edges or contours of an object into center lines or thin lines, and removing unnecessary details in the object; and during the skeletonization processing, the connectivity and topological features of the object are maintained as much as possible, so that the shape properties of the object are not affected. The path intersection point represents the point where at least two paths intersect in the vinov graph.

[0052] In some embodiments, the robot works in an indoor environment, first, the sensor installed on the robot acquires data of the position of the robot, and constructs a local grid map; then, the local grid map is preprocessed to remove noise in the local grid map; next, the preprocessed local grid map is skeletonized to obtain an initial vinov graph; then, a coordinate system is constructed based on the current position of the robot, and the coordinate value of the path intersection point in the initial vinov graph is determined; finally, based on the initial vinov graph, the coordinate value of the path intersection point and the coordinate value of the current position of the robot, the local vinov graph of the current position of the robot is determined.

[0053] In some embodiments, the method for obtaining the panoramic map of the current position of the robot in step S101 can be implemented by the following method: first, acquiring scene images collected after the robot rotates at different rotation angles at the current position; then, performing image stitching on all scene images according to the rotation angle corresponding to each scene image by using a pre-set image stitching algorithm to obtain a stitched image; finally, performing image fusion on the image overlap area in the stitched image to obtain the panoramic map of the current position of the robot.

[0054] The image stitching algorithm refers to an algorithm for merging multiple images into a continuous image, which can usually use feature matching and geometric transformation; the image overlap region refers to the overlapping part between different images in the stitched image, and the overlapping part usually contains the same scene features; image fusion refers to combining image information in the image overlap region together to eliminate visual discontinuity and differences, so that the final image is more natural and coherent.

[0055] In some embodiments, a mobile robot performs panoramic image acquisition within a limited range. First, the mobile robot rotates at different angles at the current location, where the angle of each rotation can be the same or different. For example, the mobile robot rotates 30 degrees at the current location each time to take a picture, and completes the shooting after rotating 360 degrees, a total of 12 images. Image stitching is performed on the 12 images to obtain a stitched image. Since there may be the same object or scene in the adjacent two images of the 12 images, as shown in Figure 1, which contains chair 1, and chair 1 and the scene near chair 1 are also contained in the shooting of picture 2. Therefore, after completing the image stitching, image fusion needs to be performed on the overlapping region in the stitched image to generate the final panoramic map of the current location of the mobile robot.

[0056] In some embodiments, the local semantic map of the current location of the robot can be constructed by the following method: first, the data of the surrounding environment is obtained by the sensor installed on the robot, including depth information, color information, etc.; then, meaningful features are extracted from the obtained data, for example, furniture and other objects in the room are identified through image recognition technology; then, the extracted features are matched with predefined semantic categories to generate labeling information, such as marking the identified table as furniture; finally, according to the labeling information, each feature and its relative position are integrated into the local semantic map; the local semantic map not only shows the spatial layout of the current location of the robot, but also contains the semantic information of each object.

[0057] In some embodiments, the local semantic map of the current location of the robot can also be manually set by the user, manually labeling the objects and specific positions in the environment around the current location of the robot in the corresponding map to generate the local semantic map.

[0058] Step S102, based on the local Vinotemp graph and the panoramic map, determining the feasible region of the robot at the current location and the region description of the feasible region.

[0059] A feasible region refers to an area with a walking path, determined by a local Venn diagram and a panoramic map. The region description is a specific description of the feasible region, including its size, objects and features contained within it, such as: feasible region 1 contains a three-legged table with a transparent water glass on it.

[0060] In some embodiments, see Figure 3 , Figure 3 This is a schematic diagram of the process for determining the feasible area of ​​the robot's current location and the regional description of the feasible area provided in the embodiments of this application; Figure 3 The step S102, which determines the feasible region of the robot's current location and its regional description based on the local Venn diagram and the panoramic map, can be achieved through the following steps S1021 to S1023:

[0061] Step S1021: Construct a coordinate system with the robot's current position as the origin.

[0062] In some embodiments, the robot's position is the origin of the coordinate system. The constructed coordinate system can be a two-dimensional coordinate system or a three-dimensional coordinate system. When constructing a two-dimensional coordinate system, i.e., the robot's current position is (0, 0), the x-axis and y-axis directions of the coordinate system are determined based on the robot's current position. For example, it can be assumed that the x-axis points to the right of the robot and the y-axis points directly in front of the robot. When constructing a three-dimensional coordinate system, i.e., the robot's current position is (0, 0, 0), the x-axis, y-axis, and z-axis directions of the coordinate system are determined based on the robot's current position. For example, it can be assumed that the x-axis points to the right of the robot, the y-axis points directly in front of the robot, and the z-axis points upwards on the robot. Here, the directions of the x-axis, y-axis, and z-axis are not limited in this application.

[0063] In some embodiments, when constructing a coordinate system, the current position of other objects can be used as the origin of the coordinate system, and the coordinates of the robot's current position can be determined based on the distance between the robot and other objects.

[0064] Step S1022: Based on the coordinate system, perform map matching between the local Venn map and the panoramic map to obtain the feasible area of ​​the robot at its current location.

[0065] Map matching refers to aligning the data of two or more maps according to a constructed coordinate system in order to compare and integrate data information.

[0066] In some embodiments, step S1022 can be implemented by the following method: first, based on the coordinate system, the coordinate values of each point in the panoramic map and the local Vino map are determined; then, the panoramic map and the local Vino map are matched point by point based on the coordinate values to obtain a matching area of the panoramic map and the local Vino map; finally, the feasible area of the robot at the current position is determined from the matching area.

[0067] Point-by-point matching refers to a process of matching points with the same coordinate values in the panoramic map and the local Vino map according to the coordinate system.

[0068] In some embodiments, based on the constructed coordinate system, the panoramic map and the local Vino map are placed in the same direction on the coordinate system, and then the coordinate values of each point on the panoramic map and the local Vino map are matched. For example, the coordinates of the points included in path 1 in the panoramic map are (2, 3), (2, 4) and (2, 5). According to the coordinates of each point of path 1, the area of the coordinate points corresponding to path 1 in the local Vino map is found, and so on, to obtain a plurality of matching areas. Finally, the area in which the walking path exists is searched for in the plurality of matching areas, and the area in which the walking path exists is determined as the feasible area.

[0069] Step S1023, generating the area description of the feasible area through the pre-constructed first language model.

[0070] The pre-constructed first language model refers to a pre-trained language model that can generate natural language descriptions. The specific model can be a deep learning-based architecture such as transformer or LSTM, and the specific model is not limited in the present application.

[0071] In some embodiments, the determined feasible area is input into the pre-constructed first language model to generate the area description of the feasible area. For example, after the feasible area is input into the pre-constructed first language model, the generated area description can be “there is path 1, door frame 1 and water cup 1 in the feasible area 1”.

[0072] Step S103, determining the feasible path of the robot at the current position and the path description of the feasible path based on the local semantic map and the area description.

[0073] In some embodiments, the local Vino map includes a plurality of paths; see Figure 4 , Figure 4 is a flowchart provided by the embodiments of the present application for determining the feasible path of the robot at the current position and the path description of the feasible path based on the local semantic map and the area description; Figure 4It is shown that in step S103, based on the local semantic map and the region description, the feasible path of the robot at the current position and the path description of the feasible path can be determined by the following steps S1031 to S1033:

[0074] In step S1031, the first description text of each path in the local Venn diagram is determined through the local semantic map; the first description text includes the objects included in each path and the coordinate values of each object.

[0075] In some embodiments, since the local Venn diagram contains multiple paths, first, a coordinate system is constructed at the current position of the robot; then the local semantic map and the local Venn diagram are matched according to the coordinate system, and the specific matching method is described in step S1022, which will not be repeated here. After matching, the objects included in each path in the local Venn diagram and the coordinate values of each object can be determined according to the local semantic map, such as the generated first description text {door frame 1; (4, 4); water cup 1; (5, 5)}.

[0076] In step S1032, the first description text and the region description are text spliced to obtain the first prompt word.

[0077] In some embodiments, after obtaining the first description text and the region description, the first description text and the region description are filled into the preset first prompt word template to obtain the first prompt word, such as the first prompt word: "According to the following content description of this path, {there are path 1, door frame 1 and water cup 1 in feasible region 1}, {door frame 1; (4, 4); water cup 1; (5, 5)}".

[0078] In step S1033, the second language model is constructed in advance, and the feasible path of the robot at the current position and the path description of the feasible path are determined from the multiple paths based on the first prompt word.

[0079] The second language model constructed in advance is a large language model pre-trained for generating path descriptions of feasible paths, and the specific model is not limited herein.

[0080] In some embodiments, the first prompt words are input into a pre-constructed second language model, where the first prompt words are at least one; when there are multiple first prompt words, the feasible path of the current position of the robot is determined according to the multiple first prompt words, for example, there are three first prompt words in total, the first first prompt word is: "describe the path as follows, {there are path 1, door frame 1 and water cup 1 in the feasible area 1}, {door frame 1; (4, 4); water cup 1; (5, 5)}", the second first prompt word is: "describe the path as follows, {there are path 2, bookshelf 1 and water cup 1 in the feasible area 2}, {bookshelf 1; (3, 3); water cup 1; (5, 5)}", and the third first prompt word is: "describe the path as follows, {there are path 3, chair 1; and water cup 1 in the feasible area 3}, {chair 1; (2, 2); water cup 1; (5, 5)}"; the pre-constructed second language model is used to analyze the three prompt words, and it is determined that the feasible path is path 1 because there is no obstacle in path 1 and the robot can pass through directly. After determining the feasible path, the path description of the feasible path is generated, for example: "the feasible path is path 1, there are door frame 1 and water cup 1 in path 1, the coordinates of door frame 1 are (4, 4), the coordinates of water cup 1 are (5, 5), and the robot can take water cup 1 through door frame 1 in path 1".

[0081] In step S104, the path score of the feasible path is determined based on the control instruction, the path description and the local Veno graph.

[0082] The path score of the feasible path refers to the quantitative evaluation of the feasible path, and the optimal path of the robot can be determined through the path score.

[0083] In some embodiments, step S104 can be implemented by the following steps: first, text conversion is performed on the control instruction to obtain a second description text; then, based on the path description and the second description text, a second prompt word is determined; then, through a pre-constructed third language model, the text similarity between the path description and the second description text is determined based on the second prompt word; finally, the path score of the feasible path is determined based on the local Veno graph and the text similarity.

[0084] Text conversion refers to converting the voice control instruction input by the user into text data; the second description text is the text data obtained after text conversion of the voice control instruction input by the user; the second prompt word is a prompt word used to guide the third language model to calculate the text similarity; the pre-constructed third language model is a large language model pre-trained for calculating the text similarity between the path description and the second description text; the text similarity is an index for measuring whether two short texts are similar in semantics.

[0085] In some embodiments, the control instruction is text converted to obtain a second description text: "Help me get the water cup"; then, the path description and the second description text are filled into a second prompt word template constructed in advance to obtain a second prompt word, the second prompt word is: "User demand: 'Help me get the water cup'. According to the following path description, determine the text similarity between each path description and the user demand. The path description is 'The feasible path is path 1, there are door frame 1 and water cup 1 in path 1, the coordinates of door frame 1 are (4, 4), the coordinates of water cup 1 are (5, 5), and the robot can take water cup 1 through door frame 1 in path 1'"; then, the path description and the second description text are preprocessed, such as removing punctuation, removing stop words, etc.; then, the preprocessed path description and the second description text are input into a third language model constructed in advance to generate a text vector; finally, the text similarity between the description text and the second description text is calculated using the cosine similarity; the specific method of calculating the text similarity is not limited herein.

[0086] In some embodiments, the feasible path includes N nodes; the path score of the feasible path is determined based on the local Venn diagram and the text similarity, which can be achieved by the following method: first, determine the access state of each node in the feasible path based on the local Venn diagram; the access state includes node visited and node not visited; then, score the corresponding node based on the access state of each node to obtain a score value of each node; then, the score values of all nodes in the feasible path are weighted and summed to obtain a comprehensive score of the feasible path; finally, the product of the text similarity and the comprehensive score is determined as the path score of the feasible path.

[0087] The node refers to a key point in the path, which can be a cross point or a turning point, which is not limited herein; the access state refers to whether the robot has passed through the node before the current time, if yes, the access state is visited; if not, the access state is not visited.

[0088] In some embodiments, the feasible path 1 includes 5 nodes, namely nodes A, B, C, D and E, the access states of nodes C, D and E are determined to be visited through the Venn diagram, and nodes A and B are not visited, then nodes A and B are respectively assigned a score of 1, and nodes C, D and E are respectively assigned a score of 0; then, the scores of nodes A, B, C, D and E are summed and multiplied by a pre-set weighted weight, such as a weighted weight of 0.4, and the comprehensive score of the feasible path 1 is 0.4x(1+1+0+0+0)=0.8; finally, the product of the text similarity and the comprehensive score is determined as the path score of the feasible path 1, such as a text similarity of 0.7, and the path score of the feasible path 1 is 0.7x0.8=0.56.

[0089] At step S105, the target navigation point of the robot is determined based on the path scores, and the robot is controlled to move to the target navigation point.

[0090] The target navigation point refers to a target node to which the robot needs to move next.

[0091] In some embodiments, the target navigation point of the robot is determined based on the path scores at step S105, which can be achieved by the following method: first, the feasible path with the highest path score is determined as the target path; then, the node in the target path closest to the current position is determined as the target navigation point.

[0092] In some embodiments, after the path score of each feasible path is calculated, the feasible path with the highest path score is determined as the target path; for example, the path score of the feasible path 1 is 0.78, and the path score of the feasible path 2 is 0.56, so the feasible path 1 is determined as the target path; next, the distance between each node in the target path and the current position of the robot can be determined by the sensor, and the node closest to the current position is determined as the target navigation point.

[0093] In some embodiments, after the target navigation point is determined, the local wino graph can also be updated according to the determined target navigation point, which can be achieved by the following method: first, when the robot moves to the target navigation point, the access state of the target navigation point is updated; then, the local wino graph is updated based on the updated access state of the target navigation point.

[0094] In some embodiments, when the target navigation point has not been accessed by the robot before, after the robot reaches the target navigation point, the access state of the target navigation point is updated to be accessed, and the score of the target navigation point is set to 0; then, according to the updated access state of the target navigation point, the original access state of the target navigation point stored in the local wino graph is replaced by the updated access state.

[0095] The robot control method provided in the embodiments of the present application can be widely applied to various scenes, for example, warehouse and logistics management, intelligent home cleaning, indoor navigation and service. In the warehouse and logistics management scene, the robot can be a carrying robot. The carrying robot can analyze and determine the feasible path and target navigation point by acquiring the local Vinoview, panoramic map and pre-constructed local semantic map of the current position of the carrying robot, can evaluate the feasible area in real time, avoid obstacles and improve the carrying efficiency. In the intelligent home cleaning scene, the robot can be a sweeping robot. The sweeping robot can determine the feasible area and feasible path by acquiring the local Vinoview, panoramic map and pre-constructed local semantic map of the current position of the sweeping robot, thereby determining the target navigation point and controlling the sweeping robot to move, thereby improving the work efficiency. In the indoor navigation and service scene, the robot can be a service robot. The service robot can find the nearest target position (such as a sickroom or a store) and control the robot to move to the target position by acquiring the local Vinoview, panoramic map and pre-constructed local semantic map of the current position of the carrying robot, thereby improving the service quality.

[0096] The robot control method provided in the embodiments of the present application can make the robot accurately understand the environment around the robot by acquiring the local Vinoview, panoramic map and local semantic map of the current position of the robot, thereby improving the accuracy of the robot path planning through multi-dimensional information fusion. Moreover, the robot can determine the feasible area in the current environment by acquiring the local Vinoview and panoramic map, thereby enabling the robot to quickly respond when performing a task, avoiding invalid paths and improving the navigation efficiency. Furthermore, the robot can select the best walking path by calculating the path score of each path, thereby reducing the energy consumption and improving the efficiency of task completion. In addition, the present application only uses the local map of the current position of the robot to determine the target navigation point of the robot, without the need for the robot to pre-scan the entire environment, thereby improving the intelligent degree and control efficiency of the robot.

[0097] In the following, an exemplary application of the embodiments of the present application in an actual application scene will be described.

[0098] The embodiments of the present application provide a robot control method. Through the robot control method provided in the embodiments of the present application, a more intelligent intelligent interaction and autonomous navigation system can be constructed for the robot, so that the robot can understand the human ambiguous instructions, for example, "help me find where the cup is?" According to this instruction, the robot can autonomously explore the environment in an unknown environment and record the exploration state of the environment until the required item is found.

[0099] Referring to Figure 5 , Figure 5is a system architecture diagram of a robot control method provided by an embodiment of the present application; the specific implementation steps are as follows:

[0100] Step one: first, the local grid map 301 of the current position of the robot is skeletonized to obtain an initial vinov graph of the current position of the robot; then, the coordinate values of each path intersection point in the initial vinov graph are determined; then, the local vinov graph 302 is generated in combination with the coordinate values of the current position of the robot; and the local vinov graph 302 is updated in an incremental manner, and the local vinov graph 302 is used to store the access state of each node and the scoring situation.

[0101] Step two: through the local semantic map 305 of the current position of the robot, the objects included in each path 303 in the local vinov graph 302 and the coordinate values of the objects are determined, and a first description text 304 is generated. In addition, the panoramic map 306 is regionally divided according to the path, the feasible region 307 is determined, and the regional description 308 of the feasible region 307 is obtained through the multi-modal visual language model (the first language model described above). The first prompt word 309 is generated according to the first description text 304 and the regional description 308, and the first prompt word 309 is as follows: “According to the following content description of this path, {regional description}, {first description text}”; the first prompt word 309 is input into the second large language model 310 to obtain the path description 311.

[0102] Step three: the control instruction 312 input by the user may be: “I want to eat an apple”. Then the control instruction 312 is converted into a second description text 313, the second prompt word 314 is generated according to the second description text 313 and the path description 311, and the second prompt word is as follows: “User demand: {second description text}. According to the following description of the path, output a probability (text similarity) for each path that meets the user demand. Path description: {path description}”. Then, the second prompt word 314 is input into the third language model 315, and the text similarity between the second description text 313 and the path description 311 is determined through the third language model 315.

[0103] Step four: according to the local vinov graph 302, the access state of each node of the feasible path and the corresponding score are determined, if the current node has been accessed, the score is 0; if not accessed, the score is 1, the scores of all nodes are added and multiplied by a pre-set weighted weight to obtain the comprehensive score of each path. The path score 316 of the feasible path is obtained by multiplying the comprehensive score and the text similarity output by the third language model 315, and the path with the highest path score is selected as the target path 317; the node closest to the current position of the robot in the target path 317 is taken as the target navigation point 318.

[0104] Step five: when the robot moves to the target navigation point 318, the access state of the target navigation point is updated to obtain an updated local Veno graph 319.

[0105] The robot control method provided by the embodiments of the present application enables the mobile robot to better achieve intelligent interaction with the human user in a real artificial environment. Without manually setting a complex configuration, the mobile robot can achieve autonomous navigation behavior by means of voice and visual semantic information fusion. Due to the incorporation of the large language model, the navigation process interaction becomes more intelligent, and the mobile robot can navigate in an environment without a priori map.

[0106] The robot control method described in the above embodiments, Figure 6 A structural block diagram of a robot control device 100 provided by an embodiment of the present application is shown. The robot control device can be a device in an electronic device (for example, a robot). The robot control device can be implemented in a software manner, which can be in the form of software such as a program and a plug-in. The robot control device includes the following software modules: an acquisition module 101, a first determination module 102, a second determination module 103, a third determination module 104, and a control module 105. These modules are logical, and thus can be combined or further split according to the implemented functions.

[0107] The acquisition module 101 is configured to acquire, in response to a control instruction for a robot, a local Veno graph of a current position of the robot, a panoramic map of the current position, and a pre-constructed local semantic map of the current position. The first determination module 102 is configured to determine, based on the local Veno graph and the panoramic map, a feasible region of the robot at the current position and a region description of the feasible region. The second determination module 103 is configured to determine, based on the local semantic map and the region description, a feasible path of the robot at the current position and a path description of the feasible path. The third determination module 104 is configured to determine, based on the control instruction, the path description, and the local Veno graph, a path score of the feasible path. The control module 105 is configured to determine a target navigation point of the robot based on the path score, and control the robot to move to the target navigation point.

[0108] In some embodiments, the first determination module is further configured to: construct a coordinate system with the current position of the robot as a coordinate origin; perform map matching on the local Veno graph and the panoramic map based on the coordinate system to obtain the feasible region of the robot at the current position; and generate the region description of the feasible region by using a pre-constructed first language model.

[0109] In some embodiments, the first determining module is further configured to: determine coordinate values of respective points in the panoramic map and the local Vinov map based on the coordinate system; perform point-by-point matching on the panoramic map and the local Vinov map based on the coordinate values to obtain a matching area of the panoramic map and the local Vinov map; and determine a feasible area of the robot at the current position from the matching area.

[0110] In some embodiments, the local Vinov map includes a plurality of paths; the second determining module is further configured to: determine, by the local semantic map, a first description text of each path in the local Vinov map; the first description text includes objects included in the each path and coordinate values of each of the objects; perform text splicing on the first description text and the area description to obtain a first prompt word; determine, by a pre-constructed second language model, a feasible path of the robot at the current position from the plurality of paths based on the first prompt word, and a path description of the feasible path.

[0111] In some embodiments, the third determining module is further configured to: perform text conversion on the control instruction to obtain a second description text; determine a second prompt word based on the path description and the second description text; determine, by a pre-constructed third language model, a text similarity between the path description and the second description text based on the second prompt word; and determine a path score of the feasible path based on the local Vinov map and the text similarity.

[0112] In some embodiments, the feasible path includes N nodes; the third determining module is further configured to: determine an access state of each of the nodes in the feasible path based on the local Vinov map; the access state includes a node that has been accessed and a node that has not been accessed; score the corresponding node based on the access state of each of the nodes to obtain a score value of each of the nodes; perform weighted summation on the score values of all the nodes in the feasible path to obtain a comprehensive score of the feasible path; and determine a product of the text similarity and the comprehensive score as the path score of the feasible path.

[0113] In some embodiments, the third determining module is further configured to: determine a feasible path with a highest path score as a target path; and determine a node in the target path that is closest to the current position as the target navigation point.

[0114] In some embodiments, the apparatus further includes an updating module configured to: update an access state of the target navigation point when the robot moves to the target navigation point; and update the local Vinov map based on the updated access state of the target navigation point.

[0115] In some embodiments, the obtaining module is further configured to: obtain position data of a current position of the robot and environment perception data of the current position; determine, by a grid map generation tool, a local grid map of the current position of the robot based on the position data and the environment perception data; and convert the local grid map into a local Vinovision map of the current position of the robot.

[0116] In some embodiments, the obtaining module is further configured to: pre-process the local grid map to obtain a pre-processed grid map; perform skeletonization processing on the pre-processed grid map to obtain an initial Vinovision map; determine coordinate values of each path intersection point in the initial Vinovision map and a coordinate value of the current position of the robot; and determine, based on the initial Vinovision map, the coordinate values of the path intersection points and the coordinate value of the current position, a local Vinovision map of the current position of the robot.

[0117] In some embodiments, the obtaining module is further configured to: obtain scene images collected by the robot after rotating at different rotation angles at the current position; perform image stitching on all the scene images according to the rotation angle corresponding to each scene image by using a preset image stitching algorithm to obtain a stitched image; and perform image fusion on an image overlap area in the stitched image to obtain a panoramic map of the current position of the robot.

[0118] It should be noted that the description of the device embodiments of the present application is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments, and thus will not be described herein. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0119] The electronic device provided in the embodiments of the present application can be a robot, Figure 7 is a structural schematic diagram of the electronic device provided in the embodiments of the present application. As shown in Figure 7 the electronic device 130 includes at least one processor 131 (only one processor is shown in the figure), a memory 132, and computer executable instructions 133 stored in the memory 132 and executable on the at least one processor 131, and the processor 131 executes the executable instructions 133 to implement the steps in any of the above robot control method embodiments. Figure 7

[0120] The electronic device can include but is not limited to the processor 131 and the memory 132. Those skilled in the art can understand that, Figure 7 ​The electronic device 130 is merely an example and does not limit the electronic device 130, which can include more or fewer components than shown, or combine some components, or have different components, such as an input / output device, a network access device, and the like.

[0121] The processor 131 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor.

[0122] The memory 132 can be an internal storage unit of the electronic device 130, such as a hard disk or a memory of the electronic device 130 in some embodiments. The memory 132 can also be an external storage device of the electronic device 130, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like in other embodiments. Further, the memory 132 can include both an internal storage unit and an external storage device of the electronic device 130. The memory 132 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of computer programs, and the like. The memory 132 can also be used to temporarily store data that has been output or will be output.

[0123] The embodiment of the present application provides a computer readable storage medium, wherein computer executable instructions are stored. When the computer executable instructions are executed by a processor, the processor executes the robot control method provided by the embodiment of the present application, for example, as shown in the robot control method. Figure 1 The embodiment of the present application provides a computer program product, which includes computer executable instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the robot control method provided by the embodiment of the present application.

[0124] The embodiment of the present application provides a computer program product, which includes computer executable instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the robot control method provided by the embodiment of the present application.

[0125] In some embodiments, the computer-readable storage media can be a memory such as a RAM, a ROM, a flash memory, a magnetic surface memory, an optical disk, or a CD-ROM, etc. It can also be various devices including one or any combination of the above memories.

[0126] In some embodiments, the computer-executable instructions can be in the form of programs, software, software modules, scripts, or code that are written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and that are deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0127] As an example, computer-executable instructions can, but need not, reside in a file system's files, can be stored in documents, for example, Hyper Text Markup Language (HTML) documents, in a single file dedicated to the program in question, or in multiple files (e.g., files that store one or more modules, sub programs, or code portions).

[0128] As an example, computer-executable instructions can be deployed to be executed on one electronic device or on multiple electronic devices that are located at one site or distributed across multiple sites and that are interconnected by a communication network.

[0129] The above description is only some embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A robot control method, characterized in that, The method includes: In response to control commands to the robot, a local Venn diagram of the robot's current location, a panoramic map of the current location, and a pre-constructed local semantic map of the current location are acquired. Based on the local Venn diagram and the panoramic map, the feasible area of ​​the robot at its current location and the regional description of the feasible area are determined. Based on the local semantic map and the region description, a feasible path for the robot at its current location and a path description for the feasible path are determined. Based on the control command, the path description, and the local Venn diagram, the path score of the feasible path is determined; Based on the path score, the target navigation point of the robot is determined, and the robot is controlled to move to the target navigation point.

2. The method according to claim 1, characterized in that, The process of determining the feasible region of the robot at its current location and the regional description of the feasible region based on the local Venn diagram and the panoramic map includes: A coordinate system is constructed with the robot's current position as the origin. Based on the coordinate system, map matching is performed on the local Venn diagram and the panoramic map to obtain the feasible area of ​​the robot at its current location; A regional description of the feasible region is generated using a pre-built first language model.

3. The method according to claim 2, characterized in that, The process of performing map matching between the local Venn diagram and the panoramic map based on the coordinate system to obtain the feasible area of ​​the robot at its current location includes: Based on the coordinate system, determine the coordinate values ​​of each point in the panoramic map and the local Venn diagram; Based on the coordinate values, the panoramic map and the local Veno map are matched point by point to obtain the matching area between the panoramic map and the local Veno map; Determine the feasible area of ​​the robot at its current location from the matching area.

4. The method according to claim 1, characterized in that, The local Vinograph includes multiple paths; The step of determining a feasible path for the robot at its current location and a path description of the feasible path based on the local semantic map and the region description includes: Using the local semantic map, a first descriptive text is determined for each path in the local Vinograph; the first descriptive text includes the objects included in each path and the coordinate values ​​of each object; The first descriptive text and the region description are concatenated to obtain the first prompt word; Using a pre-built second language model, a feasible path for the robot at its current location is determined from the multiple paths based on the first prompt word, and a path description of the feasible path is also determined.

5. The method according to claim 1, characterized in that, The process of determining the path score of the feasible path based on the control command, the path description, and the local Venn diagram includes: The control commands are converted into text to obtain a second description text; Based on the path description and the second description text, a second prompt word is determined; Using a pre-built third language model, the text similarity between the path description and the second description text is determined based on the second prompt word; The path score of the feasible path is determined based on the local Venn diagram and the text similarity.

6. The method according to claim 5, characterized in that, The feasible path includes N nodes; the path score determination based on the local Venn diagram and the text similarity includes: The access status of each node in the feasible path is determined based on the local Vinograph; the access status includes whether the node has been visited or not. The corresponding node is scored based on the access status of each node, and a score value is obtained for each node. The comprehensive score of the feasible path is obtained by weighted summing of the scores of all nodes in the feasible path. The product of the text similarity and the comprehensive score is determined as the path score of the feasible path.

7. The method according to claim 6, characterized in that, Based on the path score, the target navigation point of the robot is determined, including: The feasible path with the highest path score is identified as the target path; The node in the target path that is closest to the current location is determined as the target navigation point.

8. The method according to claim 7, characterized in that, The method further includes: When the robot moves to the target navigation point, the access status of the target navigation point is updated; The local Venn diagram is updated based on the updated access status of the target navigation point.

9. The method according to any one of claims 1 to 8, characterized in that, The step of obtaining the local Vinograph of the robot's current position includes: Obtain the current location data of the robot and the environmental perception data of the current location; Based on the location data and the environmental perception data, a local grid map of the robot's current location is determined using a grid map generation tool. The local grid map is converted into a local Veno map of the robot's current location.

10. The method according to claim 9, characterized in that, The step of converting the local grid map into a local Vino map of the robot's current location includes: The local raster map is preprocessed to obtain a preprocessed raster map; The preprocessed raster map is then skeletalized to obtain the initial Veno map; Determine the coordinates of each path intersection point in the initial Vinograph, as well as the coordinates of the robot's current position; Based on the initial Veno map, the coordinates of the path intersections, and the coordinates of the current position, a local Veno map of the robot's current position is determined.

11. The method according to any one of claims 1 to 8, characterized in that, Obtain a panoramic map of the robot's current location, including: Acquire scene images of the robot at its current position after rotating at different angles; By using a preset image stitching algorithm, all scene images are stitched together according to the rotation angle corresponding to each scene image to obtain a stitched image; Image fusion is performed on the overlapping areas of the stitched image to obtain a panoramic map of the robot's current location.

12. A robot control device, characterized in that, include: The acquisition module is used to acquire, in response to control commands for the robot, a local Venn diagram of the robot's current location, a panoramic map of the current location, and a pre-built local semantic map of the current location. The first determining module is used to determine, based on the local Venn diagram and the panoramic map, a feasible region of the robot at its current location and a regional description of the feasible region. The second determining module is used to determine, based on the local semantic map and the region description, a feasible path for the robot at its current location and a path description for the feasible path. The third determining module is used to determine the path score of the feasible path based on the control command, the path description and the local Venn diagram; The control module is used to determine the target navigation point of the robot based on the path score, and control the robot to move to the target navigation point.

13. An electronic device, characterized in that, include: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the robot control method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions or computer programs, which, when executed by a processor, implement the robot control method according to any one of claims 1 to 11.

Images