A semantic vine map based en route multi-task planning method

Abstract

The application provides a semantic Venn diagram-based on-route multi-task planning method, which comprises the following steps: in the process of executing a first task instruction by a robot, in response to at least one second task instruction, determining a target scene area for executing each second task instruction and a target area passing point based on a semantic Venn diagram; determining whether the target area passing point is in a first path node sequence corresponding to the first task instruction; if the target area passing point is not in the first path node sequence, evaluating a cost of executing the second task instruction, updating the first path node sequence based on the cost evaluation result, and controlling the robot to move according to the updated first path node sequence. By using the above-mentioned semantic Venn diagram-based on-route multi-task planning method, the problems of poor task scheduling flexibility, high response delay and low overall multi-task execution efficiency in the prior art are solved.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a multi-task planning method along the way based on semantic Venographs. Background Technology

[0002] With the development of artificial intelligence and autonomous mobile robot technology, robots have been widely applied in industrial production, home services, public services, and other scenarios, replacing or assisting humans in performing highly repetitive and dangerous tasks. During autonomous movement, robots require accurate scene representation and navigation point planning to achieve reliable path control. Existing robot navigation and task planning technologies mostly focus on single-target point navigation or fixed-sequence multi-target task planning: at the scene representation level, grid maps and 3D semantic scene graphs are commonly used; at the task planning level, large language models are often used to convert natural language instructions into executable action sequences to complete complex task reasoning.

[0003] However, existing methods are mostly based on static task sequence design, which makes it difficult to respond to dynamic tasks added temporarily during the execution of the main path. When faced with new tasks or task changes, global replanning is usually required. It is impossible to smoothly insert intermediate tasks without interrupting or resetting the main process, resulting in insufficient task scheduling flexibility, high computational overhead, high response latency, and the path is prone to a large number of detours and repetitions, which reduces the overall execution efficiency of multi-task. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a multi-task planning method along the way based on semantic Veno maps to overcome at least one of the above-mentioned defects.

[0005] In a first aspect, embodiments of this application provide a multi-task planning method along a route based on semantic Vinographs, including: During the execution of the first task instruction by the robot, in response to at least one second task instruction, the target scene region and the path points of the target region are determined based on the semantic Venn diagram. Determine whether the path points through the target area are in the first path node sequence corresponding to the first task instruction; If the target area's waypoints are not in the first path node sequence, evaluate the cost of executing the second task instruction, update the first path node sequence based on the cost evaluation result, and control the robot to move according to the updated first path node sequence.

[0006] In an optional implementation, the first path node sequence is updated based on the cost evaluation results in the following manner: the detour cost and the repetition cost corresponding to the second task instruction are determined. The detour cost refers to the detour distance required for the robot to cut off from the current path to execute the second task instruction and then return to the path. The repetition cost refers to the path repetition length caused by delaying the execution of the second task instruction and revisiting the target area waypoint. The first path node sequence is updated based on the comparison results of the detour cost and the repetition cost.

[0007] In an optional implementation, the step of updating the first path node sequence based on the comparison result of detour cost and repetition cost includes: if the detour cost is less than the repetition cost, inserting the second task instruction into the task execution queue under the first task instruction, and inserting the second path node sequence corresponding to the second task instruction into the first path node sequence.

[0008] In an optional implementation, the method further includes: if the detour cost is greater than or equal to the repetition cost, then delaying the execution of the second task instruction.

[0009] In an optional implementation, the method further includes: if the target area waypoint is in the first path node sequence, then directly insert the second task instruction into the task execution queue under the first task instruction.

[0010] In an optional implementation, the step of determining the target scene region and the path points of the target region for executing each second task instruction based on the semantic Veno diagram includes: parsing the second task instruction to determine the target object; selecting the target scene region where the target object is located in the semantic Veno diagram, and determining the path points of the target scene region as the path points of the target region.

[0011] In an optional implementation, before determining the target scene region and target region waypoints for executing each second task instruction based on the semantic Veno map in response to at least one second task instruction, the method further includes: constructing a raster map based on 3D semantic point cloud data of the global scene, and performing connected component segmentation on the raster map to determine multiple geometric regions; integrating the multiple geometric regions to obtain at least one integrated region carrying a semantic label; generating region waypoints for each integrated region based on the region center position and axis-aligned bounding box, and fusing the region waypoints with the Veno map to generate a semantic Veno map, wherein the Veno map is determined based on the raster map.

[0012] In an optional implementation, the step of integrating multiple geometric regions to obtain at least one integrated region carrying a semantic label includes: within each geometric region, selecting an anchor point for the geometric region based on the physical size of the object within the geometric region; assigning a target object within a preset range of the anchor point to the anchor point to determine the initial region corresponding to the geometric region; and merging the different initial regions based on the geometric overlap rate and semantic compatibility of the different initial regions to obtain at least one integrated region carrying a semantic label.

[0013] In an optional implementation, the method further includes: inputting the category, location information and boundary information of objects within the initial region into a large language model to determine the semantic description of the initial region, so as to determine the semantic compatibility of different initial regions based on the semantic description.

[0014] In an optional implementation, the step of generating regional waypoints for each integrated region based on the region center location and axis-aligned bounding box includes: for each integrated region, determining candidate connectivity directions based on the region center location and axis-aligned bounding box of the integrated region; performing visibility detection along the candidate connectivity directions, and determining the intersection points that meet preset conditions and intersect with the corresponding Vinno skeleton in the Vinno diagram as candidate waypoints; for the target connectivity direction that passes the visibility detection, projecting the midpoint of the boundary line corresponding to the target connectivity direction onto the target Vinno skeleton to obtain the regional waypoints.

[0015] The embodiments of this application bring the following beneficial effects: This application provides a multi-task planning method based on semantic Venographs, which enables dynamic task insertion and path updates during the execution of the robot's main task through semantic Venographs, eliminating the need for global replanning and significantly improving task scheduling flexibility and reducing response latency. Simultaneously, by combining cost evaluation for new tasks, the method optimizes the path node sequence, reducing path detours and repetitions, and greatly improving the overall execution efficiency of the robot's multi-task tasks. Compared with existing multi-task planning methods based on semantic Venographs, this method solves the problems of poor task scheduling flexibility, high response latency, and low overall multi-task execution efficiency.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of the multi-task planning method based on semantic Vinograph provided in this application embodiment is shown; Figure 2 A flowchart illustrating the construction steps of the semantic Venn diagram provided in this application embodiment is shown; Figure 3 This illustration shows a schematic diagram of the structure of the multi-task planning device based on semantic Venograph provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of the electronic device provided in the embodiments of this application is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0020] To facilitate understanding of this embodiment, the following uses the example of applying the semantic Venograph-based multi-task planning method provided in this application to a terminal device to illustrate the exemplary steps provided in this application.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multi-task planning method based on semantic Venographs, provided as an embodiment of this application. Figure 1 As shown in the embodiments of this application, the multi-task planning method along the route based on semantic Vinographs includes: Step S101: During the process of the robot executing the first task instruction, in response to at least one second task instruction, the target scene region for executing each second task instruction is determined based on the semantic Venn diagram.

[0022] A semantic Venograph refers to a semantic-geometric fusion navigation map of the global scene. It precisely aligns the Venotopic skeleton (geometrically accessible main roads) of the global scene with hierarchical semantic regions (rooms / functional areas) through region waypoints, supporting both high-level task reasoning and low-level real-time navigation. Semantic Venographs enable robots to achieve lightweight planning, optimal paths, and real-time response when dynamically inserting tasks along the way.

[0023] The first task instruction can refer to the main task instruction that the robot initially receives and is executing, used to control the robot to complete the main objective according to a preset path. For example, the first task instruction can be a movement task from the starting point to a specified target point, or it can be a continuous task such as inspection, delivery, or cleaning that is completed along a fixed route, or a long-term / main process task executed according to the first path node sequence. The first task instruction is used to drive the robot to move along the first path node sequence.

[0024] The first path node sequence includes multiple path nodes that the robot sequentially traverses when executing the first task instruction. Path nodes can refer to nodes in a semantic Vinograph, including regional waypoints and nodes within a region. Regional waypoints can be nodes that are necessary to enter or leave a certain region. The first path node sequence is generated based on the semantic Vinograph before the robot executes the first task instruction. For example, the first task instruction is parsed to determine the starting point, target scene region, and target object corresponding to the first task instruction. Then, regional waypoints and nodes within a region that the robot moves to the target scene region are selected from the semantic Vinograph, and the path nodes are connected in sequence according to the navigation order to form the first path node sequence.

[0025] The following reference Figure 2 This section will introduce the construction process of semantic Vinographs.

[0026] Figure 2 A flowchart illustrating the construction steps of the semantic Venn diagram provided in this application embodiment is shown, as follows: Figure 2 As shown, the steps for constructing a semantic Vinograph include: Step S1011: Construct a raster map based on the 3D semantic point cloud data of the global scene, and perform connected component segmentation on the raster map to determine multiple geometric regions.

[0027] Before determining the target scene region for executing the second task instruction based on the semantic Veno map in response to the second task instruction, the robot scans the global scene using an RGB-D camera (such as Kinect, RealSense) or LiDAR to obtain the raw point cloud data of the global scene. Visual SLAM (such as ORB-SLAM, RTAB-Map) is used to stitch together multiple frames of point cloud data. At the same time, a deep learning semantic segmentation model is used to assign semantic labels to each point in the stitched point cloud data. For example, a point is assigned the semantic label "table". This results in the final 3D semantic point cloud data of the global scene. The 3D semantic point cloud data includes, but is not limited to, the three-dimensional coordinates, semantic labels, color information and instances of each point in the global scene space.

[0028] The 3D semantic point cloud data of the global scene is projected and converted into a 2D raster map.

[0029] In one embodiment, after the raster map is determined, the raster map can be segmented into connected components.

[0030] For example, using connected component analysis algorithms (such as 4-neighbor or 8-neighbor labeling), mutually reachable continuous free spaces are labeled as the same connected component, so as to obtain multiple geometric regions by connected component segmentation of the raster map. Each connected component is a coarse-grained geometric region.

[0031] In one embodiment, after the raster map is determined, a Vinio map can also be constructed based on the raster map.

[0032] First, for each grid point in the grid map, calculate the Euclidean distance (ESDF) from the grid point to the nearest obstacle boundary and assign a sign according to the location: the distance is positive when the grid point is in free space; negative when the grid point is inside an obstacle; and 0 when the grid point is at the obstacle boundary.

[0033] Then, for each target grid point in free space, determine whether the target grid point is a local distance maximum. If it is a distance maximum, then the target grid point is determined to be a Vino skeleton point. For example, select the ESDF distance values ​​of all grids in a 3×3 or 5×5 neighborhood. If the ESDF distance value of the target grid point is greater than or equal to the ESDF distance values ​​of all grid points in the neighborhood, then the target grid point is determined to be a Vino skeleton point.

[0034] Next, the detected local distance maxima points (Vino skeleton points) are connected into continuous polylines according to the 8-neighborhood / 4-neighborhood connectivity rules to obtain the Vino skeleton that is equidistant from the boundaries of all obstacles.

[0035] Finally, by applying local free space consistency constraints, such as eliminating isolated, too short, or disconnected skeleton fragments, we ensure that the skeleton is passable, without breaks, and without dead ends, so as to construct an effective Veno graph. The Veno graph includes vertices V and edges E. Vertex V is the endpoint, bifurcation point, corner point, and intersection point of the Veno skeleton polyline; edge E is the Veno skeleton line segment connecting two vertices.

[0036] Furthermore, to improve navigation efficiency, the Vinograph can be sparsified, retaining its largest connected component as the topological skeleton of the global scene, representing the topological skeleton as a set of polyline paths. Each path This represents a skeletal broken line, used to depict the skeletal structure of free space.

[0037] Step S1012: Integrate multiple geometric regions to obtain at least one integrated region carrying semantic labels.

[0038] To obtain a structured semantic representation, a hierarchical region partitioning method can be used to obtain integrated regions.

[0039] The first step is to select anchor points for each geometric region based on the physical dimensions of the objects within that region, assign target objects within a preset range of the anchor points to the anchor points, and determine the initial region corresponding to that geometric region.

[0040] For example: Calculate the 3D bounding box volume of each object within the geometric region. Sort all objects in descending order of their 3D bounding box volumes. Select the object with the largest 3D bounding box volume as the dominant object (anchor point) for that region. If two objects have similar 3D bounding box volumes, select the one closer to the center of the geometric region as the dominant object. Using the 3D bounding box of the dominant object (anchor point) as a reference, extend upwards in the vertical direction (Z-axis) to form a cylindrical space. All objects within this cylindrical space whose height falls within the vertical coverage area of ​​the anchor point are assigned to that anchor point, forming an initial region.

[0041] The second step involves merging different initial regions based on their geometric overlap rate and semantic compatibility to obtain at least one integrated region carrying semantic labels.

[0042] For example, the IDs, category labels, location information (such as 3D coordinates), and boundary information (such as 3D bounding boxes) of objects within the initial region are input into a large language model. The large language model outputs a functional description (e.g., a cup – used for drinking / eating), spatial relationships, region name, and region description for each object within the initial region, thus determining the semantic description of the initial region. Then, based on the semantic description, it is determined whether different initial regions have semantic compatibility. For example, based on the region name and region description, it is determined whether two initial regions belong to the same semantic category. If they belong to the same semantic category, then the two initial regions are determined to have semantic compatibility.

[0043] The area name is a generic name given based on the function of the objects it contains. The area name must be determined based on the category of the dominant object in that initial area. For example, an area name could be a decorating area, rest area, dining area, work area, entertainment area, storage area, bathroom area, kitchen area, or functional area. The area description describes which key objects the area contains and its overall function.

[0044] The geometric overlap ratio K is calculated using the following formula: ; In the above formula, Initial region The axis is aligned with the bounding box; Initial region The axis is aligned with the bounding box; This is the area calculation function.

[0045] If the geometric coverage is greater than or equal to the set threshold and semantic compatibility is satisfied, the two initial regions are merged to obtain the integrated region; if the geometric coverage is less than the set threshold or semantic compatibility is not satisfied, the initial regions are not merged.

[0046] Step S1013: Based on the region center location and axis-aligned bounding box, generate region waypoints for each integrated region, and merge the region waypoints with the Veno map to generate a semantic Veno map.

[0047] First, for each integrated region, candidate connectivity directions are determined based on the region center location and axis-aligned bounding box of that integrated region.

[0048] For example, the axis-aligned bounding box of the integrated region includes four boundary lines. The midpoint of each boundary line is selected in pixel space as a point to determine the candidate connectivity direction. The center position of the integrated region is used as another point for the candidate connectivity direction. The ray directions extending from the center position of the region to the midpoint of each boundary line are determined as the candidate connectivity directions.

[0049] Secondly, visibility detection is performed along the candidate connectivity direction, and the intersection points on the ray that meet the preset conditions and intersect with the Vinno skeleton corresponding to the Vinno diagram are determined as candidate path points.

[0050] For example, visibility detection is performed along candidate connectivity directions. If a candidate connectivity direction intersects with the Vino skeleton without crossing obstacles or boundaries, it is determined that the candidate connectivity direction has effective connectivity and meets the preset conditions. If a candidate connectivity direction crosses obstacles or boundaries, or does not intersect with the Vino skeleton, it is determined that the candidate connectivity direction does not have effective connectivity and does not meet the preset conditions. The intersection points along the candidate connectivity direction that meet the preset conditions and intersect with the Vino skeleton corresponding to the Vino diagram are determined as candidate path points.

[0051] Then, for the target connectivity direction that passes through the visibility detection, the midpoint of the boundary line corresponding to the target connectivity direction is projected onto the Vino skeleton to obtain the region path points.

[0052] For example: For each target connection direction, project the midpoint of the boundary line corresponding to the target connection direction onto the Vino skeleton, calculate the distance between the candidate path points on all Vino skeleton line segments in the target connection direction and the midpoint of the boundary line corresponding to the target connection direction, and select the candidate path point closest to the midpoint of the boundary line corresponding to the target connection direction as the regional path point.

[0053] The projection is defined by the following formula: ; ; Among them, clip( ) is used to constrain the projection parameters within the interval [0,1]; a and b represent the two endpoints of a line segment on the Vino skeleton, respectively; This indicates the midpoint of the boundary line.

[0054] In addition, there may be multiple candidate waypoints located on the same Vino skeleton line segment. Therefore, in order to reduce redundancy and improve stability, the candidate waypoints can be further screened.

[0055] For example, when multiple candidate waypoints are located on the same Vino skeleton line segment, the point closest to the regional center of the integrated area is selected from the multiple candidate waypoints located on the same Vino skeleton line segment as the regional waypoint.

[0056] Finally, the selected regional waypoints are incorporated into the Veno graph: if a regional waypoint is within a preset range of an existing vertex V in the Veno graph, then vertex V in the Veno graph is directly reused as the regional waypoint; otherwise, the regional waypoint is inserted into the Veno graph by dividing the corresponding Veno skeleton line segment, that is, the Veno skeleton line segment is cut into two segments at the waypoint location, changing the original one edge into two edges. This results in a semantic navigation graph whose regional semantics are consistent with the traversable topological skeleton, providing support for subsequent path planning and connectivity reasoning.

[0057] In one embodiment, when determining the target scene region and the path points of the target region for executing the second task instruction based on the semantic Veno diagram, the second task instruction can be parsed after receiving the second task instruction to determine the target object corresponding to the second task instruction, and the target scene region and the path points of the target region where the target object is located can be selected in the semantic Veno diagram.

[0058] For example, if the second task instruction is the most recently received task instruction besides the first task instruction, and the second task instruction is to instruct the robot to fetch a water glass from the kitchen, then the water glass can be taken as the target object. The kitchen area where the water glass is located can be selected as the target scene area in the semantic Venno graph. A target area waypoint can be selected from at least one area waypoint in the target scene area of ​​the semantic Venno graph. Here, the area waypoint is a navigation node for entering and exiting the target scene area; for example, an area waypoint can be the entrance point of the target scene area.

[0059] Step S102: Determine whether the target area waypoint of the target scene area is in the first path node sequence corresponding to the first task instruction.

[0060] Traverse the first path node sequence corresponding to the first task instruction and check if there is a path node that matches the target area pass-through point. If the first path node sequence contains a target area pass-through point or the distance between the path node in the first path node sequence and the target area pass-through point is less than a set distance threshold, then it is determined that there is a path node that matches the target area pass-through point.

[0061] If the target region's waypoint is in the first path node sequence, or if there is a path node in the first path node sequence that matches the target region's waypoint, then the second task instruction is a concurrent task and can be accepted. The second task instruction is inserted into the task execution queue under the first task instruction, and a second path node sequence is generated based on the target region's waypoint to execute the second task instruction. In this case, only child node sequence markers need to be added to the path nodes in the first path node sequence that match the target region's waypoint, and these child node sequence markers are associated with the second path node sequence. There is no need to modify or reset the first path node sequence, thus completing task planning with extremely low computational cost. At this time, the second path node sequence does not include the target region's waypoint.

[0062] If the target area waypoint is not in the first path node sequence and there is no path node in the first path node sequence that matches the target area waypoint, it indicates that the second task instruction is not a follow-the-way task, and step S103 continues to be executed.

[0063] In one embodiment, it is not necessary to consider whether there are path nodes in the first path node sequence that match the path points of the target area, but only whether the path points of the target area of ​​the target scene are in the first path node sequence corresponding to the first task instruction.

[0064] Step S103: If the target area waypoint is not in the first path node sequence, evaluate the cost of executing the second task instruction, update the first path node sequence based on the cost evaluation result, and control the robot to move according to the updated first path node sequence.

[0065] If the target area waypoint is not in the first path node sequence and there is no path node in the first path node sequence that matches the target area waypoint, then the cost of executing the second task instruction is evaluated. The cost includes detour cost and repetition cost. Based on the comparison result of detour cost and repetition cost, the first path node sequence is updated.

[0066] Alternatively, if the pathpoints to the target area are not in the first path node sequence, the cost of executing the second task instruction is evaluated, and the first path node sequence is updated based on the comparison between detour cost and repetition cost.

[0067] The detour cost refers to the detour distance required for the robot to switch from the current path corresponding to the first task instruction to execute the second task instruction and then return to the path. For example, it includes the distance from the robot to the target area passing point from the switch-out node in the first path node sequence, and the distance from the target area passing point to the switch-back node in the first path node sequence.

[0068] A cut-out node refers to a node in the first path node sequence where the robot leaves the main path to execute the second task instruction; a cut-back node refers to a node in the first path node sequence that returns to the main path after completing the second task instruction. Cut-out and cut-back nodes can be selected based on the distance between each node in the first path node sequence and the point along the way to the target area.

[0069] The repetition cost refers to the path repetition length caused by delaying the execution of the second task instruction and revisiting the target area waypoint. For example, it includes the distance the robot moves from the last node in the first path node sequence to the target area waypoint and the distance the target area waypoint returns to the last node in the first path node sequence. These two target area waypoints may be the same or different.

[0070] If the cost of detour is less than the cost of repetition, then the second task instruction is inserted into the task execution queue under the first task instruction, and at the same time, the second path node sequence corresponding to the second task instruction is inserted into the first path node sequence.

[0071] For example, the second task instruction is bound to a cut-out node in the first path node sequence to insert the second task instruction into the task execution queue under the first task instruction. Simultaneously, the second path node sequence corresponding to the second task instruction is inserted at the cut-out node of the first path node sequence. This generates a local sub-path from the cut-out node → second path node sequence → cut-back node based on a semantic Vinograph. When the robot moves to the cut-out node, the subsequent execution of the first task instruction is paused, and the robot moves according to the second path node sequence to execute the second task instruction. This local sub-path allows for dynamic multi-task response through local planning only, without triggering global replanning. The second path node sequence includes waypoints within the target area.

[0072] If the cost of taking a detour is greater than or equal to the cost of repeating the process, the execution of the second task instruction is postponed until the first task instruction has been completed before the second task instruction is executed.

[0073] It should be noted that when there are multiple second task instructions, each second task instruction performs the same processing flow and updates the first path node sequence to control the robot's movement according to the updated first path node sequence.

[0074] The multi-task planning method along the way based on semantic Vinograph provided in this application has the following beneficial effects: First, it breaks through the limitations of traditional static task planning, allowing robots to flexibly respond to new tasks during the execution of the main task, greatly improving adaptability to dynamic environments.

[0075] Second, it adopts a two-stage lightweight decision-making approach, which only updates local paths and does not require global replanning, significantly reducing computing power consumption and response latency.

[0076] Third, by quantifying and weighing the costs of detours and repetitions, the system intelligently selects the optimal execution time, reducing redundant travel and ineffective detours, and improving the efficiency of multitasking.

[0077] Fourth, by constructing a semantic Venograph, it simultaneously supports high-level task reasoning and low-level safe navigation, thus solving the problem of semantic and topological separation in traditional methods.

[0078] Based on the same inventive concept, this application also provides a semantic Venograph-based multi-task planning device for multi-task planning along the way, which corresponds to the semantic Venograph-based multi-task planning method for multi-task planning along the way. Since the principle of the device in this application is similar to the semantic Venograph-based multi-task planning method for multi-task planning along the way described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0079] Please see Figure 3 , Figure 3 This is a schematic diagram of a multi-task planning device based on semantic Venographs provided in an embodiment of this application. Figure 3 As shown, the semantic Venograph-based multi-task planning device 200 includes: The target determination module 201 is used to determine the target scene region and the target region waypoints for executing each second task instruction based on a semantic Venn diagram in response to a second task instruction during the process of the robot executing at least one first task instruction. The waypoint determination module 202 is used to determine whether the waypoints of the target area are in the first path node sequence corresponding to the first task instruction; The sequence update module 203 is used to evaluate the cost of executing the second task instruction if the target waypoint is not in the first path node sequence, update the first path node sequence based on the cost evaluation result, and control the robot to move according to the updated first path node sequence.

[0080] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0081] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate via the bus 330. When the machine-readable instructions are executed by the processor 310, they can perform the operations described above. Figure 1 The steps of the multi-task planning method based on semantic Venographs in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0082] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the multi-task planning method based on semantic Venographs in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0083] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-task planning method along a route based on semantic Vinographs, characterized in that, include: During the execution of the first task instruction by the robot, in response to at least one second task instruction, the target scene region and the path points of the target region are determined based on the semantic Venn diagram. Determine whether the path points of the target area are in the first path node sequence corresponding to the first task instruction; If the target area waypoint is not in the first path node sequence, the cost of executing the second task instruction is evaluated, the first path node sequence is updated based on the cost evaluation result, and the robot is controlled to move according to the updated first path node sequence.

2. The method according to claim 1, characterized in that, The first path node sequence is updated based on the cost evaluation results in the following manner: Determine the detour cost and repetition cost corresponding to the second task instruction. The detour cost refers to the detour distance required for the robot to cut off from the current path to execute the second task instruction and then return to the path. The repetition cost refers to the path repetition length caused by delaying the execution of the second task instruction and revisiting the path point of the target area. Based on the comparison between the detour cost and the repetition cost, the first path node sequence is updated.

3. The method according to claim 2, characterized in that, The step of updating the first path node sequence based on the comparison result of the detour cost and the repetition cost includes: If the detour cost is less than the duplication cost, then the second task instruction is inserted into the task execution queue under the first task instruction, and the second path node sequence corresponding to the second task instruction is inserted into the first path node sequence.

4. The method according to claim 3, characterized in that, The method further includes: If the detour cost is greater than or equal to the repetition cost, then the execution of the second task instruction is postponed.

5. The method according to claim 1, characterized in that, The method further includes: If the target area's transit point is in the first path node sequence, then the second task instruction is directly inserted into the task execution queue under the first task instruction.

6. The method according to claim 1, characterized in that, The step of determining the target scene region and the path points of the target region for executing each second task instruction based on the semantic Venn diagram includes: The second task instruction is parsed to determine the target object; In the semantic Veno diagram, the target scene region where the target object is located is selected, and the regional waypoints of the target scene region are determined as target region waypoints.

7. The method according to claim 1, characterized in that, Before determining the target scene region and the path points of the target region for executing each second task instruction based on the semantic Venn diagram in response to at least one second task instruction, the method further includes: A raster map is constructed based on 3D semantic point cloud data of the global scene, and the raster map is segmented by connected components to determine multiple geometric regions. The multiple geometric regions are integrated to obtain at least one integrated region carrying semantic labels; Based on the region center location and axis-aligned bounding box, region waypoints are generated for each integrated region. The region waypoints are then fused with the Veno map to generate a semantic Veno map, which is determined based on the raster map.

8. The method according to claim 7, characterized in that, The step of integrating the multiple geometric regions to obtain at least one integrated region carrying a semantic label includes: Within each geometric region, anchor points are selected based on the physical dimensions of the objects within that region. Assign the target object within the preset range of the anchor point to the anchor point, and determine the initial region corresponding to the geometric region; Based on the geometric overlap rate and semantic compatibility of different initial regions, different initial regions are merged to obtain at least one integrated region carrying semantic labels.

9. The method according to claim 8, characterized in that, The method further includes: The category, location information, and boundary information of objects within the initial region are input into the large language model to determine the semantic description of the initial region, and the semantic compatibility of different initial regions is determined based on the semantic description.

10. The method according to claim 7, characterized in that, The step of generating region waypoints for each integrated region based on the region center location and axis-aligned bounding box includes: For each integrated region, candidate connectivity directions are determined based on the region center location and axis-aligned bounding box of that integrated region; Visibility detection is performed along the candidate connectivity direction, and the intersection points that meet the preset conditions and intersect with the Vinno skeleton corresponding to the Vinno diagram are determined as candidate path points. For the target connectivity direction detected by visibility, the midpoint of the boundary line corresponding to the target connectivity direction is projected onto the target Vino skeleton to obtain the region path point.

Images