Mechanical arm path planning method and device based on digital twinning

Abstract

The invention provides a mechanical arm path planning method and device based on digital twinning. The method comprises the steps that a digital twinning space of a mechanical arm is constructed; generating a historical environment model and a corresponding expert track in the digital twin space; acquiring a real environment model; and the real environment model is mapped into the digital twin space, and a planned path of the mechanical arm is generated. According to the method and the device, the expert trajectory is simulated by constructing the digital twin space, and the real environment model is planned, so that the planning experience of the expert trajectory is learned, and a better planning effect is achieved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and more specifically, to a robotic arm path planning method and device based on digital twins. Background Art

[0002] The core issue in current surgical robot control is how to plan the path of the robotic arm. However, current conventional path planning methods have difficulty learning from the planning experience of experts. Summary of the Invention

[0003] The problem that this application solves is that it is difficult for current path planning to learn the planning experience of expert trajectories.

[0004] To solve the above problems, the first aspect of the present application provides a robot arm path planning method based on digital twins, which includes:

[0005] Build a digital twin space for the robotic arm;

[0006] Generate historical environment models and corresponding expert trajectories in the digital twin space;

[0007] Obtain a real environment model;

[0008] Map the real environment model into the digital twin space to generate the planned path of the robotic arm.

[0009] The second aspect of the present application provides a manufacturing system for a robot arm path planning method based on digital twins, which includes:

[0010] The spatial construction module is used to construct the digital twin space of the robotic arm;

[0011] Trajectory generation module, which is used to generate historical environment models and corresponding expert trajectories in the digital twin space;

[0012] A model acquisition module, which is used to obtain a real environment model;

[0013] The path planning module is used to map the real environment model into the digital twin space and generate the planned path of the robotic arm.

[0014] A third aspect of the present application provides an electronic device, comprising: a memory and a processor; the memory may be configured to store a program; the processor is coupled to the memory and is configured to execute the program in the memory, so as to:

[0015] Build a digital twin space for the robotic arm;

[0016] Generate historical environment models and corresponding expert trajectories in the digital twin space;

[0017] Obtain a real environment model;

[0018] Map the real environment model into the digital twin space to generate the planned path of the robotic arm.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the aforementioned digital twin-based robotic arm path planning method.

[0020] In this application, a digital twin space is constructed to simulate expert trajectories and plan the real environment model, thereby learning the planning experience of expert trajectories and achieving better planning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flowchart of a robot arm path planning method based on digital twin according to an embodiment of the present application;

[0022] Figure 2 2 is an architectural diagram of a robot arm path planning device based on digital twin according to an embodiment of the present application;

[0023] Figure 3 2 is a diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, specific embodiments of the present application are described in detail below with reference to the accompanying drawings. Although the accompanying drawings show exemplary embodiments of the present application, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0025] It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the common meanings understood by those skilled in the art in this application.

[0026] The embodiment of the present application provides the above-mentioned robot arm path planning method based on digital twin, the specific scheme of the method is as follows: Figure 1 As shown, the method can be performed by a robot arm path planning device based on digital twins, and the robot arm path planning device based on digital twins can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 As shown, the robot arm path planning method based on digital twin includes:

[0027] S101, building a digital twin space for the robotic arm;

[0028] In this application, a virtual space is constructed, and the robotic arm and environment model are imported into the virtual space to obtain a digital twin space of the robotic arm.

[0029] In this application, the CAD model or URDF file of the robotic arm is imported, and its kinematic parameters (such as joint limits, speed, acceleration, etc.) are configured, and the behavior of the virtual robotic arm is ensured to be consistent with the real robotic arm; thereby modeling the robotic arm.

[0030] S102, generating a historical environment model and corresponding expert trajectory in the digital twin space;

[0031] In this application, the historical environment model refers to the historical data of the intraoperative environment of the robotic arm and the historical data of the intraoperative environment of manual surgery; the robotic arm parameters can be added based on the latter to generate a historical environment model to increase the total amount of historical data of the environment.

[0032] In the present application, the expert trajectory corresponding to the historical environment model can be the trajectory data of the robotic arm controlled by the doctor in real situations, or it can be the trajectory data inferred according to predetermined logic.

[0033] S103, obtaining a real environment model;

[0034] In this application, the real environment model is a three-dimensional model of the environment currently in use. Specifically, it can be obtained by using an RGB-D camera, LiDAR or other sensors to collect point cloud data of the real environment and using SLAM technology to construct a three-dimensional model of the environment.

[0035] S104, mapping the real environment model into the digital twin space to generate a planned path for the robotic arm.

[0036] In this application, a digital twin space is constructed to simulate expert trajectories and plan the real environment model, thereby learning the planning experience of expert trajectories and achieving better planning results.

[0037] In one embodiment, the step S104 of mapping the real environment model into the digital twin space to generate a planned path for the robotic arm includes:

[0038] Mapping the real environment model to the digital twin space;

[0039] Based on the mapped real environment model, similar historical environment models are selected in the digital twin space;

[0040] Based on the expert trajectory corresponding to the historical environment model, the planned path of the robotic arm is generated.

[0041] In this application, information such as the geometric structure and obstacle distribution in the real environment is synchronized to the digital twin space to ensure a high degree of consistency between the virtual environment and the real environment.

[0042] In this application, feature extraction refers to extracting key features of the real environment model, including geometric features (such as obstacle layout and spatial topology) and dynamic features (such as the speed and direction of moving obstacles). Deep learning models (such as PointNet or Graph Neural Networks) can be used to extract high-dimensional features.

[0043] In this application, similarity matching involves searching a historical environment database for the historical environment model that most closely matches the current environment's characteristics. The similarity between the real environment model and the historical environment model is calculated. This comparison is performed using a distance metric (e.g., Euclidean distance, cosine similarity) or a learning-based matching algorithm (e.g., Siamese Network).

[0044] In this application, expert trajectories corresponding to the selected historical environment model are loaded.

[0045] In this application, expert trajectories in the historical environment model are used to generate a robotic arm planning path suitable for the current real environment.

[0046] In this application, efficiency: the computational complexity of path planning is reduced by learning historical environment models and expert trajectories.

[0047] In this application, robustness: combined with the simulation capabilities of the digital twin space, it can effectively deal with dynamic obstacles and complex scenarios.

[0048] In this application, real-time performance: ensures the real-time and adaptability of the path planning process through dynamic update and fast retrieval technology.

[0049] In one embodiment, generating a planned path for the robotic arm based on an expert trajectory corresponding to a historical environment model includes:

[0050] Generate an obstacle density map based on the historical environment model;

[0051] Generate candidate paths based on historical environment models;

[0052] Based on the obstacle dense map, the best path is selected from the candidate paths as the planned path of the robotic arm.

[0053] In this application, a probability occupancy grid reflecting the distribution of obstacles in the current environment is generated as the basis for path planning.

[0054] In this application, multiple candidate paths are generated based on the expert trajectories in the historical environment model: the expert trajectories in the historical environment model are used as the initial reference to generate a basic path, and the expert trajectories are smoothed (such as B-spline curve interpolation) to improve the path quality.

[0055] Preferably, in the present application, multiple expert trajectories corresponding to the historical environment model are obtained, and each expert trajectory is used as a candidate path; and similarity calculation is performed on the expert trajectories, and the expert trajectories whose similarity exceeds a preset threshold are comprehensively processed (for example, averaged) to obtain the corresponding candidate path.

[0056] In this application, the optimal path is selected from the candidate paths as the planned path of the robotic arm.

[0057] In one embodiment, generating an obstacle density map based on a historical environment model includes:

[0058] Obtain moving obstacle information for multiple historical environment models;

[0059] Projecting moving obstacle information into the real environment model;

[0060] The projected moving obstacle information is statistically analyzed, and a probability occupancy map is generated in the real environment model as the obstacle density map.

[0061] In this application, the historical environment database is traversed to extract the mobile obstacle information in each historical environment model.

[0062] In this application, the key features of each moving obstacle are extracted, including: Position: the three-dimensional coordinates of the obstacle in the environment; Speed: the speed of the obstacle (magnitude and direction); Trajectory: the historical movement path of the obstacle.

[0063] Preferably, moving obstacles are classified (e.g., pedestrians, vehicles, etc.) and their motion modes are annotated (e.g., linear motion, curved motion). Obstacles are detected and classified using a deep learning model (e.g., Mask R-CNN or YOLO).

[0064] In this application, the moving obstacle information in the historical environment model is mapped to the current real environment model. Based on the position and speed of the moving obstacle in the historical environment model, its projected position in the real environment model is calculated. For dynamic obstacles, the future position is predicted by combining their movement direction and speed.

[0065] In this application, for multiple historical environment models, the moving obstacle information in all models is fused.

[0066] In this application, based on the projected mobile obstacle information, a probability occupancy grid reflecting the obstacle distribution is generated: the real environment model is divided into regular grid cells (Grid Cells), and the size of each grid cell can be adjusted according to the accuracy requirements; for each grid cell, the frequency of the projected mobile obstacle covering the area is counted; the occupancy probability of each grid cell is stored as a matrix to form a probability occupancy map. Among them, the higher the frequency of the grid cell being covered by the mobile obstacle, the larger the data in the matrix (which can also be a probability data).

[0067] In one embodiment, selecting the best path from the candidate paths as the planned path of the robotic arm based on the obstacle density map includes:

[0068] Based on the probabilistic occupancy map, the collision probability of each candidate path is calculated;

[0069] Select several candidate paths with the lowest collision probability and simulate them in the real environment model;

[0070] The candidate path passed by the simulation is used as the planning path of the robotic arm.

[0071] In the present application, based on the probability occupancy map, the grid cells passed by each candidate path are counted; based on the matrix / probability data of the passed grid cells, the collision probability of the candidate path is calculated (for example, normalized after accumulation).

[0072] Preferably, the candidate path is discretized into a series of key points. For each sampling point on the path, the probability occupancy value of the grid cell where it is located is queried, and the occupancy probabilities of all sampling points on the path are accumulated as the collision probability of the path.

[0073] In this application, several candidate paths with the lowest collision probability are screened out and simulated and verified in the digital twin space.

[0074] In this application, candidate paths are sorted by collision probability, and the paths with the lowest collision probability are selected. These candidate paths are then simulated in the digital twin space to detect collisions or constraint violations. The simulation includes determining whether collisions occur at each sampling point along the path, and whether joint limits or speed limits are exceeded during the robot's motion. If a path fails the simulation (e.g., collisions or constraint violations), it is excluded from the candidate paths.

[0075] In this application, the optimal path is selected from the candidate paths verified by simulation as the final planned path of the robotic arm.

[0076] Preferably, candidate paths from the simulation are comprehensively evaluated based on the following indicators: Safety: The collision probability on the path is the lowest. Efficiency: The path length is the shortest or the execution time is the shortest. Energy consumption: The energy consumption of the robot movement is the lowest.

[0077] In this application, the candidate path with the highest comprehensive score is selected as the planned path of the robotic arm, and the key points of the path (such as the joint angle sequence) are output and sent to the robotic arm controller.

[0078] In one embodiment, it further includes:

[0079] Monitor the real environment model and map it to the digital twin space in real time;

[0080] Update obstacle information based on the real environment model mapped in real time;

[0081] Based on the updated obstacle information, the planned path of the robotic arm is updated.

[0082] In this application, changes in the real environment are captured in real time and synchronized into the digital twin space.

[0083] Based on real-time mapping data, the obstacle distribution and status in the digital twin space are updated, including:

[0084] Detect new obstacles: Use change detection algorithms (such as background subtraction or deep learning segmentation models) to identify new obstacles.

[0085] Tracking dynamic obstacles: For moving obstacles, the future position is predicted based on their historical trajectory, and Kalman filter or particle filter is used for trajectory prediction.

[0086] Update the probabilistic occupancy map: Based on the information of newly added or moved obstacles, update the probabilistic occupancy map in the digital twin space and recalculate the occupancy probability for each grid cell.

[0087] In this application, path replanning is triggered: when an environmental change is detected (such as a new obstacle or a change in the position of an obstacle), the path replanning process is initiated. A threshold (such as a collision probability exceeding a certain value) is set to trigger replanning.

[0088] In this application, based on the updated obstacle density map, the path planning algorithm is called to generate new candidate paths; the collision probability of each candidate path is calculated, and several paths with the lowest collision probability are screened out; these candidate paths are simulated in the digital twin space to detect whether a collision occurs or exceeds the limit.

[0089] In one embodiment, the step S102 of generating a historical environment model and corresponding expert trajectories in the digital twin space includes:

[0090] Obtain expert trajectories and the historical environment model corresponding to the expert trajectories;

[0091] Mapping historical environment models and expert trajectories into the digital twin space;

[0092] Set multiple optimization goals;

[0093] Generate multiple expert trajectories in a historical environment model based on the optimization objective.

[0094] In this application, the historical environment model and expert trajectory are mapped into the digital twin space to form a virtual scene. The working scene of the robot arm, including obstacles, target objects, etc., is reproduced in the digital twin space.

[0095] In this application, multiple optimization goals are set according to task requirements, such as:

[0096] Shortest path length: Minimize the distance the robot moves from the starting point to the end point.

[0097] Minimum energy consumption: Reduce the energy consumed during the operation of the robot arm.

[0098] Obstacle avoidance capability: Ensures that the robotic arm does not collide with obstacles during operation.

[0099] Time efficiency: Reduce the time required for the robotic arm to complete a task.

[0100] In the digital twin space, multiple different expert trajectories are generated based on the historical environment model and optimization objectives. Each optimization objective or a combination of multiple optimization objectives can generate a corresponding trajectory as the expert trajectory.

[0101] Preferably, the generated expert trajectory is randomly perturbed (such as fine-tuning trajectory points, changing obstacle positions) to generate more diverse training data.

[0102] In this application, by constructing a digital twin space of the robotic arm, obtaining historical environment models and expert trajectories, and combining optimization objectives to generate diverse expert trajectories, high-quality data support can be provided for the training of the strategy model.

[0103] In one embodiment, the historical environment model is a depth image; after obtaining the historical environment model, the historical environment model is also updated; the specific process of the update includes:

[0104] Divide the historical environment model into blocks to obtain independent blocks;

[0105] For each independent block, obtain a first neighboring block and a second neighboring block with different spacings;

[0106] generating a first feature block based on the independent block and the first neighborhood block;

[0107] generating a second feature block based on the independent block and the second neighborhood block;

[0108] Performing feature compression on the first feature block and the second feature block to obtain a compressed block;

[0109] Iterate over all independent blocks and generate an updated historical environment model based on the resulting compressed blocks.

[0110] In this application, the historical environment model is divided into blocks, that is, the historical environment model is divided into corresponding image blocks through a checkerboard grid; wherein, the image block can be at the pixel level (that is, each pixel is an image block) or at other levels, and the specific division shall be based on the actual processing situation.

[0111] In this application, a sliding window or a fixed step size is used to divide the image into blocks of the same size.

[0112] It should be noted that the historical environment model is a three-dimensional image, so one surface is selected for chessboard division, and each grid is a strip grid with a lot of depth (the depth is the depth of the three-dimensional image), and the strip grid is an image block.

[0113] Preferably, in the present application, each image block is 1001000 pixels, so that more feature calculations between local areas can be performed on the basis of ensuring generation accuracy and reducing the amount of calculation.

[0114] In this application, an image block is selected as an independent block. The adjacent image blocks above, below, to the left, and to the right of the independent block are the first neighboring blocks. The image blocks above, below, to the left, and to the right of the independent block, separated by one grid, are the second neighboring blocks. The first and second neighboring blocks have different spacing from the independent block.

[0115] In this application, the neighborhood information of each independent block is extracted to capture the local structure.

[0116] In the present application, the first feature block is generated to generate a local feature representation using an independent block and its first neighborhood block. Specifically, the independent block and the first neighborhood block are processed by a convolution layer and an attention layer to obtain the first feature block.

[0117] In this application, the specific structure and specific parameters of the convolutional layer and the attention layer can be obtained according to the training data or determined according to the actual situation.

[0118] It should be noted that, in this application, there are four first neighborhood blocks and multiple first feature blocks.

[0119] In this application, the independent block and the first neighborhood block are processed by the convolution layer and the attention layer to obtain the first feature block. The specific process is: the independent block and the four neighborhood blocks are spliced ​​together to form a multi-channel input, and the convolution layer is used to extract features from the spliced ​​block; the self-attention mechanism or the channel attention mechanism is used to enhance important features, calculate the attention weight, and weight the convolution layer output to enhance important features; the output of the attention layer is split into multiple feature blocks, each feature block corresponds to the processing results of the independent block and at least one neighborhood block.

[0120] In this application, a second feature block is generated to generate a broader local feature representation using an independent block and its second neighborhood block. The specific generation process is the same as that of the first feature block, except that the parameters of the convolution layer and the attention layer are different.

[0121] In this application, the generated feature blocks are compressed into a more compact representation to reduce the amount of computation and retain key information. Feature compression is performed using pooling operations (such as maximum pooling or average pooling) or fully connected layers.

[0122] In this way, through compression, the first feature blocks and the second feature blocks are compressed into a compressed block, which has the same size and position as the independent block and is used to replace the independent block. All image blocks are replaced by the compressed block to obtain an updated historical environment model.

[0123] In this application, each image block of the historical environment model is traversed by traversal to obtain the corresponding compressed block.

[0124] In this application, for image blocks / independent blocks near edges, their first and second neighboring blocks are incomplete. In this case, they are completed by copying the first and second neighboring blocks at relatively similar positions. For example, if the first neighboring block above the independent block does not exist, the first neighboring block below is copied and used as the block above.

[0125] In this application, the processing accuracy of the adjacent image blocks is greatly improved by completion.

[0126] In this application, the similarity relationship between local regions is captured through an adaptive adjustment module, thereby enhancing feature representation.

[0127] In this application, through updating, on the one hand, the features in the historical environment model are enhanced, and on the other hand, the edge sharpness of the historical environment model is blunted, thereby increasing the difficulty of identifying the historical environment model, thereby artificially adding difficulty to the training process in the entire digital twin space, in order to reversely improve (the real environment model has higher clarity than the historical environment model) the accuracy of the planned path of the robotic arm generated based on the expert trajectory corresponding to the historical environment model.

[0128] An embodiment of the present application provides a robotic arm path planning device based on digital twins, which is used to execute the robotic arm path planning method based on digital twins described above in the present application. The robotic arm path planning device based on digital twins is described in detail below.

[0129] like Figure 2 As shown, the robot arm path planning device based on digital twin includes:

[0130] A space construction module 101 is used to construct a digital twin space of the robotic arm;

[0131] a trajectory generation module 102 for generating a historical environment model and corresponding expert trajectories in the digital twin space;

[0132] A model acquisition module 103 is used to acquire a real environment model;

[0133] The path planning module 104 is used to map the real environment model into the digital twin space and generate a planned path for the robotic arm.

[0134] In one embodiment, the path planning module 104 is further configured to:

[0135] The real environment model is mapped to the digital twin space. Based on the mapped real environment model, a similar historical environment model is selected in the digital twin space. Based on the expert trajectory corresponding to the historical environment model, the planned path of the robotic arm is generated.

[0136] In one embodiment, the path planning module 104 is further configured to:

[0137] Based on the historical environment model, an obstacle density map is generated; based on the historical environment model, candidate paths are generated; based on the obstacle density map, the best path is selected from the candidate paths as the planned path of the robotic arm.

[0138] In one embodiment, the path planning module 104 is further configured to:

[0139] Obtaining mobile obstacle information of multiple historical environment models; projecting the mobile obstacle information into the real environment model; counting the projected mobile obstacle information, and generating a probability occupancy map in the real environment model as the obstacle density map.

[0140] In one embodiment, the path planning module 104 is further configured to:

[0141] Based on the probabilistic occupancy map, the collision probability of each candidate path is calculated; several candidate paths with the lowest collision probability are selected and simulated in the real environment model; the candidate paths that pass the simulation are used as the planned path of the robot arm.

[0142] In one embodiment, the path planning module 104 is further configured to:

[0143] Monitor the real environment model and map it to the digital twin space in real time; update obstacle information based on the real environment model mapped in real time; and update the planned path of the robotic arm based on the updated obstacle information.

[0144] In one embodiment, the trajectory generation module 102 is further configured to:

[0145] Obtain expert trajectories and the historical environment model corresponding to the expert trajectories; map the historical environment model and expert trajectories into the digital twin space; set multiple optimization goals; and generate multiple expert trajectories in the historical environment model based on the optimization goals.

[0146] The digital twin-based robotic arm path planning device provided in the above-mentioned embodiment of the present application has a corresponding relationship with the digital twin-based robotic arm path planning method provided in the embodiment of the present application. Therefore, the specific content in the system has a corresponding relationship with the digital twin-based robotic arm path planning method. The specific content can refer to the records in the digital twin-based robotic arm path planning method, and will not be repeated in this application.

[0147] The digital twin-based robotic arm path planning device provided in the above-mentioned embodiments of the present application and the digital twin-based robotic arm path planning method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0148] The above describes the internal functions and structure of the robot path planning device based on digital twins, such as Figure 3 As shown, in practice, the digital twin-based robotic arm path planning device can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0149] The memory 301 may be configured to store programs.

[0150] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0151] The memory 301 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The processor 303 is coupled to the memory 301 and is used to execute the program in the memory 301 to:

[0152] Build a digital twin space for the robotic arm;

[0153] Generate historical environment models and corresponding expert trajectories in the digital twin space;

[0154] Obtain a real environment model;

[0155] Map the real environment model into the digital twin space to generate the planned path of the robotic arm.

[0156] In one embodiment, the processor 303 is further configured to:

[0157] The real environment model is mapped to the digital twin space. Based on the mapped real environment model, a similar historical environment model is selected in the digital twin space. Based on the expert trajectory corresponding to the historical environment model, the planned path of the robotic arm is generated.

[0158] In one embodiment, the processor 303 is further configured to:

[0159] Based on the historical environment model, an obstacle density map is generated; based on the historical environment model, candidate paths are generated; based on the obstacle density map, the best path is selected from the candidate paths as the planned path of the robotic arm.

[0160] In one embodiment, the processor 303 is further configured to:

[0161] Obtaining mobile obstacle information of multiple historical environment models; projecting the mobile obstacle information into the real environment model; counting the projected mobile obstacle information, and generating a probability occupancy map in the real environment model as the obstacle density map.

[0162] In one embodiment, the processor 303 is further configured to:

[0163] Based on the probabilistic occupancy map, the collision probability of each candidate path is calculated; several candidate paths with the lowest collision probability are selected and simulated in the real environment model; the candidate paths that pass the simulation are used as the planned path of the robot arm.

[0164] In one embodiment, the processor 303 is further configured to:

[0165] Monitor the real environment model and map it to the digital twin space in real time; update obstacle information based on the real environment model mapped in real time; and update the planned path of the robotic arm based on the updated obstacle information.

[0166] In one embodiment, the processor 303 is further configured to:

[0167] Obtain expert trajectories and the historical environment model corresponding to the expert trajectories; map the historical environment model and expert trajectories into the digital twin space; set multiple optimization goals; and generate multiple expert trajectories in the historical environment model based on the optimization goals.

[0168] In this application, Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 Components shown.

[0169] The electronic device provided in this embodiment is based on the same inventive concept as the digital twin-based robotic arm path planning method provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored therein.

[0170] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CDROM, optical storage, etc.) that contain computer-usable program code.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0173] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of a computer-readable medium.

[0174] The present application also provides a computer-readable storage medium corresponding to the digital twin-based robotic arm path planning method provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the interactive image analysis auxiliary method for 3D aerial imaging provided in any of the aforementioned embodiments.

[0175] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CDROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0176] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the interactive image analysis auxiliary method for 3D aerial imaging provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0177] It should be noted that, in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0178] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0179] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A robot arm path planning method based on digital twin, characterized in that: include: Build a digital twin space for the robotic arm; Generate historical environment models and corresponding expert trajectories in the digital twin space; Obtain a real environment model; Map the real environment model into the digital twin space to generate the planned path of the robotic arm.

2. The robot arm path planning method based on digital twin according to claim 1, characterized in that: Mapping the real environment model into the digital twin space to generate the planned path of the robotic arm includes: Mapping the real environment model to the digital twin space; Based on the mapped real environment model, similar historical environment models are selected in the digital twin space; Based on the expert trajectory corresponding to the historical environment model, the planned path of the robotic arm is generated.

3. The robot arm path planning method based on digital twin according to claim 2, characterized in that: The expert trajectory corresponding to the historical environment model is used to generate a planned path for the robotic arm, including: Generate an obstacle density map based on the historical environment model; Generate candidate paths based on historical environment models; Based on the obstacle dense map, the best path is selected from the candidate paths as the planned path of the robotic arm.

4. The robot arm path planning method based on digital twin according to claim 3, characterized in that: The generation of an obstacle density map based on the historical environment model includes: Obtain moving obstacle information for multiple historical environment models; Projecting moving obstacle information into the real environment model; The projected moving obstacle information is statistically analyzed, and a probability occupancy map is generated in the real environment model as the obstacle density map.

5. The robot arm path planning method based on digital twin according to claim 3, characterized in that: The method of selecting the best path from the candidate paths based on the obstacle density map as the planned path of the robotic arm includes: Based on the probabilistic occupancy map, the collision probability of each candidate path is calculated; Select several candidate paths with the lowest collision probability and simulate them in the real environment model; The candidate path passed by the simulation is used as the planning path of the robotic arm.

6. The robot arm path planning method based on digital twin according to any one of claims 1 to 5, characterized in that: Also includes: Monitor the real environment model and map it to the digital twin space in real time; Update obstacle information based on the real environment model mapped in real time; Based on the updated obstacle information, the planned path of the robotic arm is updated.

7. The robot arm path planning method based on digital twin according to any one of claims 1 to 5, characterized in that: The generation of the historical environment model and the corresponding expert trajectory in the digital twin space includes: Obtain expert trajectories and the historical environment model corresponding to the expert trajectories; Mapping historical environment models and expert trajectories into the digital twin space; Set multiple optimization goals; Generate multiple expert trajectories in a historical environment model based on the optimization objective.

8. A robot arm path planning device based on digital twin, characterized in that: include: The spatial construction module is used to construct the digital twin space of the robotic arm; Trajectory generation module, which is used to generate historical environment models and corresponding expert trajectories in the digital twin space; A model acquisition module, which is used to obtain a real environment model; The path planning module is used to map the real environment model into the digital twin space and generate the planned path of the robotic arm.

9. An electronic device, characterized in that: include: memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Build a digital twin space for the robotic arm; Generate historical environment models and corresponding expert trajectories in the digital twin space; Obtain a real environment model; Map the real environment model into the digital twin space to generate the planned path of the robotic arm.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the digital twin-based robotic arm path planning method described in any one of claims 1 to 7.

Images