Method for trajectory optimization of a robot arm based on model memory data mapping and related device

Abstract

The application provides a mechanical arm trajectory optimization method based on model memory data mapping and related equipment, and relates to the technical field of robot control. The application uses a visual language model to quickly identify and position the object to be worked on in the current work scene, quickly understand the expected work task issued by the user for the target mechanical arm, and then combine the model memory data mapping technology, the relationship key point constraint technology and the diffusion strategy technology in the mechanical arm trajectory optimization process. The expected mechanical arm trajectory required for the target mechanical arm to execute the expected work task on the object to be worked on in the current work scene is quickly and accurately optimized, so as to effectively improve the trajectory optimization precision, reduce the trajectory optimization time consumption and the trajectory optimization difficulty, improve the trajectory optimization real-time performance, achieve the synchronous improvement effect of the trajectory optimization real-time performance and the trajectory optimization precision, and ensure that the corresponding mechanical arm can more flexibly adapt to different work requirements of different objects.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a method and related equipment for optimizing the trajectory of a robotic arm based on model memory data mapping. Background Technology

[0002] With the continuous development of science and technology, robotics has received widespread attention from various industries due to its immense research and application value. Intelligent robotic arms are a significant application area for robotics in industries such as industrial production, biomedicine, and services. In practical applications, robotic arm trajectory optimization is typically required to ensure that the robotic arm, following its optimized trajectory, avoids obstacles in the working environment and performs desired tasks on specific objects (e.g., object grasping, object handling, workpiece welding, etc.).

[0003] Currently, the industry's commonly used robotic arm trajectory optimization solutions mainly employ reinforcement learning techniques, which typically require high computational complexity and time costs. Furthermore, the corresponding model prediction accuracy is not high, making it difficult to achieve an effective balance between real-time performance and trajectory optimization accuracy. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a robotic arm trajectory optimization method, computer device and readable storage medium based on model memory data mapping. It can effectively improve trajectory optimization accuracy, reduce trajectory optimization time and difficulty, and improve trajectory optimization real-time performance by organically combining model memory data mapping technology, relational keypoint constraints (ReKep) technology and diffusion policy technology during the robotic arm trajectory optimization process. This achieves the effect of simultaneously improving trajectory optimization real-time performance and trajectory optimization accuracy, ensuring that the corresponding robotic arm can more flexibly adapt to different operational needs of different items.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0006] In a first aspect, this application provides a method for optimizing the trajectory of a robotic arm based on model memory data mapping, the method comprising:

[0007] Obtain natural language control instructions for the target robotic arm, and call the visual language model to parse the natural language control instructions to determine the object to be operated and the expected operation task of the target robotic arm in the current operation scenario.

[0008] Based on the object observation image of the object to be operated in the current operation scene, the task execution start image of the target reference object that matches the category of the object to be operated is searched in the target model memory data of the target robotic arm;

[0009] The effective operation key point starting position of the target reference object in the task execution start image is mapped to the object observation image to obtain the estimated operation key point starting position of the object to be operated in the object observation image;

[0010] Based on the effective operation key point position change data of the target reference object during the execution of the expected operation task recorded by the target model memory data, and based on the estimated starting position of the operation key point and the end point position of the robotic arm in the object observation image, the relationship key point constraint conditions of the target robotic arm when performing the expected operation task for the object to be operated are constructed.

[0011] Based on the constraints of the key relationship points, the diffusion strategy model is invoked to optimize the original robotic arm trajectory data of the target robotic arm in the expected task execution process of the target reference object, which is recorded in the memory data of the target model, so as to obtain the expected robotic arm trajectory required by the target robotic arm to perform the expected task for the object to be worked on in the current work scenario.

[0012] In an optional implementation, the target model memory data records the initial execution images of multiple reference object samples during different tasks performed by the target robotic arm. The step of searching the target model memory data of the target robotic arm for the initial execution image of a target reference object matching the category of the object to be worked, based on the object observation image of the object to be worked in the current work scenario, includes:

[0013] The initial execution images of each of the reference object samples recorded in the target model memory data are matched with the object observation images.

[0014] The initial execution image with the highest image similarity is used as the starting image for task execution, and the reference object sample corresponding to the initial execution image is used as the target reference object.

[0015] In an optional implementation, if all key feature points and their feature descriptors are pre-detected in the target model memory data in the task execution start image, then the step of mapping the effective operation key point start position of the target reference object in the task execution start image to the object observation image to obtain the estimated operation key point start position of the object to be operated in the object observation image includes:

[0016] The scale-invariant feature transformation matching algorithm is used to detect feature points in the object observation image to obtain all key feature points and their feature descriptors in the object observation image.

[0017] Based on the feature descriptors of all key feature points included in the object observation image and the task execution start image, the best matching key feature point pair between the object observation image and the task execution start image is determined using a fast approximate nearest neighbor matching algorithm;

[0018] Based on all the identified key feature point pairs, construct the feature point position mapping relationship between the task execution start image and the object observation image;

[0019] The image position transformation is performed on the effective operation key point starting position in the task execution start image using the feature point position mapping relationship to obtain the estimated operation key point starting position in the object observation image.

[0020] In an optional implementation, the step of constructing relational key point constraints for the target robotic arm to perform the desired task on the object based on the effective operational key point position change data of the target reference object recorded by the target model memory data during the execution of the desired task, and based on the estimated starting position of the operational key point and the position of the robotic arm end-effector key point in the object observation image, includes:

[0021] Based on the feature point position mapping relationship and the first relative positional relationship between the effective operation key point position change data and the effective operation key point starting position, the effective operation key point position change data is transformed by position mapping to obtain the estimated operation key point position change data of the object to be operated during the expected operation task execution process; wherein the second relative positional relationship between the estimated operation key point position change data and the estimated operation key point starting position is consistent with the first relative positional relationship;

[0022] Based on the estimated operation key point position change data, the estimated operation key point start position and the robotic arm end point position, construct the relationship key point constraint function between all estimated operation key points of the object to be operated and the robotic arm end point of the target robotic arm, corresponding to the expected operation task, and use the constructed relationship key point constraint function as the relationship key point constraint condition.

[0023] In an optional implementation, the step of optimizing the original robotic arm trajectory data of the target robotic arm during the expected task execution process on the target reference object, as recorded in the target model memory data, based on the relational key point constraints, by invoking the diffusion strategy model, to obtain the expected robotic arm trajectory required for the target robotic arm to perform the expected task on the object to be worked on in the current work scenario, includes:

[0024] Based on the constraints of the key relationship points, the pose of the robotic arm end effector is predicted to obtain the estimated pose distribution data of the target robotic arm end effector during the execution of the expected task of the object to be worked on.

[0025] Random noise was added to the original robotic arm trajectory data to obtain multiple noisy trajectories;

[0026] For each noise trajectory, the estimated pose distribution data of the robotic arm end effector is used as a denoising constraint. The diffusion strategy model is called to perform reverse denoising on the noise trajectory to obtain the optimized robotic arm trajectory corresponding to the noise trajectory.

[0027] The trajectory quality of each of the multiple optimized robotic arm trajectories is evaluated, and the optimized robotic arm trajectory with the best corresponding trajectory quality is selected as the desired robotic arm trajectory.

[0028] In an optional implementation, the method further includes:

[0029] The object observation image of the object to be operated is used as an initial execution image of the task and the data is entered into the target model memory data. The estimated starting position of the key point of the operation in the object observation image is used as the effective starting position of the key point of the operation of the object to be operated.

[0030] The estimated changes in the position of key points of the object to be worked on during the execution of the expected task are recorded as valid changes in the position of key points of the object to be worked on and entered into the target model memory data.

[0031] In an optional implementation, the method further includes:

[0032] Obtain the original operation video and original robot arm trajectory data of the target robot arm performing different tasks for multiple reference object samples respectively;

[0033] For each original operation video, determine the starting position of the effective operation key point of the corresponding reference object sample in the initial execution image of the task involved in the original operation video, and the change data of the effective operation key point position of the corresponding reference object sample during the execution of the operation task represented by the original operation video.

[0034] Based on the initial execution image of the task involved in each of the original operation videos and the starting position of its effective operation key points, the position change data of the effective operation key points of each of the original operation videos, and the original robotic arm trajectory data corresponding to each of the original operation videos, model memory data is constructed to obtain the target model memory data of the target robotic arm.

[0035] In an optional implementation, for each original work video, the step of determining the starting position of the effective work key point of the corresponding reference object sample in the initial execution image of the task involved in the original work video, and the change data of the position of the effective work key point of the corresponding reference object sample during the execution of the work task represented by the original work video, includes:

[0036] The original operation video was subjected to image preprocessing and keyframe image extraction processing in sequence to obtain the initial execution image of the task and multiple task execution process images involved in the original operation video;

[0037] The scale-invariant feature transformation matching algorithm is used to detect all key feature points and their feature descriptors of the initial execution image of the corresponding task and multiple task execution process images;

[0038] Based on the feature point positions of all key feature points in the initial execution image and multiple execution process images of the corresponding task, the task key points are labeled in the initial execution image and multiple execution process images respectively. This yields the starting position of the effective task key point of the corresponding reference object sample in the initial execution image of the task, as well as the change data of the effective task key point position of the corresponding reference object sample in the task execution process represented by the original task video. The change data of the effective task key point position consists of the distribution positions of the effective task key points matched with the corresponding reference object sample in each of the multiple execution process images.

[0039] Secondly, this application provides a computer device, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the robotic arm trajectory optimization method based on model memory data mapping as described in any of the foregoing embodiments.

[0040] Thirdly, this application provides a readable storage medium storing a computer program thereon, which, when executed by a computer device, implements the robotic arm trajectory optimization method based on model memory data mapping as described in any of the foregoing embodiments.

[0041] In this case, the beneficial effects of the embodiments of this application may include the following:

[0042] This application utilizes a visual language model to quickly identify and locate the object to be worked on in the current work scenario, and rapidly understands the user's expected work task for the target robotic arm. Then, it maps the starting positions of the effective work key points of the target reference object matching the type of the object to be worked on from the target model memory data of the target robotic arm onto the object to be worked on, thereby determining the estimated starting positions of the work key points of the object in the object observation image. This, combined with the change data of the effective work key point positions of the target reference object recorded in the target model memory data during the execution of the expected work task, constructs the relational key point constraints of the target robotic arm when performing the expected work task on the object in the current work scenario. Finally, it uses the target robotic arm recorded in the target model memory data... Using the original robotic arm trajectory data during the expected task execution process of the target reference object as the reference trajectory, the diffusion strategy model is called for trajectory optimization to reduce the processing time and difficulty of robotic arm trajectory generation and improve the accuracy of robotic arm trajectory optimization. Finally, the expected robotic arm trajectory that meets the relationship key point constraint conditions is obtained. Thus, through the organic combination of model memory data mapping technology, relationship key point constraint technology and diffusion strategy technology in the robotic arm trajectory optimization process, the trajectory optimization accuracy is effectively improved, the trajectory optimization time and difficulty are reduced, and the trajectory optimization real-time performance is improved. This achieves the effect of simultaneously improving the trajectory optimization real-time performance and trajectory optimization accuracy, ensuring that the corresponding robotic arm can more flexibly adapt to the different operation requirements of different items.

[0043] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A schematic diagram of the composition of a computer device provided in the embodiments of this application;

[0046] Figure 2 One of the flowcharts for the robotic arm trajectory optimization method based on model memory data mapping provided in this application embodiment;

[0047] Figure 3 for Figure 2 A flowchart illustrating the sub-steps included in step S230;

[0048] Figure 4 for Figure 2 A flowchart illustrating the sub-steps included in step S240;

[0049] Figure 5 for Figure 2 A flowchart illustrating the sub-steps included in step S250;

[0050] Figure 6 The second flowchart illustrates the robotic arm trajectory optimization method based on model memory data mapping provided in this application embodiment.

[0051] Figure 7 The third flowchart illustrates the robotic arm trajectory optimization method based on model memory data mapping provided in this application embodiment.

[0052] Figure 8 for Figure 7 A flowchart illustrating the sub-steps included in step S320.

[0053] Icons: 10-Computer equipment; 11-Memory; 12-Processor; 13-Communication unit. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0056] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0057] In the description of this application, it should be understood that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are used only for the convenience of describing this application and simplifying the description, and are not intended to indicate or imply that the equipment or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0058] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0059] Furthermore, it is understood in the description of this application that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances.

[0060] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0061] Please refer to Figure 1 , Figure 1This is a schematic diagram of the device composition of the computer device 10 provided in this application embodiment. In this application embodiment, the computer device 10 can optimize the trajectory of any intelligent robotic arm to perform different tasks on different work objects in any working environment, so as to ensure that each robotic arm can perform the expected work task on the corresponding work object according to the corresponding optimized robotic arm trajectory. The computer device 10 can be integrated with the intelligent robotic arm, or it can be an electronic device independent of the intelligent robotic arm. The electronic device can be, but is not limited to, a server, a personal computer, a laptop computer, etc.

[0062] In this embodiment of the application, the computer device 10 may include a memory 11, a processor 12, and a communication unit 13. The memory 11, the processor 12, and the communication unit 13 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0063] In this embodiment, the memory 11 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 11 is used to store computer programs, and the processor 12 can execute the computer programs accordingly after receiving execution instructions.

[0064] Furthermore, the memory 11 can also be used to store target model memory data for various intelligent robotic arms. Each intelligent robotic arm corresponds to a unique joint composition, and the joint compositions of different intelligent robotic arms are different. The target model memory data of any intelligent robotic arm records the initial execution images of multiple reference object samples during different tasks of the intelligent robotic arm (i.e., the object sample image of the corresponding reference object sample before the execution of the corresponding task), the effective operation key point position change data of the aforementioned multiple reference object samples during different task executions, and the original robotic arm trajectory data of the intelligent robotic arm during different task executions for each reference object sample. In this context, any initial execution image of a task is marked with the starting position of each of the effective operational key points (i.e., key points that can be acted upon by the corresponding intelligent robotic arm end effector) of the corresponding reference object sample; the change data of the position of the effective operational key points of any reference object sample during the execution of any task consists of the distribution position of each of the effective operational key points of the reference object sample at each operational time point in different task sub-stages of the corresponding task; and the original robotic arm trajectory data during the execution of any task for any reference object sample includes the robotic arm trajectory of the corresponding intelligent robotic arm for the reference object sample in different task sub-stages of the corresponding task, wherein the robotic arm trajectory can be represented by a three-dimensional spatial trajectory function; for any intelligent robotic arm, the same original robotic arm trajectory data is used when performing the same task on multiple reference object samples of the same object type.

[0065] In this embodiment, the processor 12 can be an integrated circuit chip with signal processing capabilities. The processor 12 can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment.

[0066] In this embodiment, the communication unit 13 is used to establish a communication connection between the computer device 10 and other electronic devices via a network, and to send and receive data via the network, wherein the network includes wired communication networks and wireless communication networks. For example, the computer device 10 can communicate with a task execution monitoring device through the communication unit 13 to obtain object observation images taken by the corresponding task execution monitoring device for the object to be worked on. The task execution monitoring device can be deployed in the robot working scenario where the intelligent robotic arm is located, or it can be directly deployed on the intelligent robotic arm near the end of the robotic arm.

[0067] In this embodiment of the application, the computer device 10 may pre-store a specific computer program related to the robotic arm trajectory optimization function in the memory 11, and by driving the processor 12 to execute the specific computer program, the trajectory optimization process is carried out through the organic combination of model memory data mapping technology, relation key point constraint technology and diffusion strategy technology, which effectively improves the trajectory optimization accuracy, reduces the trajectory optimization time and difficulty, and improves the trajectory optimization real-time performance. This achieves the effect of simultaneously improving the trajectory optimization real-time performance and trajectory optimization accuracy, ensuring that the corresponding robotic arm can more flexibly adapt to the different operational needs of different items.

[0068] Understandable, Figure 1 The block diagram shown is only a schematic diagram of one configuration of the computer device 10. The computer device 10 may also include components such as... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0069] In this application, to ensure that the computer device 10 can achieve fast and accurate robotic arm trajectory optimization for any type of intelligent robotic arm, and to simultaneously improve the real-time performance and accuracy of trajectory optimization, ensuring that the corresponding robotic arm can more flexibly adapt to the different operational needs of different items, this application provides a robotic arm trajectory optimization method based on model memory data mapping to achieve the aforementioned objective. The robotic arm trajectory optimization method based on model memory data mapping provided in this application will be described in detail below.

[0070] Please refer to Figure 2 , Figure 2 This is one of the flowcharts illustrating a robotic arm trajectory optimization method based on model memory data mapping provided in this application embodiment. In this application embodiment, the robotic arm trajectory optimization method may include steps S210 to S250.

[0071] Step S210: Obtain the natural language control instructions for the target robotic arm, and call the visual language model to parse the natural language control instructions to determine the object to be operated and the expected operation task of the target robotic arm in the current operation scenario.

[0072] In this embodiment, the computer device 10 can parse the natural language control instructions by calling a visual language model to determine the expected task that the target robotic arm needs to perform and the target task object that the target robotic arm is currently processing. Then, it can search for the target task object in the object observation image collected by the task execution monitoring device for the current work scene to determine the object to be worked on by the target robotic arm in the current work scene, thereby realizing the rapid identification and positioning of the object to be worked on and quickly understanding the expected task given by the user to the target robotic arm.

[0073] Step S220: Based on the object observation image of the object to be operated in the current operation scene, search for the task execution start image of the target reference object that matches the category of the object to be operated in the target model memory data of the target robotic arm.

[0074] In this embodiment, the computer device 10 can obtain the task execution start image of a target reference object that matches the category of the object to be worked by searching for the task execution start image most similar to the observed image of the object in the target model memory data of the target robotic arm. Since the target model memory data records the task execution start images of multiple reference object samples during different task execution processes of the target robotic arm, step 220 may include sub-steps A and B to quickly and accurately find the task execution start image most suitable for the object to be worked.

[0075] Sub-step A: Perform image similarity matching between the initial execution images of all reference object samples recorded in the target model's memory data and the object observation images.

[0076] In one embodiment of this invention, the computer device 10 can implement image similarity matching by calling a large visual model. This large visual model uses a contrastive loss function to measure the feature similarity between two input images and ultimately finds the initial task execution image in the target model's memory data that is most similar to the observed image of the object to be processed. In this process, assuming the feature vectors of the two input images x1 and x2 are f(x1) and f(x2) respectively, and a label y is used to indicate whether the two images match (y = 1 when they match, and y = 0 when they do not), the contrastive loss function can be expressed as: Where D = ||f(x1)-f(x2)||2, D represents the Euclidean distance between the two feature vectors, and m represents the preset lower limit of the distance between mismatched images.

[0077] Sub-step B: Take the initial execution image of the task with the highest image similarity as the starting image of the task execution, and take the reference object sample corresponding to the initial execution image of the task as the target reference object.

[0078] Therefore, by executing the above sub-steps A to B, this application can quickly and accurately find the task execution start image that best matches the object to be worked on.

[0079] Step S230: Map the starting position of the effective operation key point of the target reference object in the task execution start image to the object observation image to obtain the estimated starting position of the operation key point of the object to be operated in the object observation image.

[0080] Alternatively, please refer to Figure 3 , Figure 3 yes Figure 2 A flowchart illustrating the sub-steps included in step S230. In this embodiment of the application, step S230 may include sub-steps S231 to S234, which accurately determine the starting positions (i.e., the starting positions of the estimated operation key points) of all the estimated operation key points (i.e., the predicted key points that can be acted upon by the end effector of the target robotic arm) that should be marked in the corresponding object observation image of the object to be operated on by mapping the positions of the effective operation key points in the model memory data.

[0081] Sub-step S231: Use the scale-invariant feature transformation matching algorithm to detect feature points in the object observation image, and obtain all key feature points and their feature descriptors in the object observation image.

[0082] Sub-step S232: Based on the feature descriptors of all key feature points included in the object observation image and the task execution start image, use the fast approximate nearest neighbor matching algorithm to determine the most matching key feature point pair between the object observation image and the task execution start image.

[0083] In this embodiment, before labeling the starting position of effective task key points in each task initial execution image recorded by the target model memory data, all key feature points and their feature descriptors in the corresponding task initial execution image are detected in advance using the SIFT (Scale Invariant Feature Transform) matching algorithm; each key feature point pair consists of a key feature point in the object observation image and a key feature point in the task execution start image.

[0084] Sub-step S233: Based on all the determined key feature point pairs, construct the feature point position mapping relationship between the task execution start image and the object observation image.

[0085] In this embodiment, the feature point position mapping relationship can be represented by an affine transformation matrix and a perspective transformation matrix.

[0086] Sub-step S234: Using the feature point position mapping relationship, perform image position transformation on the starting position of the effective operation key points in the task execution start image to obtain the estimated starting position of the operation key points in the object observation image.

[0087] Therefore, by executing the above sub-steps S231 to S234, this application can accurately determine the starting positions of all the estimated key points of the operation that should be marked in the corresponding object observation image by mapping the positions of the effective key points of the operation in the model memory data.

[0088] Step S240: Based on the target model memory data recording the effective operation key point position change data of the target reference object during the expected operation task execution process, and based on the estimated operation key point start position and the robot arm end key point position in the object observation image, construct the relationship key point constraint conditions when the target robot arm performs the expected operation task for the object to be operated.

[0089] In this embodiment, the computer device 10 can search for effective operational key point position change data corresponding to the expected task and the target reference object in the target model memory data of the target robotic arm. Then, based on the found effective operational key point position change data, and in conjunction with the feature point position mapping relationship between the task execution start image and the object observation image, it constructs the estimated operational key point position change data of the object to be operated during the execution of the expected task. Then, combined with the estimated operational key point start position of the object to be operated in the object observation image, and the position of the robotic arm end cap key point of the target robotic arm in the object observation image, it uses the relational key point constraint technology to deduce the relational key point constraint conditions between all the estimated operational key points of the object to be operated and the robotic arm end cap key points of the target robotic arm, corresponding to the expected task. The relational key point constraint conditions include the stage target constraint conditions and movement path constraint conditions that each of the different task sub-stages of the expected task needs to satisfy, so as to characterize the spatial relationship between all the estimated operational key points and the robotic arm end cap key points in the time dimension.

[0090] Alternatively, please refer to Figure 4 , Figure 4 yes Figure 2A flowchart illustrating the sub-steps included in step S240. In this embodiment of the application, step S240 may include sub-steps S241 to S242, to deduce the relevant constraints of the target robotic arm performing the desired task on the object to be worked in the current working scenario using relational key point constraint technology.

[0091] Sub-step S241: Based on the feature point position mapping relationship and the first relative position relationship between the effective operation key point position change data and the effective operation key point starting position, perform position mapping transformation on the effective operation key point position change data to obtain the estimated operation key point position change data of the object to be operated during the expected operation task execution process.

[0092] In this embodiment, the first relative positional relationship may include the relative positional relationship between the starting position of each of the effective operational key points of the target reference object and the distribution position of the effective operational key points in different task sub-stages of the expected task; the second relative positional relationship between the estimated operational key point position change data and the estimated operational key point starting position is consistent with the first relative positional relationship, wherein the second relative positional relationship includes the relative positional relationship between the starting position of each of the estimated operational key points of the object to be operated and the distribution position of the estimated operational key points in different task sub-stages of the expected task, wherein the estimated operational key point position change data consists of the approximate estimated operational key point distribution positions of each of the estimated operational key points of the object to be operated at each operation time point in different task sub-stages of the expected task.

[0093] Sub-step S242: Based on the estimated operation key point position change data, the estimated operation key point start position and the robot arm end point position, construct the relationship key point constraint function between all the estimated operation key points of the object to be operated and the robot arm end point of the target robot arm, and use the constructed relationship key point constraint function as the relationship key point constraint condition.

[0094] Therefore, by executing the above sub-steps S241 to S242, this application can deduce the relevant constraints of the target robotic arm performing the desired task on the object to be worked in the current working scenario using relational key point constraint technology.

[0095] Step S250: Based on the key relationship constraints, the diffusion strategy model is invoked to optimize the original robotic arm trajectory data of the target robotic arm in the expected task execution process of the target reference object, which is recorded in the target model memory data, so as to obtain the expected robotic arm trajectory required by the target robotic arm to perform the expected task for the object to be worked on in the current work scenario.

[0096] In this embodiment, the computer device 10 can use the original robotic arm trajectory data of the target robotic arm recorded by the target model memory data for the target reference object and the desired task as a reference trajectory, and use the combination of diffusion strategy technology and relation key point constraint technology to realize the robotic arm trajectory optimization function, so as to reduce the processing time and difficulty of the robotic arm trajectory generation problem, improve the accuracy of robotic arm trajectory optimization, and finally obtain the desired robotic arm trajectory that satisfies the relation key point constraint conditions.

[0097] Alternatively, please refer to Figure 5 , Figure 5 yes Figure 2 A flowchart illustrating the sub-steps included in step S250. In this embodiment, step S250 may include sub-steps S251 to S254 to quickly and accurately optimize the desired robotic arm trajectory to fit the target robotic arm, the object to be worked on, and the desired task.

[0098] Sub-step S251: Based on the constraint conditions of the key points of the relationship, predict the pose of the end effector of the robotic arm to obtain the estimated pose distribution data of the end effector of the target robotic arm during the execution of the expected task of the object to be worked on.

[0099] In this embodiment, the estimated pose distribution data of the robotic arm end effector includes the estimated end effector pose of the target robotic arm at various operation time points within different task sub-stages of the desired task.

[0100] Sub-step S252 involves adding random noise to the original robotic arm trajectory data to obtain multiple noisy trajectories.

[0101] In this embodiment, the specific noise magnitudes of different noise trajectories relative to the original robotic arm trajectory data are different.

[0102] In sub-step S253, for each noise trajectory, the estimated pose distribution data of the robotic arm end effector is used as a denoising constraint, and the diffusion strategy model is called to perform reverse denoising on the noise trajectory to obtain the optimized robotic arm trajectory corresponding to the noise trajectory.

[0103] Sub-step S254: Evaluate the trajectory quality of each of the multiple optimized robotic arm trajectories, and select the optimized robotic arm trajectory with the best trajectory quality as the desired robotic arm trajectory.

[0104] In this embodiment, the trajectory quality indicators involved in the above trajectory quality assessment operation may include, but are not limited to: robotic arm movement time, robotic arm movement space size, specific trajectory length, number of obstacles near the trajectory, and trajectory smoothness.

[0105] Therefore, by executing the above sub-steps S251 to S254, this application can quickly and accurately optimize the desired robotic arm trajectory that is adapted to the target robotic arm, the object to be worked on, and the desired task.

[0106] By executing the above steps S210 to S250, this application effectively improves the trajectory optimization accuracy, reduces the trajectory optimization time and difficulty, and enhances the trajectory optimization real-time performance through the organic combination of model memory data mapping technology, relational key point constraint technology, and diffusion strategy technology during the robotic arm trajectory optimization process. This achieves the effect of simultaneously improving the trajectory optimization real-time performance and trajectory optimization accuracy, ensuring that the corresponding robotic arm can more flexibly adapt to the different operational needs of different items.

[0107] Alternatively, please refer to Figure 6 , Figure 6 This is the second flowchart illustrating the robotic arm trajectory optimization method based on model memory data mapping provided in this application embodiment. In this application embodiment, with... Figure 2 Compared to the robotic arm trajectory optimization method shown, Figure 6 The robotic arm trajectory optimization method shown may also include steps S260 to S270 to effectively improve the data diversity of the target model memory data of the target robotic arm.

[0108] Step S260: The object observation image of the object to be operated on is used as an initial execution image of the task and the data is entered into the target model memory data. The estimated starting position of the key point of the operation in the object observation image is used as the effective starting position of the key point of the operation of the object to be operated on.

[0109] Step S270: The estimated changes in the position of the key points of the object to be worked on during the execution of the expected task are recorded as valid changes in the position of the key points of the task in the target model memory data.

[0110] Therefore, by executing the above steps S260 to S270, this application can continuously improve the diversity of the target model memory data of the target robotic arm by utilizing the model memory data update mechanism.

[0111] Alternatively, please refer to Figure 7 , Figure 7 This is the third flowchart illustrating the robotic arm trajectory optimization method based on model memory data mapping provided in this application embodiment. In this application embodiment, with... Figure 2 or Figure 6 Compared to the robotic arm trajectory optimization method shown, Figure 7 The robotic arm trajectory optimization method shown may also include steps S310 to S330 to construct a suitable target model memory data for any type of intelligent robotic arm.

[0112] Step S310: Obtain the original operation video and original robot arm trajectory data of the target robot arm performing different operation tasks for multiple reference object samples.

[0113] Each original operation video can be acquired by the task execution monitoring device. The original operation video acquired by the computer device 10 should take into account various monitoring angles, lighting conditions and operation backgrounds to increase the model's generalization ability.

[0114] Step S320: For each original job video, determine the starting position of the effective job key point of the corresponding reference object sample in the initial execution image of the task involved in the original job video, and the position change data of the effective job key point of the corresponding reference object sample during the execution of the job task represented by the original job video.

[0115] Alternatively, please refer to Figure 8 , Figure 8 yes Figure 7 A flowchart illustrating the sub-steps included in step S320. In this embodiment of the application, step S320 may include sub-steps S321 to S323 to determine the starting position of the effective operation key point of the corresponding reference object sample at the initial execution image of the task, and the position change data of the effective operation key point of the corresponding reference object sample at the original operation video, respectively, for different original operation videos.

[0116] Sub-step S321 involves sequentially performing image preprocessing and keyframe image extraction on the original job video to obtain the initial execution image of the task and multiple task execution process images involved in the original job video.

[0117] In this embodiment, the image preprocessing operation may include uniform sampling operation and image normalization operation; the initial execution image of the task and multiple task execution process images involved in a single original job video are all key frame images of the original job video, and the multiple task execution process images are key frame images of the corresponding reference object sample at various job time points in different task sub-stages of the corresponding job task.

[0118] Sub-step S322: Use the scale-invariant feature transformation matching algorithm to detect all key feature points and their feature descriptors of the corresponding task initial execution image and multiple task execution process images.

[0119] Sub-step S323: Based on the feature point positions of all key feature points of the corresponding task initial execution image and multiple task execution process images, perform task key point annotation on the task initial execution image and multiple task execution process images respectively, to obtain the starting position of the effective task key point of the corresponding reference object sample in the task initial execution image, and the change data of the effective task key point position of the corresponding reference object sample in the task execution process represented by the original task video.

[0120] The effective operation key point position change data consists of the distribution positions of the effective operation key points matched with the corresponding reference object samples of each of the multiple task execution process images.

[0121] Therefore, by executing the above sub-steps S321 to S323, this application can determine the starting position of the effective operation key point of the corresponding reference object sample at the initial execution image of the task, and the change data of the effective operation key point position of the corresponding reference object sample at the original operation video for different original operation videos.

[0122] Step S330: Based on the initial execution image of the task involved in each of the original operation videos and the starting position of its effective operation key point, the position change data of the effective operation key point of each of the original operation videos, and the original robot arm trajectory data corresponding to each of the original operation videos, model memory data is constructed to obtain the target model memory data of the target robot arm.

[0123] Therefore, by executing the above steps S310 to S330, this application can construct a suitable target model memory data for any kind of intelligent robotic arm.

[0124] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of the apparatus, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0125] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0126] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for optimizing the trajectory of a robotic arm based on model memory data mapping, characterized in that, The method includes: Obtain natural language control instructions for the target robotic arm, and call the visual language model to parse the natural language control instructions to determine the object to be operated and the expected operation task of the target robotic arm in the current operation scenario. Based on the object observation image of the object to be operated in the current operation scene, the task execution start image of the target reference object that matches the category of the object to be operated is searched in the target model memory data of the target robotic arm; The effective operation key point starting position of the target reference object in the task execution start image is mapped to the object observation image to obtain the estimated operation key point starting position of the object to be operated in the object observation image; Based on the effective operational key point position change data of the target reference object recorded in the target model memory data during the execution of the expected task, and based on the estimated starting position of the operational key point and the position of the end-effector key point in the object observation image, relational key point constraints are constructed for the target robotic arm to perform the expected task on the object to be worked. Specifically, based on the feature point position mapping relationship between the task execution start image and the object observation image, and the first relative positional relationship between the effective operational key point position change data and the effective operational key point starting position, a position mapping transformation is performed on the effective operational key point position change data to obtain... The estimated position change data of the key points of the object to be operated during the execution of the expected task is obtained. Then, based on the estimated position change data of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the key points of the target key arm, a relationship key point constraint function corresponding to the expected task is constructed. The constructed relationship key point constraint function is used as the relationship key point constraint condition. The second relative position relationship between the estimated position change data of the key points ... Based on the aforementioned key relationship constraints, a diffusion strategy model is invoked to optimize the original robotic arm trajectory data recorded in the target model memory data during the expected task execution process of the target robotic arm on the target reference object. This yields the desired robotic arm trajectory required for the target robotic arm to perform the expected task on the object to be worked in the current work scenario. Specifically, by predicting the robotic arm end-effector pose based on the aforementioned key relationship constraints, the estimated end-effector pose distribution data of the target robotic arm during the expected task execution process on the object to be worked is obtained. Then, random noise is added to the original robotic arm trajectory data to obtain multiple noisy trajectories. For each noisy trajectory, the estimated end-effector pose distribution data is used as a denoising constraint, and the diffusion strategy model is invoked to perform reverse denoising on that noisy trajectory, resulting in the optimized robotic arm trajectory corresponding to that noisy trajectory. Finally, the trajectory quality of each of the multiple optimized robotic arm trajectories is evaluated, and the optimized robotic arm trajectory with the best corresponding trajectory quality is selected as the desired robotic arm trajectory.

2. The method according to claim 1, characterized in that, The target model memory data records the initial execution images of multiple reference object samples during different tasks performed by the target robotic arm. The step of searching the target model memory data of the target robotic arm for the initial execution image of a target reference object matching the category of the object to be worked, based on the object observation image of the object to be worked in the current work scene, includes: The initial execution images of each of the reference object samples recorded in the target model memory data are matched with the object observation images. The initial execution image with the highest image similarity is used as the starting image for task execution, and the reference object sample corresponding to the initial execution image is used as the target reference object.

3. The method according to claim 1, characterized in that, If all key feature points and their feature descriptors are pre-detected in the target model memory data in the task execution start image, then the step of mapping the effective operation key point start position of the target reference object in the task execution start image to the object observation image to obtain the estimated operation key point start position of the object to be operated in the object observation image includes: The scale-invariant feature transformation matching algorithm is used to detect feature points in the object observation image to obtain all key feature points and their feature descriptors in the object observation image. Based on the feature descriptors of all key feature points included in the object observation image and the task execution start image, the best matching key feature point pair between the object observation image and the task execution start image is determined using a fast approximate nearest neighbor matching algorithm; Based on all the identified key feature point pairs, construct the feature point position mapping relationship between the task execution start image and the object observation image; The image position transformation is performed on the effective operation key point starting position in the task execution start image using the feature point position mapping relationship to obtain the estimated operation key point starting position in the object observation image.

4. The method according to claim 1, characterized in that, The method further includes: The object observation image of the object to be operated is used as an initial execution image of the task and the data is entered into the target model memory data. The estimated starting position of the key point of the operation in the object observation image is used as the effective starting position of the key point of the operation of the object to be operated. The estimated changes in the positions of key points of the object to be worked on during the execution of the expected task are recorded as valid changes in the positions of key points of the object to be worked on and entered into the target model memory data.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the original operation video and original robot arm trajectory data of the target robot arm performing different tasks for multiple reference object samples respectively; For each original operation video, determine the starting position of the effective operation key point of the corresponding reference object sample in the initial execution image of the task involved in the original operation video, and the change data of the position of the effective operation key point of the corresponding reference object sample during the execution of the operation task represented by the original operation video. Based on the initial execution image of the task involved in each of the original operation videos and the starting position of its effective operation key points, the position change data of the effective operation key points of each of the original operation videos, and the original robotic arm trajectory data corresponding to each of the original operation videos, model memory data is constructed to obtain the target model memory data of the target robotic arm.

6. The method according to claim 5, characterized in that, For each original job video, the step of determining the starting position of the effective job key point of the corresponding reference object sample in the initial execution image of the task involved in the original job video, and the change data of the position of the effective job key point of the corresponding reference object sample during the execution of the job task represented by the original job video, includes: The original operation video was subjected to image preprocessing and keyframe image extraction processing in sequence to obtain the initial execution image of the task and multiple task execution process images involved in the original operation video; The scale-invariant feature transformation matching algorithm is used to detect all key feature points and their feature descriptors in the initial execution image of the corresponding task and multiple task execution process images. Based on the feature point positions of all key feature points in the initial execution image and multiple execution process images of the corresponding task, the task key points are labeled in the initial execution image and multiple execution process images respectively. This yields the starting position of the effective task key point of the corresponding reference object sample in the initial execution image of the task, as well as the change data of the effective task key point position of the corresponding reference object sample in the task execution process represented by the original task video. The change data of the effective task key point position consists of the distribution positions of the effective task key points matched with the corresponding reference object sample in each of the multiple execution process images.

7. A computer device, characterized in that, The device includes a processor and a memory, the memory storing a computer program executable by the processor, which can execute the computer program to implement the robotic arm trajectory optimization method based on model memory data mapping as described in any one of claims 1-6.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a computer device, it implements the robotic arm trajectory optimization method based on model memory data mapping as described in any one of claims 1-6.

Images