Cloud edge collaborative dexterous hand tool body intelligent operation method and device and server
By using a cloud-edge collaborative data generation and model training and evaluation system, the problem of dexterous hands being unable to quickly adapt to different products in industrial flexible production has been solved, achieving efficient adaptation and rapid adaptation of dexterous hands in industrial flexible production.
Patent Information
- Application Number
- CN202511716659.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-01-23
AI Technical Summary
Existing dexterous hands cannot be quickly adapted to different products in industrial flexible manufacturing, and their high cost and long cycle time prevent them from being applied to industrial flexible manufacturing.
By adopting a cloud-edge collaboration approach, sample data is acquired through a cloud-side data generation and model training evaluation system to expand the data and train the model, generating an embodied large model. The embodied intelligent model is then distributed to the edge robot via edge hot updates to control the dexterous hand to perform intelligent operations.
It significantly enhances the rapid adaptability of dexterous hands, improves the adaptability and efficiency of flexible industrial production, and reduces equipment replacement costs and time.
Smart Images
Figure CN121374597A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of industrial flexible production, and particularly relates to a cloud-edge collaborative dexterous hand body intelligent operation method and device and a server. BACKGROUND
[0002] At present, with the rapid development of body intelligence, robots and dexterous hands, the dexterous hand is gradually used as an end effector of a robot and a mechanical arm due to the advantages of high degree of freedom and flexible action. According to the related technology, the dexterous hand includes a hardware part and a software part. The hardware part is used to realize the gripping and other operations of the dexterous hand, and the software part integrates the manual operation, pre-programmed control and AI model control functions of the dexterous hand. However, in industrial flexible production, the product type and product model frequently change. The existing dexterous hand cannot be quickly adapted to different products, and has high cost and long cycle, so it cannot be applied to industrial flexible production. SUMMARY
[0003] Therefore, the present application aims to provide a cloud-edge collaborative dexterous hand body intelligent operation method and device and a server, which can significantly improve the rapid adaptation ability of the dexterous hand.
[0004] In a first aspect, an embodiment of the present application provides a cloud-edge collaborative dexterous hand body intelligent operation method. The method is applied to a cloud-side data generation and model training and evaluation system, which is in communication connection with an upper computer of an end-side robot. The end-side robot includes a mechanical arm, a dexterous hand, a third-view camera and the upper computer. The method includes: obtaining sample data, and performing data augmentation processing on the sample data to obtain target sample data. The sample data includes joint state information of the mechanical arm and the dexterous hand, and third-view camera image information, fingertip visual-haptic sensor image information and palm eye visual sensor image information for a product to be operated. The method further includes: performing model training processing on a general model by using the target sample data to obtain a body large model, performing model evaluation processing on the body large model, determining a body large model that passes the evaluation as a target body intelligent model, and downloading the target body intelligent model to the upper computer of the end-side robot to control the dexterous hand to perform intelligent operation processing on the product to be operated.
[0005] In an embodiment, the step of performing data augmentation processing on the sample data to obtain the target sample data includes: performing data augmentation processing on the sample data and a three-dimensional model of the product to be operated by using a preset data generation model and a simulation engine to obtain the target sample data.
[0006] In an implementation, the step of generating target sample data based on sample data and a three-dimensional model of a product to be operated by using a preset data generation model and a simulation engine, comprises: inputting the sample data into the preset data generation model for data augmentation to obtain first augmented data based on the sample data; inputting the three-dimensional model of the product to be operated into the simulation engine to generate second augmented data by randomly replacing surface texture and illumination; and combining the first augmented data and the second augmented data to obtain the target sample data.
[0007] In an implementation, the step of training a general model based on the target sample data to obtain a personalized large model, comprises: pre-training the personalized operation model based on the target sample data and historical data to obtain the general model, and fine-tuning the general model in a new scene corresponding to the target sample data to obtain the personalized large model.
[0008] In an implementation, the step of evaluating the personalized large model and determining the personalized large model that passes the evaluation as the target personalized intelligent model, comprises: evaluating the personalized large model in a simulation environment to obtain an operation success rate of the personalized large model in the new scene; and determining the personalized large model as the target personalized intelligent model when the operation success rate is greater than a preset success rate threshold.
[0009] In an implementation, the method comprises: when the operation success rate is not greater than the preset success rate threshold, re-performing data augmentation to increase the amount of data for model training, and re-performing model evaluation on the trained personalized large model.
[0010] In an implementation, the step of issuing the target personalized intelligent model to an upper computer of an end-side robot to control the dexterous hand to perform intelligent operation on a product to be operated, comprises: replacing the original model of the end-side robot with the target personalized intelligent model through edge hot updating in a non-stop state, so that the upper computer controls the dexterous hand to perform intelligent operation on the product to be operated based on the target personalized intelligent model.
[0011] In a second aspect, the embodiment of the present application further provides a cloud-edge collaborative dexterous hand body intelligent operation device, which is applied to a cloud-side data generation and model training and evaluation system, the cloud-side data generation and model training and evaluation system is in communication connection with an upper computer of an end-side robot, the end-side robot comprises a mechanical arm, a dexterous hand, a third-view camera and the upper computer, and the device comprises: a data expansion module, which acquires sample data and performs data expansion processing on the sample data to obtain target sample data, wherein the sample data comprises joint state information of the mechanical arm and the dexterous hand, and third-view camera image information, fingertip visual and tactile sensor image information and palm eye visual sensor image information for a product to be operated; a model training module, which performs model training processing on a general model by using the target sample data to obtain a body large model, and performs model evaluation processing on the body large model, and determines the body large model that passes the evaluation as a target body intelligent model; and an intelligent operation module, which issues the target body intelligent model to the upper computer of the end-side robot to control the dexterous hand to perform intelligent operation processing on the product to be operated.
[0012] In a third aspect, the embodiment of the present application further provides a server, comprising a processor and a memory, the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method of any one of the first aspect.
[0013] In a fourth aspect, the embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions, and when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to implement the method of any one of the first aspect.
[0014] The embodiment of the present application brings the following beneficial effects: The cloud-edge collaborative dexterous hand body intelligent operation method, device and server provided by the embodiment of the present application can significantly improve the rapid adaptation capability of the dexterous hand.
[0015] Other features and advantages of the present application will be further described in the following description, and some will become apparent from the description, or will be learned through implementation of the present application. The objectives and other advantages of the present application will be realized and obtained by the structure particularly pointed out in the specification, claims and drawings.
[0016] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS
[0017] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.
[0018] Figure 1 A structural schematic diagram of a cloud-side data generation and model training and evaluation system provided by an embodiment of the present application is shown in the figure. Figure 2 A flowchart of a cloud-edge collaborative dexterous hand embodiment intelligent operation method provided by an embodiment of the present application is shown in the figure. Figure 3 A structural schematic diagram of an end-side robot provided by an embodiment of the present application is shown in the figure. Figure 4 A structural schematic diagram of a body operation model provided by an embodiment of the present application is shown in the figure. Figure 5 A structural schematic diagram of a cloud-edge collaborative dexterous hand embodiment intelligent operation device provided by an embodiment of the present application is shown in the figure. Figure 6 A structural schematic diagram of a server provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION
[0019] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the present application will be described in detail below in combination with embodiments. Obviously, the described embodiments are some embodiments of the present application, but not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0020] At present, with the rapid development of embodiment intelligence, robots and dexterous hands, due to the advantages of high degree of freedom and flexible motion, dexterous hands are gradually used as end effectors of robots and mechanical arms, and are preliminarily applied in flexible production such as goods carrying, feeding, part assembly and testing in 3C electronic, automobile and energy industries, which is an important means to improve the automation and intelligence level of industrial flexible production. Generally, a dexterous hand includes hardware and software. The hardware is the mechanical and electrical body of the dexterous hand, which is the basis for realizing the gripping and operation of the dexterous hand, and the software integrates the functions of manual operation, pre-programmed control and AI model control of the dexterous hand.
[0021] In industrial flexible production, product types and product models change frequently, and existing dexterous hands cannot adapt to such changes, resulting in that such devices cannot be applied to industrial flexible production. The main reasons are: 1. Traditional pre-programmed control cannot quickly adapt to different products, and has extremely poor adaptability and low success rate; 2. Artificial intelligence model control, such as imitation learning, has extremely low generalization ability, is severely dependent on data acquisition in actual production process, has high cost and long cycle, and cannot adapt to flexible production. Based on this, the cloud-edge collaborative dexterous hand body intelligent operation method, device and server provided by the embodiment of the application can significantly improve the rapid adaptation ability of the dexterous hand.
[0022] In order to facilitate the understanding of the present embodiment, first, a cloud-edge collaborative dexterous hand body intelligent operation method disclosed by the embodiment of the application is introduced in detail. The method is applied to a cloud-side data generation and model training and evaluation system. The cloud-side data generation and model training and evaluation system is in communication connection with an upper computer of an end-side robot. The end-side robot includes a mechanical arm, a dexterous hand, a third perspective camera and an upper computer. In order to facilitate the understanding of the cloud-side data generation and model training and evaluation system, the embodiment of the application provides a structural schematic diagram of a cloud-side data generation and model training and evaluation system, as shown in Figure 1 The end-side robot is deployed in a factory. Through body model reasoning, actual handling, sorting, assembly, testing and other work are completed. A small amount of data of a new scene collected in a production site is uploaded to the cloud side through a network. The cloud-side system (i.e., the cloud-side data generation and model training and evaluation system) generates a large amount of data, trains a body operation model and evaluates a model by using the data uploaded by each end-side robot and a stored database. The model that passes the evaluation is downloaded to the end-side robot through a network, and the adaptation of the new scene and the new workpiece is completed.
[0023] Based on the structural schematic diagram of the cloud-side data generation and model training and evaluation system shown in Figure 1 The embodiment of the application introduces the cloud-edge collaborative dexterous hand body intelligent operation method in detail. Referring to a flowchart of a cloud-edge collaborative dexterous hand body intelligent operation method shown in Figure 2 The method mainly includes the following steps S202 to S206: In step S202, sample data is obtained, and the sample data is subjected to data expansion processing to obtain target sample data. The sample data includes joint state information of the mechanical arm and the dexterous hand, and third perspective camera image information, fingertip visual tactile sensor image information and palm eye visual sensor image information for the product to be operated.
[0024] In an implementation, when collecting sample data, a master clock installed in the host computer can be used as a reference to send periodic synchronization pulses to the robot arm controller, the third-view camera, the fingertip visual-haptic sensor and the palm eye visual sensor, so that the time error of the four-way data is less than 1 millisecond. After time synchronization, data collection is performed.
[0025] In step S204, the general model is trained using the target sample data to obtain a body model, and the body model is evaluated to determine the target body intelligent model that passes the evaluation.
[0026] In an implementation, when performing model evaluation, the body model can be subjected to Monte Carlo sampling and geometric parameter random disturbance evaluation. The workpiece pose, size, friction coefficient, illumination and other parameters in the simulation environment are sampled, 100 times of grabbing are performed for each parameter combination, and the success rate of grabbing in the simulation environment is recorded, that is, the model evaluation result.
[0027] In step S206, the target body intelligent model is sent to the host computer of the end-side robot to control the dexterous hand to perform intelligent operation on the product to be operated.
[0028] In an implementation, when updating the original model of the end-side robot using the target body intelligent model, hot updating can be used. Hot updating has three core elements: 1, no downtime; 2, replace the original model; 3, immediately take effect to execute new tasks.
[0029] The above cloud-edge collaborative dexterous hand body intelligent operation method provided by the embodiment of the application uses the powerful data modeling capability of the artificial intelligence large model to train the body intelligent model with high generalization capability and deploy it on the dexterous hand, thereby performing carrying, sorting, assembling and testing in the flexible production process of industrial products. In order to improve the adaptation efficiency when switching flexible production products and reduce manufacturing costs, the cloud-edge collaborative method is used to perform data production and model training in the cloud and perform data collection and model inference on the edge side. The operation precision and efficiency of the dexterous hand can be greatly improved, and the adaptability of the product to different types of industrial products can be improved.
[0030] The embodiment of the application also provides a cloud-edge collaborative dexterous hand body intelligent operation platform (i.e., a cloud-edge collaborative platform) for industrial flexible production automation. It can be understood that the cloud-side data generation and model training and evaluation system and the end-side robot jointly constitute the cloud-edge collaborative platform to solve the problem of rapid adaptation in the industrial flexible production scene. Referring to Figure 3 , a structural schematic diagram of an end-side robot and Figure 4A structure diagram of a body operation model is shown, and the end-side robot is composed of a mechanical arm, a dexterous hand, a third perspective camera and an upper computer. The dexterous hand is installed at the end of the mechanical arm, and the two together complete the operation action; the third perspective camera is fixed beside the mechanical arm, and is used for visual observation of the positions of the mechanical arm, the dexterous hand and the workpiece; the dexterous hand is configured with a fingertip visual tactile sensor and a palm eye visual sensor; the upper computer is configured with end-side operation software corresponding to the robot, and the body operation model is deployed in the software; by transmitting the joint state of the mechanical arm and the dexterous hand, the image collected by the third perspective camera, the image of the fingertip visual tactile sensor and the image of the palm eye visual sensor to the upper computer software, the upper computer software calls the body operation model to predict the next joint action of the robot in real time, so as to realize the operation action of the robot:
[0031] wherein, represents the third perspective camera image, represents the palm eye visual camera image, represents the image of the fingertip visual tactile sensor, represents the joint state of the robot, represents the body model reasoning process, represents the next time joint state output by the model.
[0032] In an embodiment, the upper computer software interface of the end-side robot includes three main interfaces of the dexterous hand, simulation and body reasoning, wherein the dexterous hand interface completes basic motion control, teleoperation, gesture control and other functions of the dexterous hand; the simulation interface is responsible for two works, one is to upload the target operation workpiece physical picture of the industrial scene to the cloud system to generate data, and the other is to directly generate data at the end side; and the body reasoning interface completes the body operation reasoning process of the robot.
[0033] Through the cloud-edge collaborative platform, the problem of rapid adaptation of the industrial flexible production scene can be solved, in addition, combined with real machine data generation and simulation data generation of the physical simulation engine, rapid data production after product switching can be realized, and the rapid adaptation ability of the dexterous hand can be improved.
[0034] The embodiment of the application further provides an implementation of an intelligent operation dexterous hand, and specific reference is made to (1) to (4) as follows: (1) Utilize the data uploaded on the end side to generate a large amount of realistic data. Specifically, a preset data generation model and a simulation engine can be used to perform data augmentation processing based on sample data and a three-dimensional model of the product to be operated to obtain target sample data. In an implementation manner, the sample data is input into the preset data generation model to perform data augmentation, and first augmented data based on the sample data is obtained. The three-dimensional model of the product to be operated is input into the simulation engine, and second augmented data is generated by randomly changing the surface texture and illumination. Finally, the first augmented data and the second augmented data are combined to obtain the target sample data.
[0035] In addition, whether the data generation model is pre-trained to generate a large amount of realistic data uploaded, to realize data augmentation, or the simulation engine is used to import the three-dimensional model of the workpiece to be operated into the simulation engine and randomly change the surface texture and illumination to generate a large amount of simulation data, both data augmentation methods can be used alone or in combination, and there is no order restriction when used in combination.
[0036] (2) The target sample data and the historical data are used to pre-train the embodied operation model to obtain a general model, and the target sample data is used to fine-tune the general model in the new scene corresponding to the target sample data to obtain an embodied large model. In an implementation manner, the new data and the historical data are stored in the data storage unit of the cloud-side data generation and model training and evaluation system. All the data stored are used to pre-train the embodied operation model to improve the knowledge and generalization ability of the model. Then, the newly uploaded end-side data (i.e., sample data) and newly generated data (i.e., data augmented by the preset data generation model and the simulation engine) are used to fine-tune the model in the new scene, thereby further improving the success rate of the model in the new scene. The new scene refers to the grasping scene of the product to be operated when the sample data is uploaded by the end-side robot.
[0037] (3) The embodied large model is evaluated in the simulation environment to obtain the operation success rate of the embodied large model in the new scene: when the operation success rate is greater than a preset success rate threshold, the embodied large model is determined as a target embodied intelligent model; when the operation success rate is not greater than the preset success rate threshold, data augmentation processing is performed again to increase the amount of data for model training, and the trained embodied large model is evaluated again.
[0038] (4) the target embodiment intelligent model is replaced with the original model of the end-side robot through edge hot updating in a non-stop state, so that the upper computer controls the dexterous hand to perform intelligent operation processing on the product to be operated based on the target embodiment intelligent model. In an implementation manner, through the hot updating manner, the model can be deployed to the end-side robot system without restarting and without power failure, the rapid adaptation of new scenes and new models is realized, and the changeover in minutes and the non-stop line adaptation are realized.
[0039] To sum up, the cloud can use powerful computing devices to quickly generate dexterous hand operation data and train models for industrial production scenarios. Compared with the prior art which needs to rely on industrial field operation dexterous hands to collect data, the data production and model training efficiency is greatly improved, and the data and models are shared on different edge dexterous hand devices, which also significantly improves the replication capability of the device. Further, the cloud-edge collaborative platform of the present application continuously collects industrial operation data and uploads it to the cloud under the premise of permission of the customer during the industrial production process, gradually improves the data scale and quality, and also gradually improves the generalization capability of the embodiment operation model.
[0040] For the cloud-edge collaborative dexterous hand embodiment intelligent operation method provided by the foregoing embodiment, an embodiment of the present application provides a cloud-edge collaborative dexterous hand embodiment intelligent operation device. The device is applied to an upper computer, and the upper computer is in communication connection with an end-side robot. The end-side robot includes a mechanical arm, a dexterous hand and a third-view camera. Referring to a structural schematic diagram of a cloud-edge collaborative dexterous hand embodiment intelligent operation device shown in Figure 5 The device is applied to a cloud-side data generation and model training and evaluation system, and the cloud-side data generation and model training and evaluation system is in communication connection with an upper computer of an end-side robot. The end-side robot includes a mechanical arm, a dexterous hand, a third-view camera and the upper computer. The device includes the following parts: A data expansion module 502 acquires sample data and performs data expansion processing on the sample data to obtain target sample data. The sample data includes joint state information of the mechanical arm and the dexterous hand, and third-view camera image information, fingertip visual-haptic sensor image information and palm eye visual sensor image information for the product to be operated. A model training module 504 performs model training processing on a general model by using the target sample data to obtain an embodiment large model, and performs model evaluation processing on the embodiment large model. The embodiment large model that passes the evaluation is determined as a target embodiment intelligent model. An intelligent operation module 506 issues the target embodiment intelligent model to the upper computer of the end-side robot to control the dexterous hand to perform intelligent operation processing on the product to be operated.
[0041] The cloud-edge collaborative smart hand body intelligent operation device provided by the embodiment of the application can significantly improve the rapid adaptation capability of the smart hand.
[0042] In an implementation, in the step of performing data augmentation processing on the sample data to obtain target sample data, the data augmentation module 502 is further configured to perform data augmentation processing on the sample data and a three-dimensional model of the product to be operated based on a preset data generation model and a simulation engine to obtain the target sample data.
[0043] In an implementation, in the step of performing data augmentation processing on the sample data and the three-dimensional model of the product to be operated based on the preset data generation model and the simulation engine to obtain the target sample data, the data augmentation module 502 is further configured to input the sample data into the preset data generation model to perform data augmentation to obtain first augmented data based on the sample data; input the three-dimensional model of the product to be operated into the simulation engine to generate second augmented data by randomly replacing surface texture and illumination; and combine the first augmented data and the second augmented data to obtain the target sample data.
[0044] In an implementation, in the step of performing model training processing on the general model based on the target sample data to obtain the large model, the model training module 504 is further configured to perform pre-training processing on the body operation model based on the target sample data and historical data to obtain the general model, and perform model fine-tuning processing on the general model in a new scene corresponding to the target sample data based on the target sample data to obtain the large model.
[0045] In an implementation, in the step of performing model evaluation processing on the large model and determining the large model that passes the evaluation as the target intelligent model, the model training module 504 is further configured to perform model evaluation processing on the large model in a simulation environment to obtain an operation success rate of the large model in the new scene; and when the operation success rate is greater than a preset success rate threshold, determine the large model as the target intelligent model.
[0046] In an implementation, the model training module 504 is further configured to, when the operation success rate is not greater than the preset success rate threshold, re-perform data augmentation processing to increase the amount of data for model training, and re-perform model evaluation processing on the trained large model.
[0047] In an implementation, in the step of performing the target embodiment intelligent model distribution to the upper computer of the end-side robot to control the dexterous hand to perform intelligent operation processing on the product to be operated, the intelligent operation module 506 is further configured to replace the original model of the end-side robot with the target embodiment intelligent model through edge hot update in a non-stop state, so that the upper computer controls the dexterous hand to perform intelligent operation processing on the product to be operated based on the target embodiment intelligent model.
[0048] The device provided by the embodiment of the present application has the same implementation principle and technical effects as the foregoing method embodiment. For brevity, the part not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiment.
[0049] The embodiment of the present application provides a server, specifically, the server comprises a processor and a storage device; the storage device stores a computer program, and the computer program performs the method of any one of the foregoing embodiments when executed by the processor.
[0050] Figure 6 A structural diagram of a server provided by the embodiment of the present application is provided, and the server 100 comprises a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is configured to execute an executable module stored in the memory 61, for example, a computer program.
[0051] The memory 61 can include a high-speed random access memory (RAM) and can also include a non-volatile memory, for example, at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.
[0052] The bus 62 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.
[0053] The memory 61 is configured to store a program, and the processor 60 executes the program after receiving an execution instruction. The method executed by the device defined by the flow process of any one of the foregoing embodiments of the present application can be applied to the processor 60 or realized by the processor 60.
[0054] The processor 60 can be an integrated circuit chip with signal processing capability. In implementation, each step of the above method can be completed by integrated logic circuit of hardware in the processor 60 or by instructions in the form of software. The processor 60 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61, and combines the hardware to complete the steps of the above method.
[0055] The computer program product of the readable storage medium provided by the embodiments of the present application comprises a computer readable storage medium storing program codes, and the program codes comprise instructions for executing the method described in the foregoing method embodiments. For specific implementation, reference can be made to the foregoing method embodiments, which will not be described here.
[0056] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.
[0057] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited to this. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A method for cloud-edge collaborative dexterous hand body intelligence operation, characterized in that, The method is applied to a cloud-side data generation and model training and evaluation system, the cloud-side data generation and model training and evaluation system being in communication connection with an upper computer of an end-side robot, the end-side robot comprising a mechanical arm, a dexterous hand, a third-view camera and the upper computer, and the method comprising: obtaining sample data and performing data augmentation processing on the sample data to obtain target sample data, wherein the sample data comprises joint state information of the mechanical arm and the dexterous hand, and third-view camera image information, fingertip visual-haptic sensor image information and palm eye visual sensor image information for a product to be operated; performing model training processing on a general model using the target sample data to obtain a large-bodied model, and performing model evaluation processing on the large-bodied model, and determining a large-bodied model that passes the evaluation as a target large-bodied intelligent model; downloading the target large-bodied intelligent model to the upper computer of the end-side robot to control the dexterous hand to perform intelligent operation processing on the product to be operated. 2.The cloud-edge collaboration dexterous hand avatar intelligence operation method of claim 1, wherein, The step of performing data augmentation processing on the sample data to obtain target sample data comprises: performing data augmentation processing on the sample data and a three-dimensional model of the product to be operated based on a preset data generation model and a simulation engine to obtain the target sample data. 3.The cloud-edge collaboration dexterous hand avatar intelligence operation method of claim 2, wherein, The step of performing data augmentation processing on the sample data and the three-dimensional model of the product to be operated based on a preset data generation model and a simulation engine to obtain the target sample data comprises: inputting the sample data into the preset data generation model to perform data augmentation, and obtaining first augmented data based on the sample data; inputting the three-dimensional model of the product to be operated into the simulation engine to generate second augmented data by randomly changing surface texture and illumination; combining the first augmented data and the second augmented data to obtain the target sample data. 4.The cloud-edge collaboration dexterous hand avatar intelligence operation method of claim 1, wherein, The step of performing model training processing on a general model using the target sample data to obtain a large-bodied model comprises: performing pre-training processing on a large-bodied operation model using the target sample data and historical data to obtain the general model, and performing model fine-tuning processing on the general model in a new scene corresponding to the target sample data to obtain the large-bodied model. 5.The cloud-edge collaboration dexterous hand avatar intelligence operation method of claim 1, wherein, The step of performing model evaluation processing on the large-bodied model and determining a large-bodied model that passes the evaluation as a target large-bodied intelligent model comprises: performing model evaluation processing on the large-bodied model in a simulation environment to obtain an operation success rate of the large-bodied model in the new scene; when the operation success rate is greater than a preset success rate threshold, determining the large-bodied model as the target large-bodied intelligent model. 6.The cloud-edge collaboration dexterous hand avatar intelligence operation method of claim 5, wherein, The method comprises: when the operation success rate is not greater than the preset success rate threshold, re-performing data augmentation processing to increase the amount of data for model training, and re-performing model evaluation processing on the trained large-bodied model. 7.The cloud-edge collaboration dexterous hand avatar intelligence operation method of claim 1, wherein, The step of issuing the target embodied intelligent model into the host computer of the end-side robot to control the dexterous hand to perform intelligent operation processing on the product to be operated includes: The target embodied intelligent model is replaced with the original model of the end-side robot through edge hot updating in a non-stop state, so that the host computer controls the dexterous hand to perform intelligent operation processing on the product to be operated based on the target embodied intelligent model.
8. A cloud-edge collaborative dexterous hand exoskeleton intelligent operation device, characterized in that, The device is applied to a cloud-side data generation and model training evaluation system, which is in communication connection with a host computer of an end-side robot, and the end-side robot includes a mechanical arm, a dexterous hand, a third-view camera and the host computer, and the device includes: A data expansion module acquires sample data and performs data expansion processing on the sample data to obtain target sample data, wherein the sample data includes joint state information of the mechanical arm and the dexterous hand, and third-view camera image information, fingertip visual and tactile sensor image information and palm eye visual sensor image information for the product to be operated; A model training module performs model training processing on a general model using the target sample data to obtain an embodied large model, and performs model evaluation processing on the embodied large model, and determines an embodied large model that passes the evaluation as a target embodied intelligent model; An intelligent operation module issues the target embodied intelligent model into the host computer of the end-side robot to control the dexterous hand to perform intelligent operation processing on the product to be operated.
9. A server, characterized by The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when called and executed by the processor, cause the processor to implement the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when called and executed by the processor, cause the processor to implement the method of any one of claims 1 to 7.