Intelligent flexible assembly method for 3C products

Abstract

The application provides a kind of intelligent flexible assembly method for 3C product, comprising: constructing intelligent assembly line;Based on the intelligent assembly line, a multi-modal virtual environment is constructed;In multi-modal virtual environment, a perception model containing vision, touch and depth is established, and the collection of multi-modal teaching data is realized;Based on the analysis of the multi-modal teaching data, a skill knowledge base is established, and based on the skill knowledge base, a primitive strategy sequence of operation task is generated;The migration of the primitive strategy sequence from digital twin environment to real environment is realized by using randomization method, which is used for robot assembly in the intelligent assembly line.The method proposed by the application completes the construction of intelligent assembly system and improves the intelligent level of the system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of system control. BACKGROUND

[0002] With the rapid development of information technology, people's demand for computer, communication and consumer electronics products (3C products) is increasing. The intelligent assembly of rigid parts in existing 3C assembly has been basically realized, and the assembly of flexible deformable parts, especially variable linear objects (DLO), can only rely on manual assembly, and cannot realize the switching and migration of different production lines.

[0003] In the 3C assembly industry, it is still necessary to build an intelligent flexible assembly production line to solve the problems of material heterogeneity and easy deformation, complex assembly process, etc. in flexible assembly, improve assembly precision and efficiency, realize the demand of intelligent production, break through the limitations of traditional manual assembly, and adapt to the rapid development of intelligent assembly.

[0004] Skill learning and development is an important means for robots to achieve precise control and fine operation, and knowledge representation is a core element of skill learning. The current knowledge representation for assembly scene lacks precision and effectiveness, and there are still great challenges in robot's task understanding, motion control, memory, and new skills and tasks in intelligent assembly.

[0005] First of all, a large amount of skill knowledge (geometry, physics, function, process, data, etc.) is needed to support intelligent assembly modeling, planning, simulation, control, optimization, etc. The existing knowledge representation system is mainly based on static knowledge base, and the description of robot manipulation objects and behaviors is not detailed and flexible enough. Moreover, the structure design of the system is not reasonable, resulting in low query efficiency. In short, there is no professional operation skill knowledge base available, and data collection and analysis are difficult.

[0006] Due to the lack of data, complex environment and safety risks, it is usually not feasible to directly implement intelligent assembly strategies. It is necessary to build a learning and training environment based on the digital twin environment and an intelligent algorithm training and testing platform, and the framework platform needs to have physical properties consistent with the real scene, and migrate the trained deep reinforcement learning strategy to the physical robot. However, there are still problems such as weak task understanding ability, poor environmental adaptability, and low execution efficiency in the virtual-real migration of operation skills from simulation environment to real assembly environment. It is necessary to improve the robot's ability to understand tasks and learn skills, enhance its adaptability to various complex environments, improve execution efficiency, reduce learning cost, and improve its generalization ability.

[0007] Secondly, unlike humans who can naturally learn from videos and natural language to improve language instructions to complete tasks. Robots cannot link language and visual observations and achieve fine-grained understanding learning. Precise and effective knowledge representation methods not only help improve the operation accuracy of robots facing complex assembly tasks and remember previous experiences, but also facilitate the robot to achieve automatic programming of the underlying hardware.

[0008] Therefore, how to split different knowledge representation layers to facilitate the robot to understand the task and skill learning, and how to make the robot understand the language instruction and make the correct learning and imitation, are still a big challenge for the robot. SUMMARY

[0009] The present application aims to at least partially solve one of the technical problems in the related art.

[0010] To this end, the purpose of the present application is to propose an intelligent flexible assembly method for 3C products, for the construction of an intelligent assembly system.

[0011] To achieve the above purpose, the first aspect of the present application proposes an intelligent flexible assembly method for 3C products, comprising:

[0012] constructing an intelligent assembly line;

[0013] Based on the intelligent assembly line, a multi-modal virtual environment is constructed; in the multi-modal virtual environment, a perception model containing vision, touch and depth is established to realize the collection of multi-modal teaching data;

[0014] Based on the analysis of the multi-modal teaching data, a skill knowledge base is established, and based on the skill knowledge base, a primitive strategy sequence of operation task is generated;

[0015] The primitive strategy sequence is migrated from the digital twin environment to the real environment by using a randomization method, and is used for robot assembly in the intelligent assembly line.

[0016] In addition, the intelligent flexible assembly method for 3C products according to the above embodiments of the present application can also have the following additional technical features:

[0017] Further, in an embodiment of the present application, the intelligent assembly line comprises:

[0018] a cycle line cooperative assembly control unit, a cycle guide rail control unit, a 3C flexible mobile phone carrier table feeding and assembly unit, a 3C flexible mobile phone soft cable intelligent assembly unit, a 3C flexible mobile phone front camera intelligent assembly unit, a 3C flexible mobile phone SIM card slot intelligent assembly unit, and a 3C flexible mobile phone coaxial cable intelligent assembly unit.

[0019] Further, in an embodiment of the present application, the establishing a skill knowledge base based on the analysis of the multi-modal teaching data comprises:

[0020] The multi-modal teaching data is decoupled in different types of knowledge, and the robot assembly operation skill knowledge is represented from six levels of scene level, agent level, entity level, task level, skill level and action level.

[0021] Further, in an embodiment of the present application, the generating a primitive strategy sequence of an operation task based on the skill knowledge base comprises:

[0022] The skill learning is performed by using deep reinforcement learning, and the strategy is optimized.

[0023] Further, in an embodiment of the present application, after the generating a primitive strategy sequence of an operation task, further comprising:

[0024] The cross-task migration based on deep reinforcement learning is implemented by constructing a network structure of multi-task learning in a manner of shared layer and exclusive layer to realize feature extraction and feature adaptation between different tasks.

[0025] Further, in an embodiment of the present application, further comprising:

[0026] The common features of the input data are extracted by the shared layer, and the shared layer is represented as:

[0027] f s (x)=W s x+b s ,

[0028] Wherein x is the input data, W s and b s are parameters of the shared layer;

[0029] According to the specific requirements of the task, the features extracted by the shared layer are adapted and optimized through the exclusive layer, and the exclusive layer is represented as:

[0030] f t (x)=W t x+b t ,

[0031] Wherein x is the output of the new task, W t and b t are parameters of the exclusive layer, and t is the index of the task;

[0032] Different outputs are defined for different tasks, and the predicted value of each task is obtained through the output layer, and the output layer is represented as:

[0033] y t =gt (f t (f s (x))),

[0034] where y t is the predicted value of the tth task, g t is the activation function of the tth task;

[0035] An optimizer is defined for updating the network parameters, and the mathematical representation of the optimizer is:

[0036] W s ,b s ,W t ,b t =Adam(L t ,W s ,b s ,W t ,b t ),

[0037] where L t is the loss function of the tth task.

[0038] To achieve the above purpose, the second aspect embodiment of the present application proposes an intelligent flexible assembly device for 3C products, comprising the following modules:

[0039] A construction module is configured to construct an intelligent assembly line.

[0040] A conversion module is configured to construct a multi-modal virtual environment based on the intelligent assembly line, and to establish a perception model containing vision, touch and depth in the multi-modal virtual environment to realize the collection of multi-modal teaching data.

[0041] An analysis module is configured to establish a skill knowledge base based on the analysis of the multi-modal teaching data, and to generate a primitive strategy sequence of an operation task based on the skill knowledge base.

[0042] An assembly module is configured to realize the migration of the primitive strategy sequence from a digital twin environment to a real environment by using a randomization method, and to realize the assembly of a robot in the intelligent assembly line.

[0043] To achieve the above purpose, the third aspect embodiment of the present application proposes a computer device, characterized in that it comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, an intelligent flexible assembly method for 3C products is realized.

[0044] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the intelligent flexible assembly method for 3C products.

[0045] The intelligent flexible assembly method for 3C products provided by the embodiments of the present application is based on the construction of the intelligent flexible assembly production line and the complex operation skill knowledge base system, uses the virtual environment training robot with physical properties to learn and understand the knowledge and experience, and then migrates the knowledge and experience to new skills through the migration algorithm to cope with complex and variable unknown tasks. The construction of the intelligent assembly system is completed, and the intelligent level of the system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0047] Figure 1 A flowchart of the intelligent flexible assembly method for 3C products provided by the embodiments of the present application;

[0048] Figure 2 A network structure diagram for task migration in the knowledge-based skill migration provided by the embodiments of the present application;

[0049] Figure 3 A network structure diagram of meta-optimization in the knowledge-based skill migration provided by the embodiments of the present application;

[0050] Figure 4 A cross-scene and cross-task large model transfer learning system diagram of the large model-based transfer learning provided by the embodiments of the present application;

[0051] Figure 5 A skill transfer mechanism diagram of the large model-based transfer learning provided by the embodiments of the present application;

[0052] Figure 6 A video action analysis method diagram in the large model-based action analysis provided by the embodiments of the present application;

[0053] Figure 7 A skill learning diagram of multi-modal data fusion in the large model-based action analysis provided by the embodiments of the present application;

[0054] Figure 8 And Figure 9 A GPT executable function generation diagram in the large model-based programming-free provided by the embodiments of the present application;

[0055] Figure 10 A structural schematic diagram of a 3C product-oriented intelligent flexible assembly device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0057] A 3C product-oriented intelligent flexible assembly method of an embodiment of the present application is described below with reference to the accompanying drawings.

[0058] Figure 1 A flowchart of a 3C product-oriented intelligent flexible assembly method provided by an embodiment of the present application.

[0059] As shown in Figure 1 , the 3C product-oriented intelligent flexible assembly method comprises the following steps:

[0060] S101: constructing an intelligent assembly line;

[0061] Further, in an embodiment of the present application, the intelligent assembly line comprises:

[0062] a circulation line collaborative assembly control unit, a circulation guide rail control unit, a 3C flexible mobile phone carrier table feeding and discharging assembly unit, a 3C flexible mobile phone flexible cable intelligent assembly unit, a 3C flexible mobile phone front camera intelligent assembly unit, a 3C flexible mobile phone SIM card slot intelligent assembly unit, and a 3C flexible mobile phone coaxial cable intelligent assembly unit.

[0063] The circulation guide rail control system comprises a control cabinet, a conveying mechanism, and a mobile phone material carrier table secondary positioning mechanism. The circulation guide rail control system receives external motion control signals and sends its own state information through a communication protocol. By sending a motion instruction to the circulation guide rail control system, the servo motor drives the conveying mechanism to move the mobile phone material carrier table, and the circulation guide rail control system controls the positioning latch of the mobile phone material carrier to perform secondary positioning. The material transmission and assembly control collaboration of the 3C assembly circulation line are realized.

[0064] 3C flexible mobile phone carrier table feeding and assembling system includes an assembling workbench, a mobile phone carrier table material rack, a six-degree-of-freedom mechanical arm assembly, a flexible pneumatic two-finger clamp, a soft rubber on the flexible pneumatic clamp, and a magnetic attraction device on the flexible pneumatic two-finger clamp. The six-degree-of-freedom mechanical arm carries the flexible pneumatic two-finger clamp to the mobile phone carrier table material rack to clamp the mobile phone carrier table, and the magnetic attraction device on the flexible pneumatic two-finger clamp carried by the six-degree-of-freedom mechanical arm calibrates the clamping pose. The pin on the assembling rack and the magnetic attraction device calibrate the pose of the placed mobile phone carrier table, realizing the feeding and discharging of the mobile phone carrier table.

[0065] 3C flexible mobile phone soft cable intelligent assembly system includes an assembling workbench, a six-degree-of-freedom mechanical arm assembly, a secondary positioning table, and a vision-based tactile sensor. The end of the six-degree-of-freedom mechanical arm is equipped with an intelligent assembly end. The intelligent assembly end is internally provided with a vision camera, a tactile sensor, a sliding table cylinder, and a negative pressure suction device. The secondary positioning table is installed at the front end of the assembling workbench and includes a positioning plate sliding table cylinder and a positioning groove. The vision sensor of the intelligent assembly end uses a target detection algorithm to detect the position of the soft cable in the soft cable material disc, and then controls the mechanical arm to suck the soft cable. The soft cable is preliminarily positioned on the secondary positioning jig. The vision and tactile sensing algorithm calculates the offset and rotation angle of the male head of the soft cable to achieve accurate calibration of the pose of the male head. The tactile perception of the intelligent assembly end realizes flexible and accurate buckling, achieving multi-modal scene perception and accurate buckling of the flexible intelligent assembly of the mobile phone soft cable.

[0066] 3C flexible mobile phone front camera intelligent assembly system includes an assembling workbench, a six-degree-of-freedom mechanical arm assembly, an end effector, and a vision recognition system. The vision recognition system includes a high-precision camera and an advanced image processing algorithm, which is used for fast and accurate recognition and positioning of the front camera position on the material disc. The end effector is equipped with a suction head device, a positioning device, and a buckling device, realizing the suction, calibration, and buckling of the mobile phone front camera. The entire assembly process uses reinforcement learning algorithm to train the mechanical arm, enabling it to autonomously learn and optimize the movement path and action sequence based on vision information and hand-eye calibration results, achieving higher efficiency and precision. After reaching the target position, the system can quickly identify the mobile phone panel to be assembled and the target point to be buckled, realizing accurate assembly operation.

[0067] 3C flexible phone SIM card slot intelligent assembly system includes an assembly workbench, a six-degree-of-freedom mechanical arm assembly, an end effector, a visual recognition system, and a secondary positioning assembly. The visual recognition system uses a high-precision camera and advanced image processing algorithms to quickly and accurately identify and position the SIM card slot in the material tray. The end effector includes a suction device and a gripper to grasp the end of the SIM card slot on the secondary positioning table to ensure the accuracy of its position and orientation. The end effector is equipped with a suction device, a gripping device, and a front push rod for sucking and inserting the SIM card slot into the phone. Additionally, a secondary positioning table is provided in the assembly scene to perform secondary positioning of the SIM card slot.

[0068] The core of the 3C flexible phone coaxial cable intelligent assembly system is the use of the flexibility and stability of the mechanical arm combined with advanced visual recognition and reinforcement learning algorithms to achieve intelligent assembly of the phone coaxial cable. Due to the complexity of the phone coaxial cable assembly, two mechanical arms are used to work together to achieve precise and stable coaxial cable assembly. One mechanical arm is equipped with a positioning device, a gripping device, and a wire straightening device, while the other mechanical arm is equipped with a buckling device. Combined with a specially designed wire straightening device, the intelligent assembly of the phone coaxial cable is achieved.

[0069] S102: Based on the intelligent assembly production line, a multi-modal virtual environment is constructed. In the multi-modal virtual environment, a perception model containing vision, touch, and depth is established to achieve the collection of multi-modal teaching data.

[0070] Further, in an embodiment of the present application, the skill knowledge base is established based on the analysis of the multi-modal teaching data, which includes:

[0071] The multi-modal teaching data is decoupled into different types of knowledge, starting from six levels of scene layer, agent layer, entity layer, task layer, skill layer, and action layer, to represent the robot assembly operation skill knowledge.

[0072] The present application builds a digital twin simulation environment with a 1:1 geometric size of the real scene. Through the self-designed synchronization code, real-time synchronization between the real scene and the virtual scene can be achieved, and remote monitoring of physical equipment is realized. The entire digital twin environment model is imported by self-modeling, and the model is exactly the same as the real scene, including its material, mass, and center of gravity properties. The position and pose of the entire assembly line in the digital twin environment are consistent with the real scene, achieving 1:1 consistency. By introducing the digital twin of the 3C assembly scene, the neural radiation field (NeRFs) is used as a bridge between virtual and reality:

[0073] The present application builds each assembly scene in detail, and the position and pose of each mechanical arm are adjusted in real time according to the real scene. The scene includes feeding and discharging, soft wire assembly, SIM card assembly, front camera assembly, and coaxial cable assembly.

[0074] The feeding and discharging scene includes a mechanical arm, a mobile phone panel and a carrier table. The flexible flat cable scene includes a mechanical arm, a flexible flat cable, a secondary positioning table, visual touch detection and a carrier table. The SIM card assembly scene includes a mechanical arm, a SIM card, a secondary positioning table and a carrier table. The front camera assembly scene includes a mechanical arm, a front camera and a carrier table. The coaxial cable scene includes two mechanical arms, a coaxial cable and a carrier table.

[0075] The control mode of the mechanical arm in the whole digital twin environment is divided into two kinds. The inverse kinematics is written to realize the inverse solution code of the mechanical arm, so that the joint angle is obtained by knowing the end position and the mechanical arm is directly controlled by the joint angle. When virtual and real synchronization is required, the mechanical arm action in the digital twin environment is consistent with the real scene. The joint position information of the mechanical arm in the real scene is read through the script designed by the application, and the information is synchronized to the mechanical arm in the digital twin environment. The positions of various parts in the environment are consistent with the real scene. Therefore, the mechanical arm in the real scene can also be synchronized to suck the flexible flat cable in the digital twin environment.

[0076] The digital twin environment of the application also has a real physical engine, including collision detection, gravity, moment of inertia, visual detection and force sensor feedback.

[0077] S103: establishing a skill knowledge base based on the analysis of the multi-modal teaching data, and generating a primitive strategy sequence of the operation task based on the skill knowledge base;

[0078] Further, in an embodiment of the application, establishing a skill knowledge base based on the analysis of the multi-modal teaching data comprises:

[0079] The multi-modal teaching data is decoupled in different types of knowledge, and the robot assembly operation skill knowledge is comprehensively, effectively and accurately represented from six levels of scene layer, agent layer, entity layer, task layer, skill layer and action layer.

[0080] S104: realizing the migration of the primitive strategy sequence from the digital twin environment to the real environment by using a randomization method, for the assembly of the robot in the intelligent assembly line.

[0081] On the one hand, the application provides a reinforcement learning paradigm for environment interaction and strategy optimization by combining deep reinforcement learning, reinforcement learning and deep learning. The low-cost and unlimited number of samples obtained from the simulation environment can realize efficient learning of the robot control strategy.

[0082] As Figure 2The diagram shows the network structure for task transfer in knowledge-based skill transfer. First, multiple 3C assembly datasets for different tasks are prepared. The input and output data are preprocessed according to the network structure requirements, ensuring that the data format and dimensions match the network structure. Then, features of various modalities are extracted through a graph general layer. This general layer includes a deep convolutional module to extract visual features, a text encoding module to extract text information, and a fully connected layer to encode action information. Next, a feature fusion module fuses information from various dimensions and levels into a fused modal information. Finally, semantic information under the fused features is learned. The fused information is input into the decision layer to learn a general action strategy.

[0083] like Figure 3 The diagram shows the meta-optimization network structure in knowledge-based skill transfer. When a new task arrives, the optimizer fine-tunes the parameters of the existing model in the general layer based on the new task data, forming a task-specific layer model, such as... Figure 2 As shown, the model then fine-tunes its action strategy using the semantic information learned from the new task, transferring the learned strategy to the new task, and finally applying the learned skills to the scenario.

[0084] On the other hand, this invention transfers pre-trained strategies in a virtual environment to a real scene through virtual-real migration. It extracts and transfers knowledge from the strategies in the virtual environment through methods such as strategy distillation and strategy reuse, enabling the robot to achieve high-efficiency and high-accuracy grasping and assembly in a real physical scene.

[0085] like Figure 4 As shown, this is a cross-scene, cross-task large model transfer learning system based on large model transfer learning. The Visual Model Learning (VLM) effectively extracts semantic knowledge related to the environment and task through deep analysis of visual information such as images and videos, identifying objects, actions, and relationships in the scene and transforming them into general semantic representations. Then, the multimodal world information is passed to the Large Model Learning (LLM) for summarization and task inference. The semantic information extracted by the VLM is passed to the LLM, which is responsible for summarizing and inferring this information. The LLM, through deep understanding of textual information, summarizes the visual information extracted by the VLM and performs higher-level task inference, forming a comprehensive understanding of the environment and task. A multi-modal spatiotemporal alignment technique is introduced to integrate information from multiple sources such as text, images, sensors, and videos. This spatiotemporal alignment technique ensures that this information can effectively interact and collaborate during the learning process. Through the collaborative work of the VLM and LLM, the information obtained from their respective analysis and inference is fused into a multimodal feature representation. This representation includes a combination of visual and linguistic information, providing richer and more abstract input for subsequent skill learning.

[0086] like Figure 5As shown in the figure, it is a skill transfer mechanism diagram based on large model transfer learning. For the same skill, the system directly transfers the skill. By directly applying the skill representation obtained through reasoning to the new scene or task, the direct transfer of the skill is realized. This is applicable to skills with the same execution mode and semantic meaning in different environments; for similar skills, the system adopts the strategy of modified transfer. By further utilizing the large model, the system further reasons and modifies the similar skills to adapt to the special requirements of the new scene or task. This process involves fine-tuning and adjusting the skills to ensure better adaptation and execution in different backgrounds.

[0087] As shown in the figure, it is a video action analysis method in action analysis based on large models. By preprocessing the given video and combining it with ChatGPT questions, the multi-modal large model LLaVA is given to obtain the answer to the question, which is convenient for understanding the scene information. After multiple rounds of dialogue, ChatGPT summarizes and selects appropriate actions from the action library, and generates an instruction for each action. Figure 6

[0088] As shown in the figure, it is a skill learning based on multi-modal data fusion in action analysis based on large models. The multi-modal fusion module is used to fuse language and visual input to achieve unified representation. The knowledge enhanced learning module uses the single-step instructions planned by the LLM to improve skill learning. Finally, the action prediction module uses the fused data to predict the next assembly action. Figure 7

[0089] As shown in the figures, it is a GPT-generated executable function in programming-free based on large models. The output of this description is generated, where task_sequence is the executable sequence. In order to convert the executable sequence into an executable function, an explanatory manner is adopted, first some programming common sense is given, for example, get a variable a, define the function to be generated at the time, through natural language guidance + example, let GPT-4 learn this knowledge, and finally generate an executable function for all operation sequences. Figure 8 Figure 9

[0090] Further, in an embodiment of the present application, after generating the primitive strategy sequence of the operation task, it further includes:

[0091] Cross-task transfer based on deep reinforcement learning, by constructing a multi-task learning network structure using shared layers and dedicated layers, feature extraction and feature adaptation between different tasks are realized.

[0092] Further, in an embodiment of the present application, it further includes:

[0093] The shared layer is used to extract the general features of the input data, and the shared layer is represented as:​​​​

[0094] f s (x) = W s x + b s ,

[0095] where x is the input data, W s and b s are the parameters of the shared layer;

[0096] According to the specific requirements of the task, the features extracted by the shared layer are adapted and optimized by a dedicated layer, which is represented as:

[0097] f t (x) = W t x + b t ,

[0098] where x is the output of the new task, W t and b t are the parameters of the dedicated layer, and t is the index of the task;

[0099] Different outputs are defined for different tasks, and the prediction value of each task is obtained through an output layer, which is represented as:

[0100] y t = g t (f t (f s (x))),

[0101] where y t is the prediction value of the tth task, and g t is the activation function of the tth task;

[0102] An optimizer is defined to update the network parameters, and the mathematical representation of the optimizer is:

[0103] W s , b s , W t , b t = Adam (L t , W s , b s , W t , b t ),

[0104] where L t is the loss function of the tth task.

[0105] The meta-optimizer is an optimizer that can automatically adjust the parameters in the network structure according to the feedback of different tasks, and realize the knowledge transfer between different tasks.

[0106] Suppose there are n source tasks, each task has a dataset and a loss function, denoted as T. i =(D i ,L i Let i = 1, 2, ..., N. Assume the parameters in the network structure are θ, the meta-knowledge is ω, and the meta-optimizer is MetaOpt. The goal of the meta-optimizer is to minimize the average meta-loss of all source tasks, i.e.

[0107]

[0108] Where MetaOpt(ω,D) i ) indicates the use of meta-knowledge ω in dataset D i The parameters obtained from the training.

[0109] When there are M new tasks, each task has a dataset and a loss function, denoted as T, j =(D j ,L j ), where j = 1, 2, ..., M. The goal of the meta-optimizer is to evaluate the meta-knowledge ω. * Performance on new tasks, i.e.

[0110]

[0111] Where MetaOpt(ω) * D j ) indicates the use of meta-knowledge ω * In dataset D j The parameters obtained from the training.

[0112] If the performance evaluation on a new task is unsatisfactory, the network parameters are updated until it performs well on the new task. Ultimately, this achieves efficient learning and transfer across different tasks, reduces the learning cost for different tasks, and improves generalization ability.

[0113] The intelligent flexible assembly method for 3C products proposed in this invention, based on the construction of an intelligent flexible assembly production line and a knowledge base system for complex operation skills, utilizes a virtual environment with physical attributes to train robots to learn and understand knowledge and experience. Then, a transfer algorithm is used to transfer this knowledge and experience to new skills to cope with complex and ever-changing unknown tasks. This completes the construction of an intelligent assembly system and improves the system's intelligence level.

[0114] To achieve the above embodiments, the present invention also proposes an intelligent flexible assembly device for 3C products.

[0115] Figure 10 This is a structural schematic diagram of an intelligent flexible assembly device for 3C products provided in an embodiment of the present invention.

[0116] As Figure 10 shown, the smart flexible assembly device for 3C products comprises a construction module 100, a conversion module 200, an analysis module 300, and an assembly module 400, wherein,

[0117] The construction module is configured to construct a smart assembly line.

[0118] The conversion module is configured to construct a multi-modal virtual environment based on the smart assembly line, and to establish a perception model containing vision, touch, and depth in the multi-modal virtual environment to realize collection of multi-modal teaching data.

[0119] The analysis module is configured to establish a skill knowledge base based on analysis of the multi-modal teaching data, and to generate a primitive strategy sequence of an operation task based on the skill knowledge base.

[0120] The assembly module is configured to realize migration of the primitive strategy sequence from a digital twin environment to a real environment by using a randomization method, and to realize assembly of a robot in the smart assembly line.

[0121] To achieve the above object, the third aspect of the present application provides a computer device, characterized in that it comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the smart flexible assembly method for 3C products.

[0122] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium, characterized in that it stores a computer program, and the computer program is executable on a processor to realize the smart flexible assembly method for 3C products.

[0123] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0124] Furthermore, the terms "first", "second", etc. are used only for descriptive purposes and do not connote or imply relative importance or a quantity of the indicated technical features. Thus, a feature defined with "first", "second", etc. can include at least one of the features implicitly or explicitly. In the description of the present application, the meaning of "a plurality" is at least two, for example, two, three, etc., unless otherwise specifically defined.

[0125] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary only and are not to be taken in a limiting sense, but the scope of the present application is not to be interpreted in a limiting sense from the above-described embodiments and can be changed, modified, replaced and varied by those skilled in the art within the scope of the present application.

Claims

1. A smart flexible assembly method for 3C products, characterized in that, The method comprises the following steps: constructing an intelligent assembly production line; based on the intelligent assembly production line, constructing a multi-modal virtual environment; in the multi-modal virtual environment, a perception model containing vision, touch, and depth is established to realize the collection of multi-modal teaching data; based on the analysis of the multi-modal teaching data, a skill knowledge base is established, and based on the skill knowledge base, a primitive strategy sequence of an operation task is generated; a randomization method is used to realize the migration of the primitive strategy sequence from a digital twin environment to a real environment for the assembly of a robot in the intelligent assembly production line; after generating the primitive strategy sequence of the operation task, the method further comprises: cross-task migration based on deep reinforcement learning, which realizes feature extraction and feature adaptation between different tasks by constructing a multi-task learning network structure using shared layers and exclusive layers; the method further comprises: extracting general features of input data through a shared layer, which is represented as: f s (x) = W s x + b s , where x is the input data, W s and b s are parameters of the shared layer; according to the specific requirements of the task, the features extracted by the shared layer are adapted and optimized through an exclusive layer, which is represented as: f t (x) = W t x + b t , where x is the output of the new task, W t and b t are parameters of the specialized layer, and t is the index of the task. different outputs are defined for different tasks, and the predicted values of each task are obtained through an output layer, which is represented as: y t = g t (f t (f s (x))) where y t is the predicted value for the tth task, g t is the activation function for the tth task; an optimizer is defined to update network parameters, and the mathematical representation of the optimizer is: W s ,b s ,W t ,b t = Adam(L t ,W s ,b s ,W t ,b t ), where L t is the loss function for the tth task.

2. The method of claim 1, wherein, the intelligent assembly production line comprises: a circulating production line cooperative assembly control unit, a circulating guide rail control unit, a 3C flexible mobile phone carrier table feeding and disassembly assembly unit, a 3C flexible mobile phone soft cable intelligent assembly unit, a 3C flexible mobile phone front camera intelligent assembly unit, a 3C flexible mobile phone SIM card slot intelligent assembly unit, and a 3C flexible mobile phone coaxial cable intelligent assembly unit.

3. The method of claim 1, wherein, the establishment of the skill knowledge base based on the analysis of the multi-modal teaching data comprises: layered decoupling of different types of knowledge from the multi-modal teaching data, which is represented from six levels of scene level, agent level, entity level, task level, skill level, and action level.

4. The method of claim 1, wherein, the generation of the primitive strategy sequence of the operation task based on the skill knowledge base comprises: skill learning and strategy optimization using deep reinforcement learning.

5. A smart flexible assembly device for 3C products, characterized in that, The device implements the method of claim 1, and the device comprises the following modules: a construction module for constructing an intelligent assembly production line; a conversion module for constructing a multi-modal virtual environment based on the intelligent assembly production line; in the multi-modal virtual environment, a perception model containing vision, touch, and depth is established to realize the collection of multi-modal teaching data; an analysis module for establishing a skill knowledge base based on the analysis of the multi-modal teaching data, and generating a primitive strategy sequence of an operation task based on the skill knowledge base; an assembly module for realizing the migration of the primitive strategy sequence from a digital twin environment to a real environment using a randomization method for the assembly of a robot in the intelligent assembly production line.

6. A computer device, comprising: The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the method of claim 1-4 is realized for the intelligent flexible assembly of 3C products. The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the method of claim 1-4 is realized for the intelligent flexible assembly of 3C products.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by a processor to realize the intelligent flexible assembly method for 3C products as claimed in any one of claims 1-4.

Images