A multi-person collaborative manufacturing system and method based on digital twin and extended reality

Abstract

The present application discloses a multi-person collaborative manufacturing system and method based on digital twin and extended reality, the system comprising: a digital twin model construction module, which is used to integrate the manufacturing demand information, manufacturing process information and manufacturing process information of the product to construct a digital twin model of the whole process; a cross-platform virtual platform construction module, which is used to process the digital twin model in a standardized data exchange format to obtain a cross-platform virtual model digital platform; a process information collection and push module, which is used to use the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product; a real-time process information push module, which is used to perform federated learning algorithm training on the process information and construct a real-time process information push system for multi-person collaborative manufacturing. By constructing a real-time process information push system for multi-person collaborative manufacturing, the scattered and isolated modules are effectively integrated to achieve full process management.

Description

Technical Field

[0001] The present application relates to the technical field of virtual reality application of human-computer interaction in intelligent manufacturing, and in particular to a multi-person collaborative manufacturing system and method based on digital twins and extended reality. Background Art

[0002] In today's manufacturing industry, the trend of informatization, intelligence and networking is becoming increasingly obvious. However, traditional manufacturing systems often have many limitations in information integration, collaboration efficiency and real-time performance. For example, different manufacturing participants (such as designers, production operators, quality inspectors, etc.) may operate on different platforms or terminals, resulting in difficulties in the circulation and sharing of information; manufacturing information at various stages (such as design information, process information, process information, etc.) may be scattered in different systems or databases, requiring cumbersome data integration to achieve full process management. In addition, traditional manufacturing systems often lack real-time and interactivity, and cannot effectively achieve multi-person collaboration and real-time decision-making. For example, when a problem occurs in a certain production link, it takes a lot of time and energy to collect relevant information, analyze the cause of the problem, and then develop a solution, which greatly reduces manufacturing efficiency.

[0003] In the existing technology, the above problems are being solved by applying digital twins and extended reality technologies (including virtual reality VR and augmented reality AR) to the manufacturing industry. However, the current application of digital twins and extended reality technologies in manufacturing systems is still scattered and isolated, lacking effective integration. Therefore, how to effectively combine digital twins and extended reality technologies to realize a cross-terminal, multi-person collaborative manufacturing system is an issue that needs to be solved urgently. Summary of the invention

[0004] In view of this, the present application provides a multi-person collaborative manufacturing system and method based on digital twin and extended reality to solve the problem of scattered and isolated application of digital twin and extended reality technologies in the manufacturing industry in the prior art, and lack of effective integration.

[0005] This application proposes a multi-person collaborative manufacturing system based on digital twins and augmented reality, including:

[0006] The digital twin model building module is used to integrate the product's manufacturing demand information, manufacturing process information, and manufacturing process information to build a digital twin model of the entire process;

[0007] A cross-platform virtual platform construction module is used to process the digital twin model in a standardized data exchange format to obtain a cross-platform virtual model digital platform;

[0008] A process information collection and push module, which is used to use the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product;

[0009] The real-time process information push module is used to perform federated learning algorithm training on the process information and build a real-time process information push system for multi-person collaborative manufacturing.

[0010] Optionally, the digital twin model building module includes:

[0011] Multimodal information perception unit, used to obtain product design documents, process cards, equipment parameters and operator feedback for the entire manufacturing process;

[0012] A data classification and processing unit, used to classify and process the product manufacturing process information and convert it into modeling data;

[0013] A physical model building unit is used to build a three-dimensional model using computer-aided design software based on the whole process information of product manufacturing;

[0014] Constructing a digital twin model unit, which is used to use digital twin modeling software to combine the modeling data and the three-dimensional model to construct a digital twin model of the entire process;

[0015] The model verification and optimization unit is used to verify and optimize the digital twin model by using simulation experiments and parameter adjustment methods.

[0016] Optionally, the cross-platform virtual platform building module includes:

[0017] A preset unified data format unit, used to preset a unified data exchange format of the digital twin model;

[0018] A unified data format unit, used to convert the digital twin model into the unified data exchange format;

[0019] A cross-platform interface development unit, used to develop a data display interface, a model operation interface and a collaborative communication interface, wherein the data display interface, the model operation interface and the collaborative communication interface are compatible with VR devices, AR devices, PCs, tablets and mobile phones;

[0020] Data synchronization and model collaboration unit, used to use a unified data exchange format and interface to achieve data synchronization updates and model collaboration operations on all terminals;

[0021] The testing and optimization unit is used to test the virtual model digital platform and perform optimization and adjustment based on the test results.

[0022] Optionally, the process information collection and push module includes:

[0023] A process information viewing unit is used to view the production process of a product using the virtual model digital platform, adjust the product design according to the feedback of problems in the production process, view the operating status of the equipment, adjust the production parameters, view the quality inspection results of the product, and solve the quality problems;

[0024] Data analysis and decision-making unit, which is used to analyze data and predict product quality trends based on historical data and experience of product production using machine learning algorithms;

[0025] The status assessment and work allocation unit is used to assess the fatigue level of the staff in real time and dynamically coordinate the work intensity of the staff according to the fatigue level.

[0026] Optionally, the real-time process information push module includes:

[0027] A federated learning unit, used for performing local model training on the data of the process information of each terminal, and integrating and updating the results of the local model training of all terminals;

[0028] The data sharing unit is used to share the integrated update results across terminals to realize a real-time process information push system for multi-person collaborative manufacturing.

[0029] This application also proposes a multi-person collaborative manufacturing method based on digital twin and augmented reality, comprising the following steps:

[0030] The digital twin model building module integrates the product's manufacturing demand information, manufacturing process information, and manufacturing process information to build a digital twin model of the entire process;

[0031] The cross-platform virtual platform construction module processes the digital twin model in a standardized data exchange format to obtain a cross-platform virtual model digital platform;

[0032] The process information collection and push module uses the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product;

[0033] The real-time process information push module performs federated learning algorithm training on the process information to build a real-time process information push system for multi-person collaborative manufacturing.

[0034] Optionally, the digital twin model construction module integrates the manufacturing demand information, manufacturing process information and manufacturing process information of the product to construct a digital twin model of the entire process, specifically including:

[0035] The multimodal information perception unit obtains product design documents, process cards, equipment parameters, and operator feedback on the entire manufacturing process;

[0036] The data classification and processing unit classifies and processes the product manufacturing process information and converts it into modeling data;

[0037] The physical model building unit builds a three-dimensional model using computer-aided design software according to the whole process information of the product manufacturing;

[0038] The digital twin model building unit uses digital twin modeling software to combine the modeling data and the three-dimensional model to build a digital twin model of the entire process;

[0039] The model verification and optimization unit verifies and optimizes the digital twin model using simulation experiments and parameter adjustment methods.

[0040] Optionally, the cross-platform virtual platform construction module uses a standardized data exchange format for the digital twin model to construct a cross-platform virtual model digital platform, specifically including:

[0041] Presetting a unified data format unit to preset a unified data exchange format of the digital twin model;

[0042] The unified data format unit converts the digital twin model into the unified data exchange format;

[0043] The cross-platform interface development unit develops a data display interface, a model operation interface, and a collaborative communication interface, wherein the data display interface, the model operation interface, and the collaborative communication interface are compatible with VR devices, AR devices, PCs, tablets, and mobile phones;

[0044] The data synchronization and model collaboration unit uses a unified data exchange format and interface to achieve data synchronization updates and model collaboration operations on all terminals;

[0045] The testing and optimization unit tests the virtual model digital platform and makes optimizations and adjustments based on the test results.

[0046] Optionally, the process information collection and push module uses the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product, specifically including:

[0047] The process information viewing unit uses the virtual model digital platform to view the production process of the product, adjust the product design according to the problem feedback in the production process, view the operation status of the equipment, adjust the production parameters, view the quality inspection results of the product, and solve the quality problems;

[0048] The data analysis and decision-making unit uses machine learning algorithms to analyze data and predict product quality trends based on historical data and experience of product production;

[0049] The status assessment and work allocation unit assesses the fatigue level of the staff in real time, and dynamically coordinates the work intensity of the staff according to the fatigue level.

[0050] Optionally, the real-time process information push module performs federated learning algorithm training on the process information to build a real-time process information push system for multi-person collaborative manufacturing, specifically including:

[0051] The federated learning unit performs local model training on the data of the process information of each terminal, and integrates and updates the results of the local model training of all terminals;

[0052] The data sharing unit shares the integrated update results across terminals to implement a real-time process information push system for multi-person collaborative manufacturing.

[0053] The beneficial effects of the present application are as follows: Different from the prior art, the present application constructs a digital twin model of the entire process by constructing a digital twin model for integrating the manufacturing demand information, manufacturing process information and manufacturing process information of the product, simulating the real form of the physical entity, so that the system has both product performance; secondly, the present application constructs a cross-platform virtual platform construction module for constructing a cross-platform virtual model digital platform using a standardized data exchange format for the digital twin model, and the standardized data source can provide different data formats for synchronous display and collaboration across terminals; thirdly, the present application constructs a process information collection and push module for using the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product, thereby improving production efficiency, improving product quality, and improving the use efficiency of equipment, reducing maintenance costs, and thus improving the economic benefits of the entire manufacturing process; in addition, the present application constructs a real-time process information push module for constructing a real-time process information push system for multi-person collaborative manufacturing by training the process information through a federated learning algorithm, which can effectively utilize the data and computing power of each terminal device while protecting the privacy of the data, realizing real-time and personalized demand information push, and effectively integrating the scattered and isolated modules to achieve full process management.

[0054] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 It is a schematic diagram of the principle of a preferred embodiment of the multi-person collaborative manufacturing system based on digital twin and augmented reality of the present application;

[0057] Figure 2 It is a framework diagram of a preferred embodiment of the multi-person collaborative manufacturing system based on digital twin and augmented reality of the present application;

[0058] Figure 3 It is a multi-person collaboration flow chart across terminals of a preferred embodiment of the multi-person collaborative manufacturing system based on digital twin and augmented reality of the present application;

[0059] Figure 4 It is a flowchart of on-site multi-person collaborative optimization of a preferred embodiment of the multi-person collaborative manufacturing system based on digital twin and augmented reality of the present application;

[0060] Figure 5 It is a flowchart of a preferred embodiment of the multi-person collaborative manufacturing method based on digital twin and augmented reality of the present application. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the technical solution of the present application, the following is a further detailed description of a multi-person collaborative manufacturing system and method based on digital twins and extended reality provided by the present application in combination with the accompanying drawings and specific implementation methods. It is understandable that the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative labor are within the scope of protection of this application.

[0062] The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0063] The present application provides a multi-person collaborative manufacturing system and method based on digital twin and extended reality to solve the problem in the prior art of scattered and isolated application of digital twin and extended reality technologies in the manufacturing industry and lack of effective integration.

[0064] See also Figures 1 to 4 , Figure 1 It is a schematic diagram of the principle of a preferred embodiment of the multi-person collaborative manufacturing system based on digital twin and augmented reality of the present application; Figure 2 It is a framework diagram of a preferred embodiment of the multi-person collaborative manufacturing system based on digital twin and augmented reality of the present application; Figure 3 It is a multi-person collaboration flow chart across terminals of a preferred embodiment of the multi-person collaborative manufacturing system based on digital twin and augmented reality of the present application; Figure 4 It is an on-site multi-person collaborative optimization flow chart of a preferred embodiment of the multi-person collaborative manufacturing system based on digital twin and augmented reality of the present application.

[0065] This application proposes a multi-person collaborative manufacturing system based on digital twin and augmented reality, in which Figures 1 to 4 As shown, the multi-person collaborative manufacturing system based on digital twin and augmented reality includes a digital twin model construction module 51, a cross-platform virtual platform construction module 52, a process information collection and push module 53, and a real-time process information push module 54. Among them, the digital twin model construction module 51 is used to integrate the manufacturing demand information, manufacturing process information and manufacturing process information of the product to build a digital twin model of the whole process; the cross-platform virtual platform construction module 52 is used to process the digital twin model in a standardized data exchange format to obtain a cross-platform virtual model digital platform; the process information collection and push module 53 is used to use the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product; the real-time process information push module 54 is used to train the process information with a federated learning algorithm to build a real-time process information push system for multi-person collaborative manufacturing.

[0066] In the above embodiments, the present application constructs a digital twin model of the entire process by constructing a product for integrating manufacturing demand information, manufacturing process information and manufacturing process information, simulating the real form of the physical entity, so that the system has both product performance; secondly, the present application constructs a cross-platform virtual platform construction module for constructing a cross-platform virtual model digital platform using a standardized data exchange format for the digital twin model, and the standardized data source can provide different data formats for synchronous display and collaboration across terminals; thirdly, the present application constructs a process information collection and push module for using the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product, which can improve production efficiency, improve product quality, and improve the use efficiency of equipment and reduce maintenance costs, thereby improving the economic benefits of the entire manufacturing process; in addition, the present application constructs a real-time process information push module for constructing a real-time process information push system for multi-person collaborative manufacturing by training the process information through a federated learning algorithm, which can effectively utilize the data and computing power of each terminal device, while protecting the privacy of the data, to achieve real-time and personalized demand information push, effectively integrate the scattered and isolated modules, and achieve full process management.

[0067] The digital twin model construction module 51 includes a multimodal information perception unit, a data classification and processing unit, a physical model construction unit, a digital twin model construction unit, and a model verification and optimization unit. Figure 2 As shown, the multimodal information perception unit is used to obtain the product design documents, process cards, equipment parameters and operator feedback of the product manufacturing process information; the data classification and processing unit is used to classify and process the product manufacturing process information and convert it into modeling data; the physical model construction unit is used to construct a three-dimensional model based on the product manufacturing process information using computer-aided design software; the digital twin model construction unit is used to use digital twin modeling software to combine the modeling data and the three-dimensional model to construct a digital twin model of the whole process; the model verification and optimization unit is used to verify and optimize the digital twin model using simulation experiments and parameter adjustment methods.

[0068] Specifically, a high-fidelity digital twin modeling method is used to form a full-process digital twin model. The digital twin model integrates the manufacturing demand information, manufacturing process information, and manufacturing process information of the product. Among them, the data types include structured data, unstructured data, and geometric model data. Through high-precision modeling of physical entities, it is possible to simulate not only the form of physical entities, but also a model that represents product performance.

[0069] The multimodal information perception unit is used to obtain the manufacturing demand information, manufacturing process information and manufacturing process information of the product, such as Figure 3 As shown, this information can be product design documents, process cards, equipment parameters, operator feedback and other product manufacturing process information.

[0070] The data classification and processing unit is used to classify and process the information of the entire manufacturing process of the product to determine the data type. The data type includes structured data (such as data in a database), unstructured data (such as text, pictures, audio, etc.) and geometric model data (such as CAD drawings). For unstructured data, it may be necessary to pre-process it through machine learning and other methods, such as image recognition, speech-to-text, etc., and convert it into modeling data.

[0071] Build a physical model unit and use computer-aided design software to build a high-precision three-dimensional model of the physical entity based on the information of the entire product manufacturing process, such as product design documents and process parameters. This model not only simulates the shape of the physical entity, but also needs to be able to reflect the main performance parameters of the product.

[0072] A digital twin model unit is constructed to use the digital twin modeling software or platform to combine the structured data obtained in the above steps, the modeling data of the unstructured data, and the three-dimensional model to construct a digital twin model of the entire process. In this model, various information of the product can be viewed and analyzed by operating the physical model (such as rotation, scaling, sectioning, etc.).

[0073] Among them, the digital twin modeling software or platform can be Siemens MindSphere: Siemens' MindSphere is a cloud-based open Internet of Things (IoT) operating system that provides rich digital twin functions and services; it can also be GE Predix: GE's Predix platform is a platform designed for industrial applications and can be used to create complex digital twin models; it can also be IBM Watson IoT: IBM's Watson IoT platform provides digital twin functions, which can be used to create, test and deploy models of smart devices and systems; it can also be PTC ThingWorx: PTC's ThingWorx platform provides a series of tools and services for creating and managing digital twin models; it can also be Dassault Systèmes: Dassault Systèmes' 3DEXPERIENCE platform provides a complete set of digital twin solutions; it can also be Microsoft Azure Digital Twins: Microsoft's Azure Digital Twins is an IoT service that can be used to create, run and manage digital twins and global-scale digital twin environments; ANSYS Twin Builder: ANSYS's Twin Builder is a tool for creating, verifying and deploying system-level digital twin models.

[0074] The model verification and optimization unit is used to verify the constructed digital twin model to ensure the accuracy and effectiveness of the model. It can be verified by comparing it with the actual situation on site or through simulation experiments. According to the verification results, the model is optimized, such as adjusting model parameters, improving modeling methods, etc., until the model can accurately reflect the manufacturing process of the product and provide effective support for manufacturing decisions.

[0075] Among them, the cross-platform virtual platform construction module 52 includes: a preset unified data format unit, used to preset the unified data exchange format of the digital twin model; a unified data format unit, used to convert the digital twin model into the unified data exchange format; a cross-platform interface development unit, used to develop a data display interface, a model operation interface and a collaborative communication interface, and the data display interface, the model operation interface and the collaborative communication interface are compatible with VR devices, AR devices, PCs, tablets and mobile phones; a data synchronization and model collaboration unit, used to use a unified data exchange format and interface to realize data synchronization update and model collaborative operation of all terminals; a testing and optimization unit, used to test the platform and optimize and adjust it according to the test results.

[0076] Specifically, by using standardized data exchange formats, seamless integration of VR and AR systems is achieved, thereby realizing cross-platform data sharing and model collaboration. The unified data source can provide different data formats to facilitate synchronous display and collaboration across terminals.

[0077] Preset a unified data format unit. In order to achieve seamless integration of VR (virtual reality) and AR (augmented reality) systems, a unified data exchange format needs to be preset for the digital twin model. This format needs to be compatible with various platforms and devices to facilitate cross-platform data sharing and model collaboration. For example, an open data exchange format such as XML or JSON can be used.

[0078] The unified data format unit is used to convert the digital twin model into a unified data exchange format. The unified data exchange format facilitates synchronous display and collaboration across terminals. The data includes the data of the digital twin model and other related manufacturing data. The data source needs to have efficient data processing and query capabilities to provide real-time data support.

[0079] In order to achieve cross-platform collaboration, the cross-platform interface development unit needs to develop a set of cross-platform data display interfaces, model operation interfaces and collaborative communication interfaces. The data display interface, the model operation interface and the collaborative communication interface are compatible with VR devices, AR devices, PCs, tablets, mobile phones, etc.

[0080] The data synchronization and model collaboration unit is used to use a unified data exchange format and interface to achieve data synchronization updates and model collaboration operations on all terminals. For example, when one terminal operates a model, all other terminals can see the results of the operation in real time. At the same time, all terminals can operate the model to achieve multi-person collaboration.

[0081] The testing and optimization unit is used to test the virtual model digital platform to ensure that the cross-platform data sharing and model collaboration functions work properly, and to perform optimization and adjustments based on the test results.

[0082] Through the above steps, a cross-platform virtual model digital platform is constructed to realize cross-platform sharing and collaborative operation of digital twin models, thereby improving the efficiency and quality of the manufacturing process.

[0083] The process information collection and push module 53 includes:

[0084] The process information viewing unit is used to use the virtual model digital platform to view the production process of the product, adjust the product design according to the feedback of the problems in the production process, view the operating status of the equipment, and adjust the production parameters, such as Figure 4As shown, check the quality test results of products and solve quality problems;

[0085] Data analysis and decision-making unit, which is used to analyze data and predict product quality trends based on historical data and experience of product production using machine learning algorithms;

[0086] The status assessment and work allocation unit is used to assess the fatigue level of the staff in real time and dynamically coordinate the work intensity of the staff according to the fatigue level.

[0087] Specifically, the process information collection and push module also includes full life cycle collaborative management, which refers to the collaborative management of all participants in the entire life cycle of the product, including design, manufacturing, quality inspection and control, product maintenance and service. In this system, participants at any stage, no matter what platform or terminal they use, can interact and communicate through this system. For example, designers can view the production process of the product through the system and adjust the product design based on feedback from problems in the production process. Production operators can view the operating status of the equipment through the system and adjust production parameters as needed. Quality inspectors can view the quality inspection results of the product through the system to promptly discover and solve quality problems. This system can not only realize real-time data sharing and collaboration, but also perform data analysis and prediction based on historical data and experience to assist decision-making. For example, historical data can be analyzed through machine learning algorithms to predict product quality trends or predict equipment maintenance needs. Whether participants use VR systems, head-mounted AR systems, or PCs, tablets, mobile phones and other terminals, they can receive process information collected by the system, such as manufacturing pictures, on-site sounds and other multimodal information. The system can evaluate the status of personnel, analyze their fatigue level in real time, and guide the intensity of collaboration between different personnel, including dynamically allocating work intensity.

[0088] Through such full life cycle collaborative management and process information collection and push management, not only can production efficiency and product quality be improved, but also the utilization efficiency of equipment can be improved and maintenance costs can be reduced, thereby improving the economic benefits of the entire manufacturing process.

[0089] Wherein, the real-time process information push module 54 includes:

[0090] The federated learning unit is used to perform local model training on the process information data of each terminal and integrate and update the results of the local model training of all terminals; the data sharing unit is used to share the results of the integration and update across terminals to realize a real-time process information push system for multi-person collaborative manufacturing.

[0091] Specifically, it is necessary to establish a federated learning unit to achieve cross-terminal data sharing and local model training. In the federated learning unit, each terminal device (such as workstations, mobile phones, tablets, etc.) can share their data, but there is no need to send the data directly to the central server. Instead, they will train the local model, then integrate and update the results of the local model training and send them to the central server. The central server will share the integrated and updated results across terminals, update the global model, and then send the updated model back to each terminal device. This method can protect the privacy of the data and utilize the computing power of each device to realize a real-time process information push system for multi-person collaborative manufacturing. The multimodal information collected by the system is trained through the federated learning unit to build a real-time process information push system. The system can combine the computing power of different terminals and different platforms according to the needs and work intensity of the on-site staff, train in real time, and push manufacturing recommendation strategies on demand: including product manufacturing strategy recommendation, manufacturing status detection and early warning, etc.

[0092] Among them, the federated learning unit also includes: a data preprocessing and feature extraction unit, which is used to perform data cleaning, data formatting and feature extraction on the data of the process information of each terminal to obtain first data and first features; a model training and updating unit, which is used to use the first data and the first feature to perform repeated local model training, integrate and update the results of each local model training of all terminals, and continuously generate a global model until the global model reaches the preset performance; a demand information pushing unit, which is used to push product manufacturing strategy recommendations, manufacturing status detection and early warning information to the global model according to the needs and work intensity of on-site staff, as well as the training and prediction results of the global model; an iteration and optimization unit, which is used to continuously iterate and optimize the global model according to the update of the process information data of each terminal.

[0093] The data preprocessing and feature extraction unit, in the process of federated learning, needs to perform data cleaning, data formatting and feature extraction on the data of the process information described by each terminal. Data cleaning includes removing invalid, duplicate and erroneous data. Data formatting is to convert all data into a unified processable format, and feature extraction is to extract useful features from the original data to obtain first data and first features. These tasks can be performed on each terminal device to utilize their computing power.

[0094] The model training and updating unit uses the first data and the first feature to perform repeated local model training on each terminal device, integrates and updates the results of each local model training of all terminals, and continuously generates and updates the global model. This process is repeated until the global model meets the preset performance requirements.

[0095] The demand information push unit can be used to push demand information once the global model training is completed. The system pushes product manufacturing strategy recommendations, manufacturing status detection and early warning information to the global model based on the needs and work intensity of on-site staff and the training and prediction results of the global model. Then, the system can push these strategies to each device to guide their work.

[0096] Iteration and optimization unit, with the accumulation of more data and the addition of more equipment, the global model is continuously iterated and optimized according to the update of process information data of each terminal, and the performance of the model will be further improved. At the same time, the system also needs to adjust the push strategy according to the actual situation to meet the needs of on-site staff.

[0097] In general, multi-terminal demand information push based on federated learning can effectively utilize the data and computing power of each terminal device, while protecting the privacy of the data and realizing real-time and personalized demand information push.

[0098] See also Figure 5 , Figure 5 It is a flowchart of a preferred embodiment of the multi-person collaborative manufacturing method based on digital twin and augmented reality of the present application.

[0099] In some embodiments, Figure 5 As shown, based on the above-mentioned multi-person collaborative manufacturing system based on digital twin and extended reality, the present application also proposes a multi-person collaborative manufacturing method based on digital twin and extended reality, and the multi-person collaborative manufacturing method based on digital twin and extended reality includes:

[0100] Step S100: The digital twin model construction module integrates the manufacturing demand information, manufacturing process information and manufacturing process information of the product to construct a digital twin model of the entire process;

[0101] Step S200: The cross-platform virtual platform construction module processes the digital twin model in a standardized data exchange format to obtain a cross-platform virtual model digital platform;

[0102] Step S300: The process information collection and push module uses the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product;

[0103] Step S400: The real-time process information push module performs federated learning algorithm training on the process information to build a real-time process information push system for multi-person collaborative manufacturing.

[0104] Wherein, the step S100 specifically includes:

[0105] The multimodal information perception unit obtains product design documents, process cards, equipment parameters, and operator feedback on the entire manufacturing process;

[0106] The data classification and processing unit classifies and processes the product manufacturing process information and converts it into modeling data;

[0107] The physical model building unit builds a three-dimensional model using computer-aided design software according to the whole process information of the product manufacturing;

[0108] The digital twin model building unit uses digital twin modeling software to combine the modeling data and the three-dimensional model to build a digital twin model of the entire process;

[0109] The model verification and optimization unit verifies and optimizes the digital twin model using simulation experiments and parameter adjustment methods.

[0110] Wherein, the step S200 specifically includes:

[0111] Presetting a unified data format unit to preset a unified data exchange format of the digital twin model;

[0112] The unified data format unit converts the digital twin model into the unified data exchange format;

[0113] The cross-platform interface development unit develops a data display interface, a model operation interface, and a collaborative communication interface, wherein the data display interface, the model operation interface, and the collaborative communication interface are compatible with VR devices, AR devices, PCs, tablets, and mobile phones;

[0114] The data synchronization and model collaboration unit uses a unified data exchange format and interface to achieve data synchronization updates and model collaboration operations on all terminals;

[0115] The testing and optimization unit tests the virtual model digital platform and makes optimizations and adjustments based on the test results.

[0116] Wherein, the step S300 specifically includes:

[0117] The process information viewing unit uses the virtual model digital platform to view the production process of the product, adjust the product design according to the problem feedback in the production process, view the operation status of the equipment, adjust the production parameters, view the quality inspection results of the product, and solve the quality problems;

[0118] The data analysis and decision-making unit uses machine learning algorithms to analyze data and predict product quality trends based on historical data and experience of product production;

[0119] The status assessment and work allocation unit assesses the fatigue level of the staff in real time, and dynamically coordinates the work intensity of the staff according to the fatigue level.

[0120] Wherein, the step S400 specifically includes:

[0121] The federated learning unit performs local model training on the process information data of each terminal, and integrates and updates the results of the local model training of all terminals; the data sharing unit shares the integrated and updated results across terminals to realize a real-time process information push system for multi-person collaborative manufacturing.

[0122] Wherein, the step S400 further includes:

[0123] A data preprocessing and feature extraction unit, used for performing data cleaning, data formatting and feature extraction on the data of the process information of each terminal to obtain first data and first features;

[0124] A model training and updating unit, configured to perform repeated local model training using the first data and the first feature, integrate and update the results of each local model training of all terminals, and continuously generate a global model until the global model reaches a preset performance;

[0125] A demand information push unit, used to push product manufacturing strategy recommendations, manufacturing status detection and early warning information to the global model according to the needs and work intensity of the on-site staff and the training and prediction results of the global model;

[0126] The iteration and optimization unit is used to continuously iterate and optimize the global model according to the update of the process information data of each terminal.

[0127] In summary, this application sets a digital twin model construction module to integrate the manufacturing demand information, manufacturing process information and manufacturing process information of the product to build a digital twin model of the whole process; sets a cross-platform virtual platform construction module to use a standardized data exchange format for the digital twin model to build a cross-platform virtual model digital platform; sets a process information collection and push module to use the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product; sets a real-time process information push module to train the process information with a federated learning algorithm, build a real-time process information push system for multi-person collaborative manufacturing, effectively integrate the scattered and isolated modules, and realize full process management. Secondly, this application trains through a federated learning method based on the multimodal information in the collection process to build a real-time process information push system. The system can be trained in real time according to the needs and work intensity of the on-site staff, combined with the computing power of different terminals and different platforms, and push manufacturing recommendation strategies on demand, including product manufacturing strategy recommendations, manufacturing status detection and early warning, etc.

[0128] It should be noted that the various optional implementations introduced in the embodiments of the present application can be implemented in combination with each other or can be implemented separately, and the embodiments of the present application are not limited to this.

[0129] In the description of the present application, it should be understood that the terms "upper", "lower", "left", "right", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, and a specific orientation structure and operation. Therefore, it cannot be understood as a limitation on the present application. In addition, "first" and "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "multiple" means two or more.

[0130] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0131] The above embodiments are described with reference to the accompanying drawings, and other different forms and embodiments are also feasible without departing from the principles of the present application, so the present application should not be constructed as a limitation of the embodiments proposed herein. More specifically, these embodiments are provided so that the present application will be perfect and complete, and the scope of the present application will be conveyed to those skilled in the art. In the accompanying drawings, the component sizes and relative sizes may be exaggerated for clarity. The terms used herein are only based on the purpose of describing specific embodiments and are not intended to be limiting. The terms "comprising" and / or "including" when used in this specification indicate the presence of the features, integers, components and / or components, but do not exclude the presence or increase of one or more other feature integers, components, components and / or their groups. Unless otherwise indicated, when stated, the numerical range includes the upper and lower limits of the range and any sub-ranges therebetween.

[0132] The above descriptions are only some embodiments of the present application, and do not limit the protection scope of the present application. Any equivalent device or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-person collaborative manufacturing system based on digital twin and augmented reality, characterized in that: include: The digital twin model building module is used to integrate the product's manufacturing demand information, manufacturing process information, and manufacturing process information to build a digital twin model of the entire process; The digital twin model building module includes: Multimodal information perception unit, used to obtain product design documents, process cards, equipment parameters and operator feedback for the entire manufacturing process; A data classification and processing unit, used to classify and process the product manufacturing process information and convert it into modeling data; A physical model building unit is used to build a three-dimensional model using computer-aided design software based on the whole process information of product manufacturing; Constructing a digital twin model unit, which is used to use digital twin modeling software to combine the modeling data and the three-dimensional model to construct a digital twin model of the entire process; A model verification and optimization unit, used to verify and optimize the digital twin model using simulation experiments and parameter adjustment methods; A cross-platform virtual platform construction module is used to process the digital twin model in a standardized data exchange format to obtain a cross-platform virtual model digital platform; The cross-platform virtual platform building module includes: A preset unified data format unit, used to preset a unified data exchange format of the digital twin model; A unified data format unit, used to convert the digital twin model into the unified data exchange format; A cross-platform interface development unit, used to develop a data display interface, a model operation interface and a collaborative communication interface, wherein the data display interface, the model operation interface and the collaborative communication interface are compatible with VR devices, AR devices, PCs, tablets and mobile phones; Data synchronization and model collaboration unit, used to use a unified data exchange format and interface to achieve data synchronization updates and model collaboration operations on all terminals; Testing and optimization unit, used to test the virtual model digital platform and make optimization and adjustments based on the test results; A process information collection and push module, which is used to use the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product; The process information collection and push module includes: A process information viewing unit is used to view the production process of a product using the virtual model digital platform, adjust the product design according to the feedback of problems in the production process, view the operating status of the equipment, adjust the production parameters, view the quality inspection results of the product, and solve the quality problems; Data analysis and decision-making unit, which is used to analyze data and predict product quality trends based on historical data and experience of product production using machine learning algorithms; A status assessment and work allocation unit, used to assess the fatigue level of the staff in real time and dynamically coordinate the work intensity of the staff according to the fatigue level; A real-time process information push module is used to perform federated learning algorithm training on the process information and build a real-time process information push system for multi-person collaborative manufacturing; The real-time process information push module includes: A federated learning unit, used for performing local model training on the data of the process information of each terminal, and integrating and updating the results of the local model training of all terminals; A data sharing unit, used to share the integrated update results across terminals to implement a real-time process information push system for multi-person collaborative manufacturing; The federated learning unit also includes: a data preprocessing and feature extraction unit, which is used to perform data cleaning, data formatting and feature extraction on the data of the process information of each terminal to obtain first data and first features; a model training and updating unit, which is used to use the first data and the first features to perform repeated local model training, integrate and update the results of each local model training of all terminals, and continuously generate a global model until the global model reaches the preset performance; a demand information pushing unit, which is used to push product manufacturing strategies, manufacturing status detection and early warning information through the global model according to the needs and work intensity of on-site staff, as well as the training and prediction results of the global model; an iteration and optimization unit, which is used to continuously iterate and optimize the global model according to the update of the process information data of each terminal.

2. A multi-person collaborative manufacturing method based on digital twin and augmented reality, characterized in that: The following steps are involved: The digital twin model building module integrates the product's manufacturing demand information, manufacturing process information, and manufacturing process information to build a digital twin model of the entire process; The digital twin model building module integrates the manufacturing demand information, manufacturing process information and manufacturing process information of the product to build a digital twin model of the entire process, specifically including: The multimodal information perception unit obtains product design documents, process cards, equipment parameters, and operator feedback on the entire manufacturing process; The data classification and processing unit classifies and processes the product manufacturing process information and converts it into modeling data; The physical model building unit builds a three-dimensional model using computer-aided design software according to the whole process information of the product manufacturing; The digital twin model building unit uses digital twin modeling software to combine the modeling data and the three-dimensional model to build a digital twin model of the entire process; The model verification and optimization unit verifies and optimizes the digital twin model using simulation experiments and parameter adjustment methods; The cross-platform virtual platform construction module processes the digital twin model in a standardized data exchange format to obtain a cross-platform virtual model digital platform; The cross-platform virtual platform construction module processes the digital twin model in a standardized data exchange format to obtain a cross-platform virtual model digital platform, which specifically includes: Presetting a unified data format unit to preset a unified data exchange format of the digital twin model; The unified data format unit converts the digital twin model into the unified data exchange format; The cross-platform interface development unit develops a data display interface, a model operation interface, and a collaborative communication interface, wherein the data display interface, the model operation interface, and the collaborative communication interface are compatible with VR devices, AR devices, PCs, tablets, and mobile phones; The data synchronization and model collaboration unit uses a unified data exchange format and interface to achieve data synchronization updates and model collaboration operations on all terminals; The testing and optimization unit tests the virtual model digital platform and makes optimizations and adjustments based on the test results; The process information collection and push module uses the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product; The process information collection and push module uses the virtual model digital platform to perform full life cycle collaborative management and process information collection and push for each stage of the product, specifically including: The process information viewing unit uses the virtual model digital platform to view the production process of the product, adjust the product design according to the problem feedback in the production process, view the operation status of the equipment, adjust the production parameters, view the quality inspection results of the product, and solve the quality problems; The data analysis and decision-making unit uses machine learning algorithms to analyze data and predict product quality trends based on historical data and experience of product production; The status assessment and work allocation unit assesses the fatigue level of the staff in real time and dynamically coordinates the work intensity of the staff according to the fatigue level; The real-time process information push module performs federated learning algorithm training on the process information to build a real-time process information push system for multi-person collaborative manufacturing; The real-time process information push module performs federated learning algorithm training on the process information to build a real-time process information push system for multi-person collaborative manufacturing, specifically including: The federated learning unit performs local model training on the data of the process information of each terminal, and integrates and updates the results of the local model training of all terminals; The data sharing unit shares the integrated update results across terminals to realize a real-time process information push system for multi-person collaborative manufacturing; The federated learning unit also includes: a data preprocessing and feature extraction unit that performs data cleaning, data formatting and feature extraction on the data of the process information of each terminal to obtain first data and a first feature; a model training and updating unit that uses the first data and the first feature to repeatedly train the local model, integrates and updates the results of each local model training of all terminals, and continuously generates a global model until the global model reaches the preset performance; a demand information push unit pushes product manufacturing strategies, manufacturing status detection and early warning information through the global model according to the needs and work intensity of the on-site staff, as well as the training and prediction results of the global model; and an iteration and optimization unit that continuously iterates and optimizes the global model according to the update of the process information data of each terminal.

Images