An augmented reality based manufacturing plant maintenance assistance method

Abstract

The embodiment of the application discloses a kind of based on augmented reality's manufacturing workshop maintenance auxiliary method, it is related to intelligent manufacturing field, can alleviate the problem that existing scheme is higher in network quality and local equipment, network communication is greater in pressure etc. when cloud maintenance auxiliary scheme is applied on local workshop, the application includes: local computing server and AR glasses end are in same local area network, improve the stability of entire maintenance auxiliary system on task push.The maintenance auxiliary system of manufacturing workshop is formed by integrating local client, big data analysis cloud, effectively deal with abnormal state in workshop, improve the safety of workshop equipment operation and the efficiency of maintenance process.So as to avoid cloud big data system directly manages AR glasses terminal, reduce the management difficulty of cloud big data system, meanwhile, local computing server can be all-weather online, can keep connection with cloud big data system at any time, listen to maintenance task.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing, and in particular to a manufacturing workshop maintenance assistance method based on augmented reality. Background Technology

[0002] With the continuous development of the manufacturing industry, products are becoming increasingly diverse and varied, and more and more complex production equipment with complex functions and structures are being put into production. These factors have greatly increased the stability of production and the difficulty of maintenance, posing significant challenges to operators on the shop floor. Maintenance assistance can improve the survival and competitiveness of manufacturing enterprises and enhance the ability of manufacturing workshops to handle abnormal conditions within the workshop.

[0003] Traditional workshop maintenance support typically relies on a centralized monitoring system to detect data from various devices within the workshop, identify anomalies, and then dispatch personnel to handle them. Traditional paper or electronic maintenance manuals are inconvenient to carry, lack intuitive guidance, and require operators to sift through large amounts of text to extract useful information, making the process inconvenient and time-consuming.

[0004] Currently, some cloud-based maintenance assistance solutions have been put into practical use. However, these solutions often have some problems. For example, in medium to large-sized smart workshops with a large number of devices, multiple shifts of technicians are often required for maintenance. Each technician needs to use a local device and connect to the cloud, using a real-time communication method similar to live streaming to achieve a front-end maintenance + back-end support approach. However, this method also has some issues, such as high requirements for network quality and local devices. The cloud server needs to connect to many local devices simultaneously, which is equivalent to running multiple live streaming devices in a small area at the same time, putting a lot of pressure on network communication. Although some workshops have deployed 5G equipment, due to the large number of devices, the complex electromagnetic environment of the workshop, and communication blind spots caused by equipment obstruction, the data transmission from local devices to the outside is also affected. Ultimately, this results in less than ideal network communication quality between the cloud and local devices, making it difficult to effectively implement some maintenance assistance solutions that require real-time performance. Summary of the Invention

[0005] The embodiments of the present invention provide a manufacturing workshop maintenance assistance method based on augmented reality, which can alleviate the problems that occur when applying cloud-based maintenance assistance solutions on local workshops, such as high requirements for network quality and local equipment and high pressure on network communication in existing solutions.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] The method is used in a manufacturing workshop maintenance assistance system, which comprises: an AR glasses terminal, a local computing server, and a cloud-based big data system; the AR glasses terminal is worn by operators in the manufacturing workshop, the local computing server is deployed in the computer room of the manufacturing workshop, and the local computing server and the AR glasses terminal are connected via TCP / IP protocol; the cloud-based big data system is connected to the AR glasses terminal via a mobile communication network, and the cloud-based big data system is connected to the local computing server via a mobile communication network or the Internet.

[0008] The method includes:

[0009] S1. The local computing server pushes the maintenance task to the AR glasses terminal that is online;

[0010] S2. After the AR glasses terminal receives the maintenance task, the local computing server receives the data fed back by the AR glasses terminal and creates a task record corresponding to the maintenance task, and uploads the task record to the cloud big data system.

[0011] S3. The local computing server receives fault diagnosis information fed back by the cloud big data system, and the fault diagnosis information records the fault type;

[0012] S4. The local computing server matches the fault type in the fault diagnosis information with the existing fault types in the local maintenance expert database.

[0013] S5. If the matching is successful after S4, obtain the guidance information corresponding to the fault type; if the matching fails, start the remote expert system.

[0014] The augmented reality-based maintenance assistance method for manufacturing workshops provided in this invention ensures a reliable connection between the local computing server and the AR glasses on the same local area network, thus improving the stability of the entire maintenance assistance system in task delivery. By using the AR glasses as an augmented reality terminal, integrating the local client with cloud-based big data analysis, a maintenance assistance system for the manufacturing workshop is formed. This enables effective handling of abnormal states within the workshop, improving the safety of equipment operation and the efficiency of the maintenance process. This avoids direct management of the AR glasses terminal by the cloud-based big data system, reducing the management complexity of the cloud-based big data system. Simultaneously, the local computing server can be online 24 / 7, maintaining a constant connection with the cloud-based big data system and monitoring maintenance tasks. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 An architecture diagram of an augmented reality-based manufacturing workshop provided for embodiments of the present invention;

[0017] Figure 2 A timing diagram of an augmented reality-based manufacturing workshop system provided for embodiments of the present invention;

[0018] Figure 3 A flowchart for troubleshooting and repair guidance provided in this embodiment of the invention;

[0019] Figure 4 This is a schematic diagram of the method flow provided in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Embodiments of the present invention will be described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of the present invention means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0021] This invention provides an augmented reality-based manufacturing workshop maintenance assistance method. The method is used in a manufacturing workshop maintenance assistance system, which comprises: AR glasses, a local computing server, and a cloud-based big data system. The AR glasses are worn by operators in the manufacturing workshop. The local computing server is deployed in the workshop's server room and connects to the AR glasses via TCP / IP. The local computing server monitors the AR glasses' online status in real time using heartbeat packets. The cloud-based big data system connects to the AR glasses via a mobile communication network and to the local computing server via a mobile communication network or the internet.

[0022] In practical applications, such as Figure 1-3As shown, the AR glasses terminal is responsible for perceiving the surrounding environment, interacting with operators, and displaying virtual models and auxiliary prompts. The AR glasses terminal is a secondary development based on Microsoft HoloLens, utilizing open voice, gesture, and eye-tracking interfaces to build a human-computer interaction interface and acquire personnel operation information. It transmits the collected image data to the computing server for 3D registration and simultaneously receives maintenance tasks and fault information pushed from the local computing server. The human-computer interaction interface is developed in the Unity editor environment using the Microsoft Augmented Reality Development Kit. It calls interfaces such as voice, gesture, and eye-tracking to read operator information, including modules for personnel login, information viewing, anomaly display, fault repair guidance, and remote expert guidance. During maintenance, it can gradually and meticulously display maintenance content based on the current scene and personnel operations, improving equipment maintenance efficiency. The local computing server is responsible for managing the AR glasses terminal, coordinating workshop fault information and maintenance tasks sent from the cloud; managing the AR terminal is responsible for establishing a reliable push mechanism, realizing 3D registration, and storing a local maintenance expert database. The cloud-based big data system is responsible for processing equipment data in the workshop, analyzing potential faults, and also serves as a cloud-based expert system, facilitating communication between on-site and remote experts. The main function of this system is to analyze equipment operation data collected in the intelligent manufacturing workshop, utilize fault diagnosis algorithms such as deep transfer learning to predict faults in key equipment, and send predictive maintenance tasks to the workshop. Simultaneously, the cloud-based big data system also acts as a video server, providing a web interface for remote experts. The web interface also includes real-time viewing of operational data, management of existing datasets, and scheduling of maintenance tasks. The cloud-based big data system primarily processes equipment operation data collected in the intelligent manufacturing workshop, such as spindle vibration, temperature, current, and voltage—parameters closely related to the processing status. This data first undergoes data preprocessing, data denoising, and other data fusion processing, followed by fault feature extraction and fault feature matching to extract abnormal fault information. Finally, it calls fault diagnosis algorithms, such as deep transfer learning, to predict potential faults or existing anomalies in the workshop, generating corresponding maintenance tasks. These tasks are sent to the local computing server within the workshop via JSON or other formats, and then forwarded to the AR glasses terminal within the workshop.

[0023] like Figure 4 As shown, the method includes:

[0024] S1. The local computing server pushes the maintenance task to the AR glasses terminal that is online.

[0025] When a maintenance task needs to be performed, it is pushed to an online AR glasses terminal. If no AR glasses are online, the maintenance task is stored locally for later execution. The room management and push mechanism establishes virtual rooms based on the manufacturing workshop. After operators wear the glasses, they connect to a local computing server in the same workshop and go online in the virtual room. The local computing server determines the status of the AR glasses by receiving heartbeat detection packets from the AR glasses terminals. When a maintenance task needs to be performed, it is pushed to an online AR glasses terminal. If no AR glasses are online, the maintenance task is stored locally for later execution. The local computing server establishes a room management and push mechanism, creating virtual rooms for AR glasses within the same workshop. Glasses in the room can receive maintenance tasks from the computing server. The local server establishes a connection with the glasses via TCP / IP protocol. After obtaining camera access permissions, the interactive interface program installed on the glasses transmits the captured image data to the local computing server. The local computing server then calls 3D registration algorithms, such as ORB-SLAM2 and RGB-D SLAM, to achieve real-time tracking and positioning of objects in the environment. SLAM (simultaneous localization and mapping) is a technique that combines localization with map building.

[0026] S2. After the AR glasses terminal receives the maintenance task, the local computing server receives the data fed back by the AR glasses terminal and creates a task record corresponding to the maintenance task, and uploads the task record to the cloud big data system.

[0027] S3. The local computing server receives fault diagnosis information fed back by the cloud big data system, and the fault diagnosis information records the fault type;

[0028] S4. The local computing server matches the fault type in the fault diagnosis information with the existing fault types in the local maintenance expert database.

[0029] S5. If the matching is successful after S4, obtain the guidance information corresponding to the fault type; if the matching fails, start the remote expert system.

[0030] In this embodiment, after S1, the method further includes: the local computing server receiving an image sent by the AR glasses terminal, the image being captured by the AR glasses terminal through the glasses camera; the local computing server establishing a virtual room corresponding to the manufacturing workshop; and invoking a 3D registration algorithm and identifying the pose of the AR glasses terminal in the virtual room based on the image uploaded by the AR glasses terminal.

[0031] In this embodiment, identifying the pose of the AR glasses terminal in the virtual room includes: extracting RGB images and depth images from images sent by the AR glasses terminal, extracting features from the RGB images and performing feature matching, and estimating the pose of the AR glasses terminal using the matched feature points. The estimation of the pose of the AR glasses terminal includes: determining the optimal pose of the AR glasses terminal by solving for the minimum distance between pixel positions and spatial point positions.

[0032] Specifically, the process of determining the optimal pose of the AR glasses terminal includes:

[0033] The matched feature points are grouped to obtain at least two sets of point clouds, and one set of point clouds is p = {p1, p2, ..., p...} n} serves as a reference group, and the location of the reference group serves as a coordinate reference; another point cloud q = {q1, q2, ..., q n The points in} correspond one-to-one with the points in the reference group, p1 to p2. n For points in the reference set, q1~q n For another set of points in the point cloud, n is the total number of points; the error term for obtaining the i-th pair of points is e. i =p i -(Rq i +t), R represents the Euclidean transformation, and t satisfies The matching parameters R and t that minimize the error function are obtained by iteratively using the least squares method. This allows us to find the best camera pose, making camera pose estimation more accurate and providing a foundation for subsequent optimization and mapping.

[0034] Specifically: The ICP algorithm is typically used to solve for camera pose. One set of point clouds is used as the coordinate reference, and another set of point clouds is used to obtain the optimal transformation relationship through nonlinear iterative calculation. Assume two point sets p = {p1, p2, ..., p...} n},q={q1,q2,…,q n If there exists a Euclidean transformation R, t satisfying Then the error term for the i-th pair of points is defined as: e i =p i -(Rq i +t), and then use the least squares method to iteratively calculate the matching parameters R, t that minimize the error function. After acquiring the RGB and depth images of the environment, features of the RGB images are extracted and matched. The camera pose is estimated by the matched feature points. Then, the camera pose is optimized by a nonlinear optimization method to make the map globally consistent. Finally, the impact of tracking loss and accumulated error is reduced by loop closure and back-end.

[0035] Next, assume the coordinates of a certain spatial point are P. i =[X i ,Y i Z i ] T Its projected pixel coordinates are u i =[u i ,v i ] T Let K be the intrinsic parameter matrix of the camera, and ξ be the Lie algebra of the transformation matrix. Then the relationship between the pixel position and the spatial point position is: s i u i =Kexp(ζ^)P i Then, sum the errors to find the minimum value: Therefore, by solving for the minimum distance, the best camera pose can be found, making the camera pose estimation more accurate and providing a good foundation for subsequent optimization and mapping.

[0036] In this embodiment, the fault diagnosis information is obtained through a fault diagnosis model in the cloud-based big data system. This fault diagnosis model is a convolutional neural network (CNN), where the initial analysis phase of the CNN alternates between convolutional layers and subsampling layers. The fault diagnosis algorithm inputs the production process signal into the CNN. The network input graph is a 2D feature map or raw data graph. The initial analysis algorithm of the CNN alternates between convolution and subsampling, while the final algorithm approaching the output layer uses a conventional multilayer neural network.

[0037] In the convolutional layer, a convolution kernel is used to perform convolution operations on the feature map of the previous layer, and an activation function is used to construct the output feature map. Where M j Let represent the input feature map, l represent the l-th layer of the network, k represent the convolutional kernel, and b represent the network bias. Indicates the output of layer l. Indicates the input of layer l. This represents the convolution kernel in the l-th layer network, where i and j are both positive integers. Indicates the network bias of layer l;

[0038] After convolution, the activation function performs a non-linear transformation on each convolution output value, mapping the originally linearly inseparable multidimensional features to another space where the linear separability of the features is enhanced. There are three common activation functions: the Sigmoid function, the hyperbolic tangent function (Tanh), and the Rectified Linear Unit (ReLU). Their specific mathematical expressions are shown below:

[0039]

[0040]

[0041] a l(i，j) =f(y l(i，j) )=max{0,y l(i，j)}

[0042] a in the formula l(i,j) This indicates the output y of the convolutional layer. l(i,j) The activation value. Both Sigmoid and Tanh activation functions exhibit reciprocals approaching 0 when the input value is large, preventing error propagation and resulting in incomplete training of the lower-level network. The ReLU function, however, maintains a derivative of 1 for input values ​​greater than 0, effectively overcoming this issue. This embodiment employs this activation function. The calculation method used in the sampling layer is as follows: Where down() is the subsampling function and β is the network multiplicative bias. This represents the multiplicative bias of layer l in the network. The sampling layer scales the data map of the previous layer to reduce data dimensionality, ensuring the extracted features are scale-invariant and preventing overfitting. The computation method of its neurons can be represented as follows: Where down() is the subsampling function and β is the network multiplicative bias. The output layer of a CNN typically uses the cascaded feature map from the previous layer for a fully connected layer before the CNN, and usually employs a softmax classifier. Its advantage is that it can solve degree-based classification problems. The model can be represented as O = f(b o +w o f v ). Where f v b is the eigenvector. o w o Let be the deviation vector and the weight matrix.

[0043] Furthermore, a local maintenance expert database is established in the local computing server, and the local maintenance expert database is divided into at least three sub-databases, including: a fault collection database, a historical maintenance database, and a maintenance knowledge database;

[0044] The fault acquisition database stores fault type information;

[0045] The historical repair warehouse stores troubleshooting records for device malfunctions;

[0046] The maintenance knowledge base stores data explaining the working principles of the equipment and data guiding the disassembly and assembly of the equipment.

[0047] If a match is successful, the equipment information matching the fault type, the equipment operating principle manual, and the equipment disassembly and assembly guide will be sent to the AR glasses terminal as guidance information. Specifically, the obtained fault type can be matched with existing fault types in the local maintenance expert database. If a matching fault type is found, the corresponding equipment operating principle and common equipment disassembly and assembly guides will be shown to the operator, and a further guidance plan will be developed. If no type is matched with the local expert maintenance database, the remote expert system will be activated to request remote expert guidance to complete the maintenance work. This system is responsible for processing workshop equipment data, analyzing potential faults, and also for managing the cloud-based expert system, facilitating communication between on-site and remote experts. The local maintenance expert database includes a fault collection database, a historical maintenance database, and a maintenance knowledge base. The fault collection database is established based on common abnormal states in the workshop, the historical maintenance database records the investigation records of workshop abnormalities, and the maintenance knowledge base contains corresponding guidance content for common abnormalities in the fault collection database, including text, models, and animations, making it intuitive and efficient.

[0048] The local computing server establishes a task record for each maintenance task and saves images and video data captured during the maintenance process in the task record. When the maintenance task is completed, the local computing server stores the established task record in the historical maintenance database and / or uploads it to the cloud-based big data system. The task record stores photos of the maintenance process and the completion status of the maintenance task as feedback from the AR glasses terminal. The room management mechanism also receives feedback from the AR glasses during the maintenance process, can create a task record for each maintenance task, store photos of the maintenance process and the completion status of the maintenance task, and upload them to the cloud-based big data system, achieving effective feedback on maintenance tasks and providing a basis for subsequent maintenance task scheduling in the cloud-based big data system.

[0049] Specifically, after the remote expert system is activated, the process includes: the AR glasses terminal's interactive interface program acquiring local audio and video media and establishing a local audio and video track; then, initiating an SDP proposal request to the cloud-based big data system. The proposal request information includes Media Description Information (SDP) and the AR glasses terminal's device identification code. After obtaining the AR glasses terminal's mapping address on the public network, the WebRTC signaling server on the cloud-based big data system forwards the Media Description Information (SDP). Subsequently, the remote expert interface of the cloud-based big data system establishes WebRTC audio and video communication with the AR glasses terminal. The video server provides a convenient way to connect the workshop site with remote experts for audio and video communication. It utilizes WebRTC technology, which runs stably on both the glasses platform and the web interface. The process of establishing a connection using WebRTC technology mainly involves the following steps: the interactive interface program on the AR glasses obtains local audio and video media, establishes local audio and video tracks, initiates a request to the remote expert system, sends the local media description information (SDP), which mainly includes the terminal media functions and preferred preferences, obtains its own mapping address in the public network from the open STUN server, forwards the SDP through the signaling server mounted on the cloud big data system, and establishes a network connection using ICE technology.

[0050] In one possible practical application of this embodiment, the human-computer interaction interface program on the AR glasses terminal is mainly developed using the Unity editor and Microsoft's open augmented reality development kit. It is then published as a UWP platform application and installed on the glasses. The developed interface can be published on different platforms by changing the development environment. The human-computer interaction interface includes functions such as personnel login, information viewing, anomaly display, fault repair guidance, and remote expert guidance. The human-computer interaction interface program also includes an image transmission script. After obtaining camera permissions on the glasses, it transmits color and depth images to a local computing server for 3D registration. The local computing server receives the images transmitted by the glasses, calls the 3D registration algorithm, identifies the current scene, and tracks and locates objects. It monitors diagnostic results and repair tasks from the cloud-based big data system and pushes them to the online AR glasses terminals in the workshop. When operators wear AR glasses to perform repair tasks, the local computing server can identify the repair scene based on the acquired images and retrieve corresponding repair assistance content from the local repair expert database according to the repair task. The local repair expert database stores intuitive guidance text and animations, mainly including equipment models, part models, workpiece models, key part animations, and repair process animations. 3D reconstruction employs RGB-D SLAM technology for 3D registration. After acquiring RGB and depth images of the environment, features are extracted from the RGB images and matched. Camera pose is estimated using the matched feature points. Then, a nonlinear optimization method is used to optimize the camera pose. Finally, the least squares method is used iteratively to calculate the matching parameters R and t that minimize the error function. The cloud-based big data system primarily collects equipment operation data within the smart manufacturing workshop, such as spindle vibration, temperature, current, and voltage—parameters closely related to the processing status. This data first undergoes data preprocessing, noise reduction, and other data fusion processing. Then, it goes through fault feature extraction and matching processes to extract abnormal fault information. Finally, it calls fault diagnosis algorithms, such as deep migration algorithms, to predict potential faults or existing anomalies in the workshop, generating corresponding maintenance tasks. These tasks are sent to the local computing server within the workshop via JSON or similar formats, and then forwarded to AR glasses terminals within the workshop. The cloud-based big data system also acts as the signaling server in a remote expert system based on WebRTC technology, providing a web interface for remote experts to communicate between on-site and remote experts, thus shortening the repair time for complex anomalies.

[0051] Augmented reality (AR) is a novel display technology that uses computer graphics to overlay equipment models, text, images, and other prompts onto real equipment, providing intuitive guidance during maintenance. While AR technology holds great promise, many challenges remain in practical applications. For example, how to analyze equipment information within the workshop, reliably push potential fault information and maintenance tasks to AR terminals, and provide effective maintenance guidance to form a closed-loop maintenance assistance process—in other words, current maintenance assistance functions are not yet effectively implemented. Therefore, AR-based maintenance assistance systems still require further improvement to better meet the needs of modern smart manufacturing workshops.

[0052] In this embodiment, based on the actual needs of handling abnormal states in the manufacturing workshop, an auxiliary system comprising a local computing server and a cloud data server is constructed using HoloLens glasses as the AR terminal platform. This can be understood as establishing a practically applicable room management mechanism, avoiding direct management of the AR glasses terminal by the cloud big data system, reducing the management difficulty of the cloud big data system. Simultaneously, the local computing server can be online 24 / 7, maintaining a constant connection with the cloud big data system to monitor maintenance tasks. The local computing server and the AR glasses are on the same local area network, ensuring a reliable connection and thus improving the stability of the entire maintenance assistance system in task delivery. By using HoloLens glasses as an augmented reality terminal, integrating the local client, and combining big data analysis with the cloud to form the manufacturing workshop's maintenance assistance system, effective handling of abnormal states within the workshop is achieved, improving the safety of equipment operation and the efficiency of the maintenance process. The main advantages of this implementation scheme are: relying on the AR terminal, the form of maintenance assistance can be changed, the guidance process is more intuitive, eliminating the need to read large amounts of text to obtain maintenance instructions, freeing the operators' hands, and improving maintenance efficiency. By transmitting images acquired by AR to a local computing server, the computational task of 3D registration is handled by the server, reducing the load on the glasses and ensuring the speed of 3D registration and the stability of the terminal's human-machine interface. Through a room management mechanism, the responsibility of directly pushing maintenance tasks to AR glasses terminals is delegated to the local computing server within the workshop, reducing the complexity of big data cloud management of AR terminals and improving the reliability of maintenance task pushes. Simultaneously, the maintenance process and task completion status are recorded. A cloud database is used to monitor the manufacturing workshop's operational status, avoiding the need for large servers in the workshop and ensuring stable server operation. Unified cloud monitoring of multiple manufacturing workshops is possible. The big data cloud serves as a remote video streaming server, establishing a communication channel between on-site experts in the manufacturing workshop and remote experts. The operator's first-person perspective is transmitted to experts in real time, providing the most intuitive information and reducing communication barriers during the maintenance process.

[0053] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The above descriptions are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A manufacturing workshop maintenance assistance method based on augmented reality, characterized in that, The method is used in a manufacturing workshop maintenance assistance system, the components of which include: an AR glasses terminal, a local computing server, and a cloud big data system; The method includes: S1. The local computing server pushes the maintenance task to the AR glasses terminal that is online; S2. After the AR glasses terminal receives the maintenance task, the local computing server receives the data fed back by the AR glasses terminal, creates a task record corresponding to the maintenance task, and uploads the task record to the cloud big data system. S3. The local computing server receives fault diagnosis information fed back by the cloud big data system, and the fault diagnosis information records the fault type; S4. The local computing server matches the fault type in the fault diagnosis information with the existing fault types in the local maintenance expert database. S5. If the matching is successful after S4, obtain the guidance information corresponding to the fault type; if the matching fails, activate the remote expert system. The cloud-based big data system obtains the fault diagnosis information through a fault diagnosis model, which is a convolutional neural network. The initial analysis stage of the convolutional neural network is performed alternately by convolutional layers and subsampling layers. In the convolutional layer, a convolution kernel is used to perform convolution operations on the feature map of the previous layer, and an activation function is used to construct the output feature map. ,in Indicates the input feature map, Indicates the first Layered network, Represents the convolution kernel. Indicates network bias. express Layer output, express Layer input, This represents the convolution kernel in the l-th layer network, where i and j are both positive integers. express Layer network bias; The calculation method used in the sampling layer is as follows: ,in For subsampling functions, For network multiplicative bias, express Multiplicative bias of the network layer; When the remote expert system is activated, the process also includes: the AR glasses terminal's interactive interface program acquiring local audio and video media and establishing a local audio and video track, then initiating an SDP proposal application to the cloud-based big data system. The proposal application information includes media description information (SDP) and the AR glasses terminal's device identification code. After obtaining the AR glasses terminal's mapping address on the public network, the WebRTC signaling server on the cloud-based big data system forwards the media description information (SDP), and then the remote expert interface of the cloud-based big data system establishes WebRTC audio and video communication with the AR glasses terminal.

2. The method according to claim 1, characterized in that, Following S1, it also includes: The local computing server receives images sent by the AR glasses terminal, the images being captured by the AR glasses terminal through the glasses' camera. The local computing server establishes a virtual room corresponding to the manufacturing workshop; The 3D registration algorithm is invoked, and the pose of the AR glasses terminal in the virtual room is identified based on the image uploaded by the AR glasses terminal.

3. The method according to claim 2, characterized in that, The process of identifying the pose of the AR glasses terminal in the virtual room includes: From the image sent by the AR glasses terminal, RGB image and depth image are extracted, features of the RGB image are extracted and feature matching is performed, and the pose of the AR glasses terminal is estimated using the matched feature points. The estimation of the pose of the AR glasses terminal includes: The optimal pose of the AR glasses terminal is determined by solving for the minimum distance between the pixel position and the spatial point position.

4. The method according to claim 3, characterized in that, The process of determining the optimal pose of the AR glasses terminal includes: The matched feature points are grouped to obtain at least two sets of point clouds, and one set of point clouds is then... As a reference group, the location of the reference group serves as the coordinate reference; another set of point clouds The points in the reference group correspond one-to-one with the points in the reference group. For the points in the reference set, Let n be the number of points in another point cloud; Get the The error term for the point is R represents the Euclidean transformation, and t represents any pair of points; The matching parameters that minimize the error function are obtained by iteratively using the least squares method. , , .

5. The method according to claim 1, characterized in that, The local computing server establishes the local maintenance expert database, which is divided into at least three sub-databases, including: a fault collection database, a historical maintenance database, and a maintenance knowledge database. The fault acquisition database stores fault type information; The historical repair warehouse stores troubleshooting records for device malfunctions; The maintenance knowledge base stores data explaining the working principles of the equipment and data guiding the disassembly and assembly of the equipment. If a match is successful, the device information matching the fault type, the device operating principle manual, and the device disassembly and assembly guide will be sent to the AR glasses terminal as guidance information.

6. The method according to claim 1, characterized in that, Also includes: The local computing server establishes a task record for the corresponding maintenance task and saves the image and video data taken during the maintenance process in the task record; Once the maintenance task is completed, the local computing server stores the established task record in the historical maintenance database and / or uploads it to the cloud big data system.

7. The method according to claim 1, characterized in that, The AR glasses terminal is worn by operators in the manufacturing workshop, and the local computing server is deployed in the computer room of the manufacturing workshop. The local computing server and the AR glasses terminal establish a connection through the TCP / IP protocol. The cloud-based big data system connects to the AR glasses terminal via a mobile communication network, and the cloud-based big data system connects to the local computing server via a mobile communication network or the Internet.

Images