Image recognition system based on AR intelligent glasses

Abstract

The invention belongs to the field of industrial intelligent maintenance, and mainly relates to an image recognition system based on AR intelligent glasses, which comprises a data acquisition module, a data processing module and an AR intelligent glasses module, the data processing module dispatches the AR intelligent glasses module to a fault device, obtains a field video through a video synchronization function, identifies and confirms a problem by using a central image processing algorithm, and generates a maintenance scheme; the AR intelligent glasses module identifies key features of the equipment through voice guidance and a local image processing algorithm, marks key points, and assists an operator in completing maintenance work; image recognition is applied to an industrial scene, and the image recognition is combined with equipment data to comprehensively perceive the equipment operation state, so that the recognition accuracy is improved; in addition, maintenance guidance is provided for operators in combination with image recognition, and the operation difficulty of the operators is reduced.

Description

An image recognition system based on AR smart glasses Technical Field

[0001] This invention belongs to the field of industrial intelligent maintenance and mainly relates to an image recognition system based on AR smart glasses. Background Technology

[0002] An image recognition system is an intelligent system that acquires images of target scenes or objects through image acquisition devices, and uses algorithms to analyze and process the images to identify the type, location, state, or abnormal information of objects in the images. This system can perform information extraction, state judgment, and result output, and is widely used in fields such as industrial inspection, medical diagnosis, and security monitoring.

[0003] Chinese patent application number CN202111093228.6 discloses an image recognition system based on AR smart glasses. This system, belonging to the AR smart technology field, includes an infrared capture module, an image acquisition module, an image preprocessing module, a first feature acquisition module, a second feature acquisition module, a storage module, a database, a comparison module, a search import module, a GPS positioning module, a route planning module, a size acquisition module, a judgment module, an image generation module, and a display module. The GPS positioning module is communicatively connected to the route planning module; the infrared capture module is communicatively connected to the image acquisition module. This invention can quickly recognize images, reduce lag in AR smart glasses, improve the user experience, accurately pinpoint the user's location, and provide clear and intuitive guidance.

[0004] The aforementioned invention has provided a relatively complete image recognition system. However, in the field of industrial intelligent maintenance, the complex and diverse industrial environment, such as changes in lighting, dust obstruction, and diverse equipment types, makes it difficult for image recognition systems to capture key abnormal features in a timely and accurate manner. Secondly, image recognition systems can only extract features from surface images and cannot comprehensively perceive the operating status of equipment. Even if key abnormal features are captured, the lack of specific equipment data can lead to inaccurate fault identification results, posing a risk of misjudgment or omission. Furthermore, during equipment fault maintenance, different equipment models and fault types vary significantly, often requiring operators to consult equipment manuals extensively and rely on experience to make judgments and operations. This results in low maintenance efficiency, poor accuracy, and a high risk of misoperation or repeated repairs, affecting the stability of the entire production process. Therefore, an image recognition system is needed that has an image acquisition mechanism adaptable to complex environments, the ability to integrate images and equipment operating data for joint judgment, and provides real-time feedback and interactive maintenance assistance functions. Summary of the Invention

[0005] This invention provides an image recognition system based on AR smart glasses, aiming to solve the problems of inaccurate recognition and inability to provide real-time maintenance assistance to operators caused by the single recognition dimension and lack of comprehensive perception of equipment in the field of industrial intelligent maintenance of existing image recognition technology.

[0006] To address the aforementioned issues, this invention employs the following technology: an image recognition system based on AR smart glasses: a data acquisition module is deployed, sensors are installed on industrial equipment, and sensor data is transmitted to a data processor. The data processor preprocesses the data, encapsulates it into a structured message format, and transmits it to a data processing module. The data processing module receives the structured messages transmitted by the data acquisition module, parses them into equipment data and equipment location information, uses anomaly feature identification methods to determine if the equipment is faulty, receives real-time video footage from the site, calls a central image recognition algorithm to extract key features from the video footage, uses a clustering offset algorithm for analysis to determine the specific fault, and formulates a maintenance plan based on a solution library and a historical maintenance record database. After maintenance is completed, the maintenance information is stored, and deep learning is performed on the maintenance information using a convolutional neural network (CNN) combined with a recurrent neural network (RNN). The deep learning results are input into a solution induction model to update the solution library. An instruction set is sent to the AR smart glasses module. The AR smart glasses module receives and parses the instruction set from the data processing module, records and synchronizes the on-site video to the data processing module via a wireless network, and parses the maintenance plan from the instruction set to assist operators in maintenance work.

[0007] In a preferred embodiment, the data acquisition module specifically includes: several data processors, each corresponding to an industrial device; several sensors under each data processor, the type of which is determined according to the type of industrial device; a preprocessing unit built into the data processor, the specific steps of which are: unit conversion and format regularization of the raw data, appending corresponding timestamps and device identifiers, performing noise reduction processing, and encapsulating all preprocessed device data into a structured message format; the data acquisition module uploads the structured message to the data processing module through an industrial optical fiber communication network.

[0008] In a preferred implementation, the data processing module specifically includes: equipment analysis function, personnel scheduling function, central image recognition algorithm, solution formulation function, and solution library self-updating and learning function; the equipment analysis function specifically parses structured messages into equipment data and specific location information, determines whether the equipment has malfunctioned based on a comparison of the equipment data with normal data range, and locates the malfunctioning equipment based on the specific location information; the personnel scheduling function specifically dispatches the operator with the shortest straight-line distance to the malfunctioning equipment for maintenance; the central image recognition algorithm specifically performs image content analysis, feature extraction, and target recognition on the acquired video footage to pinpoint the problem; the solution formulation function specifically retrieves matching maintenance steps from a preset method library, formulates a maintenance solution suitable for the current problem, and sends it to the AR smart glasses module in real time; the solution library self-updating and learning function specifically records the processing solution to the historical maintenance record database; and uses a convolutional neural network (CNN) combined with a recurrent neural network (RNN) to perform deep learning on the records to dynamically supplement and optimize the existing method library.

[0009] In a preferred embodiment, the AR smart glasses module specifically includes: video synchronization function, brightness detection function, dual-camera switching function, interface display and voice function, local image recognition function, and automatic cleaning component; the video synchronization function specifically includes: video recording and synchronous uploading, continuously acquiring on-site images and recording video through the built-in camera component; simultaneously, uploading the video content during the recording process to the data processing module via wireless network; the brightness detection function specifically includes: implemented by a photosensor installed at the front end of the AR smart glasses to obtain the ambient brightness value of the current scene; triggering the dual-camera switching function based on the ambient brightness value; the dual-camera switching function specifically includes: the AR smart glasses include a visible light camera and an infrared camera, enabling image acquisition under different lighting conditions. Automatic adaptation; the interface display and voice functions specifically involve: visually presenting the maintenance plans, equipment status, and operation instructions generated by the data processing module by overlaying them onto the operator's field of vision using augmented reality (AR); allowing voice recording and calls; the local image recognition function specifically involves: integrating a lightweight image recognition algorithm model, deploying it on the local processing chip of the AR smart glasses, and recognizing and processing images within the field of vision based on the local computing resources of the AR smart glasses; using the local image recognition algorithm to perform grayscale analysis and control the on / off state of the automatic cleaning component; the automatic cleaning component specifically includes: a flexible dust removal strip and a drive mechanism set in front of the camera to clean particles or stains attached to the lens surface; the positioning function specifically refers to: a built-in Wi-Fi module that uses Wi-Fi positioning technology to upload the location of the AR device in real time.

[0010] In a preferred embodiment, the specific steps for determining whether the device is faulty include: the data processing module receives the structured message uploaded by the data processor, parses the message, extracts the device data and device location information, and locates the specific faulty device entity; the data processing module compares and analyzes the currently collected device data with the normal parameter range through abnormal feature identification to determine the device status; and the data processing module infers the device module or component area that may be involved in the fault.

[0011] As a preferred implementation, the specific steps for receiving real-time video feeds from the site include: real-time monitoring of the status information of all currently online workers, and obtaining the location data of online workers via Wi-Fi positioning. Specifically, the AR smart glasses module's built-in Wi-Fi module periodically scans the signal strength information (RSSI) of surrounding wireless access points (APs) and uploads the scan results to the data processing module. The data processing module calculates the three-dimensional coordinates of the AR smart glasses' location based on a real-time three-point ranging algorithm. Using the principle of closest spatial distance, the worker closest to the target device is selected as the scheduling target. The three-dimensional Euclidean distance calculation method is used to calculate the spatial distance between each worker and the faulty equipment. The three-dimensional Euclidean distance calculation formula is: Where D is the straight-line distance between the dispatcher and the faulty equipment. The horizontal coordinates of the dispatcher. Let be the vertical coordinates of the dispatcher. The coordinates of the dispatcher in the vertical direction. The horizontal coordinates of the faulty equipment. The coordinates of the faulty device are in the vertical direction. The system determines the height coordinates of the faulty equipment; identifies the dispatcher and issues the task instruction to the AR smart glasses module they are wearing; the operator wearing the AR smart glasses arrives at the faulty equipment site after receiving the task instruction; the operator activates the video synchronization function of the AR smart glasses module; the AR smart glasses module uses its brightness detection function to detect the light intensity of the current working environment in real time; the AR smart glasses module activates the dual-camera switching function based on the brightness detection results; the local image processing algorithm analyzes the grayscale in the image in real time and controls the automatic cleaning components; and the recorded video is uploaded to the data processing module in real time.

[0012] As a preferred implementation, the specific steps for determining the specific fault include: the data processing module analyzes the video footage using a central image recognition algorithm; the central image recognition algorithm uses the YOLOv8 general object detection model as its basic network structure and pre-trains it using a large-scale industrial equipment image dataset; representative on-site image samples are collected from the target deployment environment to construct an enhanced training set; the on-site images are input into the model, the detection layer is fine-tuned and trained, and evaluated according to the evaluation metric Precision; key parts of the equipment in the images are located and labeled; a clustering offset algorithm is used to identify whether there are abnormal features, and a K-means clustering algorithm is used to cluster the image features of the key structures of the equipment; the collected image features are vectorized, and the Euclidean distance between the image feature vector and the center of each standard feature cluster is calculated; when the image features are in the fuzzy boundary region between multiple feature clusters, the voice function of the AR smart glasses prompts the on-site operators to adjust the recording range and continue the recognition and comparison process to locate the fault location.

[0013] As a preferred implementation, the specific steps for formulating a maintenance plan by combining the solution library and the historical maintenance record database include: the data processing module extracts the core diagnostic results of the current fault, retrieves the preset maintenance plan from the method library, and filters out standardized maintenance plans through Boolean rules; the data processing module calls the historical maintenance record database to find similar successful maintenance cases, performs correlation analysis on key behaviors, and optimizes the maintenance plan; the data processing module defines the maintenance plan in the form of a step chain, including equipment data and equipment information, packages it in the form of a structured message, and sends it to the corresponding AR smart glasses module.

[0014] As a preferred implementation, the specific steps for assisting maintenance personnel include: the AR smart glasses module receiving and parsing the maintenance plan sent by the data processing module, and assisting professionals according to the maintenance plan, specifically including two parts: voice guidance and image interaction prompts; the voice guidance part specifically involves the AR smart glasses module playing the voice instructions for each maintenance operation step in sequence; the image interaction prompts specifically involve the AR smart glasses module calling the built-in local image recognition algorithm to identify the key structural features involved in the current step; combining device information, and based on the lightweight target detection model Tiny-YOLO, detecting the device screen and proposing candidate regions; further extracting its image feature vector, and identifying the image region closest to the standard component features through cosine similarity, the cosine similarity calculation formula is: Where A is the current image feature vector, B is the standard template feature vector, and n is the feature dimension; the AR smart glasses module marks key structural areas on the interface in a layered manner; the operator performs maintenance operations according to the prompts; after maintenance is completed, the operator confirms via voice through the AR smart glasses, and the video synchronization function is turned off.

[0015] As a preferred implementation, the specific steps of updating the solution library include: the data processing module writes the maintenance process records as new entries into the historical maintenance record database; the built-in deep learning model is called to train and summarize the historical maintenance records, and maintenance records are extracted in batches from the historical database to generate sample data. Specifically, for image data, a convolutional neural network (CNN) is used to extract deep feature vectors of key region images; for speech content and step sequences, a recurrent neural network (RNN) is used to learn the logical relationship between semantic flow and steps, and to discover common effective processing paths; the obtained sample data is input into the solution summarization model to evaluate the effectiveness index of each type of maintenance path, and to prioritize and label the maintenance solutions in the existing method library. The solution summarization model is a classification optimization mechanism based on rule learning and feature aggregation, which constructs a mapping relationship between fault types and processing procedures; the optimized maintenance solutions are rewritten into the solution library.

[0016] The beneficial effects of this invention are as follows: 1. By using brightness detection, dual-camera switching, and automatic cleaning components, it adapts to industrial environments, ensuring the acquisition of clear video images and image recognition quality. Through image recognition algorithms combined with data acquisition and analysis functions, it comprehensively perceives the operating status of equipment and accurately locates equipment faults; 2. It automatically formulates maintenance plans based on the solution library, displays the operation parts and provides voice prompts through the AR smart glasses interface, and achieves standardized and visual maintenance guidance, reducing the operational difficulty for operators; 3. It stores data such as video, voice, fault types, and operation steps from each maintenance process to build a historical database. Through inductive analysis, it continuously optimizes the method library, achieving self-learning evolution.

[0017] Figure 1 is a flowchart of an image recognition system based on AR smart glasses according to the present invention. Detailed Implementation

[0018] To make the technical means, creative features, and achieved objectives and effects of this invention easier to understand, the invention is further described below with reference to specific embodiments. However, the following embodiments are merely preferred embodiments of this invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments described herein without creative effort are all within the protection scope of this invention. Unless otherwise specified, the experimental methods in the following embodiments are conventional methods, and the materials and reagents used in the following embodiments are commercially available unless otherwise specified.

[0019] Example 1, as shown in Figure 1, illustrates a flowchart of an image recognition system based on AR smart glasses. This example provides an image recognition system based on AR smart glasses, specifically including the following steps: Step 1: Constructing an image recognition system, including a data acquisition module, an AR smart glasses module, and a data processing module; wherein, the data acquisition module includes: a data processor, which corresponds to an industrial device, and the data processor oversees multiple sensors; the sensor types correspond to the industrial device; the data processor has a built-in preprocessing unit; in this invention, the industrial device may include the following categories: such as conveying equipment, rotating equipment, fluid control equipment, processing equipment, etc.; specific sensor types include: current sensors, temperature sensors, vibration sensors, speed sensors, pressure sensors, flow sensors, etc.

[0020] The AR smart glasses module includes: video synchronization, brightness detection, dual-camera switching, interface display and voice functions, local image recognition, and an automatic cleaning component. Specifically, the video synchronization function allows for video recording and simultaneous uploading. While workers are wearing the glasses and performing maintenance tasks, the built-in camera continuously captures images and records video. Simultaneously, the recorded video is wirelessly uploaded to the data processing module. The brightness detection function is implemented using a photosensor mounted on the front of the AR smart glasses to obtain the ambient brightness value. Based on the ambient brightness value, the dual-camera switching function is triggered. This dual-camera switching function includes both a visible light camera and an infrared camera, used to automatically adapt image acquisition under different lighting conditions. The interface display and voice functions are also included. The visualization and voice functions specifically involve: overlaying information such as maintenance plans, equipment status, and operation instructions generated by the data processing module onto the operator's field of vision in an augmented reality (AR) manner for visualization; allowing voice recording and calls; the local image recognition function specifically involves: integrating a lightweight image recognition algorithm model, deploying it on the local processing chip of the AR smart glasses, and recognizing and processing images within the field of vision based on the glasses' local computing resources; performing grayscale analysis through the local image recognition algorithm to control the on / off state of the automatic cleaning component; the automatic cleaning component specifically includes: a flexible dust removal strip and a drive mechanism set in front of the camera, the flexible dust removal strip can move horizontally along the lens surface under the drive mechanism to clean particles or stains attached to the lens surface; the positioning function specifically refers to: using Wi-Fi positioning technology to upload the location of the AR device in real time.

[0021] The data processing module specifically includes: equipment analysis, personnel scheduling, a central image recognition algorithm, solution formulation, and a solution library self-updating and learning function. Specifically, the equipment analysis function combines equipment type, operating parameters, and historical status to determine if the target equipment is faulty; after determination, it locates the faulty equipment based on identifiers in the equipment data. The personnel scheduling function dispatches personnel with the shortest linear distance to the faulty equipment for maintenance. The central image recognition algorithm analyzes the acquired video footage, extracts features, and identifies targets, pinpointing key structural areas, damaged parts, or abnormal component conditions causing equipment malfunctions such as leaks, cracks, loosening, and wear. The solution formulation function specifically involves: after a device malfunction is confirmed, based on the identified malfunction type, abnormal characteristics, and corresponding device information, automatically retrieving and matching maintenance steps from a preset method library using multi-field matching and Boolean rule filtering, formulating a maintenance plan applicable to the current problem, including the operational objectives, tool requirements, safety tips, and key structural parts requiring attention for each step, and sending it to the AR smart glasses module in real time; the solution library self-updating and learning function specifically involves: recording processing plans to the historical maintenance record database; recording key steps and final results during the execution process, using a convolutional neural network (CNN) combined with a recurrent neural network (RNN) to perform deep learning on the records, and dynamically supplementing and optimizing the existing method library.

[0022] Step Two: The data processor in the data acquisition module receives and integrates raw monitoring data from multiple sensors, including electrical data, temperature data, vibration data, etc., and performs preprocessing operations on the raw data through the built-in preprocessing unit. The specific preprocessing operations include: the data processor first converts and standardizes the units of the data collected by different types of sensors, converting them all into a unified digital expression structure and attaching corresponding timestamps and device identifiers; it then performs noise reduction processing on the raw signals, using a sliding window-based average filtering method or median filtering algorithm to eliminate abrupt changes and high-frequency interference; finally, it encapsulates all the preprocessed device data into a structured message format, which contains fields such as unique device identifier, acquisition time, sensor type, and device data.

[0023] After preprocessing, the data acquisition module uploads the structured message containing the equipment data to the data processing module in real time through the industrial fiber optic communication network.

[0024] Step 3: After receiving the structured message uploaded by the data processor, the data processing module first parses the message and extracts the device unique identifier, acquisition time, and device data.

[0025] The data processing module extracts the normal operating parameter ranges of the equipment, including electrical data, temperature data, vibration data, etc., from the equipment information database. It compares and analyzes the currently collected equipment data with the normal parameter ranges and judges the equipment status by identifying abnormal features. Specifically, it presets the normal threshold range and the rate of change range of each type of key parameter. If it detects that one or more key parameter data values ​​have exceeded the normal limit range, it determines that the equipment has malfunctioned.

[0026] In complex scenarios, specifically when the equipment operating status, environmental conditions, or data patterns are unusually complex and there is no single abnormal parameter that can be used to make an accurate judgment, the data processing module can also use a historical maintenance record database or a rule-based expert system to compare the current data features with existing fault modes to identify whether they match a specific fault type.

[0027] After determining that the target device has malfunctioned, the data processing module extracts the device's unique identifier and region information from the parsed message. The data processing module uses the device identifier as a search keyword to quickly locate the specific device entity that has malfunctioned by searching the existing device database, and at the same time obtains the device's location information.

[0028] Meanwhile, the data processing module calculates the difference between abnormal equipment data and normal values, identifying anomalies that deviate significantly from the normal state. Specifically, this refers to deviations between the current operating parameters of the equipment and the reference parameters under normal conditions that exceed a set threshold. For example, continuous variables such as temperature, pressure, and current may deviate by more than ±10% of the normal value. Based on the specific type and degree of deviation of the anomaly, including but not limited to temperature-related anomalies such as equipment casing temperature and bearing temperature, electrical anomalies such as current and voltage, pressure anomalies such as air pressure and flow rate, mechanical motion anomalies such as vibration and rotational speed, and structural image anomalies such as cracks and loosening, the module infers the equipment modules and component areas that may be involved in the fault.

[0029] The data processing module initiates the personnel scheduling function, monitoring the status information of all currently online workers in real time. It acquires the location data of online workers via Wi-Fi positioning. The specific steps are as follows: the glasses' built-in Wi-Fi module periodically scans the signal strength information (RSSI) of surrounding wireless access points (APs) and uploads the scan results to the data processing module. Based on a real-time three-point ranging algorithm, the data processing module calculates the three-dimensional coordinate information of the AR smart glasses' location, including horizontal position (x, y) and vertical height (z). Using the principle of closest spatial distance, the module selects the worker closest to the target device as the scheduling target. Specifically, the data processing module receives the location information uploaded by each online worker's AR smart glasses in real time, including planar coordinates and vertical height data. Using a three-dimensional Euclidean distance calculation method, considering both coordinate position and floor height, it calculates the spatial distance between each worker and the faulty equipment. The three-dimensional Euclidean distance calculation formula is: Where D is the straight-line distance between the dispatcher and the faulty equipment. The horizontal coordinates of the dispatcher. Let be the vertical coordinates of the dispatcher. The coordinates of the dispatcher in the vertical direction. The horizontal coordinates of the faulty equipment. The coordinates of the faulty device are in the vertical direction. The coordinates of the faulty equipment's location in the vertical direction are used; the person closest in spatial distance is selected as the dispatch target; after the dispatcher is determined, the task instruction is sent to the AR smart glasses module worn by the dispatcher, and the task content includes the equipment number, task number and subsequent operation prompts; after receiving the task, the person goes to the faulty equipment site to perform the maintenance procedure.

[0030] Step 4: After arriving at the site of the faulty equipment, the operator wearing AR smart glasses activates the video synchronization function through voice control commands on the AR smart glasses. Once activated, the AR smart glasses begin recording the video of the current work site in real time and simultaneously uploads the recorded video of the current work site to the data processing module via a wireless network connection.

[0031] The specific steps for activating the video synchronization function of AR smart glasses include: First, the brightness detection function starts running. The light sensor at the front of the AR smart glasses detects the light intensity of the current working environment in real time. Based on the detected brightness value, it determines whether to activate the dual-camera switching function: when the light intensity is higher than the set threshold, the default visible light camera is kept on for recording; when the light intensity is lower than the threshold, it automatically switches to the infrared camera to enhance the image clarity in low-light environments and ensure that the video image has good recognizability.

[0032] Then, after recording starts, the local image processing algorithm begins to run. This image processing algorithm has a grayscale detection function, which is used to analyze the grayscale distribution of the target area in the image in real time. Specifically, firstly, a preset target area image block is extracted in each frame of the image, the color image is converted into a grayscale image, and the grayscale value of each pixel is statistically calculated. By calculating the average value, variance value, and grayscale histogram distribution of the grayscale values ​​in the area, it is determined whether there are abnormalities such as the image being too dark, too bright, or grayscale concentration. If the average grayscale value is lower than the set threshold, or the grayscale variance is significantly low, it is determined that the image quality in that area is poor, possibly due to lens stains, dust, or other obstructions. By statistically analyzing and evaluating the image grayscale values, it is determined whether there is image blurring caused by dust, oil, or other obstructions on the lens. When the grayscale detection result indicates that the overall brightness of the image is insufficient or the grayscale is concentrated in the low grayscale area, it is determined that there may be lens contamination or obstruction, and the automatic cleaning component is activated.

[0033] Subsequently, the automatic cleaning component is activated, and the drive mechanism drives the flexible dust removal strip to move horizontally along the lens surface once to wipe away dust, oil, or moisture adhering to the lens surface.

[0034] The system ensures the clarity of recorded and uploaded videos through brightness detection, dual-camera switching, local image processing algorithms, and automatic cleaning functions, and finally uploads the videos to the data processing module simultaneously.

[0035] After receiving the video content uploaded by the AR smart glasses, the data processing module calls the central image recognition algorithm to analyze the video frame by frame. The central image recognition algorithm includes four core stages: image preprocessing, target detection, key region extraction and anomaly comparison.

[0036] First, in the image preprocessing stage, the uploaded video frames undergo denoising, grayscale normalization, and edge enhancement to improve subsequent recognition accuracy. Then, in the target detection stage, the algorithm calls the YOLO deep learning-based image recognition model to locate and label key parts of the equipment in the image. This YOLO model is pre-trained based on historical image data of the equipment type and adapted to the specific factory scene image style and feature distribution through transfer learning to ensure target detection accuracy. Specifically, the YOLO general target detection model is selected as the basic network structure, and pre-trained using a large-scale industrial equipment image dataset to give the model preliminary general feature extraction capabilities. Next, representative on-site image samples are collected from the target deployment environment. The images cover real scenes under various angles, working conditions, and lighting conditions for different equipment to maximize coverage of the image variation range in the factory environment. After collection, the image data undergoes standardization processing, including size normalization, brightness correction, and denoising, and is then processed using image processing algorithms. Enhancement techniques, such as rotation, flipping, blurring, and adding noise, are used to construct an enhanced training set, thereby improving the robustness of the model under different perturbations. This means that a system or algorithm can still maintain stable and reliable operation when faced with interference, changes, or uncertainties. During the model transfer phase, the weight parameters of the YOLO model's backbone network are kept frozen, and only the detection head is fine-tuned. By inputting on-site images into the model and combining them with the bounding boxes of labeled equipment parts or key components, the detection layer is fine-tuned and trained, enabling the model to more accurately identify key equipment parts in specific scenarios. After training, the fine-tuned model is evaluated on the validation set, with precision as the main evaluation metric. Once all metrics meet the set accuracy requirements, the model is deployed to the data processing module for operation. The specific accuracy requirements are: on the validation set, the average precision of the object detection task is not less than 90%; and the recognition accuracy for key parts such as motors and valves is not less than 95%.

[0037] In the key region extraction stage, local image features are extracted from the identified target regions, including but not limited to texture direction, color uniformity, edge continuity, and structural contour integrity, and converted into feature vectors. Finally, in the anomaly comparison stage, the extracted feature vectors are compared with the image features of the corresponding devices in normal state in the database. Clustering offset algorithm is used to identify whether there are abnormal features. The specific steps are as follows: when judging anomalies in video images uploaded by AR smart glasses, based on the set of device image features in normal state, the K-means clustering algorithm is used in advance to cluster the image features of the key structures of the device to form multiple standard feature clusters in normal state; each feature cluster represents a known healthy structural state.

[0038] In actual operation, the acquired image features are vectorized and the Euclidean distance between the image feature vector and the center of each standard feature cluster is calculated. When the distance exceeds the set offset threshold, the image feature is judged to deviate from the normal feature cluster, which is considered to be a structural anomaly. The offset threshold can be obtained by multiplying the average radius of the feature vectors in the normal feature cluster by the calibration coefficient, or by setting it jointly by human experience and the automatic optimization results of the model, supporting dynamic adjustment. It adapts to the natural fluctuations of the equipment under different operating conditions, reduces the false alarm rate, and improves the sensitivity to boundary anomalies or early faults.

[0039] When an image feature is detected to be in a blurred boundary region between multiple feature clusters, the situation is marked as "requiring further image acquisition and confirmation," and a voice prompt command is issued to the AR smart glasses. The voice function of the AR smart glasses prompts the on-site operators to adjust the shooting angle, move closer to a specific part, or supplement the recording of the internal structure of the equipment. The above recognition and comparison process continues to be performed on the video content until the abnormal image feature causing the problem is located.

[0040] After determining the cause of the fault, the data processing module initiates the solution formulation function. It integrates relevant content from the method library and the historical maintenance record database to formulate a matching maintenance solution. Specifically, it first extracts the core diagnostic results of the current fault, including the fault type, the names of the involved components, the equipment model number, and the equipment data, as a combination of search conditions. Subsequently, the data processing module retrieves the preset maintenance solutions from the method library, performs multi-field matching based on the above conditions, and uses Boolean rules to select the standardized maintenance solution that best meets the current conditions.

[0041] At the same time, we further accessed the historical maintenance record database to find successful maintenance cases that had been performed on the same equipment under similar fault conditions. We conducted correlation analysis on the execution steps, processing time and final results, extracted the operational details and optimization paths that have reference value, and optimized the maintenance plan.

[0042] The selected maintenance plan is defined in the form of a chain of steps, including: the operation objectives, tool requirements, safety tips, and key structural parts to be focused on for each step; these steps are organized into a maintenance step list as the basis for subsequent voice guidance and interface guidance, while marking the current step number and work progress status to support step-by-step execution and interactive prompts on the AR smart glasses.

[0043] The established maintenance plan is packaged in the form of a structured message and sent to the AR smart glasses module worn by the corresponding operators. After receiving the task, the AR smart glasses module displays the required operation content step by step on its interface and guides the operators to perform maintenance operations according to the established process through voice prompts.

[0044] The maintenance work is carried out under the guidance of AR smart glasses, which includes two parts: voice guidance and image interaction prompts. First, after receiving the maintenance plan from the data processing module, the AR smart glasses play the voice instructions for each maintenance operation in sequence. The voice content includes the purpose of the operation, the required tools and precautions, to ensure that the operators can understand the theoretical basis and key points of the current task.

[0045] Simultaneously, the AR smart glasses utilize built-in local image recognition algorithms to perform real-time analysis of the on-site equipment image, identifying key structural features involved in the current step. Specifically, based on the equipment component information corresponding to the current step, the target area is quickly located from the image. This localization process is based on the lightweight object detection model Tiny-YOLO, combined with the visual template of the target component in the current task, to efficiently detect the equipment image and extract multiple candidate regions for subsequent analysis.

[0046] For candidate regions, image feature vectors are further extracted, such as texture direction, edge contours, and structural contour integrity, and matched with a locally built-in standard feature template library. The matching process uses cosine similarity to identify the image region most closely matching the features of the standard component. The cosine similarity calculation formula is as follows: Where A is the current image feature vector, B is the standard template feature vector, and n is the feature dimension; in order to improve the recognition accuracy, the context information of the current step is introduced during the comparison, such as the structural level of the component and the expected operation order, as a priori conditions to participate in the comprehensive judgment, so as to avoid misidentifying similar but unrelated structural components.

[0047] Once the key structural features have been accurately identified, the target area will be visually annotated in the AR smart glasses interface using an overlay method. This annotation can be in the form of point markers, block markers, etc., to guide operators to accurately locate the operation position, realize the automatic identification and confirmation of key features, and assist maintenance personnel in performing precise operations in an intuitive way, thereby improving the efficiency and safety of fault handling.

[0048] The interface displays and voice prompts are updated in sync, ensuring that the theoretical explanations and operational instructions for each step are coordinated, improving the clarity and accuracy of the maintenance process, and reducing the risk of misoperation.

[0049] If maintenance personnel encounter operational difficulties or unclear situations during maintenance tasks, they can initiate a manual assistance request through the AR smart glasses' interface and voice function. Upon receiving the request, the data processing module automatically synchronizes the current equipment number, task number, and real-time video to the backend support, allowing remote technicians with professional backgrounds to access the system. Remote personnel can view the on-site situation in real time through the video feed and provide operational suggestions; guidance information can also be transmitted to the AR smart glasses in real time via voice.

[0050] After the operator completes the current maintenance task, they can confirm it via voice using the AR smart glasses. Upon receiving the voice confirmation signal, the current task status is recorded as "completed," and the AR smart glasses disable the video synchronization function, thus terminating the video synchronization function.

[0051] Step 5: After receiving the operator's voice confirmation command and completing the video synchronization shutdown operation, the data processing module writes the record of this maintenance process as a new entry into the historical maintenance record database. This record includes: equipment identification, fault type, the sequence of maintenance steps performed, key structural images identified in the AR smart glasses, voice prompts, maintenance duration, and other information. Based on this, the solution library is periodically modeled and optimized using deep learning. Specific steps include: the data processing module extracts multiple maintenance records with similar fault types or equipment categories from the historical database in batches, performs data cleaning and structural standardization, and constructs a training sample set with a unified format. Each sample contains structured fields such as equipment type, fault label, and processing time, and unstructured fields such as image features and voice prompt transcription text. For image data, a convolutional neural network (CNN) is used to extract deep feature vectors from key areas of the image. For voice content and step sequences, a recurrent neural network (RNN) is used to learn the semantic flow and logical relationships between steps, and to discover common effective processing paths.

[0052] The data processing module inputs the cleaned sample data into the scheme induction model. The scheme induction model is a classification optimization mechanism based on rule learning and feature aggregation, used to construct the mapping relationship between fault types and processing procedures. Specifically, it classifies and compares the historical maintenance steps of the same type of fault under different equipment models and environmental parameters, extracting the step sequences with high processing efficiency and low misjudgment rate as "high-quality maintenance paths". Through statistical analysis, path fusion and other means, it refines the maintenance schemes that occur frequently and have effective results in various fault scenarios, and forms condition-triggered processing templates. The induction model evaluates the success rate, execution time, and usage frequency of each type of maintenance path, and prioritizes and labels the maintenance schemes in the existing method library accordingly.

[0053] After completing the inductive analysis, the data processing module rewrites the optimized maintenance solution into the solution library, and adds version information and a description of the applicable scope to it, so as to realize the dynamic updating of the method library. Thereafter, when a scenario similar to the historical fault is identified, the optimal processing path output by the inductive model is called first, thereby improving the overall response speed and maintenance accuracy, realizing the self-evolution from task execution to knowledge accumulation to solution optimization, and enhancing intelligent maintenance capabilities and practicality.

[0054] Example 2: The image recognition system described in this invention is deployed in a smart manufacturing plant. Data processors are installed on key production equipment, each controlling various types of sensors, including temperature sensors, vibration sensors, and current sensors, responsible for real-time acquisition of status parameters during equipment operation. The acquired data is formatted and normalized by a preprocessing module, packaged into structured messages, and uploaded to the data processing module via fiber optic communication.

[0055] During a certain operation, the data processing module received uploaded equipment status data. Through the equipment analysis function, it performed anomaly detection and fault judgment on the data. The analysis revealed that the motor operating current of the equipment was continuously and abnormally rising, exceeding the preset safety threshold, and was accompanied by abnormal motor temperature rise feedback from the temperature sensor. The system comprehensively judged that the equipment might have a motor fault and triggered the fault judgment mechanism. It then called the personnel scheduling function to filter the operator closest to the equipment from the real-time personnel status table in the background and synchronized the task information to the AR smart glasses module.

[0056] Meanwhile, the data processing module further analyzed the data and, combined with equipment parameters and historical cases, preliminarily deduced that the faults might be concentrated in three structural areas: short circuit in the motor's internal windings, blockage of the cooling fan, or jamming of the drive shaft. This information was used to define the fault range and a preliminary analysis report was generated.

[0057] After the operators arrive at the site, they activate the video synchronization function of the AR smart glasses, start video recording, and upload it to the data processing module in real time. Due to insufficient lighting and some dust interference, the brightness detection function detects that the light intensity is below the threshold, triggering the dual-camera switching mechanism and activating the infrared camera to enhance the image brightness. At the same time, the local image recognition algorithm in the AR smart glasses analyzes the grayscale of the image in real time, detects that the image clarity is affected by dust on the lens surface, and then automatically activates the automatic cleaning component, driving the flexible dust removal strip to clean the lens.

[0058] Next, the data processing module calls the central image recognition algorithm to analyze the external structure of the motor frame by frame, and identifies obvious dust accumulation and rotation obstruction on the surface of the cooling fan; at the same time, it prompts the operator to open the motor cover and take supplementary pictures of the inside through AR.

[0059] Subsequently, the data processing module continued to execute the image analysis process. Combining the equipment images and equipment data in the video, it further compared and identified the images with standard feature images under normal operating conditions. It found that there were no obvious burn marks in the motor winding area, but the position of the drive shaft was slightly offset, and oil stains accumulated in the bearing area. Finally, it was determined that the problem was caused by the drive shaft jamming.

[0060] Subsequently, the data processing module invokes the solution formulation function to query the standard maintenance path in the method library that matches the current equipment type and fault type. At the same time, it optimizes the steps by combining historical maintenance records. After identifying the cause of the fault as drive shaft jamming, the data processing module activates the solution formulation function and calls the maintenance method template corresponding to the equipment model and fault type. The template contains standard operating steps: power off protection → remove the casing → clean the bearing area → lubricate or replace the bearing → correct shaft misalignment → reset and check the operating status.

[0061] The data processing module then automatically matches the maintenance operation data of this model of equipment in the historical maintenance record database under similar "drive shaft jamming" problems, compares and analyzes parameters such as step sequence, tool use, and maintenance time, and identifies the path with higher operation efficiency and better success rate in previous successful cases. Based on the results, the data processing module adjusts and optimizes the original method template, merging the cleaning and lubrication steps and optimizing the shaft center alignment guidance action.

[0062] Then, the data processing module generates an adaptive maintenance path for the current fault situation and sends it to the AR smart glasses module through structured task instructions to guide the operators to execute the maintenance process.

[0063] After receiving the maintenance plan, the AR smart glasses module starts the local image recognition algorithm to analyze the current video screen, identify the key operation parts involved in the current maintenance task, and mark these key areas in the AR smart glasses interface in the form of layer overlay.

[0064] At the same time, the built-in voice prompt function of the AR smart glasses is activated, broadcasting the actions to be performed in the order of steps, such as: "Please loosen the two hex bolts on the right drive shaft" and "Check if there are any foreign objects blocking the bearing", and automatically proceeding to the next step based on the recognized operation completion status.

[0065] In some steps, if the local image recognition algorithm determines that a key part is not aligned, it will guide the operator to adjust the shooting angle or move closer to the operation area through voice prompts to ensure that the operation instructions accurately cover the target part and guide the operator to complete the maintenance process in sequence.

[0066] After maintenance is completed, the operator confirms the completion of the task via voice and stops video synchronization. The data processing module archives the data collected during the maintenance process, such as image features, maintenance steps, maintenance duration, and voice commands, and writes the maintenance plan into the historical maintenance record database. This triggers the solution library learning module to summarize and optimize existing solutions, thereby improving the response and matching capabilities for similar faults.

[0067] The foregoing has shown and described the basic principles, features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An image recognition system based on AR smart glasses, characterized in that: A data acquisition module is deployed, and sensors are installed on industrial equipment. The sensor data is transmitted to a data processor, which preprocesses the data, encapsulates it into a structured message format, and transmits it to the data processing module. The data processing module receives the structured messages transmitted by the data acquisition module, parses them into equipment data and equipment location information, uses anomaly feature identification methods to determine if the equipment is faulty, receives real-time video footage from the site, calls a central image recognition algorithm to extract key features from the video footage, uses a clustering offset algorithm for analysis to determine the specific fault, and formulates a maintenance plan based on the solution library and historical maintenance record database. After maintenance is completed, the maintenance information is stored, and deep learning is performed on the maintenance information using a convolutional neural network (CNN) combined with a recurrent neural network (RNN). The deep learning results are then input into a scheme induction model to update the scheme library. The instruction set is then sent to the AR smart glasses module. The AR smart glasses module receives and parses the instruction set from the data processing module, records and synchronizes the on-site video to the data processing module via a wireless network, and parses the maintenance scheme from the instruction set to assist the operators in performing maintenance work.

2. The image recognition system based on AR smart glasses according to claim 1, characterized in that: The data acquisition module specifically includes: several data processors, each corresponding to an industrial device; several sensors under each data processor, the type of which is determined according to the type of industrial device; a preprocessing unit built into the data processor, the specific steps of which are: unit conversion and format regularization of the raw data, addition of corresponding timestamps and device identifiers, noise reduction processing, and encapsulation of all preprocessed device data into a structured message format; the data acquisition module uploads the structured message to the data processing module through an industrial optical fiber communication network.

3. The image recognition system based on AR smart glasses according to claim 1, characterized in that: The data processing module specifically includes: equipment analysis function, personnel scheduling function, central image recognition algorithm, solution formulation function, and solution library self-updating and learning function. The equipment analysis function specifically parses structured messages into equipment data and specific location information, compares the equipment data with normal data ranges to determine if the equipment has malfunctioned, and locates the faulty equipment based on the specific location information. The personnel scheduling function specifically dispatches the personnel with the shortest straight-line distance to the faulty equipment for maintenance. The central image recognition algorithm specifically performs image content analysis, feature extraction, and target recognition on the acquired video footage to pinpoint the problem. The solution formulation function specifically retrieves matching maintenance steps from a preset method library, formulates a maintenance solution suitable for the current problem, and sends it to the AR smart glasses module in real time. The solution library self-updating and learning function specifically records the processing solution to the historical maintenance record database; it uses a convolutional neural network (CNN) combined with a recurrent neural network (RNN) to perform deep learning on the records, dynamically supplementing and optimizing the existing method library.

4. The image recognition system based on AR smart glasses according to claim 1, characterized in that: The AR smart glasses module specifically includes: video synchronization function, brightness detection function, dual-camera switching function, interface display and voice function, local image recognition function, and automatic cleaning component; the video synchronization function specifically includes: video recording and synchronous uploading, continuously capturing on-site images and recording video through the built-in camera component; simultaneously, the video content during the recording process is synchronously uploaded to the data processing module via wireless network; the brightness detection function specifically uses a photosensor installed at the front of the AR smart glasses to obtain the ambient brightness value of the current scene; based on the ambient brightness value, the dual-camera switching function is triggered; the dual-camera switching function specifically includes: the AR smart glasses include a visible light camera and an infrared camera, achieving automatic adaptation of image acquisition under different lighting conditions; the interface display and voice ... The surface display and voice functions specifically involve: overlaying maintenance plans, equipment status, and operation instructions generated by the data processing module onto the operator's field of vision using augmented reality (AR) for visualization; allowing voice recording and calls; the local image recognition function specifically involves: integrating a lightweight image recognition algorithm model, deploying it on the AR smart glasses' local processing chip, and recognizing and processing images within the field of vision based on the AR smart glasses' local computing resources; using the local image recognition algorithm to perform grayscale analysis and control the on / off state of the automatic cleaning component; the automatic cleaning component specifically includes a flexible dust removal strip and drive mechanism located in front of the camera to clean particles or stains adhering to the lens surface; the positioning function specifically refers to: a built-in Wi-Fi module that uses Wi-Fi positioning technology to upload the AR device's location in real time.

5. The image recognition system based on AR smart glasses according to claim 1, characterized in that: The specific steps for determining whether the equipment is faulty include: the data processing module receives the structured message uploaded by the data processor, parses the message, extracts the equipment data and equipment location information, and locates the specific faulty equipment entity; the data processing module compares and analyzes the currently collected equipment data with the normal parameter range through abnormal feature identification to determine the equipment status; and the data processing module infers the equipment module or component area that may be involved in the fault.

6. The image recognition system based on AR smart glasses according to claim 1, characterized in that: The specific steps for receiving real-time video feeds from the site include: real-time monitoring of the status information of all online workers, and obtaining the location data of online workers via Wi-Fi positioning. Specifically, the AR smart glasses module's built-in Wi-Fi module periodically scans the signal strength information (RSSI) of surrounding wireless access points (APs) and uploads the scan results to the data processing module. The data processing module calculates the three-dimensional coordinates of the AR smart glasses' location based on a real-time three-point ranging algorithm. Using the principle of closest spatial distance, the worker closest to the target device is selected as the scheduling target. The three-dimensional Euclidean distance calculation method is used to calculate the spatial distance between each worker and the faulty equipment. The three-dimensional Euclidean distance calculation formula is as follows: Where D is the straight-line distance between the dispatcher and the faulty equipment. The horizontal coordinates of the dispatcher. Let be the vertical coordinates of the dispatcher. The coordinates of the dispatcher in the vertical direction. The horizontal coordinates of the faulty equipment. The coordinates of the faulty device are in the vertical direction. The system determines the height coordinates of the faulty equipment; identifies the dispatcher and issues the task instruction to the AR smart glasses module they are wearing; the operator wearing the AR smart glasses arrives at the faulty equipment site after receiving the task instruction; the operator activates the video synchronization function of the AR smart glasses module; the AR smart glasses module uses its brightness detection function to detect the light intensity of the current working environment in real time; the AR smart glasses module activates the dual-camera switching function based on the brightness detection results; the local image processing algorithm analyzes the grayscale in the image in real time and controls the automatic cleaning components; and the recorded video is uploaded to the data processing module in real time.

7. The image recognition system based on AR smart glasses according to claim 1, characterized in that: The specific steps for determining the specific fault include: the data processing module analyzes the video footage using a central image recognition algorithm; the central image recognition algorithm uses the YOLOv8 general object detection model as its basic network structure and pre-trains it using a large-scale industrial equipment image dataset; representative on-site image samples are collected from the target deployment environment to construct an enhanced training set; the on-site images are input into the model, the detection layer is fine-tuned and trained, and evaluated according to the evaluation metric Precision; key parts of the equipment in the images are located and labeled; a clustering offset algorithm is used to identify whether there are abnormal features, and a K-means clustering algorithm is used to cluster the image features of the key structures of the equipment; the collected image features are vectorized, and the Euclidean distance between the image feature vector and the center of each standard feature cluster is calculated; when the image features are in the fuzzy boundary region between multiple feature clusters, the voice function of the AR smart glasses prompts the on-site operators to adjust the recording range and continue the recognition and comparison process to locate the fault location.

8. The image recognition system based on AR smart glasses according to claim 1, characterized in that: The specific steps for formulating a maintenance plan by combining the solution library and the historical maintenance record database include: the data processing module extracts the core diagnostic results of the current fault, retrieves the preset maintenance plan from the method library, and filters out the standardized maintenance plan through Boolean rules; the data processing module calls the historical maintenance record database to find similar successful maintenance cases, performs correlation analysis on key behaviors, and optimizes the maintenance plan; the data processing module defines the maintenance plan in the form of a step chain, including equipment data and equipment information, packages it in the form of a structured message, and sends it to the corresponding AR smart glasses module.

9. The image recognition system based on AR smart glasses according to claim 1, characterized in that: The specific steps for assisting maintenance personnel include: the AR smart glasses module receives and parses the maintenance plan sent by the data processing module, and assists the professionals according to the maintenance plan, specifically including two parts: voice guidance and image interaction prompts; the voice guidance part specifically involves the AR smart glasses module playing the voice instructions for each maintenance operation step in sequence; the image interaction prompts specifically involve the AR smart glasses module calling the built-in local image recognition algorithm to identify the key structural features involved in the current step; combining the device information, based on the lightweight target detection model Tiny-YOLO, detecting the device screen and proposing candidate regions; further extracting its image feature vector, and using cosine similarity, identifying the image region that is closest to the features of the standard component, the cosine similarity calculation formula is: Where A is the current image feature vector, B is the standard template feature vector, and n is the feature dimension; the AR smart glasses module marks key structural areas on the interface in a layered manner; the operator performs maintenance operations according to the prompts; after maintenance is completed, the operator confirms via voice through the AR smart glasses, and the video synchronization function is turned off.

10. An image recognition system based on AR smart glasses according to claim 1, characterized in that: The specific steps for updating the solution library include: the data processing module writes the maintenance process records as new entries into the historical maintenance record database; it calls the built-in deep learning model to train and summarize the historical maintenance records, extracts maintenance records in batches from the historical database, and generates sample data. Specifically, for image data, a convolutional neural network (CNN) is used to extract deep feature vectors of key regions; for speech content and step sequences, a recurrent neural network (RNN) is used to learn the logical relationship between semantic flow and steps, and to discover common effective processing paths; the obtained sample data is input into the solution summarization model to evaluate the performance indicators of each type of maintenance path, prioritize and label the maintenance solutions in the existing method library, where the solution summarization model is a classification optimization mechanism based on rule learning and feature aggregation, constructing a mapping relationship between fault types and processing procedures; and the optimized maintenance solutions are rewritten into the solution library.

Images