Mechanical and electrical product running state monitoring system based on computer vision

Abstract

The invention discloses an electromechanical product operation state monitoring system based on computer vision, and relates to the technical field of neural network monitoring, and the system comprises an image collection module which is composed of a distributed high-definition industrial camera and a collection card, and is used for synchronously obtaining a high-definition image of a part; the image preprocessing module outputs a standardized image through filtering, enhancement, segmentation and other algorithms; the feature extraction module fuses static and dynamic features based on CNN and LSTM architecture to generate a high-dimensional vector; the neural network analysis module loads a pre-training model to identify the operation state and the fault type; the running state evaluation module quantifies the health degree and divides the health degree into four stages of states; the intelligent early warning module triggers multi-channel early warning according to grades; the data storage module adopts a distributed architecture for classified storage; the remote monitoring module constructs a visual platform to support remote configuration and interaction. According to the invention, the multi-modal data and the neural network are fused, the fault identification is accurate, and the response is real-time; the method has environment self-adaption and model self-optimization capabilities, and supports preventive maintenance.

Description

Technical Field

[0001] This invention relates to the field of neural network monitoring technology, and in particular to a computer vision-based electromechanical product operation status monitoring system. Background Technology

[0002] As core equipment in industrial production, intelligent manufacturing, and daily life, the stability of electromechanical products directly determines production efficiency, product quality, and operational safety. With the advancement of Industry 4.0, electromechanical products are developing towards higher speeds, greater complexity, and greater intelligence. Traditional methods of monitoring operational status are no longer sufficient to meet the demands for refined and real-time monitoring. Traditional monitoring relies heavily on manual inspections and contact sensors. Manual inspections are inefficient, subjective, and difficult to detect early potential faults. Contact sensors are susceptible to interference from installation space and moving parts, and suffer from complex wiring, high maintenance costs, and easy wear, making it impossible to achieve stable monitoring throughout the entire lifecycle, especially limiting their application in scenarios involving rotating parts and high-temperature environments.

[0003] The rise of computer vision technology has provided a new path for non-contact monitoring, but existing computer vision-based monitoring systems still face many technical bottlenecks. Most systems rely solely on single image features for fault identification, lacking deep integration of spatial and temporal features, making it difficult to comprehensively capture the complex changes in the operating status of electromechanical products. Neural network models often have fixed structures, making it difficult to adapt to dynamic scenarios such as changes in the operating environment and performance degradation of electromechanical products after training. Their generalization ability is insufficient, leading to false positives and false negatives. Some systems do not fully integrate multimodal data, relying solely on image information and ignoring key auxiliary data such as sound, vibration, and temperature, resulting in insufficient comprehensiveness and reliability of fault identification. Furthermore, the threshold settings in existing systems are mostly fixed values, unable to be dynamically adjusted according to the product's operating stage and environmental changes, easily leading to false or false alarms and affecting monitoring effectiveness.

[0004] Existing monitoring systems have significant shortcomings in deployment and practical application. Centralized computing architectures result in high data transmission latency, making it difficult to meet real-time monitoring needs; model updates rely on manual intervention and cannot automatically optimize based on new fault data, exhibiting poor adaptability; fault location accuracy is low, only identifying fault types but not pinpointing the fault location, causing inconvenience for maintenance work; insufficient environmental adaptability leads to decreased image acquisition quality in complex environments such as low light, dust, and high temperatures, directly affecting monitoring accuracy. Furthermore, there is a lack of effective trend prediction and preventative maintenance guidance functions, with alarms mostly triggered after faults occur, failing to anticipate fault risks and hindering preventative maintenance. These technical pain points make existing systems unable to meet the high reliability and safety monitoring requirements of modern electromechanical products. There is an urgent need to develop an intelligent monitoring system integrating deep learning, multimodal data, and edge computing to achieve accurate, real-time, and comprehensive monitoring of the operating status of electromechanical products. Summary of the Invention

[0005] This invention proposes a computer vision-based electromechanical product operation status monitoring system to solve the problems mentioned in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a computer vision-based electromechanical product operation status monitoring system, comprising the following modules: The image acquisition module consists of multiple sets of high-definition industrial cameras, lens modules and image acquisition cards. The cameras are distributed according to the key operating parts of the electromechanical products, and support real-time acquisition under different lighting conditions. The image acquisition card acquires data synchronously at a preset frame rate. The image preprocessing module integrates denoising, enhancement, alignment and segmentation units. It uses a combined filtering algorithm to remove noise, enhances details through image enhancement technology, uses a semantic segmentation algorithm to separate the target from the background, and outputs standardized image data. The feature extraction module is based on a hybrid architecture of convolutional neural networks and recurrent neural networks. The ResNet structure extracts static features, and the LSTM structure analyzes dynamic features. The neural network analysis module includes a pre-trained model library and an online inference unit. The model library stores various fault identification models, and the online inference unit loads matching models. The operational status assessment module constructs a status level classification standard and feature comparison library, and calculates the similarity between real-time and standard feature vectors, combined with fault identification results; The intelligent early warning module is equipped with a tiered early warning mechanism and multi-channel notification units. It triggers corresponding early warnings according to the assessment level and pushes early warning information and fault and risk alerts through multiple methods. The data storage and management module adopts a distributed storage architecture, classifies and stores various types of monitoring data, supports multi-dimensional index queries, and is compatible with multiple standard data formats; The remote monitoring and interaction module is based on an industrial internet-based visualization platform that supports remote parameter configuration, threshold adjustment, and report viewing.

[0007] Furthermore, it also includes a fault feature similarity calculation unit, which constructs a hybrid similarity calculation model based on cosine similarity and weighting factors. The calculation model is as follows: in The feature similarity value, For spatial feature weighting coefficients, The weight coefficients of the time series features and + =1, For real-time feature vectors, For standard feature vectors, For the real-time feature vector, the first Feature values ​​in each dimension For the standard eigenvector, the first Feature values ​​in each dimension The number of feature vector dimensions; this unit also has a dynamic weight adjustment mechanism that automatically optimizes based on the type of electromechanical product and the stage of operation. and Values ​​are increased for products with rotating components as the main feature. Values ​​are increased for products with primarily static structures. Value selection; it also integrates real-time feature vector outlier detection functionality.

[0008] Furthermore, it also includes a dynamic health assessment module, which integrates a health decay model and a fault impact factor analysis unit. The health decay model dynamically adjusts the health assessment benchmark based on the electromechanical product's operating time, environmental conditions, and historical fault records. The fault impact factor analysis unit quantifies the impact of different fault types on product operation and generates a dynamic health score by combining the feature similarity calculation results. This module also supports product lifecycle stage adaptation, setting differentiated health calculation rules for the break-in period, stable operation period, and aging period.

[0009] Furthermore, it also includes a dynamic threshold adjustment module, which optimizes the state assessment threshold in real time based on statistical learning algorithms. By analyzing the distribution of normal state characteristics in historical monitoring data, it calculates the confidence interval of feature similarity and dynamically adjusts the judgment threshold for different operating state levels. This module has an adaptive learning cycle function and presets multiple scenario threshold templates. The system can automatically call the matching template according to the real-time operating mode of electromechanical products and record the threshold adjustment history and the reasons for the adjustment.

[0010] Furthermore, it also includes a multimodal data fusion module, which integrates data interfaces for sound sensors and vibration sensors to simultaneously collect sound signals and vibration data during the operation of electromechanical products. It performs spectral analysis on sound signals to extract audio features and performs time-domain and frequency-domain analysis on vibration data to extract vibration features. A new temperature sensor interface is added to collect temperature data of key components and extract temperature change trend features. An attention mechanism is used to complete multimodal feature fusion, with modes that have prominent fault-related features receiving higher weights. Audio features, vibration features, temperature features, and image features extracted by computer vision are fused together.

[0011] Furthermore, it also includes a neural network model self-optimization module, comprising an online incremental learning unit and a model update mechanism. The online incremental learning unit continuously collects new monitoring data and manually labeled results, establishes a data quality screening mechanism, and initiates the incremental training process when the accumulated effective new data reaches a preset quantity. Based on the transfer learning algorithm, it updates the parameters of the pre-trained model, retains the effective feature recognition capability of the original model, and learns new fault feature patterns. The model update mechanism supports automatic replacement of outdated models or the addition of new model versions. A model version management library is established to record the training data, parameter configurations, and performance of each version.

[0012] Furthermore, it also includes a fault location and visualization module, which marks the fault location on the product's 3D model on the monitoring platform based on image segmentation results and feature extraction of regional information, and uses different colors to distinguish the severity of the fault; it synchronously displays real-time images of the part, historical image comparisons, and fault feature details, and generates a time-series image sequence; it integrates a maintenance guidance knowledge base link, and clicking on the fault location can view relevant maintenance cases, operation steps, and parts replacement suggestions; it supports the functions of zooming in, rotating, and annotating the fault area image.

[0013] Furthermore, it also includes a historical data analysis and trend prediction module, which uses time series analysis algorithms to mine the patterns of changes in operational status in historical monitoring data, and combines long short-term memory networks to build a trend prediction model. Based on recent operational status data and health scores, it conducts multi-factor correlation analysis, identifies key influencing factors, and highlights them in the report. The prediction results include confidence interval outputs, supporting managers in developing preventive maintenance plans. It automatically generates trend analysis reports and supports regular automatic pushes to designated email addresses or management platforms.

[0014] Furthermore, it also includes an environmental adaptive adjustment module, which integrates a light sensor and a temperature sensor to collect and monitor the light intensity and temperature data of the environment in real time. When the light intensity is lower than the preset value, it automatically controls the supplementary lighting equipment to turn on; a new dust sensor monitors the dust concentration in the environment. When the concentration exceeds the standard, it activates the automatic cleaning device for the camera lens; when the temperature exceeds the normal operating range of the equipment, it activates heat dissipation or heat preservation measures; and it adjusts the image acquisition parameters and preprocessing algorithm parameters at the same time.

[0015] Furthermore, it also includes an edge computing deployment module, which deploys core computing tasks on edge computing nodes; edge nodes and the cloud monitoring platform use a lightweight communication protocol to synchronize key data; it supports offline working mode for edge nodes, enabling local monitoring and early warning even when the network is disconnected; it adds collaborative functions between edge nodes, allowing multiple edge nodes to share data and load balance through a local area network; and it supports an offline model update mechanism.

[0016] Compared with existing technologies, the beneficial effects of this invention are: The computer vision-based electromechanical product operation status monitoring system of the present invention focuses on the core pain points in the field of neural network monitoring technology. Through multi-module collaboration and technological innovation, it realizes intelligent, accurate and practical monitoring of electromechanical products, with significant core beneficial effects.

[0017] In terms of fault identification accuracy, the system adopts a hybrid architecture of convolutional neural networks and recurrent neural networks, deeply integrating spatial and temporal features to comprehensively capture the static details and dynamic changes in the operating status of electromechanical products, significantly improving the comprehensiveness and effectiveness of feature extraction. The multimodal data fusion module integrates multi-dimensional data such as images, sound, vibration, and temperature, automatically allocating feature weights through an attention mechanism to enhance the identification capability of fault-related features, solving the problem of insufficient information from a single data source and significantly improving the accuracy and reliability of fault identification. The fault feature similarity calculation unit further optimizes feature matching accuracy through dynamic weight adjustment and outlier detection, reducing the occurrence of misjudgments and missed judgments.

[0018] In terms of system adaptability and flexibility, the neural network model self-optimization module supports online incremental learning and automatic model updates. It continuously optimizes model parameters based on new monitoring data and fault modes, adapting to changes in the operating status of electromechanical products and new fault types without manual intervention, significantly improving the system's generalization and adaptability. The dynamic threshold adjustment module, based on statistical learning algorithms, automatically optimizes judgment thresholds according to historical data and operating scenarios, avoiding false alarms and missed alarms caused by fixed thresholds, and adapting to different product types, operating stages, and environmental conditions. The environmental adaptive adjustment module, through functions such as supplementary lighting, lens cleaning, and parameter compensation, ensures image acquisition quality and monitoring stability in complex environments, expanding the system's application scenarios.

[0019] In terms of monitoring response and practical application, the edge computing deployment module deploys core computing tasks to edge nodes, significantly reducing data transmission latency and enabling real-time monitoring and rapid response. It also supports offline operation and inter-node collaboration, ensuring no data loss and stable system operation. The fault location and visualization module achieves pixel-level fault location, combining 3D model markings and time-series image comparison to intuitively display the fault location, type, and development process. It integrates a maintenance guidance knowledge base, providing precise guidance for maintenance work and significantly improving maintenance efficiency. The historical data analysis and trend prediction module, based on time-series algorithms and neural network models, accurately predicts future operating trends and fault risks, supporting the formulation of preventative maintenance plans, transforming passive alarms into proactive early warnings, and reducing downtime losses.

[0020] In terms of operation and maintenance management and ease of use, the remote monitoring and interaction module builds a visual platform, supporting remote parameter configuration, data viewing, and equipment control, reducing on-site operation and maintenance costs. The data storage and management module adopts a distributed architecture, realizing multi-dimensional data classification storage and rapid indexing, ensuring data traceability and long-term utilization. The intelligent early warning module adopts a hierarchical early warning mechanism and multi-channel notification methods to ensure that relevant personnel receive fault information and risk alerts in a timely manner and take targeted measures. Overall, through technological innovation and functional integration, the system comprehensively solves the problems of low accuracy, poor adaptability, slow response, and insufficient practicality of traditional monitoring systems, providing an efficient and reliable technical solution for the full life cycle monitoring and maintenance of electromechanical products, and has significant industrial application value and promotional significance. Attached Figure Description

[0021] Figure 1 This is a schematic block diagram of a computer vision-based electromechanical product operation status monitoring system proposed in this invention. Figure 2 A bar chart comparing the overall system performance; Figure 3 A bar chart comparing the costs of different maintenance strategies; Figure 4 This is a line graph showing how the system model's performance changes over time. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0024] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0025] Reference Figures 1 to 4 A computer vision-based electromechanical product operation status monitoring system includes the following modules: The image acquisition module consists of multiple sets of high-definition industrial cameras, lens modules, and image acquisition cards. The cameras are distributed according to the key operating parts of the electromechanical products, covering core areas such as rotating parts, transmission mechanisms, and connection nodes. It supports real-time image acquisition under different lighting conditions. The image acquisition card synchronously acquires image data at a preset frame rate and outputs high-definition RGB images and grayscale images. The image preprocessing module integrates image denoising, enhancement, alignment and segmentation units. It uses a combination of Gaussian filtering and median filtering algorithms to remove image noise, enhances image details through histogram equalization and adaptive contrast enhancement, achieves multi-camera image alignment based on feature point matching, uses semantic segmentation algorithms to separate the target region from the background, and outputs standardized image data. The feature extraction module is based on a hybrid architecture of convolutional neural networks and recurrent neural networks. The convolutional neural network uses the ResNet structure to extract spatial features of the image, including static features such as part contours, texture changes, and color deviations. The recurrent neural network uses the LSTM structure to analyze the temporal features of continuous frame images, including dynamic features such as motion trajectory, vibration amplitude, and rotation speed changes. The static and dynamic features are fused to generate a high-dimensional feature vector. The neural network analysis module includes a pre-trained model library and an online inference unit. The pre-trained model library stores fault identification models for different types of electromechanical products and is generated through training with a large number of normal and fault state image samples. The online inference unit loads the matching model and performs inference calculations on the high-dimensional feature vectors output by the feature extraction module to identify the operating state category and potential fault type. The operational status assessment module constructs a status level classification standard and feature comparison library. The feature comparison library stores standard feature vectors of electromechanical products under normal operating conditions. By calculating the similarity between real-time feature vectors and standard feature vectors, and combining the fault type identification results, the operational status health is quantitatively assessed and classified into four levels: normal, slightly abnormal, moderately abnormal, and severely abnormal. The intelligent early warning module is equipped with a tiered early warning mechanism and a multi-channel notification unit. It triggers corresponding early warnings based on the operational status assessment level. Minor anomalies trigger a Level 1 early warning, moderate anomalies trigger a Level 2 early warning, and severe anomalies trigger a Level 3 early warning. The notification unit pushes early warning information through audible and visual alarms, SMS messages, and pop-ups on the industrial control platform, along with fault type and risk warnings. The data storage and management module adopts a distributed storage architecture to classify and store information such as raw image data, preprocessed images, feature vectors, inference results, and operation status evaluation reports. It supports data indexing and querying by dimensions such as timestamp, product number, and fault type. The storage format is compatible with JSON, HDF5, and industrial standard data formats, ensuring long-term data traceability. The remote monitoring and interaction module is built on the Industrial Internet to create a visual monitoring platform that displays the real-time operating status of electromechanical products, images, early warning information, and historical data statistics charts. It supports managers to remotely configure image acquisition parameters, adjust early warning thresholds, and view detailed monitoring reports, and provides manual triggering of image acquisition and status assessment functions.

[0026] This invention also includes a fault feature similarity calculation unit, which constructs a hybrid similarity calculation model based on cosine similarity and weighting factors. The calculation model is as follows: in The feature similarity value takes a range of 0-1. The spatial feature weighting coefficients are set to values ​​between 0.6 and 0.8. The weighting coefficients for time-series features take values ​​between 0.2 and 0.4. + =1, For real-time feature vectors, For standard feature vectors, For the real-time feature vector, the first Feature values ​​in each dimension For the standard eigenvector, the first Feature values ​​in each dimension The feature vector dimension is used to improve the accuracy of operational status assessment by quantifying the matching degree between real-time features and standard features; this unit also has a dynamic weight adjustment mechanism that automatically optimizes based on the type of electromechanical product and the operational stage. and Values ​​are increased for products with rotating components as the main feature. Values ​​are increased for products with primarily static structures. The system also integrates real-time feature vector outlier detection, using the isolated forest algorithm to identify abnormal feature dimensions and automatically reduce the computational weight of abnormal dimensions to avoid abnormal data interfering with similarity evaluation results.

[0027] This invention also includes a dynamic health assessment module, which integrates a health decay model and a fault impact factor analysis unit. The health decay model dynamically adjusts the health assessment benchmark based on the operating time of the electromechanical product, environmental conditions, and historical fault records. The longer the operating time and the more times the product is exposed to harsh environments, the lower the benchmark health threshold. The fault impact factor analysis unit quantifies the impact of different fault types on product operation, assigning higher impact weights to serious faults. For example, the weight of transmission mechanism jamming is higher than that of slight surface wear. Combined with the feature similarity calculation results, a dynamic health score is generated, with a score range of 0-100. A score below 60 is considered an abnormal state, and the lower the score, the higher the fault risk. This module also supports product life cycle stage adaptation, setting differentiated health calculation rules for the break-in period, stable operation period, and aging period. The abnormal judgment criteria are appropriately relaxed during the break-in period, while the threshold is tightened and the assessment frequency is increased during the aging period.

[0028] This invention also includes a dynamic threshold adjustment module, which optimizes the state assessment threshold in real time based on a statistical learning algorithm. By analyzing the distribution of normal state characteristics in historical monitoring data and calculating the confidence interval of feature similarity, it dynamically adjusts the judgment threshold for different operating state levels. When the operating environment of electromechanical products changes or slight performance degradation occurs, it automatically adapts to the threshold range to avoid false and missed warnings. This module has an adaptive learning cycle function, shortening the learning cycle to 1-3 days when data fluctuations are large and extending it to 7-15 days when data is stable, ensuring the timeliness and accuracy of threshold adjustment. At the same time, it presets multiple scenario threshold templates, covering typical scenarios such as high-speed operation, low-speed load, and start-up and shutdown phases. The system can automatically call the matching template according to the real-time operating mode of electromechanical products to further improve threshold adaptability, record the threshold adjustment history and adjustment reasons, and support retrospective analysis and manual fine-tuning.

[0029] This invention also includes a multimodal data fusion module, which integrates data interfaces for sound and vibration sensors to simultaneously collect sound signals and vibration data during the operation of electromechanical products. The module performs spectral analysis on the sound signals to extract audio features, including noise frequency and sound intensity changes, and performs time-domain and frequency-domain analysis on the vibration data to extract vibration features, including vibration amplitude, frequency components, and phase changes. A new temperature sensor interface is added to collect temperature data from key components and extract temperature change trend features. An attention mechanism is used to achieve multimodal feature fusion, automatically assigning weights to different modal features, with higher weights given to modes with prominent fault-related features. Audio features, vibration features, temperature features, and image features extracted by computer vision are fused and input into a neural network analysis module to construct a multi-dimensional feature space, improving the comprehensiveness and reliability of fault identification. Feature normalization processing is used during the fusion process to eliminate dimensional differences between different modal data, ensuring the consistency of the fused features.

[0030] This invention also includes a neural network model self-optimization module, comprising an online incremental learning unit and a model update mechanism. The online incremental learning unit continuously collects new monitoring data and manually labeled results, establishes a data quality screening mechanism to filter out fuzzy and mislabeled data, and initiates an incremental training process when the accumulated effective new data reaches a preset quantity. Based on the transfer learning algorithm, the pre-trained model parameters are updated, retaining the effective feature recognition capability of the original model while learning new fault feature patterns. The model update mechanism supports automatic replacement of outdated models or the addition of new model versions, using model performance evaluation metrics such as accuracy and recall to judge the update effect, retaining only the better-performing model. A new model compression function is added, reducing the model parameter size through pruning and quantization techniques, reducing computational resource consumption, and adapting to edge computing deployment requirements. A model version management library is established to record the training data, parameter configuration, and performance of each version, supporting historical version backtracking and comparative analysis.

[0031] This invention also includes a fault location and visualization module. Based on image segmentation results and feature extraction of regional information, it accurately locates the specific location of the fault, with a positioning accuracy down to the pixel level. The fault location is marked on the product's 3D model on the monitoring platform, and different colors are used to distinguish the severity of the fault. Real-time images of the location, historical image comparisons, and fault feature details are displayed simultaneously, generating a time-series image sequence to intuitively show the fault development process. It integrates a maintenance guidance knowledge base link, allowing users to view relevant maintenance cases, operation steps, and component replacement suggestions by clicking on the fault location. It supports image zooming, rotation, and annotation of the fault area, and managers can add notes and maintenance records. This module can also generate a fault location report, clearly identifying the fault location, type, scope of impact, and recommended processing time, providing maintenance personnel with intuitive maintenance guidance and improving maintenance efficiency.

[0032] This invention also includes a historical data analysis and trend prediction module. It employs time series analysis algorithms to mine patterns in operational status changes within historical monitoring data, and combines this with a long short-term memory network to construct a trend prediction model. Based on recent operational status data and health scores, it predicts the operational status trend of electromechanical products over the next 1-30 days, outputting a health change curve and failure risk probability. It conducts multi-factor correlation analysis to explore the correlation between factors such as ambient temperature, load size, and operating time and failure occurrence, identifying key influencing factors and highlighting them in the report. The prediction results include confidence intervals, clearly defining the probability of failure occurrence under different probabilities, supporting managers in developing differentiated preventative maintenance plans. It automatically generates trend analysis reports containing key information such as health change curves, failure risk probabilities, key influencing factors, and maintenance recommendations. These reports can be exported as PDF or Excel files and can be automatically pushed to a designated email address or management platform on a regular basis.

[0033] This invention also includes an environmental adaptive adjustment module, integrating a light sensor and a temperature sensor to collect and monitor ambient light intensity and temperature data in real time. When the light intensity is lower than a preset value, it automatically controls the supplementary lighting device to turn on, adjusting the supplementary lighting angle and brightness to ensure image acquisition quality and avoid image blurring caused by backlighting or low light. A new dust sensor monitors the ambient dust concentration. When the concentration exceeds the standard, it activates an automatic camera lens cleaning device to remove lens dust through airflow or gentle wiping. When the temperature exceeds the normal operating range of the device, it activates heat dissipation or heat preservation measures to ensure stable operation of the sensor and camera. Simultaneously, it adjusts image acquisition parameters and preprocessing algorithm parameters, such as increasing exposure time and improving image gain under low light conditions, and optimizing noise reduction algorithm parameters under high temperature conditions, to compensate for the impact of environmental factors on monitoring results, ensuring stable operation of the system in complex environments and adapting to different application scenarios such as factory workshops and outdoor sites.

[0034] This invention also includes an edge computing deployment module, which deploys core computing tasks such as image preprocessing, feature extraction, and neural network inference on edge computing nodes. These edge computing nodes are deployed close to the image acquisition module to reduce data transmission latency and improve real-time monitoring response speed, with response time controlled within 100ms. Edge nodes and the cloud monitoring platform use a lightweight communication protocol to synchronize key data, uploading only operational status evaluation results, early warning information, and important image data, reducing network bandwidth consumption. Offline working mode for edge nodes is supported, allowing local monitoring and early warning to be completed even when the network is down, storing local monitoring data, and automatically synchronizing it to the cloud after network recovery to ensure no data loss. A new collaborative function between edge nodes is added, enabling multiple edge nodes to share data and achieve load balancing through a local area network. When a node is overloaded, some computing tasks are automatically allocated to idle nodes. An offline model update mechanism is supported; after a new model is pushed from the cloud, edge nodes complete model updates and deployment during idle periods, without affecting normal monitoring work, ensuring that the edge node's model is always in optimal condition.

[0035] The following two examples further illustrate the specific implementation of this system: Example 1: Application of Industrial Motor Operation Status Monitoring Scenarios This embodiment is applied to the operational status monitoring of high-power industrial motors in factory workshops. These motors are core power equipment on the production line, and key monitoring components include the rotor, bearings, drive belts, and connecting shafts. The system focuses on identifying fault types such as rotor imbalance, bearing wear, loose belts, and shaft misalignment. The system employs a distributed image acquisition + edge computing deployment scheme, integrating multimodal data and neural network analysis to achieve real-time monitoring and accurate early warning of the motor's operational status.

[0036] I. Core Implementation Details Image acquisition module deployment: Six sets of high-definition industrial cameras are distributed across key parts of the motor. The lens focal length is adjusted according to the monitoring distance. Two sets are aimed at the rotor end face to monitor speed and vibration amplitude, two sets are focused on the bearing end cover to monitor abnormal temperature and leakage, one set is aimed at the drive belt to monitor tension and wear, and one set is aimed at the connecting shaft to monitor coaxiality deviation. The image acquisition card is set to a frame rate of 30 frames / second, a resolution of 1920×1080, and simultaneously outputs RGB and grayscale images. It supports automatic gain control in low light to adapt to changes in workshop lighting.

[0037] Image preprocessing and feature extraction: The image preprocessing module first uses Gaussian filtering to remove image noise caused by workshop dust, then median filtering to eliminate motion blur, and subsequently histogram equalization to enhance details such as abnormal bearing temperature areas and belt wear marks. Six sets of camera images are aligned using the SIFT feature point matching algorithm, and the U-Net semantic segmentation algorithm is used to separate motor components from the workshop background. The feature extraction module uses a ResNet50 structure to extract spatial features, including static features such as rotor end face scratches, bearing end cover color changes, and belt cracks; and analyzes the temporal features of 30 consecutive frames of images using an LSTM network, including dynamic features such as rotor vibration trajectory, belt jitter amplitude, and connecting shaft offset, fusing them to generate a 2048-dimensional high-dimensional feature vector.

[0038] Neural Network Analysis and Status Assessment: The neural network analysis module loads a pre-trained fault identification model for industrial motors. This model was generated through training on over 100,000 normal and faulty state image samples, covering 8 typical fault modes. The online inference unit performs inference calculations on the high-dimensional feature vectors to identify the fault type and confidence level. The status assessment module calls standard feature vectors from the feature comparison library to calculate real-time feature similarity. Combined with fault influencing factor analysis, it generates a health score of 0-100. A health score of 85 or above is considered normal, 70-85 is slightly abnormal, 60-70 is moderately abnormal, and below 60 is severely abnormal.

[0039] Multimodal data fusion and dynamic threshold adjustment: The multimodal data fusion module synchronously collects vibration data from the motor vibration sensor, operating noise data from the sound sensor, and bearing temperature data from the temperature sensor via an interface. It extracts features such as vibration amplitude, noise frequency, and temperature change rate, and assigns weights using an attention mechanism: vibration feature weight 0.3, sound feature weight 0.2, temperature feature weight 0.1, and image feature weight 0.4. The fused data is then input into the neural network to improve recognition accuracy. The dynamic threshold adjustment module analyzes 30 days of historical monitoring data, calculates the feature similarity confidence interval, and automatically calls different threshold templates for the motor start-up / shutdown phase and stable operation phase. During the start-up / shutdown phase, the threshold for minor anomalies is lowered by 5 points to avoid false alarms.

[0040] Edge computing deployment and intelligent early warning: Edge computing nodes are deployed next to the motor control cabinet, integrating core computing tasks such as image preprocessing, feature extraction, and neural network inference, with a response time controlled within 80ms. Edge nodes synchronize data with the cloud monitoring platform via the lightweight MQTT protocol, uploading only health scores, fault types, early warning levels, and key image data. The intelligent early warning module triggers warnings based on the assessment level: minor anomalies activate workshop audible and visual alarms; moderate anomalies simultaneously send SMS messages to equipment maintenance personnel; and severe anomalies trigger a pop-up warning on the industrial control platform, marking the fault location and providing an indication of the fault's impact range.

[0041] Fault Location and Remote Monitoring: Based on segmentation results and feature region information, the fault location module accurately pinpoints the fault location to the pixel level. On the motor's 3D model on the monitoring platform, bearing wear locations are marked in red, and loose belt areas are marked in yellow. Real-time images of the fault location are simultaneously displayed for comparison with historical normal images, generating a time-series image sequence to show the fault development process. Clicking on marked locations allows viewing maintenance cases such as bearing replacement and belt adjustment. Management personnel can view the motor's operating status, health curve changes, and warning records in real time through the remote monitoring platform. It supports remote adjustment of image acquisition frame rate and warning thresholds, as well as manual triggering of image acquisition and status assessment.

[0042] Model self-optimization and trend prediction: The neural network model self-optimization module continuously collects new monitoring data, filters out fuzzy and mislabeled data, and starts incremental training when the effective data reaches 10,000 records. It updates model parameters based on transfer learning and adds the ability to identify combined rotor imbalance and bearing wear faults. The historical data analysis and trend prediction module combines health data from the past 90 days and uses an LSTM network to predict the operating trend for the next 15 days. If the predicted health score is below 60, a preventative maintenance warning is triggered 3 days in advance, generating a maintenance recommendation report and pushing it to the management platform.

[0043] Table 1 Comparison of Monitoring Results for Industrial Motors Evaluation indicators Traditional monitoring methods include manual labor and contact sensors. This invention system Fault identification accuracy 75% 98% Monitoring response time 30 seconds 80ms Fault location accuracy Component level Pixel level False alarm rate 18% 2% Relative value of annual maintenance costs 100 35 Table 1 clearly demonstrates the advantages of this invention in industrial motor monitoring scenarios. Traditional monitoring methods rely on manual inspection and contact sensors, with a fault identification accuracy of only 75% and a response time of 30 seconds, failing to detect early faults in a timely manner; fault location can only reach the component level, causing inconvenience for maintenance; the false alarm rate is 18%, interfering with normal production; and frequent wear and replacement of contact sensors leads to high maintenance costs. This invention, through neural network and multimodal data fusion, improves the identification accuracy to 98% and shortens the response time to 80ms, achieving real-time monitoring; pixel-level fault location accurately guides maintenance; the false alarm rate is reduced to 2%, ensuring production stability; non-contact monitoring and edge computing deployment significantly reduce maintenance costs, adapting to the high reliability monitoring needs of industrial motors.

[0044] Example 2: Application of CNC Machine Tool Operation Status Monitoring Scenarios This embodiment is applied to the operational status monitoring of CNC machine tools in a precision machining workshop. It focuses on monitoring key components such as the spindle, guideways, tool turret, and worktable, identifying faults such as abnormal spindle vibration, guideway wear, tool breakage, and worktable positioning deviation to ensure machining accuracy and production efficiency. The system integrates computer vision, multimodal sensing, and edge computing to enhance environmental adaptability and trend prediction capabilities, adapting to the complex environments of cutting machining scenarios.

[0045] I. Core Implementation Details Image Acquisition and Environmental Adaptive Adjustment: Five sets of high-definition industrial cameras are deployed along the circumference of the CNC machine tool. One set is aimed at the spindle end face to monitor rotational speed and vibration; one set is aimed at the guide rail running surface to monitor wear and lubrication status; two sets are focused on the tool turret to monitor tool breakage and clamping status; and one set is aimed at the worktable to monitor positioning deviation. The environmental adaptive adjustment module integrates light sensors, temperature sensors, and dust sensors. When the concentration of cutting dust in the workshop exceeds the standard, the automatic camera lens cleaning device is activated to remove lens dust by airflow. In low-light environments, the supplementary lighting equipment is automatically activated, and the supplementary lighting angle is adjusted to avoid reflection. The supplementary lighting brightness is dynamically adjusted according to the light intensity. The image acquisition card is set to a frame rate of 25 frames per second and a resolution of 2048×1536 to ensure clear acquisition of even minor tool damage.

[0046] Image preprocessing and feature extraction: The image preprocessing module uses a combination of Gaussian filtering and median filtering algorithms to remove image noise caused by cutting dust. Adaptive contrast enhancement is used to highlight details such as guide rail scratches and tool notches. ORB feature point matching algorithm is used to align multi-camera images, and MaskR-CNN semantic segmentation algorithm is used to separate target regions such as the spindle, guide rail, and tool. The feature extraction module uses a ResNet101 structure to extract spatial features, including static features such as tool edge damage, guide rail surface scratches, and spindle end face runout marks. A bidirectional LSTM network is used to analyze the temporal features of 25 consecutive frames, including dynamic features such as spindle vibration amplitude, guide rail motion smoothness, and table positioning deviation, fusing these features to generate a 2048-dimensional high-dimensional feature vector.

[0047] Neural Network Analysis and Health Assessment: The neural network analysis module loads a pre-trained model specifically for CNC machine tools. This model is trained using over 150,000 normal and faulty image samples, covering 10 typical fault categories. The online inference unit infers from the feature vectors, outputting the fault type, confidence level, and degree of impact on machining accuracy. The operational status assessment module combines feature similarity calculation and fault impact factors to generate a health score: above 80 points is normal, 70-80 points is slightly abnormal, 65-70 points is moderately abnormal, and below 65 points is severely abnormal. For progressive faults such as tool wear, the health decay model dynamically adjusts the assessment benchmark based on machining time and cutting load; for every 100 hours increase in machining time, the benchmark threshold is lowered by 2 points.

[0048] Multimodal fusion and edge computing collaboration: The multimodal data fusion module simultaneously collects vibration data from the spindle vibration sensor, spindle temperature data from the temperature sensor, and cutting noise data from the sound sensor. It extracts features such as vibration frequency, temperature change, and noise peak value, and assigns weights using an attention mechanism: image features 0.4, vibration features 0.3, temperature features 0.15, and sound features 0.15. The fused data is then input into the neural network to improve the comprehensiveness of fault identification. Edge computing nodes are deployed within the CNC machine tool control cabinet, supporting collaborative work between two edge nodes. When the spindle monitoring task load is too high, some feature extraction tasks are automatically allocated to idle nodes. In the event of a network outage, an offline working mode is activated to store local monitoring data, which is automatically synchronized to the cloud upon network recovery.

[0049] Fault Location and Remote Maintenance: The fault location module accurately pinpoints faulty areas based on segmentation results and feature region information, such as tool edge breakage locations and guideway wear areas. The fault location is marked on the CNC machine tool's 3D model on the monitoring platform, with different colors distinguishing different fault types. Real-time images of the faulty area are displayed simultaneously, along with historical image comparisons and machining accuracy deviation data. A maintenance guidance knowledge base is integrated, covering tool replacement, guideway lubrication, and other related aspects. The remote monitoring platform allows administrators to remotely view machine tool operating status, health curves, and fault records, configure image acquisition parameters, adjust warning thresholds, and generate monitoring reports for machining periods. The intelligent warning module triggers warnings based on the assessment level: minor anomalies activate workshop audible and visual alarms, moderate anomalies send SMS messages to maintenance personnel, and severe anomalies suspend machine tool operation and push fault reports.

[0050] Model self-optimization and trend prediction: The neural network model self-optimization module continuously collects new monitoring data and machining quality feedback. After filtering out invalid data, incremental training is initiated when 8,000 valid data points are accumulated, updating model parameters to adapt to the fault characteristics corresponding to new tool types and machining materials. The historical data analysis and trend prediction module, based on monitoring data from the past 60 days, predicts the CNC machine tool's operating trend for the next 20 days, focusing on predicting tool wear rate and guideway accuracy degradation. It outputs a health status change curve and fault risk probability, triggering preventative maintenance warnings 5 ​​days in advance and generating maintenance plans such as tool replacement and guideway maintenance.

[0051] Table 2 Comparison of Monitoring Results for CNC Machine Tools Evaluation indicators Traditional monitoring methods: offline detection + single sensor This invention system Tool breakage identification accuracy 70% 99% Fault warning lead time none 3-5 days Machining accuracy guarantee rate 85% 99.5% Environmental adaptability Poor excellent Model update efficiency Artificial retraining Automatic incremental update Table 2 highlights the application value of this invention in CNC machine tool monitoring scenarios. Traditional monitoring methods rely on offline detection and single sensors, achieving a tool breakage identification accuracy of only 70%, failing to provide early warnings of faults, and resulting in a machining accuracy guarantee rate of only 85%. They are also susceptible to environmental changes such as cutting dust and lighting variations, exhibiting poor environmental adaptability. Model updates require manual retraining, leading to low efficiency. This invention, through computer vision and multimodal data fusion, improves the tool breakage identification accuracy to 99%; the trend prediction function provides 3-5 day advance warnings, allowing sufficient time for preventative maintenance; the machining accuracy guarantee rate is increased to 99.5%, reducing scrap generation; the environmental adaptive adjustment module ensures monitoring stability in complex machining environments; and the automatic incremental model update requires no manual intervention, adapting to different machining scenarios and tool types, significantly improving the production efficiency and operational reliability of CNC machine tools.

[0052] Reference Figure 2 This diagram comprehensively showcases the overall performance advantages of the system of this invention. Traditional monitoring methods rely on manual inspection and a single sensor, with a fault identification accuracy of only 75% and a response time as long as 2000ms, failing to capture early faults in a timely manner; fault location can only reach the component level, making maintenance difficult; the false alarm rate is 18%, frequently disrupting normal production; contact sensors require regular replacement, resulting in high maintenance costs. This invention, through computer vision and multimodal data fusion, improves the fault identification accuracy to 98%, shortens the response time to 80ms, and achieves real-time monitoring; pixel-level fault location accurately pinpoints the problem area; the false alarm rate is reduced to 2%, ensuring production continuity; non-contact monitoring significantly reduces maintenance costs, significantly outperforming traditional methods.

[0053] Reference Figure 3 This figure highlights the advantages of the system of this invention in terms of maintenance cost control. Under the traditional reactive maintenance model, failure losses amount to 500,000 yuan, maintenance labor costs to 300,000 yuan, and equipment downtime to 200 hours. Sudden failures can lead to production interruptions and significant losses. The system of this invention, through preventative maintenance, reduces failure losses to 100,000 yuan, maintenance labor costs to 150,000 yuan, and equipment downtime to 50 hours, significantly reducing direct losses from failures. While ideal predictive maintenance is even cheaper, it relies on high-precision prediction technology, making it difficult to implement. The system of this invention, through trend prediction and health assessment, approaches the effect of ideal predictive maintenance, achieving an optimal balance between cost control and production continuity, significantly outperforming traditional maintenance models.

[0054] Reference Figure 4This figure illustrates the self-optimization capability of the system model of this invention. Traditional models experience a continuous decline in accuracy from 85% to 65% over time, failing to adapt to equipment aging and changes in fault modes, leading to a gradual performance degradation. The system of this invention, through its neural network model self-optimization module, continuously collects new data and performs incremental training, gradually increasing the model accuracy from 90% to 95%, maintaining an upward trend even after 12 months of operation. The orange line trend indicates that the system model can continuously learn new fault characteristics, adapt to changes in equipment operating conditions, and consistently maintain high-precision identification capabilities, significantly outperforming the performance degradation trend of traditional models and providing technical support for long-term stable monitoring.

[0055] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A computer vision-based electromechanical product operating condition monitoring system, characterized by, The system comprises the following modules: An image acquisition module composed of multiple sets of high-definition industrial cameras, lens modules, and image acquisition cards. The cameras are distributed according to the key operating parts of electromechanical products and support real-time acquisition under different lighting conditions. The image acquisition cards synchronously acquire data at a preset frame rate. An image preprocessing module integrating denoising, enhancement, alignment, and segmentation units. It uses a combination filtering algorithm to remove noise, enhances details through image enhancement techniques, and separates targets from backgrounds using semantic segmentation algorithms to output standardized image data. A feature extraction module based on a hybrid architecture of convolutional neural networks and recurrent neural networks. The ResNet structure extracts static features, and the LSTM structure analyzes dynamic features. A neural network analysis module containing a pre-trained model library and an online inference unit. The model library stores various fault identification models, and the online inference unit loads matching models. An operating state evaluation module that establishes state level division standards and a feature comparison library. It calculates the similarity between real-time and standard feature vectors and combines fault identification results. An intelligent early warning module that sets up a hierarchical warning mechanism and a multi-channel notification unit. It triggers corresponding warnings according to evaluation levels and pushes warning information and fault and risk prompts through various means. A data storage and management module that uses a distributed storage architecture to store various monitoring data and supports multi-dimensional index queries, compatible with multiple standard data formats. A remote monitoring and interaction module based on the industrial internet to build a visual platform that supports remote parameter configuration, threshold adjustment, and report viewing.

2. A computer vision based electromechanical product operating condition monitoring system as claimed in claim 1, wherein, Also include fault feature similarity calculation unit, based on cosine similarity and weight factor to build hybrid similarity calculation model, calculation model is: Wherein is the feature similarity value, is the spatial feature weight coefficient, is the time sequence feature weight coefficient and =1, is the real-time feature vector, is the standard feature vector, is the feature value of the first dimension of the real-time feature vector, is the feature value of the first dimension of the standard feature vector, is the feature vector dimension; the unit also has dynamic weight adjustment mechanism, according to the type and operation stage of electromechanical product automatically optimize and value, the product of the rotating part is increased value, the product of the static structure is increased value; at the same time, integrate real-time feature vector outlier detection function.​ 3. The computer vision based electromechanical product operating condition monitoring system according to claim 1, wherein, A state health degree dynamic evaluation module that integrates a health degree decay model and a fault influence factor analysis unit. The health degree decay model dynamically adjusts health degree evaluation benchmarks based on product operation time, environmental conditions, and historical fault records. The fault influence factor analysis unit quantifies the impact of different fault types on product operation and generates dynamic health degree scores based on feature similarity calculation results. The module also supports product life cycle stage adaptation, setting differentiated health degree calculation rules for the running-in period, stable operation period, and aging period.

4. The computer vision-based electromechanical product operating condition monitoring system according to claim 1, wherein, A dynamic threshold adjustment module that optimizes state evaluation thresholds in real time based on statistical learning algorithms. It calculates the confidence interval of feature similarity by analyzing the normal state feature distribution in historical monitoring data and dynamically adjusts the judgment thresholds of different operating state levels. This module has an adaptive learning period function and pre-set multiple scenario threshold templates. The system can automatically call matching templates based on the real-time operation mode of electromechanical products and record threshold adjustment history and reasons.

5. The computer vision based electromechanical product operating condition monitoring system according to claim 1, wherein, A multi-modal data fusion module that integrates sound and vibration sensor data interfaces to synchronously acquire sound signals and vibration data during product operation. It extracts audio features through spectral analysis of sound signals and vibration features through time and frequency domain analysis of vibration data. A temperature sensor interface is added to collect key component temperature data and extract temperature trend features. An attention mechanism is used to complete multi-modal feature fusion, giving higher weights to modalities with prominent fault-related features. Audio, vibration, and temperature features are fused with image features extracted by computer vision.

6. The computer vision based electromechanical product operating condition monitoring system according to claim 1, wherein, Also includes a neural network model self-optimization module, including an online incremental learning unit and a model updating mechanism, the online incremental learning unit continuously collects new monitoring data and artificial annotation results, establishes a data quality screening mechanism, when the effective new data accumulation reaches the preset number, starts the incremental training process, updates the pre-trained model parameters based on the transfer learning algorithm, retains the effective feature recognition ability of the original model, and learns new fault feature patterns at the same time; The model updating mechanism supports automatic replacement of outdated models or addition of new model versions; a model version management library is established to record the training data, parameter configuration and performance of each version.

7. The computer vision based electromechanical product operating condition monitoring system of claim 1, wherein, Also includes a fault location and visualization module, based on image segmentation results and feature extraction area information, marks the fault location on the product three-dimensional model of the monitoring platform, and uses different colors to distinguish the severity of the fault; synchronously displays the real-time image, historical image comparison and fault feature details of the part, generates a time sequence image sequence; integrates a maintenance guidance knowledge base link, clicking on the fault location can view related maintenance cases, operation steps and component replacement suggestions; supports fault area image zooming, rotating, labeling functions.

8. The computer vision based electromechanical product operating condition monitoring system of claim 1, wherein, Also includes a historical data analysis and trend prediction module, which uses time series analysis algorithm to mine the running state change law in historical monitoring data, combines long short-term memory network to build a trend prediction model based on recent running state data and health degree score; carries out multi-factor correlation analysis, identifies key influencing factors and highlights them in the report; the prediction result contains confidence interval output, which supports management personnel to make preventive maintenance plan; automatically generates trend analysis report, supports regular automatic push to designated mailbox or management platform.

9. The computer vision based electromechanical product operating condition monitoring system of claim 1, wherein, Also includes an environment self-adaptive adjustment module, integrates light sensor and temperature sensor, real-time collects light intensity and temperature data of monitoring environment, when the light intensity is lower than the preset value, automatically controls the light supplement equipment to start; adds dust sensor to monitor the environmental dust concentration, when the concentration exceeds the standard, starts the camera lens automatic cleaning device; when the temperature exceeds the normal working range of the device, starts the heat dissipation or heat preservation measures; at the same time, adjusts the image acquisition parameters and preprocessing algorithm parameters.

10. The computer vision based electromechanical product operating condition monitoring system of claim 1, wherein, Also includes an edge computing deployment module, which deploys core computing tasks to edge computing nodes; edge nodes and cloud monitoring platform use lightweight communication protocol to synchronize key data; Supports edge node offline working mode, which can still complete local monitoring and early warning in offline state; Newly added edge node inter-coordination function, multiple edge nodes complete data sharing and load balancing through local area network; supports offline model updating mechanism.

Images