Magnesium alloy electric drive main shell mold wear detection method and system based on machine vision

Abstract

The invention discloses a magnesium alloy electric drive main shell mold wear detection method and system based on machine vision, and belongs to the technical field of mold predictive maintenance. According to the method, three-dimensional point cloud, a surface texture image and an infrared thermogram of a mold are synchronously collected, and point cloud data fused with multi-modal information is generated through multi-sensor joint calibration and high-precision registration; geometric, texture and thermodynamic features are extracted and fused from the features, and a cross-modal feature vector is constructed; intelligent judgment of the wear level of the mold and accurate prediction of the remaining service life are achieved through the trained LightGBM ensemble classification model and the long-short-term memory network time sequence regression model; and finally, associating and visualizing the evaluation result and the three-dimensional model, and automatically generating a structured detection report. According to the method, comprehensive quantification, intelligent evaluation and accurate prediction of mold wear are realized, and the detection efficiency and the scientificity of maintenance decision are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of mold wear detection and predictive maintenance technology, specifically relating to a method and system for detecting wear of magnesium alloy electric drive main housing molds based on machine vision. Background Technology

[0002] With the rapid development of the new energy vehicle industry, the demand for large, complex, and high-precision die-cast parts such as magnesium alloy electric drive main housings is increasing daily. This places extremely high demands on the lifespan of die-casting molds and the stability of product quality. As a core process equipment, the mold cavity surface is subjected to repeated impacts, pressures, and thermal cycles from the high-temperature magnesium alloy molten material, leading to wear, cracking, corrosion, and other degradation. This directly affects the dimensional accuracy and surface quality of the castings, and may even cause unplanned downtime or scrapping of the mold. Therefore, accurate, online, and non-destructive detection and evaluation of mold wear conditions, and prediction of its remaining service life, are of great significance for achieving predictive maintenance, reducing production costs, and ensuring product quality.

[0003] Currently, mold wear detection mainly relies on manual visual inspection, offline contact coordinate measuring machine (CMM) measurement, or visual inspection based on a single two-dimensional image. Manual inspection is highly subjective, inefficient, and difficult to quantify; offline measurement requires disassembling the mold from the production line, which is time-consuming and labor-intensive, and cannot meet the needs of online, high-frequency monitoring. While traditional two-dimensional machine vision methods can achieve non-contact inspection, their information dimension is limited, making it difficult to characterize the depth, volume, and other three-dimensional geometric features of the wear area, and also unable to capture minute temperature changes during the wear process. This results in insufficient sensitivity for detecting early wear, minor indentations, or thermal fatigue damage, limiting the accuracy of wear assessment and life prediction.

[0004] The existing technology has the following main drawbacks:

[0005] The detection methods are limited and the information is incomplete: relying on single visual or geometric information, it is impossible to simultaneously obtain high-precision three-dimensional morphology, surface texture and temperature distribution. It has a weak ability to characterize complex wear modes and is difficult to fully reflect the true wear state of the mold.

[0006] Insufficient intelligence and weak predictive ability: Most existing methods remain at the stage of wear detection and simple quantification, lacking the ability to deeply integrate multimodal data and use machine learning models for intelligent state classification and accurate prediction of remaining life. Poor system adaptability and difficulty in updating: Detection models are usually trained based on specific batch or process data, making it difficult to adapt to different mold materials, changes in process parameters, or new failure modes, and lacking mechanisms for online learning and continuous evolution.

[0007] In view of the shortcomings of the existing technologies, such as the single dimension of detection information, insufficient intelligence in wear assessment and life prediction, and lack of adaptive update capability, this invention aims to provide a mold wear detection method and system that can achieve high-precision fusion of multimodal information, intelligent classification and prediction, and has online learning capability. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide an automated wear detection method and system for magnesium alloy molds, integrating high-precision multimodal data registration, cross-modal feature fusion and extraction, intelligent wear condition assessment, and remaining life prediction. This method can comprehensively and accurately identify the wear condition of molds online and predict their remaining life, effectively solving the problems of limited information dimensions, insufficient intelligence, and poor system adaptability in existing technologies. It provides a scientific and reliable technical means for predictive maintenance of molds.

[0009] To achieve the above objectives, the present invention adopts the following technical solution;

[0010] A machine vision-based method for detecting wear on magnesium alloy electric drive main housing molds, comprising:

[0011] Acquire the 3D point cloud, surface texture image and infrared thermal image of the mold surface. Based on the multi-sensor calibration parameters, register the surface texture image and infrared thermal image to the spatial coordinate system of the 3D point cloud to generate a spatially aligned multimodal dataset.

[0012] Based on a multimodal dataset, geometric, textural, and thermodynamic features representing wear areas are extracted and fused to identify wear areas on the mold surface and generate a cross-modal feature vector that comprehensively represents the current wear state.

[0013] Wear status and life assessment are performed based on cross-modal feature vectors; the cross-modal feature vectors are input into a pre-trained ensemble classification model to output the current wear level or health status of the mold; and the time-series feature sequence containing the current feature vectors is input into a degradation prediction model to output the remaining working life of the mold before it reaches a preset failure threshold.

[0014] The wear level, remaining life prediction results and the identified wear area information are correlated and integrated into a 3D model of the mold for visualization rendering, generating a structured inspection report containing quantitative assessment results and maintenance recommendations.

[0015] As a further aspect of the present invention, the generation of spatially aligned multimodal datasets includes:

[0016] Multi-sensor joint calibration is performed using a stereo calibration board with visible markings in both visible and infrared bands. The bundle adjustment algorithm is used to jointly optimize and solve the intrinsic parameter matrix and distortion coefficient of the industrial camera, the intrinsic parameter matrix and distortion coefficient of the infrared thermal imager, the rigid body transformation matrix from the 3D scanner coordinate system to the camera coordinate system, and optimize the homography matrix between the camera and thermal imager images.

[0017] After the mold completes the die-casting cycle and is opened, it is transferred to the inspection station, where the 3D scanner, industrial camera and infrared thermal imager are triggered to collect data simultaneously.

[0018] For each point in the 3D point cloud, the rigid body transformation matrix is ​​used to transform it to the camera coordinate system, the corresponding pixel coordinates are obtained by projecting it onto the texture image plane through the camera intrinsic parameter matrix, and the color value is obtained from the color texture image by bilinear interpolation.

[0019] The infrared thermal image is transformed to the camera image coordinate system through the homography matrix, and the projection and interpolation of the same point are realized to obtain the temperature value. Finally, a multimodal fused point cloud containing three-dimensional coordinates, three-channel color and one-dimensional temperature is generated for each point.

[0020] As a further aspect of the present invention, before or after the infrared thermal imager acquires temperature data, the thermal imager is rapidly calibrated on-site using a blackbody reference source with a known temperature. The calibration formula is as follows:

[0021] ;

[0022] This is the original temperature value of the thermal imager; and These are calibration coefficients.

[0023] As a further aspect of the present invention, the extraction and fusion of geometric features, texture features, and thermodynamic features characterizing the wear area includes:

[0024] The normal vector and curvature of the downsampled point cloud are estimated based on principal component analysis, where the curvature is calculated by the ratio of the smallest eigenvalue of the covariance matrix to the sum of all eigenvalues.

[0025] The current point cloud is iteratively registered with the nearest point in the CAD digital model of the mold, the normal sign distance from each point to the surface of the CAD model is calculated, and all points below the set threshold are marked as potential wear points.

[0026] Potential wear points are segmented by Euclidean clustering to obtain several independent candidate wear regions. For each region, its three-dimensional projected area, volume, maximum depth, average depth, and surface roughness are calculated.

[0027] For each geometric region, the gray-level co-occurrence matrix statistical features and rotation-invariant unified mode local binary mode histogram are extracted from the corresponding texture image, and the average color difference between it and the intact reference region is calculated in the CIELab color space.

[0028] For each region, the average temperature, maximum temperature, temperature standard deviation, and temperature difference with the overall average temperature of the mold are calculated from the temperature attributes of its point cloud.

[0029] As a further aspect of the present invention, the calculation of the normal symbol distance from each point to the CAD model surface uses the following formula:

[0030] ;

[0031] The first point cloud in the actual measurement The three-dimensional coordinates of each point; Distance on the surface of the CAD model The coordinates of the nearest point; For point The unit normal vector on the surface of the CAD model points outward from the mold. This indicates that the measured point is located on the outer side of the CAD model surface, which is the raised area; This indicates that the measured point is located inside the surface of the CAD model, which is the recessed area; This indicates that the measured point coincides with the surface of the CAD model.

[0032] As a further aspect of the present invention, the cross-modal feature vector includes:

[0033] All geometric, textural, and thermodynamic features of each candidate wear region are concatenated into a high-dimensional region feature vector.

[0034] Calculate global statistical characteristics, including total wear area, total volume loss, maximum depth, highest abnormal temperature, and average color difference;

[0035] The feature vectors of all regions are concatenated with the global statistical features to form the original feature vector, and then dimensionality reduction is performed using principal component analysis to obtain the cross-modal feature vector.

[0036] As a further aspect of the present invention, the ensemble classification model is a multi-class classification model trained based on the LightGBM algorithm, and its loss function adopts multi-class log loss:

[0037] ;

[0038] For the number of categories, One-hot encoding of the actual label; To predict probabilities;

[0039] The degradation prediction model is a regression model based on a long short-term memory network, and its loss function uses mean squared error.

[0040] ;

[0041] To predict remaining lifespan; This represents the actual remaining lifespan. This represents the number of samples.

[0042] As a further aspect of the present invention, the wear status and life assessment also includes an online update mechanism for establishing an integrated classification model and a degradation prediction model: every time a certain amount of new detection data is accumulated, the integrated classification model and / or degradation prediction model are automatically started in the background to perform incremental training or fine-tuning, so that they continue to evolve and adapt to changes in mold materials or process parameters of different batches.

[0043] Another aspect of this application provides a wear detection system for magnesium alloy electric drive main housing molds based on machine vision, the system comprising:

[0044] The multimodal data acquisition and registration module is used to acquire and register 3D point clouds, surface texture images and infrared thermal images to generate spatially aligned multimodal datasets.

[0045] The cross-modal feature extraction and wear recognition module is used to extract and fuse geometric, texture and thermodynamic features from the multimodal dataset, identify wear areas and generate cross-modal feature vectors;

[0046] The wear condition assessment and life prediction module is used to perform classification and regression prediction based on the cross-modal feature vector, and output the wear level and remaining working life;

[0047] The 3D visualization and report generation module is used to associate the evaluation results with the wear area information and perform 3D visualization rendering to automatically generate a structured inspection report.

[0048] A third aspect of this application provides a computer read storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the machine vision-based magnesium alloy electric drive main housing mold wear detection method described above.

[0049] In summary, due to the adoption of the above technical solution, the beneficial technical effects of the invention are as follows:

[0050] This invention effectively overcomes the shortcomings of existing technologies and achieves significant technological progress and beneficial effects by integrating multimodal data such as 3D point cloud, surface texture image and infrared thermal image, and using advanced machine learning and deep learning models for comprehensive analysis.

[0051] Firstly, regarding the comprehensiveness and accuracy of detection, this invention addresses the problem of limited information dimensions in existing technologies. Through high-precision multi-sensor calibration and registration, a seven-dimensional multimodal fusion point cloud containing three-dimensional coordinates, color, and temperature information is generated, achieving strict spatial alignment and unified characterization of geometric deformation, surface texture changes, and thermodynamic anomalies. It can not only accurately identify the geometric features of wear areas, such as area, depth, and volume, but also sensitively capture surface texture degradation and abnormal temperature zones caused by wear, thereby achieving more comprehensive, earlier, and more accurate detection and quantification of complex wear modes, significantly improving detection sensitivity and reliability.

[0052] Secondly, regarding the intelligence and accuracy of assessment and prediction, this invention addresses the shortcomings of existing technologies in terms of insufficient intelligence and weak predictive capabilities. By constructing cross-modal feature vectors and training an ensemble classification model based on LightGBM and a time-series prediction model based on Long Short-Term Memory networks, the system can automatically map multi-source heterogeneous features to specific wear levels and quantified remaining working life. This data-driven intelligent assessment method eliminates the subjectivity and uncertainty of manual judgment, providing reliable data support for developing accurate predictive maintenance plans and effectively avoiding over-maintenance or under-maintenance.

[0053] Finally, regarding the system's adaptability and practicality, this invention solves the problems of poor adaptability and difficulty in updating existing systems. By establishing an online model update mechanism, the system can automatically perform incremental training or fine-tuning using accumulated new detection data, enabling the classification and prediction models to continuously evolve and adapt to the challenges of different batches of mold material characteristics, process parameter fluctuations, and new wear patterns. This self-improvement capability ensures the system's long-term accuracy, stability, and generalization ability, reducing maintenance costs. Simultaneously, combined with 3D visualization rendering and automatic generation of structured reports, abstract detection data is presented to users intuitively and clearly, and maintenance suggestions are automatically associated, greatly improving the understandability of detection results and decision support efficiency, facilitating rapid integration into existing production management and asset maintenance workflows.

[0054] In summary, this invention provides a comprehensive solution integrating high-precision multimodal data acquisition, intelligent feature fusion, accurate condition assessment and life prediction, and adaptive updating. It achieves a leap from perception to cognition and then to prediction of the wear state of magnesium alloy electric drive main housing mold, providing strong technical support for intelligent manufacturing and predictive maintenance in the die casting industry. Attached Figure Description

[0055] Figure 1 This is a flowchart of a machine vision-based method for detecting wear on magnesium alloy electric drive main housing molds. Figure 2 This is a flowchart of S100, a method for detecting wear of magnesium alloy electric drive main housing molds based on machine vision. Figure 3 This is a flowchart of S200, a machine vision-based method for detecting wear on magnesium alloy electric drive main housing molds. Figure 4 S300 is a flowchart of a machine vision-based method for detecting wear of magnesium alloy electric drive main housing mold. Figure 5 S400 is a flowchart of a machine vision-based method for detecting wear on magnesium alloy electric drive main housing molds. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0058] The core objective of this invention is to provide a comprehensive solution capable of online, automated, and high-precision detection and evaluation of the wear state of magnesium alloy electric drive main housing molds, and accurate prediction of their remaining service life. This solution aims to achieve early detection, precise quantification, intelligent grading, and trend prediction of mold wear by integrating multimodal sensing information and machine learning technology. This provides a scientific basis for predictive maintenance decisions in the production process, ultimately extending mold life, improving product quality, reducing maintenance costs, and minimizing unplanned downtime.

[0059] The core idea of ​​this method is:

[0060] First, multi-source sensing and high-precision registration technology are employed. Overcoming the limitations of single-sensor information dimensions, multi-modal data are simultaneously acquired, including 3D geometric topography point clouds, 2D surface texture color images, and 1D temperature distribution infrared thermograms, comprehensively reflecting the wear state. Through precise multi-sensor joint calibration and spatial registration technology, these data from different sources and with different physical meanings are unified into a single 3D spatial coordinate system. This constructs a multi-modal fused point cloud where each data point contains seven attributes: shape, color, and temperature, providing a precise and consistent data foundation for subsequent analysis.

[0061] Second, cross-modal feature fusion and wear characterization are achieved. From the registered multimodal data, geometric features such as indentation depth, volume, roughness, texture and color features such as grayscale statistics, local texture patterns, and color difference, as well as thermodynamic features such as average temperature, temperature difference, and temperature standard deviation are collaboratively extracted and fused. These features together constitute a comprehensive description of the physical, chemical, and thermal state of the wear area. Through feature dimensionality reduction and fusion, a low-dimensional, dense cross-modal feature vector that can comprehensively quantify the overall wear state of the current mold is generated.

[0062] Third, construct intelligent condition assessment and lifespan prediction models based on machine learning. Two core machine learning models are trained using historical data. The condition assessment model inputs cross-modal feature vectors into an ensemble classification model based on algorithms such as LightGBM, automatically outputting the current wear level of the mold, such as normal, slight, moderate, or severe failure, achieving intelligent and standardized judgment of wear status. The lifespan prediction model constructs a sequence of current and historical cross-modal feature vectors according to time series, inputting it into a time-series prediction model based on a long short-term memory network, capturing wear degradation trends, and outputting the remaining working lifespan modes and confidence intervals of the mold before reaching the failure threshold, achieving a leap from condition monitoring to lifespan prediction.

[0063] Fourth, it provides result visualization and decision support. Abstract conclusions such as wear levels and remaining lifespan obtained from intelligent analysis are correlated and integrated with specific wear areas identified in 3D space for unified visualization. Through 3D model coloring and linked information panels, it intuitively displays where wear has occurred, how severe the wear is, and how long it can be used. It also automatically generates structured inspection reports, including quantitative data, trend charts, and specific maintenance recommendations, efficiently transforming technical analysis results into actionable maintenance instructions that directly support production decisions.

[0064] like Figure 1 As shown, this application illustrates an exemplary machine vision-based method for detecting wear on magnesium alloy electric drive main housing molds, specifically including the following steps:

[0065] The specific details of the above technical solution are as follows:

[0066] A machine vision-based method for detecting wear on magnesium alloy electric drive main housing molds includes the following steps:

[0067] S100. Multimodal data acquisition and high-precision registration;

[0068] S200. Cross-modal feature extraction, fusion, and wear region identification;

[0069] S300. Wear condition classification and remaining life prediction based on machine learning;

[0070] S400. 3D visualization rendering and structured report generation.

[0071] Please refer to Figure 2 The flowchart illustrates an exemplary method S100 for detecting wear of magnesium alloy electric drive main housing mold based on machine vision.

[0072] In a machine vision-based method for detecting wear of magnesium alloy electric drive main housing molds, the purpose of S100 is to accurately acquire the three-dimensional morphology, two-dimensional texture, and temperature distribution information of the mold surface through the collaborative work of multiple sensors, and to strictly align the data of different modes in three-dimensional space, so as to provide a data foundation for subsequent multi-feature fusion and accurate analysis.

[0073] The specific steps include:

[0074] S110. During the installation phase of the detection system, calibration is performed using a stereo calibration plate with special markings that is visible in both the visible and infrared bands. The sensor group is fixed, and the calibration plate is moved to multiple different poses while simultaneously acquiring 3D point clouds, texture images, and infrared thermal images. Figure 3 Using the bundle adjustment algorithm, we jointly optimize and solve the intrinsic parameter matrix and distortion coefficient of the industrial camera, the intrinsic parameter matrix and distortion coefficient of the infrared thermal imager, and the rigid body transformation matrix from the 3D scanner coordinate system to the camera coordinate system.

[0075] Meanwhile, the homography matrix between the camera and thermal imager images is optimized. The optimization goal is to minimize the reprojection error of all observation points in the image and three-dimensional space. The final calibration error needs to be controlled to be less than 0.05 mm in the linear direction and less than 0.05 degrees in the angular direction.

[0076] In practical implementation: The objective function of the bundle adjustment algorithm is expressed as:

[0077] ;

[0078] The coordinates of the point in the 3D scanner coordinate system. For industrial camera image coordinates, For the coordinates of the infrared thermal image, and These are the projection functions for the camera and the thermal imager, respectively. Let be the rigid body transformation matrix. It is a homography matrix.

[0079] S120. After the mold completes one cycle and opens in the die-casting machine, it is transferred to the inspection station and fixed by the transfer mechanism; the host computer sends a synchronous trigger pulse, and at the same time starts the grating projection of the 3D scanner, the global shutter exposure of the industrial camera, and the frame capture of the infrared thermal imager; the 3D scanner acquires the three-dimensional coordinates of the mold cavity surface; the industrial camera captures high-resolution color texture images; the infrared thermal imager acquires the temperature distribution image of the mold in the current state.

[0080] S130. For each point in the 3D point cloud, use the rigid body transformation matrix obtained from the calibration to transform it to the camera coordinate system, and then project it onto the texture image plane through the camera intrinsic parameter matrix to obtain the corresponding pixel coordinates; use bilinear interpolation to obtain the corresponding color value in the color texture image based on the corresponding pixel coordinates.

[0081] Similarly, to obtain the temperature value of each point in the 3D point cloud in the infrared image, the infrared image is first transformed to the camera image coordinate system through a homography matrix to achieve spatial alignment with the texture image. Then, the same point is projected and interpolated to obtain its temperature value. Finally, a multimodal fused point cloud is generated, in which each point contains seven attributes: three-dimensional coordinates, three-channel color, and one-dimensional temperature. Subsequently, the multimodal fused point cloud is downsampled by voxel mesh filtering, with the voxel side length set to 0.1 mm, which greatly improves the subsequent processing speed while maintaining accuracy.

[0082] The formula for transforming each point in the 3D point cloud to the camera coordinate system using the rigid body transformation matrix obtained from calibration is expressed as:

[0083] ;

[0084] For point Homogeneous coordinates in the camera coordinate system; In 3D point cloud The homogeneous coordinates of a point in the scanner coordinate system; The rigid body transformation matrix describes the rotation and translation relationship from the scanner coordinate system to the camera coordinate system;

[0085] The formula for projecting the camera intrinsic parameter matrix onto the texture image plane to obtain the corresponding pixel coordinates is expressed as:

[0086] ;

[0087] This is the camera intrinsic parameter matrix; This is the depth scaling factor; For point Pixel coordinates on the image plane;

[0088] In one possible implementation, to ensure the accuracy of the temperature data collected by the infrared thermal imager, the thermal imager is quickly calibrated on-site using a blackbody reference source with a known temperature before or after the data collection.

[0089] The quick calibration formula is:

[0090] ;

[0091] This is the original temperature value of the thermal imager; and These are calibration coefficients;

[0092] Please refer to Figure 3 The flowchart illustrates an exemplary method S200 for detecting wear of magnesium alloy electric drive main housing mold based on machine vision.

[0093] In a machine vision-based method for detecting wear of magnesium alloy electric drive main housing molds, the purpose of S200 is to systematically extract and fuse geometric features, surface texture features, and thermodynamic features that characterize the wear area of ​​the mold from the registered multimodal fusion data, accurately identify and segment each wear area on the mold surface, and finally generate a cross-modal feature vector that can comprehensively and quantitatively characterize the current overall wear state of the mold.

[0094] The specific steps include:

[0095] S210. Geometric Feature Extraction and Preliminary Segmentation:

[0096] First, the normal vector and curvature of the downsampled point cloud are estimated based on principal component analysis. For each point, the covariance matrix is ​​calculated in its neighborhood, and the eigenvector corresponding to the smallest eigenvalue is the normal vector. The curvature is calculated by the ratio of the smallest eigenvalue to the sum of all eigenvalues.

[0097] Wherein, the covariance matrix The calculation formula is:

[0098] ;

[0099] The number of neighboring points; For the neighboring region The coordinates of the point; The average coordinates of the neighboring points; This represents the offset vector of each point relative to the average position of its neighborhood. This represents the transpose of the average position offset vector of each point relative to its neighborhood.

[0100] curvature The calculation formula is:

[0101] ;

[0102] It is the smallest eigenvalue; Covariance matrix The first in One eigenvalue;

[0103] Next, a comparative analysis based on the CAD model is performed, and the current point cloud and the CAD digital model of the mold are iteratively registered with the nearest point. For each registered point, the sign of the normal to the surface of the CAD model is calculated, and the concavity is negative. A threshold is set, and all points below the threshold are marked as potential wear points.

[0104] In one possible implementation, the CAD digital model of the mold is a precise three-dimensional digital model created during the mold design phase, which includes the theoretical geometry and dimensional tolerances of the mold cavity, core, and other structural features.

[0105] Calculate the sign normal distance from the current point cloud to the surface of the CAD model. The formula is:

[0106] ;

[0107] The first point cloud in the actual measurement The three-dimensional coordinates of each point; Distance on the surface of the CAD model The coordinates of the nearest point; For point The unit normal vector on the surface of the CAD model points outward from the mold. This indicates that the measured point is located on the outer side of the CAD model surface, which is the raised area; This indicates that the measured point is located inside the surface of the CAD model, which is the recessed area; This indicates that the measured point coincides with the surface of the CAD model;

[0108] Then, Euclidean clustering is performed on the potential wear points to obtain several independent candidate wear regions; for each region, its three-dimensional projected area, volume, maximum depth, average depth, and surface roughness are calculated; the roughness is obtained by calculating the average absolute deviation of the distance from the point in the region to the local fitting plane.

[0109] S220. Texture and Color Feature Extraction:

[0110] For each geometric region, its region on the texture image is reconstructed using the color attributes of its point cloud and the corresponding image coordinates; the region image is converted into a grayscale image, its grayscale co-occurrence matrix is ​​calculated, the distance parameter is set to 1, and the angle is taken as the average of 0 degrees, 45 degrees, 90 degrees and 135 degrees, and four core statistical features of contrast, correlation, energy and homogeneity are extracted.

[0111] Simultaneously, the rotation-invariant unified mode local binary mode histogram of the image patch is calculated as a texture descriptor;

[0112] In terms of color, the region image is converted to the CIELab color space, the average color of the region is calculated, and the color difference is calculated with the average color of a predefined intact reference region; the color difference is obtained by taking the square root of the sum of the squares of the luminance difference and the chromaticity difference.

[0113] The formula for calculating color difference is:

[0114] ;

[0115] The brightness value for the current area, taken from the CIELab color space; This is the chromaticity value for the current region, taken from the CIELab color space; The chromaticity values ​​of the current region are taken from the CIELab color space; The brightness value for the reference area is usually taken from an intact mold area; The chromaticity value is for the reference area; The chromaticity values ​​of the reference area; The color difference value represents the Euclidean distance between the current area and the reference area in the CIELab color space, and is used to quantify color differences.

[0116] S230. Thermodynamic Feature Extraction:

[0117] For each region, extract the temperature attributes from its point cloud and calculate the average temperature, maximum temperature, and temperature standard deviation; at the same time, calculate the temperature difference between the temperature of this region and the overall average temperature of the mold or the temperature of a specific well-cooled region.

[0118] S240. Feature Fusion and Cross-Modal Vector Generation:

[0119] All geometric, textural, and thermodynamic features of each region are concatenated to form a high-dimensional region feature vector.

[0120] Then, global statistical characteristics are calculated, such as total wear area, total volume loss, maximum depth, highest abnormal temperature, and average color difference.

[0121] The feature vectors of all regions and the global statistical features are concatenated again to obtain a high-dimensional original feature vector. Finally, principal component analysis is used to reduce the dimensionality of the original feature vector to obtain a low-dimensional, information-dense cross-modal feature vector. This vector is a comprehensive quantitative description of the current mold wear state.

[0122] In one possible implementation, the principal component analysis dimensionality reduction process is represented as:

[0123] ;

[0124] The resulting low-dimensional feature vectors retain the most discriminative information from the original data, while significantly reducing the dimensionality. The mean vector of the feature vectors is obtained by calculating the mean value of the original feature vectors of all samples in the training set; Principal component projection matrix; This is the transpose of the projection matrix, used to project the centered original features onto the principal component space;

[0125] Please refer to Figure 4 The flowchart illustrates an exemplary method S300 for detecting wear of magnesium alloy electric drive main housing mold based on machine vision.

[0126] In a machine vision-based method for detecting wear of magnesium alloy electric drive main housing molds, step S300 aims to use machine learning and deep learning models to intelligently analyze and model the cross-modal feature vectors generated in step S200, thereby achieving accurate classification of mold wear status and accurate time-series prediction of remaining service life.

[0127] This is the core stage of moving from data perception to intelligent decision-making, aiming to transform the digital characteristics of wear into quantitative conclusions that can directly guide production and maintenance. Specifically, it includes two core objectives: first, to standardize and objectively assess the current health status of molds through classification models, replacing subjective human judgment; and second, to proactively estimate the number of cycles the mold can safely operate before failure through time-series prediction models based on historical and current wear trends, providing key data support for the development of predictive maintenance plans.

[0128] The specific steps include:

[0129] S310. Construction and Deployment of Wear Level Classification Model:

[0130] We collected a large number of cross-modal feature vectors obtained from historical molds at different stages of their life cycle, and invited experienced mold engineers to label each sample with a wear level label according to industry standards, which were divided into five levels: normal, slight, moderate, severe, and failure.

[0131] An ensemble classification model was trained using the LightGBM algorithm. The dataset was divided into training and test sets in a 7:3 ratio. Five-fold cross-validation and grid search were used to optimize the hyperparameters. After training, the model could output the probability of a given feature vector belonging to each level. During online detection, the level with the highest probability was taken as the wear level of the current mold.

[0132] Loss function of LightGBM algorithm Employing multi-class logarithmic loss:

[0133] ;

[0134] For the number of categories, One-hot encoding of the actual label; To predict probabilities;

[0135] S320. Construction and Deployment of Remaining Useful Life Prediction Model:

[0136] Construct a time-series dataset. For each mold that eventually fails, arrange the cross-modal feature vectors obtained from its previous detections in chronological order to form a feature sequence. The label corresponding to each sequence sample is its remaining useful life, that is, the number of die-casting cycles from the detection to failure.

[0137] A degradation prediction model is constructed using a long short-term memory network. The input layer of the long short-term memory network receives a variable-length feature sequence, which is then processed through several layers of memory units to capture long-term dependencies in the sequence. Finally, a scalar value, i.e., the predicted remaining lifetime value, is output through a fully connected layer. Mean squared error is used as the loss function, and early stopping is employed to prevent overfitting.

[0138] Mean squared error loss function for:

[0139] ;

[0140] To predict remaining lifespan; This represents the actual remaining lifespan. The number of samples;

[0141] During online prediction, the system retrieves all historical feature vectors of the current model and combines them with the feature vectors detected in this test to form an input sequence, which is then fed into a trained long short-term memory network model to obtain the remaining lifetime prediction value, in units of modulus. At the same time, the system uses the Monte Carlo random dropout method to perform multiple forward propagations, calculates the mean and standard deviation of the prediction value, and provides the confidence interval.

[0142] In one possible implementation, the classification and prediction model is updated online. Every time a certain amount of new detection data is accumulated, especially data containing new failure modes, the wear level classification model and / or remaining life prediction model are automatically started in the background for incremental training or fine-tuning, so that they can continuously evolve and adapt to changes in mold materials or process parameters of different batches.

[0143] Please refer to Figure 5 The flowchart illustrates an exemplary method S400 for detecting wear of magnesium alloy electric drive main housing mold based on machine vision.

[0144] In a machine vision-based method for detecting wear of magnesium alloy electric drive main housing molds, S400 aims to comprehensively and intuitively present and output the abstract data results obtained in the aforementioned steps, such as wear detection, grade assessment, and life prediction, in a structured manner, and transform the analysis conclusions into information products that can be directly used for maintenance decisions and engineering practices in a highly visualized form.

[0145] Specific objectives include: using 3D visualization technology to accurately overlay identified wear areas and corresponding wear levels onto the 3D model of the mold, providing an intuitive display of spatial distribution and severity; enabling the correlation query between assessment results and specific wear area details through an interactive interface; and finally, automatically generating a structured inspection report containing all key findings, quantitative data, trend analysis, and specific maintenance recommendations, completing the process from data collection to report delivery, and providing equipment managers with clear, accurate, and actionable decision-making support.

[0146] The specific steps include:

[0147] S410. 3D visualization rendering of the wear area:

[0148] The color point cloud generated in S100 is used as the visualization base; based on each wear area segmented in S210 and its corresponding overall or local wear level in S310, color mapping is applied to map the five wear levels defined in S310 to the corresponding distinguishing colors, and each area is colored; in the 3D display engine, the smooth rotation, scaling, translation, and cross-section viewing functions of the model are realized, allowing users to cut open the mold to view the wear of the internal cavity.

[0149] S420. Evaluation of information overlay and interaction:

[0150] The overall assessment results are displayed prominently in the side information panel of the visualization interface, including mold number, inspection time, current wear level, and predicted remaining life; at the same time, the three to five most severely worn areas are listed in tabular form, including their location description, area, and maximum depth.

[0151] To achieve interactive linkage, when the user clicks on any shaded area on the 3D model, the side panel will simultaneously highlight and display the detailed feature data of that area, including geometric parameters, texture feature values, and temperature values.

[0152] S430. Automatic generation of structured inspection reports:

[0153] After the system completes the detection, it automatically calls the report template engine to generate a structured detection report in a portable document format;

[0154] The report mainly includes a header, executive summary, detailed analysis and data, maintenance recommendations, and appendices. The executive summary directly presents the core conclusions and action recommendations. The detailed analysis section includes a 3D wear distribution rendering, a quantitative statistical summary table of wear areas, and historical trend charts of key characteristic values. The maintenance recommendations provide specific and actionable maintenance guidance based on wear level and remaining life prediction. The report can be automatically sent to a pre-configured email address or uploaded to the enterprise asset management system to complete data archiving and workflow triggering.

[0155] Example 2:

[0156] A machine vision-based wear detection system for magnesium alloy electric drive main housing molds, the system comprising:

[0157] The multimodal data acquisition and registration module is used to acquire the three-dimensional point cloud, surface texture image and infrared thermal image of the mold surface, and based on the multi-sensor calibration parameters, register the surface texture image and infrared thermal image to the spatial coordinate system of the three-dimensional point cloud to generate a spatially aligned multimodal dataset.

[0158] The cross-modal feature extraction and wear recognition module is used to extract and fuse geometric features, texture features and thermodynamic features that characterize the wear area based on the multimodal dataset, so as to identify the wear area on the mold surface and generate a cross-modal feature vector that comprehensively characterizes the current wear state.

[0159] The wear condition assessment and life prediction module is used to assess wear condition and life based on the cross-modal feature vector: including inputting the cross-modal feature vector into a pre-trained ensemble classification model and outputting the current wear level or health status of the mold; and inputting the time-series feature sequence containing the current feature vector into a degradation prediction model based on time-series analysis and outputting the remaining working life of the mold before it reaches a preset failure threshold.

[0160] The 3D visualization and report generation module is used to associate wear level, remaining life prediction results and identified wear area information, perform integrated visualization rendering on the mold 3D model, and generate a structured inspection report containing quantitative assessment results and maintenance recommendations.

[0161] The multimodal data acquisition and registration module includes:

[0162] The sensor group control unit is used to coordinate and control the synchronous triggering and data acquisition of the 3D scanner, industrial camera and infrared thermal imager;

[0163] The calibration and registration unit is used to perform joint calibration of multiple sensors, calculate the rigid body transformation matrix, camera and thermal imager intrinsic parameter matrices and homography matrix, and realize the spatial alignment and fusion of multimodal data.

[0164] The cross-modal feature extraction and wear identification module includes:

[0165] The geometric analysis unit is used to estimate the normal vector and curvature of the point cloud, perform comparative analysis and iterative nearest point registration based on the CAD model, segment potential wear areas and extract geometric features;

[0166] The texture and color analysis unit is used to extract gray-level co-occurrence matrix features, local binary mode histograms, and CIELab color space color difference features from texture images.

[0167] Thermodynamic analysis unit is used to extract regional temperature statistical features and temperature difference features from infrared thermal images;

[0168] The feature fusion unit is used to concatenate geometric, textural, and thermodynamic features and reduce their dimensionality to generate cross-modal feature vectors.

[0169] The wear condition assessment and life prediction module includes:

[0170] The wear classification unit deploys a multi-classification model based on LightGBM and outputs the wear level based on cross-modal feature vectors.

[0171] The lifespan prediction unit deploys a time-series regression model based on a long short-term memory network to predict the remaining working life based on the feature sequence.

[0172] The model update unit supports online incremental learning and model fine-tuning, adapting to changes in mold materials and processes in different batches.

[0173] The 3D visualization and report generation module includes:

[0174] A visualization engine for rendering worn areas on 3D models, supporting rotation, scaling, translation, and cross-sectional viewing;

[0175] The interactive interface displays assessment results and regional details, and supports click-based interaction and data highlighting.

[0176] The report generator automatically generates structured inspection reports based on templates, and supports email sending and system archiving.

[0177] The system also includes a data management and communication module, which manages historical testing data, model parameters and system configuration, and supports data exchange and command issuance with the die-casting machine control system and the enterprise asset management system.

[0178] Example 3:

[0179] In a practical application verification of a wear detection method for magnesium alloy electric drive main housing mold based on machine vision in a die-casting enterprise, the following system configuration and test scheme were adopted:

[0180] I. Test Environment and Equipment Configuration:

[0181] Mold type: magnesium alloy electric drive main housing die casting mold, cavity size approximately 400mm×300mm×150mm;

[0182] Die casting equipment: 850T cold chamber die casting machine, with a production cycle of 80-100 molds per hour;

[0183] Detection system configuration:

[0184] 3D scanner: laser line scanning type, accuracy ±0.02mm, resolution 0.05mm;

[0185] Industrial camera: 5-megapixel global shutter CMOS camera, 30fps;

[0186] Infrared thermal imager: resolution 640×480, temperature measurement range 0-500℃, accuracy ±1℃;

[0187] Computing platform: Industrial edge computing server (Intel i7 processor, 32GB memory, NVIDIA RTX 3060 graphics card).

[0188] Software system: Developed based on Python and Open3D, integrating OpenCV, Scikit-learn, and TensorFlow libraries.

[0189] II. Testing Process and Methods:

[0190] (1) System installation and calibration;

[0191] A detection system was installed next to the mold opening station of the die-casting machine, and a double-sided visible-infrared calibration plate was used for multi-sensor joint calibration. After calibration, the system reprojection error was controlled within 0.03mm, meeting the detection accuracy requirements.

[0192] (2) Data acquisition and model training;

[0193] The complete lifecycle data of the mold from new mold to failure was collected, with a total of 32 test data collections, including samples of five levels: normal, slight wear, moderate wear, severe wear, and failure.

[0194] Training set: the first 25 detection data;

[0195] Test set: Online detection and validation of the last 7 test data;

[0196] (3) Online detection and verification;

[0197] During the production process, an online inspection is automatically performed after every 50 die-casting cycles, and the monitoring continues until the mold reaches the preset failure threshold.

[0198] III. Test Results and Performance Analysis:

[0199] (1) Accuracy of wear level classification;

[0200] Evaluate the performance of the classification model using five-fold cross-validation:

[0201] Overall accuracy: 96.3%;

[0202] F1-score for each category: Normal 0.97, Slight Wear 0.95, Moderate Wear 0.96, Severe Wear 0.94, Failure 0.98;

[0203] The confusion matrix shows that the main misclassifications occur between adjacent wear levels, which is within the acceptable range in engineering practice.

[0204] (2) Remaining lifetime prediction accuracy;

[0205] Evaluate the performance of the prediction model using the mean absolute percentage error (MAPE):

[0206] Early stage (remaining lifetime > 2000 cycles): MAPE = 12.5%;

[0207] Mid-term phase (500-2000 cycles): MAPE = 8.2%;

[0208] Later stage (<500 cycles): MAPE = 5.7%;

[0209] The overall MAPE is 9.8%, which meets the requirements for engineering applications.

[0210] (3) Detection efficiency and stability;

[0211] Time taken per test: 15 seconds for data acquisition + 25 seconds for data processing = 40 seconds per test;

[0212] System availability: After 30 days of continuous operation, the failure-free time rate is 99.2%.

[0213] Temperature effect test: When the ambient temperature changes from 15-35℃, the test results fluctuate by <2%;

[0214] IV. Application Results and Value Demonstration:

[0215] (1) The effect of preventive maintenance;

[0216] Early warning: A moderate wear alarm is issued 800 times before the final mold failure, and a severe wear alarm is issued 300 times before the final mold failure;

[0217] Maintenance decision support: Based on the forecast results, the mold maintenance plan was optimized, and the original scheduled maintenance every 5,000 mold cycles was changed to condition-based predictive maintenance;

[0218] Unplanned downtime reduced: Unplanned downtime related to molds decreased by 67%;

[0219] (2) Improvement in product quality;

[0220] Product defect rate: After implementation, the product dimensional deviation rate caused by mold wear decreased from 1.2% to 0.3%;

[0221] Reduced scrap: Approximately 150 scrapped items per month due to mold issues;

[0222] (3) Economic benefit analysis;

[0223] Extended mold life: Through precise maintenance, the total service life of molds is extended by 28%;

[0224] Reduced maintenance costs: Annual mold maintenance expenses reduced by 35%;

[0225] Overall benefits: Considering factors such as improved production efficiency and quality, the annual overall economic benefit is approximately RMB 420,000 per mold.

[0226] Typical application cases:

[0227] During a production run, the system detected abnormal wear in the lower left corner of the mold. The system determined it to be moderate wear and predicted a remaining lifespan of 420 cycles. Maintenance personnel applied a reinforcing coating to the area as recommended. The mold continued to be used for 460 cycles before reaching severe wear, verifying the accuracy of the prediction.

[0228] Example 4:

[0229] According to Embodiment 1 of this application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can implement all the steps of the wear detection method for magnesium alloy electric drive main housing mold based on machine vision described in Embodiment 1.

[0230] The sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0231] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting wear of magnesium alloy electric drive main housing molds based on machine vision, characterized in that, include: Acquire the 3D point cloud, surface texture image and infrared thermal image of the mold surface. Based on the multi-sensor calibration parameters, register the surface texture image and infrared thermal image to the spatial coordinate system of the 3D point cloud to generate a spatially aligned multimodal dataset. Based on a multimodal dataset, geometric, textural, and thermodynamic features representing wear areas are extracted and fused to identify wear areas on the mold surface and generate a cross-modal feature vector that comprehensively represents the current wear state. Wear condition and life assessment are performed based on cross-modal feature vectors; The cross-modal feature vector is input into a pre-trained ensemble classification model to output the current wear level or health status of the mold, and the temporal feature sequence containing the current feature vector is input into a degradation prediction model to output the remaining working life of the mold before it reaches a preset failure threshold. The wear level, remaining life prediction results, and identified wear area information are correlated and integrated into a 3D model of the mold for visualization rendering, generating a structured inspection report containing quantitative assessment results and maintenance recommendations.

2. The method for detecting wear of magnesium alloy electric drive main housing mold based on machine vision according to claim 1, characterized in that, The generated spatially aligned multimodal dataset includes: Multi-sensor joint calibration is performed using a stereo calibration board with visible markings in both visible and infrared bands. The bundle adjustment algorithm is used to jointly optimize and solve the intrinsic parameter matrix and distortion coefficient of the industrial camera, the intrinsic parameter matrix and distortion coefficient of the infrared thermal imager, the rigid body transformation matrix from the 3D scanner coordinate system to the camera coordinate system, and optimize the homography matrix between the camera and thermal imager images. After the mold completes the die-casting cycle and is opened, it is transferred to the inspection station, where the 3D scanner, industrial camera and infrared thermal imager are triggered to collect data simultaneously. For each point in the 3D point cloud, the rigid body transformation matrix is ​​used to transform it to the camera coordinate system, the corresponding pixel coordinates are obtained by projecting it onto the texture image plane through the camera intrinsic parameter matrix, and the color value is obtained from the color texture image by bilinear interpolation. Infrared thermal images are transformed to the camera image coordinate system using a homography matrix, enabling projection and interpolation of the same point to obtain temperature values. Ultimately, a multimodal fusion point cloud is generated, where each point contains three-dimensional coordinates, three-channel color, and one-dimensional temperature.

3. The method for detecting wear of magnesium alloy electric drive main housing mold based on machine vision according to claim 2, characterized in that, Before or after the infrared thermal imager acquires temperature data, the imager is rapidly calibrated on-site using a blackbody reference source with a known temperature. The calibration formula is as follows: ； This is the original temperature value of the thermal imager; and These are calibration coefficients.

4. The method for detecting wear of magnesium alloy electric drive main housing mold based on machine vision according to claim 1, characterized in that, The extraction and fusion of geometric, textural, and thermodynamic features characterizing the wear region includes: The normal vector and curvature of the downsampled point cloud are estimated based on principal component analysis, where the curvature is calculated by the ratio of the smallest eigenvalue of the covariance matrix to the sum of all eigenvalues. The current point cloud is iteratively registered with the nearest point in the CAD digital model of the mold, the normal sign distance from each point to the surface of the CAD model is calculated, and all points below the set threshold are marked as potential wear points. Potential wear points are segmented by Euclidean clustering to obtain several independent candidate wear regions. For each region, its three-dimensional projected area, volume, maximum depth, average depth, and surface roughness are calculated. For each geometric region, the gray-level co-occurrence matrix statistical features and rotation-invariant unified mode local binary mode histogram are extracted from the corresponding texture image, and the average color difference between it and the intact reference region is calculated in the CIELab color space. For each region, the average temperature, maximum temperature, temperature standard deviation, and temperature difference with the overall average temperature of the mold are calculated from the temperature attributes of its point cloud.

5. The method for detecting wear of magnesium alloy electric drive main housing mold based on machine vision according to claim 4, characterized in that, The formula for calculating the symbolic normal distance from each point to the CAD model surface is as follows: ； The first point cloud in the actual measurement The three-dimensional coordinates of each point; Distance on the surface of the CAD model The coordinates of the nearest point; For point The unit normal vector on the surface of the CAD model points outward from the mold. This indicates that the measured point is located on the outer side of the CAD model surface, which is the raised area; This indicates that the measured point is located inside the surface of the CAD model, which is the recessed area; This indicates that the measured point coincides with the surface of the CAD model.

6. The method for detecting wear of magnesium alloy electric drive main housing mold based on machine vision according to claim 1, characterized in that, The cross-modal feature vector includes: All geometric, textural, and thermodynamic features of each candidate wear region are concatenated into a high-dimensional region feature vector. Calculate global statistical characteristics, including total wear area, total volume loss, maximum depth, highest abnormal temperature, and average color difference; The feature vectors of all regions are concatenated with the global statistical features to form the original feature vector, and then dimensionality reduction is performed using principal component analysis to obtain cross-modal feature vectors.

7. The method for detecting wear of magnesium alloy electric drive main housing mold based on machine vision according to claim 1, characterized in that, The wear condition and life assessment includes: The cross-modal feature vector is input into a pre-trained ensemble classification model to output the current wear level or health status of the mold, and the time-series feature sequence containing the current feature vector is input into a time-series analysis-based degradation prediction model to output the remaining working life of the mold before it reaches a preset failure threshold. The ensemble classification model is a multi-class classification model trained based on the LightGBM algorithm, and its loss function adopts multi-class log loss: ； For the number of categories, One-hot encoding of the actual label; To predict probabilities; The degradation prediction model is a regression model based on a long short-term memory network, and its loss function uses mean squared error. ； To predict remaining lifespan; This represents the actual remaining lifespan. This represents the number of samples.

8. The method for detecting wear of magnesium alloy electric drive main housing mold based on machine vision according to claim 7, characterized in that, The wear condition and life assessment also includes: Establish an online update mechanism for the ensemble classification model and the degradation prediction model, and perform incremental training or fine-tuning on the ensemble classification model and / or the degradation prediction model based on the accumulated new detection data.

9. A wear detection system for magnesium alloy electric drive main housing mold based on machine vision, characterized in that, The system for implementing the method of any one of claims 1 to 8 comprises: The multimodal data acquisition and registration module is used to acquire and register 3D point clouds, surface texture images and infrared thermal images to generate spatially aligned multimodal datasets. The cross-modal feature extraction and wear recognition module is used to extract and fuse geometric, texture and thermodynamic features from the multimodal dataset, identify wear areas and generate cross-modal feature vectors; The wear condition assessment and life prediction module is used to perform classification and regression prediction based on cross-modal feature vectors, and output the wear level and remaining working life. The 3D visualization and report generation module is used to associate the evaluation results with the wear area information and perform 3D visualization rendering to automatically generate a structured inspection report.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images