Artificial intelligence monitoring method and inspection equipment for fastener loosening status

Abstract

A method for real-time artificial intelligence monitoring of the loosening status of fasteners uses laser irradiation and an area array camera system to collect the light intensity distribution of the fastener and adjacent surfaces. The light intensity distribution of the measured surface represents the depth information of the fastener and adjacent surfaces. Through artificial intelligence machine learning, a correspondence between speckle fringe patterns and the fastener status is directly established. The loosening status of the fastener is detected by directly identifying the obtained speckle fringe patterns, eliminating the impact of mechanical vibration from railway lines and monitoring vehicles on monitoring accuracy. This method eliminates the use of traditional cameras and the phase extraction technology used in the prior art. According to the present invention, not only can the surface status of fasteners be monitored, but also the status at a certain depth below the surface, and even the state of impending loosening that has not yet occurred can be detected in advance. The detection speed and accuracy can surpass the detection capabilities of skilled workers.

Description

Technical Field

[0001] The present invention relates to an artificial intelligence monitoring method and inspection equipment for the loosening state of fasteners, and specifically to a real-time detection method and device for the connection firmness of fasteners, and more particularly to the application of artificial intelligence machine learning in the field of upgrading real-time, non-contact detection methods for the loosening state of railway rail fasteners. Background Art

[0002] Fasteners are essential components for equipment in aerospace, rail transit, and machinery manufacturing. They effectively connect multiple components to form a new system, making them one of the primary connection methods currently in use. While many fasteners offer the advantages of being removable and replaceable, they can also easily loosen and pose a risk. Therefore, regular fastener loosening inspections are necessary to ensure the proper and safe operation of the entire system.

[0003] Railway fasteners secure rails to sleepers, maintaining track gauge and preventing movement relative to the sleepers. They are a representative category of fasteners, with their standardized design and widespread application. They ensure safe railway operations and reduce track vibration. However, fasteners can become loose due to vibration from passing trains, seriously impacting railway safety. Especially with today's high-speed and heavily loaded railways, rails are subject to greater and more frequent vibrations, necessitating regular inspection of fasteners. Currently, fastener inspections rely primarily on manual labor, but manual inspections have numerous drawbacks: they require significant manpower, are slow, and are subject to subjective judgment and experience, as well as potential risks.

[0004] With the rapid development of railways and high-speed rail in recent years, the development of reliable and practical automated fastener loosening detection technology has become increasingly urgent. Currently, the automatic detection technology for fastener loosening can be divided into sensor-based and 3D vision-based detection methods.

[0005] Traditional sensor-based detection methods use the fact that loose fasteners cause changes in the rail's physical parameters and structural characteristics to identify loose fasteners. However, this method requires the pre-installation of a large number of specialized sensors along the track, making deployment inconvenient and costly.

[0006] 3D vision-based inspection methods utilize a structured light system to capture a 3D point cloud image of the fastener and restore its 3D shape. Looseness can then be determined based on factors such as bolt height or the distance from the center of the spring clip to the rail bottom. However, this method is susceptible to interference from ambient light, and the calculation process for extracting the 3D shape of the fastener and the centerline of the metal spring clip is extremely time-consuming, making it incapable of meeting the requirements for fast, real-time inspection.

[0007] In addition, in actual working conditions, there are situations where the fasteners have become loose but have not yet deformed. Traditional three-dimensional visual inspection methods based on depth information cannot effectively identify such loose conditions.

[0008] In the existing method for detecting damaged or missing fasteners using two-dimensional images, the two-dimensional image is taken from directly above and does not contain depth information of the fastener and its surroundings, making it impossible to complete fastener loosening detection that requires depth information. Summary of the Invention

[0009] The purpose of the present invention is to provide a method and equipment for detecting the loose status of fasteners, which is a contactless, high-precision, fast, and even real-time artificial intelligence detection method. A set of devices is used to monitor countless fasteners along the way while moving, and can also perform early identification of working conditions that have actually begun to loosen but have not yet displaced.

[0010] According to a first aspect of the present invention, there is provided an artificial intelligence real-time monitoring method for fastener loosening, characterized in that:

[0011] Monitor stress or strain in and around fasteners, replacing traditional detection of displacement or deformation;

[0012] Abandoning traditional cameras, the light intensity distribution of fasteners and adjacent surfaces is collected through laser illumination and an area array camera system. The light intensity distribution of the measured surface is used to represent the depth information of the fasteners and adjacent surfaces.

[0013] Abandoning the phase extraction technology in the existing technology, the correspondence between the speckle fringe pattern and the fastener status is directly established through artificial intelligence machine learning; the loose state of the fastener is detected by directly identifying the obtained speckle fringe pattern.

[0014] In the implementation of this method, through machine learning of speckle fringe patterns under different degrees of looseness of fasteners, including machine learning of loosened but not yet deformed states, the artificial intelligence model will not only be able to determine whether the fasteners are loose, but also to judge and identify the degree of looseness of the fasteners and issue a warning or alarm.

[0015] Preferably, a laser is emitted by a laser, which passes through a beam expander to form a laser output beam of larger diameter, irradiating the fastener and the rough surface around it; the reflected light from the fastener and the rough surface around it passes through a shearing device to form a shearing speckle; the shearing speckle is recorded by a CCD and transmitted to a computer for storage and image processing.

[0016] Preferably, the CCD is used to collect an original shear speckle pattern of irregular random distribution before the fastener under test is deformed, and a shear speckle pattern after the fastener under test is deformed; the shear speckle pattern after deformation is subtracted from the original shear speckle pattern to obtain a speckle fringe pattern that records the phase information, i.e., the depth information, of the surface of the fastener under test.

[0017] Preferably, a shearing device is used to cause a misalignment between the reference light and the interference light on the imaging plane, thereby generating a sheared speckle pattern after interference; preferably, a reference mirror of the Michelson interferometer is rotated by an angle so that the two beams of reflected light are misaligned on the imaging plane, thereby forming interference.

[0018] Preferably, the training process of the artificial intelligence machine learning includes:

[0019] Applying a load to the system in which the fastener is installed, while simultaneously collecting an image of the physical parameter changes of the fastener (e.g., a speckle fringe image) and transmitting it to a computer; then, stopping the load application after obtaining the near state until the fastener system returns to the initial state, and stopping image collection; repeating the loading and unloading operations multiple times until sufficient images of the physical parameter changes of the fastener in the normal state are obtained;

[0020] Loosening the fastener to make it loose; repeating the loading, unloading, and image acquisition operations multiple times until sufficient images of changes in physical parameters of the fastener in the loose state are acquired;

[0021] A large number of images are randomly selected from the collected physical parameter change images as the original data set, ensuring that the images of the fasteners in normal state and the images in loose state each account for 50%;

[0022] Using a computer to perform operations such as filtering and normalization on the physical parameter change images; labeling all physical parameter change images as normal or loose; or directly labeling the images of the fasteners that humans believe are in a normal state as normal, and directly labeling the images of the fasteners that humans believe are in a loose state as loose;

[0023] The original data set is used as a training set to be put into the artificial intelligence judgment model for training, and the artificial intelligence judgment model is saved.

[0024] According to a second aspect of the present invention, there is provided an artificial intelligence real-time monitoring device for fastener loosening conditions, characterized in that the device comprises:

[0025] a laser mounted in a position to illuminate the fastener;

[0026] a spatial filter and beam expander disposed in a laser path between the laser and the fastener to filter and expand the laser beam;

[0027] A monitoring image forming device that forms two identical, interfering images with a phase difference;

[0028] A monitoring image receiving device using an area array camera; and

[0029] The computer has an artificial intelligence judgment model for real-time monitoring of the looseness status of fasteners. The monitoring image is filtered and normalized and then input into the trained artificial intelligence judgment model, which detects the tightness status of the fasteners in real time.

[0030] Preferably, the laser is a single longitudinal mode semiconductor laser with a wavelength of 532 nm.

[0031] Preferably, the monitoring image forming device includes two polarizing plates and a Roger prism.

[0032] According to a third aspect of the present invention, there is provided an artificial intelligence real-time monitoring and inspection device for the loosening state of fasteners, characterized in that the inspection device is installed on an operating vehicle and comprises:

[0033] A laser that sequentially illuminates each of the monitored fasteners while the vehicle is in operation;

[0034] A spatial filter and beam expander, which is arranged in the irradiation path of the laser to filter and expand the laser beam;

[0035] A monitoring image forming device that forms two identical, interfering images with a phase difference;

[0036] A monitoring image receiving device using an area array camera; and

[0037] The computer has an artificial intelligence judgment model for real-time monitoring of the looseness status of fasteners. The monitoring image is filtered and normalized and then input into the trained artificial intelligence judgment model, which detects the tightness status of the fasteners in real time.

[0038] According to a fourth aspect of the present invention, there is provided an artificial intelligence inspection vehicle for real-time monitoring of the loosening status of fasteners, wherein the inspection vehicle comprises:

[0039] A laser that sequentially illuminates each of the monitored fasteners while the inspection vehicle is driving;

[0040] A spatial filter and beam expander, which is arranged in the irradiation path of the laser to filter and expand the laser beam;

[0041] A monitoring image forming device that forms two identical, interfering images with a phase difference;

[0042] A monitoring image receiving device using an area array camera; and

[0043] The computer has an artificial intelligence judgment model for real-time monitoring of the looseness status of fasteners. The monitoring image is filtered and normalized and then input into the trained artificial intelligence judgment model, which detects the tightness status of the fasteners in real time.

[0044] According to the present invention, laser technology exhibits a "speckle" phenomenon. This phenomenon occurs when a laser illuminates a diffusely reflective surface. The reflected light from the surface coherently superimposes in space, interfering throughout the space and forming randomly distributed bright and dark spots. In recent years, speckle interferometry has been widely used to measure physical quantities such as vibration, distance, velocity, flow rate, and displacement due to its advantages, including high accuracy, high speed, minimal measurement equipment requirements, and the ability to achieve full-field and non-contact measurement.

[0045] The digital shearing speckle interferometry technique of the present invention is highly sensitive to minute out-of-plane deformation gradients of the measured surface and can be used to measure out-of-plane and in-plane displacement components, strain, slope, curvature, and vibration of diffusely reflective objects. The captured shearing speckle pattern contains depth information about the measured surface. Therefore, image processing-based fastener loosening detection can achieve full-field, high-precision, rapid, and contactless measurement. However, in the prior art, the use of digital shearing speckle interferometry in fastener loosening detection technology, particularly in railway rail fastener loosening detection technology, has never been considered or attempted.

[0046] Machine learning is an advanced image classification method that can extract features from samples to achieve image classification. In various embodiments of the present invention, three machine learning algorithms—decision trees, support vector machines, and convolutional neural networks—were used to verify the reliability and universality of the present invention. Furthermore, each algorithm has its own advantages, and different algorithms can be selected based on specific practical needs. However, in the prior art, no one has used machine learning-based image classification methods to automatically detect the looseness of fasteners.

[0047] This method combines digital shearing speckle interferometry with machine learning to detect loose fasteners. Compared to traditional methods, it offers the advantages of non-contact, high precision, and significantly improved detection speed.

[0048] In particular, the present invention produces unexpected technical effects and can even detect situations in actual working conditions where the fastener has become loose but has not yet undergone actual deformation, which is something that traditional naked eye recognition and computer 3D vision-based methods cannot detect.

[0049] In the prior art, no one has ever used digital shearing speckle interferometry to automatically detect the loose state of fasteners (not only monitoring the surface state of the fasteners, but also monitoring the state at a certain depth below the surface), nor has anyone used a machine learning-based image classification method to automatically detect the loose state of fasteners (not only reaching the level of traditional manual detection, but also surpassing traditional manual detection in speed and quality; on the contrary, traditional phase extraction algorithms restrict the real-time and predictive nature of loose state detection), and no one has simultaneously used digital shearing speckle interferometry and machine learning-based image classification methods to automatically detect the loose state of fasteners, and no prior art can achieve the technical effects of the present invention.

[0050] It is particularly important to point out that the technical solution of the present invention involves not only mechanical engineering (including metal material engineering), but also optical engineering, artificial intelligence technology, etc., which involves multiple technical fields such as mechanics, optics, and electricity. Therefore, before the present invention, there was no person who knew multiple technical fields such as mechanics, optics, and electricity at the same time. The "technical personnel in this field" only knew one of mechanics, optics, and electricity; on the contrary, in the present invention, the new "technical personnel in this field" must be a mechanical engineer, an optical engineer, and an artificial intelligence engineer at the same time; therefore, the inventor of this application is the first "technical personnel in this field" who is familiar with all the above fields at the same time. In the sense of patent law, the hybridization and combination of technologies in different fields to produce unprecedented technical solutions and achieve unexpected technical effects are undoubtedly a manifestation of creativity.

[0051] According to the present invention, the influence of mechanical vibration of railway lines and monitoring vehicles on monitoring accuracy is eliminated. The present invention abandons the use of traditional cameras and the phase extraction technology in the prior art.

[0052] In particular, the present invention overcomes traditional technological prejudices and reverses traditional and backward thinking habits; it transforms from a traditional logic-based mathematical operation scheme to an artificial intelligence sensory judgment scheme based on image comparison; that is, the monitoring device in the present invention directly "checks" whether the fastener has or is about to loosen, instead of relying on "calculations" to make judgments as in the past. This is another manifestation of the creativity of the present invention.

[0053] According to the present invention, the unexpected technical effects are: not only can the surface status of fasteners be monitored, but also the status at different depths below the surface can be monitored; not only can the loosening state that has already occurred be monitored, but also the loosening state that is about to occur can be predicted; one set of equipment can monitor or predict in real time the countless fasteners in four rows on both sides of two rails. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1This is the schematic diagram of the Michelson transverse shear device.

[0055] Figure 2a It is the first stage of the speckle field processing process in the prior art and the present invention, the original speckle pattern captured by the CCD;

[0056] Figure 2b It is the second stage in the speckle field processing process of existing technologies and medium, the interference pattern after the subtraction algorithm;

[0057] Figure 2c It is the third stage in the speckle field processing process in the prior art, the interference pattern after filtering and binarization;

[0058] Figure 2d It is the fourth stage in the speckle field processing process in the prior art, and is the phase distribution diagram obtained from the interference pattern.

[0059] Figure 3 Light path diagram of a fastener loosening detection system based on digital shearing speckle interferometry and machine learning.

[0060] Figure 4 It is a schematic diagram of two wave fronts after transverse shearing.

[0061] Figure 5 The structure of a fastener for a railway rail is shown.

[0062] Figure 6 It is a schematic diagram of the overall test process according to an embodiment of the present invention.

[0063] Figure 7 4 is a flowchart of fastener loosening based on a decision tree according to one embodiment of the present invention.

[0064] Figure 8 2 is a schematic diagram of the support vector machine principle according to an embodiment of the present invention.

[0065] Figure 9 2 is a schematic diagram of an improved VGG-16 network structure according to an embodiment of the present invention.

[0066] Figure 10a FIG. 1 is a layout diagram of a laser system according to a first embodiment of the inspection device of the present invention.

[0067] Figure 10b FIG. 4 is a layout diagram of a laser system according to a second embodiment of the inspection device of the present invention.

[0068] Figure 10c FIG. 4 is a layout diagram of a laser system according to a third embodiment of a patrol inspection device of the present invention.

[0069] Figure 10d FIG. 4 is a layout diagram of a laser system according to a fourth embodiment of a patrol inspection device of the present invention. DETAILED DESCRIPTION

[0070] The preferred embodiment will be described in detail below with reference to the accompanying drawings. It should be emphasized that the following description is merely illustrative and is not intended to limit the scope of the present invention and its application.

[0071] Basic principles of the present invention:

[0072] The present invention transforms depth information (photosensitive phase information) beneath the fastener surface into the light intensity distribution of a speckle fringe pattern through steps such as laser irradiation, "shearing," and "subtraction." This overcomes traditional technical biases and abandons conventional, outdated thinking (i.e., reliance on the mathematical correspondence between light intensity distribution and phase information in a speckle fringe pattern; therefore, phase information must first be derived from the light intensity distribution before depth information beneath the measured surface can be obtained, resulting in slow detection speed and low accuracy).

[0073] On the contrary, in the prior art, the frequency of natural light or flash is too high, exceeding the resolution capability of ordinary cameras. Therefore, ordinary cameras cannot directly acquire the photosensitive phase information about the fasteners.

[0074] According to one embodiment of the present invention, a laser is emitted from a laser, which, after passing through a beam expander, forms a laser output beam of a larger diameter, which is irradiated onto the rough surface of the fastener. The reflected light from the rough surface of the fastener passes through a shearing device to form a shearing speckle. The shearing speckle is recorded by a CCD and transmitted to a computer for storage.

[0075] The speckle fringe pattern obtained through the above-mentioned "shearing" and "subtraction" steps converts the depth information (phase information) under the measured surface into the light intensity distribution change of the speckle fringe pattern.

[0076] Therefore, according to the relationship between different states of fasteners (with different depth information) and different types of speckle fringe patterns (light intensity distribution) established in the machine learning training model, the state of the fasteners can be directly judged by the speckle fringe patterns.

[0077] Interference image formation principle:

[0078] The so-called "shearing" is generated by the shearing device, which causes a certain misalignment between the reference light and the interference light on the imaging surface, thereby generating interference. In one embodiment, Figure 1 As shown in FIG, one of the reference mirrors M1 of the Michelson interferometer is rotated by a certain angle so that the two reflected light beams are slightly misaligned on the imaging plane, thereby forming interference.

[0079] In the present invention, common shearing devices include but are not limited to Michelson interferometer; prisms such as biprism, Rochon prism, and Wollaston prism; gratings such as Ronchi grating and cross-grating; and liquid crystal spatial light modulator, etc.

[0080] Speckle treatment:

[0081] In the present invention, the original shear speckle pattern (irregularly randomly distributed speckles, as shown below) is collected by CCD. Figure 2a ) and stored in the computer. The speckle pattern before the tested fastener is deformed is recorded as F1; after deformation, the speckle pattern of the Nth frame is recorded as F N Subtract F from F1 N , get the speckle fringe pattern F corresponding to the Nth frame N '(Regular black and white stripes, as shown below Figure 2b ), the speckle fringe pattern records the phase information of the surface of the fastener being tested, that is, the depth information; the depth difference between adjacent dark lines (or bright lines) is the wavelength of the emitted laser.

[0082] Data processing:

[0083] In traditional shear speckle measurement, the interference fringe pattern after subtraction is filtered, normalized, binarized, etc. ( Figure 2c ), using the phase extraction technology, the wrapped phase information corresponding to the fringe pattern can be obtained ( Figure 2d ); then, using phase unwrapping technology, the wrapped phase is unfolded to obtain the unfolded phase information corresponding to the actual surface of the fastener under test. The corresponding relationship between the unfolded phase information and the depth information (every 2π phase change corresponds to a depth change of one laser wavelength) is then used to recover the depth change information of the measured surface.

[0084] However, in the use of the model of the present invention, strict mathematical methods are no longer used to restore the above depth change information, and steps such as binarization and subsequent unwrapping are no longer included; the interference fringe pattern after subtraction is directly compared with the memory image in the model after processing, allowing the machine to "see" directly instead of letting the computer "calculate". This is a clear difference between the present invention and the prior art.

[0085] The term "shear" means:

[0086] After the original wavefront is “sheared”, two wavefronts are formed that are identical in waveform but slightly different in spatial position. The sheared speckle pattern ( Figure 4 The dark area in the middle is the result of shearing of two wavefronts. The term "shearing" does not mean selecting or cutting off a part of the whole.

[0087] Model training and application

[0088] For a new system formed by connecting various components together using fasteners, in order to ensure that the fasteners are working properly, the method for automatically detecting the loose state of fasteners according to the present invention includes:

[0089] 1) Data Acquisition Equipment: For example, a single-longitudinal-mode semiconductor laser with a wavelength of 532 nm is selected as the light source. The laser light output is filtered and expanded by a spatial filter before being irradiated onto the surface of the new system to form laser speckle. The laser speckle reflected by the measured surface passes through a shearing device (e.g., polarizer 1, Roger prism, and polarizer 2 in sequence). The sheared speckle pattern is captured by an area array camera and transmitted to a computer.

[0090] 2) Model training: Apply an appropriate load to the fastener system while simultaneously capturing images and transferring them to a computer. Stop applying the load after reaching a critical state, and continue until the fastener system returns to its initial state, at which point image capture ceases. Repeat the above steps until sufficient shear speckle images of the fastener in its normal state are obtained. Loosen the fastener to loosen it. Repeat the above steps until sufficient shear speckle images of the fastener in its loose state are obtained. Randomly select a large number of images from the acquired speckle images as the original dataset, ensuring that images of the fastener in its normal state and images in its loose state each account for 50%. Use a computer to perform operations such as filtering and normalization on the shear speckle images. Label the images as normal and loose states, respectively. Use the original dataset as the training set for training the machine learning algorithm model, and save the model.

[0091] 3) Model Application: Capture speckle fringe patterns from an actual fastener system. After filtering and normalizing the images, they are fed into the trained model, which directly outputs the detection results, completing the fastener status inspection.

[0092] Example

[0093] 1. Design goals and design principles

[0094] Railway fasteners are standardized and have a large and representative application market. Temperature fluctuations are a typical load that fasteners often experience. For example, the maximum temperature of a rail system in summer is around 60°C.

[0095] Spring Clip II fasteners are widely used on my country's ballastless tracks, offering advantages such as high clamping force, a large safety reserve, and minimal residual deformation. For rails, 60kg / m standard rails are currently widely used on mainline railways.

[0096] 2. Fasteners and rails

[0097] Assemble the rails, heating plates, insulation boards and cast iron pads together with fasteners, such as Figure 5 shown.

[0098] 3. Overall test system

[0099] The overall process of fastener loosening detection is as follows Figure 6 As shown in the figure, the image acquisition system first captures shear speckle fringe patterns generated by rail deformation in both tightened and loosened states. The images are then labeled (tightened or loose). After preprocessing, three different machine learning algorithms—decision tree, support vector machine, and convolutional neural network—are used to process the images and output detection results, completing the detection of fastener looseness.

[0100] 3.1 Image acquisition part

[0101] The present invention adopts Figure 3 The digital shearing speckle interferometry system shown is used to collect shearing speckle patterns under different states.

[0102] The laser light emitted by the laser passes through a beam expander and is scattered onto the surface of the object being measured. After being reflected by the surface of the object being measured, it passes through polarizer 1, a Roger prism, and polarizer 2 (an embodiment of a shearing device) in sequence before being captured by a CCD and stored in a computer. After the two sheared beams interfere on the CCD surface, the light intensity I on the CCD surface can be expressed as:

[0103]

[0104] Among them, I0 represents the background light intensity, γ represents the fringe contrast, represents the random phase factor.

[0105] Those skilled in the art know that when the rail surface is deformed, the optical path difference and phase difference will change; the intensity distribution of the speckle pattern will also change. However, there are technical difficulties: due to the random phase distribution factor The existence of the light intensity map does not allow for intuitive observation of alternating light and dark fringes. The speckle pattern directly captured by the CCD is still a speckle pattern with irregular distribution of bright and dark spots. In the present invention, the speckle fringe pattern can be obtained by subtracting the intensity map before and after deformation. Since light intensity cannot be negative, the absolute value of the intensity after subtraction is:

[0106]

[0107] Among them, I1 represents the light intensity distribution before deformation, and I2 represents the light intensity distribution after deformation. Indicates the relative phase difference caused by rail deformation.

[0108] From formula (2), it can be seen that the rail will deform under the influence of different temperature changes (and / or load changes), which will cause different phase differences. Phase difference The change of the light intensity gray value I captured by the CCD will cause s Therefore, by simply recording a series of speckle fringe images, the phase difference of the measured area can be obtained through these grayscale values, and then the deformation value of the rail can be further obtained.

[0109] The light intensity I on the CCD surface is very sensitive to the gradient of the out-of-plane deformation of the surface of the object being measured. If the light intensity I is sheared along the x direction (vertical direction, which allows an angle with the symmetry line of the rail section; during the monitoring process, the angle is allowed to fluctuate within a small range without affecting the monitoring accuracy, which is another feature of the present invention that distinguishes it from the prior art), the gradient and phase difference of the out-of-plane deformation ω (i.e., the out-of-plane displacement ω) The relationship between can be expressed as:

[0110]

[0111] in, is the derivative of the out-of-plane displacement along the x-direction, λ is the wavelength of the laser, and Δx is the distance between the two corresponding shear points ( Figure 3 The distance between A and A', or between B and B'). It represents the relative phase difference caused by rail deformation. The light intensity I of the speckle fringe pattern mentioned above is calculated by the distribution formula (2).

[0112] Δx is a known quantity, so when we get the phase difference Then the derivative of the off-plane displacement along the x direction can be solved according to the above formula (3): Derivative of the out-of-plane displacement ω After integration along the x-direction, the out-of-plane displacement ω (i.e., depth information ω) can be obtained.

[0113] That is, the phase difference can be calculated by Thus, the out-of-plane displacement ω is obtained.

[0114] This shows that the collected shear speckle pattern contains depth information v that is not contained in traditional two-dimensional images, which can solve the problem of fastener loosening detection that requires depth information ω.

[0115] However, according to the existing technology, in the traditional shear speckle interferometry, the phase difference must be restored by To calculate the out-of-plane displacement ω. The problem is that phase extraction is a key step in interferometry, but phase unwrapping is very complex and has many limitations, especially for two-dimensional surfaces. According to formula (2), the phase difference The change of will cause the intensity distribution of the speckle fringe pattern after subtraction|I s |The change occurs, manifesting itself in a speckle fringe pattern on the image that differs from the fringe pattern obtained when the fastener is in its normal state. Fasteners in different states produce different speckle fringe patterns, but humans have difficulty summarizing and classifying them.

[0116] In contrast, the present invention bypasses the complex phase extraction steps of the prior art and instead utilizes machine learning to directly establish a relationship between the speckle fringe pattern and the fastener's condition. Fastener operating status detection is achieved by directly classifying the resulting speckle fringe pattern. In other words, the present invention avoids complex mathematical calculations, accelerating recognition speed. Furthermore, by directly performing image recognition and implementing artificial intelligence, the present invention can even identify critical states before loosening, easily surpassing human capabilities in both image interpretation speed and judgment accuracy.

[0117] According to the present invention, the monitoring device can be installed on a dedicated vehicle or an operating vehicle. A single set of devices can monitor numerous fasteners in real time and over a large area. This avoids the drawbacks of the prior art, which requires the installation of multiple sets of monitoring devices, which can only monitor a limited area and incur high costs.

[0118] 3.2 Image processing part

[0119] 3.2.1 Fastener loosening detection based on decision tree

[0120] The decision tree classifies images by extracting the features of the target image. In the decision tree, each node represents a feature, and each branch represents the classification result that can be determined by this node. The decision tree of the present invention is as follows Figure 7 As shown, the loose state of the fastener can be directly determined by simply inputting the collected shear speckle pattern into the decision tree classifier.

[0121] In one embodiment of the present invention, the grayscale histogram of the acquired image is selected as a feature of the decision tree model. During the early training process, the decision tree model calculates the grayscale histograms of all training set samples and establishes a connection with the labels of the corresponding samples (tightened or loose). The calculated grayscale histograms are automatically divided into two types (the judgment basis is automatically learned and selected by the model), which respectively represent the tightened state or loose state of the fastener. By directly establishing the association between the grayscale histogram type and the fastener state, the task of completing the state detection by calculating the grayscale histogram is completed. In actual work, it is only necessary to input the acquired speckle fringe image into the decision tree model. The model calculates the grayscale histogram features of the image and classifies the grayscale histogram according to the judgment basis learned from the training set to complete the state detection of the fastener.

[0122] 3.2.2 Fastener loosening detection based on support vector machine

[0123] Support vector machine is a powerful mathematical model for classification and regression. Its basic idea is to find a hyperplane that can correctly divide the data set and has the largest geometric interval L. The schematic diagram is as follows Figure 8 In the present invention, the linear classification of the speckle pattern is achieved by extracting the grayscale histogram features of the speckle pattern.

[0124] In one embodiment of the present invention, the grayscale histogram of the acquired image is selected as the feature of the support vector machine model. During the early training process, the support vector machine model calculates the grayscale histograms of all training set samples and establishes a connection with the labels of the corresponding samples (tightened or loose). The calculated grayscale histograms are automatically divided into two types (the judgment basis is automatically learned and selected by the model), which respectively represent the tightened state or loose state of the fastener. By directly establishing the association between the grayscale histogram type and the fastener state, the task of completing the state detection by calculating the grayscale histogram is completed. In actual work, it is only necessary to input the acquired speckle fringe image into the support vector machine model. The model calculates the grayscale histogram features of the image and classifies the grayscale histogram according to the judgment basis learned from the training set to complete the state detection of the fastener.

[0125] 3.2.3 Fastener Loosening Detection Based on Convolutional Neural Network

[0126] Convolutional neural network is one of the representative algorithms of deep learning and is currently the most advanced image classification algorithm. Unlike the previous two algorithms, it has feature learning capabilities, that is, it does not require manual feature extraction, but the algorithm itself extracts features. As a classic algorithm for image classification, VGG-16 has shown excellent performance in the field of image classification. The present invention uses an improved VGG-16 model or other convolutional neural network model to complete the binary classification of the input shear speckle pattern to achieve the detection of the loose state of fasteners. The schematic diagram of the improved VGG-16 model of the present invention is as follows Figure 9 As shown in the figure, it contains a total of 14 convolutional layers, 4 maximum pooling layers, 4 fully connected layers and one Softmax layer.

[0127] The convolutional layer is composed of several convolutional units, each of which has parameters optimized using the backpropagation algorithm. Its function is to extract information from the input image, that is, to extract image features. The maximum pooling layer selects image features that are not affected by position in the convolutional layer; reduces the dimensionality of these features to increase the receptive field of subsequent features; and reduces the number of variables in the feature map, reducing the amount of computation. The fully connected layer converts all feature matrices from the pooling layer into large one-dimensional feature vectors, performing dimensionality reduction on the data. The softmax layer converts the previously output vector values ​​into probabilistic representations, aiming to express the classification results in the form of probabilities and complete the image classification.

[0128] Unlike the two previously demonstrated algorithms, convolutional neural networks possess the ability to learn features. During the initial training of the convolutional neural network model, there's no need to specify specific features as the basis for judgment. Instead, the model automatically selects one or more features (learned by the model itself, without humans knowing the specific features) as the optimal basis for judgment during the iteration process. It then establishes a direct link between this or these features and the fastener's condition, relying on the optimal features selected by the model to classify the image and detect the fastener's condition. In practice, the acquired speckle fringe pattern is input into the convolutional neural network model, which calculates and classifies the selected features of the image, ultimately detecting the fastener's condition.

[0129] For loosening detection of fasteners in railway systems, the trained model can be used to complete the loosening detection of fasteners in railway systems.

[0130] For loose fastener detection in other situations, such as factory piping, it's difficult to train a single model that can handle all fastener loosening detection scenarios. Different models need to be trained for different use cases. Alternatively, a single model can be trained to handle multiple scenarios.

[0131] The fasteners in the railway system may have different positions or angles during installation, which may cause the collected images to have certain deformations. Therefore, during the early model training process, new images can be added by scaling (cropping), flipping, and randomly cropping the original images to adapt to this problem (this method is also applicable when there is not enough data set).

[0132] The more training data an artificial intelligence model has, the better the model will be, the higher the judgment accuracy will be, and the better the universality will be.

[0133] Regarding the present invention, continuous learning can be achieved to continuously improve oneself. In other embodiments of the present invention, for example,

[0134] 1) For decision tree models, support vector machine models, or other machine learning models, other features or multiple combined features can be selected as the basis for judgment to find the best feature to explain the relationship between the target variable and the independent variable;

[0135] 2) Multi-scale training of convolutional neural network models. This involves inputting images of different scales and leveraging the unique nature of convolutional pooling to allow the neural network to fully learn the features of images at different resolutions, thereby improving machine learning performance.

[0136] 3) Use more different algorithms or models;

[0137] 4) Integrate multiple algorithms or models to achieve better results.

[0138] According to the present invention, not only the surface condition of the fastener can be monitored, but also the condition at a certain depth below the surface can be monitored; not only can the level of traditional manual inspection be achieved, but also traditional manual inspection can be surpassed in speed and quality). Compared with the existing technology, this is an unexpected technical effect.

[0139] The inspection device according to the present invention can be installed on a vehicle, and even a dedicated inspection vehicle can be designed.

[0140] like Figure 10a As shown, two sets of laser irradiation and area array camera systems can be set up, one set is used to simultaneously irradiate the fasteners on the outside of the left rail and the fasteners on the inside of the right rail, and the other set is used to simultaneously irradiate the fasteners on the outside of the right rail and the fasteners on the inside of the left rail.

[0141] like Figure 10b As shown, three sets of laser irradiation and area array camera systems can also be provided, one set for irradiating the fasteners on the outside of the left rail, one set for irradiating the fasteners on the outside of the right rail, and one set for simultaneously irradiating the fasteners on the inside of the left rail and the fasteners on the inside of the right rail.

[0142] like Figure 10cAs shown, four sets of laser irradiation and area array camera systems can be set up, one set for irradiating the fasteners on the outside of the left rail, one set for irradiating the fasteners on the inside of the left rail, one set for irradiating the fasteners on the inside of the right rail, and one set for irradiating the fasteners on the outside of the right rail.

[0143] like Figure 10d As shown, two sets of laser irradiation and area array camera systems can be set up, one set is used to simultaneously irradiate the fasteners on the outside of the left rail and the fasteners on the inside of the left rail, and the other set is used to simultaneously irradiate the fasteners on the inside of the right rail and the fasteners on the outside of the right rail.

[0144] In actual operation, it is preferred to Figure 10c The embodiment shown.

[0145] The technical solution of the present invention is not easily conceived, and is also reflected in the following: in the existing two-dimensional image detection method for fastener damage or loss, the two-dimensional image is captured from directly above and does not contain depth information about the fastener and its surroundings, making it impossible to detect loose fasteners, which requires depth information. Therefore, no researchers have previously used the detection or classification of two-dimensional images to detect the loose state of fasteners. However, the light intensity distribution of the speckle fringe pattern obtained by the digital shearing speckle interferometry technology in the present invention does contain depth information, which provides theoretical feasibility for fastener loosening detection requiring depth information.

[0146] 2D images do not contain depth information, so no one uses this for fastener loosening detection, as depth information is required.

[0147] In addition, two-dimensional image classification does not require calculation and can be directly "seen", which is much faster than three-dimensional vision methods.

[0148] The digital shearing speckle interferometry technology of the present invention directly measures stress or strain.

[0149] When the state of a fastener changes but has not yet deformed, the stress or strain on the measured surface has changed and can be directly detected by this technology; however, since deformation has not yet occurred at this time, traditional 3D vision-based methods cannot detect that the state of the fastener has changed.

[0150] In contrast, traditional thinking focuses on displacement or deformation, making it impossible to perceive or measure the special state where stress has changed but deformation has not yet occurred. This special state can essentially be regarded as a state of minimal or very little looseness.

[0151] According to the present invention, during use, by later learning the speckle fringe patterns of fasteners at different degrees of looseness, including the learning of states where the fasteners have been loose but have not yet been deformed, the model will not only be able to determine whether the fasteners are loose, but also to determine and identify the degree of looseness of the fasteners, and issue a warning or alarm.

Claims

1. A method for real-time monitoring of fastener loosening status using artificial intelligence, characterized in that: The monitoring object is the stress or strain in and around the fastener; The light intensity distribution of the fastener and adjacent surfaces is collected through laser irradiation and an area array camera system; the depth information of the fastener and adjacent surfaces is represented by the light intensity distribution of the measured surface; Through artificial intelligence machine learning, the corresponding relationship between the speckle fringe pattern and the fastener status is directly established; the loosening status of the fastener is detected by directly identifying the obtained speckle fringe pattern; By learning from the speckle fringe patterns of fasteners at different degrees of looseness, including those that have already loosened but not yet deformed, the artificial intelligence model can not only determine whether the fastener is loose, but also determine and identify the degree of looseness, and issue a warning or alarm; The training process of the artificial intelligence machine learning includes: Applying loads of different working conditions from zero load to critical load to the system in which the fastener is installed, while simultaneously collecting images of changes in physical parameters of the fastener and transmitting them to a computer; stopping the load application after each cyclic loading reaches a critical state until the fastener system returns to an initial state, at which point image collection is stopped; repeating the loading and unloading operations multiple times until sufficient images of changes in physical parameters of the fastener under normal conditions are obtained; Loosening the fastener to make it loose; repeating the loading, unloading and image acquisition operations multiple times until sufficient images of changes in physical parameters of the fastener in the loose state are acquired; A large number of images are randomly selected from the collected physical parameter change images as the original data set, ensuring that the images of the fasteners in normal state and the images in loose state each account for 50%; Using a computer to filter and normalize the physical parameter change images; labeling all physical parameter change images as normal or loose; or directly labeling the images of the fasteners that humans believe are in a normal state as normal, and directly labeling the images of the fasteners that humans believe are in a loose state as loose; The original data set is used as a training set to be put into the artificial intelligence judgment model for training, and the artificial intelligence judgment model is saved.

2. The artificial intelligence real-time monitoring method for fastener loosening status according to claim 1, characterized in that: The laser is emitted by the laser, which forms a larger diameter laser output beam after passing through the beam expander and irradiates the rough surface of the fastener and its adjacent area; the reflected light from the rough surface of the fastener and its adjacent area passes through the shearing device to form shear speckle; the shear speckle is recorded by the CCD and transmitted to the computer for storage and image processing.

3. The artificial intelligence real-time monitoring method for fastener loosening status according to claim 1, characterized in that: The CCD is used to collect the irregularly randomly distributed original shear speckle pattern of the tested fastener before deformation and the shear speckle pattern of the tested fastener after deformation. The shear speckle pattern after deformation is subtracted from the original shear speckle pattern to obtain the speckle fringe pattern that records the phase information, i.e., the depth information, of the surface of the tested fastener.

4. The artificial intelligence real-time monitoring method for fastener loosening status according to claim 2, characterized in that: A shearing device is used to cause a misalignment between the reference light and the interference light on the imaging plane, thereby producing a speckle fringe pattern after interference; a reference mirror of the Michelson interferometer is rotated by an angle, causing the two beams of reflected light to be misaligned on the imaging plane, thereby forming interference.

5. The artificial intelligence real-time monitoring method for fastener loosening status according to claim 1, characterized in that: The image of changes in the physical parameters of the fastener is a speckle fringe image.

6. A device for implementing the method for real-time artificial intelligence monitoring of fastener loosening status according to any one of claims 1 to 5, characterized in that: The device includes: a laser mounted in a position to illuminate the fastener; a spatial filter and beam expander disposed in a laser path between the laser and the fastener to filter and expand the laser beam; A monitoring image forming device that forms two identical, interfering images with a phase difference; A monitoring image receiving device using an area array camera; and The computer has an artificial intelligence judgment model for real-time monitoring of the looseness status of fasteners. The monitoring image is filtered and normalized and then input into the trained artificial intelligence judgment model, which detects the tightness status of the fasteners in real time.

7. The implementation device according to claim 6, characterized in that: The laser is a single longitudinal mode semiconductor laser with a wavelength of 532 nm.

8. The implementation device according to claim 6, characterized in that: The monitoring image forming device includes two polarizing plates and a Roger prism.

9. An inspection device for implementing the artificial intelligence real-time monitoring method for fastener loosening status according to any one of claims 1 to 5, characterized in that: The inspection equipment is installed on the operating vehicle and includes: A laser that sequentially illuminates each of the monitored fasteners while the vehicle is in operation; A spatial filter and beam expander, which is arranged in the irradiation path of the laser to filter and expand the laser beam; A monitoring image forming device that forms two identical, interfering images with a phase difference; A monitoring image receiving device using an area array camera; and The computer has an artificial intelligence judgment model for real-time monitoring of the looseness of fasteners. The monitoring image is filtered and normalized and then input into the trained artificial intelligence judgment model, which detects the tightness of the fasteners in real time.

10. A patrol vehicle implementing the artificial intelligence real-time monitoring method for fastener loosening status according to any one of claims 1 to 5, characterized in that: The inspection vehicle includes: A laser that sequentially illuminates each of the monitored fasteners while the inspection vehicle is driving; A spatial filter and beam expander, which is arranged in the irradiation path of the laser to filter and expand the laser beam; A monitoring image forming device that forms two identical, interfering images with a phase difference; A monitoring image receiving device using an area array camera; and The computer has an artificial intelligence judgment model for real-time monitoring of the looseness status of fasteners. The monitoring image is filtered and normalized and then input into the trained artificial intelligence judgment model, which detects the tightness status of the fasteners in real time.

Images