A laser impact spot size monitoring method based on multi-modal signal fusion

Abstract

The application belongs to the technical field of laser processing manufacturing and online monitoring, and specifically discloses a laser shock spot size monitoring method based on multi-modal signal fusion, which comprises the following steps: synchronously collecting LIBS spectral signals, ion beam explosion image signals and acoustic electric signals; performing feature extraction to obtain spectral line features, image topographic features and acoustic time-frequency features; through cross-modal feature alignment and modal attention weighted fusion, multi-modal data sets associated with spot size are obtained; a deep neural network is trained to obtain a spot size prediction model; after the training is completed, the predicted spot size is obtained through the spot size prediction model based on the collected multi-modal signals; the spot size deviation is obtained, and the workpiece surface spot size is adjusted to keep the target spot size. Through the application, real-time perception, intelligent judgment and closed-loop precise regulation and control of the spot size can be realized.

Description

Technical Field

[0001] This application belongs to the field of laser processing and manufacturing and online monitoring technology. More specifically, it relates to a laser shock spot size monitoring method based on multimodal signal fusion, which can be used for real-time sensing and closed-loop control of spot size during laser shock strengthening processing of curved components, weld areas and complex irregular components. Background Technology

[0002] In the field of laser processing and manufacturing, laser shock peening technology uses high-power-density, short-pulse lasers to act on the surface of materials, causing the coated energy absorption layer to rapidly vaporize and ionize, forming high-temperature and high-pressure plasma. The plasma expands rapidly under the constraint of the confinement layer and generates an ion beam explosion, which in turn forms a high-intensity shock wave that acts on the surface of the workpiece, causing controllable plastic deformation of the material surface and introducing a stable residual compressive stress field, thereby improving the service life and reliability of the components.

[0003] Laser shock peening is a key modification technology for improving the fatigue performance and service life of alloy materials. Its processing quality is highly sensitive to the laser spot size and the position of the laser beam. Especially in the strengthening of curved components, weld areas, and complex irregular components, abrupt changes in workpiece surface curvature, undulating weld morphology, and variable structural angles can lead to continuous dynamic changes in the working distance between the laser point of application and the optical system. This directly causes problems such as unstable laser spot size, uneven energy density distribution, and shift in the impact pressure field. Ultimately, this results in quality defects such as uneven residual stress distribution, insufficient strengthening layer depth, and weak strengthening effect at the interface between the weld and the curved surface, thereby reducing processing quality and component service performance.

[0004] Currently, laser rangefinders are commonly used in industrial production for monitoring processing distances. However, these devices can only perform static measurements of geometric distances and cannot characterize the spot state and impact strengthening quality in real time during processing. They also cannot establish a quantitative correlation between distance deviation and the effect of compressive stress injection, thus relying on post-processing detection and passive control. For laser shock strengthening scenarios involving curved surfaces, welds, and complex components, the spot size needs to be dynamically matched to the surface morphology in real time. Traditional static ranging methods cannot adjust the focal length in real time according to the curvature of the surface and the height of the weld, easily leading to problems such as excessively large spots or excessively strong focusing, resulting in localized over-strengthening, under-strengthening, or surface damage to the component.

[0005] Meanwhile, the laser shock strengthening process is subject to strong interference from plasma, sputtering, and acoustic vibrations. Laser rangefinders are prone to measurement deviations and cannot compensate for spot shifts caused by differences in surface conditions. They also have inherent defects such as detection lag, passive control, and poor adaptability, making it difficult to meet the high-precision and high-consistency strengthening requirements of complex components.

[0006] Current online monitoring technologies for laser shock strengthening mostly employ single-signal detection or dual-modal signal fusion, which suffers from insufficient feature dimensions and limited anti-interference capabilities. Traditional laser ranging can only achieve static measurement of geometric distance, failing to characterize the spot state and compressive stress injection effect in real time during processing. Furthermore, it lacks the capability for real-time dynamic adjustment of the spot size and multi-modal intelligent judgment and closed-loop control for curved surfaces, welds, and complex components. Therefore, how to achieve real-time perception, intelligent judgment, and precise closed-loop control of the spot size in the dynamic changing working distance conditions of laser shock strengthening of curved surfaces, welds, and complex components is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the purpose of this application is to realize real-time perception, intelligent judgment and closed-loop precise control of the spot size.

[0008] To achieve the above objectives, in a first aspect, this application provides a method for monitoring the size of laser shock spot based on multimodal signal fusion, the method comprising: During the laser shock peening process, LIBS spectral signals, ion beam explosion image signals, and acoustic electrical signals are acquired simultaneously. Based on LIBS spectral signals, ion beam explosion image signals and acoustic electrical signals, feature extraction is performed to obtain spectral line features, image morphology features and acoustic time-frequency features; Based on spectral line features, image morphology features, and acoustic wave time-frequency features, a multimodal dataset associated with spot size is obtained through cross-modal feature alignment and modal attention weighted fusion. Based on a multimodal dataset, a deep neural network is trained to obtain a spot size prediction model; After training is completed, based on the collected LIBS spectral signals, ion beam explosion image signals and acoustic signals, the predicted spot size is obtained through cross-modal feature alignment, modal attention weighted fusion and spot size prediction model. Based on the predicted spot size and the target spot size, the spot size deviation is obtained. The spot size deviation is used to adjust the spot size on the workpiece surface to maintain the target spot size.

[0009] Specifically, during laser shock peening (LSP) processing, multiple signals are simultaneously acquired from the processing area, including LIBS spectral signals, ion beam explosion image signals, and acoustic signals. This allows for the simultaneous characterization of the real-time interaction between the laser and the workpiece from three complementary dimensions: energy radiation, morphological contour, and mechanical acoustic waves. Feature extraction is then performed on each of the three acquired signals to obtain spectral line features reflecting plasma radiation, image morphology features reflecting the spot shape, and acoustic time-frequency features reflecting the impact. This transforms signals with different physical properties into effective feature representations related to the spot size. Since changes in spot size with working distance simultaneously cause coordinated changes in spectral line intensity, spot morphology, and acoustic time-frequency distribution, these three features together constitute a multi-dimensional perception of the spot size, providing more comprehensive feature basis for real-time spot size determination than a single signal.

[0010] After acquiring spectral line features, image morphology features, and acoustic wave time-frequency features, this application eliminates temporal deviations and dimensional differences in different modal signals across the time and feature dimensions through cross-modal feature alignment. Then, through modal attention weighted fusion, it adaptively learns the contribution weights of the three modalities to the spot size, weakening the weights of interfered modes and strengthening the weights of effective modes to obtain a multimodal dataset associated with the spot size. Based on this multimodal dataset, a deep neural network is trained to learn the mapping relationship between multimodal features and spot size, thus obtaining a spot size prediction model. In this way, the spot size prediction process retains the complementary information of the three modalities while suppressing the influence of interference such as plasma splashes and vibrations at the processing site on the recognition results, giving the spot size determination high accuracy and anti-interference capability.

[0011] After training is completed, this application obtains the predicted spot size based on the real-time acquired LIBS spectral signal, ion beam explosion image signal and acoustic electrical signal through cross-modal feature alignment, modal attention weighted fusion and spot size prediction model; based on the predicted spot size and the target spot size, the spot size deviation is obtained, and the spot size deviation is used to adjust the spot size on the workpiece surface to maintain the target spot size.

[0012] By leveraging the synchronous acquisition, feature extraction, cross-modal fusion, model prediction, and deviation adjustment in the aforementioned processing, this application connects the real-time perception of the spot state with the deviation determination and closed-loop adjustment of the spot size into a continuous processing link. In the processing of curved surfaces, welds, and complex components, the spot size is corrected in real time according to the dynamic changes in the working distance, thereby realizing the real-time perception, intelligent determination, and precise closed-loop control of the spot size.

[0013] This section describes the data structure of the multimodal dataset. The dataset consists of several sample records, each corresponding one-to-one with a light spot size. Each record contains a fused multimodal feature field and a light spot size label field. The fused multimodal feature field is a fused multimodal feature vector obtained by cross-modal feature alignment and modal attention weighted fusion based on spectral line features, image morphology features, and acoustic time-frequency features. It comprehensively represents the processing state information of three complementary dimensions: energy, morphology, and acoustic waves. The light spot size label field is the actual light spot size value corresponding to that sample record. A correlation mapping relationship exists between the fused multimodal feature field and the light spot size label field. The multimodal dataset is used to train a deep neural network to establish a prediction mapping from the fused multimodal features to the light spot size. The input dimension of the deep neural network matches the dimension of the fused multimodal feature vector.

[0014] For example, the deep neural network is a multilayer perceptron (MLP) structure, including an input layer, several hidden layers, and an output layer. The node dimensions of the input layer match the feature field dimensions of a single sample record in the multimodal dataset, and are used to receive the aligned and fused multimodal features. The hidden layers use nonlinear activation functions to map and abstract the multimodal features layer by layer, establishing a nonlinear mapping relationship between the multimodal features and the spot size. The output layer outputs a single predicted value for the spot size. The deep neural network can also be a ResNet deep learning model. It should be noted that this application does not limit the specific network structure of the deep neural network.

[0015] The input and output of the spot size prediction model are explained here. The input of the spot size prediction model is the fused multimodal features obtained by feature extraction, cross-modal feature alignment and modal attention weighted fusion based on LIBS spectral signal, ion beam explosion image signal and acoustic electrical signal. The output of the spot size prediction model is the predicted spot size value corresponding to the current processing time, so as to realize the input of multimodal signal and the direct output of the current spot size prediction result.

[0016] In one possible implementation, the above-mentioned feature extraction based on LIBS spectral signals, ion beam explosion image signals, and acoustic electrical signals to obtain spectral line features, image morphology features, and acoustic time-frequency features includes: Based on the LIBS spectral signal, baseline correction and denoising are performed to obtain spectral line characteristics (spectral line characteristics of ion beam explosion radiation). The spectral line characteristics include characteristic peak position, peak intensity and spectral line broadening. Based on the ion beam explosion image signal, gray value division and image segmentation processing are performed to obtain image morphology features (two-dimensional morphology features). The image morphology features include the outline, size distribution, gray level gradient and edge morphology parameters of the light spot. Based on the acoustic wave electrical signal, filtering and noise reduction processing is performed to obtain the time-frequency characteristics of the acoustic wave, which include the amplitude, energy, and frequency distribution parameters of the electrical signal.

[0017] The process of acquiring spectral line features is illustrated here. First, baseline correction is performed on the LIBS spectral signal to remove baseline drift introduced by continuous background radiation. Then, denoising is performed to suppress the interference of random noise on the spectral lines. On the corrected and denoised spectrum, the feature spectral lines corresponding to the ion beam explosion radiation are located, and their feature peak positions, peak intensities, and spectral line broadening are extracted. The feature peak positions correspond to the processed alloy materials, while the peak intensities and spectral line broadening are characteristic quantities for judging the ion beam explosion state under the same material. The three together constitute the spectral line features related to the energy dimension.

[0018] The process of acquiring image morphology features is illustrated here. First, grayscale values ​​are divided into the ion beam explosion image signal to distinguish the spot area from the background area. Then, image segmentation is performed to extract the morphology area of ​​the two-dimensional plasma spot. On the segmented spot area, the outline, size distribution, grayscale gradient and edge morphology parameters of the spot are extracted to form image morphology features related to the morphological dimension.

[0019] The process of obtaining the time-frequency characteristics of sound waves is illustrated here. First, the sound wave electrical signal is filtered and noise-reduced to remove the interference components introduced by vibration and environmental noise at the processing site. Then, the amplitude, energy and frequency distribution parameters of the electrical signal are extracted in the time domain and frequency domain respectively to form the time-frequency characteristics of sound waves related to the sound wave dimension.

[0020] In one possible implementation, the aforementioned cross-modal feature alignment includes: Based on spectral line features, image morphology features, and acoustic wave time-frequency features, alignment features are obtained by unifying the time dimension and feature dimension to eliminate temporal deviations and dimensional differences.

[0021] This section provides an illustrative example of the cross-modal feature alignment process. In the time dimension, using the same processing time as a benchmark, spectral line features, image morphology features, and acoustic wave time-frequency features are mapped to the same time scale, eliminating the temporal deviation caused by different acquisition channels for the three types of modes, and ensuring that the three types of features at the same time correspond to the same spot state. In the feature dimension, the three types of modal features are standardized to eliminate the numerical scale differences caused by different physical dimensions, thereby obtaining aligned features that are consistent in both the time and feature dimensions.

[0022] In one possible implementation, the above modal attention weighted fusion includes: Based on alignment features, a modal attention mechanism is used to learn the contribution weights of the three modalities to spot size prediction, and dynamic adaptive weighted fusion is performed to obtain the features of the fused multimodal system.

[0023] This section provides an illustrative explanation of the modal attention weighted fusion process. Based on the aligned spectral line features, image morphology features, and acoustic wave time-frequency features, the modal attention mechanism adaptively learns and assigns contribution weights to each modality according to the correlation between the three modalities and the spot size prediction. It assigns higher weights to effective modalities with high correlation and lower weights to interference-affected and low-correlation modalities. Subsequently, the three aligned features are dynamically weighted and fused according to the learned contribution weights to obtain fused multimodal features. This allows the fused multimodal features to weaken interference modalities while strengthening effective feature modalities, thus constructing a multimodal dataset strongly correlated with spot size.

[0024] For example, let the alignment features of the three modalities be respectively , , These correspond to spectral line features, image morphology features, and acoustic wave time-frequency features, respectively. (Through a learnable attention parameter matrix) With bias The alignment features of each modality are linearly transformed and mapped using an activation function to obtain the attention score for each modality. The calculation relationship is as follows: ; Parameter explanation: : indicates the first Modality-based attention score; This reflects the correlation between the mode and the predicted spot size; : Represents a non-linear activation function; Used to introduce nonlinear mappings to enhance the model's ability to express the degree of modal correlation; : Represents the attention parameter matrix; These are learnable parameters that are adaptively updated as the model trains. : indicates the first Alignment feature vectors of class modalities These correspond to spectral line features, image morphology features, and acoustic wave time-frequency features, respectively.

[0025] : Indicates attentional bias; These are learnable parameters that are adaptively updated as the model trains.

[0026] Subsequently, the attention scores of the three modalities were softmax normalized to obtain the contribution weight of each modality in the fusion. The calculation relationship is as follows: ; Parameter explanation: : indicates the first The contribution weights of the modal types; after Softmax normalization, they satisfy... A higher weight value indicates a greater contribution of the mode to the prediction of the spot size; : indicates the first Modality-based attention score; : indicates the first Modality-based attention score, , used for normalizing the summation of the denominator.

[0027] Then, the three types of aligned features are weighted and summed according to the contribution weight of each modality to obtain the fused multimodal features. The calculation relationship is as follows: ; Parameter explanation: : Represents a feature vector fused from multiple modalities; during the construction of the dataset, It is stored as the fused multimodal feature field for each sample record in the multimodal dataset; at the same time As input to a deep neural network / spot size prediction model; : indicates the first Contribution weights of different modalities; : indicates the first Alignment feature vectors of class modalities; , , These correspond to spectral line features, image morphology features, and acoustic wave time-frequency features, respectively. It is understandable that the attention parameter matrix is ​​involved in the model training process. With bias Adaptive updates during parameter iteration training, thus improving contribution weights. Ultimately, the algorithm converges to an optimal allocation that reflects the true contribution of the three modes to the spot size prediction, thereby suppressing the weights of interfering modes and enhancing the weights of effective modes.

[0028] It should be noted that the above attention scores The calculation method is not limited to the form of linear transformation plus activation function mentioned above.

[0029] In one possible implementation, the above-mentioned method of training a deep neural network based on a multimodal dataset to obtain a spot size prediction model includes: Based on the multimodal dataset, the training set, validation set, and test set are obtained according to a preset ratio. Based on the training set, validation set, and test set, the parameters of the deep neural network are iteratively trained, hyperparameters are tuned, and accuracy is verified to obtain a convergent and stable spot size prediction model.

[0030] Specifically, before dividing the dataset according to a preset ratio, the multimodal dataset is first preprocessed by standard normalization to eliminate differences in dimensions. Then, the multimodal dataset is divided into a training set, a validation set, and a test set according to a preset ratio of 7:1:2. The training set is used to perform iterative parameter training on the deep neural network, the validation set is used to perform hyperparameter tuning and overfitting suppression, and the test set is used to perform model accuracy verification. Finally, a spot size prediction model that is convergent and stable and whose accuracy meets the process requirements is obtained.

[0031] In one possible implementation, the aforementioned LIBS spectral signal, ion beam explosion image signal, and acoustic electrical signal are obtained through the following steps: Based on the spectral and image information generated by the ion beam explosion, the beam is reflected by the scanning galvanometer and then split into two beams by the beam splitter after passing through the transmission and reflection mirror. One beam enters the camera to acquire the ion beam explosion image signal, and the other beam enters the spectral analyzer through the spectral signal receiver to acquire the LIBS spectral signal. The acoustic waves generated by the ion beam explosion on the workpiece surface are collected by acoustic wave sensors placed symmetrically on both sides of the processing surface to obtain acoustic electrical signals.

[0032] This section describes the hardware composition of the real-time monitoring and control system for laser shock peening spot size. The system includes: The laser source unit includes a laser, which is a solid-state nanosecond laser used to emit laser light with a wavelength of 1064nm or 532nm. The spot size adjustment unit includes a focusing lens and a voice coil motor. The focusing lens is fixed above the voice coil motor. The position of the focusing lens is adjusted by changing the voltage input of the voice coil motor, thereby adjusting the spot size on the workpiece surface. The optical path transmission unit includes a reflector, a transmission mirror, and a scanning galvanometer arranged in sequence. After being focused by a focusing lens, the laser beam passes through the reflector, transmission mirror, and scanning galvanometer in sequence to converge into a laser spot on the surface of the workpiece. The reflector reflects beams of wavelengths of 1064nm and 532nm. The transmission mirror reflects beams of wavelengths of 1064nm and 532nm while transmitting beams of wavelength ... The plasma imaging coaxial acquisition unit includes a beam splitter and a camera. The spectral and image information generated by the ion beam explosion is reflected by the scanning galvanometer and passes through the transmission and reflection mirror. The beam splitter then splits the spectral and image information into two beams. One beam enters the camera to acquire the ion beam explosion image signal, and the other beam enters the LIBS signal coaxial acquisition unit. The LIBS signal coaxial acquisition unit includes a spectral signal receiver and a spectral analyzer. The spectral signal receiver collects the light beam, which then enters the spectral analyzer to acquire the LIBS spectral signal. The acoustic wave acquisition unit includes acoustic wave sensors symmetrically placed on both sides of the processing surface, used to acquire acoustic waves generated by ion beam explosion on the workpiece surface and obtain acoustic wave electrical signals. The central processing unit (computer) is connected to the LIBS signal coaxial acquisition unit, the plasma imaging coaxial acquisition unit, and the acoustic wave acquisition unit, respectively. It is used to receive LIBS spectral signals, ion beam explosion image signals, and acoustic electrical signals, and to complete real-time monitoring (monitoring and control) of the spot size during model training and processing. Among them, the surface of the workpiece being processed is a curved surface or an irregular weld seam of metal material.

[0033] In one possible implementation, after obtaining the spot size deviation based on the predicted spot size and the target spot size, the method further includes: Based on the spot size deviation, a closed-loop control voltage signal with corresponding amplitude is generated, and the voice coil motor is driven to move the focusing lens to dynamically adjust the spot size on the workpiece surface to maintain the target spot size.

[0034] Specifically, a closed-loop control voltage signal with corresponding amplitude is generated based on the spot size deviation. The closed-loop control voltage signal drives the voice coil motor to move the focusing lens, thereby changing the optical path from the focusing lens to the reflector in the optical path relationship. This, in turn, changes the total optical path (from the focusing lens to the workpiece surface) in real time, dynamically adjusting the spot size on the workpiece surface to maintain the target spot size, thus realizing real-time closed-loop control of the spot size during laser shock strengthening processing.

[0035] It is understood that the above specific implementation methods can be combined with each other without conflict. The listed processing procedures, structural components and parameter values ​​are all illustrative examples and do not constitute a limitation on the scope of protection of this application.

[0036] In a second aspect, this application provides an electronic device, including: a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions; the one or more processors invoke the computer instructions to cause the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect.

[0037] Thirdly, this application provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect or any possible implementation thereof.

[0038] Fourthly, this application provides a computer program product, including a computer program or instructions that, when run on an electronic device, cause the electronic device to perform the method described in the first aspect or any possible implementation thereof.

[0039] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0040] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art.

[0041] (1) Multimodal complementary sensing improves the anti-interference capability and accuracy of state recognition. This application simultaneously acquires three types of data: LIBS spectrum, ion beam explosion image, and acoustic wave signal. It characterizes the processing state from three complementary dimensions: energy, morphology, and mechanics. This overcomes the shortcomings of traditional single-modal sensing, which is easily affected by plasma splash, vibration, etc., and achieves high-precision and high-robust sensing of spot state and processing quality.

[0042] (2) Adaptive weighted fusion of multimodal features combined with a deep neural network prediction model to achieve end-to-end closed-loop intelligent control. This application models the correlation between multimodal data and the position of the focusing lens driven by the voice coil motor. The model can not only determine the state deviation, but also directly output executable control signals to drive the focusing lens to adjust the focal position in real time. This breaks through the limitations of traditional static calibration and manual adjustment, and realizes dynamic focusing and adaptive control in the process of complex curved surface processing.

[0043] (3) Improve the processing consistency and process stability of complex curved surface components. In view of the problem that the working distance changes dynamically with the workpiece undulation in the processing of curved surfaces, this application uses millisecond-level real-time data acquisition, model inference and closed-loop adjustment to keep the laser spot size and energy density in the desired state throughout the processing, which effectively solves the problems of uneven compressive stress layer depth and unstable strengthening effect of curved surface components.

[0044] (4) Reduce process dependence and improve the automation level of laser shock strengthening. Traditional processes rely heavily on manual experience for offline calibration and parameter adjustment. This application realizes unmanned and intelligent control of the processing process by automatically learning the mapping relationship between multimodal features and process states through the model, which reduces the dependence on the operator's experience and improves the repeatability and portability of the process. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the laser shock enhancement spot size real-time monitoring and control system provided in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the construction of the multimodal spot size prediction model provided in an embodiment of this application; Figure 3 This is a schematic diagram of the dynamic focusing strategy during the actual processing provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0046] To facilitate a clearer understanding of the various embodiments of this application, some relevant background knowledge will be introduced as follows.

[0047] In the field of laser shock peening, the control of the laser spot state mainly relies on offline process calibration combined with static geometric measurement using a laser rangefinder. This method can only complete the static parameter setting before processing and cannot perceive the laser energy injection state, plasma morphology, and shock peening mechanical effects in real time during processing. In the processing of complex curved surface components, it is easily affected by workpiece surface undulations, vibrations, plasma splashes, etc., causing working distance deviations to go undetected, resulting in problems such as unstable laser spot size, insufficient impact pressure, and uneven compressive stress layer depth.

[0048] Some existing technologies attempt to introduce single- or dual-modal sensing methods. For example, they may use only spectral signals to monitor plasma characteristics, or only acquire spot morphology through images, or combine the above methods to observe and record the plasma, and then correlate it with the processed results. Such single-modal or dual-modal sensing methods improve the perception capability of the processing state to a certain extent. However, due to the single signal dimension, they cannot comprehensively characterize the processing process from the three dimensions of energy, morphology, and mechanics simultaneously. They have shortcomings such as weak anti-interference ability, insufficient accuracy of state determination, and difficulty in establishing a direct mapping relationship with the actual processing quality. They cannot achieve real-time closed-loop control of spot size and cannot meet the requirements of high-precision and high-consistency industrial production.

[0049] Regarding specific existing technical documents, patent document CN120536712A uses a digital twin-driven approach to enhance the monitoring of laser shock reinforcement of aircraft landing gear. Although it is a multimodal monitoring method, it only monitors the processing result of a single component and cannot adjust the processing effect during the processing. Patent document CN120336919A uses an acousto-optic dual-modality to monitor the sound and light signals during the processing, but it can only assess the processing quality and also cannot adjust it during the processing. Furthermore, dual-modal monitoring may result in inaccurate monitoring. In contrast, this application connects the real-time perception of the spot state with the determination of the spot size deviation and closed-loop adjustment into a continuous processing link, enabling real-time perception, intelligent determination, and precise real-time closed-loop control of the spot size.

[0050] Therefore, existing technologies for laser shock peening of curved surfaces, welds, and complex components generally suffer from the following technical defects: First, static geometric ranging can only set parameters before processing and cannot detect the laser energy injection state, plasma morphology, and mechanical effects in real time during processing. Second, single-mode or dual-mode sensing lacks sufficient signal dimensions, failing to comprehensively characterize the processing state from the three dimensions of energy, morphology, and mechanics simultaneously, resulting in weak anti-interference capabilities and insufficient accuracy in state determination. Third, existing monitoring methods struggle to establish a quantitative mapping relationship between monitoring characteristics and actual processing quality, making it impossible to achieve real-time closed-loop control of the laser spot size. These defects collectively lead to problems such as unstable laser spot size, uneven compressive stress layer depth, and unstable strengthening effects during the processing of complex curved components.

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0053] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0054] The embodiments of this application are described below with reference to the accompanying drawings.

[0055] Example 1: Real-time monitoring and control system for laser shock enhancement spot size.

[0056] This application proposes a method for real-time monitoring and control of laser shock blasting spot size based on multimodal signal fusion and deep learning. This method acquires three types of process characteristic signals: LIBS (Laser-Induced Breakdown Spectroscopy), acoustic emission signals, and ion beam explosion images. Combined with a deep neural network, it achieves online determination, intelligent evaluation, and closed-loop fine-tuning of the spot size during processing. This overcomes the shortcomings of traditional detection methods, which cannot correlate processing quality in real time and are insufficient to meet the demands of automated, high-precision production.

[0057] To achieve the above objectives, this application provides a real-time monitoring and control system for laser shock enhancement spot size based on a combination of multimodal signal fusion and deep learning, such as... Figure 1 As shown. The system includes a laser source unit, a spot size control unit, a LIBS signal coaxial acquisition unit, a plasma imaging coaxial acquisition unit, an acoustic wave acquisition unit, and a central processing unit. Specifically, it includes a laser 1, a focusing lens 2, a voice coil motor 3 (an electromagnetic drive device that uses the Lorentz force principle to achieve linear motion), a reflector 4, a transmission mirror 5, a scanning galvanometer 6, a workpiece 7, an acoustic wave sensor 8, a beam splitter 9, an ICCD camera 10 (Intensified Charge-Coupled Device Camera), a spectral signal receiver 11, a spectral analyzer 12, and a computer 13. The scanning galvanometer 6 includes two reflecting mirrors, designated as the first reflecting mirror 6-1 and the second reflecting mirror 6-2.

[0058] Laser 1 is a solid-state nanosecond laser. The emitted laser beam is first focused by focusing lens 2, then passes through reflecting mirror 4, transmission mirror 5, and scanning galvanometer 6 before converging into a laser spot of a specific size on the surface of the workpiece 7. The optical path A of the laser beam from focusing lens 2 to workpiece 7 is calculated as follows: Optical path A is composed of the sum of three segments: x, y, and z. x is the optical path from focusing lens 2 to reflecting mirror 4, y is the optical path from reflecting mirror 4 to scanning galvanometer 6 (first reflecting mirror 6-1), and z is the optical path from scanning galvanometer 6 (first reflecting mirror 6-1) to workpiece 7. The calculation relationship for optical path A is as follows: ; Parameter explanation: : Indicates the total optical path (i.e., working distance) of the laser beam from the focusing lens 2 to the workpiece 7. : This represents the optical path from the focusing lens 2 to the reflecting mirror 4, and its value can be changed by the movement of the focusing lens 2 driven by the voice coil motor 3; : This represents the optical path from mirror 4 to scanning galvanometer 6 (first reflecting mirror 6-1), and its value remains constant. : This represents the optical path from the scanning galvanometer 6 (first reflecting mirror 6-1) to the workpiece 7, and its value is determined by the surface morphology of the workpiece 7.

[0059] As shown in the above equation, the change in optical path A ultimately affects the size of the laser spot on the surface of the workpiece 7; y remains constant, the value of x can be changed by the movement of the focusing lens 2 driven by the voice coil motor 3, and z is determined by the surface morphology of the workpiece 7. Therefore, by adjusting x through the voice coil motor 3, the optical path A can be changed, thereby achieving the adjustment of the spot size.

[0060] When the laser beam acts on the surface of the workpiece 7, it generates an ion beam explosion. The emitted spectral and image information is reflected by the scanning galvanometer 6, passes through the transmissive mirror 5, and is split into two beams on the surface of the beam splitter 9: one beam directly enters the ICCD camera 10 to acquire the ion beam explosion image information; the other beam is collected by the spectral signal receiver 11 and then enters the spectral analyzer 12 to acquire the ion beam explosion spectrum. Simultaneously, the acoustic wave signal of the ion beam explosion on the surface of the workpiece 7 is collected by the acoustic wave sensors 8 placed symmetrically on the left and right sides to acquire the acoustic wave electrical signal data. Finally, the data of the three modes are input into the computer 13 (central processing unit) for machine learning and real-time adjustment of the spot size.

[0061] The components are described in further detail below. The laser 1 emits a beam with a wavelength of 1064nm or 532nm. The focusing lens 2 allows beams of 1064nm and 532nm wavelengths to pass through and be focused; its focal length is related to the processing distance. The focusing lens 2 is fixed above the voice coil motor 3, and its position is adjusted by changing the voltage input of the voice coil motor 3. The reflecting mirror 4 reflects beams of 1064nm and 532nm wavelengths. The transmission mirror 5 reflects beams of 1064nm and 532nm wavelengths while allowing beams of other wavelengths to pass through. These other wavelengths are the characteristic line spectrum wavelengths generated by the plasma explosion when the laser acts on the workpiece 7. The mirrors inside the scanning galvanometer 6 can reflect 1064nm and 532nm beams, as well as the characteristic line spectrum wavelengths of the plasma explosion.

[0062] The surface of the workpiece 7 is a metal material with a certain curvature, an arc shape, or an irregular weld. The acoustic sensor 8 can receive the time and frequency information of the ion beam explosion. The beam splitter 9 splits the ion beam explosion beam into two beams, which enter the ICCD camera 10 to obtain a two-dimensional image of the ion beam explosion, and the spectral information of the corresponding band is collected by the spectral analyzer 12 through the spectral signal receiver 11. Then, the spectral data, two-dimensional image data, and acoustic signal data are transmitted to the computer 13 for learning and optimization to obtain a multimodal spot size prediction model.

[0063] Example 2: Method for constructing a multimodal spot size prediction model.

[0064] Figure 2 This diagram illustrates the construction of a multimodal spot size prediction model. This application uses a deep neural network based on multimodal signal fusion to simulate and train the spot size prediction model; the construction process is as follows: Figure 2 As shown, the steps include the following.

[0065] Step S101: Perform laser shock peening on the alloy workpiece under different spot sizes, and simultaneously acquire multiple sets of initial LIBS spectral data, initial ion beam explosion image data and initial acoustic wave electrical signal data to complete the synchronous acquisition of multimodal raw data.

[0066] Step S102 involves extracting targeted features from the three types of raw data: extracting spectral line features from LIBS spectral data, extracting two-dimensional morphological image features from ion beam explosion image data, and extracting time-frequency features from acoustic wave electrical signal data, thereby achieving effective feature representation of different modal signals.

[0067] Step S103: Cross-modal feature alignment is performed on the spectral line information, morphological information and time-frequency information extracted from the three modes. Modal attention mechanism is used to realize adaptive weighted feature fusion of the three types of signals: spectrum, image and sound wave, and a unified multimodal dataset strongly correlated with the spot size is constructed.

[0068] Step S104: Standardize and preprocess the unified multimodal dataset, and then divide it into training set, validation set and test set according to a preset ratio; use deep neural network to complete model training and parameter tuning, and finally obtain a high-precision spot size prediction model, which provides algorithm support for real-time spot size determination and dynamic focusing control in subsequent processing.

[0069] The specific features of each of the above steps are as follows. Step S101.1: Under the same laser single-pulse energy, processing parameters with different spot sizes are preset to conduct a comparative laser shock strengthening test on the alloy workpiece, and an ion beam explosion is generated on the alloy surface. Step S101.2: A coaxial acquisition system is used to acquire initial plasma spectral data and transient initial image data of the ion beam explosion, with the optical path as follows... Figure 1 As shown; simultaneously, acoustic wave sensors 8 placed on the left and right sides of the processing surface collect acoustic wave electrical signal data generated by the ion beam explosion. Step S101.3: through a spot size gradient experiment covering the entire process range, multiple sets of corresponding original sample data are obtained to complete the construction of the original multimodal sample library for model training.

[0070] Step S102.1: For the LIBS spectral data, after baseline correction and denoising, extract the spectral features of the ion beam explosion radiation, including core spectral parameters such as characteristic peak position, peak intensity, and spectral line broadening. The characteristic peak position corresponds to the selected alloy material of the processing surface, and the core parameters such as peak intensity and spectral line broadening are the standards for judging the ion beam explosion state under the same material. Step S102.2: For the ion beam explosion image data, after grayscale value division and image segmentation, extract the two-dimensional plasma spot morphology image features, including spot contour, size distribution, grayscale gradient, and edge morphology parameters. Step S102.3: For the acoustic signal data, after filtering and denoising, extract the time-frequency domain feature parameters such as signal amplitude, energy, and frequency distribution. Step S102.4: For the three types of modal data with completely different physical properties, use corresponding dedicated algorithms for feature extraction, remove redundant information from the original noise, and retain effective features strongly correlated with the spot size.

[0071] Step S103.1: The spectral line information, image morphology information, and acoustic wave time-frequency information extracted in step S102 are uniformly aligned in terms of time dimension and feature dimension to eliminate temporal deviations and dimensional differences between different modal data. Step S103.2: A modal attention mechanism is introduced to automatically learn the contribution weights of the three modalities to the spot size prediction, realizing dynamic adaptive weighted feature fusion of spectral, image, and acoustic wave signals, weakening the weights of interfering modalities and strengthening the weights of effective feature modalities, and finally constructing a unified multimodal association dataset that corresponds one-to-one with the spot size.

[0072] Step S104.1: Perform standardization and normalization preprocessing on the fused unified multimodal association dataset to eliminate dimensional differences. Step S104.2: Divide the unified multimodal association dataset into three datasets: a training set, a validation set, and a test set, in a ratio of 7:1:2. Step S104.3: Build a prediction model using a deep neural network. Iteratively train the model parameters using the training set, fine-tune hyperparameters and suppress overfitting using the validation set, and validate the model's accuracy using the test set. Finally, obtain a convergent, stable, and highly accurate spot size prediction model.

[0073] After the above four steps (S101~S104), a spot size prediction model can be obtained. This model can input multimodal signals and directly output the current spot size prediction result, providing an algorithmic basis for the dynamic focusing closed-loop control of laser shock strengthening of complex curved surface components.

[0074] Example 3: Dynamic focusing and closed-loop control method in actual processing.

[0075] Figure 3 This is a schematic diagram of the dynamic focusing strategy during actual processing. This embodiment illustrates the process of dynamic focusing and closed-loop control in the actual laser shock peening process of this application, such as... Figure 3 As shown, the specific steps are as follows.

[0076] Step S201: During the actual laser shock strengthening process, the LIBS spectral signal data corresponding to the plasma on the surface of the workpiece, the transient morphological image data of the ion beam explosion, and the acoustic-electric signal data converted from the shock wave are simultaneously collected by the multi-modal sensing unit to provide multi-modal characteristic data support for subsequent spot size analysis.

[0077] Step S202: Input the real-time acquired LIBS spectral data, ion beam explosion image data, and acoustic signal data into... Figure 2 The predicted spot size is output in the spot size prediction model obtained in the paper.

[0078] Step S203: Obtain the spot size value output by the spot size prediction model in step S202, compare it with the spot size required by the actual process, and obtain the quantitative deviation value of the spot size to provide a decision basis for subsequent control.

[0079] Step S204: Based on the comparison results of the light spot size deviation, generate a closed-loop control voltage signal with the corresponding amplitude; based on... Figure 1 The calculation relationship of the total optical path length in the middle processing (i.e.) The voltage signal drives the voice coil motor 3 to move the focusing lens 2 precisely to change the size of x, and adjusts the total processing optical path A between the focusing lens 2 and the workpiece 7 in real time, dynamically adjusting the spot size to maintain the target spot size expected by the process, thus realizing real-time closed-loop precise control of the spot size during laser shock strengthening processing.

[0080] Based on the methods in the above embodiments, this application provides an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.

[0081] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0082] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0083] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0084] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0085] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0086] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0087] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0088] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for monitoring the size of a laser shock spot based on multimodal signal fusion, characterized in that, include: During the laser shock peening process, LIBS spectral signals, ion beam explosion image signals, and acoustic electrical signals are acquired simultaneously. Based on LIBS spectral signals, ion beam explosion image signals, and acoustic electrical signals, feature extraction is performed to obtain spectral line features, image morphology features, and acoustic time-frequency features. Based on spectral line features, image morphology features, and acoustic wave time-frequency features, a multimodal dataset associated with spot size is obtained through cross-modal feature alignment and modal attention weighted fusion. A deep neural network was trained based on a multimodal dataset to obtain a spot size prediction model. After training is completed, based on the collected LIBS spectral signals, ion beam explosion image signals and acoustic signals, the predicted spot size is obtained through cross-modal feature alignment, modal attention weighted fusion and spot size prediction model. Based on the predicted spot size and the target spot size, the spot size deviation is obtained. The spot size deviation is used to adjust the spot size on the workpiece surface to maintain the target spot size.

2. The laser shock spot size monitoring method based on multimodal signal fusion according to claim 1, characterized in that, The method involves feature extraction based on LIBS spectral signals, ion beam explosion image signals, and acoustic electrical signals to obtain spectral line features, image morphology features, and acoustic time-frequency features, including: Based on the LIBS spectral signal, baseline correction and denoising are performed to obtain spectral line features, including characteristic peak positions, peak intensities, and spectral line broadening. Based on the ion beam explosion image signal, gray value division and image segmentation processing are performed to obtain image morphology features, including the outline, size distribution, gray level gradient and edge morphology parameters of the light spot. Based on the acoustic wave electrical signal, filtering and noise reduction processing is performed to obtain the time-frequency characteristics of the acoustic wave, which include the amplitude, energy, and frequency distribution parameters of the electrical signal.

3. The laser shock spot size monitoring method based on multimodal signal fusion according to claim 1, characterized in that, The cross-modal feature alignment includes: Based on spectral line features, image morphology features, and acoustic wave time-frequency features, alignment features are obtained by unifying the time dimension and feature dimension to eliminate temporal deviations and dimensional differences.

4. The laser shock spot size monitoring method based on multimodal signal fusion according to claim 3, characterized in that, The modal attention weighted fusion includes: Based on alignment features, a modal attention mechanism is used to learn the contribution weights of the three modalities to spot size prediction, and dynamic adaptive weighted fusion is performed to obtain the features of the fused multimodal system.

5. The laser shock spot size monitoring method based on multimodal signal fusion according to claim 1, characterized in that, The process of training a deep neural network based on a multimodal dataset to obtain a spot size prediction model includes: Based on the multimodal dataset, the training set, validation set, and test set are obtained according to a preset ratio. Based on the training set, validation set, and test set, the parameters of the deep neural network are iteratively trained, hyperparameters are tuned, and accuracy is verified to obtain a convergent and stable spot size prediction model.

6. The laser shock spot size monitoring method based on multimodal signal fusion according to claim 1, characterized in that, The LIBS spectral signal, ion beam explosion image signal, and acoustic electrical signal were obtained through the following steps: Based on the spectral and image information generated by the ion beam explosion, the beam is reflected by the scanning galvanometer and then split into two beams by the beam splitter after passing through the transmission and reflection mirror. One beam enters the camera to acquire the ion beam explosion image signal, and the other beam enters the spectral analyzer through the spectral signal receiver to acquire the LIBS spectral signal. The acoustic waves generated by the ion beam explosion on the workpiece surface are collected by acoustic wave sensors placed symmetrically on both sides of the processing surface to obtain acoustic electrical signals.

7. The laser shock spot size monitoring method based on multimodal signal fusion according to claim 1, characterized in that, After obtaining the spot size deviation based on the predicted spot size and the target spot size, the process also includes: Based on the spot size deviation, a closed-loop control voltage signal with corresponding amplitude is generated, and the voice coil motor is driven to move the focusing lens to dynamically adjust the spot size on the workpiece surface to maintain the target spot size.

8. An electronic device, characterized in that, include: Memory and one or more processors; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions; The one or more processors invoke the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instructions are run on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-7.

Images