Shaft part composite straightening method and device based on multi-mode camera and laser self-mixing displacement meter

Abstract

The invention relates to a shaft part composite straightening method and device based on a multi-mode camera and a laser self-mixing displacement meter. The method comprises the steps of collecting a 2D image, 3D point cloud data, straightness deviation data and a vibration time domain signal of a shaft part, and performing feature extraction processing to generate a multi-modal observation vector; performing data fusion processing through a Kalman filtering algorithm to obtain a comprehensive state estimation vector; the method comprises the following steps of: predicting a straightening parameter through a pre-trained AI (Artificial Intelligence) prediction network of a lightweight Transform architecture; and on the basis of the straightening parameters, a servo pressure head control instruction is generated to instruct a servo pressure head to execute the straightening action according to the rolling reduction, the pressure maintaining time and the supporting span in the straightening parameter instruction set, and the shaft part subjected to composite straightening is obtained. By the adoption of the method, the internal and external states of the shaft parts can be sensed in all directions and precisely regulated, and the straightening precision and the one-time success rate are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of shaft part straightening technology, and in particular relates to a composite straightening method and device for shaft parts based on a multimodal camera and a laser self-mixing displacement meter. Background Technology

[0002] As the high-end equipment manufacturing industry develops towards precision and intelligence, extremely stringent requirements are being placed on the straightness accuracy of shaft parts such as motor shafts in new energy vehicles and joint shafts in industrial robots. Precision straightening, as a crucial follow-up process to ensure the shape accuracy of shaft parts, directly impacts the performance and reliability of the final product. Currently, the straightening of shaft parts mainly relies on various automatic or semi-automatic straightening machines.

[0003] In traditional technology, mainstream straightening equipment generally uses mechanical or hydraulic pressure heads in conjunction with contact displacement sensors. The typical process is as follows: first, the straightness deviation of the shaft is measured using the displacement sensor; then, the operator sets a preliminary straightening parameter (such as the pressure reduction) based on experience, or the system calculates it using a simple mathematical model; next, the pressure head is driven to perform a straightening action; afterward, the shaft needs to be removed from the equipment and re-inspected at a separate measurement station. If the straightness does not meet the standard, it needs to be re-clamped and the above trial-and-error process of "measurement-straightening-offline re-inspection" repeated.

[0004] However, current straightening methods have several core problems that urgently need to be addressed: First, because they rely solely on geometric dimension measurements, they cannot perceive the residual stress distribution inside the shaft, leading to over- or under-calibration during the straightening process. Furthermore, stress release after straightening can easily cause re-bending, requiring repeated trial calibrations, resulting in low efficiency. Second, the lack of online detection capabilities for surface defects (such as indentations and scratches) may further damage the workpiece during straightening. Third, traditional measurement systems exhibit poor stability in industrial environments with vibration interference, leading to distorted measurement data and further affecting straightening accuracy. Summary of the Invention

[0005] Therefore, it is necessary to provide an intelligent alignment method that can achieve high precision, one-time success, and adaptability to complex industrial environments to address the aforementioned technical problems.

[0006] Firstly, this application provides a composite straightening method for shaft-type parts based on a multimodal camera and a laser self-mixing displacement meter, including:

[0007] Multimodal data of shaft parts are collected; feature extraction processing is performed on the multimodal data to generate multimodal observation vectors; the multimodal data includes 2D images of shaft parts, 3D point cloud data of shaft parts, straightness deviation data of shaft parts, and vibration time-domain signals of the surface of shaft parts;

[0008] Based on the multimodal observation vectors, a comprehensive state estimation vector is obtained by data fusion processing using the Kalman filter algorithm.

[0009] Based on the comprehensive state estimation vector, a pre-trained AI prediction network is used to predict the straightening parameters and generate a straightening parameter instruction set. The straightening parameter instruction set includes the compression amount, holding time, and support span. The pre-trained AI prediction network is a lightweight Transformer architecture.

[0010] Based on the straightening parameter instruction set, servo pressure head control instructions are generated. The servo pressure head control instructions are used to instruct the servo pressure head to perform straightening actions according to the pressing amount, holding time and support span in the straightening parameter instruction set, so as to obtain the shaft parts after composite straightening.

[0011] Furthermore, the method also includes:

[0012] Repeat the steps of collecting multimodal data of the composite-corrected shaft parts and generating servo pressure head control commands until the composite-corrected shaft parts meet the preset accuracy requirements, thus completing the composite correction of the shaft parts.

[0013] Furthermore, based on the multimodal observation vectors, data fusion processing is performed using the Kalman filter algorithm to obtain a comprehensive state estimation vector, including:

[0014] Use the multimodal observation vectors as the observation values ​​for the Kalman filter algorithm;

[0015] Calculate the prior state estimate at the current time step based on the posterior state estimate and state transition matrix from the previous time step.

[0016] Based on the covariance of the state prior estimate, the observation model matrix, and the observation noise covariance matrix, the Kalman gain is calculated using the Kalman gain calculation formula:

[0017]

[0018] in, For Kalman gain, The covariance of the prior state estimate. For the observation model matrix, The noise covariance matrix is ​​observed, and T is the transpose.

[0019] Based on the Kalman gain, observations, observation model matrix, and prior state estimate, calculate the posterior state estimate at the current time step; and use the posterior state estimate at the current time step as the comprehensive state estimate vector:

[0020]

[0021] in, This is the comprehensive state estimation vector. For state prior estimation, For Kalman gain, For the observed values, This is the observation model matrix.

[0022] Furthermore, feature extraction processing is performed on the multimodal data to generate multimodal observation vectors, including:

[0023] Surface defect identification processing is performed on 2D images in multimodal data to obtain surface defect features;

[0024] Geometric shape features are extracted from 3D point cloud data in multimodal data.

[0025] The vibration time-domain signal is processed by spectral analysis to obtain the vibration feature vector;

[0026] By combining surface defect features, geometric features, straightness deviation data, and vibration feature vectors, a multimodal observation vector is generated.

[0027] Furthermore, the vibration time-domain signal is subjected to spectral analysis to obtain vibration characteristics, including:

[0028] Bandpass filtering is applied to the vibration time-domain signal to obtain a clean vibration time-domain signal;

[0029] The vibration spectrum is obtained by performing a fast Fourier transform on the pure vibration time-domain signal.

[0030] Extract the frequency and amplitude corresponding to the peak with the largest amplitude from the vibration spectrum; and use the frequency and amplitude as vibration characteristics.

[0031] Furthermore, based on the comprehensive state estimation vector, a pre-trained AI prediction network is used to predict the calibration parameters, generating a calibration parameter instruction set, including:

[0032] The comprehensive state estimation vector is subjected to high-dimensional mapping to obtain a high-dimensional feature vector;

[0033] The dependencies between features in the high-dimensional feature vector are calculated using a multi-head self-attention mechanism to obtain an enhanced feature representation.

[0034] The enhanced feature representation is subjected to a linear transformation to generate a preliminary parameter set;

[0035] The initial parameter set is subjected to physical constraint optimization to generate a calibration parameter instruction set.

[0036] Secondly, this application also provides a composite straightening device for shaft parts based on a multimodal camera and a laser self-mixing displacement meter, including:

[0037] The data acquisition module is used to collect multimodal data of shaft parts; and to perform feature extraction processing on the multimodal data to generate multimodal observation vectors; the multimodal data includes 2D images of shaft parts, 3D point cloud data of shaft parts, straightness deviation data of shaft parts, and vibration time-domain signals of the surface of shaft parts;

[0038] The data fusion module is used to perform data fusion processing based on multimodal observation vectors using the Kalman filter algorithm to obtain a comprehensive state estimation vector;

[0039] The instruction set generation module is used to generate a straightening parameter instruction set based on the comprehensive state estimation vector and through a pre-trained AI prediction network. The straightening parameter instruction set includes the compression amount, holding time, and support span. The pre-trained AI prediction network is a lightweight Transformer architecture.

[0040] The instruction generation module is used to generate servo pressure head control instructions based on the straightening parameter instruction set. The servo pressure head control instructions are used to instruct the servo pressure head to perform straightening actions according to the pressing amount, holding time and support span in the straightening parameter instruction set, so as to obtain the shaft parts after composite straightening.

[0041] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the composite straightening method for shaft parts based on a multimodal camera and a laser self-mixing displacement meter as described in any of the embodiments of this application.

[0042] Fourthly, this application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement the composite straightening method for shaft parts based on a multimodal camera and a laser self-mixing displacement meter as described in any of the embodiments of this application.

[0043] The aforementioned composite straightening method and apparatus for shaft parts based on a multimodal camera and a laser self-mixing displacement meter acquires 2D images, 3D point cloud data, straightness deviation data, and vibration time-domain signals of the shaft parts; performs feature extraction processing to generate multimodal observation vectors; uses a Kalman filter algorithm for data fusion processing to obtain a comprehensive state estimation vector; predicts a straightening parameter set through a pre-trained AI prediction network with a Transformer architecture, and generates servo pressure head control commands to instruct the servo pressure head to perform straightening actions according to the pressure reduction, holding time, and support span in the straightening parameter command set, resulting in a composite straightened shaft part. This method enables comprehensive detection of surface defects, geometric deviations, and internal stresses of the parts, significantly improves straightening accuracy, prevents post-straightening bending, enhances the processing quality and service life of shaft parts, adapts to the processing needs of various precision shaft parts, and enhances the stability and applicability of the straightening process. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a composite straightening method for shaft parts based on a multimodal camera and a laser self-mixing displacement meter in one embodiment.

[0046] Figure 2 This is a schematic diagram of a composite straightening device for shaft parts based on a multimodal camera and a laser self-mixing displacement meter in one embodiment.

[0047] Figure 3 In one embodiment Figure 3 This is a schematic diagram of the computer equipment used in a remote monitoring and early warning method for wheat diseases and pests based on smart agriculture, as shown in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] In one embodiment, a composite straightening method for shaft parts based on a multimodal camera and a laser self-mixing displacement meter is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. Figure 1 As shown, in this embodiment, the method includes the following steps:

[0050] Step S101: Collect multimodal data of shaft parts; and perform feature extraction processing on the multimodal data to generate multimodal observation vectors; the multimodal data includes 2D images of shaft parts, 3D point cloud data of shaft parts, straightness deviation data of shaft parts, and vibration time-domain signals of the surface of shaft parts.

[0051] Feature extraction is the process of extracting key information (features) from raw data, transforming it into a structured form that can be used for model analysis or task processing. Its purpose is to: simplify data dimensions and remove redundant information; retain key information and highlight the essential features of the data; and unify data formats so that different types of raw data (such as images, text, and audio) can be adapted to the same processing flow.

[0052] For example, 2D images and 3D point cloud data of shaft-like parts can be simultaneously acquired using a multimodal camera. The 2D images are used to obtain texture and defect information of the part's surface, while the 3D point cloud data can be used to reconstruct the part's three-dimensional geometry using laser scanning technology, thereby accurately capturing its spatial contour. Straightness deviation data can be obtained by scanning along the part's axis using a high-precision laser rangefinder, and the degree of bending is quantified by calculating the distance deviation between each measuring point and the ideal axis. Vibration time-domain signals can be measured non-contactly using a laser self-mixing displacement meter. Feature extraction processing is performed on the acquired 2D images, 3D point cloud data, straightness deviation data, and vibration time-domain signals, and the processed features are combined into a multimodal observation vector. Among them, multimodal cameras are cameras capable of acquiring various types of image information. Their core is the integration of multiple sensing modalities, breaking through the limitations of traditional single-type image acquisition. They can simultaneously acquire visual data from different dimensions, rather than relying on a single imaging method. Laser scanning technology is a technique that scans targets by emitting laser beams, capturing their spatial information and converting it into digital data. The core principle is: a laser emitter emits a laser beam of a specific wavelength, pointing it towards the surface of the target object. The laser beam is reflected after encountering the object, and a receiver captures the reflected signal, recording the time or phase change of the laser propagation. Combined with parameters such as the angle and position of the scanning device, the three-dimensional coordinates of the target point are calculated, ultimately forming point cloud data or a two-dimensional image. High-precision laser rangefinders are instruments that accurately measure distances by emitting laser beams and receiving reflected signals, calculating time or phase differences. Laser self-mixing displacement gauges are devices that measure displacement based on the laser self-mixing effect. The laser self-mixing effect captures the microscopic vibrations of the part's surface by detecting the signal change caused by the interference between the laser beam reflected from the part's surface and the original beam, thus indirectly reflecting the internal stress state.

[0053] Step S102: Based on the multimodal observation vector, the data is fused using the Kalman filter algorithm to obtain the comprehensive state estimation vector.

[0054] Among them, the Kalman filter algorithm is a recursive state estimation algorithm that fuses multi-sensor data through two stages: prediction and update, thereby reducing noise and improving estimation accuracy; data fusion processing refers to the process of collecting, cleaning, associating, integrating and analyzing data from different sources and in different formats, and finally outputting unified data results with higher value and credibility; the comprehensive state estimation vector is defined as the comprehensive state of shaft parts, including parameters such as curvature and stress level.

[0055] For example, the multimodal observation vector is used as the observation input, and the data is fused using the Kalman filter algorithm to obtain the comprehensive state estimation vector.

[0056] Step S103: Based on the comprehensive state estimation vector, the straightening parameter prediction is performed through a pre-trained AI prediction network to generate a straightening parameter instruction set; the straightening parameter instruction set includes the compression amount, holding time and support span; the pre-trained AI prediction network is a lightweight Transformer architecture.

[0057] The pre-trained AI prediction network is based on a self-attention mechanism, which can automatically identify key features in the input data without manual intervention to filter important information. By capturing hidden feature correlations in the data, it transforms the comprehensive state estimation vector into accurate straightening parameters such as compression amount and holding time. The lightweight Transformer architecture is a variant of Transformer designed to adapt to edge / low-resource scenarios. While retaining the core mechanism, it reduces computation and storage overhead. The core optimization directions are: simplifying the attention mechanism: replacing global attention with local attention (such as window attention) and sparse attention to reduce computation; lightweighting network components: replacing fully connected layers with depthwise separable convolutions, or reducing the number of model layers and the dimension of hidden layers; optimizing computational efficiency: using techniques such as quantization and knowledge distillation to further compress the model size and improve inference speed; pre-training refers to training the model based on historical straightening data, which can adapt to the characteristics of different shaft parts and achieve accurate parameter recommendation.

[0058] For example, the comprehensive state estimation vector is input into a pre-trained lightweight Transformer architecture AI prediction network, which outputs a preliminary set of straightening parameters, including the reduction amount, holding time, and support span. These parameters are adjusted through physical constraint optimization to ensure that they meet the mechanical limits and working range of the servo head, such as the reduction amount not exceeding the maximum stroke and the holding time being set based on the material creep characteristics.

[0059] Step S104: Based on the straightening parameter instruction set, generate servo pressure head control instructions; the servo pressure head control instructions are used to instruct the servo pressure head to perform straightening actions according to the pressing amount, holding time and support span in the straightening parameter instruction set, so as to obtain the shaft parts after composite straightening.

[0060] For example, the pressure reduction parameter in the straightening parameter instruction set is converted into the target displacement signal of the servo pressure head, and the pressure head is moved to the specified position through closed-loop control; the pressure holding time parameter sets the pressure holding duration to eliminate the elastic recovery of the material; the support span parameter adjusts the relative distance between the pressure head and the support point to optimize the force distribution. After the straightening action is performed, the shaft part after composite straightening is obtained.

[0061] In this embodiment, a comprehensive state estimation vector is generated by collecting and extracting features from multimodal data of shaft-type parts. A Kalman filter algorithm is then used for data fusion, and the fused data is input into a pre-trained AI prediction network to generate a straightening instruction set. Based on this instruction set, composite straightening of the shaft-type parts is performed. This enables comprehensive perception and precise control of the internal and external states of shaft-type parts, significantly improving straightening accuracy and first-pass success rate.

[0062] In one exemplary embodiment, the method further includes:

[0063] Step S201: Repeat the steps of collecting multimodal data of the composite straightened shaft parts and generating servo pressure head control commands until the composite straightened shaft parts meet the preset accuracy requirements, thus completing the composite straightening of the shaft parts.

[0064] Among them, the preset accuracy requirement is a precision threshold set in advance based on product design, usage scenario or processing standard, such as a straightness error threshold of ≤0.02mm.

[0065] For example, after the servo indenter completes one straightening action, multimodal data of the straightened shaft-like part is immediately re-acquired, including 2D images, 3D point clouds, straightness deviation, and vibration time-domain signals. These data are processed through feature extraction to generate new multimodal observation vectors. A Kalman filter algorithm is used to fuse the new observation vectors, resulting in an updated comprehensive state estimation vector that reflects the real-time state changes of the straightened part. The new comprehensive state estimation vector is input into a pre-trained lightweight Transformer architecture AI prediction network to generate an adjusted straightening parameter instruction set. Based on the new parameters, servo indenter control commands are generated to drive the indenter to perform corrective straightening actions. This process is repeated. After each iteration, the straightness deviation component is analyzed from the comprehensive state estimation vector and compared with a preset accuracy threshold. If the deviation is greater than the threshold, the loop continues until the deviation meets the preset accuracy requirement, at which point the process terminates.

[0066] In this embodiment, a closed-loop iterative mechanism is used to connect multimodal data acquisition, fusion, prediction and execution into a self-optimizing system, thereby effectively solving the problem of multiple trial calibrations caused by the inaccuracy of a single calibration in traditional methods, improving efficiency and avoiding over-calibration or under-calibration. At the same time, online retesting reduces the risk of re-bending and adapts to industrial field environments.

[0067] In one embodiment, based on the multimodal observation vectors, a Kalman filter algorithm is used for data fusion processing to obtain a comprehensive state estimation vector, including:

[0068] Step S301: Use the multimodal observation vector as the observation value of the Kalman filter algorithm.

[0069] For example, the multimodal observation vector is read and formatted into a standard vector form, ensuring that its dimensions match the observation space preset by the Kalman filter algorithm. The formatted multimodal observation vector is then used as the observation value for the Kalman filter algorithm. This observation value is a composite vector formed by feature extraction from multi-sensor data, including dimensions such as visual features, straightness deviation, and vibration features. Here, the observation value represents the actual measurement data.

[0070] Step S302: Calculate the prior state estimate for the current time step based on the posterior state estimate and state transition matrix of the previous time step.

[0071] Among them, the posterior state estimate comes from the filtering result of the previous cycle and represents the optimal state value based on historical data; the state transition matrix predefines the system dynamics model, describing the evolution of the state (such as curvature and stress) of shaft parts over time; the system dynamics model is a modeling method based on feedback mechanism to analyze the dynamic changes of complex systems and reveal the interaction law of internal elements of the system.

[0072] For example, based on the posterior state estimate and state transition matrix from the previous time step, matrix multiplication is performed to generate a prior state estimate vector. This prior state estimate reflects state predictions based solely on the model and does not incorporate new observation data. The formula for matrix multiplication is as follows:

[0073]

[0074] This is the state prior estimation vector. This is a posterior estimate of the state from the previous time step. The state transition matrix needs to be pre-calibrated based on the material properties of the parts (such as elastic modulus) and motion constraints.

[0075] Step S303: Based on the covariance of the state prior estimate, the observation model matrix, and the observation noise covariance matrix, calculate the Kalman gain using the Kalman gain calculation formula:

[0076]

[0077] in, For Kalman gain, The covariance of the prior state estimate. For the observation model matrix, The noise covariance matrix is ​​observed, and T is the transpose.

[0078] Among them, the covariance of the state prior estimate is used to characterize the prediction uncertainty; the observation model matrix is ​​used to map the state space to the observation space and define the linear relationship between the state and the observation value, which needs to be set in advance in conjunction with the sensor calibration data; the observation noise covariance matrix is ​​used to quantify the sensor measurement error, which also needs to be set in advance in conjunction with the sensor calibration data; the calibration data can determine the correspondence between the sensor output and the real physical quantity.

[0079] For example, the product of the state prior covariance matrix and the transpose of the observation model matrix is ​​first calculated, then summed with the observation noise covariance matrix, and finally inverted and multiplied to obtain the Kalman gain.

[0080] Step S304: Calculate the posterior state estimate at the current time based on the Kalman gain, observed values, observation model matrix, and prior state estimate; and use the posterior state estimate at the current time as the comprehensive state estimate vector.

[0081]

[0082] in, This is the comprehensive state estimation vector. For state prior estimation, For Kalman gain, For the observed values, This is the observation model matrix.

[0083] For example, the innovation is first calculated, which is the residual between the observed value and the predicted observed value. Then, multiply it by the Kalman gain and add it to the state prior estimate to obtain the state posterior estimate. The innovation is used to reflect unexpected changes in the observed data and correct prediction biases; the posterior state estimate, as the final output, represents the current optimal state estimate. For example, the state covariance matrix is ​​updated using a closed-loop filtering process to ensure recursion in the next cycle.

[0084] In this embodiment, the multimodal observation vectors are recursively fused into a comprehensive state estimation vector using the Kalman filter algorithm. This effectively reduces the interference of sensor noise and model errors, generating a high-precision and reliable state vector.

[0085] In one embodiment, feature extraction processing is performed on the multimodal data to generate a multimodal observation vector, including:

[0086] Step S401: Perform surface defect recognition processing on the 2D image in the multimodal data to obtain surface defect features.

[0087] Among them, surface defect identification processing is based on digital image processing technology, which detects defect areas that differ from normal surfaces by analyzing the brightness, color or texture changes of image pixels. Digital image processing technology is a technology that uses computers to acquire, process and analyze images to achieve goals such as image optimization and feature extraction. Essentially, it converts images into digital signals (pixel matrices) and relies on algorithms to perform calculations on pixel data.

[0088] For example, the acquired 2D image is converted into a grayscale image to simplify computational complexity. A threshold segmentation algorithm can be used to binarize the grayscale image, dividing it into target and background regions. Morphological operations are then performed on the binary image, including opening and closing operations. Opening eliminates minor noise and smooths the target contour, while closing fills internal voids and connects adjacent regions. Contour features are extracted from the processed image, and geometric parameters such as the area, perimeter, and roundness of the defect region are calculated. These parameters are then normalized and combined into a surface defect feature vector. This feature vector can quantify the severity and distribution characteristics of defects such as scratches and pits on the surface of shaft-type parts. Among them, threshold segmentation algorithm sets one or more grayscale thresholds based on the distribution characteristics of image pixel grayscale values ​​to divide image pixels and achieve separation of target and background; morphological processing takes the pixel set as the processing object, focuses on the shape, size and other morphological features of the target in the image or achieves image modification or feature extraction through the interaction operation of structural elements with the image; opening operation refers to erosion followed by dilation to remove small noise while preserving the overall shape of the target; closing operation refers to dilation followed by erosion to fill small holes inside the target without changing the approximate size of the target; dilation refers to expanding the target area to fill small gaps or connect broken parts; erosion refers to shrinking the target area to eliminate small noise or refine the target edge.

[0089] Step S402: Extract geometric shape features from the 3D point cloud data in the multimodal data to obtain geometric shape features.

[0090] Among them, geometric shape feature extraction is the process of extracting key information that can characterize the essence of shape from image or entity geometric data, which can transform visual / spatial shape into quantifiable and analyzable parameters.

[0091] For example, 3D point cloud data in multimodal data is downsampled using voxel meshes to reduce the data volume while preserving the overall shape characteristics. Next, a random sampling consensus algorithm is used for planar detection to remove interfering point clouds such as those from support fixtures. The point cloud is then registered to a reference coordinate system using an iterative nearest-point algorithm to eliminate pose deviations. Based on the registered point cloud, principal component analysis is performed to extract three principal directions. The first principal component direction is used as the initial estimate of the axis, and the mean, variance, skewness, and kurtosis of all points to this axis are calculated as equidistant distribution statistics. These statistics collectively constitute the geometric features describing the degree of bending of shaft-like parts. Voxel grid downsampling reduces data volume by merging points within adjacent voxels while preserving overall shape features. Specifically, it divides the 3D space into regular cubes (voxels) of fixed size, covering the entire point cloud. All points are assigned to corresponding voxels, each containing several neighboring points. Points within each voxel are aggregated (e.g., by taking the center point, average point, or randomly selecting a point), and a single point replaces the entire set of points within that voxel. The random sampling consensus algorithm robustly estimates model parameters from noisy / outlier data: a small number of samples are randomly selected from the dataset as initial model data. An initial model, such as a line or plane, is fitted using these samples. The number of samples in the dataset that conform to this initial model (within a threshold error) is counted; these are called "inliers," and the rest are "outliers." This sampling-fitting-testing process is repeated, and the model with the most inliers is considered the optimal model. Iterative nearest-neighbor calculation... The first method is to quickly and accurately register two sets of 3D point clouds. It finds the optimal transformation matrix through iterative optimization to align the two point sets. Specifically, it sets an initial transformation matrix, usually an identity matrix, assuming the initial positions of the two point sets are close. For each point in the target point set, it finds the closest point in the source point set to form a point pair. Based on the matched point pairs, it solves for the translation and rotation matrices that minimize the sum of squared distances between the point pairs. It updates the position of the source point set according to the new transformation matrix and repeats the "matching-calculating transformation" steps until the distance error between the point pairs is less than a threshold or the maximum number of iterations is reached. The second method is Principal Component Analysis (PCA), a commonly used data dimensionality reduction method. While preserving the main information of the data, it transforms high-dimensional data into low-dimensional data. It finds "principal components," i.e., the directions with the largest variance in the data. These directions reflect the main changing characteristics of the data. Each principal component is independent of the others, avoiding information duplication. The first k principal components with the largest variance are selected to replace the original high-dimensional variables, thus achieving dimensionality reduction.

[0092] Step S403: Perform spectral analysis on the vibration time-domain signal to obtain the vibration feature vector.

[0093] Among them, spectrum analysis processing is a signal processing technique that converts a signal from the time domain to the frequency domain and analyzes its frequency composition and distribution. Through Fourier transform, the time-varying signal (such as sound and radio waves) is decomposed into sine / cosine components of different frequencies, and the amplitude, phase and other parameters of each frequency component are calculated to form a spectrum diagram corresponding to "frequency-amplitude", which intuitively presents the frequency characteristics of the signal.

[0094] For example, the vibration time-domain signal is subjected to spectral analysis to obtain the vibration feature vector.

[0095] Step S404: Combine surface defect features, geometric shape features, straightness deviation data and vibration feature vectors to generate a multimodal observation vector.

[0096] For example, min-maximum scaling can be used to map surface defect features, geometric features, straightness deviation data, and vibration feature vectors of different dimensions to a unified numerical range; the normalized features are then concatenated into a high-dimensional vector, and the arrangement of the features must be consistent with the observation matrix structure of the Kalman filter algorithm. Min-maximum scaling refers to mapping the data to [0,1] using the following formula:

[0097]

[0098] denoted as the normalized feature value, x as the original data, and min and max as the minimum and maximum values ​​of the data, respectively.

[0099] In this embodiment, quantized features are extracted from the raw data of visual and vibration sensors, standardized, and then integrated into a multimodal observation vector. This provides comprehensive and standardized multi-dimensional observation data for subsequent state estimation, thereby significantly improving the overall perception capability of the shaft's surface quality, geometry, and internal stress state.

[0100] In one embodiment, the vibration time-domain signal is subjected to spectral analysis to obtain vibration characteristics, including:

[0101] Step S501: Bandpass filtering is performed on the vibration time-domain signal to obtain a clean vibration time-domain signal.

[0102] Bandpass filtering is a signal processing technique that allows signals within a specific frequency range to pass through while filtering out low-frequency and high-frequency signals outside that range.

[0103] For example, the raw vibration time-domain signal acquired by a laser self-mixer displacement meter is read. This signal typically contains irrelevant components such as high-frequency noise, low-frequency drift, and power frequency interference. A digital filter can be used to bandpass filter the vibration time-domain signal to obtain a clean vibration time-domain signal. The digital filter is based on frequency selectivity, retaining the effective frequency band reflecting the inherent vibration characteristics of the shaft part by setting the passband frequency range, while attenuating interference signals outside the passband. The effective frequency band is predefined according to the material resonance characteristics and typical defect frequency distribution of the shaft part, including the lower and upper limits of the passband. These predefined parameters ensure the targeted nature of the filtering.

[0104] Step S502: Perform a fast Fourier transform on the clean vibration time-domain signal to obtain the vibration spectrum.

[0105] Among them, the Fast Fourier Transform (FFT) reduces the computational complexity of the Discrete Fourier Transform from O(n log n) to O(n log n) based on a divide-and-conquer strategy. ) decreased to O( This achieves efficient time-frequency conversion by decomposing the time-domain signal into a series of sinusoidal components to generate amplitude and phase spectra in the frequency domain. The number of points in the Fast Fourier Transform needs to be set in advance according to the sampling rate and analysis requirements, and this number of points determines the frequency resolution. The divide-and-conquer strategy is an algorithm that breaks down a complex problem into multiple identical / similar subproblems, recursively solves each subproblem (if the subproblem is small enough, it can be solved directly), and combines the solutions of the subproblems to obtain the final solution of the original problem.

[0106] For example, a Hanning window is applied to a clean vibration time-domain signal to reduce spectral leakage. The Hanning window is a commonly used cosine window function that reduces sideband effects caused by truncation by smoothly weighting the two ends of the signal. For example, a Fast Fourier Transform is performed on the signal after applying the Hanning window, outputting the vibration spectrum. This spectrum, with frequency on the horizontal axis and amplitude on the vertical axis, visually displays the energy distribution characteristics of the signal.

[0107] Step S503: Extract the frequency and amplitude corresponding to the peak with the largest amplitude from the vibration spectrum; and use the frequency and amplitude as vibration features.

[0108] For example, local maxima are identified by comparing the amplitude of each frequency point in the vibration spectrum with its neighborhood values. Before comparison, a peak threshold needs to be predefined to eliminate spurious peaks caused by noise, ensuring that only significant vibration components are retained. After identifying the local maxima, the peak with the largest amplitude is located, and its corresponding frequency and amplitude are recorded. The frequency reflects the dominant vibration frequency of the shaft-like part and is related to the internal stress state, while the amplitude characterizes the vibration intensity and indicates the energy level. Finally, the frequency and amplitude are combined into a vibration feature vector.

[0109] In this embodiment, noise interference from the original vibration time-domain signal is removed by a preset passband filter to obtain a clean signal. A fast Fourier transform is then used to convert the signal from the time domain to the frequency domain, generating a vibration spectrum. The main peak frequency and amplitude are extracted as quantization features. This effectively improves the signal-to-noise ratio and feature representativeness, providing accurate and reliable vibration dimension input for multimodal data fusion, thereby enhancing the ability to perceive the stress state of the component.

[0110] In one embodiment, based on the comprehensive state estimation vector, a pre-trained AI prediction network is used to perform calibration parameter prediction processing to generate a calibration parameter instruction set, including:

[0111] Step S601: Perform high-dimensional mapping processing on the comprehensive state estimation vector to obtain a high-dimensional feature vector.

[0112] Among them, high-dimensional mapping processing refers to projecting the input vector onto a higher-dimensional feature space based on linear transformation or nonlinear activation function, thereby enhancing the expressive power and separability of the features; the comprehensive state estimation vector contains multi-dimensional information such as the curvature and stress state of shaft parts.

[0113] For example, a comprehensive state estimation vector is read, which integrates multi-dimensional state information such as the bending degree and stress level of the shaft part. A preset embedding layer is invoked. The weight matrix of this embedding layer is trained using massive historical straightening data and is used to establish a mapping relationship between the low-dimensional state vector and the high-dimensional feature space. The bias term of this embedding layer is used to optimize the mapping accuracy. The comprehensive state estimation vector is multiplied by the weight matrix through matrix multiplication, and the bias term is added to output a high-dimensional feature vector with richer feature expression capabilities. Here, the preset embedding layer is a basic component in the lightweight Transformer architecture AI prediction network, used to complete the transformation from low-dimensional data to high-dimensional features; the matrix multiplication operation realizes the mapping transformation of vectors or matrices in different spaces by multiplying rows and columns accordingly.

[0114] Step S602: The dependencies between features in the high-dimensional feature vector are calculated through a multi-head self-attention mechanism to obtain an enhanced feature representation.

[0115] Among them, the multi-head self-attention mechanism is the core of the lightweight Transformer architecture. It splits features, calculates correlations independently, and then fuses the results to achieve accurate mining of dependencies between high-dimensional features.

[0116] For example, the number of attention heads and the feature dimension partitioning rules for each attention head are predefined to ensure that each attention head focuses on the association mining of different feature subsets. Based on the predefined number of attention heads and the feature dimension partitioning rules for each attention head, the high-dimensional feature vector is first subjected to three linear transformations to generate query vectors, key vectors, and value vectors respectively. Attention weights are obtained by calculating the similarity between the query vector and the key vector. Then, the attention weights are weighted and summed with the value vector. Finally, the outputs of multiple attention heads are concatenated and fused to obtain the enhanced feature representation. Here, linear transformation is a mapping method in vector space that preserves linear operations such as addition and scalar multiplication. Through a predefined transformation matrix, linear operations (matrix multiplication) are performed on the input feature vector, without changing the linear association characteristics of the data. This is a fundamental method of feature mapping in AI prediction networks. The elements of the transformation matrix are determined through model training optimization, directly defining the correspondence between input features and output results (such as calibration parameters). Similarity calculation refers to calculating the similarity between the query vector and each key vector using methods such as inner product and scaled dot product to obtain an initial association score. The initial score is normalized, and the attention weight is output. A higher weight indicates a stronger association between the corresponding features.

[0117] Step S603: Perform linear transformation on the enhanced feature representation to generate a preliminary parameter set.

[0118] For example, after obtaining the enhanced feature representation, a preset linear transformation layer is invoked to perform processing operations. The transformation matrix of this linear transformation layer has been optimized and determined through the model training process, and is used to establish a linear correspondence between the enhanced features and the straightening parameters. The enhanced feature representation is multiplied by the transformation matrix through matrix multiplication to obtain a preliminary parameter set containing the initial values ​​of the compression amount, holding time, and support span. This parameter set directly reflects the preliminary prediction results of the straightening parameters based on the current state of the part, but does not yet consider the physical constraints in actual industrial scenarios. Among them, the preset linear transformation layer is the core component in the lightweight Transformer architecture that realizes the transformation from features to straightening parameters. By performing matrix multiplication on the high-dimensional enhanced feature vector and the transformation matrix, it directly outputs the preliminary parameter set containing the initial values ​​of the straightening parameters, completing the mapping of the vector space.

[0119] Step S604: Perform physical constraint optimization on the preliminary parameter set to generate a straightening parameter instruction set.

[0120] Among them, physical constraint optimization refers to making reasonable corrections to parameters based on the actual physical rules of industrial straightening scenarios to ensure that the parameters meet the operating limits of the equipment and the processing characteristics of the parts.

[0121] For example, after reading the initial parameter set, the system checks whether each parameter in the initial parameter set meets the constraints, based on multiple preset physical constraints. Parameters exceeding the limits are adjusted, such as correcting the pressure reduction exceeding the maximum stroke to the maximum value allowed by the equipment, and adjusting the holding time to a reasonable range according to the material creep characteristics. Finally, a straightening parameter instruction set that meets the actual processing requirements is generated. The preset physical constraints are based on the mechanical structure parameters of the servo indenter, the material properties of shaft parts, and industry processing standards, including the maximum stroke limit of the servo indenter, the minimum holding time threshold, the safe range of the support span, and the elastic deformation limit of the material.

[0122] In this embodiment, the comprehensive state estimation vector is transformed into a high-dimensional feature vector with rich expressive power through high-dimensional mapping. An enhanced feature representation is obtained by mining the dependencies between features using a multi-head self-attention mechanism. A preliminary straightening parameter set is generated through linear transformation, and a straightening parameter instruction set that meets actual processing requirements is obtained through physical constraint optimization. This effectively improves the accuracy of straightening parameter prediction, avoids over- and under-calibration problems caused by insufficient feature mining in traditional straightening parameter settings, and avoids equipment damage or part processing defects caused by ignoring physical constraints. It significantly improves the reliability and adaptability of straightening parameter prediction.

[0123] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0124] Based on the same inventive concept, this application also provides a composite straightening device for shaft parts based on a multimodal camera and a laser self-mixing displacement meter for implementing the aforementioned composite straightening method for shaft parts based on a multimodal camera and a laser self-mixing displacement meter. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the composite straightening device for shaft parts based on a multimodal camera and a laser self-mixing displacement meter provided below can be found in the limitations of the composite straightening method for shaft parts based on a multimodal camera and a laser self-mixing displacement meter described above, and will not be repeated here.

[0125] In one exemplary embodiment, such as Figure 2 As shown, a composite straightening device 200 for shaft parts based on a multimodal camera and a laser self-mixing displacement meter is provided, comprising:

[0126] The data acquisition module 201 is used to collect multimodal data of shaft parts; and to perform feature extraction processing on the multimodal data to generate multimodal observation vectors; the multimodal data includes 2D images of shaft parts, 3D point cloud data of shaft parts, straightness deviation data of shaft parts, and vibration time-domain signals of the surface of shaft parts;

[0127] Data fusion module 202 is used to perform data fusion processing based on multimodal observation vectors using the Kalman filter algorithm to obtain a comprehensive state estimation vector;

[0128] The instruction set generation module 203 is used to generate a straightening parameter instruction set based on the comprehensive state estimation vector and through a pre-trained AI prediction network. The straightening parameter instruction set includes the compression amount, holding time, and support span. The pre-trained AI prediction network is a lightweight Transformer architecture.

[0129] The instruction generation module 204 is used to generate servo pressure head control instructions based on the straightening parameter instruction set. The servo pressure head control instructions are used to instruct the servo pressure head to perform straightening actions according to the pressing amount, holding time and support span in the straightening parameter instruction set, so as to obtain the shaft parts after composite straightening.

[0130] In one exemplary embodiment, the apparatus further includes:

[0131] The online retesting module is used to repeatedly execute the steps from collecting multimodal data of the composite-corrected shaft parts to generating servo pressure head control commands until the composite-corrected shaft parts meet the preset accuracy requirements, thus completing the composite correction of the shaft parts.

[0132] In one embodiment, the data fusion module 202 is further configured to:

[0133] Use the multimodal observation vectors as the observation values ​​for the Kalman filter algorithm;

[0134] Calculate the prior state estimate at the current time step based on the posterior state estimate and state transition matrix from the previous time step.

[0135] Based on the covariance of the state prior estimate, the observation model matrix, and the observation noise covariance matrix, the Kalman gain is calculated using the Kalman gain calculation formula:

[0136]

[0137] in, For Kalman gain, The covariance of the prior state estimate. For the observation model matrix, The noise covariance matrix is ​​observed, and T is the transpose.

[0138] Based on the Kalman gain, observations, observation model matrix, and prior state estimate, calculate the posterior state estimate at the current time step; and use the posterior state estimate at the current time step as the comprehensive state estimate vector:

[0139]

[0140] in, This is the comprehensive state estimation vector. For state prior estimation, For Kalman gain, For the observed values, This is the observation model matrix.

[0141] In one embodiment, the data acquisition module 201 is further configured to:

[0142] Surface defect identification processing is performed on 2D images in multimodal data to obtain surface defect features;

[0143] Geometric shape features are extracted from 3D point cloud data in multimodal data.

[0144] The vibration time-domain signal is processed by spectral analysis to obtain the vibration feature vector;

[0145] By combining surface defect features, geometric features, straightness deviation data, and vibration feature vectors, a multimodal observation vector is generated.

[0146] In one embodiment, the data acquisition module 201 is further configured to:

[0147] Bandpass filtering is applied to the vibration time-domain signal to obtain a clean vibration time-domain signal;

[0148] The vibration spectrum is obtained by performing a fast Fourier transform on the pure vibration time-domain signal.

[0149] Extract the frequency and amplitude corresponding to the peak with the largest amplitude from the vibration spectrum; and use the frequency and amplitude as vibration characteristics.

[0150] In one embodiment, the instruction set generation module 203 is further configured to:

[0151] The comprehensive state estimation vector is subjected to high-dimensional mapping to obtain a high-dimensional feature vector;

[0152] The dependencies between features in the high-dimensional feature vector are calculated using a multi-head self-attention mechanism to obtain an enhanced feature representation.

[0153] The enhanced feature representation is subjected to a linear transformation to generate a preliminary parameter set;

[0154] The initial parameter set is subjected to physical constraint optimization to generate a calibration parameter instruction set.

[0155] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the composite straightening method for shaft parts based on a multimodal camera and a laser self-mixing displacement meter as described above.

[0156] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0157] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0158] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A compound straightening method for shaft parts based on a multi-modal camera and a laser self-mixing displacement meter, characterized in that, The method comprises: Collecting multi-modal data of the shaft part; and performing feature extraction processing on the multi-modal data to generate a multi-modal observation vector; the multi-modal data includes a 2D image of the shaft part, 3D point cloud data of the shaft part, straightness deviation data of the shaft part, and vibration time domain signals of the surface of the shaft part; Based on the multi-modal observation vector, data fusion processing is performed through a Kalman filtering algorithm to obtain a comprehensive state estimation vector; Based on the comprehensive state estimation vector, straightening parameter prediction processing is performed through a pre-trained AI prediction network to generate a straightening parameter instruction set; the straightening parameter instruction set includes a press-down amount, a pressure holding time, and a support span; the pre-trained AI prediction network is a lightweight Transformer architecture; Based on the straightening parameter instruction set, a servo pressure head control instruction is generated; the servo pressure head control instruction is used to instruct the servo pressure head to perform a straightening action according to the press-down amount, the pressure holding time, and the support span in the straightening parameter instruction set to obtain a composite straightened shaft part.

2. The method of claim 1, wherein, The method further comprises: Repeating the steps of collecting the multi-modal data of the composite straightened shaft part to generating the servo pressure head control instruction until the composite straightened shaft part meets the preset accuracy requirement, and completing the composite straightening of the shaft part.

3. The method of claim 1, wherein, The data fusion processing based on the multi-modal observation vector through the Kalman filtering algorithm to obtain the comprehensive state estimation vector comprises: Taking the multi-modal observation vector as the observation value of the Kalman filtering algorithm; According to the state posterior estimation of the last time and the state transition matrix, the state prior estimation of the current time is calculated; Based on the covariance of the state prior estimation, the observation model matrix, and the observation noise covariance matrix, the Kalman gain is calculated through the Kalman gain calculation formula: wherein, is the Kalman gain, is the covariance of the state a priori estimate, is the observation model matrix, is the observation noise covariance matrix, T is the transpose; According to the Kalman gain, the observation value, the observation model matrix, and the state prior estimation, the state posterior estimation of the current time is calculated; and the state posterior estimation of the current time is taken as the comprehensive state estimation vector: wherein, is a combined state estimate vector, is a state prior estimate, is a Kalman gain, is an observation value, is an observation model matrix.

4. The method of claim 1, wherein, The feature extraction processing on the multi-modal data to generate the multi-modal observation vector comprises: Performing surface defect recognition processing on the 2D image in the multi-modal data to obtain surface defect features; Performing geometric shape feature extraction processing on the 3D point cloud data in the multi-modal data to obtain geometric shape features; Performing frequency spectrum analysis processing on the vibration time domain signal to obtain vibration features; Combining the surface defect features, the geometric shape features, the straightness deviation data, and the vibration features to generate a multi-modal observation vector.

5. The method of claim 4, wherein, The frequency spectrum analysis processing on the vibration time domain signal to obtain vibration features comprises: Performing band-pass filtering processing on the vibration time domain signal to obtain a pure vibration time domain signal; Performing fast Fourier transform on the pure vibration time domain signal to obtain a vibration frequency spectrum; Extracting the frequency and amplitude corresponding to the peak with the maximum amplitude from the vibration frequency spectrum; and taking the frequency and the amplitude as vibration features.

6. The method of claim 1, wherein, The straightening parameter instruction set is generated by performing straightening parameter prediction processing on the comprehensive state estimation vector based on a pre-trained AI prediction network, including: The high-dimensional feature vector is obtained by performing high-dimensional mapping processing on the comprehensive state estimation vector; The dependency relationship between each feature in the high-dimensional feature vector is calculated through a multi-head self-attention mechanism to obtain an enhanced feature representation; The enhanced feature representation is linearly transformed to generate a preliminary parameter set; The straightening parameter instruction set is generated by performing physical constraint optimization processing on the preliminary parameter set.

7. The shaft part compound straightening device based on a multi-modal camera and a laser self-mixing displacement meter, characterized in that, The device comprises: A data acquisition module is configured to collect multi-modal data of the shaft part, and perform feature extraction processing on the multi-modal data to generate a multi-modal observation vector. The multi-modal data includes a 2D image of the shaft part, 3D point cloud data of the shaft part, straightness deviation data of the shaft part, and vibration time domain signals of the surface of the shaft part. A data fusion module is configured to perform data fusion processing on the multi-modal observation vector based on a Kalman filtering algorithm to obtain a comprehensive state estimation vector. An instruction set generation module is configured to generate a straightening parameter instruction set by performing straightening parameter prediction processing on the comprehensive state estimation vector based on a pre-trained AI prediction network. The straightening parameter instruction set includes a press-down amount, a pressure holding time, and a support span. The pre-trained AI prediction network is a lightweight Transformer architecture. An instruction generation module is configured to generate a servo pressure head control instruction based on the straightening parameter instruction set. The servo pressure head control instruction is used to instruct the servo pressure head to perform a straightening action according to the press-down amount, the pressure holding time, and the support span in the straightening parameter instruction set to obtain a composite straightened shaft part.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images