A method and system for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion

Abstract

This invention discloses a method and system for workpiece in-situ state recognition based on multi-source heterogeneous data fusion, belonging to the field of sensor information fusion and visual recognition. It includes: acquiring image streams and point cloud streams to generate geometrically invariant vectors; acquiring fixture commands and current waveforms to construct driving invariant vectors; mapping the vectors to a high-dimensional space and generating temporal correlation embedding vectors by calculating the distance between positive and negative sample pairs; updating classification parameters based on feedback to output state confidence and calculating the distance residual relative to the reference vector to output rule verification labels; determining weighting coefficients based on the confidence entropy value and the historical matching rate of the verification labels; and performing multi-channel signal fusion to output the results. This invention employs a dual-path intelligent decision-making method that integrates visual geometric features and fixture driving features, combined with machine learning and rule verification, which can improve the accuracy and robustness of workpiece in-situ state recognition.

Description

Technical Field

[0001] This invention relates to the field of sensor information fusion and visual recognition, and in particular to a method and system for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion. Background Technology

[0002] In intelligent manufacturing and automated machining, workpiece in-situ identification is a crucial step in ensuring production safety and processing quality. Before CNC machine tool processing, automated assembly, or welding operations, it is essential to confirm that the workpiece is accurately placed and reliably fixed in its designated position, i.e., in its in-situ state. If the workpiece is not correctly positioned or the clamping fails before starting the operation, it can easily lead to machining defects, tool damage, or even equipment failure.

[0003] In related technologies, Chinese utility model patent with announcement number CN119762444A discloses an artificial intelligence-based method and system for detecting the condition of automotive fasteners, including: a machine vision module, an infrared thermal imaging module, an acoustic analysis module, a multimodal data fusion processing module, and a decision output module.

[0004] Regarding the aforementioned technologies, the inventors believe that in complex industrial environments filled with oil, iron filings, and coolant, contact-based detection methods are prone to sensor failure due to mechanical wear or foreign object jamming, and are also difficult to detect minute tilts of workpieces. On the other hand, single-vision detection methods are easily affected by changes in ambient lighting, cutting fluid splashes, or obstructions from robotic arms, leading to false alarms or missed alarms in the identification results. The robustness of the detection is poor, making it difficult to meet the requirements of high-precision and high-reliability automated production. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method and system for identifying the in-situ state of a workpiece by fusing multi-source heterogeneous data. It employs a dual-path intelligent decision-making method that integrates visual geometric features and fixture-driven features, combined with machine learning and rule verification, thereby improving the accuracy and robustness of workpiece in-situ state identification.

[0006] The above objectives can be achieved through the following approach:

[0007] A method for workpiece in-situ status recognition through multi-source heterogeneous data fusion includes: acquiring image streams and point cloud streams via a workstation vision sensor; extracting the geometric relationships of image corner points to generate visual micro-motion feature vectors; statistically analyzing the spatial distribution of the point cloud to generate three-dimensional structural feature vectors; and concatenating a geometric invariant vector based on the visual micro-motion feature vectors and the three-dimensional structural feature vectors; acquiring position command sequences and current waveforms via a fixture controller; calculating the position command transition interval to generate a timing beat feature vector; performing frequency domain integration on the current waveform to generate a driving energy feature vector; combining the timing beat feature vector and the driving energy feature vector to construct a driving invariant vector; and mapping the geometric invariant vector and the driving invariant vector to a high-dimensional feature space. Data augmentation is performed on feature vectors within the same time window to construct positive sample pairs, and random sampling is performed on feature vectors within different time windows to construct negative sample pairs. The distance between sample pairs is calculated to generate a temporal correlation embedding vector. Based on feedback data from the next process, the classification boundary parameters of the temporal correlation embedding vector are updated using gradient descent to output state confidence. The distance residual between the geometric invariant vector and the preset benchmark vector is calculated to output a rule verification label. The uncertainty entropy value of the state confidence and the historical matching rate of the rule verification label are statistically analyzed. Based on the uncertainty entropy value and the historical matching rate, a weighting coefficient is determined using a sliding time window. Multi-channel signal fusion is performed to output the in-situ state determination result.

[0008] Optionally, the step of concatenating the visual micro-motion feature vector and the three-dimensional structural feature vector to form a geometric invariant vector includes: acquiring a continuous frame RGB image sequence using a workstation vision sensor as an image stream, and simultaneously acquiring a sparse point cloud sequence using a depth camera as a point cloud stream; detecting a set of corner points in the image stream, calculating the Euclidean distance ratio sequence and the angle difference sequence between any two corner points, and performing variance statistics to generate a visual micro-motion feature vector; performing voxelization meshing on the point cloud stream, statistically analyzing the point cloud density distribution histogram and the histogram of the angle between the average normal vector within each voxel, generating a three-dimensional structural feature vector, and concatenating the visual micro-motion feature vector and the three-dimensional structural feature vector to output a geometric invariant vector.

[0009] Optionally, the step of constructing a driving invariant vector by combining the timing beat feature vector and the driving energy feature vector includes: using a fixture controller to collect the I / O port level state sequence as a bit instruction sequence, and simultaneously using a servo driver to collect the three-phase current instantaneous value sequence as a current waveform; identifying the rising and falling edges in the bit instruction sequence, calculating the time difference between adjacent edges, and statistically analyzing the time difference distribution histogram to generate a timing beat feature vector; performing a Fourier transform on the current waveform to obtain the power spectral density, performing a definite integral calculation on the power spectral density to generate a driving energy feature vector, and performing a vector combination of the timing beat feature vector and the driving energy feature vector to output a driving invariant vector.

[0010] Optionally, generating the timing beat feature vector includes: extracting time coordinates and arranging them in timing order based on the signal level transition points of the bit instruction sequence, calculating the absolute difference between adjacent time coordinates, and generating a process action beat sequence; calculating the logarithm of the total number of elements based on the total number of elements in the process action beat sequence to determine the number of feature bins; performing equal-width discretization statistics on the process action beat sequence based on the number of bins, and calculating the frequency proportion within each bin interval to generate the timing beat feature vector.

[0011] Optionally, the step of calculating the distance between sample pairs to generate a temporally correlated embedding vector includes: concatenating the geometric invariant vector and the driving invariant vector to obtain an original fusion vector; performing element-wise squaring and product cross operations on the original fusion vector; concatenating the result with the original fusion vector to construct a high-dimensional extended vector; performing amplitude linear scaling on the high-dimensional extended vector to generate a homologous enhancement vector; pairing the high-dimensional extended vector with the homologous enhancement vector to construct a positive sample pair; performing a sign flipping operation on the high-dimensional extended vector to generate an inverse feature vector; pairing the high-dimensional extended vector with the inverse feature vector to construct a negative sample pair; deriving the Euclidean distance using the positive sample pair and the negative sample pair; and multiplying the Euclidean distance element-wise with the high-dimensional extended vector to output a temporally correlated embedding vector.

[0012] Optionally, the method further includes: calculating the Pearson correlation coefficient between the distribution features based on the three-dimensional structural feature vector and the temporal beat feature vector; performing absolute value operation and moving average smoothing on the Pearson correlation coefficient to generate a smoothed correlation intensity sequence.

[0013] Optionally, calculating the distance residual between the geometric invariant vector and a preset reference vector to output a rule verification label includes: receiving a pass / fail marker and a feedback timestamp from the feedback data of the next process; mapping the pass / fail marker to a supervision label; mapping the feedback timestamp to a time window index and binding it to the temporal correlation embedding vector; constructing classification boundary parameters for in-situ and out-of-situ states; performing gradient descent updates on the classification boundary parameters based on the supervision label; calculating the Euclidean distance between the temporal correlation embedding vector and the classification boundary parameters; normalizing to obtain the state confidence; calculating the distance residual from the geometric invariant vector to the reference vector; using the distance residual to divide the upper quartile threshold; using the smoothed correlation strength sequence to perform inverse proportional correction on the upper quartile threshold to generate a dynamic discrimination threshold; and determining whether the distance residual exceeds the dynamic discrimination threshold to generate a rule verification label.

[0014] Optionally, binding the time-series correlation embedding vector includes: parsing the communication message of the feedback data from the next process to extract a Boolean qualified status bit and a long integer feedback time value; performing numerical quantization conversion on the Boolean qualified status bit, assigning a positive value when the status bit is true and a negative value when the status bit is false, and generating a supervision label; calculating the ratio of the long integer feedback time value to the sliding time window step size and performing a rounding operation to generate a time window index; locating the time-series correlation embedding vector based on the time window index; and binding the supervision label to the time-series correlation embedding vector.

[0015] Optionally, the step of performing multi-channel signal fusion to output an in-situ state determination result includes: calculating the probability distribution based on the state confidence level to obtain an uncertainty entropy value; statistically analyzing the proportion of time points where the label converted from the state confidence level matches the rule verification label as a historical matching rate; performing numerical normalization calculation on the reciprocal of the uncertainty entropy value and the historical matching rate to generate a weighting coefficient; and performing an optimal selection between the state confidence level and the rule verification label based on the weighting coefficient to complete the multi-channel signal fusion and output an in-situ state determination result.

[0016] Based on the same inventive concept, this invention also provides a workpiece in-situ status recognition system based on multi-source heterogeneous data fusion. The system includes: a visual point cloud feature module, used to acquire image streams and point cloud streams through a workstation visual sensor, extract the geometric relationships of image corner points to generate visual micro-motion feature vectors, statistically analyze the spatial distribution of the point cloud to generate three-dimensional structural feature vectors, and concatenate a geometric invariant vector based on the visual micro-motion feature vectors and the three-dimensional structural feature vectors; a fixture drive feature module, used to acquire bit command sequences and current waveforms through a fixture controller, calculate the bit command transition interval to generate a timing beat feature vector, perform frequency domain integration on the current waveform to generate a drive energy feature vector, and combine the timing beat feature vector and the drive energy feature vector to construct a drive invariant vector; and a fusion embedding generation module, used to combine the geometric invariant vector with the... The system maps the driving invariant vector to a high-dimensional feature space, performs data augmentation on feature vectors within the same time window to construct positive sample pairs, performs random sampling on feature vectors within different time windows to construct negative sample pairs, and calculates the distance between sample pairs to generate a temporal correlation embedding vector. A feedback boundary update module updates the classification boundary parameters of the temporal correlation embedding vector using gradient descent based on feedback data from the next process to output state confidence, and calculates the distance residual between the geometric invariant vector and a preset benchmark vector to output a rule verification label. An entropy matching and fusion module calculates the historical matching rate between the uncertainty entropy value of the state confidence and the rule verification label, determines the weighting coefficient using a sliding time window based on the uncertainty entropy value and the historical matching rate, and performs multi-channel signal fusion to output the in-situ state determination result.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] 1. By fusing two-dimensional images and three-dimensional point cloud data, a comprehensive set of geometrically invariant features was constructed, capable of simultaneously capturing minute dynamic changes of the workpiece in a plane and its static structural contours in space. This combination of multimodal visual information overcomes the perception limitations of a single visual sensor when faced with complex lighting, occlusion, or changes in workpiece posture, making the geometric description of the workpiece's in-situ state more accurate and robust, and improving the accuracy of recognition.

[0019] 2. The timing and energy characteristics of the fixture drive system are introduced as auxiliary judgment criteria to correlate the physical state of the workpiece with the dynamic behavior of the actuator. By mining the cycle pattern in the bit instruction sequence and the energy consumption characteristics in the current waveform, the changes in equipment load caused by improper workpiece placement or abnormal posture are perceived from a new dimension. This enables the effective identification of hidden anomalies that are difficult to observe directly with visual information, and enhances the overall system's identification capability.

[0020] 3. Through nonlinear feature expansion and self-referential contrast enhancement, the original features after multi-source fusion are transformed into a deeper representation that is more sensitive to time changes. This process not only improves the discriminative power of the features but also provides higher-quality input for subsequent supervised learning, enabling the classification model to learn more effectively the subtle differences between qualified and unqualified states, further optimizing recognition performance.

[0021] Other features and advantages of the invention will be set forth in the following description. The objects and other advantages of the invention can be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion, according to an embodiment of the present invention.

[0024] Figure 2 This is a comparison diagram of the distribution of multi-source features under different workpiece states according to an embodiment of the present invention.

[0025] Figure 3 This is a time-series comparison diagram of the geometric distance residual and the dynamic discrimination threshold in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of a workpiece in-situ status identification system based on multi-source heterogeneous data fusion according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Reference Figure 1One embodiment of the present invention proposes a workpiece in-situ state recognition method based on multi-source heterogeneous data fusion. It adopts a dual-path intelligent decision-making method that integrates visual geometric features and fixture driving features, and combines machine learning and rule verification, which can improve the accuracy and robustness of workpiece in-situ state recognition.

[0029] The method in this embodiment specifically includes:

[0030] Image streams and point cloud streams are acquired by workstation vision sensors. Geometric relationships of image corner points are extracted to generate visual micro-motion feature vectors. The spatial distribution of point cloud is statistically analyzed to generate three-dimensional structural feature vectors. Geometric invariant vectors are then concatenated based on visual micro-motion feature vectors and three-dimensional structural feature vectors.

[0031] The fixture controller acquires the bit instruction sequence and current waveform, calculates the bit instruction transition interval to generate a timing beat feature vector, performs frequency domain integration on the current waveform to generate a driving energy feature vector, and combines the timing beat feature vector and the driving energy feature vector to construct a driving invariant vector.

[0032] Geometric invariant vectors and driving invariant vectors are mapped to a high-dimensional feature space. Data augmentation is performed on feature vectors within the same time window to construct positive sample pairs. Random sampling is performed on feature vectors within different time windows to construct negative sample pairs. The distance between sample pairs is calculated to generate temporally related embedding vectors.

[0033] Based on the feedback data from the next process, the classification boundary parameters of the temporal correlation embedding vector are updated by gradient descent to output the state confidence, and the distance residual between the geometric invariant vector and the preset benchmark vector is calculated to output the rule verification label.

[0034] The uncertainty entropy value of the statistical state confidence is compared with the historical matching rate of the rule verification label. The weighting coefficient is determined by using a sliding time window based on the uncertainty entropy value and the historical matching rate. Multi-channel signal fusion is then performed to output the in-situ state determination result.

[0035] Optionally, the geometric invariant vector concatenated from the visual micro-motion feature vector and the three-dimensional structural feature vector includes:

[0036] A continuous frame RGB image sequence is acquired using a workstation vision sensor as an image stream, and a sparse point cloud sequence is acquired simultaneously using a depth camera as a point cloud stream.

[0037] The RGB camera and depth camera are synchronously controlled via hardware triggering. The RGB camera captures two-dimensional color information reflecting the surface texture of the workpiece, while the depth camera captures three-dimensional depth information reflecting the spatial geometry of the workpiece. To ensure data alignment, each frame of the RGB image and its corresponding point cloud frame are assigned a timestamp from the same clock. Sparse point cloud refers to a set of three-dimensional points that has undergone downsampling processing. Its sampling density is set based on the minimum feature size of the workpiece to reduce computational load while preserving contour features.

[0038] For example, in an engine block inspection station, a vision sensor array operates at a frame rate of 30 fps. When the workpiece arrival signal is triggered, an RGB camera acquires an image sequence with a resolution of 1920×1080, while a ToF depth camera acquires a point cloud sequence containing approximately 30,000 coordinate points. The timestamp deviation between the two is controlled within 5 milliseconds, forming a strictly corresponding image stream and point cloud stream.

[0039] Detect a set of corner points in an image stream, calculate the Euclidean distance ratio sequence between any two corner points and the difference sequence of the angle between the connecting lines, and perform variance statistics to generate a visual micro-motion feature vector.

[0040] In each frame of the image stream, a set of corner points is extracted using a corner detection algorithm, such as the FAST algorithm. Any two corner points are selected to form a point pair, and the ratio of the Euclidean distance between the current frame and the previous frame, as well as the difference in the angle between the connecting lines, are calculated. To quantify this subtle change, the variance of the ratio sequence within the sliding time window is calculated using the following formula:

[0041] ,

[0042] in, This represents the calculated visual micro-motion feature components, used to characterize the stability of the point's geometric relationship; This indicates the number of image frames included in the sliding time window. The time is set according to the workstation cycle time, typically covering 0.2 to 0.5 seconds of data. For example, taking... ; Indicates the first The Euclidean distance ratio of this pair of points in the frame image; This represents the arithmetic mean of all ratios within the time window. The smaller this variance statistic, the more visually stable the workpiece; conversely, a larger value indicates the presence of vibration or looseness.

[0043] For example, corner points A and B are detected. Over 10 consecutive frames, the distance ratio sequence between A and B is calculated as [1.00, 1.00, 1.01, 0.99, ...]. If the workpiece is well clamped, these ratios are extremely close to 1.0, and the calculated variance... Possibly If the workpiece is not clamped and moves slightly, the ratio fluctuation will increase, and the variance may rise to [a certain value]. The variance values ​​of all detected point pairs are combined into a vector, which is the visual micro-motion feature vector.

[0044] Voxelization meshing is performed on the point cloud flow. The histogram of the point cloud density distribution and the histogram of the angle between the average normal vector and the point cloud density distribution in each voxel are statistically analyzed to generate a three-dimensional structural feature vector. The visual micro-motion feature vector and the three-dimensional structural feature vector are concatenated and spliced ​​to output a geometric invariant vector.

[0045] First, construct a cube space enclosing the point cloud, and then cut it into sections with sides of length [missing information]. tiny voxels, The value is set based on the workpiece machining accuracy. For each non-hollow voxel, the angle between the average normal vector and the reference axis is calculated using the following formula:

[0046] ,

[0047] in, The average normal angle of a voxel represents the spatial orientation of a local surface. This represents the average vector of all point cloud normals within a voxel; This represents the reference axis vector, whose value is the direction of gravitational acceleration on the bottom plane of the tooling fixture. Subsequently, the values ​​of all voxels are statistically analyzed. The distribution forms a histogram, which is then merged with the point cloud density histogram to generate a 3D structural feature vector. Finally, the visual micro-motion feature vector and the 3D structural feature vector are concatenated along the channel dimension to form the final geometrically invariant vector.

[0048] For example, the workpiece is a metal block with an inclined plane. Its point cloud space is divided into... The mesh. For voxels in the slope region, the calculated included angle. The angles are concentrated around 45 degrees; for voxels on the top plane, the included angles are concentrated around 0 degrees. The number of these angles falling into different intervals is counted, forming an 18-dimensional histogram vector. This vector is concatenated with the previously calculated 50-dimensional visual variance vector to finally output a 68-dimensional geometric invariant vector.

[0049] Optionally, constructing a driving invariant vector by combining the temporal tick feature vector and the driving energy feature vector includes:

[0050] The fixture controller is used to collect the IO port level state sequence as a bit instruction sequence, and the servo driver is used to collect the three-phase current instantaneous value sequence as a current waveform.

[0051] During the data acquisition phase, the fixture controller PLC and servo drive are accessed in parallel via an industrial fieldbus interface. For bit instruction sequences, a fixed low-frequency sampling period is used. For example, 10ms is used to read the logic state of the digital output port controlling the fixture's movement, forming a discrete time series. For current waveforms, a high-frequency sampling period is used. For example, 0.1ms corresponds to 10kHz. The instantaneous values ​​of the U, V, and W phase currents of the servo motor are read, typically selecting the phase with the heaviest load or calculating the magnitude of the composite current vector. The key is "synchronization," using a unified hardware clock to trigger the acquisition, ensuring that each bit command state corresponds to a precise current waveform data.

[0052] For example, in a hydraulic clamp controller, the clamp controller controls the opening and closing of a solenoid valve. A data acquisition card records the on / off state sequence of the solenoid valve at a frequency of 50 Hz. Simultaneously, the motor current driving the hydraulic pump is acquired at a frequency of 2kHz via the analog monitoring port of the servo driver. The data streams of both are strictly aligned on the time axis, with the error controlled within one instruction sampling cycle.

[0053] Identify the rising and falling edges in the bit instruction sequence, calculate the time difference between adjacent edges, and statistically analyze the histogram of the time difference distribution to generate a timing beat feature vector.

[0054] First, traverse the bit instruction sequence. Detect state transition points. Define time. For rising edge if and only if and Define the time. For the falling edge if and only if and Record all identified transition moments as an ordered set. Next, the time difference between adjacent edges is calculated. Finally, construct the distribution histogram. (Settings...) Divide the data into time intervals, i.e., bins, and count the number of items falling into each interval. The quantity is normalized, and the calculation formula is as follows:

[0055] ,

[0056] in, It is the first of the temporal beat feature vectors The nth component represents the nth component. The frequency percentage of each time interval; It represents the total number of detected edges; It is a conditional function; it takes the value 1 if the condition is met, and 0 otherwise. and They are the first The lower and upper bounds of each container. Container boundaries. It is based on the cycle time of the standard process. For example, if the standard clamping action takes 2 seconds, then the high-density compartmenting can be set between 1.8 seconds and 2.2 seconds.

[0057] For example, a complete clamping cycle consists of four stages: "rapid traverse - feed - hold pressure - release". Three key time intervals were identified: rapid traverse takes 0.5s, feed takes 1.2s, and hold pressure takes 5.0s. Ten time bins were set. Statistical analysis showed that a count of 1 corresponding to feed occurs within the [1.0s, 1.5s] interval, and a count of 1 corresponding to hold pressure occurs within the [4.0s, 6.0s] interval. The resulting timing cycle feature vector reflects the time structure of this process.

[0058] Perform a Fourier transform on the current waveform to obtain the power spectral density, perform a definite integral calculation on the power spectral density to generate the driving energy feature vector, and perform a vector combination of the timing beat feature vector and the driving energy feature vector to output the driving invariant vector.

[0059] Energy is driven by frequency domain quantization to eliminate time domain noise interference. First, a current waveform segment corresponding to the operating cycle is selected. Performing a Fast Fourier Transform (FFT) on it yields the spectrum. Next, the power spectral density (PSD) is calculated and integrated over the characteristic frequency band. The calculation formula is as follows:

[0060] ,

[0061] in, It is an element in the driving energy feature vector, representing the signal energy within a specific frequency band; It is the amplitude of the current signal's spectrum; This is the total number of sampling points; It is the sampling frequency; and These are the starting and ending frequencies of the integral. and The selection of frequency bands is determined based on the mechanical and load characteristics of the motor. For example, the low-frequency band (0-50Hz) reflects the main driving work, while the high-frequency band (above 500Hz) reflects mechanical friction or chatter. Finally, the calculated frequencies of multiple frequency bands are... The driving energy feature vector is composed of a driving energy feature vector and concatenated with the temporal beat feature vector to output the driving invariant vector.

[0062] For example, the current waveform of the servo motor fluctuates drastically at the moment of clamping the workpiece. An FFT analysis is performed on this waveform, and the energy integrals for three frequency bands are calculated: the low frequency (0-100Hz) represents the clamping torque maintenance energy, the mid-frequency (100-300Hz) represents the response of the mechanical transmission chain, and the high frequency (300-1000Hz) represents the vibration energy transmitted through tool cutting. Three values ​​are obtained from these calculations. This forms the driving energy feature vector. It is then compared with the previous time-series vector. By concatenating the components, we obtain a six-dimensional driving invariant vector. This vector serves as the basis for determining the workpiece's in-situ status.

[0063] Optionally, generating the temporal tick feature vector includes:

[0064] Based on the bit instruction sequence, scan the signal level transition points, extract the time coordinates and arrange them in time sequence, calculate the absolute value of the difference between adjacent time coordinates, and generate the process action cycle sequence.

[0065] In the timing analysis phase, edge detection is first performed on the acquired bit instruction sequence. The bit instruction sequence is a binary state stream that changes over time. The entire sequence is traversed, identifying all rising edges (level change from 0 to 1) and falling edges (level change from 1 to 0), and the timestamps of these transitions are extracted. These timestamps are stored in a one-dimensional array in chronological order to form a time coordinate set. Next, to obtain the duration of each sub-action in the process, the absolute difference between adjacent time coordinates is calculated. The calculation formula is as follows:

[0066] ,

[0067] in, This indicates the first step in the sequence of action beats in a process. The element represents the first element. The duration of a sub-action or waiting state; and These represent the first in the time coordinate set. The and the first A timestamp. Composed of all The resulting sequence is the cycle time sequence of the process actions. Its significance lies in converting absolute time points into relative action durations, thereby eliminating interference caused by different start times of the processes.

[0068] For example, in a typical pneumatic clamp control process, the acquired bit command sequence contains three transition points, corresponding to timestamps of: 100ms valve opening, 150ms cylinder positioning, and 650ms valve closing. The difference between adjacent times is calculated: Corresponding action response time This corresponds to the pressure holding and clamping time. The generated process action cycle sequence is as follows: .

[0069] Based on the total number of statistical elements in the process action beat sequence, the logarithm of the total number of elements is calculated to determine the number of feature bins. Based on the number of bins, equal-width discretization statistics are performed on the process action beat sequence, and the frequency ratio in each bin interval is calculated to generate a temporal beat feature vector.

[0070] To balance feature resolution and computational dimensionality, an adaptive binning strategy based on sequence length is adopted. First, the total number of elements in the process action beat sequence is counted. Next, the number of segmentations for the feature bins is calculated using a logarithmic function. The calculation formula is as follows:

[0071] ,

[0072] in, This represents the final number of feature bins, i.e., the vector dimension, and its value is an integer. It is the total number of elements in the sequence of process actions; It is the base of the logarithm; It is a correction constant, to prevent when When the number of boxes is small, the number of boxes is 0, which is usually set. For example, take ; This represents the rounding function. (The last part is incomplete and likely refers to a specific function or rule.) Next, perform equal-width discretization statistics to find the maximum value in the cycle time sequence of the process actions. and minimum value Calculate the width of each sub-box. The statistics fell into the first place. Each container section number of elements Calculation frequency percentage Ultimately, by The resulting vector is the temporal beat feature vector.

[0073] For example, suppose that in a complex machining cycle, the sequence of operation steps contains An action element. Set the logarithmic base. , corrected constant Then the number of boxes Assuming the minimum duration in this sequence is 0ms and the maximum duration is 300ms, then the bin width... The three compartments are respectively , , Statistics show that 20 actions fall into the short motion category (0-100ms), 50 actions fall into the medium motion category (100-200ms), and 30 actions fall into the long motion category (200-300ms). The percentages were calculated. , , The final generated temporal beat feature vector is: This vector not only has a fixed dimension, but also reflects the time structure characteristic of the process, which is dominated by "medium-duration actions".

[0074] Optionally, calculating the distance between sample pairs to generate temporally associated embedding vectors includes:

[0075] The geometric invariant vector and the driving invariant vector are concatenated and concatenated to obtain the original fused vector. The original fused vector is then subjected to element-wise squaring and product cross-operation. The result of the operation is concatenated with the original fused vector to construct a high-dimensional extended vector.

[0076] In the feature mapping stage, the geometric invariant vector output from the upstream is concatenated with the driving invariant vector to form an original fusion vector containing both static and dynamic information of the workpiece. To uncover the implicit nonlinear coupling relationships between features, such as whether visual micro-movements lead to a quadratic increase in the driving current, a second-order polynomial feature is explicitly constructed. This involves two core operations: calculating the square of each feature dimension in the original fusion vector to amplify the influence of large-amplitude features; and calculating the product between features of different dimensions in the original fusion vector. The final high-dimensional extended vector... The construction logic is shown in the following formula:

[0077] ,

[0078] in, It is the original fusion vector; This represents the new vector obtained by squaring all elements of the original vector. The cross term vector is formed by the product of all pairwise distinct elements in the original vector, i.e. This operation is equivalent to explicitly mapping the feature space to a high-dimensional second-order space in engineering, which can improve the boundary partitioning ability of linear classifiers.

[0079] For example, suppose the geometric invariant vector is The driving invariant vector is Original fusion vector Squared term vector Cross term vector The final concatenated high-dimensional extended vector for The dimensions have been expanded from 3 to 9.

[0080] Perform magnitude linear scaling on the high-dimensional extended vector to generate homologous enhancement vectors, and pair the high-dimensional extended vectors with the homologous enhancement vectors to construct positive sample pairs;

[0081] Simulate sensor noise or minor environmental disturbances to construct positive samples for measuring local stability. Generate a scaling factor. This is then applied to each element of the high-dimensional extended vector, and the calculation formula is as follows:

[0082] ,

[0083] in, Represents a homologous enhancement vector; This is the current high-dimensional extended vector; The amplitude linear scaling factor is in the interval Random numbers sampled uniformly within the range, The perturbation tolerance is typically set to 0.01 to 0.05, representing a fluctuation of 1% to 5%. The original high-dimensional extended vector is paired with this homologous augmentation vector, which has only a small amplitude difference; this is defined as a positive sample pair. In the feature space, these two vectors are essentially aligned and very close to each other.

[0084] Perform a sign-flipping operation on the high-dimensional extended vector to generate an inverse feature vector, and pair the high-dimensional extended vector with the inverse feature vector to construct a negative sample pair;

[0085] Multiply each element of the high-dimensional extended vector by -1 to generate the inverse eigenvector. Pairing the high-dimensional extended vector with the inverse eigenvector is defined as a negative sample pair. In engineering terms, the inverse eigenvector represents the "maximum difference" reference of the current state in the feature space. Although it may not correspond to actual working conditions physically, it provides a clear distant reference for distance measurement mathematically.

[0086] The Euclidean distance is obtained by using positive and negative sample pairs. The Euclidean distance is then multiplied element-wise with the high-dimensional extended vector to output the temporal correlation embedding vector.

[0087] The feature vectors are weighted and modulated using the distance information between sample pairs to generate the final embedding representation. First, the Euclidean distance within each positive sample pair is calculated. Euclidean distance between negative sample pairs The final temporal correlation embedding vector The calculation formula is as follows:

[0088] ,

[0089] in, It is the output temporal correlation embedding vector; It is the Euclidean distance between the high-dimensional extended vector and the homologous augmented vector. This value actually quantifies the absolute drift of the feature vector after being subjected to random scaling perturbation. This represents the L2 norm operation. This operation is equivalent to a "self-attention mechanism." If the magnitude of the current feature vector is large and the signal is strong, then the same scaling will result in a larger absolute distance. This assigns greater weight to the vector. This results in the generated embedding vector containing not only directional information but also implicit confidence information about the signal strength. For example... Figure 2 As shown, in-situ and out-of-situ samples exhibit significant distribution differences in feature dimensions such as visual micro-motion, three-dimensional structure, temporal beat, and driving energy, verifying the separability and effectiveness of multi-source feature fusion for state recognition.

[0090] For example, suppose the current high-dimensional expansion vector Simplified to Randomly select scaling factor Homologous enhancement vector Calculate the Euclidean distance between positive sample pairs. The final output is a temporal correlation embedding vector. As can be seen, this output vector not only encodes the features. The magnitude of this value is also directly related to the sensitivity of the feature to disturbances.

[0091] Optionally, the method further includes:

[0092] The Pearson correlation coefficient between distribution features is calculated based on the three-dimensional structural feature vector and the temporal beat feature vector.

[0093] From an engineering logic perspective, a specific workpiece geometry should induce a specific set of action cycles within a standard process flow. Therefore, the numerical distributions of these two should exhibit a certain statistical correlation. First, the 3D structural feature vector generated by the current time window is obtained. and temporal beat feature vector If the vector dimensions of the two vectors are inconsistent, a zero-padding strategy is used, padding the lower-dimensional vector with zeros at the end to make its dimension equal. Consistent with higher-dimensional vectors, the Pearson correlation coefficient between these two feature vectors is then calculated using the following formula:

[0094] ,

[0095] in, This represents the calculated Pearson correlation coefficient, with a value range of [value missing]. This is used to characterize the degree of linear correlation between spatial and temporal features; Indicates the dimension of the aligned feature vector; and These represent the first eigenvectors in the three-dimensional structural feature vector and the second eigenvector in the temporal beat feature vector, respectively. Each element value; and Let represent the arithmetic mean of all elements in these two vectors, respectively. A value approaching 1 indicates a high degree of matching between the "structure" and the "cycle time," signifying a stable process; if... A value close to 0 indicates a lack of correlation between the two, potentially indicating an inconsistent state where "the workpiece is correct but the action is abnormal" or "the action is normal but the workpiece is incorrect".

[0096] For example, suppose that in a certain detection, the aligned 3D structural feature vector for The main energy is concentrated in the first component, the temporal beat eigenvector. for The main energy is concentrated in the first component. The distribution patterns of the two are highly similar; substituting them into the formula yields... This indicates that the current workpiece structure perfectly matches the expected action cycle. Conversely, if... Become If the action is delayed, the correlation coefficient will decrease significantly, or even become negative.

[0097] The absolute value of the Pearson correlation coefficient is calculated and smoothed by moving average to generate a smoothed correlation strength sequence.

[0098] The aim is to eliminate statistical noise in a single measurement and generate an auxiliary discrimination signal that reflects the long-term process stability. First, the absolute value of the Pearson correlation coefficient is calculated to generate the instantaneous correlation strength. This step is to ignore the direction of the correlation, i.e., whether it is positive or negative, and only focus on the strength of the association. Next, a moving average filter is used to smooth the instantaneous correlation strength sequence; the calculation formula is as follows:

[0099] ,

[0100] in, Indicates the first The values ​​in the smooth correlation intensity sequence output at each time point are used to dynamically adjust the discrimination threshold; The length of the sliding window is determined based on the inertial time constant of the process. It is usually taken as the amount of data covering 3 to 5 workpiece detection cycles to ensure that the signal can both filter out sudden noise and respond to changes in the working condition trend in a timely manner. Indicates the current moment and the past. The instantaneous correlation strength calculated at each time point.

[0101] For example, five workpieces were processed consecutively, and the calculated Pearson correlation coefficient sequence was as follows: Note that the fourth data point, 0.4, indicates an abnormal fluctuation. The first step is to take the absolute value to obtain the instantaneous correlation strength sequence: The second step is to perform a moving average, assuming... For the third time step, the smoothing value For the 5th time step, the smoothing value... As can be seen, although a low value of 0.4 appeared at the fourth time step, after smoothing, the output sequence value changed relatively smoothly, avoiding drastic changes in the judgment logic due to a single anomaly.

[0102] Optionally, calculating the distance residual between the geometric invariant vector and a preset reference vector to output rule validation labels includes:

[0103] Receive the pass mark and feedback timestamp from the feedback data of the next process, map the pass mark to the supervision label, map the feedback timestamp to the time window index and bind the time-series association embedding vector;

[0104] To establish a link between "post-event feedback" and "pre-event prediction," feedback data packets from the next process stage, such as robot grasping or final inspection stations, are received. First, Boolean-type pass / fail markers are mapped to numerical supervision labels. If marked as "qualified", then If marked as "unqualified", then Secondly, in order to locate the corresponding feature vector, the feedback timestamp is used. Calculate the time window index The calculation logic is as follows:

[0105] ,

[0106] in, This is the start time; The step size of the sliding time window; The average transfer time of a workpiece from the current inspection station to the next process is a fixed value calculated by the speed of the production line conveyor belt and the distance between stations. This is the floor function. Based on this index... Find the corresponding temporal association embedding vector in the historical cache and add the supervision label. Write the attribute fields of the vector to complete the data binding.

[0107] For example, the vision station collects data at 10:00:00, with a step size of 1 second. It takes 5 seconds for the workpiece to reach the gripping robot. The robot reports "Grip failed" at 10:00:05. According to the formula... Locate the embedding vector generated in time window 0 and mark it as -1.

[0108] Construct classification boundary parameters for in-situ and out-of-situ states, perform gradient descent updates on the classification boundary parameters based on supervised labels, calculate the Euclidean distance between the temporal association embedding vector and the classification boundary parameters, and normalize to obtain the state confidence.

[0109] Two prototype vectors are constructed as classification boundary parameters, representing the prototype vector of "in-situ state". and the prototype vector representing the "absent state" Both have dimensions consistent with the temporally correlated embedding vector. When a new supervision label is received... Then, gradient descent is performed on these two prototype vectors to update them, with the update formula as follows:

[0110] ,

[0111] in, and These are the prototype vectors before and after the update, respectively. renew ,against renew ; The learning rate is usually set to a small value, such as 0.01, to ensure stability. This is the embedding vector for the currently bound temporal association. After the update is complete, the current vector is calculated. To the "in-situ prototype" Euclidean distance The state confidence is obtained by normalization using the Sigmoid function. :

[0112] ,

[0113] in, This is the distance from the vector to the "absent prototype". The confidence level approaches 1 as the vector gets closer to the "present prototype" and further away from the "absent prototype".

[0114] The distance residual from the geometric invariant vector to the reference vector is calculated. The distance residual is used to divide the upper quartile threshold. The upper quartile threshold is inversely corrected by the smoothed correlation strength sequence to generate a dynamic discrimination threshold. It is then determined whether the distance residual exceeds the dynamic discrimination threshold to generate a rule verification label.

[0115] First, calculate the distance residual. Call the baseline vector The reference vector is obtained by collecting the geometric invariant vectors of 100 standard qualified workpieces during the debugging phase and calculating their arithmetic mean. The calculation formula is:

[0116] ,

[0117] in, Let this be the current geometric invariant vector. Next, we statistically analyze the most recent... The distance residuals of each qualified sample are used to calculate the upper quartile (75th quartile) of their statistical distribution, which is denoted as the base threshold. Subsequently, the smoothed correlation strength sequence generated upstream was used. Perform an inverse correction on the base threshold to generate a dynamic discrimination threshold. The calculation formula is as follows:

[0118] ,

[0119] in, It is the dynamic threshold ultimately used for judgment; It is the current smooth correlation strength value, ranging from 0 to 1, representing the consistency between vision and action; It is a small constant to prevent the denominator from being zero, for example ; This is the adjustment coefficient, usually set to the historical average correlation strength value. When the value is close to 1, it indicates that the visual features and the movement rhythm are highly consistent, and the operating condition is stable. At this time, the denominator becomes larger, and the dynamic threshold increases. To tighten or shrink means to perform more stringent geometric checks under stable operating conditions; conversely, when... At lower thresholds, the threshold is relaxed to tolerate non-critical disturbances. Finally, if If the condition is met, a "Pass" tag will be generated; otherwise, a "Reject" tag will be generated. For example... Figure 3 As shown, the discrimination threshold adaptively tightens as the correlation strength of the operating condition increases and loosens as the correlation strength decreases, thereby ensuring effective interception of real anomalies while avoiding false alarms caused by environmental disturbances.

[0120] For example, the upper quartile threshold of the distance residual obtained from historical statistics. Current smoothing correlation strength Excellent operating conditions, adjustment coefficient Then the dynamic threshold The threshold tightens; if the residual is 1.9 at this point, although it is less than the historical value... However, if the value exceeds the dynamic threshold, an anomaly will be reported, and the rule validation will fail, effectively blocking potential risks. Current smoothing correlation strength. The operating conditions fluctuate greatly. Therefore, the dynamic threshold... The threshold has been relaxed to avoid false alarms caused by environmental vibrations.

[0121] Optionally, the binding time-series association embedding vector includes:

[0122] Analyze the communication messages of feedback data from the next process to extract the Boolean pass status bit and the long integer feedback time value;

[0123] Real-time analysis of feedback data from the next process is achieved by listening to and receiving communication messages from downstream inspection or assembly stations via the industrial Ethernet bus protocol. These messages follow a predefined format, such as Modbus TCP or OPC UA. Two key fields are extracted from the received messages: a Boolean pass / fail status bit representing the workpiece's quality status (typically generated directly by the downstream vision inspection system or sensor; a true value indicates pass, and a false value indicates fail); and a long integer feedback time value representing the moment the feedback was generated. This is a high-precision timestamp, such as a Unix timestamp in milliseconds or microseconds, marking the precise point in time when the quality judgment was made.

[0124] Perform numerical quantization conversion on the Boolean qualified status bit, assign a positive value of one when the status bit is true and a negative value of one when the status bit is false, and generate a supervision label;

[0125] The purpose of performing numerical quantization is to transform logical states into numerical supervision labels that can be directly processed by machine learning models. The extracted Boolean valid state bits are mapped: when a state bit is True, it is assigned a value of +1; when a state bit is False, it is assigned a value of -1. This symmetrical, zero-centered label encoding method achieves better convergence performance in many gradient descent optimization algorithms. After this step, each feedback message is converted into a concise supervision label carrying a positive or negative sign.

[0126] Calculate the ratio of the long integer feedback time value to the sliding time window step size and perform a rounding operation to generate a time window index. Locate the time-series association embedding vector based on the time window index and bind the supervision label to the time-series association embedding vector.

[0127] The calculation and location of the time window index aims to accurately find the time-series association embedding vector corresponding to the feedback artifact in the historical data buffer based on the feedback timestamp. Internally, a sliding time window is maintained, scrolling with a fixed time step. The time window index is generated through a simple mathematical operation, calculated as follows:

[0128] ,

[0129] in, This represents the calculated time window index; It is the value of the long integer feedback time obtained from prior analysis; It is the sliding time window step size; This represents the floor function. The formula discretizes the continuous feedback timestamps onto corresponding time window numbers. Finally, the calculated time window index is used... The search and location are performed within the data structure that stores historical time-series embedded vectors, precisely finding the index value. The temporal correlation embedding vector is generated. Once localization is successful, the supervision label is bound to this localized vector. In engineering, this binding involves creating a data pair containing the feature vector and its corresponding supervision label, and feeding this data pair into the subsequent classification boundary parameter update module as a valid training sample.

[0130] For example, register data from the PLC of the next process is read via the Modbus TCP protocol. Address 40001 stores the pass / fail status, and the read value is 1. Addresses 40002-40003 store the Unix timestamp, and the read value is 1700000005500. Sliding window step size. The time is 1000ms. The first step is to quantize the state; if the state is true, generate a supervision label. The second step is to calculate the index, which, according to the formula, is 1700000005. The third step is to perform binding, finding the time-series association embedding vector with index 1700000005 in the in-memory database and then... Bind to this vector.

[0131] Optionally, performing multi-channel signal fusion to output the in-situ status determination result includes:

[0132] The probability distribution based on the state confidence level is used to obtain the uncertainty entropy value. The percentage of time points where the labels after state confidence level conversion are the same as the labels in the rule verification is used as the historical matching rate.

[0133] By designing a dynamic, data-driven weighting mechanism, this study combines machine learning-based state confidence with physical rule-based rule validation labels. First, uncertainty analysis is performed on the state confidence, a value between 0 and 1, which is treated as a binary probability distribution. Based on this, its uncertainty entropy is calculated using the following formula:

[0134] ,

[0135] in, This represents the uncertainty entropy value; the closer the value is to 1, the more hesitant the model is, and the closer it is to 0, the more certain the model is. This represents the confidence level of the output state at the current moment. Secondly, historical consistency is evaluated. A database of length [length missing] is maintained in memory. First-in-first-out queue, storing the most recently... The system records the judgments for each period. For each historical moment, the state confidence score is binarized with a threshold of 0.5; a score greater than 0.5 is recorded as +1, otherwise as -1. This score is then compared with the rule verification label at that time. The number of times the two scores match is counted. Calculate the historical matching rate This metric reflects the degree of synergy between "data-driven" and "physical rules" over a period of time.

[0136] Numerical normalization is performed on the reciprocal of the uncertainty entropy and the historical matching rate to generate weighting coefficients;

[0137] The aim is to generate a comprehensive weighting to determine which signal to trust more. First, the reciprocal of the uncertainty entropy is calculated as a "model determinism index." To prevent the denominator from being zero, a small constant is usually added. Next, this deterministic indicator is mapped to the same dimension as the historical matching rate, and the final weighting coefficient is calculated. The calculation formula is as follows:

[0138] ,

[0139] in, This represents the generated weighting coefficients, with a value range of 1. ; It is the Sigmoid activation function, used to smoothly map the results of linear combinations to a probability interval; It is a deterministic index normalization constant, typically taking the value of . ; and These are weight hyperparameters, set based on debugging experience, and are typically set to... This indicates a greater emphasis on the certainty of the current model, while also taking into account historical consistency.

[0140] Based on the weighting coefficients, an optimal choice is made between the state confidence level and the rule verification label to complete the fusion of multiple signals and output the in-situ state determination result.

[0141] Set a fusion threshold For example, 0.7, if the calculated weighting coefficient This indicates that the machine learning model is currently highly certain and has historically been highly consistent with physical rules. In this case, the system determines the model is in a "high-confidence zone" and directly adopts the label transformed from state confidence as the final result. If... This indicates that the model may have encountered unseen abnormal operating conditions or conflicted with physical rules. In this case, it is determined to enter the "safety net," and the rule verification label based on rigid geometric constraints is forcibly adopted as the final result. In this way, while maintaining the level of intelligence, safety under extreme operating conditions is ensured.

[0142] For example, in scenario one, normal processing occurs, and the state confidence level is... Very certain, uncertainty entropy value Historical matching rate Calculated weighting coefficients It's close to 0.95. Because... The model outputs the "in-situ" result corresponding to the confidence level. In Scene 2, abnormal lighting causes visual interference from strong light; the model outputs a confidence level... Hesitation, indecisiveness, entropy Calculated weighting coefficients It dropped to 0.4. Because Ignore uncertain model results and instead check rule validation labels, such as whether geometric residuals exceed the limit, and use the rule results as the final judgment.

[0143] Based on the same inventive concept, this invention also provides a workpiece in-situ status identification system based on multi-source heterogeneous data fusion, such as... Figure 4 As shown, the system includes:

[0144] The visual point cloud feature module is used to acquire image streams and point cloud streams through workstation vision sensors, extract the geometric relationship of image corner points to generate visual micro-motion feature vectors, statistically analyze the spatial distribution of point clouds to generate three-dimensional structural feature vectors, and concatenate geometric invariant vectors based on visual micro-motion feature vectors and three-dimensional structural feature vectors.

[0145] The fixture drive feature module is used to acquire bit command sequence and current waveform through fixture controller, calculate bit command transition interval to generate timing beat feature vector, perform frequency domain integration on current waveform to generate drive energy feature vector, and combine timing beat feature vector and drive energy feature vector to construct drive invariant vector.

[0146] The fusion embedding generation module is used to map geometric invariant vectors and driving invariant vectors to a high-dimensional feature space, perform data augmentation on feature vectors within the same time window to construct positive sample pairs, perform random sampling on feature vectors within different time windows to construct negative sample pairs, and calculate the distance between sample pairs to generate temporally related embedding vectors.

[0147] The feedback boundary update module is used to update the classification boundary parameters of the time-series associated embedding vector through gradient descent based on the feedback data from the next process to output the state confidence, and to calculate the distance residual between the geometric invariant vector and the preset benchmark vector to output the rule verification label.

[0148] The entropy matching and fusion module is used to statistically analyze the uncertainty entropy value of the state confidence and the historical matching rate of the rule verification label. Based on the uncertainty entropy value and the historical matching rate, the weighting coefficient is determined using a sliding time window, and multi-channel signal fusion is performed to output the in-situ state determination result.

[0149] It should be noted that the functional division and information interaction between the various modules described above are logical, but in terms of physical implementation, they can be integrated on the same software platform or deployed in a distributed manner. The connections between them represent data flow and control flow, aiming to collaboratively achieve the objectives of this invention. The above descriptions are merely exemplary embodiments of this invention and should not be construed as limiting the scope of protection of this invention.

Claims

1. A method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion, characterized in that, The method includes: Image streams and point cloud streams are acquired by a workstation vision sensor. Geometric relationships of image corner points are extracted to generate visual micro-motion feature vectors. The spatial distribution of point cloud is statistically analyzed to generate three-dimensional structural feature vectors. Geometric invariant vectors are then concatenated based on the visual micro-motion feature vectors and the three-dimensional structural feature vectors. The fixture controller acquires the bit instruction sequence and current waveform, calculates the bit instruction transition interval to generate a timing beat feature vector, performs frequency domain integration on the current waveform to generate a driving energy feature vector, and combines the timing beat feature vector and the driving energy feature vector to construct a driving invariant vector. The geometric invariant vector and the driving invariant vector are mapped to a high-dimensional feature space. Data augmentation is performed on the feature vectors within the same time window to construct positive sample pairs. Random sampling is performed on the feature vectors within different time windows to construct negative sample pairs. The distance between the sample pairs is calculated to generate a temporally associated embedding vector. Based on the feedback data from the next process, the classification boundary parameters of the temporal correlation embedding vector are updated by gradient descent to output the state confidence, and the distance residual between the geometric invariant vector and the preset benchmark vector is calculated to output the rule verification label. The uncertainty entropy value of the state confidence is statistically analyzed and the historical matching rate of the rule verification label is calculated. Based on the uncertainty entropy value and the historical matching rate, a weighting coefficient is determined using a sliding time window, and multi-channel signal fusion is performed to output the in-situ state determination result.

2. The method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion according to claim 1, characterized in that, The geometric invariant vector concatenated from the visual micro-motion feature vector and the three-dimensional structural feature vector includes: A continuous frame RGB image sequence is acquired using a workstation vision sensor as an image stream, and a sparse point cloud sequence is acquired simultaneously using a depth camera as a point cloud stream. In the image stream, a set of corner points is detected, the Euclidean distance ratio sequence between any two corner points and the difference sequence of the angle between the connecting lines are calculated, and variance statistics are performed to generate a visual micro-motion feature vector. The point cloud flow is subjected to voxelization meshing, and the histogram of the point cloud density distribution and the angle between the average normal vector and the point cloud density distribution within each voxel are statistically analyzed to generate a three-dimensional structural feature vector. The visual micro-motion feature vector and the three-dimensional structural feature vector are concatenated and spliced ​​to output a geometric invariant vector.

3. The method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion according to claim 1, characterized in that, The step of constructing the driving invariant vector by combining the timing beat feature vector and the driving energy feature vector includes: The fixture controller is used to collect the IO port level state sequence as a bit instruction sequence, and the servo driver is used to collect the three-phase current instantaneous value sequence as a current waveform. Identify the rising and falling edges in the bit instruction sequence, calculate the time difference between adjacent edges, and statistically analyze the histogram of the time difference distribution to generate a timing beat feature vector. A Fourier transform is performed on the current waveform to obtain the power spectral density. A definite integral is performed on the power spectral density to generate a driving energy feature vector. The timing beat feature vector and the driving energy feature vector are then combined to output a driving invariant vector.

4. The method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion according to claim 3, characterized in that, The generated temporal beat feature vector includes: Based on the bit instruction sequence, scan the signal level transition points, extract the time coordinates and arrange them in time sequence, calculate the absolute value of the difference between adjacent time coordinates, and generate the process action rhythm sequence. Based on the total number of statistical elements in the process action beat sequence, the logarithm of the total number of elements is calculated to determine the number of feature bins. Based on the number of bins, equal-width discretization statistics are performed on the process action beat sequence, and the frequency ratio in each bin interval is calculated to generate a temporal beat feature vector.

5. The method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion according to claim 1, characterized in that, The calculation of sample pair distances to generate temporally associated embedding vectors includes: The geometric invariant vector and the driving invariant vector are concatenated and concatenated to obtain the original fusion vector. The original fusion vector is then subjected to element-wise squaring and product cross-operation. The result of the operation is concatenated with the original fusion vector to construct a high-dimensional extended vector. Perform magnitude linear scaling operation on the high-dimensional extended vector to generate homologous enhancement vectors, and pair the high-dimensional extended vectors with the homologous enhancement vectors to construct positive sample pairs; Perform a sign-flipping operation on the high-dimensional extended vector to generate an inverse feature vector, and pair the high-dimensional extended vector with the inverse feature vector to construct a negative sample pair; The Euclidean distance is obtained using the positive sample pair and the negative sample pair. The Euclidean distance is then multiplied element-wise with the high-dimensional extended vector to output the temporal correlation embedding vector.

6. The method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion according to claim 1, characterized in that, The method further includes: The Pearson correlation coefficient between the distribution features is calculated based on the three-dimensional structural feature vector and the temporal beat feature vector. The absolute value of the Pearson correlation coefficient is calculated and smoothed by moving average to generate a smoothed correlation strength sequence.

7. The method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion according to claim 6, characterized in that, Calculating the distance residual between the geometric invariant vector and a preset reference vector to output rule verification labels includes: Receive the pass mark and feedback timestamp from the feedback data of the next process, map the pass mark to the supervision label, map the feedback timestamp to the time window index and bind the time-series association embedding vector; Construct classification boundary parameters for in-situ and out-of-situ states, perform gradient descent updates on the classification boundary parameters based on the supervision labels, calculate the Euclidean distance between the temporal association embedding vector and the classification boundary parameters, and normalize to obtain the state confidence. Calculate the distance residual from the geometric invariant vector to the reference vector, use the distance residual to divide the upper quartile threshold, use the smoothed correlation strength sequence to perform an inverse proportional correction on the upper quartile threshold to generate a dynamic discrimination threshold, and determine whether the distance residual exceeds the dynamic discrimination threshold to generate a rule verification label.

8. The method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion according to claim 7, characterized in that, The binding of the time-related embedding vector includes: The communication messages of the feedback data from the next process are analyzed to extract the Boolean qualified status bit and the long integer feedback time value; Perform numerical quantization conversion on the Boolean qualified status bit, assigning a positive value of one when the status bit is true and a negative value of one when the status bit is false, and generate a supervision tag; The ratio of the long integer feedback time value to the sliding time window step size is calculated and rounded down to generate a time window index. The time-series correlation embedding vector is located based on the time window index, and the supervision label is bound to the time-series correlation embedding vector.

9. The method for identifying the in-situ status of a workpiece through multi-source heterogeneous data fusion according to claim 1, characterized in that, The process of performing multi-channel signal fusion to output an in-situ status determination result includes: The probability distribution based on the state confidence level is used to obtain the uncertainty entropy value. The percentage of time points where the labels after the state confidence level conversion are the same as the labels in the rule verification is used as the historical matching rate. Numerical normalization is performed on the reciprocal of the uncertainty entropy and the historical matching rate to generate weighting coefficients; Based on the weighting coefficients, an optimal selection is performed between the state confidence level and the rule verification label to complete the fusion of multiple signals and output the in-situ state determination result.

10. A workpiece in-situ status identification system based on multi-source heterogeneous data fusion, applied to the workpiece in-situ status identification method based on multi-source heterogeneous data fusion as described in any one of claims 1-9, characterized in that, The system includes: The visual point cloud feature module is used to acquire image streams and point cloud streams through workstation visual sensors, extract the geometric relationship of image corner points to generate visual micro-motion feature vectors, statistically analyze the spatial distribution of point clouds to generate three-dimensional structural feature vectors, and concatenate geometric invariant vectors based on the visual micro-motion feature vectors and the three-dimensional structural feature vectors. The fixture drive feature module is used to acquire bit command sequence and current waveform through fixture controller, calculate bit command transition interval to generate timing beat feature vector, perform frequency domain integration on current waveform to generate drive energy feature vector, and combine the timing beat feature vector and drive energy feature vector to construct drive invariant vector. The fusion embedding generation module is used to map the geometric invariant vector and the driving invariant vector to a high-dimensional feature space, perform data augmentation on feature vectors within the same time window to construct positive sample pairs, perform random sampling on feature vectors within different time windows to construct negative sample pairs, and calculate the distance between sample pairs to generate temporally associated embedding vectors. The feedback boundary update module is used to update the classification boundary parameters of the time-series associated embedding vector by gradient descent based on the feedback data of the next process to output the state confidence, and to calculate the distance residual between the geometric invariant vector and the preset benchmark vector to output the rule verification label. The entropy matching and fusion module is used to calculate the historical matching rate between the uncertainty entropy value of the state confidence and the rule verification label, determine the weighting coefficient using a sliding time window based on the uncertainty entropy value and the historical matching rate, and perform multi-channel signal fusion to output the in-situ state determination result.

Images