Multi-modal data deep fusion and cognitive intelligent system for manufacturing industry

Abstract

The present application relates to the field of information technology, in particular to a multi-modal data deep fusion and cognitive intelligent system for manufacturing industry, comprising: a data acquisition module: obtaining first type of physical field signals by deploying a distributed sensor network, and deploying second type of physical field signal acquisition devices in the same area; a signal preprocessing and mode activation module: preprocessing the first type of physical signals, generating an activation instruction after determining that there is an anomaly, and triggering the second type of physical field signal acquisition devices to enter the acquisition mode; a data fusion module: performing space-time calibration on the first type of physical field signals and the signals of the second type of physical field signal acquisition devices, and constructing a complete evidence chain; an abnormality cognition module: calculating the spatial distribution coincidence degree of the temperature abnormal area and the acoustic emission sound source positioning area, combining the surface deformation characteristics, and determining the structure damage according to a preset coefficient. The first type and the second type of sensors are deployed synchronously, the problem of single information blind area is solved, and the comprehensiveness of abnormal characteristics is ensured.

Description

Technical Field

[0001] This invention relates to the field of information technology, and more specifically to a multimodal data deep fusion and cognitive intelligence system for the manufacturing industry. Background Technology

[0002] In the field of equipment operation and maintenance in manufacturing, health status monitoring of key equipment such as CNC machine tools, heavy machinery, and pressure vessels is crucial for production safety and efficiency. Traditional monitoring technologies have significant limitations: they often rely on single physical field signals, making it difficult to comprehensively characterize equipment anomalies. For example, vibration anomalies may be caused by mechanical imbalance or external interference, and relying solely on vibration signals can easily lead to misjudgments; temperature anomalies may originate from environmental fluctuations, and without microscopic feature support, it is difficult to attribute the cause; high-precision monitoring equipment operates continuously, resulting in high energy consumption and significant data redundancy, especially during long-term stable operation, where resource waste is prominent. The asynchronous timestamps and inconsistent spatial coordinates of multi-source data make it impossible to establish precise correlations between anomaly times and anomaly regions, leading to fragmented cross-modal features and difficulty in forming an effective chain of evidence. Anomaly judgments often rely on single feature thresholds, lacking multi-dimensional verification, are easily affected by environmental interference, and have a high misjudgment rate. Summary of the Invention

[0003] This invention addresses the technical problems existing in the prior art by providing a multimodal data deep fusion and cognitive intelligence system for the manufacturing industry.

[0004] The technical solution of this invention to solve the above-mentioned technical problems is as follows: a multimodal data deep fusion and cognitive intelligence system for manufacturing, comprising:

[0005] Data acquisition module: Acquires first-type physical field signals by deploying a distributed sensor network, transmits them in real time via an industrial bus, and deploys second-type physical field signal acquisition devices in the same area. The second-type physical field signal acquisition devices switch working states through activation commands.

[0006] Signal preprocessing and mode activation module: preprocesses the first type of physical signal, compares it with a preset threshold through a hardware comparator, and generates an activation command after determining that there is an anomaly. The command triggers the second type of physical field signal acquisition device to enter the high-speed acquisition mode.

[0007] Data fusion module: performs spatiotemporal calibration on the signals from the first and second type of physical field signals acquisition devices, extracts the temperature anomaly region, the micro-deformation spectrum corresponding to the anomaly region, and the surface texture changes corresponding to the anomaly region, and constructs a complete chain of evidence;

[0008] Anomaly Recognition Module: Calculates the spatial overlap between the temperature anomaly area and the acoustic emission source location area, and determines structural damage based on the surface deformation characteristics and preset coefficients.

[0009] In a preferred embodiment, the data acquisition module acquires a first type of physical field signal in the equipment operating area. The acquisition of the first type of physical field signal captures the multi-dimensional physical characteristics of the key operating area of ​​the equipment through a distributed sensor network. The distributed sensor network includes vibration sensors, acoustic emission sensors, and electromagnetic field sensors. The first type of physical field signal is connected to the edge computing node through an industrial bus. The edge computing node acts as a master clock source and sends synchronization pulses to each sensor. The sensor embeds a timestamp when acquiring the signal.

[0010] The second type of physical field signal acquisition device is deployed in the key operating area covered by the first type of sensor to form spatial coverage overlap. The second type of physical field signal acquisition device includes an infrared thermal imager and a high-speed vision camera. The second type of physical field signal acquisition device is in a low-power standby state by default, with only the wake-up interface retained. The activation mechanism is implemented through trigger commands. When the first type of signal detects an anomaly, the edge computing node sends an activation command, and the device switches from standby to working state in a short time.

[0011] In a preferred embodiment, the signal preprocessing and modal activation module performs preprocessing operations on the first type of physical field signal through an edge computing node. The first type of physical field signal includes at least vibration signal, acoustic emission signal, and electromagnetic field signal. Specifically, it includes: using a filtering algorithm to remove power frequency interference of a specific frequency from the vibration signal, and calculating the peak value and root mean square value. The peak value reflects the instantaneous extreme value of vibration intensity, and the root mean square value reflects the average level of vibration energy. It also applies a wavelet denoising algorithm to suppress environmental noise in the acoustic emission signal, extracts the number of acoustic emission events per unit time and the energy carried by the stress wave. The number of acoustic emission events reflects the frequency of deformation activity, and the energy reflects the severity of deformation. Finally, it calculates the difference between the real-time amplitude of the electromagnetic field signal and the normal operating condition baseline. The normal operating condition baseline is calibrated based on the equipment factory test data and historical stable operation records. The amplitude deviation directly reflects the degree of distortion of the electromagnetic field.

[0012] The first preset threshold is calibrated based on the equipment's factory standards and historical normal operating data. The specific judgment logic is as follows:

[0013] The preprocessed feature parameters are compared with the corresponding first preset threshold in real time using a hardware comparator.

[0014] When any feature parameter exceeds its associated threshold, the device status is determined to be potentially abnormal;

[0015] When a potential anomaly is detected, the edge computing node generates a modal activation instruction set, which includes the activation device ID, sampling parameters, and time window information. The modal activation instruction set is transmitted to the second type of physical field signal acquisition device via industrial Ethernet, triggering the device to enter the acquisition mode. The infrared thermal imager starts continuous temperature measurement, focuses on the spatial area of ​​the first type of physical field signal anomaly, and the high-speed vision camera starts local area capture, simultaneously recording the visual features of the anomaly area.

[0016] In a preferred embodiment, the spatiotemporal calibration of the data fusion module uses the abnormal moment of the first type of signal as the reference time point, and adjusts the acquisition timestamp of the second type of physical field signal acquisition device through a timestamp correction algorithm so that the time difference between the two is controlled within the error range.

[0017] By establishing coordinate mapping through the 3D model of the equipment, the installation position of the vibration sensor, the pixel area of ​​the infrared thermal imager, and the positioning results of the acoustic emission signal are unified to the equipment coordinate system. The 3D model of the equipment is constructed using 3D modeling tools, and the installation position of the vibration sensor is directly mapped to the physical coordinates in the model. The pixel area of ​​the infrared thermal imager is converted into physical coordinates through camera intrinsic and extrinsic parameter calibration. The positioning results of the acoustic emission signal are calculated using a multi-sensor array triangulation algorithm and mapped to the physical coordinates of the 3D model. Temperature anomaly areas, the micro-deformation spectrum corresponding to the anomaly areas, and the surface texture changes corresponding to the anomaly areas are extracted to construct a complete chain of evidence and achieve accurate identification of equipment structural damage.

[0018] Temperature anomaly areas are identified by calculating a temperature baseline based on historical infrared thermal imaging data under normal operating conditions. This baseline includes the mean and standard deviation of the normal temperature, reflecting the average temperature level of that area on the equipment surface under normal operating conditions. This serves as the benchmark for determining temperature anomalies. The standard deviation reflects the temperature fluctuation range during normal operation. Temperature gradient distribution is calculated from real-time acquired infrared thermal imaging data to locate areas of abrupt temperature gradient changes. Let the real-time acquired infrared thermal imaging data be a two-dimensional temperature matrix T(x,y), where (x,y) represents the image pixel coordinates, x represents the horizontal pixel index, and y represents the vertical pixel index. The temperature gradient is the rate of change of temperature in space, containing both horizontal and vertical components. The x-direction temperature gradient component T is calculated using the Sobel operator convolution. x The calculation of (x,y) uses a 3×3 Sobel convolution kernel G in the x-direction. x Convolution with the temperature matrix, the specific calculation formula is as follows:

[0019]

[0020] Among them, T x(x,y) represents the temperature gradient component of pixel (x,y) in the x-direction in the infrared thermal image, and T(x+i,y+j) represents the temperature value of pixel (x+i,y+j) in the infrared thermal image, where T represents the temperature, (x+i,y+j) represents the coordinates of the neighboring pixels covered by the convolution kernel centered at the current pixel (x,y), i and j represent the relative offset, and T represents the temperature gradient component in the y-direction. y The calculation of (x,y) uses a 3×3 Sobel convolution kernel G in the y-direction. y Convolution with the temperature matrix, the specific calculation formula is as follows:

[0021]

[0022] Among them, T y (x,y) represents the temperature gradient component of pixel (x,y) in the y direction in an infrared thermal image. The temperature value in the region is compared with the normal baseline, and continuous regions with temperatures exceeding twice the historical temperature standard deviation of the normal mean are selected as temperature anomaly regions.

[0023] Based on time alignment, the acoustic emission signal synchronous with the temperature anomaly region is extracted from the micro-deformation spectrum corresponding to the abnormal region. The high-frequency signal components are extracted by bandpass filtering algorithm. The selection of high-frequency band is based on the mechanical characteristics of materials. During the crack initiation and propagation process of metallic materials, stress waves in this frequency band are generated due to crystal fracture and friction. Environmental noise is mostly concentrated in the lower frequency band. Micro-deformation characteristics can be effectively preserved by frequency band screening. The energy distribution of this frequency band is calculated by short-time Fourier transform. The frequency points and time points where the energy is concentrated are the active characteristics of micro-deformation.

[0024] For abnormal areas, the surface texture changes are identified by extracting continuous image sequences from high-speed vision cameras after spatial alignment. Image registration algorithms are used to eliminate image offsets caused by normal device movement. Texture analysis operators are used to extract local texture features from the images and compare them with historical normal texture templates to identify dynamic changes in the surface. These dynamic changes are directly related to material deformation and can intuitively reflect the external manifestations of structural damage.

[0025] In a preferred embodiment, the spatial distribution overlap degree in the anomaly recognition module adopts the intersection-union ratio algorithm to quantify the overlap ratio between the temperature anomaly region and the acoustic emission source localization region. Let the temperature anomaly region be set A and the acoustic emission source localization region be set B, then the specific calculation formula is as follows:

[0026]

[0027] Wherein, contact represents the degree of overlap, |A∩B| represents the overlapping area between the temperature anomaly region and the acoustic emission source region, and |A∪B| represents the total coverage area between the temperature anomaly region and the acoustic emission source region. The value of the degree of overlap ranges from [0,1]. A value close to 1 indicates a high degree of spatial consistency between the two regions.

[0028] A preset overlap coefficient α is set. If contact→α and the high-speed vision camera observes a change in surface texture, it is determined to be structural damage, and a level 3 warning is generated. If contact≤α and only a single mode is abnormal, it is determined to be an interference signal, and a level 1 warning is generated. The decision result is pushed to the equipment management system through the industrial configuration unit to ensure that maintenance personnel can obtain the warning information in a timely manner and respond quickly to structural damage.

[0029] The beneficial effects of this invention are as follows: This invention simultaneously deploys first and second type sensors to characterize the device status from multiple dimensions, including macroscopic motion, microscopic deformation, thermal distribution, and surface morphology. The first type of sensor provides real-time monitoring and clues to anomalies, while the second type of device is activated on demand to supplement fine features, solving the problem of blind spots in single-modal information and ensuring the comprehensiveness of anomaly features. The second type of device is triggered based on the threshold judgment of the first type of signal, and high-speed acquisition is only started when there is a potential anomaly, while maintaining low-power standby under normal conditions. This mechanism reduces ineffective energy consumption and data volume, while ensuring that fine features are not lost at the moment of anomaly, balancing monitoring accuracy and economy. Spatiotemporal consistency provides a basis for cross-modal feature correlation, avoiding the breakage of the evidence chain due to misalignment. Multi-dimensional verification is achieved by quantifying spatial overlap and combining it with preset coefficients. Attached Figure Description

[0030] Figure 1 This is a flowchart of the present invention;

[0031] Figure 2 This is a logic block diagram of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0034] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0035] like Figure 1 This embodiment provides a multimodal data deep fusion and cognitive intelligence system for the manufacturing industry, including:

[0036] Data acquisition module: Acquires first-type physical field signals by deploying a distributed sensor network, transmits them in real time via an industrial bus, and deploys second-type physical field signal acquisition devices in the same area. The second-type physical field signal acquisition devices switch working states through activation commands.

[0037] In this embodiment, the data acquisition module needs to be specifically described. The data acquisition module acquires the first type of physical field signal in the equipment operating area. The acquisition of the first type of physical field signal captures the multi-dimensional physical characteristics of the key operating areas of the equipment (such as bearing housing, weld seam, transmission mechanism) through a distributed sensor network. The distributed sensor network includes vibration sensors, acoustic emission sensors and electromagnetic field sensors. The first type of physical field signal is connected to the edge computing node through an industrial bus. The edge computing node acts as the master clock source and sends synchronization pulses to each sensor. The sensor embeds a timestamp when acquiring the signal.

[0038] The second type of physical field signal acquisition equipment is deployed in the key operating area covered by the first type of sensor to form spatial coverage overlap. The second type of physical field signal acquisition equipment includes infrared thermal imagers and high-speed vision cameras. The second type of physical field signal acquisition equipment is in a low-power standby state by default, with only the wake-up interface retained. The activation mechanism is implemented through trigger commands. When the first type of signal detects an anomaly, the edge computing node sends an activation command, and the device switches from standby to working state in a short time.

[0039] It should be noted that the reason for spatially covering the critical operating area covered by the second type of physical field signal acquisition equipment and the first type of sensor is that when the equipment fails, such as when the bearing overheats, it will simultaneously exhibit abnormal vibration (first type of physical field signal), temperature rise, and surface deformation. Spatially overlapping deployment can ensure that multiple features in the same abnormal area are monitored simultaneously, avoiding feature fragmentation caused by misalignment of monitoring areas. During deployment, the field of view is calibrated through the three-dimensional model of the equipment. Based on the sensor installation position, the focal length of the infrared thermal imager and the shooting angle of the high-speed vision camera are adjusted so that the monitoring center of both coincides with the sensitive area of ​​the sensor.

[0040] Signal preprocessing and mode activation module: preprocesses the first type of physical signal, compares it with a preset threshold through a hardware comparator, and generates an activation command after determining that there is an anomaly. The command triggers the second type of physical field signal acquisition device to enter the high-speed acquisition mode.

[0041] In this embodiment, the signal preprocessing and modal activation module needs to be specifically described. The signal preprocessing and modal activation module performs preprocessing operations on the first type of physical field signal through edge computing nodes. The first type of physical field signal includes at least vibration signal, acoustic emission signal and electromagnetic field signal. Specifically, it includes: using a filtering algorithm to remove power frequency interference of a specific frequency in the vibration signal, and calculating the peak value and root mean square value. The peak value reflects the instantaneous extreme value of vibration intensity, and the root mean square value reflects the average level of vibration energy. The wavelet denoising algorithm is applied to suppress environmental noise in the acoustic emission signal. The number of acoustic emission events per unit time and the energy carried by the stress wave are extracted. The number of acoustic emission events reflects the frequency of deformation activity, and the energy reflects the severity of deformation. The difference between the real-time amplitude of the electromagnetic field signal and the normal operating condition baseline is calculated. The normal operating condition baseline is calibrated based on the equipment factory test data and historical stable operation records. The amplitude deviation directly reflects the degree of distortion of the electromagnetic field.

[0042] The first preset threshold is calibrated based on the equipment's factory standards and historical normal operating data. The specific judgment logic is as follows:

[0043] The preprocessed feature parameters are compared with the corresponding first preset threshold in real time using a hardware comparator.

[0044] When any feature parameter exceeds its associated threshold, the device status is determined to be potentially abnormal;

[0045] When a potential anomaly is detected, the edge computing node generates a modal activation instruction set, which includes the activation device ID, sampling parameters, and time window information. The modal activation instruction set is transmitted to the second type of physical field signal acquisition device via industrial Ethernet, triggering the device to enter the acquisition mode. The infrared thermal imager starts continuous temperature measurement, focuses on the spatial area of ​​the first type of physical field signal anomaly, and the high-speed vision camera starts local area capture, simultaneously recording the visual features of the anomaly area.

[0046] It should be noted that the infrared thermal imager focuses on the abnormal area to obtain temperature distribution characteristics, supplementing the thermal information that the first type of physical field signal cannot cover. After activation, the infrared thermal imager starts the continuous temperature measurement mode and sets a specific sampling interval. This interval is calibrated based on the time scale of temperature change: the temperature rise caused by equipment abnormality usually has a second-level evolution process. The specific interval can completely record the temperature change trend from normal to abnormal, avoiding the loss of key temperature points due to excessively long intervals and the data redundancy due to excessively short intervals. The core of the high-speed vision camera's local capture is to record the dynamic morphological characteristics of the abnormal area, supplementing the visual evidence that the first type of physical field signal cannot provide. After activation, the high-speed vision camera adjusts the frame rate to a specific value. This frame rate is calibrated based on the movement speed of small deformations. The speed of small deformations on the equipment surface, such as crack propagation and component loosening, is usually on the order of millimeters per second. The specific frame rate can ensure that each millimeter displacement is captured by at least multiple frames of images. The capture area is limited to the local area of ​​the first type of physical signal abnormality. High-definition imaging of the local area is achieved by adjusting the lens focal length, which can clearly identify small-scale surface defects, such as scratches and dents.

[0047] The high-speed acquisition mode of the second type of physical field signal acquisition device has high power consumption (such as the cooling module of the infrared thermal imager and the high frame rate imaging of the visual camera, both of which require continuous power supply). If it runs all the time, it will significantly increase the power consumption of the device and the pressure on data storage. The activation logic only starts the second type of physical field signal acquisition device when the first type of physical field signal is determined to be potentially abnormal, so that it dynamically switches between low power standby and high-speed acquisition states. In standby, the main functions are turned off (power consumption is reduced to a low proportion of the working state), and only the abnormal area is focused during acquisition, which greatly reduces the amount of invalid power consumption and data.

[0048] Data fusion module: performs spatiotemporal calibration on the signals from the first and second type of physical field signals acquisition devices, extracts the temperature anomaly region, the micro-deformation spectrum corresponding to the anomaly region, and the surface texture changes corresponding to the anomaly region, and constructs a complete chain of evidence;

[0049] In this embodiment, the data fusion module needs to be specifically explained. The spatiotemporal calibration of the data fusion module uses the abnormal moment of the first type of signal as the reference time point, and adjusts the acquisition timestamp of the second type of physical field signal acquisition device through the timestamp correction algorithm so that the time difference between the two is controlled within the error range.

[0050] By establishing coordinate mapping through the 3D model of the equipment, the installation position of the vibration sensor, the pixel area of ​​the infrared thermal imager, and the positioning results of the acoustic emission signal are unified to the equipment coordinate system. The 3D model of the equipment is constructed using 3D modeling tools, and the installation position of the vibration sensor is directly mapped to the physical coordinates in the model. The pixel area of ​​the infrared thermal imager is converted into physical coordinates through camera intrinsic and extrinsic parameter calibration. The positioning results of the acoustic emission signal are calculated using a multi-sensor array triangulation algorithm and mapped to the physical coordinates of the 3D model. Temperature anomaly areas, the micro-deformation spectrum corresponding to the anomaly areas, and the surface texture changes corresponding to the anomaly areas are extracted to construct a complete chain of evidence and achieve accurate identification of equipment structural damage.

[0051] Temperature baselines for abnormal temperature areas are calculated based on infrared thermal imaging data from historical normal operating conditions. Under historical normal operating conditions, the temperature sampling sequence for the equipment surface area is T. hist ={T1,T2,...,T M}, where M represents the number of samples, and the temperature baseline includes the mean and standard deviation of the normal temperature. The specific formula for calculating the mean of the normal temperature is as follows:

[0052]

[0053] Among them, T i Represents the temperature sequence T hist The i-th temperature sample value in the sequence, M represents the temperature sampling sequence T. hist The sample size, μ, represents the mean normal temperature, reflecting the average temperature level of this area on the equipment surface under normal operating conditions, and serves as the benchmark value for judging temperature anomalies. The specific formula for calculating the standard deviation is as follows:

[0054]

[0055] Where σ represents the standard deviation, which reflects the temperature fluctuation range during normal operation. The temperature gradient distribution is calculated from the real-time acquired infrared thermal image data to locate regions of abrupt temperature gradient changes. Let the real-time acquired infrared thermal image data be a two-dimensional temperature matrix T(x,y), where (x,y) represents the image pixel coordinates, x represents the horizontal pixel index, and y represents the vertical pixel index. The temperature gradient is the rate of change of temperature in space, containing both horizontal and vertical components. The x-direction temperature gradient component T is calculated using the Sobel operator convolution. xThe calculation of (x,y) (reflecting the rate of temperature change in the horizontal direction) uses a 3×3 Sobel convolution kernel G in the x-direction. x Convolution with the temperature matrix, the specific calculation formula is as follows:

[0056]

[0057] Among them, T x (x,y) represents the temperature gradient component of pixel (x,y) in the x-direction in the infrared thermal image, and T(x+i,y+j) represents the temperature value of pixel (x+i,y+j) in the infrared thermal image, where T represents the temperature, (x+i,y+j) represents the coordinates of the neighboring pixels covered by the convolution kernel centered at the current pixel (x,y), i and j represent the relative offset, and T represents the temperature gradient component in the y-direction. y The calculation of (x, y) (reflecting the rate of temperature change in the vertical direction) uses a 3×3 Sobel convolution kernel G in the y-direction. y Convolution with the temperature matrix, the specific calculation formula is as follows:

[0058]

[0059] Among them, T y (x,y) represents the temperature gradient component of pixel (x,y) in the y direction in an infrared thermal image. The temperature value in the region is compared with the normal baseline, and continuous regions with temperatures exceeding twice the historical temperature standard deviation of the normal mean are selected as temperature anomaly regions.

[0060] Based on time alignment, the acoustic emission signal s(t) synchronous with the acquisition time of the temperature anomaly region is extracted from the micro-deformation spectrum corresponding to the anomaly region. The high-frequency signal component s is then extracted using a bandpass filtering algorithm. fil (τ), the selection of high-frequency bands is based on the mechanical characteristics of materials. During the crack initiation and propagation process of metallic materials, stress waves in this frequency band are generated due to crystal fracture and friction. Environmental noise (such as low-frequency vibrations from equipment operation) is mostly concentrated in lower frequency bands. By selecting frequency bands, micro-deformation characteristics can be effectively preserved. The energy distribution of this frequency band is calculated using short-time Fourier transform. The frequency points and time points where energy is concentrated are the active characteristics of micro-deformation. The specific calculation formula of short-time Fourier transform is as follows:

[0061]

[0062] Where STFT(t,f) represents the short-time Fourier transform result, a function of time t and frequency f, reflecting the energy distribution of the acoustic emission signal at different time and frequency points, and ω(τ-t) represents the sliding time window function, where τ represents the integration variable, e -j2πfτdt' represents a complex exponential function used to convert a signal from the time domain to the frequency domain, e represents the natural constant, j represents the imaginary unit, π represents pi, f represents the frequency variable, t' represents the time variable, and dt' represents the differentiation of the time variable t'.

[0063] For the abnormal areas corresponding to the surface texture changes, after spatial alignment (which coincide with temperature anomaly areas and sound source areas), continuous image sequences captured by high-speed vision cameras are extracted. Image registration algorithms are used to eliminate image offset caused by normal device movement. Texture analysis operators are used to extract local texture features of the images (such as gray value distribution and edge contours) and compare them with historical normal texture templates to identify dynamic changes in the surface (such as increased scratch length and changes in indentation depth). Dynamic changes are directly related to material deformation (such as surface protrusions caused by crack propagation) and can intuitively reflect the external manifestations of structural damage.

[0064] It should be noted that the error range is set based on the time evolution scale of abnormal features. The key feature changes of equipment abnormalities usually occur within a millisecond time window. The error range can ensure that the correspondence between vibration peaks, acoustic emission events, temperature rise, and surface deformation on the time axis is not disrupted, avoiding misjudgment of feature correlation due to time misalignment. The continuous area of ​​twice the historical temperature standard deviation of the normal mean is used as the temperature abnormal area because the threshold of twice the historical temperature standard deviation is based on statistical laws. Under normal operating conditions, the probability of the temperature exceeding this threshold is extremely low, which can effectively distinguish between normal fluctuations and abnormal heating.

[0065] Anomaly Recognition Module: Calculates the spatial overlap between the temperature anomaly area and the acoustic emission source location area, and determines structural damage based on the surface deformation characteristics and preset coefficients.

[0066] In this embodiment, the anomaly recognition module needs to be specifically explained. The spatial distribution overlap degree in the anomaly recognition module adopts the intersection-union ratio algorithm to quantify the overlap ratio between the temperature anomaly region and the acoustic emission source localization region. Let the temperature anomaly region be set A and the acoustic emission source localization region be set B, then the specific calculation formula is as follows:

[0067]

[0068] Wherein, contact represents the degree of overlap, |A∩B| represents the overlapping area between the temperature anomaly region and the acoustic emission source region, and |A∪B| represents the total coverage area between the temperature anomaly region and the acoustic emission source region. The value of the degree of overlap ranges from [0,1]. A value close to 1 indicates a high degree of spatial consistency between the two regions.

[0069] A preset overlap coefficient α is set. If contact→α and the high-speed vision camera observes a change in surface texture, it is determined to be structural damage, and a level 3 warning is generated. If contact≤α and only a single mode (temperature or acoustic emission) is abnormal, it is determined to be an interference signal, and a level 1 warning is generated. The decision result is pushed to the equipment management system through the industrial configuration unit to ensure that maintenance personnel can obtain warning information in a timely manner (such as a level 3 warning triggering a shutdown command, and a level 1 warning prompting key monitoring) to achieve a rapid response to structural damage.

[0070] It should be noted that the preset overlap coefficient is set by analyzing the overlap distribution between temperature anomaly areas and acoustic emission areas in a large number of known loss cases to determine the critical value that can effectively distinguish between damage and interference, and this coefficient can be dynamically adjusted according to the equipment material and structural type.

[0071] The acoustic emission source localization area is obtained by extracting the acoustic emission signal corresponding to the temperature anomaly area based on spatial alignment, and extracting the characteristic frequency band of energy concentration through spectrum analysis (such as the acoustic emission signal of crack propagation in metal materials is mostly concentrated in a specific high frequency band). The selection of this frequency band is based on the material physical properties. Different structural damages (such as cracks and plastic deformation) will generate stress waves of specific frequencies, while the frequency characteristics of environmental interference (such as external vibration) are significantly different from these. By identifying the energy concentration phenomenon in this frequency band, it can be determined whether the temperature anomaly is caused by microscopic deformation inside the material.

[0072] The micro-deformation spectrum corresponding to the acoustic emission source localization region and the anomalous region has an inclusive relationship. The acoustic emission source localization region is a broad set of acoustic emission signal features, while the micro-deformation spectrum corresponding to the anomalous region is the core spectral feature used to characterize the essence of micro-deformation within the acoustic emission localization region. The acoustic emission source localization region refers to all extractable features contained in the acoustic emission signal collected within the spatially aligned temperature anomaly region, covering multiple dimensions such as the time domain and frequency domain. For example, time domain features include event count rate (the number of acoustic emission events per unit time) and peak signal amplitude, while frequency domain features include energy... The distribution of quantities, dominant frequency, etc., serve to describe the overall characteristics of acoustic emission signals in the region from multiple dimensions, and serve as the basis for judging whether there is abnormal activity. The micro-deformation spectrum corresponding to the abnormal region is the core frequency domain feature extracted from the acoustic emission signal of the above-mentioned acoustic emission source location region for the specific physical process of micro-deformation. Specifically, it is manifested as the characteristic frequency band of energy concentration (such as the high frequency band corresponding to the propagation of metal cracks). It is the spectral performance directly related to the changes in the internal microstructure of the material (such as crack initiation and propagation) and is specifically used to reflect the mechanical nature of micro-deformation (such as the frequency characteristics of stress waves).

[0073] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0079] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multimodal data deep fusion and cognitive intelligence system for manufacturing, characterized in that: include: Data acquisition module: Acquires first-type physical field signals by deploying a distributed sensor network, transmits them in real time via an industrial bus, and deploys second-type physical field signal acquisition devices in the same area. The second-type physical field signal acquisition devices switch working states through activation commands. Signal preprocessing and mode activation module: preprocesses the first type of physical field signal, compares it with a preset threshold through a hardware comparator, and generates an activation command after determining that there is an anomaly. The command triggers the second type of physical field signal acquisition device to enter the high-speed acquisition mode. Data fusion module: performs spatiotemporal calibration on the signals from the first and second type of physical field signals acquisition devices, extracts the temperature anomaly region, the micro-deformation spectrum corresponding to the anomaly region, and the surface texture changes corresponding to the anomaly region, and constructs a complete chain of evidence; Anomaly Recognition Module: Calculates the spatial overlap between the temperature anomaly area and the acoustic emission source location area, and determines structural damage based on the surface deformation characteristics and preset coefficients. The data fusion module establishes coordinate mapping through the device's 3D model, unifying the installation location of the vibration sensor, the pixel area of ​​the infrared thermal imager, and the positioning results of the acoustic emission signal into the device coordinate system. It constructs a 3D model of the device using 3D modeling tools, directly mapping the installation location of the vibration sensor to the physical coordinates in the model. The pixel area of ​​the infrared thermal imager is converted into physical coordinates through camera intrinsic and extrinsic parameter calibration. The positioning results of the acoustic emission signal are calculated using a multi-sensor array triangulation algorithm and mapped to the physical coordinates of the 3D model. It extracts temperature anomaly areas, the micro-deformation spectrum corresponding to the anomaly areas, and the surface texture changes corresponding to the anomaly areas, constructing a complete chain of evidence. Temperature anomaly areas are determined based on infrared thermographic data from historical normal operating conditions to calculate a temperature baseline. This baseline includes the mean and standard deviation of the normal temperature, reflecting the average temperature level of the area on the equipment surface under normal operating conditions. This baseline serves as the benchmark for determining temperature anomalies. The standard deviation reflects the temperature fluctuation range during normal operation. The temperature gradient distribution is calculated from real-time acquired infrared thermographic data to locate areas of abrupt temperature gradient changes. For the micro-deformation spectrum corresponding to the anomaly area, acoustic emission signals synchronized with the acquisition time of the temperature anomaly area are extracted based on time alignment. High-frequency components are extracted using a bandpass filtering algorithm. The selection of high-frequency bands is based on material mechanics characteristics. During crack initiation and propagation in metallic materials, stress waves in this frequency band are generated due to crystal fracture and friction. Environmental noise is mostly concentrated in lower frequency bands. Frequency band selection effectively preserves micro-deformation characteristics. Short-time Fourier transform is used to calculate the energy distribution of this frequency band. The frequency points and time points where energy is concentrated represent the active characteristics of micro-deformation.

2. The multimodal data deep fusion and cognitive intelligence system for manufacturing as described in claim 1, characterized in that, The data acquisition module acquires the first type of physical field signal in the equipment operating area. The acquisition of the first type of physical field signal captures the multi-dimensional physical characteristics of the key operating area of ​​the equipment through a distributed sensor network. The distributed sensor network includes vibration sensors, acoustic emission sensors and electromagnetic field sensors. The first type of physical field signal is connected to the edge computing node through an industrial bus. The edge computing node acts as the master clock source and sends synchronization pulses to each sensor. The sensor embeds a timestamp when acquiring the signal. The second type of physical field signal acquisition device is deployed in the key operating area covered by the first type of sensor to form spatial coverage overlap. The second type of physical field signal acquisition device includes an infrared thermal imager and a high-speed vision camera. The second type of physical field signal acquisition device is in a low-power standby state by default, with only the wake-up interface retained. The activation mechanism is implemented through trigger commands. When the first type of signal detects an anomaly, the edge computing node sends an activation command, and the device switches from standby to working state in a short time.

3. The multimodal data deep fusion and cognitive intelligence system for manufacturing as described in claim 1, characterized in that, The signal preprocessing and modal activation module performs preprocessing operations on the first type of physical field signal through edge computing nodes. The first type of physical field signal includes at least vibration signal, acoustic emission signal and electromagnetic field signal. Specifically, it includes: using a filtering algorithm to remove power frequency interference of a specific frequency from the vibration signal, and calculating the peak value and root mean square value. The peak value reflects the instantaneous extreme value of vibration intensity, and the root mean square value reflects the average level of vibration energy. It applies a wavelet denoising algorithm to suppress environmental noise in the acoustic emission signal, extracts the number of acoustic emission events per unit time and the energy carried by the stress wave. The number of acoustic emission events reflects the frequency of deformation activity, and the energy reflects the severity of deformation. It calculates the difference between the real-time amplitude of the electromagnetic field signal and the normal operating condition baseline. The normal operating condition baseline is calibrated based on the equipment factory test data and historical stable operation records. The amplitude deviation directly reflects the degree of distortion of the electromagnetic field. The first preset threshold is calibrated based on the equipment's factory standards and historical normal operating data. The specific judgment logic is as follows: The preprocessed feature parameters are compared with the corresponding first preset threshold in real time using a hardware comparator. When any feature parameter exceeds its associated threshold, the device status is determined to be potentially abnormal.

4. The multimodal data deep fusion and cognitive intelligence system for manufacturing as described in claim 3, characterized in that, When a potential anomaly is detected, the edge computing node generates a modal activation instruction set, which includes the activation device ID, sampling parameters, and time window information. The modal activation instruction set is transmitted to the second type of physical field signal acquisition device via industrial Ethernet, triggering the device to enter the acquisition mode. The infrared thermal imager starts continuous temperature measurement, focuses on the spatial area of ​​the first type of physical field signal anomaly, and the high-speed vision camera starts local area capture, simultaneously recording the visual features of the anomaly area.

5. The multimodal data deep fusion and cognitive intelligence system for manufacturing as described in claim 1, characterized in that, The spatiotemporal calibration of the data fusion module uses the abnormal moment of the first type of signal as the reference time point, and adjusts the acquisition timestamp of the second type of physical field signal acquisition device through the timestamp correction algorithm to keep the time difference between the two within the error range.

6. The multimodal data deep fusion and cognitive intelligence system for manufacturing as described in claim 5, characterized in that, Assume the real-time acquired infrared thermal image data is a two-dimensional temperature matrix. ,in, The image pixel coordinates are represented by x, where x represents the horizontal pixel index and y represents the vertical pixel index. The temperature gradient is the rate of change of temperature in space, containing both horizontal and vertical components. It is calculated using the Sobel operator convolution, with the x-axis temperature gradient component being the [missing information]. The calculation uses a 3×3 Sobel convolution kernel in the x-direction. Convolution with the temperature matrix, the specific calculation formula is as follows: ; in, This represents the temperature gradient component of a pixel (x, y) in the x-direction in an infrared thermal image. This represents the temperature value of a pixel in an infrared thermal image, where T represents temperature. This represents the coordinates of the neighboring pixels covered by the convolution kernel, centered at the current pixel (x, y). i and j represent the relative offsets, and the temperature gradient component in the y-direction is also represented. The calculation uses a 3×3 Sobel convolution kernel in the y-direction. Convolution with the temperature matrix, the specific calculation formula is as follows: ; in, This represents the temperature gradient component of a pixel (x, y) in the y direction in an infrared thermal image. The temperature values ​​within the region are compared with the normal baseline, and continuous regions whose temperatures exceed twice the historical temperature standard deviation of the normal average are selected as temperature anomaly regions.

7. The multimodal data deep fusion and cognitive intelligence system for manufacturing as described in claim 5, characterized in that, For abnormal areas, the surface texture changes are identified by capturing a continuous image sequence from a high-speed vision camera after spatial alignment. The image registration algorithm is used to eliminate image offset caused by normal device movement. The texture analysis operator is used to extract local texture features of the image and compare them with historical normal texture templates to identify dynamic changes in the surface. These dynamic changes are directly related to material deformation.

8. The multimodal data deep fusion and cognitive intelligence system for manufacturing as described in claim 1, characterized in that, In the anomaly recognition module, the spatial distribution overlap degree adopts the intersection-union ratio algorithm to quantify the overlap ratio between the temperature anomaly region and the acoustic emission source localization region. Let the temperature anomaly region be set A and the acoustic emission source localization region be set B. The specific calculation formula is as follows: ; in, Indicates the degree of overlap. This represents the area of ​​overlap between the temperature anomaly region and the acoustic emission source location region. This represents the total coverage area of ​​the temperature anomaly region and the acoustic emission source location region. The overlap value ranges from [0,1]. An overlap value close to 1 indicates a high degree of spatial consistency between the two regions.

9. The multimodal data deep fusion and cognitive intelligence system for manufacturing as described in claim 8, characterized in that, Set the preset overlap coefficient ,like Furthermore, if the high-speed vision camera observes changes in surface texture, it determines that this is structural damage and generates a level three warning. If an anomaly occurs in only a single mode, it is determined to be an interference signal, generating a level one warning. The decision result is then pushed to the equipment management system through the industrial configuration unit, ensuring that maintenance personnel can obtain warning information in a timely manner and respond quickly to structural damage.

Images