Multimodal resonance detection method and system adapted to different dielectric interface fit

Abstract

The present invention relates to the field of intelligent detection, and proposes a multimodal resonance detection method and system adapted to the interface fit of different media. By matching and setting initial knocking parameters for different material interfaces to adapt to different material properties and interface states, the adaptability to different material interfaces is improved. A multi-point automatic knocking rule is designed to adaptively generate knocking trajectories for different knocking surface curvatures and different component shapes, and multiple different knocking areas are divided for multi-point knocking to avoid the influence of structural blind spots, thereby improving the adaptability and accuracy of detection. The multimodal data is then feature enhanced through a spatiotemporal synchronous attention algorithm to synchronously match the features of the multimodal data in time and space orientations, thereby eliminating interference from complex combinations or composite methods of materials, and also eliminating noise interference from the detection environment, thereby improving the accuracy of overall recognition. The present invention improves the accuracy and adaptability of different medium interface fit detection.

Description

Technical Field

[0001] The present invention relates to the field of intelligent detection, and in particular to a multi-modal resonance detection method and system adapted to different medium interface fits. Background Art

[0002] With the rapid development of the automobile industry, vehicle inspection has gradually become an indispensable part of the production and maintenance process. Vehicle inspection often involves combining components or composite components of different materials. The wide variety of materials and the different shapes of components pose a great challenge to the accuracy of vehicle inspection.

[0003] In the prior art, vehicle inspection methods based on image vision such as infrared thermal imaging are usually used. However, when facing the interface fit detection of different media, the vehicle inspection methods based on image vision have poor recognition sensitivity and accuracy for hollowing, local poor fitting and complete detachment, resulting in the inability to eliminate defects. When using vehicle inspection methods based on ultrasonic technology, it is difficult to cope with the complex shapes of different components and the complex combination or composite methods of different material media, resulting in poor accuracy in identifying interface fit defects. At the same time, the existing vehicle inspection methods based on image vision and ultrasonic technology have the problems of high cost and complex operation, resulting in insufficient applicability and robustness of the inspection methods.

[0004] Therefore, how to design a detection method that is suitable for the interface fit of different media to improve the accuracy and applicability of the detection has become an urgent problem to be solved. Summary of the invention

[0005] Based on this, the present invention provides a multimodal resonance detection method and system that adapts to the fit of different medium interfaces. By matching and setting initial tapping parameters for different material interfaces to adapt to different material properties and interface states, the adaptability to different material interfaces is improved, while ensuring that the signal frequency range of the generated and collected multimodal data is within the target stable range, avoiding distortion or interference signals due to excessively high or low tapping detection energy. A multi-point automatic tapping rule is designed to adaptively generate tapping trajectories for different tapping surface curvatures and different component shapes, and at the same time divide a plurality of different tapping areas for multi-point tapping to avoid the influence of structural blind spots, thereby improving the adaptability and accuracy of detection. The multimodal data is then feature enhanced through a spatiotemporal synchronous attention algorithm to synchronously match the features of the multimodal data in time and space orientations, thereby eliminating interference from complex combinations or composite methods of materials, as well as noise interference from the detection environment, thereby improving the accuracy of overall recognition. The present invention improves the accuracy and adaptability of detection of the fit of different medium interfaces.

[0006] The present invention provides a multi-modal resonance detection method adapted to different dielectric interface adhesions, comprising:

[0007] Obtaining material properties of the target to be measured to match the initial knocking parameters, performing knocking detection on the target to be measured according to multi-point automatic knocking rules, and performing self-calibration according to the initial knocking parameters, wherein the target to be measured is two different media, and the knocking detection is applied to the fitting interface of the two different media;

[0008] Collecting multimodal data of the knock detection and preprocessing them respectively;

[0009] Extracting multimodal fusion features of the multimodal data according to a multimodal convolutional neural network, and then performing feature enhancement processing on the multimodal fusion features according to a spatiotemporal synchronous attention algorithm to obtain multimodal enhanced features;

[0010] Defect recognition is performed based on the multimodal enhanced features to mark the defective area.

[0011] In summary, according to the above-mentioned multimodal resonance detection method adapted to the fit of different medium interfaces, by matching and setting the initial tapping parameters for different material interfaces to adapt to different material properties and interface states, the adaptability to different material interfaces is improved, while ensuring that the signal frequency range of the generated and collected multimodal data is in the target stable range, avoiding distortion or interference signals due to excessively high or low tapping detection energy, and designing a multi-point automatic tapping rule to adaptively generate tapping trajectories for different tapping surface curvatures and different component shapes, while dividing multiple different tapping areas for multi-point tapping to avoid the influence of structural blind spots and improve the adaptability and accuracy of detection, and then enhancing the features of the multimodal data through a spatiotemporal synchronous attention algorithm to synchronously match the features of the multimodal data in time and space orientations, eliminating the interference of complex combinations or composite methods of materials, and also eliminating the noise interference of the detection environment, thereby improving the accuracy of overall recognition. The present invention improves the accuracy and adaptability of different medium interface fit detection. Specifically, the material properties of the target to be measured are obtained to match the initial knocking parameters, the target to be measured is knocked according to the multi-point automatic knocking rules, and self-calibration is performed according to the initial knocking parameters. The target to be measured is two different media, and the knocking detection is applied to the fitting interface of the two different media. For different material interfaces, the initial knocking parameters are matched and set to adapt to different material properties and interface states, so as to improve the adaptability to different material interfaces. The multimodal data of the knocking detection are collected and pre-processed respectively, the interference noise and interference vibration in the actual detection environment are removed, and the multimodal signals in the target range are screened. At the same time, the detection of knocking to generate a target signal is enhanced, and the multimodal fusion features of the multimodal data are extracted according to the multimodal convolutional neural network. The multimodal fusion features are then feature enhanced according to the spatiotemporal synchronous attention algorithm to obtain multimodal enhanced features, and the multimodal data features are synchronously matched in time and space orientations. The frequency offset features of the acoustic signal and the energy attenuation features of the vibration signal in the multimodal signal are jointly extracted to improve the accuracy of the overall recognition. Defect identification is performed according to the multimodal enhanced features to mark the defective area. The present invention improves the accuracy and adaptability of interface fit detection of different media.

[0012] Furthermore, the step of obtaining the material properties of the target to be measured to match the initial knocking parameters specifically includes:

[0013] Acquire a material combination of the target to be measured, and retrieve an acoustic impedance ratio parameter and an interface stiffness parameter matching the material combination according to a preset material property database;

[0014] Calculating and generating initial knocking parameters according to the acoustic impedance ratio parameter and the interface stiffness parameter;

[0015] The initial knocking parameters are subjected to initial error self-calibration according to a preset standard specimen error correction threshold.

[0016] Furthermore, the step of performing tapping detection on the target to be detected according to the multi-point automatic tapping rule and performing self-calibration according to the initial tapping parameters specifically includes:

[0017] The three-dimensional point cloud data of the target to be measured is obtained, the knocking area layout is divided according to the three-dimensional point cloud data, and the knocking paths of different knocking areas are generated based on the adaptive fractal algorithm. The knocking paths are specifically as follows:

[0018] ,

[0019] in, , , Respectively represent the three-dimensional coordinates of the next tap point, , , and They represent the fractal trajectory generation parameters of the horizontal coordinates of the tapping points, , , and They represent the fractal trajectory generation parameters of the vertical coordinates of the tapping points, , , and They represent the fractal trajectory generation parameters of the vertical coordinates of the tapping point, , , Respectively represent the three-dimensional coordinates of the current tapping point;

[0020] The tapping path is self-calibrated according to the initial tapping parameters and the trajectory curvature radius of the tapping path.

[0021] Furthermore, the step of performing self-calibration of the tapping path according to the initial tapping parameters and the trajectory curvature radius of the tapping path further includes:

[0022] Perform array acquisition according to the knocking area layout to obtain multimodal data corresponding to different knocking areas, wherein the multimodal data includes audio modal data and vibration modal data;

[0023] The audio modality data is subjected to noise separation processing, and the algorithm of the noise separation processing is specifically as follows:

[0024] ,

[0025] in, Represents the original audio modal data, and represents the noise separation matrix, represents the norm, represents the balance weight coefficient, represents the noise basis matrix;

[0026] The vibration modal data is discretized to obtain discrete vibration modal data, and then the discrete vibration modal data is decomposed based on different frequency sub-bands. The specific algorithm of the decomposition process is as follows:

[0027] ,

[0028] in, and denote the scale coefficient and wavelet packet coefficient respectively, represents the number of layers of discrete vibration modal data, represents the number of frequency bands of discrete vibration modal data, represents the total length of the filter, represents the filter length, and represent the coefficients of the low-pass filter and the high-pass filter respectively, Represents discrete vibration modal data;

[0029] Then, the correlation between the audio modal data after noise separation and the vibration modal data after decomposition is verified to filter out irrelevant multimodal data.

[0030] Furthermore, the step of extracting the multimodal fusion features of the multimodal data according to the multimodal convolutional neural network specifically includes:

[0031] Inputting the multimodal data into a multimodal convolutional neural network, wherein the multimodal convolutional neural network includes a shallow multi-scale feature extraction branch and a deep feature extraction branch;

[0032] Performing feature extraction on the audio modal data in the multimodal data according to the shallow multi-scale feature extraction branch to obtain basic audio features of different feature scales;

[0033] Extracting features from vibration modal data in the multimodal data according to the deep feature extraction branch to obtain deep vibration features;

[0034] The deep vibration features are fused with basic audio features of different feature scales according to the feature scale sequence to obtain multimodal fusion features.

[0035] Furthermore, the step of fusing the deep vibration features with basic audio features of different feature scales according to the feature scale sequence to obtain multimodal fusion features includes:

[0036] The multimodal fusion features are subjected to spatiotemporal synchronization processing according to the time dimension weight and the space dimension weight, and the time dimension weight and the space dimension weight are specifically as follows:

[0037] ,

[0038] ,

[0039] ,

[0040] ,

[0041] in, and Represent the time dimension weight and space dimension weight respectively, and Represent the distribution values ​​of temporal attention and spatial attention respectively, and Represent the temporal and spatial features respectively. Represents global features, represents the total time step, represents a single time step, represents the total space vector, represents a single space vector;

[0042] The multimodal fusion features after the spatiotemporal synchronization processing are subjected to attention enhancement processing, and the specific formula of the attention enhancement processing is as follows:

[0043] ,

[0044] in, Represent query features, key features, and value features respectively, represents the activation function, Represents the dimension, represents the prior probability of basic defects, Represents the transposed matrix.

[0045] Furthermore, the step of performing defect identification according to the multimodal enhanced features to mark the defect area specifically includes:

[0046] Defect classification is performed according to the multimodal enhanced features to obtain defect states of all knock detection areas, wherein the defect states include a fully bonded state, a partially poorly bonded state, and a completely detached state;

[0047] Marking all knock detection areas according to the defect status to generate a defect hot spot map;

[0048] Match the defect heat map with the spectrum map and time domain response curve of the multimodal data, and then send them to the data center and mobile terminal for visual display;

[0049] The data center generates and stores the defect history records of the defect heat map according to a preset historical defect database.

[0050] The present invention proposes a multi-modal resonance detection system adapted to different dielectric interface adhesions, comprising:

[0051] An automated tapping module, used to obtain material properties of the target to be tested to match the initial tapping parameters, perform tapping detection on the target to be tested according to multi-point automated tapping rules, and perform self-calibration according to the initial tapping parameters. The target to be tested is two different media, and the tapping detection is applied to the fitting interface of the two different media.

[0052] A preprocessing module, used for collecting the multimodal data of the knock detection and performing preprocessing respectively;

[0053] A feature processing module, used for extracting multimodal fusion features of the multimodal data according to a multimodal convolutional neural network, and then performing feature enhancement processing on the multimodal fusion features according to a spatiotemporal synchronous attention algorithm to obtain multimodal enhanced features;

[0054] The identification module is used to perform defect identification according to the multimodal enhanced features to mark the defect area.

[0055] The present invention also provides a storage medium, wherein the storage medium stores one or more programs, and when the programs are executed by a processor, the multi-modal resonance detection method adapted to the interface adhesion of different media as described above is implemented.

[0056] The present invention further provides a computer device, the computer device comprising a memory and a processor, wherein:

[0057] The memory is used to store computer programs;

[0058] When the processor is used to execute the computer program stored in the memory, the multi-modal resonance detection method adapted to different dielectric interface adhesions as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flow chart of a multi-modal resonance detection method adapted to different dielectric interface adhesions proposed in the first embodiment of the present invention;

[0060] Figure 2 A flow chart of a multi-modal resonance detection method adapted to different dielectric interface adhesions proposed in the second embodiment of the present invention;

[0061] Figure 3 This is a schematic structural diagram of a multi-modal resonance detection system adapted to different dielectric interface adhesions proposed in the third embodiment of the present invention.

[0062] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0063] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0064] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0066] See also Figure 1 , which is a flow chart of a multi-modal resonance detection method adapted to different dielectric interface adhesions proposed in the first embodiment of the present invention, and the multi-modal resonance detection method adapted to different dielectric interface adhesions comprises steps S01 to S04, wherein:

[0067] Step S01: obtaining material properties of the target to be tested to match the initial knocking parameters, performing knocking detection on the target to be tested according to the multi-point automatic knocking rules, and performing self-calibration according to the initial knocking parameters;

[0068] It should be noted that in this embodiment, the material combination of the target to be measured is obtained, so as to retrieve the acoustic impedance ratio parameter and the interface stiffness parameter matching the material combination according to the preset material property database;

[0069] Calculating and generating initial knocking parameters according to the acoustic impedance ratio parameter and the interface stiffness parameter;

[0070] Performing initial error self-calibration on the initial knocking parameters according to a preset standard specimen error correction threshold;

[0071] The three-dimensional point cloud data of the target to be measured is obtained, the knocking area layout is divided according to the three-dimensional point cloud data, and the knocking paths of different knocking areas are generated based on the adaptive fractal algorithm. The knocking paths are specifically as follows:

[0072] ,

[0073] in, , , Respectively represent the three-dimensional coordinates of the next tap point, , , and They represent the fractal trajectory generation parameters of the horizontal coordinates of the tapping points, , , and They represent the fractal trajectory generation parameters of the vertical coordinates of the tapping points, , , and They represent the fractal trajectory generation parameters of the vertical coordinates of the tapping point, , , Respectively represent the three-dimensional coordinates of the current tapping point;

[0074] The tapping path is self-calibrated according to the initial tapping parameters and the trajectory curvature radius of the tapping path.

[0075] Step S02: collecting multimodal data of knock detection and preprocessing them respectively;

[0076] It should be noted that in this embodiment, array acquisition is performed according to the knocking area layout to obtain multimodal data corresponding to different knocking areas, and the multimodal data includes audio modal data and vibration modal data;

[0077] The audio modality data is subjected to noise separation processing, and the algorithm of the noise separation processing is specifically as follows:

[0078] ,

[0079] in, Represents the original audio modal data, and represents the noise separation matrix, represents the norm, represents the balance weight coefficient, represents the noise basis matrix;

[0080] The vibration modal data is discretized to obtain discrete vibration modal data, and then the discrete vibration modal data is decomposed based on different frequency sub-bands. The specific algorithm of the decomposition process is as follows:

[0081] ,

[0082] in, and denote the scale coefficient and wavelet packet coefficient respectively, represents the number of layers of discrete vibration modal data, represents the number of frequency bands of discrete vibration modal data, represents the total length of the filter, represents the filter length, and represent the coefficients of the low-pass filter and the high-pass filter respectively, Represents discrete vibration modal data;

[0083] Then, the correlation between the audio modal data after noise separation and the vibration modal data after decomposition is verified to filter out irrelevant multimodal data.

[0084] Step S03: extracting multimodal fusion features of multimodal data according to a multimodal convolutional neural network, and then performing feature enhancement processing on the multimodal fusion features according to a spatiotemporal synchronous attention algorithm to obtain multimodal enhanced features;

[0085] It should be noted that in this embodiment, multimodal data is input into a multimodal convolutional neural network, and the multimodal convolutional neural network includes a shallow multi-scale feature extraction branch and a deep feature extraction branch;

[0086] Performing feature extraction on the audio modal data in the multimodal data according to the shallow multi-scale feature extraction branch to obtain basic audio features of different feature scales;

[0087] Extracting features from vibration modal data in the multimodal data according to the deep feature extraction branch to obtain deep vibration features;

[0088] According to the feature scale sequence, the deep vibration features are respectively fused with basic audio features of different feature scales to obtain multimodal fusion features;

[0089] The multimodal fusion features are subjected to spatiotemporal synchronization processing according to the time dimension weight and the space dimension weight, and the time dimension weight and the space dimension weight are specifically as follows:

[0090] ,

[0091] ,

[0092] ,

[0093] ,

[0094] in, and Represent the time dimension weight and space dimension weight respectively, and Represent the distribution values ​​of temporal attention and spatial attention respectively, and Represent the temporal and spatial features respectively. Represents global features, represents the total time step, represents a single time step, represents the total space vector, represents a single space vector;

[0095] The multimodal fusion features after the spatiotemporal synchronization processing are subjected to attention enhancement processing, and the specific formula of the attention enhancement processing is as follows:

[0096] ,

[0097] in, Represent query features, key features, and value features respectively, represents the activation function, Represents the dimension, represents the prior probability of basic defects, Represents the transposed matrix.

[0098] Step S04: performing defect recognition according to the multimodal enhanced features to mark the defective area;

[0099] It should be noted that in this embodiment, defect classification is performed according to the multimodal enhancement feature to obtain the defect status of all knock detection areas, and the defect status includes a complete bonding state, a partial poor bonding state, and a complete detachment state;

[0100] The specific algorithm of defect classification is as follows:

[0101] ,

[0102] in, represents the decision function for defect classification, represents the symbolic function, represents the total sample size, represents the sample ordinal number, represents the sample label, represents the Lagrange multiplier, represents the kernel function, represents the bias parameter;

[0103] Marking all knock detection areas according to the defect status to generate a defect hot spot map;

[0104] Match the defect heat map with the spectrum map and time domain response curve of the multimodal data, and then send them to the data center and mobile terminal for visual display;

[0105] The data center generates and stores the defect history records of the defect heat map according to a preset historical defect database.

[0106] In summary, according to the above-mentioned multimodal resonance detection method adapted to the fit of different medium interfaces, by matching and setting the initial tapping parameters for different material interfaces to adapt to different material properties and interface states, the adaptability to different material interfaces is improved, while ensuring that the signal frequency range of the generated and collected multimodal data is in the target stable range, avoiding distortion or interference signals due to excessively high or low tapping detection energy, and designing a multi-point automatic tapping rule to adaptively generate tapping trajectories for different tapping surface curvatures and different component shapes, while dividing multiple different tapping areas for multi-point tapping to avoid the influence of structural blind spots and improve the adaptability and accuracy of detection, and then enhancing the features of the multimodal data through a spatiotemporal synchronous attention algorithm to synchronously match the features of the multimodal data in time and space orientations, eliminating the interference of complex combinations or composite methods of materials, and also eliminating the noise interference of the detection environment, thereby improving the accuracy of overall recognition. The present invention improves the accuracy and adaptability of different medium interface fit detection. Specifically, the material properties of the target to be measured are obtained to match the initial knocking parameters, the target to be measured is knocked according to the multi-point automatic knocking rules, and self-calibration is performed according to the initial knocking parameters. The target to be measured is two different media, and the knocking detection is applied to the fitting interface of the two different media. For different material interfaces, the initial knocking parameters are matched and set to adapt to different material properties and interface states, so as to improve the adaptability to different material interfaces. The multimodal data of the knocking detection are collected and pre-processed respectively, the interference noise and interference vibration in the actual detection environment are removed, and the multimodal signals in the target range are screened. At the same time, the detection of knocking to generate a target signal is enhanced, and the multimodal fusion features of the multimodal data are extracted according to the multimodal convolutional neural network. The multimodal fusion features are then feature enhanced according to the spatiotemporal synchronous attention algorithm to obtain multimodal enhanced features, and the multimodal data features are synchronously matched in time and space orientations. The frequency offset features of the acoustic signal and the energy attenuation features of the vibration signal in the multimodal signal are jointly extracted to improve the accuracy of the overall recognition. Defect identification is performed according to the multimodal enhanced features to mark the defective area. The present invention improves the accuracy and adaptability of interface fit detection of different media.

[0107] See also Figure 2 , which is a flow chart of a multi-modal resonance detection method adapted to different dielectric interface adhesions proposed in the second embodiment of the present invention, the multi-modal resonance detection method adapted to different dielectric interface adhesions comprises steps S11 to S16, wherein:

[0108] Step S11: obtaining a material combination of the target to be tested, retrieving the acoustic impedance ratio parameters and interface stiffness parameters of the matching material combination according to a preset material property database, calculating and generating initial knocking parameters according to the acoustic impedance ratio parameters and the interface stiffness parameters, and performing initial error self-calibration on the initial knocking parameters according to a preset standard specimen error correction threshold;

[0109] It should be noted that the preset material property database in this embodiment includes acoustic impedance ratio parameters and interface stiffness parameters of a variety of different material combinations. In order to ensure the accuracy and reliability of the test results, the preset standard specimen error correction threshold in this embodiment is used as a limit value for judging whether the measurement error needs to be corrected. When the measurement error exceeds the preset standard specimen error correction threshold, the measurement result needs to be corrected. In this embodiment, the preset standard specimen error correction threshold is set based on the metal material standard parts with the highest material proportion in the vehicle.

[0110] Step S12: acquiring three-dimensional point cloud data of the target to be measured, dividing the knocking area layout according to the three-dimensional point cloud data, generating knocking paths of different knocking areas based on an adaptive fractal algorithm, and performing self-calibration of the knocking path according to the initial knocking parameters and the trajectory curvature radius of the knocking path;

[0111] Step S13: performing array acquisition according to the layout of the knocking areas to obtain multimodal data corresponding to different knocking areas, the multimodal data including audio modal data and vibration modal data, performing noise separation processing on the audio modal data, performing discretization processing on the vibration modal data to obtain discrete vibration modal data, then decomposing the discrete vibration modal data based on different frequency sub-bands, and then performing correlation verification on the audio modal data after the noise separation processing and the vibration modal data after the decomposition processing to filter out irrelevant multimodal data;

[0112] Step S14: inputting the multimodal data into a multimodal convolutional neural network, the multimodal convolutional neural network including a shallow multi-scale feature extraction branch and a deep feature extraction branch, performing feature extraction on the audio modal data in the multimodal data according to the shallow multi-scale feature extraction branch to obtain basic audio features of different feature scales, performing feature extraction on the vibration modal data in the multimodal data according to the deep feature extraction branch to obtain deep vibration features, and performing feature fusion on the deep vibration features with the basic audio features of different feature scales according to the feature scale sequence to obtain multimodal fusion features;

[0113] Step S15: performing spatiotemporal synchronization processing on the multimodal fusion features according to the time dimension weight and the space dimension weight, and performing attention enhancement processing on the multimodal fusion features after the spatiotemporal synchronization processing;

[0114] Step S16: Defect classification is performed according to the multimodal enhanced features to obtain the defect status of all knock detection areas, and all knock detection areas are marked according to the defect status to generate a defect hotspot map, and the defect hotspot map is matched with the spectrum map and time domain response curve of the multimodal data, and then sent to the data center and the mobile terminal for visual display. The data center generates and stores the defect history records of the defect hotspot map according to the preset historical defect database;

[0115] It should be noted that the preset historical defect database in this embodiment includes defect hot spots maps of all vehicle models and components and the spectrum diagrams and time domain response curves of the corresponding multimodal data.

[0116] In summary, according to the above-mentioned multimodal resonance detection method adapted to the fit of different medium interfaces, by matching and setting the initial tapping parameters for different material interfaces to adapt to different material properties and interface states, the adaptability to different material interfaces is improved, while ensuring that the signal frequency range of the generated and collected multimodal data is in the target stable range, avoiding distortion or interference signals due to excessively high or low tapping detection energy, and designing a multi-point automatic tapping rule to adaptively generate tapping trajectories for different tapping surface curvatures and different component shapes, while dividing multiple different tapping areas for multi-point tapping to avoid the influence of structural blind spots and improve the adaptability and accuracy of detection, and then enhancing the features of the multimodal data through a spatiotemporal synchronous attention algorithm to synchronously match the features of the multimodal data in time and space orientations, eliminating the interference of complex combinations or composite methods of materials, and also eliminating the noise interference of the detection environment, thereby improving the accuracy of overall recognition. The present invention improves the accuracy and adaptability of different medium interface fit detection. Specifically, the material properties of the target to be measured are obtained to match the initial knocking parameters, the target to be measured is knocked according to the multi-point automatic knocking rules, and self-calibration is performed according to the initial knocking parameters. The target to be measured is two different media, and the knocking detection is applied to the fitting interface of the two different media. For different material interfaces, the initial knocking parameters are matched and set to adapt to different material properties and interface states, so as to improve the adaptability to different material interfaces. The multimodal data of the knocking detection are collected and pre-processed respectively, the interference noise and interference vibration in the actual detection environment are removed, and the multimodal signals in the target range are screened. At the same time, the detection of knocking to generate a target signal is enhanced, and the multimodal fusion features of the multimodal data are extracted according to the multimodal convolutional neural network. The multimodal fusion features are then feature enhanced according to the spatiotemporal synchronous attention algorithm to obtain multimodal enhanced features, and the multimodal data features are synchronously matched in time and space orientations. The frequency offset features of the acoustic signal and the energy attenuation features of the vibration signal in the multimodal signal are jointly extracted to improve the accuracy of the overall recognition. Defect identification is performed according to the multimodal enhanced features to mark the defective area. The present invention improves the accuracy and adaptability of interface fit detection of different media.

[0117] See also Figure 3 , which is a schematic diagram of the structure of a multi-modal resonance detection system adapted to different dielectric interface adhesions proposed in the third embodiment of the present invention, the system comprises:

[0118] The automated tapping module 10 is used to obtain the material properties of the target to be tested to match the initial tapping parameters, perform tapping detection on the target to be tested according to the multi-point automated tapping rules, and perform self-calibration according to the initial tapping parameters. The target to be tested is two different media, and the tapping detection is applied to the fitting interface of the two different media.

[0119] A preprocessing module 20, used for collecting the multi-modal data of the knock detection and performing preprocessing respectively;

[0120] A feature processing module 30 is used to extract multimodal fusion features of the multimodal data according to a multimodal convolutional neural network, and then perform feature enhancement processing on the multimodal fusion features according to a spatiotemporal synchronous attention algorithm to obtain multimodal enhanced features;

[0121] The identification module 40 is used to perform defect identification according to the multi-modal enhanced features to mark the defect area.

[0122] The present invention also provides a computer storage medium on which one or more programs are stored. When the programs are executed by a processor, the multi-modal resonance detection method for adapting to the interface adhesion of different media is implemented.

[0123] The present invention also proposes a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned multi-modal resonance detection method adapted to the interface fit of different media.

[0124] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0125] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0126] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0128] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multi-modal resonance detection method adapted to different media interface adhesion, characterized in that: include: Obtaining material properties of the target to be measured to match the initial knocking parameters, performing knocking detection on the target to be measured according to multi-point automatic knocking rules, and performing self-calibration according to the initial knocking parameters, wherein the target to be measured is two different media, and the knocking detection is applied to the fitting interface of the two different media; The step of obtaining the material properties of the target to be tested to match the initial knocking parameters specifically includes: Acquire a material combination of the target to be measured, and retrieve an acoustic impedance ratio parameter and an interface stiffness parameter matching the material combination according to a preset material property database; Calculating and generating initial knocking parameters according to the acoustic impedance ratio parameter and the interface stiffness parameter; Performing initial error self-calibration on the initial knocking parameters according to a preset standard specimen error correction threshold; The step of performing tapping detection on the target to be detected according to the multi-point automatic tapping rule and performing self-calibration according to the initial tapping parameters specifically includes: The three-dimensional point cloud data of the target to be measured is obtained, the knocking area layout is divided according to the three-dimensional point cloud data, and the knocking paths of different knocking areas are generated based on the adaptive fractal algorithm. The knocking paths are specifically as follows: ， in, , , Respectively represent the three-dimensional coordinates of the next tap point, , , and They represent the fractal trajectory generation parameters of the horizontal coordinates of the tapping points, , , and They represent the fractal trajectory generation parameters of the vertical coordinates of the tapping points, , , and They represent the fractal trajectory generation parameters of the vertical coordinates of the tapping point, , , Respectively represent the three-dimensional coordinates of the current tapping point; Performing self-calibration of the knocking path according to the initial knocking parameters and the trajectory curvature radius of the knocking path; Collecting multimodal data of the knock detection and preprocessing them respectively; The multimodal data includes audio modal data and vibration modal data; Extracting multimodal fusion features of the multimodal data according to a multimodal convolutional neural network, and then performing feature enhancement processing on the multimodal fusion features according to a spatiotemporal synchronous attention algorithm to obtain multimodal enhanced features; Defect recognition is performed based on the multimodal enhanced features to mark the defective area.

2. The multi-modal resonance detection method for adapting to different media interface adhesion according to claim 1, characterized in that: The step of performing self-calibration of the knocking path according to the initial knocking parameters and the trajectory curvature radius of the knocking path further includes: Perform array acquisition according to the knocking area layout to obtain multimodal data corresponding to different knocking areas, wherein the multimodal data includes audio modal data and vibration modal data; The audio modality data is subjected to noise separation processing, and the algorithm of the noise separation processing is specifically as follows: ， in, Represents the original audio modal data, and represents the noise separation matrix, represents the norm, represents the balance weight coefficient, represents the noise basis matrix; The vibration modal data is discretized to obtain discrete vibration modal data, and then the discrete vibration modal data is decomposed based on different frequency sub-bands. The specific algorithm of the decomposition process is as follows: ， in, and denote the scale coefficient and wavelet packet coefficient respectively, represents the number of layers of discrete vibration modal data, represents the number of frequency bands of discrete vibration modal data, represents the total length of the filter, represents the filter length, and represent the coefficients of the low-pass filter and the high-pass filter respectively, Represents discrete vibration modal data; Then, the correlation between the audio modal data after noise separation and the vibration modal data after decomposition is verified to filter out irrelevant multimodal data.

3. The multi-modal resonance detection method for adapting the interface adhesion of different media according to claim 1, characterized in that: The step of extracting the multimodal fusion features of the multimodal data according to the multimodal convolutional neural network specifically includes: Inputting the multimodal data into a multimodal convolutional neural network, wherein the multimodal convolutional neural network includes a shallow multi-scale feature extraction branch and a deep feature extraction branch; Performing feature extraction on the audio modal data in the multimodal data according to the shallow multi-scale feature extraction branch to obtain basic audio features of different feature scales; Extracting features from vibration modal data in the multimodal data according to the deep feature extraction branch to obtain deep vibration features; The deep vibration features are fused with basic audio features of different feature scales according to the feature scale sequence to obtain multimodal fusion features.

4. The multi-modal resonance detection method for adapting to different media interface adhesion according to claim 3, characterized in that: The step of fusing the deep vibration features with basic audio features of different feature scales according to the feature scale sequence to obtain multimodal fusion features, then includes: The multimodal fusion features are subjected to spatiotemporal synchronization processing according to the time dimension weight and the space dimension weight, and the time dimension weight and the space dimension weight are specifically as follows: ， ， ， ， in, and Represent the time dimension weight and space dimension weight respectively, and Represent the distribution values ​​of temporal attention and spatial attention respectively, and Represent the temporal and spatial features respectively. Represents global features, represents the total time step, represents a single time step, represents the total space vector, represents a single space vector; The multimodal fusion features after the spatiotemporal synchronization processing are subjected to attention enhancement processing, and the specific formula of the attention enhancement processing is as follows: ， in, Represent query features, key features, and value features respectively. represents the activation function, Represents the dimension, represents the prior probability of basic defects, Represents the transposed matrix.

5. The multi-modal resonance detection method for adapting to different media interface adhesion according to claim 1, characterized in that: The step of performing defect recognition according to the multimodal enhanced features to mark the defect area specifically includes: Defect classification is performed according to the multimodal enhanced features to obtain defect states of all knock detection areas, wherein the defect states include a fully bonded state, a partially poorly bonded state, and a completely detached state; Marking all knock detection areas according to the defect status to generate a defect hot spot map; Match the defect heat map with the spectrum map and time domain response curve of the multimodal data, and then send them to the data center and mobile terminal for visual display; The data center generates and stores the defect history records of the defect heat map according to a preset historical defect database.

6. A multi-modal resonance detection system adapted to different media interface adhesion, characterized in that: include: An automated tapping module, used to obtain material properties of the target to be tested to match the initial tapping parameters, perform tapping detection on the target to be tested according to multi-point automated tapping rules, and perform self-calibration according to the initial tapping parameters. The target to be tested is two different media, and the tapping detection is applied to the fitting interface of the two different media. The step of obtaining the material properties of the target to be tested to match the initial knocking parameters specifically includes: Acquire a material combination of the target to be measured, and retrieve an acoustic impedance ratio parameter and an interface stiffness parameter matching the material combination according to a preset material property database; Calculating and generating initial knocking parameters according to the acoustic impedance ratio parameter and the interface stiffness parameter; Performing initial error self-calibration on the initial knocking parameters according to a preset standard specimen error correction threshold; The step of performing tapping detection on the target to be detected according to the multi-point automatic tapping rule and performing self-calibration according to the initial tapping parameters specifically includes: The three-dimensional point cloud data of the target to be measured is obtained, the knocking area layout is divided according to the three-dimensional point cloud data, and the knocking paths of different knocking areas are generated based on the adaptive fractal algorithm. The knocking paths are specifically as follows: ， in, , , Respectively represent the three-dimensional coordinates of the next tap point, , , and They represent the fractal trajectory generation parameters of the horizontal coordinates of the tapping points, , , and They represent the fractal trajectory generation parameters of the vertical coordinates of the tapping points, , , and They represent the fractal trajectory generation parameters of the vertical coordinates of the tapping point, , , Respectively represent the three-dimensional coordinates of the current tapping point; Performing self-calibration of the knocking path according to the initial knocking parameters and the trajectory curvature radius of the knocking path; A preprocessing module, used for collecting the multimodal data of the knock detection and performing preprocessing respectively; The multimodal data includes audio modal data and vibration modal data; A feature processing module, used for extracting multimodal fusion features of the multimodal data according to a multimodal convolutional neural network, and then performing feature enhancement processing on the multimodal fusion features according to a spatiotemporal synchronous attention algorithm to obtain multimodal enhanced features; The identification module is used to perform defect identification according to the multimodal enhanced features to mark the defect area.

7. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by the processor, implement the multi-modal resonance detection method adapted to the interface fit of different media as described in any one of claims 1 to 5.

8. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, the multi-modal resonance detection method adapted to the interface fit of different media as described in any one of claims 1 to 5 is implemented.

Images