A product defect analysis method based on audio experience text

Abstract

The application discloses a product defect analysis method based on audio experience text. A product defect expert experience text library is constructed, and expert listening experience knowledge is used as dynamic acoustic semantic prompt. An audio-text cross-modal alignment model is constructed, and audio timing acoustic segments and expert text are mapped to a shared embedding space. After obtaining audio of a product to be analyzed, candidate defect types and expert text prompts are determined through cross-modal matching, a frame-level positioning model is guided to perform sliding window analysis by taking the expert text prompt as a dynamic condition, types, start and end time positions and confidence of multiple defects are synchronously recognized, and a structured multi-defect diagnosis report is generated. Defect acoustic fingerprints are extracted from diagnosed audio, and the expert experience text library is automatically optimized or supplemented after cross-modal consistency verification, so that an audio-text closed-loop evolution mechanism is formed.

Description

Technical Field

[0001] This invention relates to the fields of industrial audio analysis, acoustic detection of product defects, multimodal learning, and knowledge-driven quality diagnosis, and particularly to a product defect analysis method based on audio-textual experience. The method is used in product quality inspection scenarios with historical quality inspection audio and accumulated expert experience text. It utilizes expert experience text descriptions as dynamic prompts to guide an audio understanding model to perform frame-level multi-defect localization, and automatically optimizes the expert experience text library through acoustic fingerprint feedback of audio diagnostic results, forming a closed-loop defect analysis mechanism with bidirectional audio-text enhancement. Background Technology

[0002] In the process of industrial product quality inspection, the sound of a product's operation is an important basis for judging whether a product has defects. Defects in industrial products such as motors, pumps, valves, bearings, gearboxes, and compressors often manifest abnormal operating sounds. For example, bearing wear produces periodic metallic friction sounds, abnormal gear meshing manifests as rhythmic knocking sounds, and seal failure is accompanied by a continuous whistling sound. Experienced quality inspectors usually identify the type of defect, assess its severity, and provide remedial suggestions by listening to the product's operating sounds.

[0003] Current methods for analyzing audio defects in industrial products mainly fall into two categories. The first category is based on traditional acoustic features, such as using Mel-Frequency Cepstral Coefficients (MFCC), wavelet packet energy, or spectral analysis to extract acoustic features, which are then combined with classifiers such as support vector machines, autoencoders, or Gaussian mixture models for anomaly detection. These methods typically require manual feature design and parameter calibration for specific product categories and defect types, and novel defects not present in the training data are difficult to identify effectively. In practical applications, they still heavily rely on human experts for assisted auditory judgment, resulting in low detection efficiency and high subjectivity.

[0004] The second category is based on deep learning methods, such as using convolutional neural networks or audio Transformers to perform end-to-end defect classification on product operation audio. In recent years, although some studies have attempted to input audio into large language models for defect analysis, most only output simple normal / abnormal judgments or single defect category labels, failing to accurately pinpoint the time interval of defect occurrence, and even more difficult to simultaneously identify multiple overlapping defect types in the same audio. For product operation audio containing multiple different defects, existing methods often only report the most prominent defect while missing other co-occurring or overlapping defects.

[0005] More importantly, in actual production lines, the field of quality inspection has accumulated a wealth of expert experience textual knowledge over a long period, such as "bearing wear emits a continuous high-frequency whistling sound, with the frequency increasing with rotational speed," "poor gear meshing manifests as periodic knocking, with the knocking frequency consistent with the meshing frequency," and "rotor imbalance causes low-frequency vibration accompanied by a humming sound, with the amplitude increasing with rotational speed." These expert experience texts accurately depict the acoustic characteristics and auditory performance patterns of different product types and defect types in natural language. However, existing audio defect analysis methods generally use these expert texts only as defect category labels or reference descriptions, failing to fully leverage the potential of natural language descriptions in dynamically guiding audio analysis models to focus on specific defect acoustic patterns. Deep learning methods rely on a large number of labeled audio samples, while the rich acoustic prior knowledge contained in expert texts has not been effectively utilized for cross-modal defect feature learning.

[0006] On the other hand, the number of product operation audio samples collected from industrial sites continues to grow, and new acoustic patterns of defects are constantly emerging hidden in the audio data. Existing systems lack a closed-loop mechanism to automatically feed newly discovered acoustic patterns of defects back to the knowledge base: when the audio understanding model identifies acoustic manifestations of defects in newly collected audio that are inconsistent with existing experience text descriptions, the system cannot automatically extract the acoustic fingerprint features of the defects from the audio to update or supplement the expert experience text base. This results in the knowledge base failing to evolve continuously, making it difficult to continuously improve the ability to identify new types of defects. Therefore, the static nature of the expert experience text base and the unidirectional nature of audio diagnostic results constitute the core bottleneck of existing technology: audio defect analysis systems cannot flexibly absorb expert knowledge described in natural language, lack fine-grained localization and classification capabilities for scenarios with multiple concurrent defects and acoustic aliasing, and lack a closed-loop mechanism to extract descriptions from new audio to update the experience text base.

[0007] Therefore, a product defect analysis method based on audio experience text is needed. This method should use expert experience text descriptions as dynamic semantic cues to guide the audio understanding model to achieve frame-level fine-grained localization of multiple defects based on cross-modal semantic alignment. At the same time, this method should also establish a feedback loop from audio diagnostic results to the expert experience text library, so that newly discovered defect acoustic fingerprints can automatically optimize or supplement the experience text descriptions, continuously improving the system's generalization ability for different products and defect types. Summary of the Invention

[0008] The purpose of this invention is to provide a product defect analysis method based on audio experience text, in order to solve the problems in existing methods where expert experience text is only used as a passive label and fails to fully guide audio analysis, cannot accurately locate the time intervals of multiple defects in the audio, and lacks a closed-loop feedback mechanism from audio diagnostic results to the expert experience text library.

[0009] To achieve the above objectives, this invention first constructs a product defect expert experience text library. This library stores expert experience text descriptions linked by defect type identifiers. These descriptions use natural language to depict the acoustic characteristics and auditory performance patterns of the product defects. The sources of these expert experience text descriptions may include at least one of the following: quality inspection records, repair reports, operating procedures, and interviews with domain experts.

[0010] This invention constructs an audio-text cross-modal alignment model, which includes an audio encoder and a text encoder. This model maps the temporal acoustic segment features of product audio with the expert-experienced text description to a shared embedding space. Through contrastive learning training, semantically related audio segments and text descriptions are brought closer together in the shared embedding space, providing a unified semantic representation foundation for subsequent cross-modal matching.

[0011] Upon receiving the audio of the product to be analyzed, the system extracts the temporal acoustic segment features of the audio. Based on the cross-modal alignment model, it performs cross-modal matching between the temporal acoustic segment features and the text descriptions of each defect type in the expert experience text library corresponding to the product category, determining the candidate defect type set and its corresponding expert text prompts. The cross-modal matching uses a cosine similarity metric in a shared embedding space, i.e. ,in For the feature vector of a temporal acoustic segment, For defect-type text feature vectors, For cross-modal similarity.

[0012] In the frame-level defect localization stage, this invention uses expert text prompts corresponding to the candidate defect types as dynamic conditional inputs to guide the frame-level defect localization model to perform sliding window analysis on the product audio. Unlike traditional fixed classifiers, this invention encodes expert text prompts as conditional vectors and deeply fuses them with the audio frame features within the sliding window through a cross-attention mechanism. This allows the localization model to perceive the acoustic semantic information of the defects described by the expert text at each window position, thereby simultaneously identifying multiple defects appearing in the audio and outputting the defect type, defect start and end time positions, and frame-level confidence for each defect.

[0013] In the diagnostic output stage, this invention generates a structured multi-defect diagnostic report based on the frame-level defect location results. Each defect entry records the defect type, defect time interval, frame-level confidence level, and the expert text referenced, so that the diagnostic results have traceable expert knowledge support.

[0014] In the closed-loop optimization phase, this invention extracts defect acoustic fingerprint features from the diagnosed audio and performs cross-modal consistency checks with the existing text descriptions of the corresponding defect types in the expert experience text library. When the inconsistency exceeds a preset update threshold, the expert experience text descriptions of the corresponding defect types are automatically optimized or new defect acoustic fingerprint text entries are added based on the defect acoustic fingerprint features, forming a closed-loop mechanism for audio diagnostic results feedback to optimize the expert experience text library.

[0015] In a preferred embodiment, the present invention merges the detection results of multiple different defect types in the same audio segment. When the detection results of the same defect type meet the cross-union ratio threshold in the time interval, they are merged into one defect entry. When the time intervals of different defect types overlap, their respective entries are retained and marked as co-occurrence defects or acoustic aliasing defects.

[0016] In a preferred embodiment, the present invention sets a manual review node in the closed-loop optimization stage. The text description change record automatically generated by the system must be manually confirmed before it can be formally written into the expert experience text database, so as to ensure the quality and reliability of the knowledge base.

[0017] The features of this invention are:

[0018] (1) Audio-text cross-modal alignment elevates expert experience text from passive labeling to active semantic guidance;

[0019] (2) Expert text prompts serve as dynamic conditional guidance for frame-level multi-defect synchronous localization;

[0020] (3) Acoustic fingerprint cross-modal consistency verification and feedback optimization of expert experience text library. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall process of steps S1 to S6 of the product defect analysis method based on audio experience text described in this invention.

[0022] Figure 2 This is a flowchart of step S1, "Construction of Expert Experience Text Library," and step S2, "Training of Audio-Text Cross-Modal Alignment Model," as described in this invention.

[0023] Figure 3 This is a schematic diagram of the process of cross-modal matching and candidate defect type determination in step S3 and dynamic prompting and guidance frame-level multi-defect localization in the present invention.

[0024] Figure 4 This is a flowchart of step S5, generating a structured multi-defect diagnostic report, and step S6, optimizing the audio-text closed loop, as described in this invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0026] This invention can be deployed in an industrial product quality acoustic inspection system. The industrial product quality acoustic inspection system includes at least an audio acquisition device, a product defect expert experience text library, an audio encoder, a text encoder, a cross-modal alignment module, a frame-level defect localization model, a diagnostic report generation module, and a closed-loop optimization module. The audio acquisition device is used to acquire product operating audio; the product defect expert experience text library is used to identify and store expert experience text descriptions according to defect type; the audio encoder is used to encode temporal segments of the product operating audio into acoustic feature vectors; the text encoder is used to encode expert text descriptions into text feature vectors; the cross-modal alignment module is used to map audio features and text features to a shared embedding space and calculate cross-modal matching scores; the frame-level defect localization model is used to perform sliding window frame-level defect detection under the guidance of expert text prompts; the diagnostic report generation module is used to output a structured multi-defect diagnostic report; and the closed-loop optimization module is used to extract defect acoustic fingerprint features from the diagnosed audio and perform consistency checks and text library updates.

[0027] In one embodiment, each defect type entry in the product defect expert experience text library includes at least one of the following: defect type identifier, expert experience text description, defect acoustic keyword set, defect frequency domain feature description, defect auditory performance pattern description, applicable product category list, and evidence source identifier. The expert experience text description is derived from at least one natural language description from quality inspection records, maintenance reports, operating procedures, and interviews with domain experts, and has undergone terminology standardization. The defect type may include at least one of abnormal noise, impact sound, friction sound, whistling sound, rhythmic knocking sound, and abnormal operating noise. The product category may include at least one of motors, pumps / valves, bearings, gearboxes, and compressors.

[0028] like Figure 1 As shown, the method includes steps S1 to S6. Step S1 is used to construct a product defect expert experience text library; step S2 is used to construct an audio-text cross-modal alignment model; step S3 is used to acquire the audio of the product to be analyzed and determine candidate defect types and expert text prompts based on cross-modal matching; step S4 is used to guide frame-level multi-defect localization using expert text prompts as dynamic conditions; step S5 is used to generate a structured multi-defect diagnostic report; and step S6 is used to extract defect acoustic fingerprints from the diagnosed audio and perform closed-loop optimization. Thus, the method forms a complete processing chain from expert knowledge encoding, cross-modal alignment, dynamic prompt-guided localization, multi-defect diagnostic output to audio feedback optimization knowledge base.

[0029] Step S1 includes: collecting expert experience text data related to historical product defects, including acoustic descriptions of defects in quality inspection records, fault sound records in maintenance reports, abnormal sound descriptions in operating procedures, and summaries of listening experience from interviews with domain experts; performing terminology standardization on the expert experience text data, extracting defect type identifiers and corresponding standardized natural language descriptions; establishing a hierarchical index structure according to product category and defect type, and storing the standardized expert experience text descriptions in the product defect expert experience text database. The terminology standardization process includes unifying different colloquial expressions of the same acoustic phenomenon into standardized terms, for example, standardizing "incorrect sound," "noisy operation," and "sounds abnormal" into "abnormal operating noise," and storing the corresponding acoustic feature descriptions (such as "periodic metallic friction sound, frequency increasing with rotational speed," "rhythmic knocking sound, knocking frequency consistent with gear meshing frequency," and "continuous high-frequency whistling sound, accompanied by outlet pressure fluctuations") as sub-entries in the expert experience text description of that defect type.

[0030] Step S2 includes: selecting an audio encoder architecture and a text encoder architecture; constructing a training dataset containing paired audio segments and text descriptions, wherein positive sample pairs in the training dataset consist of defective audio segments and their corresponding expert text descriptions, and negative sample pairs consist of defective audio segments randomly paired with unrelated expert text descriptions; and jointly training the audio encoder and text encoder using a contrastive learning loss function, so that positive sample pairs are close to each other in a shared embedding space, and negative sample pairs are far apart. In one embodiment, the audio encoder uses a Mel-Frequency Cepstral Coefficients (MFCC) feature extractor combined with a convolutional neural network architecture to encode the input audio segments into fixed-dimensional temporal acoustic segment feature vectors; in another embodiment, the audio encoder uses an audio Transformer or audio spectrogram Transformer architecture, utilizing a self-attention mechanism to capture the global time-frequency dependencies of the audio segments. The text encoder uses a pre-trained language model to encode the expert experience text descriptions into text feature vectors. The contrastive learning loss function can be InfoNCE (Info Noise Contrastive Estimation) loss or a variant thereof.

[0031] Step S3 includes: receiving audio of the product to be analyzed; obtaining the product category identifier corresponding to the product audio; calling the audio encoder to divide the product audio into multiple time-series segments and extracting the acoustic feature vectors of each segment; calculating the similarity between the features of each time-series acoustic segment and the text feature vectors of each defect type in the expert experience text library of the corresponding product category based on the cross-modal alignment model; determining the candidate defect type set and the expert text prompts corresponding to each candidate defect type according to the similarity ranking. In one embodiment, the product category identifier can be provided by workstation configuration, material coding, inspection task sheet, upstream product classification module, or manual selection results. When the product category identifier of the product audio to be analyzed is missing or there is no defect type entry corresponding to the product category in the expert experience text library, the system can use the general defect text description across product categories as the initial matching basis, and in the subsequent closed-loop optimization, prioritize creating exclusive defect expert text entries for the new product category.

[0032] Step S4 includes: encoding the expert text prompts corresponding to the candidate defect types into conditional vectors using the text encoder; traversing each time interval of the product audio in a sliding window manner; at each sliding window position, performing cross-attention fusion of the conditional vectors and the audio frame features within the window, enabling the frame-level defect localization model to perceive the acoustic semantic information of the defects described by the expert text prompts; and based on the fused features, simultaneously predicting whether a defect exists within the current window, the type of defect, the start time position of the defect, the end time position of the defect, and the frame-level confidence level. The audio frame features can be a sequence representation of Mel spectrogram features, short-time Fourier transform features, or wavelet transform features.

[0033] Unlike the traditional two-stage method of classification followed by localization, this invention injects dynamic acoustic semantic signals from expert text prompts into the audio frame sequence analysis process through a cross-attention mechanism. This allows the localization model to selectively focus on time-frequency regions related to specific defect acoustic patterns under the guidance of expert knowledge, thereby simultaneously completing defect detection and time-interval localization in a single stage. When multiple candidate defect types exist, the system independently performs frame-level localization for each candidate defect type, generating frame-level detection results for each candidate defect type.

[0034] Step S5 includes: receiving the frame-level detection results of each candidate defect type output in step S4; merging the detection results; and generating a structured multi-defect diagnostic report. The merging rules include: when two detection results have the same defect type and the intersection-over-union ratio (IoU) of their defect time intervals exceeds a preset merging threshold, they are merged into the same defect entry, with the higher confidence level used as the merged confidence level; when two detection results have different defect types but overlapping time intervals, their respective defect entries are retained and marked as co-occurring defects or acoustic aliasing defects in the diagnostic report. Each defect entry in the structured multi-defect diagnostic report records at least the defect type, defect time interval (start time to end time), frame-level confidence level, and the cited expert text. In one embodiment, the diagnostic report also includes a defect severity level and treatment recommendations. The defect severity level is determined based on the frame-level confidence level and the defect duration, and the treatment recommendations are retrieved and sorted from historical treatment records associated with the defect type in the expert experience text library.

[0035] Step S6 includes steps S6-1, S6-2, and S6-3. Step S6-1 is used to extract defect acoustic fingerprint features from the diagnosed audio, step S6-2 is used to perform cross-modal consistency verification, and step S6-3 is used to perform optimization or supplementation operations on the expert experience text library based on the verification results.

[0036] In step S6-1, the system extracts the acoustic fingerprint features of the defect from the diagnosed audio within the defect time interval determined in step S5. The acoustic fingerprint features include at least one of the following: Mel spectrogram features, wavelet packet energy distribution features, and spectral centroid trajectory features for the defect time interval. The Mel spectrogram features are used to characterize the energy distribution pattern of the defective sound in different Mel frequency bands; the wavelet packet energy distribution features are used to characterize the energy proportion of the defective sound in each frequency band, reflecting the spectral structure of the sound; and the spectral centroid trajectory features are used to characterize the time-varying law of the energy centroid of the defective sound in the audio domain.

[0037] In step S6-2, the system maps the acoustic fingerprint features of the defect to a shared embedding space through the cross-modal alignment model, performs cross-modal consistency verification with the existing text description features of the defect type in the expert experience text library, and calculates the acoustic fingerprint-text consistency score. The consistency score uses the same cosine similarity metric as cross-modal matching. ,in This is the acoustic fingerprint feature vector of the defect. This is the textual description feature vector for this defect type. The acoustic fingerprint-text consistency score reflects the degree of semantic matching between the actual acoustic performance of the defect heard in the current audio and the existing textual descriptions of this defect type in the expert experience text library.

[0038] In step S6-3, the system executes different update strategies based on the acoustic fingerprint-text consistency score. When the acoustic fingerprint-text consistency score is lower than a preset update threshold, it is determined that the existing text description of the defect type is insufficient to cover the acoustic performance of the defect heard in the current audio. The system automatically encodes the acoustic fingerprint features of the defect into a supplementary natural language description. The encoding process can utilize a pre-trained audio description generation model or a multimodal large language model to convert the acoustic fingerprint features of the defect into a natural language description of the acoustic performance of the defect (e.g., "Intermittent energy pulses exist in the 800 to 1200 Hz frequency band, with a pulse interval of approximately 0.2 seconds, which coincides with the second harmonic of the gear meshing frequency"). The generated supplementary description is appended to the expert experience text description entries for this defect type, or replaces the existing description with the lowest consistency score. When the acoustic fingerprint-text consistency score is higher than a preset retention threshold, the existing text description of the defect type is kept unchanged, indicating that the existing expert knowledge has well covered the acoustic performance of the current defect.

[0039] Furthermore, in step S6-3, the present invention also includes a manual review node. When the system automatically performs text description optimization or supplementation operations, a change record is generated and pushed to the manual review interface. The change record includes a defect type identifier, the original text description, the new text description, the reason for the change, and the acoustic fingerprint-text consistency score. Only after manual confirmation is the changed content officially written into the product defect expert experience text database. This mechanism ensures that the knowledge base has the ability to evolve automatically, while ensuring the accuracy and reliability of the knowledge entered into the database through manual review.

[0040] like Figure 2 As shown, Figure 2 Further explanation of steps S1 and S2 follows. Step S1 involves collecting expert experience text data from multiple sources, including quality inspection records, repair reports, operating procedures, and expert interviews. After terminology standardization, a hierarchical index is established according to product category and defect type, and the data is written into a product defect expert experience text database. Step S2 involves constructing a paired audio-text training dataset. The audio encoder and text encoder are jointly trained using a contrastive learning loss function, ensuring that positive sample pairs are close to each other in the shared embedding space and negative sample pairs are far apart, thus obtaining an audio-text cross-modal alignment model.

[0041] like Figure 3 As shown, Figure 3Steps S3 and S4 are further explained. Step S3 includes receiving the audio of the product to be analyzed, obtaining the product category identifier, extracting temporal acoustic segment features, performing cross-modal matching with the text features of each defect type in the expert experience text library corresponding to the product category, and determining the candidate defect type set and corresponding expert text prompts by sorting them by similarity. Step S4 includes encoding the expert text prompts into conditional vectors, traversing each time interval of the product audio with a sliding window, and fusing the conditional vectors with audio frame features at each window position through cross-attention to simultaneously predict whether a defect exists in the window, the defect type, the start and end time positions, and the frame-level confidence. When there are multiple candidate defect types, the above localization process is performed independently for each type.

[0042] like Figure 4 As shown, Figure 4 Further explanation of steps S5 and S6 is provided. Step S5 includes merging the frame-level detection results of each candidate defect type—merging defects of the same type with an intersection-union ratio (IU) exceeding a threshold, and marking defects of different types with overlapping time intervals as co-occurring defects or acoustic aliasing defects—and generating a structured multi-defect diagnostic report containing defect type, defect time interval, confidence level, expert textual basis, severity level, and treatment recommendations. Step S6 includes extracting the acoustic fingerprint features of defects from the defect time intervals of the diagnosed audio, performing cross-modal consistency verification with the textual descriptions of the corresponding defect types in the expert experience text library, automatically encoding and generating supplementary textual descriptions when the consistency score is lower than the update threshold, and writing them into the expert experience text library after manual review and confirmation, thus completing the audio-text closed-loop optimization.

[0043] In one embodiment, the audio of the product to be analyzed is a recording of a certain model of motor in operation. The audio simultaneously contains high-frequency whistling caused by bearing wear and low-frequency periodic vibration noise caused by fan blade imbalance. The system first performs cross-modal matching between the audio segment and an expert experience text library for motor categories in step S3, identifying "high-frequency whistling due to bearing wear" and "low-frequency vibration noise due to fan blade imbalance" as candidate defect types. In step S4, the system uses "a continuous high-frequency whistling sound is emitted when the bearing is worn, with a frequency concentrated between 2000 and 4000 Hz, increasing with increasing speed" and "fan blade imbalance manifests as periodic low-frequency humming, with frequency synchronized with speed and accompanied by amplitude modulation" as dynamic conditions to guide the frame-level localization model to locate the start and end times and confidence levels of the two defects in the audio. In step S5, the system generates a diagnostic report containing the two defect entries. In step S6, the system extracts the acoustic fingerprint features of the two defects from the located defect time intervals, performs consistency verification with existing descriptions in the expert experience text library, and decides whether to optimize the text description based on the verification results.

[0044] In another embodiment, the audio of the product to be analyzed is a recording of the operation of a certain type of pump valve, in which a continuous whistling sound caused by seal failure is heard. In step S3, the system matches the pump valve category with an expert experience text library to obtain the expert text description of "seal failure whistling sound": "When the pump valve seal fails, a continuous high-frequency whistling sound can be heard, with the frequency concentrated between 1500 and 3000 Hz, accompanied by outlet pressure fluctuations," as a dynamic prompt. In step S4, the frame-level positioning model, guided by the dynamic prompt, accurately locates the start and end times of the whistling sound along the audio time sequence. Since the spectral characteristics of the whistling sound are highly consistent with the existing description of "seal failure whistling sound" in the expert experience text library, the cross-modal consistency check score in step S6 is high, and there is no need to trigger a text library update.

[0045] In another embodiment, the audio of the product to be analyzed is a recording of a new model gearbox in operation. The audio contains an irregular, intermittent tapping pattern not recorded in the expert experience text library. In step S3, since the new product category lacks a specific defect text entry, the system uses a general defect text description for gearboxes as the matching basis. In step S4, the system can still locate the abnormal time interval based on the general description. In step S6, after extracting acoustic fingerprint features from the abnormal time interval, the system finds that the consistency score with the general text description is lower than the update threshold. It automatically encodes the newly discovered irregular, intermittent tapping pattern into a natural language description—for example, "Irregular intermittent tapping occurs within a specific speed range, the peak of the tapping spectrum is around 500Hz, the tapping interval varies randomly between 0.15 and 0.8 seconds, and there is no fixed multiple relationship with the gear meshing frequency"—and after manual review, it is added to the gearbox category defect entry in the expert experience text library. Subsequently, when encountering similar abnormal audio again, the system can directly match the updated expert text description, improving diagnostic accuracy and location precision.

[0046] This invention is not limited to a specific audio encoder architecture or text encoder architecture. The audio encoder can employ MFCC combined with convolutional neural networks, audio Transformer, AST (Audio Spectrogram Transformer), or other deep learning architectures capable of extracting audio time-frequency features; the text encoder can employ BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly Optimized BERTP Retraining Approach), GPT (Generative Pre-trained Transformer), or other pre-trained language models; the cross-modal alignment model can employ a CLAP (Contrastive Language-Audio Pretraining) style contrastive learning framework or other audio-text cross-modal alignment frameworks; the frame-level defect localization model can employ a Transformer decoder architecture based on cross-attention or other temporal localization models that support conditional input. The encoding of the defect acoustic fingerprint features into a natural language description can be achieved using a pre-trained multimodal large language model or an audio description generation model.

[0047] It should be noted that the hierarchical structure of defect types, terminology standardization rules, number of candidates for cross-modal matching, size and step size of the sliding window, merging threshold of frame-level detection results, calculation method of acoustic fingerprint-text consistency score, specific values ​​of update threshold and retention threshold, and manual review process in closed-loop optimization in the aforementioned expert experience text library can all be configured according to specific product quality acoustic testing scenarios. Those skilled in the art can achieve the same or similar technical effects by using different network architectures, acoustic feature extraction methods, similarity measurements, or audio description generation methods without departing from the inventive concept.

[0048] In summary, this invention constitutes a complete bidirectional enhanced defect analysis system through a product defect expert experience text library, an audio-text cross-modal alignment model, dynamic expert text prompts guiding frame-level multi-defect localization, structured diagnostic reports, and audio-text closed-loop optimization. Its core concept lies in elevating expert listening experience knowledge expressed in natural language from passive labels to active acoustic semantic guidance signals, and establishing a closed-loop feedback channel between audio diagnostic results and the expert experience text library. This enables the system to achieve precise multi-defect frame-level localization driven by expert knowledge, while continuously absorbing new defect acoustic patterns from audio data to evolve the knowledge base. As long as the overall technical concept remains based on using expert experience text descriptions as dynamic conditions to guide audio frame-level defect localization, and optimizing the expert experience text library based on acoustic fingerprint feedback from audio diagnostic results, it should fall within the protection scope of this invention.

Claims

1. A product defect analysis method based on audio-based experiential text, characterized in that, The method includes the following steps: Step S1, constructing a product defect expert experience text library, which includes expert experience text descriptions stored in association according to defect type identifiers. The expert experience text descriptions use natural language to characterize the acoustic features and auditory performance patterns of product defects; Step S2, constructing an audio-text cross-modal alignment model, which includes an audio encoder and a text encoder, used to map the temporal acoustic segment features of the product audio to the expert experience text descriptions to a shared embedding space, so that semantically related audio segments and text descriptions are close in distance in the shared embedding space; Step S3, obtaining the product audio to be analyzed and its product category identifier, extracting the temporal acoustic segment features of the product audio, and performing cross-modal matching between each temporal acoustic segment feature and the text descriptions of each defect type in the expert experience text library of the corresponding product category based on the cross-modal alignment model to determine the candidate defect type set and its corresponding expert text prompts; Step S4, using the candidate defect... The expert text prompts corresponding to the defect type serve as dynamic condition inputs, guiding the frame-level defect localization model to perform sliding window analysis on the product audio, simultaneously identifying multiple defects appearing in the audio, and outputting the defect type, defect start and end time positions, and frame-level confidence for each defect; Step S5: Based on the frame-level defect localization results, a structured multi-defect diagnostic report is generated. The structured multi-defect diagnostic report contains at least one defect entry, and each defect entry records the defect type, defect time interval, frame-level confidence, and the cited expert text basis; Step S6: Defect acoustic fingerprint features are extracted from the diagnosed audio, and cross-modal consistency checks are performed with the existing text descriptions of the corresponding defect types in the expert experience text library. When the degree of inconsistency in the cross-modal consistency check exceeds a preset update threshold, the expert experience text descriptions of the corresponding defect types are automatically optimized or new defect acoustic fingerprint text entries are added based on the defect acoustic fingerprint features, forming a closed-loop mechanism for audio diagnostic result feedback optimization of the expert experience text library.

2. The method according to claim 1, characterized in that, Each defect type entry in the product defect expert experience text library includes at least one of the following: defect type identifier, expert experience text description, defect acoustic keyword set, defect frequency domain feature description, defect auditory performance pattern description, applicable product category list, and evidence source identifier; the expert experience text description is derived from at least one natural language description from quality inspection records, maintenance reports, operating procedures, and interviews with domain experts, and has undergone terminology standardization processing; the defect type includes at least one of abnormal noise, impact sound, friction sound, whistling sound, rhythmic knocking sound, and abnormal operating noise; the product category includes at least one of motor, pump valve, bearing, gearbox, and compressor.

3. The method according to claim 1, characterized in that, The audio encoder employs a Mel-frequency cepstral coefficient feature extractor combined with a convolutional neural network or an audio Transformer architecture to encode the input audio segments into fixed-dimensional temporal acoustic segment feature vectors. The text encoder uses a pre-trained language model to encode the expert experience text descriptions into text feature vectors. The shared embedding space is obtained through training a contrastive learning loss function, which brings matching audio segment-text description pairs closer together in the shared embedding space and keeps mismatched audio segment-text description pairs further apart.

4. The method according to claim 1, characterized in that, In step S4, using the expert text prompt as a dynamic condition input specifically includes: encoding the expert text prompt corresponding to the candidate defect type into a condition vector using the text encoder; performing cross-attention fusion of the condition vector with the audio frame features within the sliding window; enabling the frame-level defect localization model to perceive the acoustic semantic information of the defect described by the expert text prompt at each sliding window position; and based on the features after cross-attention fusion, the frame-level defect localization model simultaneously predicts whether a defect exists in the current window, the type of defect, the start time position of the defect, the end time position of the defect, and the frame-level confidence level.

5. The method according to claim 4, characterized in that, When the same audio segment is guided by expert text prompts of multiple different defect types, step S4 performs frame-level localization independently for each candidate defect type, and in step S5, multiple defect detection results in the same time interval are merged. The merging rules include: when the defect types of two detection results are the same and the cross-union ratio of the defect time intervals exceeds the merging threshold, the two are merged into the same defect entry, and the higher confidence level is used as the confidence level after merging; when the defect types of two detection results are different but the time intervals overlap, their respective defect entries are retained and marked as co-occurring defects or acoustic aliasing defects in the diagnostic report.

6. The method according to claim 1, characterized in that, Step S6 specifically includes: Step S6-1, extracting acoustic fingerprint features of defects from the defect time interval marked by frame-level localization results in the diagnosed audio, wherein the acoustic fingerprint features of defects include at least one of Mel spectrogram features, wavelet packet energy distribution features, and spectral centroid trajectory features of the defect time interval; Step S6-2, performing cross-modal consistency verification between the acoustic fingerprint features of defects and existing text descriptions of the defect type in the expert experience text library, and calculating an acoustic fingerprint-text consistency score; Step S6-3, when the acoustic fingerprint-text consistency score is lower than a preset update threshold, determining that the existing text descriptions of the defect type are insufficient to cover the acoustic performance of the current defect, the system automatically encodes the acoustic fingerprint features of defects into supplementary natural language descriptions and appends them to the expert experience text description entries of the defect type, or replaces the existing description with the lowest consistency score; when the acoustic fingerprint-text consistency score is higher than a preset retention threshold, keeping the existing text descriptions of the defect type unchanged.

7. The method according to claim 6, characterized in that, The cross-modal consistency verification also includes a manual review node: when the system automatically performs text description optimization or supplementation operations, a change record is generated and pushed to the manual review interface. The change record includes defect type identifier, original text description, new text description, reason for change and acoustic fingerprint-text consistency score; only after manual confirmation is the change content officially written into the product defect expert experience text library.

8. The method according to claim 1, characterized in that, Step S3 further includes: when the product category identifier of the product audio to be analyzed is missing or there is no defect type entry corresponding to the product category in the expert experience text library, the system uses a general defect text description across product categories as the initial matching basis, and in step S6, it prioritizes creating a dedicated defect expert text entry for the new product category.

9. The method according to claim 1, characterized in that, Each defect entry in the structured multi-defect diagnostic report also records the defect severity level and handling recommendations. The defect severity level is determined based on the frame-level confidence level and the defect duration. The handling recommendations are retrieved and sorted from the historical handling records associated with the defect type in the expert experience text library.

10. An electronic device comprising a processor and a memory, characterized in that, The memory stores a computer program, which, when executed by the processor, causes the electronic device to perform the method described in any one of claims 1 to 9.

Images