A speech-visual fusion method and system
By constructing a topological difference metric for speech and images, the problem of speech recognition error accumulation in noisy industrial workshop environments was solved, enabling advanced perception and blocking in noisy environments and improving the reliability and accuracy of quality inspection reports.
Patent Information
- Application Number
- CN202611337624.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
In industrial workshops with strong impulse noise, existing speech-vision fusion methods suffer from the problem of accumulated speech recognition errors, which are difficult to correct effectively, making it hard to correct errors in quality inspection reports.
By simultaneously acquiring the voice signals of quality inspectors and industrial microscopic images, a topological structure difference metric value for voice and image is constructed. The output of voice recognition is adjusted using topological anomaly perception signals, and a visual verification mechanism is activated to generate defect description text.
Capturing speech features in noisy environments degrades the quality, preventing erroneous recognition results from entering quality inspection reports, improving the reliability and accuracy of defect description texts, saving computing resources, and making it suitable for online quality inspection scenarios.
Smart Images

Figure CN123116152A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of industrial defect detection technology, and in particular relates to a voice and vision fusion method and system. Background Technology
[0002] With the deepening of intelligent upgrading in the manufacturing industry, visual quality inspection of industrial parts based on human-machine collaboration has been widely used in the field of precision manufacturing. In the final quality inspection of precision parts such as bearings and gears, quality inspectors typically use a handheld high-magnification industrial camera to scan the surface of the parts and simultaneously verbally describe the types and sizes of microscopic defects observed. The system then automatically converts the verbal description into text through a speech recognition engine, generating a structured quality inspection report. To improve the accuracy of speech recognition, existing technical solutions typically employ a speech-vision fusion method based on deep learning. This method introduces simultaneously acquired microscopic images of the parts as an auxiliary modality to verify and correct the speech recognition results at the content level.
[0003] However, the aforementioned methods face severe performance degradation in industrial workshops with strong impulse noise. Sudden impulse noise generated by stamping, grinding, and other equipment in workshops has high sound pressure levels and its spectrum highly overlaps with speech. In existing serial fusion modes, the speech recognition engine first independently decodes and generates candidate text, which is then verified by the vision module. When impulse noise covers key defective syllables, the engine may produce high-confidence erroneous recognition results. The subsequent vision module, receiving text that deviates from the actual spoken text, struggles to effectively backtrack and correct, leading to errors directly entering the quality inspection record. This "listen first, then see" approach suffers from irreversible error accumulation and lacks the ability to determine the structural integrity of the speech signal before semantic generation. Therefore, we propose a speech-vision fusion method and system. Summary of the Invention
[0004] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: This invention relates to a speech-visual fusion method, comprising the following steps: Step S1: Synchronously acquire the voice signal of the quality inspector describing the defects and the industrial microscopic image of the current inspection area, and preprocess the industrial microscopic image; Step S2: The speech signal is framed and its spectral features are extracted. A high-dimensional feature point cloud is constructed based on the spectral vector sequence within the sliding window to generate a persistent graph sequence representing the evolution of the speech topology. Step S3: Select an industrial microscopic image that is synchronized with the midpoint of the current speech sliding window time, extract key points with scale and rotation invariance and their descriptor vectors, construct a local topological feature manifold based on the descriptor vectors, and generate a persistence map characterizing the microscopic texture topology of the current field of view. Step S4: Based on the characteristic that there is a statistical correspondence between the speech spectrum topology and the image texture topology under stable quality inspection conditions, calculate the topological structure difference metric between the speech persistence map and the image persistence map. The topological structure difference metric represents the degree of correlation deviation between the speech topological distribution and the image topological distribution. The topological structure difference metric is obtained by weighting the harmonic noise ratio of the current speech signal and the sensitivity weight of the real-time ambient noise sound pressure level dynamic adjustment, and the topological structure difference metric is used as a topological anomaly sensing signal. At the same time, a topological anomaly map is generated according to the spatial distribution of the topological structure difference. The topological anomaly map is used to indicate the regions in the image that are inconsistent with the speech topological structure. Step S5: When the topology anomaly perception signal is less than a preset threshold, the candidate text sequence output by the speech recognition engine is directly adopted as the defect description; when the topology anomaly perception signal is greater than or equal to the preset threshold, the output of the speech recognition engine is blocked, the visual verification mechanism is activated, and the image description generation network generates defect description text based on the industrial microscopic image and the topology anomaly map.
[0005] Further, step S1 includes the following steps: Step S11: Using a microphone array fixed at the quality inspection station, continuously collect single-channel voice signals when the quality inspector verbally describes defects at a preset sampling rate of not less than 8kHz. Step S12: Using an industrial microscope camera that is synchronized with the hardware clock of the microphone array, continuously capture images of the bearing raceway surface at a preset frame rate, the preset frame rate being between 15 frames per second and 60 frames per second. Step S13: Perform automatic white balance and histogram equalization on each frame of the image to eliminate brightness fluctuations caused by the flickering of fluorescent lights in the workshop; pass the voice signal through a high-pass filter with a cutoff frequency in the range of 50Hz to 120Hz to remove low-frequency mechanical vibration interference.
[0006] Furthermore, step S2 includes the following steps: Step S21: Using an analysis window with a length between 20ms and 40ms, the continuously flowing speech signal is divided into frames with a frame shift of 5ms to 15ms. The short-time Fourier transform of each frame signal is calculated to obtain the linear spectrum. The linear spectrum is then filtered through a set of Mel filter banks to obtain the multidimensional Mel spectrum vector. The number of Mel filter banks is between 40 and 128; Step S22: Use a sliding window to cover multiple consecutive frames, where each consecutive frame corresponds to a context of a preset duration. Treat the set of points formed by the multidimensional Mel spectrum vectors within the window as a point cloud in a high-dimensional space. Step S23: First, perform principal component analysis on the Mel spectrum vector to reduce the dimensionality to within 10 dimensions, then use the sparse Vietoris-Rips complex approximation method to generate a persistence graph, given a distance threshold. When the Euclidean distance between any two points satisfies When two points are connected by an edge, a triangle is formed when three points are connected in pairs, and so on to generate a simple complex. The range increases from 0 to the point cloud diameter, recording the birth and death information of complex topological features, and generating the first... 0-dimensional persistent graph of a window With 1D continuity graph The continuously generated persistence graph sequence constitutes a manifold describing the evolution of speech topology; the 0-dimensional and 1-dimensional persistence graphs are fused according to preset weights to obtain a unified speech persistence graph for subsequent topological difference measurement.
[0007] Furthermore, step S3 includes the following steps: Step S31: Process the current frame image captured by the camera. The scale-invariant feature transform algorithm is used to extract keypoints with scale and rotation invariance. Each keypoint is represented by a 128-dimensional descriptor vector. Characterization; Step S32: Only retain key points within a preset proportion area in the center of the image. The preset proportion range is 30% to 70% of the center area, corresponding to the core detection area of the rolling track focused by the microscope camera, to avoid global background interference. Step S33: After dimensionality reduction of the 128-dimensional descriptor vectors, the persistence graph is calculated using the approximate persistent homology algorithm. The Vietoris-Rips complex construction framework, the same as in step S23, is used to calculate the... 0D persistence map of key point cloud of frame image With 1D continuity graph Since the visual scene switching speed is slower than the audio frame rate and the image content already contains sufficient local structural information within a single frame, each frame generates an independent topological snapshot. The persistence graph represents the inherent topological structure of the micro-texture of the defect region under the current field of view.
[0008] Step S34: Treat the point cloud composed of the retained 128-dimensional descriptor vectors as a point cloud within the local topological space, and use the same Vietoris-Rips complex construction framework as in step S23 to calculate the... 0D persistence map of key point cloud of frame image With 1D continuity graph The first The frame image is an image frame that is synchronized with the hardware clock at the midpoint of the current speech sliding window time; the 0-dimensional and 1-dimensional persistence maps are fused according to preset weights to obtain a unified image persistence map for subsequent topological difference measurement; the visual scene switching speed is slower than the speech frame rate and the image content already contains sufficient local structural information in a single frame, so each frame generates an independent topological snapshot, and the persistence map represents the inherent topological structure of the micro-texture of the defect region under the current field of view.
[0009] Furthermore, in step S4, the calculation method for the topological difference metric is as follows: Take the unified persistence graph of the current voice window after fusion. With the fused unified persistence map corresponding to the current image frame , Includes A continuous point, Includes Each persistent point is mapped to a birth-death two-dimensional plane with its birth scale as the x-axis and its death scale as the y-axis. Before semantic content, malicious distortions of speech features caused by industrial impulse noise are captured at the topological level. Based on Wasserstein distance, the Euclidean distance between two points on the birth-death two-dimensional plane is used as the matching cost function. With the constraint that each persistent point participates in only one matching, a linear programming method is used to solve for the optimal transmission matching relationship. Determine the correspondence between points in two persistence graphs; calculate the weighted sum of the absolute values of the differences in persistence values for all matching point pairs, as a measure of topological dissimilarity. , The magnitude of the value directly reflects the degree of deviation of the statistical correlation between the speech spectrum topology and the image texture topology. The larger the value, the more serious the noise pollution of the speech signal and the higher the degree of destruction of the topological statistical correspondence under stable conditions. The calculation formula is as follows: In the formula, express The Middle The persistence value of each point, This is the scale at which the topological feature disappears. This is the scale at which this topological feature was generated. That is, the lifespan of this topological feature from its birth to its extinction; Indicates in via optimal transmission matching and The Middle The persistence value of each point corresponding to a given point; To act on the first Adjust the weights based on the sensitivity of the matching points.
[0010] Furthermore, the sensitivity adjustment weight The harmonic noise ratio of the current speech signal and real-time collected ambient noise sound pressure level Dynamic joint determination enables the topological difference metric to be adaptively sensitive to the drastically changing noise environment in industrial settings. Reduce or All increases will push up This amplifies the calculated topological differences at the moment of noise pollution; Calculated using the following formula: In the formula, The sigmoid function maps the input to the (0,1) interval. The more noisy the environment, the more... The closer the value is to 1, the more sensitive the topological difference measure becomes to small distortions; and It is a scaling factor obtained in advance through training with a small number of standard samples, used to balance the magnitude of influence of the two acoustic parameters, harmonic noise ratio and ambient noise sound pressure level. It is a positive number to ensure that the input increases and the weight increases when the HNR decreases.
[0011] Furthermore, step S5 includes the following steps: Step S51: Synchronously feed the speech signal into the speech recognition engine based on connection-time classification to obtain candidate text sequences. and confidence score The system combines topological anomaly perception signals with speech recognition confidence to make joint decisions. When a topological anomaly occurs and the confidence level is below a threshold, a visual review is triggered. Step S52: Preset topology anomaly threshold With the speech confidence threshold p th The threshold The threshold is determined based on the statistical distribution of the topological difference measurement values of standard samples verbally described by quality inspectors in a clean environment, so that the threshold can distinguish between normal fluctuations in voice signals and malignant topological distortions caused by noise. Step S53: When the topology difference metric value is calculated in real time And the confidence level of speech recognition ≥p th If the topological structure of the speech feature manifold is consistent with that of the current image texture manifold, and the speech signal has not undergone malignant distortions that affect the structure, the candidate text sequence is directly adopted. The defect description at the current inspection point is stored in the quality inspection report; Step S54: When the topology difference metric value is calculated in real time or speech recognition confidence p th If the current speech signal is found to be severely contaminated by noise and the speech manifold structure is mismatched with the visual scene, the output of the speech recognition engine is blocked, and a pre-trained image description generation network is activated. This image description generation network uses the current microscopic image... Using the topological anomaly map as joint input, a vision-driven defect description text is generated. .
[0012] Further, the generation method of the topology anomaly map in step S54 is as follows: calculate the weighted sum of the persistence value deviation of the persistence point corresponding to each key point in the optimal transmission matching and the spatial matching distance, as the local persistence contribution value of the point; according to the pixel coordinates of the key point, the contribution value is interpolated and projected onto the entire image to obtain the topology anomaly heatmap. The heatmap highlights the region in the image that is most inconsistent with the speech topology structure, and the image description generation network focuses on the highlighted region to generate text.
[0013] A speech and vision fusion system includes a multimodal acquisition and processing module, a speech topology manifold construction module, an image topology manifold construction module, an acoustic environment parameter monitoring module, a topology difference measurement and generation module, a speech recognition module, a visual verification and image description module, and a decision fusion and output module. The multimodal acquisition and processing module simultaneously acquires the quality inspector's spoken voice and industrial microscopic images; it includes a microphone array, an industrial microscopic camera, and a hardware clock synchronization unit; the preprocessing unit performs automatic white balance and histogram equalization on the images, and removes low-frequency vibration interference from the voice signal using a high-pass filter; the output of the multimodal acquisition and processing module is connected to the input of the voice topology manifold construction module, the image topology manifold construction module, and the acoustic environment parameter monitoring module, respectively; The speech topology manifold construction module is used to receive the preprocessed speech stream, perform framing, short-time Fourier transform and Mel filtering, and generate a multidimensional Mel spectrum vector sequence; point cloud is formed through a sliding window, and speech persistence graph sequence is generated using the Vietoris-Rips complex construction method to characterize the dynamic evolution of the speech topology. The image topology manifold construction module is used to receive the current frame microscopic image, extract key points and 128-dimensional descriptors using the scale-invariant feature transformation algorithm; retain key points in the core detection area by filtering the central region, construct descriptor point clouds, and generate image persistence maps using the same complex construction method as the speech module to characterize the inherent topological structure of the microscopic texture in the current field of view. The acoustic environment parameter monitoring module is used to calculate the harmonic noise ratio of the speech signal in real time. and collecting ambient noise sound pressure level This provides input for subsequent dynamic weight adjustment; The topology difference measurement and generation module is used to receive the speech persistence map. and image persistence graph By using optimal transmission matching to calculate the weighted Wasserstein distance, a measure of topological dissimilarity is obtained. Weight Depend on and Dynamic adjustment; simultaneously, the topology difference measurement and generation module generates a topology anomaly heatmap based on the local persistent contribution value of each key point, indicating the region where the speech and visual topologies are most inconsistent; the input of the topology difference measurement and generation module is connected to the output of the speech topology manifold construction module, the image topology manifold construction module, and the acoustic environment parameter monitoring module, respectively; The speech recognition module is responsible for receiving speech streams synchronously, independently of the topological manifold channel, and generating candidate defect description text by connecting to a temporal classification model. and confidence score; The visual verification and image description module is activated upon receiving a topology anomaly trigger signal. Using the current microscopic image and topology anomaly heatmap as input, a pre-trained image description generation network focuses on the anomaly region, generating purely vision-driven defect description text. ; The decision fusion and output module is used to compare topology difference metrics. With preset threshold Generate control signals; when When the speech recognition confidence level meets the standard, select the speech recognition result. As the final output, otherwise, disable the audio results and select visual verification text. Finally, the defect description is stored in the quality inspection report; the input end of the decision fusion and output module is connected to the output end of the topology difference measurement and generation module, the speech recognition module, and the visual verification and image description module, respectively.
[0014] The present invention has the following beneficial effects: 1. This invention measures the consistency of speech and visual features at the topological level rather than the semantic content level, thus exhibiting natural resistance to industrial impulse noise interference. When subjected to sudden, strong noise, the topological manifold within a speech signal undergoes structural fractures or void proliferation. This distortion is sensitively reflected in the persistence graph, occurring much earlier than text output errors in the speech recognition engine. Therefore, this invention can capture feature quality degradation before the speech recognition result is generated, preventing erroneous recognition results from entering subsequent quality inspection reports. This achieves proactive perception and blocking of noise interference, improving the reliability of the final defect description text.
[0015] 2. This invention introduces a sensitivity weight that is adaptively adjusted by the harmonic noise ratio and ambient noise sound pressure level, enabling the topology difference measurement to dynamically adapt to changes in the ambient noise environment. When the workshop is in a relatively quiet period, the weight automatically decreases to avoid oversensitivity to normal speech fluctuations; when strong noise sources such as stamping machines are present, the weight automatically increases, amplifying the detection sensitivity of topology distortion. This adaptive mechanism eliminates the need for manually setting fixed thresholds or frequent recalibration, ensuring stable operation of the algorithm under different production cycles and background noise conditions.
[0016] 3. The visual verification mechanism of this invention is activated only when a structural mismatch is detected between the speech feature manifold and the image texture manifold, rather than continuously running the image description network, thus significantly saving computational resources. During the verification process, a topological anomaly map guides the image description network to focus on the region in the microscopic image that is least consistent with the speech topology. This ensures that the generated purely visual defect description has clear physical correspondence and specificity, avoiding redundant information or misjudgments introduced by indiscriminate description of the entire image. This on-demand activation design balances real-time performance and accuracy, making it suitable for online quality inspection scenarios.
[0017] 4. Compared with traditional speech recognition followed by text correction or attention-weighted fusion schemes, this invention avoids the dependence on large-scale noisy spoken language corpora and the complex training of cross-modal semantic alignment. The construction of speech manifolds and image manifolds is based on unsupervised continuous homology analysis, and the topological difference measurement only relies on real-time extracted acoustic parameters for fine-tuning. The overall scheme does not require the re-collection of a large amount of labeled data when switching between different product models or defect types, and has good generalization ability and low migration cost.
[0018] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating a speech-visual fusion method according to the present invention. Figure 2 This is a schematic diagram of the architecture of a speech and vision fusion system according to the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] Please see Figure 1-2 As shown, this invention is a speech-vision fusion method, comprising the following steps: Step S1: Simultaneously acquire the voice signal of the quality inspector describing the defects and the industrial microscopic image of the current inspection area, and preprocess the industrial microscopic image; Step S2: Frame the speech signal and extract spectral features. Construct a high-dimensional feature point cloud based on the spectral vector sequence within the sliding window to generate a persistent graph sequence representing the evolution of the speech topology. Step S3: Select an industrial microscopic image that is synchronized with the midpoint of the current speech sliding window time, extract key points with scale and rotation invariance and their descriptor vectors, construct a local topological feature manifold based on the descriptor vectors, and generate a persistence map that characterizes the microscopic texture topology of the current field of view. Step S4: Based on the statistical correspondence between the speech spectrum topology and image texture topology under stable quality inspection conditions, calculate the topological structure difference metric between the speech persistence map and the image persistence map. The topological structure difference metric represents the degree of correlation deviation between the speech topological distribution and the image topological distribution. The topological structure difference metric is obtained by weighting the harmonic noise ratio of the current speech signal and the sensitivity weight of the real-time ambient noise sound pressure level, and the topological structure difference metric is used as the topological anomaly sensing signal. At the same time, a topological anomaly map is generated according to the spatial distribution of the topological structure difference. The topological anomaly map is used to indicate the regions in the image that are inconsistent with the speech topological structure. Step S5: When the topology anomaly perception signal is less than the preset threshold, the candidate text sequence output by the speech recognition engine is directly adopted as the defect description; when the topology anomaly perception signal is greater than or equal to the preset threshold, the output of the speech recognition engine is blocked, the visual verification mechanism is activated, and the image description generation network generates defect description text based on the industrial microscopic image and the topology anomaly map.
[0023] Step S1 includes the following steps: Step S11: Using a microphone array fixed at the quality inspection station, continuously collect single-channel voice signals when the quality inspector verbally describes the defects at a preset sampling rate of not less than 8kHz. Step S12: Using an industrial microscope camera that is synchronized with the hardware clock of the microphone array, continuously capture images of the bearing raceway surface at a preset frame rate, which is between 15 frames per second and 60 frames per second. Step S13: Perform automatic white balance and histogram equalization on each frame of the image to eliminate brightness fluctuations caused by fluorescent light flicker in the workshop; remove low-frequency mechanical vibration interference from the voice signal by passing it through a high-pass filter with a cutoff frequency in the range of 50Hz to 120Hz.
[0024] Step S2 includes the following steps: Step S21: Using an analysis window with a length between 20ms and 40ms, the continuously flowing speech signal is divided into frames with a frame shift of 5ms to 15ms. The short-time Fourier transform of each frame signal is calculated to obtain the linear spectrum. The linear spectrum is then filtered through a set of Mel filter banks to obtain the multidimensional Mel spectrum vector. The number of Mel filter banks ranges from 40 to 128; Step S22: Use a sliding window to cover multiple consecutive frames. The multiple consecutive frames correspond to a context of a preset duration. Treat the point set formed by the multidimensional Mel spectrum vectors within the window as a point cloud in a high-dimensional space. Step S23: First, perform principal component analysis on the Mel spectrum vector to reduce the dimensionality to within 10 dimensions, then use the sparse Vietoris-Rips complex approximation method to generate a persistence graph, given a distance threshold. When the Euclidean distance between any two points satisfies When two points are connected by an edge, a triangle is formed when three points are connected in pairs, and so on to generate a simple complex. The range increases from 0 to the point cloud diameter, recording the birth and death information of complex topological features, and generating the first... 0-dimensional persistent graph of a window With 1D continuity graph The continuously generated persistence graph sequence constitutes a manifold describing the evolution of speech topology; the 0-dimensional and 1-dimensional persistence graphs are fused according to preset weights to obtain a unified speech persistence graph for subsequent topological difference measurement.
[0025] Step S3 includes the following steps: Step S31: Process the current frame image captured by the camera. The scale-invariant feature transform algorithm is used to extract keypoints with scale and rotation invariance. Each keypoint is represented by a 128-dimensional descriptor vector. Characterization; Step S32: Only retain key points within a preset proportion area in the center of the image. The preset proportion range is 30% to 70% of the center area, corresponding to the core detection area of the rolling track focused by the microscope camera, to avoid global background interference. Step S33: After dimensionality reduction of the 128-dimensional descriptor vectors, the persistence graph is calculated using the approximate persistent homology algorithm. The Vietoris-Rips complex construction framework, the same as in step S23, is used to calculate the... 0D persistence map of key point cloud of frame image With 1D continuity graph Since the visual scene switching speed is slower than the audio frame rate and the image content already contains enough local structural information in a single frame, each frame generates an independent topological snapshot. The persistence graph represents the inherent topological structure of the micro-texture of the defect region under the current field of view.
[0026] Step S34: Treat the point cloud composed of the retained 128-dimensional descriptor vectors as a point cloud within the local topological space, and use the same Vietoris-Rips complex construction framework as in step S23 to calculate the... 0D persistence map of key point cloud of frame image With 1D continuity graph ;No. The frame image is an image frame that is synchronized with the hardware clock at the midpoint of the current speech sliding window time. The 0-dimensional and 1-dimensional persistence maps are fused according to preset weights to obtain a unified image persistence map for subsequent topological difference measurement. Since the visual scene switching speed is slower than the speech frame rate and the image content already contains sufficient local structural information in a single frame, an independent topological snapshot is generated for each frame. The persistence map represents the inherent topological structure of the micro-texture of the defect region under the current field of view.
[0027] In step S4, the topology difference metric is calculated as follows: Take the unified persistence graph of the current voice window after fusion. With the fused unified persistence map corresponding to the current image frame , Includes A continuous point, Includes Each persistent point is mapped to a birth-death two-dimensional plane with its birth scale as the x-axis and its death scale as the y-axis. Before semantic content, malicious distortions of speech features caused by industrial impulse noise are captured at the topological level. Based on Wasserstein distance, the Euclidean distance between two points on the birth-death two-dimensional plane is used as the matching cost function. With the constraint that each persistent point participates in only one matching, a linear programming method is used to solve for the optimal transmission matching relationship. Determine the correspondence between points in two persistence graphs; calculate the weighted sum of the absolute values of the differences in persistence values for all matching point pairs, as a measure of topological dissimilarity. , The magnitude of the value directly reflects the degree of deviation of the statistical correlation between the speech spectrum topology and the image texture topology. The larger the value, the more serious the noise pollution of the speech signal and the higher the degree of destruction of the topological statistical correspondence under stable conditions. The calculation formula is as follows: In the formula, express The Middle The persistence value of each point, This is the scale at which the topological feature disappears. This is the scale at which this topological feature was generated. That is, the lifespan of this topological feature from its birth to its extinction; Indicates in via optimal transmission matching and The Middle The persistence value of each point corresponding to a given point; To act on the first Adjust the weights based on the sensitivity of the matching points.
[0028] Sensitivity adjustment weight The harmonic noise ratio of the current speech signal and real-time collected ambient noise sound pressure level Dynamic joint determination enables the topological difference metric to be adaptively sensitive to the drastically changing noise environment in industrial settings. Reduce or All increases will push up This amplifies the calculated topological differences at the moment of noise pollution; Calculated using the following formula: In the formula, The sigmoid function maps the input to the (0,1) interval. The more noisy the environment, the more... The closer the value is to 1, the more sensitive the topological difference measure becomes to small distortions; and It is a scaling factor obtained in advance through training with a small number of standard samples, used to balance the magnitude of influence of the two acoustic parameters, harmonic noise ratio and ambient noise sound pressure level. It is a positive number to ensure that the input increases and the weight increases when the HNR decreases.
[0029] Step S5 includes the following steps: Step S51: Synchronously feed the speech signal into the speech recognition engine based on connection-time classification to obtain candidate text sequences. and confidence score The system combines topological anomaly perception signals with speech recognition confidence to make joint decisions. When a topological anomaly occurs and the confidence level is below a threshold, a visual review is triggered. Step S52: Preset topology anomaly threshold With the speech confidence threshold p th threshold The threshold is determined based on the statistical distribution of the topological differences in standard samples described by quality inspectors in a clean environment, so that the threshold can distinguish between normal fluctuations in the voice signal and malignant topological distortions caused by noise. Step S53: When the topology difference metric value is calculated in real time And the confidence level of speech recognition ≥p th If the topological structure of the speech feature manifold is consistent with that of the current image texture manifold, and the speech signal has not undergone malignant distortions that affect the structure, then the candidate text sequence is directly adopted. The defect description at the current inspection point is stored in the quality inspection report; Step S54: When the topology difference metric value is calculated in real time or speech recognition confidence p th At this point, it is determined that the current speech signal is severely contaminated by noise, and the speech manifold structure is mismatched with the visual scene. The output of the speech recognition engine is then blocked, and the pre-trained image description generation network is activated. The image description generation network uses the current microscopic image... Using the topological anomaly map as joint input, a vision-driven defect description text is generated. .
[0030] The generation method of the topology anomaly map in step S54 is as follows: calculate the weighted sum of the persistence value deviation of the persistence point corresponding to each key point in the optimal transmission matching and the spatial matching distance, as the local persistence contribution value of the point; according to the pixel coordinates of the key point, the contribution value is interpolated and projected onto the entire image to obtain the topology anomaly heatmap. The heatmap highlights the area in the image that is most inconsistent with the speech topology structure, and the image description generation network focuses on the highlighted area to generate text.
[0031] A speech and vision fusion system includes a multimodal acquisition and processing module, a speech topology manifold construction module, an image topology manifold construction module, an acoustic environment parameter monitoring module, a topology difference measurement and generation module, a speech recognition module, a visual verification and image description module, and a decision fusion and output module. The multimodal acquisition and processing module simultaneously acquires the quality inspector's spoken voice and industrial microscopic images; it includes a microphone array, an industrial microscopic camera, and a hardware clock synchronization unit; the preprocessing unit performs automatic white balance and histogram equalization on the images, and removes low-frequency vibration interference from the voice signal using a high-pass filter; the output of the multimodal acquisition and processing module is connected to the input of the voice topology manifold construction module, the image topology manifold construction module, and the acoustic environment parameter monitoring module, respectively; The speech topology manifold construction module receives the preprocessed speech stream, performs framing, short-time Fourier transform and Mel filtering, and generates a multidimensional Mel spectrum vector sequence; it forms a point cloud through a sliding window and generates a speech persistence graph sequence using the Vietoris-Rips complex construction method to characterize the dynamic evolution of the speech topology. The image topology manifold construction module receives the current frame microscopic image, extracts key points and 128-dimensional descriptors using the scale-invariant feature transformation algorithm, retains key points in the core detection area by filtering the central region, constructs a descriptor point cloud, and generates an image persistence map using the same complex construction method as the speech module, representing the inherent topological structure of the microscopic texture in the current field of view. The acoustic environment parameter monitoring module is used to calculate the harmonic noise ratio of the speech signal in real time. and collecting ambient noise sound pressure level This provides input for subsequent dynamic weight adjustment; The topological difference measurement and generation module is used to receive the speech persistence map. and image persistence graph By using optimal transmission matching to calculate the weighted Wasserstein distance, a measure of topological dissimilarity is obtained. Weight Depend on and Dynamic adjustment; simultaneously, the topology difference measurement and generation module generates a topology anomaly heatmap based on the local persistent contribution value of each key point, indicating the region where the speech and visual topologies are most inconsistent; the input of the topology difference measurement and generation module is connected to the output of the speech topology manifold construction module, the image topology manifold construction module, and the acoustic environment parameter monitoring module, respectively; The speech recognition module is responsible for receiving speech streams synchronously, independently of the topological manifold channel, and generating candidate defect description text by connecting to a temporal classification model. and confidence score; The visual verification and image description module is activated upon receiving a topological anomaly trigger signal. Using the current microscopic image and topological anomaly heatmap as input, a pre-trained image description generation network focuses on the anomaly region, generating purely vision-driven defect description text. ; The decision fusion and output module is used to compare topology difference metrics. With preset threshold Generate control signals; when When the speech recognition confidence level meets the standard, select the speech recognition result. As the final output, otherwise, disable the audio results and select visual verification text. Finally, the defect description is stored in the quality inspection report; the input of the decision fusion and output module is connected to the output of the topology difference measurement and generation module, the speech recognition module, and the visual verification and image description module, respectively.
[0032] One specific application of this embodiment is: This embodiment is applied to the raceway surface final inspection station in a precision bearing manufacturing workshop. Addressing the strong pulse noise environment generated by the workshop's stamping and grinding equipment, it achieves topological fusion verification between the quality inspector's verbal description of defects and industrial microscopic images. The specific implementation steps are as follows: Step S1: Synchronous acquisition and preprocessing of multimodal signals Step S11: Use a ring microphone array fixed at the quality inspection station to continuously collect single-channel voice signals when the quality inspector verbally describes defects at a sampling rate of 16kHz, so as to meet the minimum sampling accuracy requirements of industrial voice recognition.
[0033] Step S12: Use an industrial microscope camera that is synchronized with the microphone array hardware clock to continuously capture images of the bearing raceway surface at a frame rate of 30 frames per second, ensuring that the time deviation between voice and image is less than 10ms.
[0034] Step S13: Perform automatic white balance and histogram equalization processing on each frame of the microscopic image to eliminate brightness fluctuations caused by the flickering of fluorescent lights in the workshop; pass the voice signal through a high-pass filter with a cutoff frequency of 80Hz to filter out low-frequency interference caused by vibrations of the workshop structure.
[0035] Step S2: Construction of short-time spectral manifold and persistence graph sequences for speech Step S21: Using a Hamming window of length 25ms, the continuous speech signal is divided into frames with a frame shift of 10ms; the short-time Fourier transform of each frame is calculated to obtain the linear spectrum, which is then filtered by a Mel filter bank containing 64 filters to output a 64-dimensional Mel spectrum vector. .
[0036] Step S22: Use a sliding window to cover 20 consecutive frames of speech, and regard the set of points formed by all 64-dimensional Mel spectrum vectors in the window as a topological subspace point cloud in a high-dimensional space.
[0037] Step S23: Construct a complex using the Vietoris-Rips method for the point cloud within each sliding window: Set a distance threshold. When the Euclidean distance between any two points satisfies Connecting two points with an edge, and connecting three points in pairs, forms a triangle; this process continues to generate higher-dimensional simplex complexes. The process is gradually increased from 0 to the point cloud diameter, recording the birth and disappearance scales of topological features in each dimension throughout, generating the [number]th [item]. 0-dimensional persistent graph of a window With 1D continuity graph The continuous sliding sequence of graphs together constitutes a manifold representing the dynamic evolution of speech topology.
[0038] Step S3: Generation of local topological features manifold and persistence graph of the image Step S31: For the currently collected data... Frame micrographs The Scale Invariant Feature Transform (SIFT) algorithm is used to extract keypoints that are scale and rotation invariant. Each keypoint corresponds to a 128-dimensional descriptor vector. .
[0039] Step S32: Only retain key points within the 50% area of the image center, which corresponds to the core detection area of the roller track focused by the microscope camera, and remove interference from the image edge background and tooling fixtures.
[0040] Step S33: Treat the set of points formed by the filtered 128-dimensional descriptor vectors as a local topological space point cloud, and use the Vietoris-Rips complex construction framework, which is completely consistent with step S23, to calculate the... 0D persistence map of key point cloud of frame image With 1D continuity graph Since the visual scene switching speed is much lower than the audio frame rate, a single frame image already contains sufficient microscopic texture structure information. Therefore, each frame generates an independent topological snapshot, and this persistence map represents the inherent topological structure of the microscopic texture of the defect region under the current field of view.
[0041] Step S4: Measurement of Topological Differences and Dynamic Weight Adjustment Retrieve the persistence graph corresponding to the current voice window Persistence graph with synchronized image frames ,in Include A continuous point, Include There are several persistence points. Based on the Wasserstein distance, the correspondence between points in two persistence graphs is found through optimal transmission matching. The weighted sum of the absolute values of the differences in persistence values for all matching points is calculated as a measure of topological dissimilarity. The calculation formula is as follows: In the formula, ,express The Middle The persistence value of each point, This is the scale at which the topological feature disappears. This is the scale at which the topological feature was generated; express via optimal transmission mapping With the The duration value of image duration points matched with each speech duration point; To act on the first Adjust the weights based on the sensitivity of the matching points.
[0042] Sensitivity adjustment weight The harmonic noise ratio of the current speech signal With real-time ambient noise sound pressure level Dynamic joint determination, the calculation formula is as follows: In the formula The Sigmoid activation function is of the form: This maps the input to the (0,1) interval; , This is a scaling factor obtained through training with standard cleanroom samples, used to balance the magnitude of influence of the two acoustic parameters.
[0043] When the workshop is in a quiet period, , hour, Topological metrics are less sensitive to normal speech fluctuations; however, when the stamping equipment starts up, impulse noise occurs, , hour, Automatically amplify the detection sensitivity of topological distortion.
[0044] The final calculation It serves as a topological anomaly detection signal for subsequent decision-making.
[0045] Step S5: Dual-modal decision fusion and defect description output Step S51: The original speech signal is synchronously fed into the speech recognition engine based on connection-time classification, and candidate defect description text sequences are output independently. and its confidence score .
[0046] Step S52: Preset topology anomaly threshold The threshold is determined by the statistical distribution of the topological difference measure of standard defect samples verbally described by quality inspectors in a clean environment, which can effectively distinguish between normal speech fluctuations and malignant topological distortions caused by noise.
[0047] Step S53: When real-time calculation is obtained If the current speech feature manifold and image texture manifold topology are consistent and the speech signal has not undergone structural distortion, the candidate text sequence is directly adopted. The defect description at the current inspection point is stored in the structured quality inspection report.
[0048] Step S54: When real-time calculation is obtained When it is determined that the current speech is contaminated by strong impulse noise and the topology is mismatched with the visual scene, the output of the speech recognition engine is immediately blocked and the pre-trained image description generation network is activated.
[0049] While activating the visual verification, a topological anomaly map is generated: the local persistent contribution value of each key point in the image is calculated, and the contribution value is projected back into the pixel space to generate a heatmap, highlighting the areas in the image with the greatest difference from the speech topology; the image description generation network uses the current microscopic image... Using the topological anomaly heatmap as joint input, the system focuses on highlighting areas to generate purely visually driven defect description text. This is then stored in the quality inspection report as the final output.
[0050] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0051] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims
1. A speech-visual fusion method, characterized by, Includes the following steps: Step S1: Synchronously acquire the voice signal of the quality inspector describing the defects and the industrial microscopic image of the current inspection area, and preprocess the industrial microscopic image; Step S2: The speech signal is framed and its spectral features are extracted. A high-dimensional feature point cloud is constructed based on the spectral vector sequence within the sliding window to generate a persistent graph sequence representing the evolution of the speech topology. Step S3: Select an industrial microscopic image that is synchronized with the midpoint of the current speech sliding window time, extract key points with scale and rotation invariance and their descriptor vectors, construct a local topological feature manifold based on the descriptor vectors, and generate a persistence map characterizing the microscopic texture topology of the current field of view. Step S4: Based on the characteristic that there is a statistical correspondence between the speech spectrum topology and the image texture topology under stable quality inspection conditions, calculate the topological structure difference metric between the speech persistence map and the image persistence map. The topological structure difference metric represents the degree of correlation deviation between the speech topological distribution and the image topological distribution. The topological structure difference metric is obtained by weighting the harmonic noise ratio of the current speech signal and the sensitivity weight of the real-time ambient noise sound pressure level dynamic adjustment, and the topological structure difference metric is used as a topological anomaly sensing signal. At the same time, a topological anomaly map is generated according to the spatial distribution of the topological structure difference. The topological anomaly map is used to indicate the regions in the image that are inconsistent with the speech topological structure. Step S5: When the topology anomaly perception signal is less than a preset threshold, the candidate text sequence output by the speech recognition engine is directly adopted as the defect description; when the topology anomaly perception signal is greater than or equal to the preset threshold, the output of the speech recognition engine is blocked, the visual verification mechanism is activated, and the image description generation network generates defect description text based on the industrial microscopic image and the topology anomaly map.
2. The method of claim 1, wherein, Step S1 includes the following steps: Step S11: Using a microphone array fixed at the quality inspection station, continuously collect single-channel voice signals when the quality inspector verbally describes defects at a preset sampling rate of not less than 8kHz. Step S12: Using an industrial microscope camera that is synchronized with the hardware clock of the microphone array, continuously capture images of the bearing raceway surface at a preset frame rate, the preset frame rate being between 15 frames per second and 60 frames per second. Step S13: Perform automatic white balance and histogram equalization on each frame of the image to eliminate brightness fluctuations caused by the flickering of fluorescent lights in the workshop; pass the voice signal through a high-pass filter with a cutoff frequency in the range of 50Hz to 120Hz to remove low-frequency mechanical vibration interference.
3. The method of claim 1, wherein, Step S2 includes the following steps: Step S21: Using an analysis window with a length between 20ms and 40ms, the continuously flowing speech signal is divided into frames with a frame shift of 5ms to 15ms. The short-time Fourier transform of each frame signal is calculated to obtain the linear spectrum. The linear spectrum is then filtered through a set of Mel filter banks to obtain the multidimensional Mel spectrum vector. The number of Mel filter banks is between 40 and 128; Step S22: Use a sliding window to cover multiple consecutive frames, where each consecutive frame corresponds to a context of a preset duration. Treat the set of points formed by the multidimensional Mel spectrum vectors within the window as a point cloud in a high-dimensional space. Step S23: First, perform principal component analysis on the Mel spectrum vector to reduce the dimensionality to within 10 dimensions, then use the sparse Vietoris-Rips complex approximation method to generate a persistence graph, given a distance threshold. When the Euclidean distance between any two points satisfies When two points are connected by an edge, a triangle is formed when three points are connected in pairs, and so on to generate a simple complex. The range increases from 0 to the point cloud diameter, recording the birth and death information of complex topological features, and generating the first... 0-dimensional persistent graph of a window With 1D continuity graph The continuously generated persistence graph sequence constitutes a manifold describing the evolution of speech topology; the 0-dimensional and 1-dimensional persistence graphs are fused according to preset weights to obtain a unified speech persistence graph for subsequent topological difference measurement.
4. The method of claim 1, wherein, Step S3 includes the following steps: Step S31: Process the current frame image captured by the camera. The scale-invariant feature transform algorithm is used to extract keypoints with scale and rotation invariance. Each keypoint is represented by a 128-dimensional descriptor vector. Characterization; Step S32: Only retain key points within a preset proportion area in the center of the image. The preset proportion range is 30% to 70% of the center area, corresponding to the core detection area of the rolling track focused by the microscope camera, to avoid global background interference. Step S33: After dimensionality reduction of the 128-dimensional descriptor vectors, the persistence graph is calculated using the approximate persistent homology algorithm. The Vietoris-Rips complex construction framework, the same as in step S23, is used to calculate the... 0D persistence map of key point cloud of frame image With 1D continuity graph Since the visual scene switching speed is slower than the audio frame rate and the image content already contains sufficient local structural information within a single frame, each frame generates an independent topological snapshot. The persistence graph represents the inherent topological structure of the micro-texture of the defect region under the current field of view. Step S34: Treat the point cloud composed of the retained 128-dimensional descriptor vectors as a point cloud within the local topological space, and use the same Vietoris-Rips complex construction framework as in step S23 to calculate the... 0D persistence map of key point cloud of frame image With 1D continuity graph The first The frame image is an image frame that is synchronized with the hardware clock at the midpoint of the current speech sliding window time; the 0-dimensional and 1-dimensional persistence maps are fused according to preset weights to obtain a unified image persistence map for subsequent topological difference measurement; the visual scene switching speed is slower than the speech frame rate and the image content already contains sufficient local structural information in a single frame, so each frame generates an independent topological snapshot, and the persistence map represents the inherent topological structure of the micro-texture of the defect region under the current field of view.
5. The method of claim 1, wherein, In step S4, the topological difference metric is calculated as follows: Take the unified persistence graph of the current voice window after fusion. With the fused unified persistence map corresponding to the current image frame , Includes A continuous point, Includes Each persistent point is mapped to a birth-death two-dimensional plane with its birth scale as the x-axis and its death scale as the y-axis. Before semantic content, malicious distortions of speech features caused by industrial impulse noise are captured at the topological level. Based on Wasserstein distance, the Euclidean distance between two points on the birth-death two-dimensional plane is used as the matching cost function. With the constraint that each persistent point participates in only one matching, a linear programming method is used to solve for the optimal transmission matching relationship. Determine the correspondence between points in two persistence graphs; calculate the weighted sum of the absolute values of the differences in persistence values for all matching point pairs, as a measure of topological dissimilarity. , The magnitude of the value directly reflects the degree of deviation of the statistical correlation between the speech spectrum topology and the image texture topology. The larger the value, the more serious the noise pollution of the speech signal and the higher the degree of destruction of the topological statistical correspondence under stable conditions. The calculation formula is as follows: In the formula, express The Middle The persistence value of each point, This is the scale at which the topological feature disappears. This is the scale at which this topological feature was generated. That is, the lifespan of this topological feature from its birth to its extinction; Indicates in via optimal transmission matching and The Middle The persistence value of each point corresponding to a given point; To act on the first Adjust the weights based on the sensitivity of the matching points.
6. The speech-visual fusion method according to claim 5, characterized in that, The sensitivity adjustment weight The harmonic noise ratio of the current speech signal and real-time collected ambient noise sound pressure level Dynamic joint determination enables the topological difference metric to be adaptively sensitive to the drastically changing noise environment in industrial settings. Reduce or All increases will push up This amplifies the calculated topological differences at the moment of noise pollution; Calculated using the following formula: In the formula, The sigmoid function maps the input to the (0,1) interval. The more noisy the environment, the more... The closer the value is to 1, the more sensitive the topological difference measure becomes to small distortions; and It is a scaling factor obtained in advance through training with a small number of standard samples, used to balance the magnitude of influence of the two acoustic parameters, harmonic noise ratio and ambient noise sound pressure level. It is a positive number to ensure that the input increases and the weight increases when the HNR decreases.
7. The speech-visual fusion method according to claim 1, characterized in that, Step S5 includes the following steps: Step S51: Synchronously feed the speech signal into the speech recognition engine based on connection-time classification to obtain candidate text sequences. and confidence score The system combines topological anomaly perception signals with speech recognition confidence to make joint decisions. When a topological anomaly occurs and the confidence level is below a threshold, a visual review is triggered. Step S52: Preset topology anomaly threshold With the speech confidence threshold p th The threshold The threshold is determined based on the statistical distribution of the topological difference measurement values of standard samples verbally described by quality inspectors in a clean environment, so that the threshold can distinguish between normal fluctuations in voice signals and malignant topological distortions caused by noise. Step S53: When the topology difference metric value is calculated in real time And the confidence level of speech recognition ≥p th If the topological structure of the speech feature manifold is consistent with that of the current image texture manifold, and the speech signal has not undergone malignant distortions that affect the structure, the candidate text sequence is directly adopted. The defect description at the current inspection point is stored in the quality inspection report; Step S54: When the topology difference metric value is calculated in real time or speech recognition confidence p th If the current speech signal is found to be severely contaminated by noise and the speech manifold structure is mismatched with the visual scene, the output of the speech recognition engine is blocked, and a pre-trained image description generation network is activated. This image description generation network uses the current microscopic image... Using the topological anomaly map as joint input, a vision-driven defect description text is generated. .
8. The speech-visual fusion method according to claim 7, characterized in that, The generation method of the topology anomaly map in step S54 is as follows: calculate the weighted sum of the persistence value deviation of the persistence point corresponding to each key point in the optimal transmission matching and the spatial matching distance, as the local persistence contribution value of the point; according to the pixel coordinates of the key point, the contribution value is interpolated and projected onto the entire image to obtain the topology anomaly heatmap. The heatmap highlights the region in the image that is most inconsistent with the speech topology structure, and the image description generation network focuses on the highlighted region to generate text.
9. A voice-visual fusion system, characterized in that, It includes a multimodal acquisition and processing module, a speech topology manifold construction module, an image topology manifold construction module, an acoustic environment parameter monitoring module, a topology difference measurement and generation module, a speech recognition module, a visual verification and image description module, and a decision fusion and output module; The multimodal acquisition and processing module simultaneously acquires the quality inspector's spoken voice and industrial microscopic images; it includes a microphone array, an industrial microscopic camera, and a hardware clock synchronization unit. The preprocessing unit performs automatic white balance and histogram equalization on the image and removes low-frequency vibration interference from the speech signal using a high-pass filter; the output of the multimodal acquisition and processing module is connected to the input of the speech topology manifold construction module, the image topology manifold construction module, and the acoustic environment parameter monitoring module, respectively. The speech topology manifold construction module is used to receive the preprocessed speech stream, perform framing, short-time Fourier transform and Mel filtering, and generate a multidimensional Mel spectrum vector sequence; point cloud is formed through a sliding window, and speech persistence graph sequence is generated using the Vietoris-Rips complex construction method to characterize the dynamic evolution of the speech topology. The image topology manifold construction module is used to receive the current frame microscopic image, extract key points and 128-dimensional descriptors using the scale-invariant feature transformation algorithm; retain key points in the core detection area by filtering the central region, construct descriptor point clouds, and generate image persistence maps using the same complex construction method as the speech module to characterize the inherent topological structure of the microscopic texture in the current field of view. The acoustic environment parameter monitoring module is used to calculate the harmonic noise ratio of the speech signal in real time. and collecting ambient noise sound pressure level This provides input for subsequent dynamic weight adjustment; The topology difference measurement and generation module is used to receive the speech persistence map. and image persistence graph By calculating the weighted Wasserstein distance using optimal transmission matching, a measure of topological dissimilarity is obtained. ; Weight Depend on and Dynamic adjustment; simultaneously, the topology difference measurement and generation module generates a topology anomaly heatmap based on the local persistent contribution value of each key point, indicating the region where the speech and visual topologies are most inconsistent; the input of the topology difference measurement and generation module is connected to the output of the speech topology manifold construction module, the image topology manifold construction module, and the acoustic environment parameter monitoring module, respectively; The speech recognition module is responsible for receiving speech streams synchronously, independently of the topological manifold channel, and generating candidate defect description text by connecting to a temporal classification model. and confidence score; The visual verification and image description module is activated upon receiving a topology anomaly trigger signal. Using the current microscopic image and topology anomaly heatmap as input, a pre-trained image description generation network focuses on the anomaly region, generating purely vision-driven defect description text. ; The decision fusion and output module is used to compare topology difference metrics. With preset threshold Generate control signals; when When the speech recognition confidence level meets the standard, select the speech recognition result. As the final output, otherwise, disable the audio results and select visual verification text. Finally, the defect description is stored in the quality inspection report; the input end of the decision fusion and output module is connected to the output end of the topology difference measurement and generation module, the speech recognition module, and the visual verification and image description module, respectively.
