Sound detection method, device and equipment based on machine vision, and medium

Abstract

The application discloses a sound detection method and device based on machine vision, equipment and medium, and belongs to the technical field of acoustic detection. In the application, the acoustic quality of the intelligent sound in the mass production stage is detected through the automatic detection technology of machine vision, so as to ensure the product quality. First, the recorded audio of the to-be-detected sound when playing the preset test audio is converted into a time domain image and a frequency domain image. Then, the recorded audio is detected according to the time domain image, and the recorded audio is detected according to the frequency domain image. If the abnormal audio detection and the frequency spectrum detection are qualified, the to-be-detected sound is determined to be qualified.

Description

Technical Field

[0001] This application relates to the technical field of acoustic testing, and in particular to a machine vision-based sound testing method, a machine vision-based sound testing device, a machine vision-based sound testing equipment, and a computer-readable storage medium. Background Technology

[0002] Smart speakers combine traditional speakers with emerging technologies such as voice recognition and natural language processing, enabling them to simultaneously provide audio playback, intelligent voice interaction, and smart home control. Due to their rich functionality, smart speakers are considered the control hub of smart homes, thus becoming one of the fastest-growing electronic products today. Audio playback is the fundamental function of smart speakers; therefore, ensuring their acoustic performance is a basic requirement for quality, necessitating the testing of acoustic parameters during the manufacturing process.

[0003] Currently, in the traditional manufacturing of smart speakers, the acoustic performance is often evaluated by listening to the sound in a soundproof room next to the production line during the mass production stage. This testing method is labor-intensive, reduces testing efficiency, has poor robustness, is greatly affected by subjective factors, and long-term testing can lead to auditory fatigue of inspectors, affecting the hearing health of employees and making it difficult to ensure product quality consistency. Summary of the Invention

[0004] The main objective of this application is to provide a machine vision-based sound detection method, a machine vision-based sound detection device, a machine vision-based sound detection equipment, and a computer-readable storage medium, which aim to accurately detect the acoustic quality of sound.

[0005] To achieve the above objectives, this application provides a machine vision-based sound detection method, the method comprising:

[0006] Acquire the audio recording of the speaker under test while playing a preset test audio, and convert the audio recording into a time-domain image and a frequency-domain image;

[0007] Abnormal audio detection is performed on the recorded audio based on the time-domain image, and spectrum detection is performed on the recorded audio based on the frequency-domain image;

[0008] If both the abnormal audio detection and the spectrum detection of the recorded audio of the speaker under test are qualified, then the speaker under test is determined to be qualified.

[0009] For example, the step of detecting abnormal audio based on the time-domain image of the recorded audio includes:

[0010] Select the first ROI region corresponding to the preset test audio on the time domain image, and determine whether the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio;

[0011] If the image features of the first ROI region do not match the preset abnormal sound features corresponding to the preset test audio, it is determined that the recorded audio has abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is unqualified.

[0012] If the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio, then it is determined that the recorded audio does not have any abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is qualified.

[0013] For example, the step of performing spectrum detection on the recorded audio based on the frequency domain image includes:

[0014] Select the second ROI region corresponding to the preset test audio on the frequency domain image, and determine whether the image features of the second ROI region match the preset spectral features corresponding to the preset test audio.

[0015] If the image features of the second ROI region do not match the preset spectral features corresponding to the preset test audio, then the recorded audio spectrum is determined to be abnormal, and the spectrum detection of the recorded audio of the speaker under test is determined to be unqualified.

[0016] If the image features of the second ROI region match the preset spectral features corresponding to the preset test audio, then the spectrum of the recorded audio is determined to be normal, and the spectrum detection of the recorded audio of the speaker under test is determined to be qualified.

[0017] For example, before the steps of performing abnormal audio detection on the recorded audio based on the time-domain image and performing spectrum detection on the recorded audio based on the frequency-domain image, the method includes:

[0018] Obtain the basic image features of the time-domain image and the basic image features of the frequency-domain image;

[0019] If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed;

[0020] If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing spectrum detection on the recorded audio based on the frequency domain image is executed.

[0021] For example, the step of obtaining the basic image features of the time-domain image includes:

[0022] The time-domain image is segmented to obtain a time-domain waveform region image, and the waveform of the time-domain waveform region image is obtained.

[0023] The time-domain waveform region image is subjected to image binarization processing to obtain a binarized time-domain waveform image, and edge detection is performed on the binarized time-domain waveform image to determine the envelope of the waveform;

[0024] Based on the envelope of the waveform, the basic image features of the time-domain image are determined, wherein the basic image features include the zero-crossing rate and the onset time;

[0025] If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed.

[0026] For example, the step of obtaining the basic image features of the frequency domain image includes:

[0027] The frequency domain image is segmented to obtain a frequency domain waveform region image, and the waveform of the frequency domain waveform region image is obtained.

[0028] The frequency domain waveform region image is subjected to image binarization processing to obtain a binarized frequency domain waveform image, and edge detection is performed on the binarized frequency domain waveform image to determine the envelope of the waveform;

[0029] Based on the envelope of the waveform, the basic image features of the frequency domain image are determined, wherein the basic image features include the spectral centroid and the fundamental harmonic frequency;

[0030] If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing spectrum detection on the recorded audio based on the frequency domain image is executed.

[0031] For example, the method further includes:

[0032] Obtain the key test points of the speaker under test, and obtain the basic electrical parameters of the speaker under test at the key test points when playing a preset test audio.

[0033] If the abnormal audio detection and spectrum detection of the recorded audio of the speaker under test are both qualified, and the basic electrical parameters meet the preset electrical parameter standards, then the speaker under test is determined to be qualified.

[0034] This application also provides a machine vision-based sound detection device, the machine vision-based sound detection device comprising:

[0035] The acquisition module is used to acquire the audio recordings made by the speaker under test when playing a preset test audio, and to convert the audio recordings into time-domain images and frequency-domain images;

[0036] The detection module is used to perform abnormal audio detection on the recorded audio based on the time-domain image and to perform spectrum detection on the recorded audio based on the frequency-domain image.

[0037] The determination module is used to determine that the audio device under test is qualified if both the abnormal audio detection and the spectrum detection of the recorded audio of the audio device under test are qualified.

[0038] This application also provides a machine vision-based audio detection device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the machine vision-based audio detection method as described above.

[0039] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the machine vision-based sound detection method described above.

[0040] This application proposes a machine vision-based audio detection method, a machine vision-based audio detection device, a machine vision-based audio detection equipment, and a computer-readable storage medium. The method involves acquiring a recording of the audio device under test while playing a preset test audio, and converting the recording into a time-domain image and a frequency-domain image. Abnormal audio detection is performed on the recording based on the time-domain image, and spectrum detection is performed on the recording based on the frequency-domain image. If both the abnormal audio detection and the spectrum detection of the recording of the audio device under test are qualified, the audio device under test is determined to be qualified.

[0041] In this application, automated inspection technology based on machine vision is used to inspect the acoustic quality of smart speakers in the mass production stage, thereby ensuring product quality. First, the audio recordings of the speaker under test, played while playing preset test audio, are converted into time-domain and frequency-domain images. Then, abnormal audio detection is performed on the recorded audio based on the time-domain image, and spectrum analysis is performed on the recorded audio based on the frequency-domain image. If both the abnormal audio detection and the spectrum analysis are qualified, the speaker under test is deemed qualified. To ensure the quality of smart speakers, their acoustic performance needs to be tested during the production line stage. Manual inspection requires significant manpower, and prolonged testing can cause auditory fatigue, while also being prone to missed or incorrect detections. This machine vision-based smart speaker production testing method achieves automated acoustic inspection on the production line, accurately detecting the acoustic intelligence of the speakers. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the structure of the operating device of the hardware operating environment involved in the embodiments of this application;

[0043] Figure 2 This is a schematic flowchart of an embodiment of the machine vision-based sound detection method involved in the present application.

[0044] Figure 3 This is a schematic diagram of a PCB incoming material inspection algorithm according to an embodiment of the machine vision-based audio detection method involved in the present application.

[0045] Figure 4 This is a schematic diagram of the PCB incoming material inspection process of an embodiment of the machine vision-based audio detection method involved in the present application.

[0046] Figure 5 This is a schematic diagram illustrating an embodiment of the machine vision-based sound detection method involved in the embodiments of this application.

[0047] Figure 6 This is a schematic diagram of a machine vision-based audio detection device involved in an embodiment of this application.

[0048] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0050] Reference Figure 1 , Figure 1 This is a schematic diagram of the operating device structure of the hardware operating environment involved in the embodiments of this application.

[0051] like Figure 1As shown, the operating device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0052] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the operating equipment and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0053] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and computer programs.

[0054] exist Figure 1 In the illustrated operating device, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the operating device of this application can be disposed in the operating device, and the operating device calls the computer program stored in the memory 1005 through the processor 1001 and performs the following operations:

[0055] Acquire the audio recording of the speaker under test while playing a preset test audio, and convert the audio recording into a time-domain image and a frequency-domain image;

[0056] Abnormal audio detection is performed on the recorded audio based on the time-domain image, and spectrum detection is performed on the recorded audio based on the frequency-domain image;

[0057] If both the abnormal audio detection and the spectrum detection of the recorded audio of the speaker under test are qualified, then the speaker under test is determined to be qualified.

[0058] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:

[0059] The step of detecting abnormal audio based on the time-domain image includes:

[0060] Select the first ROI region corresponding to the preset test audio on the time domain image, and determine whether the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio;

[0061] If the image features of the first ROI region do not match the preset abnormal sound features corresponding to the preset test audio, it is determined that the recorded audio has abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is unqualified.

[0062] If the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio, then it is determined that the recorded audio does not have any abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is qualified.

[0063] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:

[0064] The step of performing spectrum detection on the recorded audio based on the frequency domain image includes:

[0065] Select the second ROI region corresponding to the preset test audio on the frequency domain image, and determine whether the image features of the second ROI region match the preset spectral features corresponding to the preset test audio.

[0066] If the image features of the second ROI region do not match the preset spectral features corresponding to the preset test audio, then the recorded audio spectrum is determined to be abnormal, and the spectrum detection of the recorded audio of the speaker under test is determined to be unqualified.

[0067] If the image features of the second ROI region match the preset spectral features corresponding to the preset test audio, then the spectrum of the recorded audio is determined to be normal, and the spectrum detection of the recorded audio of the speaker under test is determined to be qualified.

[0068] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:

[0069] Before the steps of performing abnormal audio detection on the recorded audio based on the time-domain image and performing spectrum detection on the recorded audio based on the frequency-domain image, the following steps are included:

[0070] Obtain the basic image features of the time-domain image and the basic image features of the frequency-domain image;

[0071] If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed;

[0072] If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing spectrum detection on the recorded audio based on the frequency domain image is executed.

[0073] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:

[0074] The step of obtaining the basic image features of the time-domain image includes:

[0075] The time-domain image is segmented to obtain a time-domain waveform region image, and the waveform of the time-domain waveform region image is obtained.

[0076] The time-domain waveform region image is subjected to image binarization processing to obtain a binarized time-domain waveform image, and edge detection is performed on the binarized time-domain waveform image to determine the envelope of the waveform;

[0077] Based on the envelope of the waveform, the basic image features of the time-domain image are determined, wherein the basic image features include the zero-crossing rate and the onset time;

[0078] If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed.

[0079] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:

[0080] The step of obtaining the basic image features of the frequency domain image includes:

[0081] The frequency domain image is segmented to obtain a frequency domain waveform region image, and the waveform of the frequency domain waveform region image is obtained.

[0082] The frequency domain waveform region image is subjected to image binarization processing to obtain a binarized frequency domain waveform image, and edge detection is performed on the binarized frequency domain waveform image to determine the envelope of the waveform;

[0083] Based on the envelope of the waveform, the basic image features of the frequency domain image are determined, wherein the basic image features include the spectral centroid and the fundamental harmonic frequency;

[0084] If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing spectrum detection on the recorded audio based on the frequency domain image is executed.

[0085] In one embodiment, the processor 1001 may invoke a computer program stored in the memory 1005 and further perform the following operations:

[0086] The method further includes:

[0087] Obtain the key test points of the speaker under test, and obtain the basic electrical parameters of the speaker under test at the key test points when playing a preset test audio.

[0088] If the abnormal audio detection and spectrum detection of the recorded audio of the speaker under test are both qualified, and the basic electrical parameters meet the preset electrical parameter standards, then the speaker under test is determined to be qualified.

[0089] This application provides a machine vision-based sound detection method, referring to... Figure 2 In one embodiment of the machine vision-based sound detection method, the method includes:

[0090] Step S10: Obtain the audio recording of the speaker under test when playing a preset test audio, and convert the audio recording into a time-domain image and a frequency-domain image;

[0091] To ensure the acoustic performance of smart speakers, quality testing is required during the production line phase. Testing is conducted on the production line by first playing a preset test audio. In one embodiment, three sets of tests are performed in parallel: manual monitoring, instrument measurement, and loopback. Manual monitoring involves recording audio using a standard microphone, with manual sampling to check acoustic performance remotely. Instrument measurement involves measuring basic parameters such as voltage and current (provided that test points are reserved during the product development of the smart speaker). Loopback uses the smart speaker's built-in microphone to record audio, sending the audio signal via serial port to a host computer. The host computer converts the audio signal into an image, and then uses a machine vision-based detection algorithm to evaluate the acoustic performance.

[0092] In one embodiment, if the test passes, the product is deemed qualified and sent to the next workstation; if the test fails, the information is uploaded to the expert system and the test data is analyzed, and subsequent repairs or other processing can be carried out.

[0093] Step S20: Perform abnormal audio detection on the recorded audio based on the time domain image, and perform spectrum detection on the recorded audio based on the frequency domain image;

[0094] The Loopback test involves recording audio using the smart speaker's microphone while the speaker is playing audio. The recorded audio signal is then transmitted to a host computer to execute an audio detection algorithm, ensuring comprehensive acoustic performance.

[0095] In one embodiment, reference is made to Figure 3 First, the host computer receives the audio signal via serial port, then converts the signal into time-domain and frequency-domain images, which are then detected in parallel. The audio signal is a digital signal, and information such as the total number of frames, sampling rate, and amplitude can be used to convert it into an image. Then, abnormal audio detection is performed on the recorded audio based on the time-domain image, and spectrum analysis is performed on the recorded audio based on the frequency-domain image.

[0096] For example, the step of detecting abnormal audio based on the time-domain image of the recorded audio includes:

[0097] Select the first ROI region corresponding to the preset test audio on the time domain image, and determine whether the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio;

[0098] If the image features of the first ROI region do not match the preset abnormal sound features corresponding to the preset test audio, it is determined that the recorded audio has abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is unqualified.

[0099] If the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio, then it is determined that the recorded audio does not have any abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is qualified.

[0100] When performing abnormal audio detection on recorded audio based on time-domain images, specific Regions of Interest (ROIs) are first selected according to the different test audio being played. Then, abnormal sound detection is performed, examining the image features of the ROIs (using template matching or machine learning) and verifying whether they match preset features. If the ROI features do not match the preset features, it is determined that there is abnormal sound, and the product is rejected; otherwise, the abnormal audio detection is considered successful.

[0101] For example, the step of performing spectrum detection on the recorded audio based on the frequency domain image includes:

[0102] Select the second ROI region corresponding to the preset test audio on the frequency domain image, and determine whether the image features of the second ROI region match the preset spectral features corresponding to the preset test audio.

[0103] If the image features of the second ROI region do not match the preset spectral features corresponding to the preset test audio, then the recorded audio spectrum is determined to be abnormal, and the spectrum detection of the recorded audio of the speaker under test is determined to be unqualified.

[0104] If the image features of the second ROI region match the preset spectral features corresponding to the preset test audio, then the spectrum of the recorded audio is determined to be normal, and the spectrum detection of the recorded audio of the speaker under test is determined to be qualified.

[0105] When performing spectrum detection on recorded audio based on the frequency domain image, the time-domain signal is first converted to the frequency domain using a Short-Time Fourier Transform (STFT) to obtain its frequency domain waveform image. Then, specific Regions of Interest (ROIs) are selected based on the different test audio being played, and spectrum detection is performed to detect the image features of the ROIs (using template matching or machine learning) and verify whether they match preset features. If the ROI features do not match the preset features, it is judged as NG; otherwise, the spectrum detection is judged as qualified.

[0106] For example, before the steps of performing abnormal audio detection on the recorded audio based on the time-domain image and performing spectrum detection on the recorded audio based on the frequency-domain image, the method includes:

[0107] Obtain the basic image features of the time-domain image and the basic image features of the frequency-domain image;

[0108] If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed;

[0109] If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing spectrum detection on the recorded audio based on the frequency domain image is executed.

[0110] In one embodiment, reference is made to Figure 3 Before performing abnormal audio detection and spectrum detection, the basic features need to be verified. Basic image features of the time-domain image and the frequency-domain image are acquired. Only if the basic image features of the time-domain image meet the preset time-domain feature standards will the step of abnormal audio detection based on the time-domain image be executed; similarly, only if the basic image features of the frequency-domain image meet the preset frequency-domain feature standards will the step of spectrum detection based on the frequency-domain image be executed.

[0111] For example, the step of obtaining the basic image features of the time-domain image includes:

[0112] The time-domain image is segmented to obtain a time-domain waveform region image, and the waveform of the time-domain waveform region image is obtained.

[0113] The time-domain waveform region image is subjected to image binarization processing to obtain a binarized time-domain waveform image, and edge detection is performed on the binarized time-domain waveform image to determine the envelope of the waveform;

[0114] Based on the envelope of the waveform, the basic image features of the time-domain image are determined, wherein the basic image features include the zero-crossing rate and the onset time;

[0115] If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed.

[0116] When acquiring the basic image features of a time-domain image, after denoising the image using a spatial filter, the image is segmented to obtain the waveform region, and its waveform can be acquired. The image is then binarized, and an edge detection algorithm (such as using a differential operator) is used to obtain the waveform envelope. Then, various basic features of the time-domain image are detected, including zero-crossing rate: the waveform curve is tracked, and the number of times it crosses the zero point (when there is no signal) is recorded. Onset time: the envelope is tracked, and the rise time is recorded. If the basic features do not meet the standards, it is judged as NG; otherwise, subsequent abnormal audio detection steps are performed.

[0117] For example, the step of obtaining the basic image features of the frequency domain image includes:

[0118] The frequency domain image is segmented to obtain a frequency domain waveform region image, and the waveform of the frequency domain waveform region image is obtained.

[0119] The frequency domain waveform region image is subjected to image binarization processing to obtain a binarized frequency domain waveform image, and edge detection is performed on the binarized frequency domain waveform image to determine the envelope of the waveform;

[0120] Based on the envelope of the waveform, the basic image features of the frequency domain image are determined, wherein the basic image features include the spectral centroid and the fundamental harmonic frequency;

[0121] If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing spectrum detection on the recorded audio based on the frequency domain image is executed.

[0122] When acquiring the basic image features of a frequency domain image, image segmentation yields waveform regions, from which the frequency domain waveform can be obtained. Image binarization and edge detection algorithms (such as those using differential operators) provide the envelope of the frequency domain waveform. Then, various basic features of the frequency domain image are detected, including the spectral centroid: traversing the envelope and calculating its first moment yields the spectral centroid, where is the x-coordinate of the envelope and is the corresponding y-coordinate. Fundamental harmonic detection: the envelope is traced by curves, and the x and y coordinates corresponding to extreme points are found to determine the frequencies and values ​​of the fundamental and harmonic waves. If the basic features do not meet the criteria, the image is considered NG; otherwise, subsequent spectral detection steps are performed.

[0123] Step S30: If both the abnormal audio detection and the spectrum detection of the recorded audio of the speaker under test are qualified, then the speaker under test is determined to be qualified.

[0124] If both the abnormal audio detection and spectrum detection pass, the product is sent to the next workstation. If an NG (Not Acceptable) result is found, the next step is to input the problem information into the expert system for analysis. The expert system is a software system that analyzes the abnormal situation, classifies the causes of acoustic performance abnormalities into circuit abnormalities, structural abnormalities, speaker abnormalities, and others, proposes repair or other handling suggestions, and determines whether manual re-inspection or manual rework is required.

[0125] For example, the method further includes:

[0126] Obtain the key test points of the speaker under test, and obtain the basic electrical parameters of the speaker under test at the key test points when playing a preset test audio.

[0127] If the abnormal audio detection and spectrum detection of the recorded audio of the speaker under test are both qualified, and the basic electrical parameters meet the preset electrical parameter standards, then the speaker under test is determined to be qualified.

[0128] In one embodiment, with the smart speaker's test points exposed, key test points of the speaker under test are obtained. The basic electrical parameters of the speaker under test at these key test points are acquired when playing a preset test audio. The voltage and current at these key test points during audio playback are measured using instruments to ensure that the electrical indicators related to audio playback are normal. If the abnormal audio detection and spectrum detection of the recorded audio of the speaker under test are both qualified, and the basic electrical parameters meet the preset electrical parameter standards, then the speaker under test is determined to be qualified.

[0129] In one application scenario of the machine vision-based sound detection method of this application, referring to... Figure 4The production testing of smart speakers based on machine vision is divided into two parts: the pre-production line preparation stage and the production line testing stage. In the pre-production line preparation stage, a stable testing environment is ensured, and metal-machine testing is performed for verification. This includes setting up the testing environment and conducting metal-machine testing. A soundproof enclosure is used to build the smart speaker production testing environment, ensuring sound insulation and eliminating sound source interference. The product under test and a standard microphone (for manual sampling) are fixed in place within the soundproof enclosure. After the testing environment is set up, it needs to be debugged. A metal-machine tester (with all parameters at their optimal level) is used to verify the consistency and stability of the testing environment and to calibrate the testing environment parameters.

[0130] During the production line testing phase, three tests are performed in parallel: manual listening, instrument measurement, and loopback, while the product under test plays audio according to the testing requirements. Manual listening is conducted on an irregular, random basis to verify the stability of the automated testing method through human intervention. Instrument measurement and loopback are the main parts of the testing; only products that pass both tests will proceed to the next workstation; otherwise, they undergo other processing.

[0131] During random manual inspections, a standard microphone is activated to remotely transmit recorded audio to a laboratory for human listening and evaluation of the speaker's acoustic performance. With the smart speaker's test points exposed, instruments are used to measure the voltage and current at key test points during audio playback to ensure that its electrical parameters related to audio playback are normal. Loopback testing is as follows... Figure 3 As shown.

[0132] Reference Figure 5 The machine vision-based audio detection process is detailed below:

[0133] (1) Set up the testing environment for the production line and conduct metalworking tests;

[0134] (2) Send the smart speaker under test into the test environment and play the test audio.

[0135] (3) Three sets of tests are conducted in parallel, and some products need to be sampled and manually monitored.

[0136] (4) Use instruments to measure basic electrical parameters in parallel. If the parameters are not qualified, they are judged as NG.

[0137] (5) Perform Loopback testing in parallel, and transmit the audio recorded by the built-in microphone of the speaker to the host computer; execute the audio detection algorithm based on machine vision in the host computer, and judge it as NG if it is unqualified.

[0138] (6) Input the information of NG products into the expert system for analysis, and perform rework or other processing according to the processing suggestions of the expert system;

[0139] (7) If all the above tests are qualified, the product is deemed PASS and allowed to flow into the subsequent work station. At this point, the production test of the smart speaker based on machine vision is completed.

[0140] This invention provides an automated testing method for intelligent speaker production lines, replacing manual audio quality inspection and improving testing efficiency and accuracy, thus ensuring product quality. Furthermore, the system integrates with expert systems to quickly handle product faults and optimize product design and processes, thereby enhancing the level of intelligence in manufacturing. This method features high integration, high reliability, high automation, and a certain degree of intelligence, reducing testing costs, improving testing efficiency, and can be extended to other products with audio playback functions.

[0141] Reference Figure 6 Furthermore, embodiments of this application also provide a machine vision-based sound detection device, which includes:

[0142] The acquisition module M1 is used to acquire the audio recording of the speaker under test when playing a preset test audio, and convert the audio recording into a time domain image and a frequency domain image;

[0143] The detection module M2 is used to perform abnormal audio detection on the recorded audio based on the time domain image and to perform spectrum detection on the recorded audio based on the frequency domain image.

[0144] The determination module M3 is used to determine that the audio device under test is qualified if both the abnormal audio detection and the spectrum detection of the recorded audio of the audio device under test are qualified.

[0145] For example, the detection module is further configured to:

[0146] Select the first ROI region corresponding to the preset test audio on the time domain image, and determine whether the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio;

[0147] If the image features of the first ROI region do not match the preset abnormal sound features corresponding to the preset test audio, it is determined that the recorded audio has abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is unqualified.

[0148] If the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio, then it is determined that the recorded audio does not have any abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is qualified.

[0149] For example, the detection module is further configured to:

[0150] Select the second ROI region corresponding to the preset test audio on the frequency domain image, and determine whether the image features of the second ROI region match the preset spectral features corresponding to the preset test audio.

[0151] If the image features of the second ROI region do not match the preset spectral features corresponding to the preset test audio, then the recorded audio spectrum is determined to be abnormal, and the spectrum detection of the recorded audio of the speaker under test is determined to be unqualified.

[0152] If the image features of the second ROI region match the preset spectral features corresponding to the preset test audio, then the spectrum of the recorded audio is determined to be normal, and the spectrum detection of the recorded audio of the speaker under test is determined to be qualified.

[0153] For example, the detection module is further configured to:

[0154] Before the steps of performing abnormal audio detection on the recorded audio based on the time-domain image and performing spectrum detection on the recorded audio based on the frequency-domain image.

[0155] Obtain the basic image features of the time-domain image and the basic image features of the frequency-domain image;

[0156] If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed;

[0157] If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing spectrum detection on the recorded audio based on the frequency domain image is executed.

[0158] For example, the detection module is further configured to:

[0159] The time-domain image is segmented to obtain a time-domain waveform region image, and the waveform of the time-domain waveform region image is obtained.

[0160] The time-domain waveform region image is subjected to image binarization processing to obtain a binarized time-domain waveform image, and edge detection is performed on the binarized time-domain waveform image to determine the envelope of the waveform;

[0161] Based on the envelope of the waveform, the basic image features of the time-domain image are determined, wherein the basic image features include the zero-crossing rate and the onset time;

[0162] If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed.

[0163] For example, the detection module is further configured to:

[0164] The frequency domain image is segmented to obtain a frequency domain waveform region image, and the waveform of the frequency domain waveform region image is obtained.

[0165] The frequency domain waveform region image is subjected to image binarization processing to obtain a binarized frequency domain waveform image, and edge detection is performed on the binarized frequency domain waveform image to determine the envelope of the waveform;

[0166] Based on the envelope of the waveform, the basic image features of the frequency domain image are determined, wherein the basic image features include the spectral centroid and the fundamental harmonic frequency;

[0167] If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing spectrum detection on the recorded audio based on the frequency domain image is executed.

[0168] For example, the determining module is further configured to:

[0169] Obtain the key test points of the speaker under test, and obtain the basic electrical parameters of the speaker under test at the key test points when playing a preset test audio.

[0170] If the abnormal audio detection and spectrum detection of the recorded audio of the speaker under test are both qualified, and the basic electrical parameters meet the preset electrical parameter standards, then the speaker under test is determined to be qualified.

[0171] The machine vision-based audio detection device provided in this application employs the machine vision-based audio detection method described in the above embodiments, aiming to accurately detect the acoustic quality of audio. Compared with conventional technologies, the beneficial effects of the machine vision-based audio detection device provided in this application are the same as those of the machine vision-based audio detection method described in the above embodiments, and other technical features in the machine vision-based audio detection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0172] Furthermore, this application embodiment also provides a machine vision-based audio detection device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the machine vision-based audio detection method as described above.

[0173] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the machine vision-based sound detection method described above.

[0174] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to conventional technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0176] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A machine vision-based sound detection method, characterized in that, The method includes: The audio recording of the speaker under test while playing a preset test audio is obtained, and the audio recording is converted into a time domain image and a frequency domain image according to the total number of frames, sampling rate and amplitude of the audio recording; Based on the time-domain image, abnormal audio detection is performed on the recorded audio using machine vision; based on the frequency-domain image, spectrum detection is performed on the recorded audio using machine vision. If both the abnormal audio detection and the spectrum detection of the recorded audio of the speaker under test are qualified, then the speaker under test is determined to be qualified. The step of performing machine vision-based spectrum detection on the recorded audio based on the frequency domain image includes: The second ROI region corresponding to the preset test audio is selected on the frequency domain image, and the image features of the second ROI region are determined to match the preset spectral features corresponding to the preset test audio by template matching or machine learning. If the image features of the second ROI region do not match the preset spectral features corresponding to the preset test audio, then the recorded audio spectrum is determined to be abnormal, and the spectrum detection of the recorded audio of the speaker under test is determined to be unqualified. If the image features of the second ROI region match the preset spectral features corresponding to the preset test audio, then the spectrum of the recorded audio is determined to be normal, and the spectrum detection of the recorded audio of the speaker under test is determined to be qualified.

2. The machine vision-based sound detection method as described in claim 1, characterized in that, The step of performing machine vision-based abnormal audio detection on the recorded audio based on the time-domain image includes: Select the first ROI region corresponding to the preset test audio on the time domain image, and determine whether the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio; If the image features of the first ROI region do not match the preset abnormal sound features corresponding to the preset test audio, it is determined that the recorded audio has abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is unqualified. If the image features of the first ROI region match the preset abnormal sound features corresponding to the preset test audio, then it is determined that the recorded audio does not have any abnormal sound, and the abnormal audio detection of the recorded audio of the speaker under test is qualified.

3. The machine vision-based sound detection method as described in claim 1, characterized in that, Before the steps of performing machine vision-based abnormal audio detection on the recorded audio based on the time-domain image and performing machine vision-based spectrum detection on the recorded audio based on the frequency-domain image, the procedure includes: Obtain the basic image features of the time-domain image and the basic image features of the frequency-domain image; If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed. If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing machine vision-based spectrum detection on the recorded audio based on the frequency domain image is executed.

4. The machine vision-based sound detection method as described in claim 3, characterized in that, The step of obtaining the basic image features of the time-domain image includes: The time-domain image is segmented to obtain a time-domain waveform region image, and the waveform of the time-domain waveform region image is obtained. The time-domain waveform region image is subjected to image binarization processing to obtain a binarized time-domain waveform image, and edge detection is performed on the binarized time-domain waveform image to determine the envelope of the waveform; Based on the envelope of the waveform, the basic image features of the time-domain image are determined, wherein the basic image features include the zero-crossing rate and the onset time; If the basic image features of the time-domain image meet the preset time-domain feature standard, then the step of performing abnormal audio detection on the recorded audio based on the time-domain image is executed.

5. The machine vision-based sound detection method as described in claim 3, characterized in that, The step of obtaining the basic image features of the frequency domain image includes: The frequency domain image is segmented to obtain a frequency domain waveform region image, and the waveform of the frequency domain waveform region image is obtained. The frequency domain waveform region image is subjected to image binarization processing to obtain a binarized frequency domain waveform image, and edge detection is performed on the binarized frequency domain waveform image to determine the envelope of the waveform; Based on the envelope of the waveform, the basic image features of the frequency domain image are determined, wherein the basic image features include the spectral centroid and the fundamental harmonic frequency; If the basic image features of the frequency domain image meet the preset frequency domain feature standard, then the step of performing machine vision-based spectrum detection on the recorded audio based on the frequency domain image is executed.

6. The machine vision-based sound detection method as described in claim 1, characterized in that, The method further includes: Obtain the key test points of the speaker under test, and obtain the basic electrical parameters of the speaker under test at the key test points when playing a preset test audio. If the abnormal audio detection and spectrum detection of the recorded audio of the speaker under test are both qualified, and the basic electrical parameters meet the preset electrical parameter standards, then the speaker under test is determined to be qualified.

7. A machine vision-based sound detection device, characterized in that, The machine vision-based sound detection device includes: The acquisition module is used to acquire the audio recordings of the speaker under test when playing preset test audio, and convert the audio recordings into time-domain images and frequency-domain images according to the total number of frames, sampling rate and amplitude of the audio recordings; The detection module is used to perform machine vision-based abnormal audio detection on the recorded audio based on the time domain image, and to perform machine vision-based spectrum detection on the recorded audio based on the frequency domain image. The determination module is used to determine that the audio device under test is qualified if both the abnormal audio detection and the spectrum detection of the recorded audio of the audio device under test are qualified. The step of performing machine vision-based spectrum detection on the recorded audio based on the frequency domain image includes: The second ROI region corresponding to the preset test audio is selected on the frequency domain image, and the image features of the second ROI region are determined to match the preset spectral features corresponding to the preset test audio by template matching or machine learning. If the image features of the second ROI region do not match the preset spectral features corresponding to the preset test audio, then the recorded audio spectrum is determined to be abnormal, and the spectrum detection of the recorded audio of the speaker under test is determined to be unqualified. If the image features of the second ROI region match the preset spectral features corresponding to the preset test audio, then the spectrum of the recorded audio is determined to be normal, and the spectrum detection of the recorded audio of the speaker under test is determined to be qualified.

8. A machine vision-based sound detection device, characterized in that, The machine vision-based audio detection device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the machine vision-based audio detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the machine vision-based sound detection method as described in any one of claims 1 to 6.

Images