Sound recognition method, apparatus, device, and storage medium
By introducing a multi-sensor correlation reasoning mechanism into the sound recognition model and utilizing a correlation calculation model, the problem of low recognition accuracy of a single sensor in industrial settings is solved, achieving higher accuracy in abnormal sound recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
- Filing Date
- 2022-08-26
- Publication Date
- 2026-05-29
AI Technical Summary
Existing sound recognition models typically rely on data from a single sensor, resulting in low accuracy in identifying unusual sounds in complex industrial environments.
A distributed multi-sensor correlation reasoning mechanism is adopted to improve the accuracy of abnormal sound recognition by comparing data from adjacent sensors and using a correlation calculation model.
By using correlation reasoning from multiple sensors, the accuracy of abnormal sound recognition has been improved and the false alarm rate has been reduced.
Smart Images
Figure CN115440245B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a voice recognition method, apparatus, device, and storage medium. Background Technology
[0002] With advancements in manufacturing technology, more and more production equipment is being installed in production workshops. The normal operation of these workshops depends on the proper functioning of this equipment; unscheduled shutdowns of large equipment can lead to equipment and material losses, impacting production efficiency. Therefore, real-time monitoring of the equipment's operating status is crucial. Traditional methods typically employ video capture devices, which can be used for review after equipment malfunctions, but cannot prevent them. However, methods now exist that use sound recognition models to detect abnormal sounds during production processes to prevent equipment failures.
[0003] However, existing sound recognition models typically only consider data from a single sensor, while the environment in industrial settings is complex. Relying on data from a single source results in low accuracy in identifying abnormal sounds. Summary of the Invention
[0004] This application provides a voice recognition method, apparatus, device, and storage medium, which can improve voice recognition accuracy. The technical solution is as follows:
[0005] In a first aspect, a sound recognition method is provided, comprising: acquiring first sound feature information of a target object collected by a first sensor; inputting the first sound feature information into a probability density estimation model and outputting a first probability value of the first sound feature information indicating the presence of abnormal sound; if the first probability value meets a preset probability condition, acquiring second sound feature information of the target object collected by a second sensor; generating target combined feature information based on the first sound feature information and the second sound feature information; inputting the target combined feature information into a correlation calculation model and outputting the correlation between the first sound feature information and the second sound feature information; and determining that the target object is emitting abnormal sound if the correlation meets a preset correlation condition.
[0006] In this application, firstly, the first sound feature information of the target object is acquired by a first sensor. Then, a first probability value of abnormal sound is calculated. If the first probability value meets a preset probability condition, second sound feature information of the target object is acquired by a second sensor, which is different from the first sensor. Based on the second sound feature information and the first sound feature information, a further judgment is made on whether the target object emits abnormal sound. Specifically, target combined feature information is generated based on the first and second sound feature information. The target combined feature information is input into a correlation calculation model, and the correlation between the first and second sound feature information is output. If the correlation meets a preset correlation condition, it is determined that the target object emits abnormal sound. Abnormal sound is identified through a two-stage judgment process of first probability value and correlation. Considering that the sound feature information of the target object acquired by a single sensor may be inaccurate, after the initial identification of abnormal sound through the first probability value, a further judgment is made on whether abnormal sound truly exists through the correlation between the first and second sound feature information. The accuracy of abnormal sound identification is high.
[0007] In one embodiment, generating target combined feature information based on first sound feature information and second sound feature information includes: calculating target similarity feature information between the first sound feature information and the second sound feature information; selecting at least one piece of information from the target similarity feature information and from the first sound feature information and the second sound feature information respectively and concatenating them to obtain target combined feature information; or, selecting at least two pieces of information from the target similarity feature information and concatenating them to obtain target combined feature information.
[0008] In one embodiment, before acquiring the first sound feature information of the target object collected by the first sensor, the method further includes: acquiring multiple sound feature information of the abnormal sound emitted by the object in the past collected by multiple sensors; generating a first combined feature information and a second combined feature information based on the multiple sound feature information, wherein the first combined feature information corresponds to the multiple sound feature information and the second combined feature information corresponds to the multiple sound feature information after perturbation; and using the first combined feature information and the second combined feature information as training samples to train a correlation network to obtain a correlation calculation model.
[0009] In one embodiment, generating first combined feature information based on multiple sound feature information includes: calculating first similarity feature information between any two sound feature information; selecting at least one piece of information from the first similarity feature information and from the multiple sound feature information respectively and concatenating them to obtain first combined feature information; or, selecting at least two pieces of information from the first similarity feature information and concatenating them to obtain first combined feature information.
[0010] In one embodiment, generating second combined feature information based on multiple sound feature information includes: perturbing the first combined feature information to obtain the second combined feature information.
[0011] In one embodiment, generating second combined feature information based on multiple sound feature information includes: perturbing multiple sound feature information; calculating second similarity feature information between any two perturbed sound feature information; selecting at least one piece of information from the second similarity feature information and the multiple perturbed sound feature information respectively and splicing them together to obtain second combined feature information; or, selecting at least two pieces of information from the second similarity feature information and splicing them together to obtain second combined feature information.
[0012] In one embodiment, acquiring multiple sound feature information includes: acquiring multiple historical sound samples of a historical object collected by multiple sensors; inputting the multiple historical sound samples into a compression network to output multiple historical sound feature information; inputting the multiple historical sound feature information into a probability density estimation model to output multiple second probability values representing abnormal sounds emitted by the historical object; and determining the historical sound feature information corresponding to the second probability values that meet preset probability conditions as multiple sound feature information.
[0013] In one possible implementation, generating first combined feature information based on multiple sound feature information includes: calculating first similarity feature information between any two sound feature information; selecting at least one piece of information from the first similarity feature information and from the multiple sound feature information respectively and concatenating them to obtain first combined feature information; or, selecting at least two pieces of information from the first similarity feature information and concatenating them to obtain first combined feature information.
[0014] In one possible implementation, a second combination of feature information is generated based on multiple sound feature information, including: perturbing the first combination of feature information to obtain the second combination of feature information.
[0015] In one possible implementation, generating second combined feature information based on multiple sound feature information includes: perturbing multiple sound feature information; calculating second similarity feature information between any two perturbed sound feature information; selecting at least one piece of information from the second similarity feature information and from the multiple perturbed sound feature information respectively and concatenating them to obtain the second combined feature information; or, selecting at least two pieces of information from the second similarity feature information and concatenating them to obtain the second combined feature information.
[0016] In one possible implementation, acquiring multiple sound feature information of historical objects emitting abnormal sounds collected by multiple sensors includes: acquiring multiple historical sound samples collected by multiple sensors; inputting the multiple historical sound samples into a compression network and outputting multiple historical sound feature information; inputting the multiple historical sound feature information into a probability density estimation model and outputting multiple probability values representing sound abnormalities; and determining the historical sound feature information corresponding to the probability values that meet preset probability conditions as multiple sound feature information.
[0017] Secondly, a training method for a sound recognition model is provided. The method includes: acquiring multiple sound feature information of historical objects emitting abnormal sounds collected by multiple sensors; generating a first combination feature information and a second combination feature information based on the multiple sound feature information, wherein the first combination feature information corresponds to the multiple sound feature information and the second combination feature information corresponds to the multiple sound feature information after perturbation; and training a correlation network using the first combination feature information and the second combination feature information as training samples to obtain a sound recognition model.
[0018] In this application, multiple sound feature information of historical objects emitting abnormal sounds is first acquired from multiple sensors. Then, a first combination of feature information and a second combination of feature information are generated based on the sound feature information. These two combination of feature information are then used as training samples to train a correlation network, resulting in a sound recognition model. The diverse sources of training samples for the sound recognition model are beneficial for improving its recognition accuracy. Furthermore, by generating first combination of feature information corresponding to the sound feature information and second combination of feature information corresponding to the perturbed sound feature information, model training can be performed without the participation of negative samples. Therefore, training the correlation network using the first and second combination of feature information to analyze the correlation between sound feature information further improves the abnormal sound recognition accuracy of the sound recognition model.
[0019] In one embodiment, generating first combined feature information based on multiple sound feature information includes: calculating first similarity feature information between any two sound feature information; selecting at least one piece of information from the first similarity feature information and from the multiple sound feature information respectively and concatenating them to obtain first combined feature information; or, selecting at least two pieces of information from the first similarity feature information and concatenating them to obtain first combined feature information.
[0020] In one embodiment, generating second combined feature information based on multiple sound feature information includes: perturbing the first combined feature information to obtain the second combined feature information.
[0021] In one embodiment, generating second combined feature information based on multiple sound feature information includes: perturbing multiple sound feature information; calculating second similarity feature information between any two perturbed sound feature information; selecting at least one piece of information from the second similarity feature information and the multiple perturbed sound feature information respectively and splicing them together to obtain second combined feature information; or, selecting at least two pieces of information from the second similarity feature information and splicing them together to obtain second combined feature information.
[0022] In one embodiment, acquiring multiple sound feature information of a historical object emitting abnormal sounds collected by multiple sensors includes: acquiring multiple historical sound samples of the historical object collected by multiple sensors; inputting the multiple historical sound samples into a compression network to output multiple historical sound feature information; inputting the multiple historical sound feature information into a probability density estimation model to output multiple probability values representing the historical object emitting abnormal sounds; and determining the historical sound feature information corresponding to the probability values that meet preset probability conditions as multiple sound feature information.
[0023] Thirdly, a voice recognition device is provided for performing any of the voice recognition methods provided in the first aspect above.
[0024] In one possible implementation, this application can divide the voice recognition device into functional modules according to the method provided in the first aspect above. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. For example, this application can divide the voice recognition device into an acquisition module, a generation module, and a training module, etc., according to their functions. The descriptions of the possible technical solutions and beneficial effects of the various functional modules described above can be found in the technical solutions provided in the first aspect above or its corresponding possible implementations, and will not be repeated here.
[0025] Fourthly, a training apparatus for a sound recognition model is provided, which is used to execute any of the sound recognition model training methods provided in the second aspect above.
[0026] In one possible implementation, this application can divide the training device for the voice recognition model into functional modules according to the method provided in the second aspect above. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. For example, this application can divide the training device for the voice recognition model into an acquisition module, a recognition module, and a determination module, etc., according to their functions. The descriptions of the possible technical solutions and beneficial effects of the various functional modules described above can be found in the technical solutions provided in the second aspect above or its corresponding possible implementations, and will not be repeated here.
[0027] Fifthly, embodiments of this application provide a computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the methods as described in the first or second aspect above.
[0028] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing at least one computer program that is loaded and executed by a processor to implement the methods described in the first or second aspect above.
[0029] In a seventh aspect, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A terminal's processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the terminal to perform the methods described in the first or second aspect of the above-described alternative implementations.
[0030] For a detailed description of aspects three through seven and their various implementations in this application, please refer to the detailed description in aspect one or two and their various implementations; and for a detailed analysis of the beneficial effects of aspects three through seven and their various implementations in aspect one or two and their various implementations, please refer to the beneficial effect analysis in aspect one or two and their various implementations, which will not be repeated here.
[0031] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the structure of a sound recognition model provided in an embodiment of this application;
[0033] Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of this application;
[0034] Figure 3 This is a schematic diagram of the deployment distribution of a sensor provided in an embodiment of this application;
[0035] Figure 4 This is a flowchart illustrating a training method for a sound recognition model provided in an embodiment of this application;
[0036] Figure 5 This is a schematic flowchart of a voice recognition method provided in an embodiment of this application;
[0037] Figure 6 This is a structural block diagram of a voice recognition device provided in an embodiment of this application;
[0038] Figure 7 This is a structural block diagram of a training device for a sound recognition model provided in an embodiment of this application;
[0039] Figure 8 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0041] In this article, "multiple" refers to two or more items. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0042] like Figure 1 As shown, conventional sound recognition models include compressed networks and probability density estimation models. The compressed network, consisting of an encoder and a decoder, processes sound samples to obtain sound feature information. The probability density estimation model estimates the probability of anomalies based on the sound feature information generated by the compressed network; if the probability exceeds a pre-set threshold T, it is considered an anomaly. However, the above-mentioned abnormal sound detection scheme only considers data from a single sensor. In complex industrial environments, relying on single data points can easily lead to false alarms, resulting in low accuracy in abnormal sound recognition.
[0043] To address the aforementioned issues, this application provides a training method for a sound recognition model. Based on a conventional sound recognition model, a distributed multi-sensor correlation reasoning mechanism is introduced. This solution assumes that the data between sensors are interconnected. By comparing adjacent sensor data, correlation reasoning is performed between multiple sound feature information from different sensors that have been identified as abnormal sounds. If the correlation is greater than a preset threshold during reasoning, the sound detection result of the conventional sound recognition model is considered reliable; otherwise, it is considered a false alarm, thereby improving the accuracy of abnormal sound recognition.
[0044] The following section first introduces the application scenarios of the embodiments of this application:
[0045] The method provided in this application can be applied to industrial sound recognition scenarios. Please refer to... Figure 2 This illustrates application scenario diagrams related to various embodiments of this application. For example... Figure 2As shown, this sound recognition scenario involves industrial equipment 200, multiple adjacently arranged sensors 210, and computer equipment 220. The sensors 210 can collect sound samples emitted by the industrial equipment 100 and send the sound samples to the computer equipment 220, which can then execute the method provided in this embodiment based on the sound samples.
[0046] For example, this industrial sound recognition scenario could be a distributed belt idler noise detection scenario, with the idler corresponding to industrial equipment 200. Idler rollers are rolling components that support the conveyor belt or wire rope and the materials it carries. Conveyor rollers are installed at certain intervals and according to requirements, below the carrying section and return section of the conveyor belt. When the conveyor belt runs on the idler rollers, it drives the rollers to rotate around their spindles. Idler rollers are a major component of belt conveyors, accounting for a significant portion of the overall cost. They are also vulnerable to wear and tear, frequently experiencing jamming, stalling, and other malfunctions. Their quality directly affects the stability of the conveyor. Damaged idler rollers can even scratch the belt, causing spontaneous combustion and resulting in greater economic losses. In severe cases, they can even cause serious accidents such as personal injury or death.
[0047] Furthermore, in the scenario of distributed belt idler noise detection, an industrial stethoscope sensor is used as sensor 210. For example, the industrial stethoscope sensor can be selected as: a contact microphone, a non-contact microphone, a vibration sensor, etc. The sensor deployment distribution is as follows... Figure 3 As shown, the sensors are deployed on one side of the belt roller according to their effective detection distance. Taking a contact microphone as an example, the effective pickup range is M, so the distance between the sensors is M.
[0048] Figure 2 and Figure 3 The application scenarios provided in this application are merely possible examples and should not be construed as limiting the application. The methods provided in this application can be applied to many more scenarios, which this application does not limit. The following, in conjunction with... Figure 2 and Figure 3 The application scenarios shown will be used to introduce the methods provided in the embodiments of this application.
[0049] Figure 4 A flowchart illustrating a training method for a sound recognition model provided in an exemplary embodiment of this application is shown. The training method for this sound recognition model can be... Figure 2 The computer device 220 performs this operation in the industrial sound recognition scenario shown. The method may include the following steps:
[0050] S410 acquires multiple sound feature information of abnormal sounds emitted by historical objects collected by multiple sensors.
[0051] Among them, reference Figure 3The sensor deployment shown refers to multiple sensors that are adjacent to each other on one side of industrial equipment in an industrial sound recognition scenario.
[0052] S420 generates a first combination of feature information and a second combination of feature information based on multiple sound feature information.
[0053] In this regard, considering that the combined feature information of corresponding normal sample pairs (representing multiple sound feature information with strong correlation) is easy to obtain, while the combined feature information of corresponding abnormal sample pairs (representing multiple sound feature information with weak correlation) is relatively scarce, multiple sound feature information is used to generate combined feature information of corresponding normal sample pairs and abnormal sample pairs at the same time, so as to obtain the first combined feature information and the second combined feature information.
[0054] S430, the first combination of feature information and the second combination of feature information are used as training samples to train the correlation network and obtain the sound recognition model.
[0055] During training, the label of the first combination of feature information can be set to 1, indicating that the multiple sound feature information corresponding to the first combination of feature information is highly correlated; the label of the second combination of feature information can be set to 0, indicating that the multiple sound feature information corresponding to the second combination of feature information is less correlated. By using the first combination of feature information and the second combination of feature information as training samples to train the correlation network, the sound recognition model (also known as the correlation calculation model) can be obtained.
[0056] The network structure of a correlation network is not limited to a specific network structure; for example, it can be a residual network (ResNet).
[0057] In this application, multiple sound feature information of historical objects emitting abnormal sounds is first acquired from multiple sensors. Then, a first combination of feature information and a second combination of feature information are generated based on the sound feature information. These two combination of feature information are then used as training samples to train a correlation network, resulting in a sound recognition model. The diverse sources of training samples for the sound recognition model are beneficial for improving its recognition accuracy. Furthermore, by generating first combination of feature information corresponding to the sound feature information and second combination of feature information corresponding to the perturbed sound feature information, model training can be performed without the participation of negative samples. Therefore, training the correlation network using the first and second combination of feature information to analyze the correlation between sound feature information further improves the abnormal sound recognition accuracy of the sound recognition model.
[0058] In one possible implementation, S420: generating first combined feature information based on multiple sound feature information may include:
[0059] Calculate the first similarity feature between any two audio feature information items.
[0060] Among them, similarity calculation algorithms can be used to calculate the similarity features between any two audio feature information, such as the cosine scoring algorithm.
[0061] At least one piece of information is selected from the first similarity feature information and from multiple sound feature information and concatenated to obtain the first combined feature information; or, at least two pieces of information are selected from the first similarity feature information and concatenated to obtain the first combined feature information.
[0062] In generating the first combined feature information, at least one piece of information can be selected from the first similarity feature information and from multiple sound feature information for concatenation to obtain the first combined feature information. Alternatively, at least two pieces of information can be selected from the first similarity feature information for concatenation to obtain the first combined feature information. Specifically, features can be concatenated using dimensional combination. For example, if the two features to be concatenated are [ab] and [cd], then the result after dimensional combination is [abcd].
[0063] For example, multiple sound feature information includes three items A, B, and C. By calculating the similarity features between each pair, the similarity feature S can be obtained. AB S AC S BC If the method of "selecting at least one piece of information from the first similarity feature information and from multiple sound feature information for concatenation" is adopted, then from S AB S AC S BC If at least one piece of information is selected from S, then at least one piece of information is selected from A, B, and C for concatenation; if the method of "selecting at least two pieces of information from the first similarity feature information for concatenation" is used, then from S... AB S AC S BC At least two pieces of information must be selected and combined.
[0064] In this embodiment, the first similarity feature information between any two sound feature information is first calculated, and then the first similarity feature information is concatenated to obtain the first combined feature information, or the first similarity feature information and multiple sound feature information are concatenated to obtain the first combined feature information. This ensures that the first combined feature information can reflect the sound features and the correlation between sounds, thereby improving the training accuracy of the subsequent model.
[0065] In one possible implementation, S420: generating second combined feature information based on multiple sound feature information may include:
[0066] The first combination of feature information is perturbed to obtain the second combination of feature information.
[0067] In this embodiment of the application, considering that the combined feature information of the corresponding abnormal sample pairs (representing multiple sound feature information with weak correlation) is relatively scarce, the second combined feature information is obtained by perturbing the first combined feature information, which can provide rich training samples for model training, thereby improving the training accuracy of the subsequent model.
[0068] In one possible implementation, S420: generating the second combined feature information based on multiple sound feature information may further include:
[0069] Disturbs multiple sound feature information.
[0070] Calculate the second similarity feature between any two perturbated sound feature information.
[0071] At least one piece of information is selected from the second similarity feature information and from the multiple sound feature information after perturbation, and then concatenated to obtain the second combined feature information; or, at least two pieces of information are selected from the second similarity feature information and then concatenated to obtain the second combined feature information.
[0072] Specifically, a second similarity feature is obtained by calculating the similarity between the perturbed sound feature information. Based on this second similarity feature information, a second combined feature information can be obtained by concatenating them. The calculation process for the second similarity feature information can be referenced from the calculation process for the first similarity feature information, and the concatenation process for the second combined feature information can be referenced from the concatenation process for the first combined feature information; therefore, it will not be repeated here.
[0073] In this embodiment of the application, considering that the combined feature information of the corresponding abnormal sample pairs (representing multiple sound feature information with weak correlation) is relatively scarce, the similarity feature between the perturbed sound feature information is calculated to obtain the second similarity feature information. Then, the second combined feature information can be obtained by splicing the second similarity feature information. This can provide rich training samples for model training, thereby improving the training accuracy of the subsequent model.
[0074] In one possible implementation, S410: acquires multiple sound feature information of historical objects emitting abnormal sounds collected by multiple sensors, including:
[0075] Acquire multiple historical sound samples of historical objects collected by multiple sensors.
[0076] Among them, several historical sound samples included both normal and abnormal sounds.
[0077] Multiple historical sound samples are input into a compression network, which outputs multiple historical sound feature information.
[0078] Among them, compressed networks such as Figure 1 As shown, the compression network has the function of feature extraction. By inputting multiple historical sound samples into the compression network, it can output multiple historical sound feature information.
[0079] Input multiple historical sound feature information into the probability density estimation model, and output multiple probability values representing the abnormal sounds emitted by historical objects.
[0080] Among them, the probability density estimation model is as follows Figure 1 As shown, the probability density estimation model has the function of probability estimation. By inputting multiple historical sound feature information into the probability density estimation model, it can output multiple probability values representing the abnormal sounds emitted by historical objects.
[0081] The historical sound feature information corresponding to the probability value that meets the preset probability condition is determined as multiple sound feature information.
[0082] The preset probability condition can be set to the probability value meeting a certain probability threshold, for example, threshold T. Once the probability value representing sound abnormality exceeds threshold T, the historical sound feature information corresponding to the probability value is determined as the sound feature information of the historical object emitting abnormal sound, so as to be used for training the sound recognition model.
[0083] In this embodiment, the sound feature information of the historical object emitting abnormal sounds is determined from historical sound samples by a compressed network and a probability density estimation model, which provides a reliable source of samples for the training of the sound recognition model and thus improves the training accuracy of the subsequent model.
[0084] The training method for a sound recognition model has been described above. This application also provides a sound recognition method that applies the trained sound recognition model to perform sound recognition, and this sound recognition method can be executed by a computer device. Figure 5 As shown, the method includes:
[0085] S510, acquire the first acoustic feature information of the target object collected by the first sensor.
[0086] Among them, reference Figure 3 The sensor deployment shown refers to any one of the multiple sensors adjacent to one side of the industrial equipment in the industrial sound recognition scenario.
[0087] S520, input the first sound feature information into the probability density estimation model, and output the first probability value of the first sound feature information containing abnormal sound.
[0088] Among them, the probability density estimation model is as follows Figure 1As shown, the probability density estimation model has the function of probability estimation. By inputting the first sound feature information into the probability density estimation model, it can output a first probability value of the existence of abnormal sound based on the first sound feature information.
[0089] S530: When the first probability value meets the preset probability condition, the second sound feature information of the target object collected by the second sensor is obtained.
[0090] The preset probability condition can be set to a probability value that meets a certain probability threshold, for example, threshold T. Once the probability value representing a sound abnormality exceeds threshold T, it is preliminarily determined that the sound feature information corresponding to that probability value contains an abnormal sound. If the first probability value meets the preset probability condition, it is preliminarily determined that the first sound feature information contains an abnormal sound. In this case, the second sound feature information of the target object is acquired by the second sensor to further determine whether the target object emits an abnormal sound based on the second sound feature information.
[0091] Furthermore, unlike the first sensor, the second sensor has at least one unit.
[0092] S540, generate target combined feature information based on the first sound feature information and the second sound feature information.
[0093] Among them, the target combination feature information can characterize the first sound feature information and the second sound feature information.
[0094] S550 inputs the target combination feature information into the correlation calculation model and outputs the correlation between the first sound feature information and the second sound feature information.
[0095] Referring to the training process of the relevance calculation model (sound recognition model) in S430, the relevance calculation model has the function of calculating the relevance between the first sound feature information and the second sound feature information corresponding to the target combined feature information. Inputting the target combined feature information into the relevance calculation model can output the relevance between the first sound feature information and the second sound feature information.
[0096] S560, if the relevance meets the preset relevance conditions, determines that the target object is emitting abnormal sounds.
[0097] If the correlation meets the preset correlation conditions, it indicates that the first sound feature information and the second sound feature information are strongly correlated, the first probability value calculated based on the first sound feature information is reliable, and it can be determined that the target object is emitting abnormal sounds.
[0098] In this application, firstly, the first sound feature information of the target object is acquired by a first sensor. Then, a first probability value of abnormal sound is calculated. If the first probability value meets a preset probability condition, second sound feature information of the target object is acquired by a second sensor, which is different from the first sensor. Based on the second sound feature information and the first sound feature information, a further judgment is made on whether the target object emits abnormal sound. Specifically, target combined feature information is generated based on the first and second sound feature information. The target combined feature information is input into a correlation calculation model, and the correlation between the first and second sound feature information is output. If the correlation meets a preset correlation condition, it is determined that the target object emits abnormal sound. Abnormal sound is identified through a two-stage judgment process of first probability value and correlation. Considering that the sound feature information of the target object acquired by a single sensor may be inaccurate, after the initial identification of abnormal sound through the first probability value, a further judgment is made on whether abnormal sound truly exists through the correlation between the first and second sound feature information. The accuracy of abnormal sound identification is high.
[0099] In one embodiment, S540: Generating target combined feature information based on the first sound feature information and the second sound feature information may include:
[0100] Calculate the target similarity feature information between the first sound feature information and the second sound feature information.
[0101] At least one piece of information is selected from the target similarity feature information, the first sound feature information, and the second sound feature information, and then concatenated to obtain the target combined feature information; or, at least two pieces of information are selected from the target similarity feature information and then concatenated to obtain the target combined feature information.
[0102] Referring to the description of the first combination feature, the generation process of the target combination feature is the same as the generation process of the first combination feature, and will not be repeated here.
[0103] Furthermore, if there is only one second sensor, then there is only one corresponding second sound feature information, and consequently, only one corresponding target similarity feature information. When adopting the method of "selecting at least two pieces of information from the target similarity feature information and concatenating them to obtain the target combined feature information", since there is only one target similarity feature information, the target similarity feature information can be directly determined as the target combined feature information.
[0104] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0105] This application embodiment can divide the corresponding device into functional modules according to the above method examples. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.
[0106] Figure 6 A structural block diagram of a voice recognition device provided in an exemplary embodiment of this application is shown. The voice recognition device includes:
[0107] The acquisition module 610 is used to acquire the first acoustic feature information of the target object collected by the first sensor.
[0108] The recognition module 620 is used to input the first sound feature information into the probability density estimation model and output the first probability value of the first sound feature information containing abnormal sound.
[0109] The acquisition module 610 is also used to acquire the second sound feature information of the target object collected by the second sensor when the first probability value meets the preset probability conditions.
[0110] The generation module 630 is used to generate target combined feature information based on the first sound feature information and the second sound feature information.
[0111] The recognition module 620 is also used to input the target combination feature information into the correlation calculation model and output the correlation between the first sound feature information and the second sound feature information.
[0112] The determination module 640 is used to determine the abnormal sound emitted by the target object when the relevance meets the preset relevance conditions.
[0113] In this application, firstly, the first sound feature information of the target object is acquired by a first sensor. Then, a first probability value of abnormal sound is calculated. If the first probability value meets a preset probability condition, second sound feature information of the target object is acquired by a second sensor (different from the first sensor). Based on the second sound feature information combined with the first sound feature information, a further judgment is made as to whether the target object emits abnormal sound. Specifically, target combined feature information is generated based on the first and second sound feature information. The target combined feature information is input into a correlation calculation model, and the correlation between the first and second sound feature information is output. If the correlation meets a preset correlation condition, it is determined that the target object emits abnormal sound. Abnormal sound identification is performed through a two-stage judgment process of first probability value and correlation. Considering that the sound feature information of the target object acquired by a single sensor may be inaccurate, after the initial identification of abnormal sound using the first probability value, a further judgment is made as to whether abnormal sound truly exists using the correlation between the first and second sound feature information. This results in high accuracy in identifying abnormal sound.
[0114] In one embodiment, the generation module 630 is specifically used for:
[0115] Calculate the target similarity feature information between the first sound feature information and the second sound feature information.
[0116] At least one piece of information is selected from the target similarity feature information, the first sound feature information, and the second sound feature information, and then concatenated to obtain the target combined feature information.
[0117] Alternatively, at least two pieces of information can be selected from the target similarity feature information and concatenated to obtain the target combined feature information.
[0118] In one embodiment, the device further includes a training module 650;
[0119] The acquisition module 610 is also used to acquire multiple sound feature information of abnormal sounds emitted by historical objects collected by multiple sensors.
[0120] The generation module 630 is also used to generate a first combination of feature information and a second combination of feature information based on multiple sound feature information.
[0121] Among them, the first combination of feature information corresponds to multiple sound feature information, and the second combination of feature information corresponds to multiple sound feature information after perturbation.
[0122] The training module 650 is used to train the correlation network using the first combination of feature information and the second combination of feature information as training samples to obtain the correlation calculation model.
[0123] In one embodiment, the generation module 630 is specifically used for:
[0124] Calculate the first similarity feature between any two audio features. Select at least one piece of information from the first similarity feature and from multiple audio features respectively, and concatenate them to obtain the first combined feature. Alternatively, select at least two pieces of information from the first similarity feature and concatenate them to obtain the first combined feature.
[0125] In one embodiment, the generation module 630 is specifically used for:
[0126] The first combination of feature information is perturbed to obtain the second combination of feature information.
[0127] In one embodiment, the generation module 630 is specifically used for:
[0128] Multiple sound feature information is perturbed. A second similarity feature information is calculated between any two perturbed sound feature information. At least one piece of information is selected from the second similarity feature information and from the multiple perturbed sound feature information, and then concatenated to obtain the second combined feature information.
[0129] Alternatively, at least two pieces of information can be selected from the second similarity feature information and concatenated to obtain the second combined feature information.
[0130] In one embodiment, the acquisition module 610 is specifically used for:
[0131] Multiple historical sound samples of a historical object are acquired from multiple sensors. These samples are then input into a compression network, which outputs multiple historical sound feature information. This historical sound feature information is then input into a probability density estimation model, which outputs multiple second probability values representing abnormal sounds emitted by the historical object. The historical sound feature information corresponding to these second probability values that meet preset probability conditions is determined as the multiple sound feature information.
[0132] In one possible implementation, the generation module 630 is specifically used for:
[0133] Calculate the first similarity feature between any two audio features. Select at least one piece of information from the first similarity feature and from multiple audio features respectively, and concatenate them to obtain the first combined feature. Alternatively, select at least two pieces of information from the first similarity feature and concatenate them to obtain the first combined feature.
[0134] In one possible implementation, a second combined feature information is generated based on multiple sound feature information, including:
[0135] The first combination of feature information is perturbed to obtain the second combination of feature information.
[0136] In one possible implementation, the generation module 630 is specifically used for:
[0137] Multiple sound feature information are perturbed. A second similarity feature information is calculated between any two perturbed sound feature information. At least one piece of information is selected from the second similarity feature information and from the multiple perturbed sound feature information, and then concatenated to obtain a second combined feature information. Alternatively, at least two pieces of information are selected from the second similarity feature information and then concatenated to obtain the second combined feature information.
[0138] In one possible implementation, module 610 is specifically used for:
[0139] Multiple historical sound samples collected by multiple sensors are acquired. These samples are then input into a compression network, which outputs multiple historical sound feature information. This historical sound feature information is then input into a probability density estimation model, which outputs multiple probability values representing sound anomalies. The historical sound feature information corresponding to the probability values that meet preset probability conditions is determined as the multiple sound feature information.
[0140] Figure 7 A structural block diagram of a training apparatus for a voice recognition model provided in an exemplary embodiment of this application is shown. The training apparatus for the voice recognition model includes:
[0141] The acquisition module 710 is used to acquire multiple sound feature information of abnormal sounds emitted by historical objects collected by multiple sensors.
[0142] The generation module 720 is used to generate a first combination feature information and a second combination feature information based on multiple sound feature information. The first combination feature information corresponds to multiple sound feature information, and the second combination feature information corresponds to multiple sound feature information after perturbation.
[0143] The training module 730 is used to train the correlation network using the first combination feature information and the second combination feature information as training samples to obtain the sound recognition model.
[0144] In this application, multiple sound feature information of historical objects emitting abnormal sounds is first acquired from multiple sensors. Then, a first combination of feature information and a second combination of feature information are generated based on the sound feature information. These two combination of feature information are then used as training samples to train a correlation network, resulting in a sound recognition model. The diverse sources of training samples for the sound recognition model are beneficial for improving its recognition accuracy. Furthermore, by generating first combination of feature information corresponding to the sound feature information and second combination of feature information corresponding to the perturbed sound feature information, model training can be performed without the participation of negative samples. Therefore, training the correlation network using the first and second combination of feature information to analyze the correlation between sound feature information further improves the abnormal sound recognition accuracy of the sound recognition model.
[0145] In one embodiment, the generation module 720 is specifically used for:
[0146] Calculate the first similarity feature between any two audio features. Select at least one piece of information from the first similarity feature and from multiple audio features respectively, and concatenate them to obtain the first combined feature. Alternatively, select at least two pieces of information from the first similarity feature and concatenate them to obtain the first combined feature.
[0147] In one embodiment, the generation module 720 is specifically used for:
[0148] The first combination of feature information is perturbed to obtain the second combination of feature information.
[0149] In one embodiment, the generation module 720 is specifically used for:
[0150] Multiple sound feature information are perturbed. A second similarity feature information is calculated between any two perturbed sound feature information. At least one piece of information is selected from the second similarity feature information and from the multiple perturbed sound feature information, and then concatenated to obtain a second combined feature information. Alternatively, at least two pieces of information are selected from the second similarity feature information and then concatenated to obtain the second combined feature information.
[0151] In one embodiment, the acquisition module 710 is specifically used for:
[0152] Multiple historical sound samples of a historical object are acquired from multiple sensors. These samples are then input into a compression network, which outputs multiple historical sound feature information. This information is then input into a probability density estimation model, which outputs multiple probability values representing the abnormal sounds emitted by the historical object. The historical sound feature information corresponding to the probability values that meet preset probability conditions is identified as the final multiple sound feature information.
[0153] Figure 8 This illustration shows a structural block diagram of a computer device provided in an exemplary embodiment of this application. The computer device may be an electronic device such as a smartphone, tablet computer, e-reader, portable personal computer, or smart wearable device. The computer device in this application may include one or more of the following components: a processor 810, a memory 820, and a screen 830.
[0154] The processor 810 may include one or more processing cores. The processor 810 connects to various parts of the terminal using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 820, and by calling data stored in the memory 820. Optionally, the processor 810 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 810 may integrate one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen 830; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 810 and may be implemented separately using a communication chip.
[0155] The memory 820 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 820 may include a non-transitory computer-readable storage medium. The memory 820 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 820 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc. The operating system may be an Android system (including systems deeply developed based on the Android system), an iOS system developed by Apple Inc. (including systems deeply developed based on the iOS system), or other systems. The data storage area may also store data created by the terminal during use (such as phonebook data, audio and video data, chat history data, etc.).
[0156] In addition, those skilled in the art will understand that the structure of the computer device shown in the above figures does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the computer device may also include radio frequency circuits, imaging components, sensors, audio circuits, wireless fidelity (WiFi) components, power supplies, Bluetooth components, etc., which will not be described in detail here.
[0157] This application also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the methods described in the above embodiments.
[0158] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A terminal's processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various alternative implementations of the above aspects.
[0159] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the foregoing examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.
[0160] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A voice recognition method, characterized in that, The method includes: Acquire the first acoustic feature information of the target object collected by the first sensor; The first sound feature information is input into the probability density estimation model, and the first probability value of the existence of abnormal sound in the first sound feature information is output. When the first probability value meets the preset probability condition, the second sound feature information of the target object is acquired by the second sensor. Target combined feature information is generated based on the first sound feature information and the second sound feature information. The target combined feature information can reflect the sound features and the correlation between the sound features. The correlation between the sound features represents the similarity between multiple sound features. The target combination feature information is input into the correlation calculation model, and the correlation between the first sound feature information and the second sound feature information is output. If the correlation meets the preset correlation conditions, it is determined that the target object is emitting abnormal sounds.
2. The method according to claim 1, characterized in that, The step of generating target combined feature information based on the first sound feature information and the second sound feature information includes: Calculate the target similarity feature information between the first sound feature information and the second sound feature information; At least one piece of information is selected from the target similarity feature information, the first sound feature information, and the second sound feature information, and then concatenated to obtain the target combined feature information; Alternatively, at least two pieces of information can be selected from the target similarity feature information and concatenated to obtain the target combined feature information.
3. The method according to claim 1, characterized in that, Before acquiring the first acoustic feature information of the target object collected by the first sensor, the method further includes: Acquire multiple sound feature information of abnormal sounds emitted by historical objects collected by multiple sensors; A first combined feature information and a second combined feature information are generated based on the plurality of sound feature information, wherein the first combined feature information corresponds to the plurality of sound feature information and the second combined feature information corresponds to the perturbed plurality of sound feature information; The first and second combined feature information are used as training samples to train the relevance network, thereby obtaining the relevance calculation model.
4. The method according to claim 3, characterized in that, Generate a first combined feature information based on the multiple sound feature information, including: Calculate the first similarity feature information between any two of the aforementioned sound feature information; At least one piece of information is selected from the first similarity feature information and the plurality of sound feature information, and then concatenated to obtain the first combined feature information; Alternatively, at least two pieces of information can be selected from the first similarity feature information and concatenated to obtain the first combined feature information.
5. The method according to claim 3, characterized in that, Generate a second combined feature information based on the multiple sound feature information, including: The first combined feature information is perturbed to obtain the second combined feature information.
6. The method according to claim 3, characterized in that, Generate a second combined feature information based on the multiple sound feature information, including: Disturb the multiple sound feature information; Calculate the second similarity feature information between any two perturbated sound feature information items; At least one piece of information is selected from the second similarity feature information and the perturbated plurality of sound feature information, and then concatenated to obtain the second combined feature information; Alternatively, at least two pieces of information can be selected from the second similarity feature information and concatenated to obtain the second combined feature information.
7. The method according to any one of claims 3-6, characterized in that, The acquisition of multiple sound feature information includes: Acquire multiple historical sound samples of the historical object from multiple sensors; The multiple historical sound samples are input into a compression network, which outputs multiple historical sound feature information. The multiple historical sound feature information is input into the probability density estimation model, and multiple second probability values representing the abnormal sound emitted by the historical object are output. The historical sound feature information corresponding to the second probability value that meets the preset probability condition is determined as the plurality of sound feature information.
8. A method for training a voice recognition model, characterized in that, The method includes: Acquire multiple sound feature information of abnormal sounds emitted by objects in history, collected by multiple sensors; A first combined feature information and a second combined feature information are generated based on the multiple sound feature information. The first combined feature information corresponds to the multiple sound feature information. The first combined feature information can reflect the sound features and the correlation between the sound features. The correlation between the sound features represents the similarity between the multiple sound features. The second combined feature information corresponds to the perturbed multiple sound feature information. The first and second combined feature information are used as training samples to train the correlation network, thereby obtaining the sound recognition model.
9. The method according to claim 8, characterized in that, Generate a first combined feature information based on the multiple sound feature information, including: Calculate the first similarity feature information between any two of the aforementioned sound feature information; At least one piece of information is selected from the first similarity feature information and the plurality of sound feature information, and then concatenated to obtain the first combined feature information; Alternatively, at least two pieces of information can be selected from the first similarity feature information and concatenated to obtain the first combined feature information.
10. The method according to claim 9, characterized in that, Generate a second combined feature information based on the multiple sound feature information, including: The first combined feature information is perturbed to obtain the second combined feature information.
11. The method according to claim 8, characterized in that, Generate a second combined feature information based on the multiple sound feature information, including: Disturb the multiple sound feature information; Calculate the second similarity feature information between any two perturbated sound feature information items; At least one piece of information is selected from the second similarity feature information and the perturbated plurality of sound feature information, and then concatenated to obtain the second combined feature information; Alternatively, at least two pieces of information can be selected from the second similarity feature information and concatenated to obtain the second combined feature information.
12. The method according to any one of claims 8-11, characterized in that, The acquisition of multiple sound feature information of historical objects emitting abnormal sounds collected by multiple sensors includes: Acquire multiple historical sound samples of the historical object from multiple sensors; The multiple historical sound samples are input into a compression network, which outputs multiple historical sound feature information. The multiple historical sound feature information is input into the probability density estimation model, and multiple probability values representing the abnormal sounds emitted by the historical object are output. The historical sound feature information corresponding to the probability value that meets the preset conditions is determined as the plurality of sound feature information.
13. A voice recognition device, characterized in that, The device includes: The acquisition module is used to acquire the first acoustic feature information of the target object collected by the first sensor; The recognition module is used to input the first sound feature information into the probability density estimation model and output a first probability value that the first sound feature information contains abnormal sound. The acquisition module is further configured to acquire second sound feature information of the target object collected by the second sensor when the first probability value meets the preset probability condition; The generation module is used to generate target combined feature information based on the first sound feature information and the second sound feature information. The target combined feature information can reflect the sound features and the correlation between the sound features. The correlation between the sound features represents the similarity between multiple sound features. The recognition module is further configured to input the target combination feature information into the correlation calculation model and output the correlation between the first sound feature information and the second sound feature information; The determination module is used to determine that the target object is emitting abnormal sounds when the correlation meets the preset correlation conditions.
14. A training device for a voice recognition model, characterized in that, The device includes: The acquisition module is used to acquire multiple sound feature information of abnormal sounds emitted by objects in history, collected by multiple sensors; The generation module is used to generate a first combined feature information and a second combined feature information based on the plurality of sound feature information. The first combined feature information corresponds to the plurality of sound feature information. The first combined feature information can reflect the sound features and the correlation between the sound features. The correlation between the sound features characterizes the similarity between the plurality of sound features. The second combined feature information corresponds to the perturbed plurality of sound feature information. The training module is used to train the correlation network using the first combined feature information and the second combined feature information as training samples to obtain the sound recognition model.
15. A computer device, characterized in that, The computer device includes a processor and a memory; the memory stores at least one computer instruction, which is loaded and executed by the processor to implement the method as described in any one of claims 1 to 12.
16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 12.