Npu-based acoustic image deep learning classification inference method and system

Abstract

The application discloses a sound image deep learning classification inference method and system based on NPU, and relates to artificial intelligence technology, comprising the following steps: firstly, sending a to-be-processed frequency spectrum into an NPU, and extracting target time-frequency features by using a trained target sound image feature recognition model; then, the NPU performs parallel calculation on the matching degree of the features and multiple reference time-frequency features in a preset sound image feature library, quickly locks the target reference time-frequency feature with the highest matching degree, and outputs the associated sound image identification code as a final classification result. Through special matrix operation and on-chip cache of the NPU, low-delay, low-power-consumption and high-concurrency local inference are realized, and the sound image real-time recognition performance of an embedded device is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to an audio-visual deep learning classification and reasoning method and system based on an NPU. Background Technology

[0002] Audio-visual classification and reasoning have important applications in audio content recognition, environmental monitoring, and intelligent interaction. Its core lies in achieving accurate classification by analyzing the spectral characteristics of audio signals. Current technologies, such as traditional audio-visual classification methods, rely on manually designed spectral features, resulting in weak generalization ability and difficulty adapting to complex and diverse audio scenarios. While deep learning-based methods can automatically extract features, they suffer from insufficient feature extraction accuracy and simplistic matching logic. For example, matching based solely on a single benchmark feature or simple distance calculation is prone to errors due to subtle differences in spectral features. Therefore, improving the effectiveness of feature extraction and the accuracy of matching degree calculation is crucial for optimizing the performance of audio-visual classification and reasoning. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for classification and reasoning based on NPU-based deep learning of audio and video.

[0004] In a first aspect, embodiments of the present invention provide an NPU-based deep learning-based classification and reasoning method for audio-visual imaging, comprising:

[0005] Obtain the spectrum to be processed;

[0006] The spectrum to be processed is input into the NPU, and the NPU performs feature extraction operation on the spectrum to be processed based on the pre-trained target acoustic image feature recognition model to obtain the target time-frequency features.

[0007] The matching degree between the target time-frequency feature and at least two reference time-frequency features in the preset audio-visual feature library is calculated to obtain the matching degree between the target time-frequency feature and each reference time-frequency feature.

[0008] Based on the matching degree between the target time-frequency feature and each reference time-frequency feature, the target reference time-frequency feature with the highest matching degree with the target time-frequency feature is obtained;

[0009] The audio-visual identification code associated with the target reference time-frequency features is used as the audio-visual classification reasoning result of the spectrum to be processed.

[0010] In a second aspect, embodiments of the present invention provide a server system, including a server, the server being used to execute the method described in the first aspect.

[0011] Compared to existing technologies, the beneficial effects provided by this invention include: Employing the NPU-based deep learning classification and inference method and system for audio-visual imaging disclosed in this invention, the spectrum to be processed is directly fed into the NPU, and target time-frequency features are extracted using a trained target audio-visual feature recognition model. Subsequently, the NPU calculates in parallel the matching degree between this feature and multiple benchmark time-frequency features in a preset audio-visual feature library, quickly identifies the target benchmark time-frequency feature with the highest matching degree, and outputs its associated audio-visual identification code as the final classification result. Through the NPU's dedicated matrix operations and on-chip caching, low-latency, low-power, and high-concurrency local inference is achieved, significantly improving the real-time audio-visual recognition performance of embedded devices. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating the steps of the NPU-based deep learning-based classification and reasoning method for audio-visual imaging provided in this embodiment of the invention;

[0014] Figure 2 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0016] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0017] In order to solve the technical problems mentioned in the background art Figure 1 This is a flowchart illustrating the NPU-based deep learning classification and reasoning method for audio-visual imaging provided in this embodiment of the present disclosure. The following is a detailed description of the NPU-based deep learning classification and reasoning method for audio-visual imaging.

[0018] Step S201: Obtain the spectrum to be processed;

[0019] Step S202: Input the spectrum to be processed into the NPU, and perform feature extraction operation on the spectrum to be processed by the NPU based on the pre-trained target acoustic image feature recognition model to obtain the target time-frequency features;

[0020] Step S203: Calculate the matching degree between the target time-frequency feature and at least two reference time-frequency features in the preset audio-visual feature library to obtain the matching degree between the target time-frequency feature and each reference time-frequency feature;

[0021] Step S204: Based on the matching degree between the target time-frequency feature and each reference time-frequency feature, obtain the target reference time-frequency feature with the highest matching degree with the target time-frequency feature;

[0022] Step S205: The audio-visual identification code associated with the target reference time-frequency features is used as the audio-visual classification reasoning result of the spectrum to be processed.

[0023] In this embodiment of the invention, exemplarily, when this method is applied to an industrial motor fault diagnosis system, the server acts as the execution entity, implementing audio-visual classification and reasoning of the motor's operating status through the following process. First, the server acquires the spectrum to be processed. Specifically, high-sensitivity microphones installed next to each motor in the factory collect the sound signals of the motor running in real time. These sound signals are transmitted to the server via industrial Ethernet after analog-to-digital conversion. The server preprocesses the received continuous audio stream, first using wavelet denoising technology to remove low-frequency background noise in the environment, such as the sound of air conditioners in the workshop. The denoised audio signal is divided into audio frames of approximately 46 milliseconds in length, and a Hamming window is superimposed on each frame to reduce spectral leakage. Subsequently, a short-time Fourier transform is performed on the windowed audio frames to convert the time-domain signal into a frequency-domain spectrum. The server performs standardization processing on this spectrum, that is, subtracting the mean of all frequency points from the value of each frequency point and then dividing by the standard deviation, finally obtaining the spectrum to be processed for subsequent processing. For example, if the server receives an audio segment from motor number 3 at a specific time, after the above preprocessing process, the corresponding spectrum to be processed is generated.

[0024] Next, the server inputs the obtained spectrum to be processed into the Neural Processing Unit (NPU). The NPU pre-deploys a trained target acoustic feature recognition model, which is a lightweight convolutional neural network. The server performs feature extraction on the input spectrum through the NPU. Internally, the spectrum first passes through multiple convolutional and pooling layers. The convolutional layers perform convolution operations on local time-frequency regions of the spectrogram using a sliding window, extracting transient features from the motor sound signal, such as the time location and frequency distribution of fault impacts. The pooling layers reduce the dimensionality of the feature map output by the convolutional layers, retaining key feature information, such as the energy peaks that bearing wear might occur in the 100-500Hz frequency band. After convolution and pooling operations, the feature map is fed into a global average pooling layer for further dimensionality reduction. Finally, a fully connected layer maps the high-dimensional features into a dense vector of fixed dimensions, thus obtaining the target time-frequency features. This process fully utilizes the parallel computing capabilities of the NPU, ensuring the efficiency of feature extraction. For example, the spectrum of motor No. 3 to be processed is processed by the NPU and outputs a 512-dimensional target time-frequency feature vector.

[0025] After acquiring the target time-frequency feature, the server calculates its matching degree with at least two reference time-frequency features in a preset audio-visual feature library. The preset audio-visual feature library, stored in the server's storage device, contains reference time-frequency features corresponding to various typical motor operating states (such as normal operation, slight bearing wear, severe bearing wear, rotor imbalance, stator winding fault, etc.). Each state type contains multiple different samples of reference time-frequency features, and each reference time-frequency feature is associated with a unique audio-visual identifier. The matching degree calculation process is as follows: The server first divides the target time-frequency feature into multiple local feature vectors according to a certain dimension. For each reference time-frequency feature in the preset audio-visual feature library, it is similarly divided into the same number and dimension of local feature vectors. Then, the cosine similarity between each local feature vector of the target time-frequency feature and all local feature vectors of the reference time-frequency feature is calculated, resulting in multiple similarity values. The server sorts these similarity values, selects the top K similarity values ​​with the largest values, and calculates the arithmetic mean of these K similarity values. This average is used as the final matching degree between the target time-frequency feature and the current reference time-frequency feature. The server iterates through all the reference time-frequency features in the preset audio-visual feature library, calculates and records the matching degree between the target time-frequency features and their respective features.

[0026] After completing all matching degree calculations, the server selects the baseline time-frequency feature with the highest matching degree to the target time-frequency feature based on the obtained matching degree values. During the selection process, the server compares all matching degrees with a preset confidence threshold. If the matching degree of the target time-frequency feature with all baseline time-frequency features is lower than the confidence threshold, the spectrum to be processed is determined to belong to an unknown audio-visual category. In this case, the server triggers the corresponding unknown category processing mechanism, such as prompting the user to perform manual annotation and storing the relevant data in a specific database for subsequent model optimization. If there are baseline time-frequency features with a matching degree higher than or equal to the confidence threshold, the one with the highest matching degree is selected.

[0027] Finally, the server uses the acoustic image identification code associated with the determined target reference time-frequency features as the acoustic image classification and inference result corresponding to the spectrum to be processed. This result is stored in a local log file by the server and simultaneously visualized on the factory's monitoring screen, for example, displaying "Motor No. 3: Minor bearing wear (M002)". In addition, the server will automatically send alarm information to the maintenance personnel's mobile APP according to preset rules, such as when a serious fault type is identified, prompting them to carry out timely maintenance. Through this series of steps, real-time and accurate classification and inference of the operating status of industrial motors is achieved, assisting maintenance decision-making.

[0028] In this embodiment of the invention, the acoustic-image feature recognition model is obtained in the following manner, and can be implemented through the following examples.

[0029] The first acoustic image feature recognition model is deployed to the NPU to extract time-frequency features of the first spectrum instance and obtain the time-frequency features of the first spectrum instance.

[0030] The first error is obtained based on the time-frequency characteristics of the first spectrum instance, the target value of the state type of the first spectrum instance, and the initial sound image category prototype vectors corresponding to at least two sound image category prototype vectors of each state type.

[0031] Based on the first error, adjust the model parameters of the first audio-visual feature recognition model and the initial audio-visual category prototype vectors corresponding to at least two audio-visual category prototype vectors of each state type to obtain the first audio-visual category prototype vectors corresponding to the second audio-visual feature recognition model and at least two audio-visual category prototype vectors of each state type.

[0032] The second acoustic image feature recognition model is deployed to the NPU to extract time-frequency features of the second spectrum instance, thereby obtaining the time-frequency features of the second spectrum instance;

[0033] The second error is obtained based on the time-frequency characteristics of the second spectrum instance, the state type target value of the second spectrum instance, and the first sound image category prototype vector corresponding to the core sound image category prototype vector of each state type; the core sound image category prototype vector of a state type is one of at least two sound image category prototype vectors of that state type.

[0034] Based on the second error, adjust the model parameters of the second audio-visual feature recognition model and the first audio-visual category prototype vector corresponding to the core audio-visual category prototype vector of each state type until the trained target audio-visual feature recognition model is obtained.

[0035] In this embodiment of the invention, for example, the server acts as the execution entity. During the training phase of the industrial motor fault diagnosis model, it first deploys the initially constructed first acoustic-image feature recognition model to the NPU. This model contains multiple convolutional layers and fully connected layers. The server retrieves a first set of spectrum instances from a local database. This set contains labeled samples of various state types, such as "normal operation," "bearing wear," and "rotor imbalance," with multiple spectrum instances under each state type. The NPU performs batch processing on these first spectrum instances, extracting the time-frequency feature vector corresponding to each spectrum instance through convolution and pooling operations of the model. These vectors are temporarily stored in the server's memory.

[0036] Next, the server initializes at least two prototype vectors for each state type, representing the feature centers of different subtypes within that state (e.g., "bearing wear" might include two sub-prototypes: "inner ring wear" and "outer ring wear"). For each first spectrum instance, the server calculates the matching degree between its feature vector and each prototype vector under its state type, selecting the prototype vector with the highest matching degree as the baseline prototype. Combining the state type target value of the instance, the server calculates the classification error (first sub-error) between the current baseline prototype and the target value, and simultaneously calculates the minimum matching degree between the instance's features and prototype vectors of other state types, constructing the inter-class separation error (second sub-error). The server dynamically adjusts the adjustment factor according to the training epochs; this factor, associated with the epoch, is used to correct the audio-visual spacing parameter and the feature scale factor. Finally, the corrected parameters are weighted and summed with the two sub-errors to obtain the first error.

[0037] Based on the first error, the server adjusts the network parameters (such as convolutional layer weights and bias terms) and the initial prototype vectors for all state types of the first acoustic image feature recognition model using the backpropagation algorithm. After multiple rounds of iterative training, the model parameters and prototype vectors gradually converge. The server then saves the model at this point as the second acoustic image feature recognition model and simultaneously updates the first acoustic image category prototype vectors for each state type.

[0038] Subsequently, the server selects a baseline spectrum instance set (i.e., samples with high classification confidence in that state) from the first spectrum instance set for each state type. For each state type, it calculates the average matching degree between the prototype vector of each first audio-visual category and all features in the corresponding baseline set, and determines the prototype vector with the highest matching degree as the core audio-visual category prototype vector for that state type. The server then deploys the second audio-visual feature recognition model to the NPU to extract features from the second spectrum instance set (which may partially overlap with the first spectrum instance set or be newly acquired samples) to obtain the time-frequency features of the second spectrum instance. Based on these features, the corresponding state type target values, and the core prototype vectors of each state type, the server calculates the second error, which mainly reflects the classification performance of the core prototype vectors. The server continues to adjust the model parameters and core prototype vectors based on the second error, iteratively optimizing until the model performance stabilizes, finally obtaining the trained target audio-visual feature recognition model.

[0039] In this embodiment of the invention, the step of obtaining the first error based on the time-frequency characteristics of the first spectrum instance, the target value of the state type of the first spectrum instance, and the initial sound-image category prototype vector vectors corresponding to at least two sound-image category prototype vectors of each state type can be implemented through the following example.

[0040] For each state type, calculate the first matching degree between the time-frequency features of the first spectrum instance and the initial sound-image category prototype vectors corresponding to at least two sound-image category prototype vectors respectively;

[0041] The initial sound image category prototype vector vector with the highest first matching degree between the at least two initial sound image category prototype vector vectors of each state type and the time-frequency features of the first spectrum instance is taken as the reference sound image category prototype vector vector.

[0042] The first sub-item error is obtained based on the benchmark acoustic image category prototype vector and the state type target value of the first spectrum instance;

[0043] The first error is obtained based on the first matching degree between the time-frequency characteristics of the first spectrum instance and the initial sound image category prototype vectors corresponding to at least two sound image category prototype vectors of each state type, and the first sub-item error.

[0044] In an embodiment of the present invention, for example, during the training process of the industrial motor fault diagnosis model, the server is processing a sample, denoted as X, from the first spectrum instance set. The target value of the state type of this sample is "slight bearing wear" (denoted as state S2). The server first retrieves the time-frequency feature vector F (512-dimensional dense vector) of X extracted by the NPU from memory, and then begins to calculate the first error.

[0045] The server first needs to calculate the first degree of matching between F and the initial prototype vectors of each state type. The current system contains 5 state types: "normal operation" (S1), "slight bearing wear" (S2), "severe bearing wear" (S3), "rotor imbalance" (S4), and "stator winding fault" (S5). Each state type is pre-initialized with 2 acoustic image category prototype vectors (e.g., S2 contains "inner ring wear prototype vector P2a" and "outer ring wear prototype vector P2b", S1 contains "smooth operation prototype vector P1a" and "load fluctuation prototype vector P1b", and so on for the other states). These initial prototype vectors are generated by the server through a random normal distribution and normalized to the interval [-1,1]. For each state type, the server calculates the degree of matching between F and all initial prototype vectors under that type (using cosine similarity, which measures directional consistency by the ratio of the vector inner product to the product of the magnitudes). For example, the matching degree of F with P2a of S2 is 0.82, and the matching degree with P2b is 0.75; the matching degree with P1a of S1 is 0.31, and the matching degree with P1b is 0.28; the matching degree with P3a of S3 is 0.68, and the matching degree with P3b is 0.62; the matching degree with P4a of S4 is 0.45, and the matching degree with P4b is 0.41; and the matching degree with P5a of S5 is 0.53, and the matching degree with P5b is 0.49.

[0046] Next, the server selects the vector with the highest matching degree with F from the initial prototype vectors of all state types as the baseline audio-visual category prototype vector. In the above calculation results, P2a (0.82) of S2 is the maximum value among all matching degrees, so the server determines P2a as the baseline prototype vector of X.

[0047] Subsequently, the server calculates the first sub-item error based on the baseline prototype vector and the target value of X's state type. Since the baseline prototype vector P2a belongs to state S2, which is consistent with the target value of X (both are S2), it indicates that the current classification direction is correct. The server calculates this classification error using the cross-entropy loss function, and the error value is relatively small (e.g., 0.15). If the baseline prototype vector belongs to other state types (such as S3), the classification is incorrect, and the error value will increase significantly.

[0048] Based on this, the server calculates the first error by combining the first matching degree and the first sub-item error for all state types. First, the server extracts the highest matching degree for each state type (e.g., 0.31 for S1, 0.82 for S2, 0.68 for S3, 0.45 for S4, and 0.53 for S5). These values ​​reflect the overall closeness of F to each state type. To enhance inter-class discriminability, the server calculates the minimum value of the highest matching degree between feature F of X and non-target state types (S1, S3, S4, S5) (e.g., 0.31, i.e., the highest matching degree of S1). An inter-class separation error is constructed using a triplet loss function to ensure that the matching degree of the same class is significantly higher than that of different classes. Simultaneously, the server generates a dynamic adjustment factor based on the current training epoch (e.g., epoch 50). This factor decreases with increasing epochs and is used to correct the acoustic-image spacing parameter (controlling the clustering degree of similar features) and the feature scale factor (adjusting the overall distribution range of the feature vector). Finally, the server weights and sums the corrected parameters with the first sub-item error (classification error) and the inter-class separation error to obtain the first error, which comprehensively reflects the classification accuracy and inter-class distinguishability, providing a basis for subsequent model parameter adjustments.

[0049] In this embodiment of the invention, the first error is obtained based on the first matching degree between the time-frequency characteristics of the first spectrum instance and the initial sound-image category prototype vectors corresponding to at least two sound-image category prototype vectors of each state type, and the first sub-item error. This can be implemented through the following example.

[0050] For each state type, based on the first matching degree between the time-frequency characteristics of the first spectrum instance and the initial sound-image category prototype vectors corresponding to at least two sound-image category prototype vectors of the state type, the second sound-image category prototype vector with the highest first matching degree between the time-frequency characteristics of the first spectrum instance under the state type is obtained.

[0051] The second sub-item error is obtained based on the first matching degree between the second acoustic image category prototype vector under each of at least two state types and the time-frequency features of the first spectrum instance;

[0052] The first error is obtained based on the first component error and the second component error.

[0053] In an embodiment of the present invention, for example, in the training process of the industrial motor fault diagnosis model, when the server calculates the first error for the first spectrum instance, it first needs to determine the prototype vector that is most similar to the features of the instance under each state type. Taking a spectrum instance of slight bearing wear (target state type S2) as an example, the server has extracted its 512-dimensional time-frequency feature vector F through the NPU, and initialized two acoustic image category prototype vectors for the five state types preset by the system (S1: normal operation, S2: slight bearing wear, S3: severe bearing wear, S4: rotor imbalance, S5: stator winding fault) respectively (e.g., S2 includes inner ring wear prototype P2a and outer ring wear prototype P2b, S1 includes stable operation prototype P1a and load fluctuation prototype P1b). The server iterates through all state types, calculating the cosine similarity (first matching degree) between F and all prototype vectors under each type. For example, S2 has a P2a matching degree of 0.81 and a P2b matching degree of 0.73; S1 has a P1a matching degree of 0.32 and a P1b matching degree of 0.29; S3 has a P3a matching degree of 0.67 and a P3b matching degree of 0.61; S4 has a P4a matching degree of 0.44 and a P4b matching degree of 0.40; and S5 has a P5a matching degree of 0.52 and a P5b matching degree of 0.48. For each state type, the prototype vector corresponding to the highest matching degree is taken as the second audio-visual category prototype vector, forming a representative prototype set for each type (e.g., S2 takes P2a, S1 takes P1a, S3 takes P3a, etc.).

[0054] Next, the server calculates the second sub-item error based on the matching degree of these representative prototype vectors. The system's goal is to ensure that the current instance feature F has a significantly higher matching degree with the representative prototype (P2a, matching degree 0.81) of its state type (S2) than with the representative prototypes of other types. The server extracts the highest matching degree for all non-target types (0.32 for S1, 0.67 for S3, 0.44 for S4, and 0.52 for S5), with S3 having the highest value of 0.67. To enhance inter-class discriminability, the server constructs the second sub-item error using a triplet loss function. The formula includes a preset boundary value (e.g., 0.2). When the difference (0.14) between the target matching degree (0.81) and the maximum non-target matching degree (0.67) is less than the boundary value, the error is the difference between the boundary value and the difference (0.06); otherwise, the error is 0. In this case, the calculated second sub-item error is 0.06.

[0055] Finally, the server weighted and summed the first sub-item error and the second sub-item error to obtain the first error. The first sub-item error comes from the classification loss of the baseline prototype vector (P2a) and the instance target type (S2), and is calculated to be 0.13 using the cross-entropy function (because the classification is correct, the error is small). The server calculates according to preset weights (e.g., classification error accounts for 60%, inter-class separation error accounts for 40%): 0.13×0.6+0.06×0.4=0.078+0.024=0.102. This value is the first error of the current first spectrum instance, which is used for subsequent model parameter adjustments. The prototype distance error is calculated using the following formula: First error = First audio-visual distance parameter × First sub-item error + First feature scale factor × Second sub-item error. Where: First sub-item error: calculated based on the classification loss of the benchmark audio-visual category prototype vector and the state type target value, using the cross-entropy loss function; Second sub-item error: calculated based on the optimal matching degree of all state types, using the triplet loss function, specifically: max(boundary value - (highest matching degree of target state - highest matching degree of non-target state), 0); The boundary value is preset to 0.2.

[0056] In this embodiment of the invention, obtaining the first error based on the first sub-item error and the second sub-item error can be implemented through the following example.

[0057] Based on the training round number corresponding to the first epoch of the first spectrum instance, a first acoustic-image spacing parameter and a first feature scale factor are obtained, and the first acoustic-image spacing parameter and the first feature scale factor have negative feedback with the first epoch number;

[0058] The prototype distance error is calculated based on the first acoustic-image spacing parameter, the first feature scale factor, the first sub-item error, and the second sub-item error to obtain the first error.

[0059] In this embodiment of the invention, for example, during the training process of the industrial motor fault diagnosis model, the server is processing a "slight bearing wear" sample X (state type target value S2) from the first spectrum instance set, currently in the 50th round of training (the first round has 50 iterations, and the total number of training rounds is preset to 200 rounds). The server first determines the first acoustic-image spacing parameter and the first feature scale factor based on the number of rounds in the first round. These two parameters have a negative feedback relationship with the number of rounds, that is, the parameters are larger when the number of rounds is smaller in the early stage of training, and gradually decrease as the number of rounds increases, thereby realizing the training strategy of "first strengthening feature aggregation and separation, and then optimizing classification accuracy". Specifically, the server has built-in parameter adjustment rules: the baseline value of the first acoustic-image spacing parameter is 1.2, and the parameter decreases by an amplitude of 0.005 for each additional round; the baseline value of the first feature scale factor is 1.0, and the factor decreases by an amplitude of 0.003 for each additional round. Therefore, in the 50th round, the first acoustic-image spacing parameter is 1.2-50×0.005=0.95, and the first feature scale factor is 1.0-50×0.003=0.85; if trained to the 150th round, the parameters will decay to 1.2-150×0.005=0.45 and 1.0-150×0.003=0.55 respectively, reflecting the negative feedback characteristics.

[0060] Next, the server combines these two parameters with the first and second sub-item errors to calculate the prototype distance error, obtaining the first error. The first sub-item error is the classification error between the baseline prototype vector of sample X (the inner wear prototype P2a of S2) and the target state type S2, previously calculated as 0.13 (small error due to correct classification); the second sub-item error is the inter-class separation error between the features of X and the highest matching degree of the non-target type (0.67 in S3), previously calculated as 0.06 (error of 0.06 because the difference between the target matching degree 0.81 and the highest non-target matching degree 0.67 is less than the preset boundary value of 0.2). The server fuses these parameters using a preset formula: Prototype distance error = First acoustic-image spacing parameter × First sub-item error + First feature scale factor × Second sub-item error. Substituting the parameters and error values ​​from round 50, we get: Prototype distance error = 0.95 × 0.13 + 0.85 × 0.06 = 0.1235 + 0.051 = 0.1745. This value is the first error of the current first spectrum instance X. The server uses this as the basis for backpropagation to adjust the model parameters and prototype vector. The larger the error value, the greater the adjustment of the model parameters, until training converges.

[0061] If the number of training rounds is increased to 150 (the first round has 150 rounds), the first acoustic-image distance parameter decays to 0.45 and the first feature scale factor decays to 0.55. Assuming that the first sub-error of X is optimized to 0.08 (more accurate classification) and the second sub-error is optimized to 0.02 (more sufficient inter-class separation), then the prototype distance error = 0.45 × 0.08 + 0.55 × 0.02 = 0.036 + 0.011 = 0.047. The error is significantly reduced, indicating that the model gradually achieves the training goal of "strengthening feature structure in the early stage and focusing on classification accuracy in the later stage" through the dynamic adjustment of negative feedback parameters.

[0062] In this embodiment of the invention, the step of obtaining the first acoustic-image spacing parameter and the first feature scale factor based on the first round number corresponding to the training round in which the first spectrum instance is located can be implemented through the following example.

[0063] Based on the number of the first round corresponding to the training round in which the first spectrum instance is located, a first adjustment factor is obtained, and the first adjustment factor has negative feedback with the number of the first round.

[0064] The first acoustic-image spacing correction value is obtained based on the first adjustment factor. The first acoustic-image spacing correction value is added to the first reference acoustic-image spacing parameter to obtain the first acoustic-image spacing parameter.

[0065] The first feature scale correction value is obtained based on the first adjustment factor, and the first feature scale correction value is added to the first reference feature scale factor to obtain the first feature scale factor.

[0066] In this embodiment of the invention, for example, during the training process of the industrial motor fault diagnosis model, the server is performing the 30th round of training on the first spectrum instance set (the first round has 30 iterations, and the total number of training rounds is preset to 200 rounds). Currently, the spectrum instance being processed is in the state of "slight bearing wear". To dynamically adjust the focus of model training, the server first calculates a first adjustment factor based on the number of rounds in the first round. This factor has a negative feedback relationship with the number of rounds, that is, the adjustment factor is larger when the number of rounds is smaller in the early stage of training, and gradually decreases as the number of rounds increases, thereby realizing the training strategy of "first strengthening the feature space structure, and then optimizing the detailed classification". The server has built-in rules for calculating the adjustment factor: the initial adjustment factor is set to 1.0 (corresponding to the first round), and for each additional round of training, the adjustment factor decreases linearly by a fixed amplitude, and the amplitude of the decrease is determined according to the total number of rounds (when the total number of rounds is 200, the decrease is 0.005 per round). Therefore, during the 30th training round, the server reads the current round number "30" from the training log and calculates the first adjustment factor as 1.0 - 30 × 0.005 = 0.85. If training progresses to the 100th round, the adjustment factor will decay to 1.0 - 100 × 0.005 = 0.5, and by the 200th round, it will decay to 0 (at which point the adjustment factor no longer affects the parameter). The first adjustment factor is generated according to a linear rule: First adjustment factor = 1.0 - (current round × single round decay); single round decay = 1.0 / total training rounds; The first sound-image spacing parameter is generated according to the following rule: First sound-image spacing parameter = first baseline sound-image spacing parameter + (first adjustment factor × spacing correction coefficient); the spacing correction coefficient is preset to 0.5; The first feature scale factor is generated according to the following rule: First feature scale factor = first baseline feature scale factor + (first adjustment factor × scale correction coefficient); the scale correction coefficient is preset to 0.3.

[0067] Next, the server calculates the first acoustic-image spacing correction value based on the first adjustment factor. The first acoustic-image spacing parameter is used to control the degree of clustering of similar acoustic-image features, and its baseline value (first baseline acoustic-image spacing parameter) is preset by the server to 1.5 (initial value, to ensure strong feature clustering in the early stages of training). The server has a built-in spacing correction coefficient of 0.5 (used to control the correction amplitude), and the correction value is obtained by multiplying the adjustment factor and the correction coefficient: In the 30th round, the first acoustic-image spacing correction value = first adjustment factor (0.85) × spacing correction coefficient (0.5) = 0.425. Subsequently, the server adds this correction value to the first baseline acoustic-image spacing parameter to obtain the first acoustic-image spacing parameter for the current round: 1.5 + 0.425 = 1.925. As the adjustment factor decreases with each round, the correction value also gradually decreases. Therefore, the first image-sound distance parameter will gradually decrease from the initial 1.925 (round 30) as training progresses. By round 100 (adjustment factor 0.5), the correction value is 0.5 × 0.5 = 0.25, and the parameter becomes 1.5 + 0.25 = 1.75, ensuring that later training focuses on subtle differences in features rather than excessive aggregation.

[0068] Simultaneously, the server calculates the first feature scale factor using the same logic. The first feature scale factor is used to adjust the overall distribution range of the feature vectors, enhancing the discriminative power of the features. Its baseline value (first baseline feature scale factor) is preset to 1.2 (initial value, ensuring a moderate feature scale). The server has a built-in scale correction coefficient of 0.3 (controlling the scale correction amplitude). The first feature scale correction value is obtained by multiplying the first adjustment factor and the scale correction coefficient: at round 30, the correction value = 0.85 × 0.3 = 0.255. The server adds this correction value to the first baseline feature scale factor to obtain the current first feature scale factor: 1.2 + 0.255 = 1.455. Similar to the sound-image spacing parameter, as the training rounds increase, the adjustment factor decreases, leading to a decrease in the correction value, and the first feature scale factor also gradually decreases. For example, at round 100, the correction value is 0.5 × 0.3 = 0.15, and the factor becomes 1.2 + 0.15 = 1.35, avoiding overfitting due to excessively large feature scales in later training.

[0069] Through the above process, the server obtains the first acoustic-image distance parameter of 1.925 and the first feature scale factor of 1.455 in the 30th round of training. These two parameters will be used to calculate the prototype distance error (first error) in the subsequent calculation, dynamically balancing the feature aggregation and inter-class separation requirements in model training, ensuring that the model can quickly build a clear feature space structure in the early stage of training, and gradually optimize the classification accuracy in the later stage.

[0070] In this embodiment of the invention, there are multiple first spectrum instances. Before obtaining the second error based on the time-frequency characteristics of the second spectrum instance, the state type target value of the second spectrum instance, and the first sound image category prototype vector corresponding to the core sound image category prototype vector of each state type, this embodiment of the invention also provides the following implementation method.

[0071] Obtain the reference spectrum instance set corresponding to each state type. The reference spectrum instance set corresponding to a state type includes multiple first spectrum instances whose state type target value is the same as that state type.

[0072] For each state type, the baseline matching degree corresponding to each sound image category prototype vector under that state type is obtained based on the matching degree between the center vector of each sound image category prototype vector in at least two sound image category prototype vectors under that state type and the time-frequency features of each first spectrum instance in the baseline spectrum instance set corresponding to that state type.

[0073] For each state type, the prototype vector of the audio-visual category with the highest baseline matching degree under that state type is taken as the core audio-visual category prototype vector of that state type.

[0074] In an embodiment of the present invention, for example, in the training process of the industrial motor fault diagnosis model, when there are multiple first spectrum instances (e.g., including 5 state types such as "normal operation", "slight bearing wear", "severe bearing wear", "rotor imbalance" and "stator winding fault", with 200 labeled samples for each state, for a total of 1000 first spectrum instances), the server needs to determine the core acoustic image category prototype vector for each state type before calculating the second error. The specific process is as follows:

[0075] The server first selects a baseline spectrum instance set corresponding to each state type from all first spectrum instances. Each first spectrum instance has a target value for the state type (e.g., a sample with "slight bearing wear" is labeled S2). The server groups the instances according to the target value, resulting in 5 state type groups. For each group (e.g., group S2), the server retrieves the time-frequency feature vectors extracted from the NPU for all first spectrum instances in that group, and combines this with the classification confidence recorded during training (determined by the softmax probability value of the model's output layer), selecting samples with a confidence level higher than a preset threshold (e.g., 0.7) as baseline spectrum instances. For example, among the 200 instances in group S2, 150 samples have a classification confidence score of over 0.7. The server uses these 150 samples to form the baseline spectrum instance set corresponding to S2. Similarly, 160 samples are selected from the "normal operation" (S1) group, 145 from the "severe bearing wear" (S3) group, 155 from the "rotor imbalance" (S4) group, and 140 from the "stator winding fault" (S5) group, forming the baseline spectrum instance sets for each of the five state types. These sets contain the typical sample features with high classification reliability in that state.

[0076] Next, for each state type, the server calculates the baseline matching degree of the prototype vectors for each sound image category. Taking S2 (slight bearing wear) as an example, this state type has two prototype vectors for sound image categories: "inner ring wear prototype vector P2a" and "outer ring wear prototype vector P2b" (both are prototype vectors of the first sound image category, which have been optimized through the first round of training). The server iterates through 150 samples in the baseline spectrum instance set of S2, calculates the cosine similarity (matching degree) between the time-frequency feature vector of each sample and P2a, and records all similarity values; then, it calculates the matching degree between the features of all samples and P2b in the same way. After completion, the server takes the arithmetic mean of the 150 matching degrees of P2a to obtain the baseline matching degree of P2a (e.g., 0.83); and takes the arithmetic mean of the 150 matching degrees of P2b to obtain the baseline matching degree of P2b (e.g., 0.76). Other state types are processed using the same logic: for example, the two prototype vectors P1a and P1b of S1 (normal operation) are calculated to have an average matching degree of 0.85 and 0.78 with the features of 160 samples in the S1 reference set, respectively; the average matching degree of P3a and P3b of S3 (severe bearing wear) is 0.81 and 0.75, and so on.

[0077] Finally, the server identifies the prototype vector of the audio-visual category with the highest baseline matching degree under each state type as the core audio-visual category prototype vector. In S2, the baseline matching degree of P2a (0.83) is higher than that of P2b (0.76), so the server sets P2a as the core prototype vector of S2; in S1, P1a (0.85) is higher than that of P1b (0.78), so P1a is identified as the core prototype; in S3, P3a (0.81) is higher than that of P3b (0.75), so P3a is identified as the core prototype, and the same applies to the other state types. These core prototype vectors represent the most representative feature centers under the corresponding state types. Subsequently, the server will calculate the second error based on these core prototypes to further optimize the model's ability to identify typical fault features.

[0078] In this embodiment of the invention, there are multiple second spectrum instances. The step of obtaining the second error based on the time-frequency characteristics of the second spectrum instance, the target value of the state type of the second spectrum instance, and the first sound-image category prototype vector corresponding to the core sound-image category prototype vector of each state type can be implemented through the following example.

[0079] The matching degree between the time-frequency features of each second spectrum instance and the first sound-image category prototype vector corresponding to the core sound-image category prototype vector of each state type is calculated to obtain the second matching degree between the time-frequency features of each second spectrum instance and the core sound-image category prototype vector of each state type.

[0080] The state type corresponding to the core acoustic image category prototype vector with the highest second matching degree between at least two state types and the time-frequency features of the second spectrum instance is taken as the inferred state type of the second spectrum instance.

[0081] The second error is obtained based on the target value of the state type of the second spectrum instance and the inferred state type of the second spectrum instance.

[0082] In this embodiment of the invention, for example, during the second round of training of the industrial motor fault diagnosis model, the second spectrum instance set processed by the server contains multiple labeled samples, covering five state types: "normal operation" (S1), "slight bearing wear" (S2), "severe bearing wear" (S3), "rotor imbalance" (S4), and "stator winding fault" (S5). Each state corresponds to 50 samples, for a total of 250 second spectrum instances. Some of these instances come from the first spectrum instance set (screened high-confidence samples), and some are newly collected motor operating sound spectra (such as the recently added "bearing cage noise" sample in the workshop), all of which have been preprocessed and converted into standardized spectrograms. The server deploys the second acoustic image feature recognition model (the model optimized in the first round of training) to the NPU and performs batch feature extraction on all second spectrum instances: the NPU performs sliding window calculation on the time-frequency region of the spectrogram through convolutional layers, and after ReLU activation and pooling processing, the fully connected layer outputs a 512-dimensional time-frequency feature vector. The server groups these vectors according to state type and stores them in memory. The second adjustment factor generation rule is as follows: Second adjustment factor = 1.0 - (current round × single round attenuation); Single round attenuation = 1.0 / total training rounds in the second round; The second sound-image spacing parameter generation rule is as follows: Second sound-image spacing parameter = second baseline sound-image spacing parameter + (second adjustment factor × spacing correction coefficient); The spacing correction coefficient is preset to 0.4; The second feature scale factor generation rule is as follows: Second feature scale factor = second baseline feature scale factor + (second adjustment factor × scale correction coefficient); The scale correction coefficient is preset to 0.3.

[0083] Next, the server calculates the second matching degree between each second spectrum instance and the core prototype vector of the state type. The core acoustic image category prototype vectors for each state type have been determined in the previous stage: S1 is the "smooth operation prototype vector P1a", S2 is the "inner race wear prototype vector P2a", S3 is the "rolling element wear prototype vector P3a", S4 is the "mass eccentricity prototype vector P4a", and S5 is the "inter-turn short circuit prototype vector P5a". These prototype vectors have been optimized through the first round of training and represent the typical feature centers of the corresponding states. The server traverses each second spectrum instance and calculates the cosine similarity (second matching degree) between its 512-dimensional time-frequency features and the five core prototype vectors. For example, a second spectrum instance belonging to S2 (target state type S2) has a feature vector matching degree of 0.85 with P2a, 0.32 with P1a, 0.68 with P3a, 0.45 with P4a, and 0.53 with P5a; another instance belonging to S4 has a feature vector matching degree of 0.82 with P4a, 0.29 with P1a, and 0 with P2a. 41 has a matching degree of 0.58 with P3a and 0.38 with P5a; instances belonging to S1 have a matching degree of 0.87 with P1a and a matching degree of less than 0.4 with other prototypes; instances belonging to S3 have a matching degree of 0.83 with P3a and a highest matching degree of 0.55 with other prototypes (P2a); instances belonging to S5 have a matching degree of 0.80 with P5a and a highest matching degree of 0.52 with other prototypes (P3a).

[0084] Subsequently, the server determines the inferred state type for each second spectrum instance. For each instance, the server selects the maximum value from the second matching degrees of the five core prototypes; the corresponding state type is the inferred state type of that instance. In the S2 instance above, P2a's 0.85 is the maximum value, and the inferred state type is S2; in the S4 instance, P4a's 0.82 is the maximum value, and the inferred state type is S4; the inferred state types of S1, S3, and S5 instances are also consistent with the target type. However, there are some special cases: a second spectrum instance labeled as S2 but actually close to S3 (a transitional state of accelerated bearing wear) has a matching degree of 0.65 with P2a and a matching degree of 0.72 (the maximum value) with P3a. In this case, the inferred state type is S3, which is inconsistent with the target type S2.

[0085] Finally, the server calculates the second error based on the target state type and the inferred state type of all second spectrum instances. For each instance, the server uses the cross-entropy loss function to calculate the classification error: when the inferred state type matches the target type (such as most instances of S1, S2, S3, S4, and S5 above), the error value is small (usually below 0.1, such as 0.08 for instance S1 and 0.07 for instance S2); when the inferred type does not match the target type (such as the transitional state instance above), the error value increases significantly (such as 0.65 for this instance). The server sums the classification errors of all 250 second spectrum instances and takes the average to obtain the total second error (for example, the average error of 248 correct instances is 0.09, the error of 2 incorrect instances is 0.65, and the total second error is (248 × 0.09 + 2 × 0.65) / 250 ≈ 0.10). This second error reflects the classification performance of the model under the guidance of the core prototype vector. The server uses it for backpropagation to further optimize the parameters and core prototype vector of the second audio-visual feature recognition model until the model converges.

[0086] In this embodiment of the invention, the step of obtaining the second error based on the target value of the state type of the second spectrum instance and the inferred state type of the second spectrum instance can be implemented through the following example.

[0087] Based on the number of the second round corresponding to the training round in which the second spectrum instance is located, the second acoustic-image spacing parameter and the second feature scale factor are obtained. The second acoustic-image spacing parameter and the second feature scale factor have negative feedback with the number of the second round. The inter-class separation error is calculated based on the second acoustic-image spacing parameter, the second feature scale factor, the target value of the state type of the second spectrum instance, and the inferred state type of the second spectrum instance, to obtain the second error.

[0088] In this embodiment of the invention, for example, during the second round of training of the industrial motor fault diagnosis model, the server is processing 250 second spectrum instances (covering 5 state types such as "normal operation" and "minor bearing wear"), and is currently in the 50th round of training (the second round has 50 iterations, and the total number of second training rounds is preset to 150). To dynamically optimize the model's control over inter-class separation in the feature space, the server first determines the second acoustic-image spacing parameter and the second feature scale factor based on the number of the second round. These two parameters have a negative feedback relationship with the number of rounds; they are larger when the number of rounds is smaller in the early stages of training and gradually decrease as the number of rounds increases, in order to achieve the training objective of "first strengthening inter-class differences, and then finely adjusting the classification boundaries".

[0089] Specifically, the server has a built-in second-round adjustment rule: the initial value of the second adjustment factor is 1.0 (corresponding to the first round). With each additional round of training, the adjustment factor decreases linearly by a fixed margin (decreasing by 0.0067 per round for a total of 150 rounds). At the 50th round, the server reads the current second-round count as "50" and calculates the second adjustment factor as 1.0 - 50 × 0.0067 ≈ 0.665. If training continues to the 100th round, the adjustment factor decreases to 1.0 - 100 × 0.0067 ≈ 0.33, and at the 150th round, it decreases to 0 (at which point the parameter is no longer affected by the adjustment factor).

[0090] Based on the second adjustment factor, the server calculates the second acoustic-image spacing parameter and the second feature scaling factor. The second acoustic-image spacing parameter is used to control the degree of separation of features of different state types. Its baseline value (second baseline acoustic-image spacing parameter) is preset to 1.2 (initial value, to ensure that the inter-class distance is large enough in the early stage of training). The server obtains the spacing correction value by multiplying the adjustment factor and the spacing correction coefficient (preset 0.4): at the 50th round, the correction value = second adjustment factor (0.665) × spacing correction coefficient (0.4) ≈ 0.266. After adding the baseline parameter, the second acoustic-image spacing parameter is obtained as 1.2 + 0.266 ≈ 1.466. The second feature scaling factor is used to adjust the overall distribution range of feature vectors, enhancing the discriminative power of subtle differences. Its baseline value (second baseline feature scaling factor) is preset to 1.0. The scaling correction value is obtained by multiplying the adjustment factor and the scaling correction coefficient (preset to 0.3): at the 50th round, the correction value = 0.665 × 0.3 ≈ 0.1995, and after adding the baseline factor, the second feature scaling factor = 1.0 + 0.1995 ≈ 1.1995. As the number of rounds increases, the adjustment factor decreases, leading to a decrease in the scaling correction value, and the parameters will gradually decrease (e.g., at the 100th round, the distance parameter ≈ 1.2 + 0.33 × 0.4 ≈ 1.332, and the scaling factor ≈ 1.0 + 0.33 × 0.3 ≈ 1.099), ensuring that later training focuses on fine optimization of the classification boundary.

[0091] Next, the server combines the second acoustic-image spacing parameter, the second feature scale factor, and the target and inferred state types of the second spectrum instance to calculate the inter-class separation error, thus obtaining the second error. Taking a specific second spectrum instance as an example: an instance labeled "slight bearing wear" (target state type S2) has an inferred state type of S2 (correct match), with a feature matching degree of 0.85 with the core prototype vector of S2, and a difference of 0.17 compared to the highest matching degree among non-target types (0.68 in S3); another instance labeled "slight bearing wear," because its feature is close to "severe bearing wear" (S3), has an inferred state type of S3 (incorrect match), with a matching degree of 0.65 with the core prototype of S2 and a matching degree of 0.72 with the core prototype of S3, a difference of -0.07 (i.e., the target matching degree is lower than the non-target); and there is also an instance of "normal operation" (S1), with an inferred type of S1 (correct), a target matching degree of 0.87, and the highest non-target matching degree of 0.42, a difference of 0.45.

[0092] The server calculates the error for each instance using an inter-class separation loss function: when the inferred type matches the target type and the difference between the target matching degree and the highest non-target matching degree is greater than a preset boundary value (e.g., 0.2), the error is 0 (e.g., for instance S1, the difference is 0.45 > 0.2, so the error is 0); when the difference is less than the boundary value but the inference is correct (e.g., for instance S2, the difference is 0.17 < 0.2), the error is the difference between the boundary value and the difference (0.2 - 0.17 = 0.03); when the inference is incorrect (e.g., for instance S2, the difference is -0.07), the error is the sum of the absolute values ​​of the boundary value and the difference (0.2 + 0.07 = 0.27). Subsequently, the server weights the error of incorrect instances with a second acoustic-image spacing parameter (enhancing the penalty for misclassification), weights the small error of correct instances with a second feature scale factor (fine-tuning the feature distribution), and sums the weighted errors of all instances and takes the average to obtain the second error. For example, among 250 instances, the average small error of 240 correct instances is 0.05, and the average error of 10 incorrect instances is 0.3. After parameter weighting (the spacing parameter of 1.466 weights the error of incorrect instances, and the scale factor of 1.1995 weights the error of correct instances), the total second error is reflected in "the larger errors of incorrect instances are amplified, and the small errors of correct instances are fine-tuned". The server uses this second error for backpropagation to further adjust the parameters and core prototype vector of the second audio-visual feature recognition model until the model's ability to separate inter-class features and the classification accuracy reach the expected level. The calculation rules for the inter-class separation error are as follows: If the inferred state type is consistent with the target state type: when the target matching degree - the highest non-target matching degree ≥ the boundary value (0.2): error = 0; otherwise: error = boundary value - (target matching degree - the highest non-target matching degree); if the inferred state type is inconsistent with the target state type: error = boundary value + |target matching degree - the highest non-target matching degree|; the final second error is the average of the errors of all instances.

[0093] In this embodiment of the invention, the step of obtaining the second acoustic-image spacing parameter and the second feature scale factor based on the number of the second round corresponding to the training round in which the second spectrum instance is located can be implemented through the following example.

[0094] A second adjustment factor is obtained based on the number of the second round corresponding to the training round in which the second spectrum instance is located. The second adjustment factor has negative feedback with the number of the second round.

[0095] The second acoustic-image spacing correction value is obtained based on the second adjustment factor. The second reference acoustic-image spacing parameter is added to the second acoustic-image spacing correction value to obtain the second acoustic-image spacing parameter.

[0096] The second feature scale correction value is obtained based on the second adjustment factor. The second baseline feature scale factor and the second feature scale correction value are added together to obtain the second feature scale factor.

[0097] In an embodiment of the invention, for example, during the second round of training of the industrial motor fault diagnosis model, the server is training a dataset containing 250 second spectrum instances. Currently, it is in the 40th round of training (the second round has 40 iterations, and the total number of second training rounds is preset to 200). These second spectrum instances cover five state types: "normal operation" (S1), "slight bearing wear" (S2), "severe bearing wear" (S3), "rotor imbalance" (S4), and "stator winding fault" (S5). Each state corresponds to 50 samples, some of which are newly collected motor operating sound spectra (such as sub-scenario samples like "abnormal bearing cage noise" and "stator insulation aging"). All samples have been converted into standardized spectrum diagrams and stored in the server's local database.

[0098] To dynamically adjust the model's control strength for inter-class feature separation, the server first calculates a second adjustment factor based on the current second round number. This factor has a negative feedback relationship with the number of rounds; in the early stages of training, when the number of rounds is small, the adjustment factor is larger to strengthen the separation of features from different state types. As the number of rounds increases, the adjustment factor gradually decreases, allowing the model to focus on fine-tuning the classification boundary. The server has a built-in calculation rule for the second adjustment factor: the initial value is set to 1.0 (corresponding to the first round of training), and with each additional round, the adjustment factor decreases linearly by a fixed amount, the decrease being determined based on the total number of second rounds (0.005 decreases per round when the total number of rounds is 200). Therefore, at the 40th round of training, the server reads the current second round number "40" from the training log and calculates the second adjustment factor as 1.0 - 40 × 0.005 = 0.8; if training progresses to the 100th round, the adjustment factor will decrease to 1.0 - 100 × 0.005 = 0.5; and at the 200th round, it decreases to 0 (at which point the adjustment factor no longer affects parameter calculation).

[0099] Next, the server calculates the second acoustic-image spacing correction value based on the second adjustment factor, thus obtaining the second acoustic-image spacing parameter. The second acoustic-image spacing parameter controls the separation distance between features of different state types. Its baseline value (the second baseline acoustic-image spacing parameter) is preset by the server to 1.3 (an initial value to ensure sufficient inter-class distance in the early stages of training to avoid feature overlap). To control the correction magnitude, the server has a built-in spacing correction coefficient of 0.5 (experiments have verified that this coefficient balances separation strength and model stability). In the 40th round, the second acoustic-image spacing correction value = second adjustment factor (0.8) × spacing correction coefficient (0.5) = 0.4. The server adds this correction value to the second baseline acoustic-image spacing parameter to obtain the second acoustic-image spacing parameter for the current round: 1.3 + 0.4 = 1.7. As the adjustment factor decreases with each round, the correction value also gradually decreases. Therefore, the second acoustic-image spacing parameter will gradually decrease from the initial 1.7 (round 40) as training progresses. By round 100 (adjustment factor 0.5), the correction value = 0.5 × 0.5 = 0.25, and the parameter becomes 1.3 + 0.25 = 1.55. By round 200, the correction value is 0, and the parameter is equal to the baseline value of 1.3. At this point, the model no longer needs to strengthen inter-class separation, but instead relies on the discriminative power of the features themselves.

[0100] Simultaneously, the server calculates the second feature scaling factor using the same logic. The second feature scaling factor is used to adjust the overall distribution range of the feature vectors and amplify subtle feature differences (such as the high-frequency energy difference between the inner and outer ring wear of a bearing). Its baseline value (the second baseline feature scaling factor) is preset to 0.9 (initial value, to avoid overfitting due to excessively large feature scales). The server has a built-in scaling correction coefficient of 0.3 (after debugging, this coefficient effectively enhances the local discriminative power of features). The second feature scaling correction value is obtained by multiplying the second adjustment factor and the scaling correction coefficient: In the 40th round, the correction value = second adjustment factor (0.8) × scaling correction coefficient (0.3) = 0.24. The server adds this correction value to the second baseline feature scaling factor to obtain the current second feature scaling factor: 0.9 + 0.24 = 1.14. Similar to the image-sound spacing parameter, as the number of rounds increases, the adjustment factor decreases, leading to a decrease in the correction value. The second feature scale factor gradually decreases. By the 100th round (adjustment factor 0.5), the correction value = 0.5 × 0.3 = 0.15, and the factor becomes 0.9 + 0.15 = 1.05. By the 200th round, the correction value is 0, and the factor equals the baseline value of 0.9, ensuring the stability of the feature distribution in later training and avoiding interference caused by excessive amplification of noise.

[0101] Through the above process, the server dynamically obtains the second acoustic-image spacing parameter (1.7) and the second feature scale factor (1.14) in the 40th round of the second training. These two parameters will be used to calculate the inter-class separation error in the subsequent calculation. By weighting the difference in feature matching degree between the target state and the non-target state, the model's ability to distinguish different fault types will be optimized, providing a basis for adjusting model parameters (such as convolutional layer weights and core prototype vectors) for backpropagation, and finally achieving accurate classification of motor fault types.

[0102] In this embodiment of the invention, the step of calculating the matching degree between the target time-frequency feature and at least two reference time-frequency features in a preset audio-visual feature library can be implemented through the following example: Obtain multiple local feature vectors of the target time-frequency feature; calculate the cosine similarity between each local feature vector of the target time-frequency feature and all local feature vectors of a reference time-frequency feature in the preset audio-visual feature library, obtaining multiple similarity values; sort the multiple similarity values ​​and select the top K largest similarity values; calculate the average of the top K largest similarity values ​​as the final matching degree between the target time-frequency feature and the reference time-frequency feature.

[0103] In this embodiment of the invention, for example, in an industrial motor fault diagnosis system, after the server completes the extraction of the target time-frequency features, it needs to calculate the matching degree with the reference time-frequency features in the preset acoustic-image feature library to determine the current operating state of the motor. The preset acoustic-image feature library is stored on the server's SSD hard drive and contains reference time-frequency features for five state types: "normal operation", "slight bearing wear", "severe bearing wear", "rotor imbalance", and "stator winding fault". Each state corresponds to 10 typical samples (e.g., "slight bearing wear" includes "inner ring wear reference sample 1", "outer ring wear reference sample 2", etc.). Each reference sample has been extracted into a 512-dimensional feature vector through previous training and divided into local feature vectors for storage.

[0104] The server first acquires multiple local feature vectors of the target time-frequency feature. The target time-frequency feature is a 512-dimensional dense vector output by the NPU (e.g., the feature vector corresponding to the current spectrum of motor No. 3). The server reads this vector from memory and divides it into local feature vectors according to a preset dimension (each group of 16 dimensions). Specifically, the 512-dimensional vector is divided into 32 local feature vectors, each containing feature values ​​of 16 consecutive dimensions (e.g., dimensions 1-16 are the first local vector, dimensions 17-32 are the second local vector, and so on), forming a set of local feature vectors. Each vector is L2 normalized (with a magnitude of 1) to ensure the stability of subsequent similarity calculations.

[0105] Next, the server calculates the cosine similarity between the target local feature vector and all local feature vectors of the benchmark time-frequency feature. Taking a benchmark sample (denoted as benchmark sample B2-3) in the state of "slight bearing wear" as an example, the 512-dimensional features of this sample are also divided into 32 16-dimensional local feature vectors (stored in the "B2-3 local vector group" in the feature library). The server iterates through the 32 local vectors of the target. For each local vector of the target (e.g., the first local vector v1), it calculates the cosine similarity with each of the 32 local vectors of the benchmark sample B2-3 (e.g., the first local vector u1, the second local vector u2, etc. of B2-3). The cosine similarity is then calculated by dividing the vector dot product by the product of the magnitudes of the two vectors (since normalized, the magnitude product is 1, so the dot product value is directly taken), resulting in 32 similarity values. Subsequently, the second local vector v2 of the target is calculated with the 32 local vectors of B2-3, yielding another 32 similarity values… and so on. A total of 32 × 32 = 1024 similarity values ​​are generated between the 32 local vectors of the target and the 32 local vectors of the benchmark sample B2-3. If there are 10 benchmark samples in this state, the server needs to repeat the above calculation for each sample, resulting in a total of 10 × 1024 = 10240 similarity values ​​(in actual calculations, the server processes benchmark samples in batches, processing all local vectors of one benchmark sample in each batch).

[0106] After similarity calculation, the server sorts all similarity values ​​and selects the top K largest values. The server uses its built-in quicksort algorithm to sort the 1024 similarity values ​​of the current benchmark sample (e.g., B2-3) in descending order, extracting the top K values ​​from largest to smallest (K is a preset parameter, set to 50 based on feature stability experiments to ensure coverage of the most similar local feature pairs). For example, the top 50 similarity values ​​after sorting are 0.92, 0.91, 0.89, ..., 0.78 (all are cosine similarity, ranging from -1 to 1; larger values ​​indicate greater similarity in local features).

[0107] Finally, the server calculates the arithmetic mean of the top K similarity values ​​as the final matching degree between the target time-frequency feature and the baseline time-frequency feature. The sum of the above 50 similarity values ​​is divided by 50 to obtain the average similarity (e.g., (0.92+0.91+…+0.78) / 50≈0.85), which is the matching degree between the target feature and the baseline sample B2-3. The server iterates through all 10 baseline samples of the "slight bearing wear" state using the same process, calculating the matching degree for each, and taking the maximum value as the overall matching degree between the target feature and this state type. The same operation is then performed on the other four state types, ultimately obtaining a list of matching degrees between the target feature and the five state types, providing a basis for subsequent classification reasoning.

[0108] In this embodiment of the invention, after obtaining the second acoustic image feature recognition model and before extracting time-frequency features from the second spectrum instance using the second acoustic image feature recognition model, the following implementation method is provided: Calculate the absolute values ​​of the weight parameters of each convolutional layer in the second acoustic image feature recognition model; according to a preset pruning rate, set the weight parameters with the smallest absolute values ​​to zero to obtain a sparsed second acoustic image feature recognition model; retrain the sparsed second acoustic image feature recognition model to restore the model accuracy lost due to pruning; wherein, the extraction of time-frequency features from the second spectrum instance using the second acoustic image feature recognition model includes: extracting time-frequency features from the second spectrum instance using the retrained sparsed second acoustic image feature recognition model.

[0109] In an embodiment of the present invention, for example, in the training process of the industrial motor fault diagnosis model, after the server completes the first round of training and obtains the second acoustic image feature recognition model, in order to optimize the deployment efficiency of the model on the NPU (reduce the consumption of computing resources and speed up the inference speed), the model will be sparsified and retrained, and then used for feature extraction of the second spectrum instance.

[0110] The second audio-visual feature recognition model comprises two convolutional blocks: Convolutional Block 1 contains two convolutional layers (both with 3×3 kernels, 3 input channels, 64 output channels, and ReLU activation), and Convolutional Block 2 contains two convolutional layers (3×3 kernels, 64 input channels, 128 output channels, and ReLU activation), followed by a global average pooling layer and a fully connected layer. The server first reads the weight parameters of each convolutional layer from the model parameter file. The weights of each convolutional layer are stored in the form of a four-dimensional tensor (e.g., the weight dimension of the first layer of Convolutional Block 1 is 3×3×3×64, representing a 3×3 spatial size, 3 input channels, and 64 output channels for the convolutional kernel parameters). The server iterates through the weight parameters of all convolutional layers, calculates the absolute value of each parameter (e.g., if a kernel parameter value is -0.023, its absolute value is 0.023; if a parameter value is 0.156, its absolute value is 0.156), and sorts all the absolute values ​​of the parameters in ascending order, storing them in a temporary array.

[0111] Based on the preset pruning rate (set to 30% by the server considering the computational resource limitations of the NPU and the model accuracy requirements), the server determines the proportion of parameters that need to be zeroed. For example, the first layer of convolutional block 1 has 3×3×3×64=1728 weight parameters, and a 30% pruning rate corresponds to 518 parameters needing to be zeroed. The server selects the first 518 parameters with the smallest absolute value (e.g., parameters with an absolute value ≤ 0.05) from the sorted absolute value array and sets their values ​​directly to zero, retaining the 70% of parameters with larger absolute values ​​(these parameters are more critical for feature extraction, such as convolutional kernel parameters that capture high-frequency impacts from bearing wear). The above operation is repeated for all convolutional layers of the model: 518 parameters in the second layer of convolutional block 1 (also with 1728 parameters) are zeroed; 22,118 parameters in the first layer of convolutional block 2 (3×3×64×128=73728 parameters) are zeroed; and 22,118 parameters in the second layer of convolutional block 2 (73728 parameters) are zeroed. Fully connected layers are not pruned due to their smaller number of parameters and their significant impact on accuracy. After zeroing, a large number of zero values ​​appear in the model's weight matrix, forming a sparse second acoustic feature recognition model. This reduces storage volume by approximately 30%, and zero-value multiplication can be skipped during computation, resulting in an expected 25% improvement in NPU inference speed.

[0112] To recover the accuracy loss caused by pruning (the model's ability to extract some subtle fault features decreased after pruning, such as the high-frequency features of minor cracks in the outer ring of a bearing), the server retrained the sparse model. The retraining dataset used a portion of the first spectrum instance set (e.g., 100 typical samples for each state type, totaling 500 samples). The number of training epochs was set to 1 / 4 of the first epoch (e.g., 50 epochs), and the learning rate was reduced to 1 / 10 of the initial value (e.g., 0.0001) to avoid overfitting due to excessive adjustment. During training, the server froze the zeroed weight parameters (which did not participate in backpropagation updates) and only adjusted the non-zero weight parameters: features were extracted from the samples using the NPU, the loss function (cross-entropy loss) was calculated, and the non-zero weights were optimized through backpropagation to gradually restore the model's ability to capture key features (e.g., the classification accuracy of the "minor bearing wear" sample dropped from 92% to 85% after pruning, but rebounded to 91.5% after 20 epochs of retraining, approaching the level before pruning).

[0113] After retraining, the server solidifies the model parameters, resulting in a retrained sparse second acoustic-image feature recognition model. When processing the second spectrum instance, the server deploys this sparse model to the NPU: the NPU loads the sparse weight matrix and uses its own sparse computation units (supporting skipping zero-valued parameters) to extract features from the standardized spectrogram of the second spectrum instance. The convolutional layers perform sparse convolution operations (only calculating the product of non-zero weights and input features), and after pooling and activation, output a 512-dimensional time-frequency feature vector, which is stored in the server's memory for error calculation and model optimization in the subsequent second round of training. Through this process, the model maintains high accuracy while achieving lightweight design, meeting the resource requirements of real-time NPU inference in industrial scenarios.

[0114] In this embodiment of the invention, the following implementation methods are also provided.

[0115] When the matching degree between the target time-frequency feature and all reference time-frequency features is lower than the preset confidence threshold, it is determined that the spectrum to be processed belongs to an unknown audio-visual category; the manual annotation result of the user on the unknown audio-visual category is received; the target time-frequency feature, its corresponding spectrum to be processed, and the manually annotated audio-visual identification code are added together as new reference samples to the preset audio-visual feature library.

[0116] In an embodiment of the present invention, for example, in an industrial motor fault diagnosis system, after the server completes the matching degree calculation of the target time-frequency features and the preset acoustic-image feature library, it will perform threshold judgment on the matching results to identify unknown acoustic-image categories and dynamically update the feature library.

[0117] At 10:15 AM on a weekday, the server received the spectrum of motor #5 to be processed. This spectrum was generated from the preprocessed sound of the motor operating, collected by a microphone, and the corresponding sound signal exhibited "intermittent metallic knocking sounds" (unlike typical fault sounds such as "bearing wear" and "rotor imbalance"). The server extracted the target time-frequency features (512-dimensional vector) of this spectrum using the NPU and calculated the matching degree with all the benchmark time-frequency features in the preset acoustic image feature library. The preset acoustic image feature library contains five state types: "normal operation" (M001), "slight bearing wear" (M002), "severe bearing wear" (M003), "rotor imbalance" (M004), and "stator winding fault" (M005). Each state corresponds to 10 benchmark samples, for a total of 50 benchmark time-frequency features. The server calculated the matching degree based on the top K average values ​​of the cosine similarity of the local feature vectors: the highest matching degree with M002 was 0.65 (corresponding to the 3rd benchmark sample of "slight bearing wear"), the highest matching degree with M003 was 0.62, and the matching degree with other states was all below 0.6. The server's preset confidence threshold is 0.7 (verified through historical fault data, this threshold can balance the false positive rate and the unknown category recognition rate). Since all matching degrees (maximum 0.65) are below 0.7, the server determines that the spectrum to be processed belongs to an unknown audio-visual category.

[0118] Subsequently, the server triggered the unknown category processing procedure: an alarm window popped up on the factory operation and maintenance monitoring platform, displaying "Unknown fault type detected in motor No. 5 (timestamp: 10:15:23), please manually annotate," and provided the original audio clip (downloadable and playable), a time-frequency feature visualization graph (showing the time-frequency energy distribution of the spectrum), and a matching degree ranking table for each state type corresponding to the spectrum. The maintenance personnel downloaded the audio clip through the platform, used a professional stethoscope to confirm the sound of motor No. 5 on-site, and combined with disassembly inspection, discovered that "bearing cage fracture caused rolling element jamming," which belonged to a new, unrecorded fault type. The maintenance personnel entered "bearing cage fracture" in the annotation interface of the monitoring platform and submitted the annotation result. The server received the manually annotated information via HTTP interface, automatically assigned a new audio-visual identifier code "M006" (generated according to the feature library encoding rules), and simultaneously associated and stored the target time-frequency feature of the unknown category, the corresponding spectrum to be processed (standardized spectrum graph data), and the annotation timestamp (10:28:15).

[0119] Finally, the server adds the new sample to the preset sound and image feature library. The preset sound and image feature library is stored in the server's distributed file system (path " / data / sound_feature_library / "), using a three-level directory structure of "status type-benchmark sample-feature data": M001 corresponds to the "normal operation" directory, which contains 10 subdirectories (each benchmark sample). The subdirectories store time-frequency feature vector files, spectrum files, and metadata (such as acquisition time, motor model). The server creates a sibling directory "M006_bearing cage fracture" for the new identifier M006, and creates the first benchmark sample subdirectory "sample_001" under this directory. The received target time-frequency feature vector (512-dimensional floating-point array, saved as .npy format), the spectrum to be processed (standardized 100×1025 matrix, saved as .csv format), and manually labeled metadata (labeler, labeling time, fault description) are stored in the subdirectory. Simultaneously, the server updates the feature library index file (JSON format), adding an entry for M006 to the "Status Type List," recording its name "Bearing Cage Fracture," the number of baseline samples "1," and the feature dimension "512." After addition, the server records in the system log "Added unknown category sample M006, added to the preset acoustic feature library," and prompts maintenance personnel to "Trigger incremental model training to update the target acoustic feature recognition model's ability to recognize M006." Subsequently, when motor No. 5 produces the same sound again, the server can match it using the newly added baseline sample and output the corresponding acoustic identification code M006, achieving dynamic identification of unknown fault types and expansion of the feature library.

[0120] This invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned NPU-based deep learning classification and reasoning method. Figure 2 As shown, Figure 2 This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a memory 111, a processor 112, and a communication unit 113. To enable data transmission or interaction, the memory 111, processor 112, and communication unit 113 are electrically connected to each other directly or indirectly. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0121] For illustrative purposes, the foregoing description is based on specific embodiments. However, the foregoing illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in accordance with the foregoing teachings. These embodiments were chosen and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the disclosure and to employ various embodiments with different modifications to suit a particular intended application.

Claims

1. A deep learning-based classification and reasoning method for audio-visual imaging based on an NPU, characterized in that, include: Obtain the spectrum to be processed; The spectrum to be processed is input into the NPU, and the NPU performs feature extraction operation on the spectrum to be processed based on the pre-trained target acoustic image feature recognition model to obtain the target time-frequency features. The matching degree between the target time-frequency feature and at least two reference time-frequency features in the preset audio-visual feature library is calculated to obtain the matching degree between the target time-frequency feature and each reference time-frequency feature. Based on the matching degree between the target time-frequency feature and each reference time-frequency feature, the target reference time-frequency feature with the highest matching degree with the target time-frequency feature is obtained; The acoustic image identification code associated with the target reference time-frequency features is used as the acoustic image classification reasoning result of the spectrum to be processed; The acoustic-image feature recognition model is obtained through the following methods: The first acoustic image feature recognition model is deployed to the NPU to extract time-frequency features of the first spectrum instance and obtain the time-frequency features of the first spectrum instance. The first error is obtained based on the time-frequency characteristics of the first spectrum instance, the target value of the state type of the first spectrum instance, and the initial sound image category prototype vectors corresponding to at least two sound image category prototype vectors of each state type. Based on the first error, adjust the model parameters of the first audio-visual feature recognition model and the initial audio-visual category prototype vectors corresponding to at least two audio-visual category prototype vectors of each state type respectively to obtain the first audio-visual category prototype vectors corresponding to the second audio-visual feature recognition model and at least two audio-visual category prototype vectors of each state type respectively. The second acoustic image feature recognition model is deployed to the NPU to extract time-frequency features of the second spectrum instance, thereby obtaining the time-frequency features of the second spectrum instance; The second error is obtained based on the time-frequency characteristics of the second spectrum instance, the target value of the state type of the second spectrum instance, and the first sound image category prototype vector corresponding to the core sound image category prototype vector of each state type. Based on the second error, adjust the model parameters of the second sound image feature recognition model and the first sound image category prototype vector corresponding to the core sound image category prototype vector of each state type until the trained target sound image feature recognition model is obtained. The first spectrum instance is multiple. Before obtaining the second error based on the time-frequency characteristics of the second spectrum instance, the state type target value of the second spectrum instance, and the first sound image category prototype vector corresponding to the core sound image category prototype vector of each state type, the method further includes: Obtain the reference spectrum instance set corresponding to each state type. The reference spectrum instance set corresponding to a state type includes multiple first spectrum instances whose state type target value is the same as that state type. For each state type, the baseline matching degree corresponding to each sound image category prototype vector under that state type is obtained based on the matching degree between the center vector of each sound image category prototype vector in at least two sound image category prototype vectors under that state type and the time-frequency features of each first spectrum instance in the baseline spectrum instance set corresponding to that state type. For each state type, the prototype vector of the audio-visual category with the highest baseline matching degree under that state type is taken as the core audio-visual category prototype vector of that state type.

2. The method according to claim 1, characterized in that, The step of obtaining the first error based on the time-frequency characteristics of the first spectrum instance, the target value of the state type of the first spectrum instance, and the initial sound image category prototype vectors corresponding to at least two sound image category prototype vectors of each state type includes: For each state type, calculate the first matching degree between the time-frequency features of the first spectrum instance and the initial sound-image category prototype vectors corresponding to at least two sound-image category prototype vectors respectively; The initial sound image category prototype vector with the highest first matching degree between the at least two initial sound image category prototype vectors of each state type and the time-frequency features of the first spectrum instance is taken as the reference sound image category prototype vector. The first sub-item error is obtained based on the benchmark acoustic image category prototype vector and the state type target value of the first spectrum instance; The first error is obtained based on the first matching degree between the time-frequency characteristics of the first spectrum instance and the initial sound image category prototype vectors corresponding to at least two sound image category prototype vectors of each state type, and the first sub-item error.

3. The method according to claim 2, characterized in that, The step of obtaining the first error based on the first matching degree between the time-frequency characteristics of the first spectrum instance and the initial sound image category prototype vectors corresponding to at least two sound image category prototype vectors of each state type, and the first sub-term error, includes: For each state type, based on the first matching degree between the time-frequency features of the first spectrum instance and the initial sound-image category prototype vectors corresponding to at least two sound-image category prototype vectors of the state type, the second sound-image category prototype vector with the highest first matching degree between the time-frequency features of the first spectrum instance under the state type is obtained. The second sub-item error is obtained based on the first matching degree between the second acoustic image category prototype vector under each of at least two state types and the time-frequency features of the first spectrum instance; Based on the number of the first round corresponding to the training round in which the first spectrum instance is located, a first adjustment factor is obtained, and the first adjustment factor has negative feedback with the number of the first round. The first acoustic-image spacing correction value is obtained based on the first adjustment factor. The first acoustic-image spacing correction value is added to the first reference acoustic-image spacing parameter to obtain the first acoustic-image spacing parameter. The first feature scale correction value is obtained according to the first adjustment factor. The first feature scale correction value is added to the first reference feature scale factor to obtain the first feature scale factor. The first sound image spacing parameter and the first feature scale factor have negative feedback with the first round number. The prototype distance error is calculated based on the first acoustic-image spacing parameter, the first feature scale factor, the first sub-item error, and the second sub-item error to obtain the first error.

4. The method according to claim 1, characterized in that, The second spectrum instance is multiple, and the step of obtaining the second error based on the time-frequency characteristics of the second spectrum instance, the state type target value of the second spectrum instance, and the first sound image category prototype vector corresponding to the core sound image category prototype vector of each state type includes: The matching degree between the time-frequency features of each second spectrum instance and the first sound image category prototype vector corresponding to the core sound image category prototype vector of each state type is calculated to obtain the second matching degree between the time-frequency features of each second spectrum instance and the core sound image category prototype vector of each state type. The state type corresponding to the core acoustic image category prototype vector with the highest second matching degree between at least two state types and the time-frequency features of the second spectrum instance is taken as the inferred state type of the second spectrum instance. A second adjustment factor is obtained based on the number of the second round corresponding to the training round in which the second spectrum instance is located. The second adjustment factor has negative feedback with the number of the second round. The second acoustic-image spacing correction value is obtained based on the second adjustment factor. The second reference acoustic-image spacing parameter is added to the second acoustic-image spacing correction value to obtain the second acoustic-image spacing parameter. The second feature scale correction value is obtained based on the second adjustment factor. The second reference feature scale factor and the second feature scale correction value are added together to obtain the second feature scale factor. The second sound image spacing parameter and the second feature scale factor have negative feedback with the second round number. The inter-class separation error is calculated based on the second sound image spacing parameter, the second feature scale factor, the state type target value of the second spectrum instance and the inferred state type of the second spectrum instance to obtain the second error.

5. The method according to claim 1, characterized in that, The step of calculating the matching degree between the target time-frequency feature and at least two reference time-frequency features in a preset audio-visual feature library includes: obtaining multiple local feature vectors of the target time-frequency feature; calculating the cosine similarity between each local feature vector of the target time-frequency feature and all local feature vectors of a reference time-frequency feature in the preset audio-visual feature library to obtain multiple similarity values; sorting the multiple similarity values ​​and selecting the top K largest similarity values; and calculating the average of the top K largest similarity values ​​as the final matching degree between the target time-frequency feature and the reference time-frequency feature.

6. The method according to claim 1, characterized in that, After obtaining the second acoustic image feature recognition model, and before extracting time-frequency features from the second spectrum instance using the second acoustic image feature recognition model, the method further includes: calculating the absolute value of the weight parameters of each convolutional layer in the second acoustic image feature recognition model; setting the weight parameters with the smallest absolute value to zero according to a preset pruning rate to obtain a sparse second acoustic image feature recognition model; retraining the sparse second acoustic image feature recognition model to restore the model accuracy lost due to pruning; wherein, extracting time-frequency features from the second spectrum instance using the second acoustic image feature recognition model includes: extracting time-frequency features from the second spectrum instance using the retrained sparse second acoustic image feature recognition model.

7. The method according to claim 1, characterized in that, The method further includes: when the matching degree between the target time-frequency feature and all reference time-frequency features is lower than a preset confidence threshold, determining that the spectrum to be processed belongs to an unknown audio-visual category; receiving the user's manual annotation result for the unknown audio-visual category; and adding the target time-frequency feature, its corresponding spectrum to be processed, and the manually annotated audio-visual identification code as new reference samples to the preset audio-visual feature library.

8. A server system, characterized in that, Includes a server, the server being used to perform the method according to any one of claims 1-7.

Images