Voice quality inspection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice quality inspection method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology and artificial intelligence. The method comprises the steps of performing network layer pruning on a candidate attention network layer according to a network layer weight, performing unit pruning on a candidate attention unit according to a unit weight, and determining a target attention network layer according to a pruned attention network layer and a pruned attention unit; and executing voice category quality inspection on the target multi-modal data by using the target attention network layer. According to the embodiment of the invention, the method comprises the steps: carrying out the network layer pruning of a candidate attention network layer according to the weight of the network layer, and removing the network layers which have the minimum influence on the voice quality inspection effect; the method comprises the following steps: firstly, performing unit pruning on candidate attention units according to unit weights, further removing redundant attention units in a network layer, and finally, performing voice category quality inspection on target multi-modal data by using a target attention network layer, thereby improving the efficiency of voice quality inspection.

Description

Technical Field

[0001] The present application relates to the fields of financial technology and artificial intelligence technology, and in particular to a voice quality inspection method and device, an electronic device and a storage medium. Background Art

[0002] At present, voice quality inspection is usually performed on labeled voice data through neural networks (such as convolutional neural networks (CNN)). For example, in the application scenario of financial technology, when the voice data is the voice of a vehicle accident, the vehicle accident voice is analyzed through a convolutional neural network to identify that the vehicle needs to be assessed for damage, and the degree of damage to the vehicle is associated with its corresponding vehicle damage assessment terms, thereby outputting a voice quality inspection report that requires vehicle claims. However, this method also relies too much on neural networks, resulting in a lot of redundant information being extracted between neural network channels, making the efficiency of voice quality inspection low. Therefore, how to improve the efficiency of voice quality inspection has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a speech quality inspection method and device, an electronic device and a storage medium, aiming to improve the efficiency of speech quality inspection.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a speech quality inspection method, the method comprising:

[0005] Acquire original multimodal data; wherein the original multimodal data includes original image data and original voice data;

[0006] Using the original attention unit of the original attention network layer to extract image features from the original image data, to obtain preliminary image features;

[0007] Performing speech recognition on the original speech data to obtain preliminary text data, and performing text feature extraction on the preliminary text data using the original attention unit to obtain preliminary text features;

[0008] Obtaining a network layer weight of a candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is an original attention network layer with a connection relationship;

[0009] Performing network layer pruning on the candidate attention network layer according to the network layer weight to obtain a pruned attention network layer;

[0010] Obtaining unit weights of candidate attention units in each layer of the candidate attention network layer according to the preliminary text features and the preliminary image features, and performing unit pruning on the candidate attention units according to the unit weights to obtain pruned attention units;

[0011] Determine a target attention network layer according to the pruned attention network layer and the pruned attention unit;

[0012] The target attention network layer is used to perform speech category quality inspection on the target multimodal data to obtain a speech quality inspection category; wherein the target multimodal data includes target speech data and target image data.

[0013] In some embodiments, the performing network layer pruning on the candidate attention network layer according to the network layer weight to obtain a pruned attention network layer includes:

[0014] Select any network layer from the candidate attention network layers as the input attention network layer, and obtain the candidate attention network layer having a connection relationship with the input attention network layer as the output attention network layer;

[0015] Obtain the number of output features of the input attention network layer; Obtain the number of input features of the output attention network layer;

[0016] Obtaining an input activation value of the input attention network layer, and determining layer feature contribution recognition of the input attention network layer according to the input activation value, the number of output features and the network layer weight, to obtain an input network layer contribution;

[0017] Obtaining an output activation value of the output attention network layer, and identifying the layer feature contribution of the output attention network layer according to the output activation value, the number of input features and the network layer weight, to obtain the output network layer contribution;

[0018] The candidate attention network layer is pruned based on the input network layer contribution, the output network layer contribution and the network layer weight to obtain the pruned attention network layer.

[0019] In some embodiments, the performing unit pruning on the candidate attention unit according to the unit weight to obtain the pruned attention unit includes:

[0020] Selecting the unit included in the pruned attention network layer from the candidate attention units as the target candidate attention unit;

[0021] Acquire candidate features of the target candidate attention unit; the candidate features are the preliminary text features or the preliminary image features acquired by the target candidate attention unit;

[0022] Screening the candidate features according to the network layer weights to determine their contribution to obtain target features;

[0023] The candidate attention unit is masked according to the unit weight to obtain the pruned attention unit.

[0024] In some embodiments, performing speech category quality inspection on target multimodal data using the target attention network layer to obtain a speech quality inspection category includes:

[0025] Performing speech recognition on the target speech data to obtain target text data, and performing feature extraction on the target text data using the target attention network layer to obtain target text features;

[0026] Using the target attention network layer to extract features of the target image data to obtain target image features;

[0027] Fusing the target text feature with the target image feature to obtain a target feature;

[0028] Performing a speech category quality check on the target feature to obtain the speech quality check category.

[0029] In some embodiments, after determining the target attention network layer according to the pruned attention network layer and the pruned attention unit, the method includes:

[0030] Obtaining a unit gradient value of the candidate attention unit according to the preliminary text feature and the preliminary image feature;

[0031] Averaging all the unit gradient values ​​to obtain the network layer gradient value of the candidate attention network layer;

[0032] Determine the network gradient variance of the candidate attention network layer according to the unit gradient value and the network layer gradient value;

[0033] Performing gradient network layer pruning on the candidate attention network layer according to the network layer gradient value and the network layer gradient variance to obtain a gradient pruned attention network layer;

[0034] Perform gradient unit pruning on the candidate attention units in the gradient pruning attention network layer according to the unit gradient value and the network layer gradient variance to obtain a gradient pruned attention unit;

[0035] The target attention network layer is updated according to the gradient pruning attention network layer and the gradient pruning attention unit to obtain an updated attention network layer, and the updated attention network layer is used as the target attention network layer.

[0036] In some embodiments, performing gradient network layer pruning on the candidate attention network layer according to the network layer gradient value and the network layer gradient variance to obtain a gradient pruned attention network layer includes:

[0037] Performing a gradient variance importance test on the network layer gradient variance to obtain the network layer gradient variance importance;

[0038] Performing a gradient importance detection on the network layer gradient value to obtain the network layer gradient importance;

[0039] Based on the network layer gradient importance and the network layer gradient importance, the candidate attention network layer is subjected to gradient network layer pruning to obtain the gradient pruned attention network layer.

[0040] In some embodiments, the step of performing gradient unit pruning on the candidate attention unit in the gradient pruning attention network layer according to the unit gradient value and the network layer gradient variance to obtain the gradient pruning attention unit includes:

[0041] Performing a gradient importance test on the unit gradient value to obtain the unit gradient importance;

[0042] Based on the unit gradient importance, the network layer gradient method importance and the network layer gradient importance, gradient unit pruning is performed on the candidate attention units in the gradient pruned attention network layer to obtain the gradient pruned attention unit.

[0043] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech quality inspection device, the device comprising:

[0044] A multimodal data acquisition module, used to acquire original multimodal data; wherein the original multimodal data includes original image data and original voice data;

[0045] An image feature extraction module, used to extract image features from the original image data using the original attention unit of the original attention network layer to obtain preliminary image features;

[0046] A text feature extraction module, configured to perform speech recognition on the original speech data to obtain preliminary text data, and to perform text feature extraction on the preliminary text data using the original attention unit to obtain preliminary text features;

[0047] A network layer weight acquisition module, used to acquire the network layer weight of the candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is an original attention network layer with a connection relationship;

[0048] A network layer pruning module, used for pruning the candidate attention network layer according to the network layer weight to obtain a pruned attention network layer;

[0049] A unit gradient pruning module, used for obtaining the unit weights of the candidate attention units in each layer of the candidate attention network layer according to the preliminary text features and the preliminary image features, and performing unit pruning on the candidate attention units according to the unit weights to obtain pruned attention units;

[0050] A target network layer acquisition module, used to determine a target attention network layer according to the pruned attention network layer and the pruned attention unit;

[0051] A speech quality inspection module is used to perform speech category quality inspection on target multimodal data using the target attention network layer to obtain a speech quality inspection category; wherein the target multimodal data includes target speech data and target image data.

[0052] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0053] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0054] The speech quality inspection method and device, electronic device and storage medium proposed in the present application obtain original multimodal data; wherein the original multimodal data includes original image data and original speech data; the original attention unit of the original attention network layer is used to extract image features from the original image data to obtain preliminary image features; speech recognition is performed on the original speech data to obtain preliminary text data, and the original attention unit is used to extract text features from the preliminary text data to obtain preliminary text features; the network layer weight of the candidate attention network layer is obtained according to the preliminary text features and preliminary image features; wherein the candidate attention network layer is an original attention network layer with a connection relationship; the network layer of the candidate attention network layer is pruned according to the network layer weight to obtain a pruned attention network layer; the unit weight of the candidate attention unit in each layer of the candidate attention network layer is obtained according to the preliminary text features and preliminary image features, and the candidate attention unit is pruned according to the unit weight to obtain a pruned attention unit; the target attention network layer is determined according to the pruned attention network layer and the pruned attention unit; the target multimodal data is subjected to speech category quality inspection using the target attention network layer to obtain a speech quality inspection category; wherein the target multimodal data includes target speech data and target image data. The present application first performs network layer pruning on candidate attention network layers according to network layer weights to obtain pruned attention network layers, which can remove those network layers that have the least impact on speech quality inspection effects, so as to reduce the redundancy and computational burden of the network layer, and facilitate the subsequent improvement of speech quality inspection efficiency; secondly, unit pruning is performed on candidate attention units according to unit weights to obtain pruned attention units, ensuring that only attention units that are most sensitive to preliminary text and preliminary image features are retained, and redundant attention units in the network layer are further removed, thereby improving the speech quality inspection efficiency while still ensuring the network layer's ability to capture key information; finally, the target attention network layer is used to perform speech category quality inspection on the target multimodal data to obtain a speech quality inspection category, thereby realizing speech quality inspection of the target attention network layer that combines network layer pruning and unit pruning, avoiding the extraction of redundant text or image features, and further improving the efficiency of speech quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a voice quality inspection method provided in an embodiment of the present application;

[0056] Figure 2 yes Figure 1 Flow chart of step S105 in FIG.

[0057] Figure 3 yes Figure 1 Flow chart of step S106 in FIG.

[0058] Figure 4 is another flow chart of the voice quality inspection method provided in an embodiment of the present application;

[0059] Figure 5 yes Figure 4 Flow chart of step S404 in FIG.

[0060] Figure 6 yes Figure 1 Flow chart of step S108 in FIG.

[0061] Figure 7 It is a structural schematic diagram of a speech quality inspection device provided in an embodiment of the present application;

[0062] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] First, some nouns involved in this application are analyzed:

[0067] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] Based on this, the embodiments of the present application provide a voice quality inspection method and device, an electronic device and a storage medium, aiming to improve the efficiency of voice quality inspection.

[0069] The speech quality inspection method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speech quality inspection method in the embodiments of the present application is described.

[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0071] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, processing / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0072] The speech quality inspection method provided in the embodiment of the present application relates to the fields of financial technology and artificial intelligence technology. The speech quality inspection method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech quality inspection method, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] Figure 1 is an optional flow chart of the voice quality inspection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S108.

[0075] Step S101, obtaining original multimodal data; wherein the original multimodal data includes original image data and original voice data.

[0076] Step S102, using the original attention unit of the original attention network layer to extract image features from the original image data to obtain preliminary image features.

[0077] Step S103, performing speech recognition on the original speech data to obtain preliminary text data, and using the original attention unit to extract text features from the preliminary text data to obtain preliminary text features.

[0078] Step S104, obtaining the network layer weight of the candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is the original attention network layer with a connection relationship.

[0079] Step S105, pruning the candidate attention network layer according to the network layer weight to obtain a pruned attention network layer.

[0080] Step S106, obtaining the unit weights of the candidate attention units in each candidate attention network layer according to the preliminary text features and the preliminary image features, and performing unit pruning on the candidate attention units according to the unit weights to obtain pruned attention units.

[0081] Step S107, determining the target attention network layer according to the pruned attention network layer and the pruned attention unit.

[0082] Step S108, using the target attention network layer to perform speech category quality inspection on the target multimodal data to obtain a speech quality inspection category; wherein the target multimodal data includes target speech data and target image data.

[0083] Steps S101 to S108 shown in the embodiment of the present application are obtained by obtaining original multimodal data; wherein the original multimodal data includes original image data and original voice data; using the original attention unit of the original attention network layer to extract image features from the original image data to obtain preliminary image features; performing voice recognition on the original voice data to obtain preliminary text data, and using the original attention unit to extract text features from the preliminary text data to obtain preliminary text features; obtaining the network layer weights of the candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is a network layer having a connection The invention relates to an original attention network layer of the relationship; the candidate attention network layer is pruned according to the network layer weights to obtain the pruned attention network layer; the unit weights of the candidate attention units in each layer of the candidate attention network layer are obtained according to the preliminary text features and the preliminary image features, and the candidate attention units are pruned according to the unit weights to obtain the pruned attention units; the target attention network layer is determined according to the pruned attention network layer and the pruned attention units; the target attention network layer is used to perform speech category quality inspection on the target multimodal data to obtain the speech quality inspection category; wherein the target multimodal data includes target speech data and target image data. The present application first performs network layer pruning on candidate attention network layers according to network layer weights to obtain pruned attention network layers, which can remove those network layers that have the least impact on speech quality inspection effects, so as to reduce the redundancy and computational burden of the network layer, and facilitate the subsequent improvement of speech quality inspection efficiency; secondly, unit pruning is performed on candidate attention units according to unit weights to obtain pruned attention units, ensuring that only attention units that are most sensitive to preliminary text and preliminary image features are retained, and redundant attention units in the network layer are further removed, thereby improving the speech quality inspection efficiency while still ensuring the network layer's ability to capture key information; finally, the target attention network layer is used to perform speech category quality inspection on the target multimodal data to obtain a speech quality inspection category, thereby realizing speech quality inspection of the target attention network layer that combines network layer pruning and unit pruning, avoiding the extraction of redundant text or image features, and further improving the efficiency of speech quality inspection.

[0084] In step S101 of some embodiments, specifically, the original multimodal data includes original image data and original voice data.

[0085] For example, in financial and technological application scenarios, when a vehicle insurance claim is required, the original voice data is the vehicle accident voice data, and the original image data is the vehicle damage photos at the vehicle accident scene and the video frame images of the vehicle accident.

[0086] Specifically, the original voice data can be obtained from a recording of a report phone call or a recording of a vehicle accident scene, and the original image data can be obtained from photos taken at the vehicle accident scene and the accident video recorded by a driving recorder of the accident vehicle.

[0087] In step S102 of some embodiments, the original attention network layer is a Multiway Transformer model, which includes a visual network layer and a language network layer, and each network layer contains at least two original attention units, which can be an Attention Head.

[0088] Specifically, the visual network layer is used to extract image features from the original image data, and the language network layer is used to extract semantic information from the preliminary text data in the original speech data.

[0089] Specifically, the original image data can be convolved through the visual network layer to obtain a feature map, and the Attention Head can be used to perform attention weight (i.e., query Q-key K-value V) operation on the feature map to obtain preliminary image features represented by a high-dimensional vector.

[0090] In step S103 of some embodiments, specifically, the original voice data is first preprocessed by automatic speech recognition technology (including removing background noise, echo cancellation, etc.) to obtain a preprocessed voice signal to ensure the clarity of the original voice data; secondly, the preprocessed voice signal is converted into a series of digital features, which can reflect the acoustic characteristics of the original voice data (such as pitch, sound intensity, sound length, etc.); finally, the possible pronunciation unit sequence is predicted based on the acoustic characteristics, and the pronunciation unit sequence is combined into meaningful words and sentences in combination with grammatical rules and context information to generate preliminary text data.

[0091] For example, in financial and technological scenarios, when a car owner encounters a car accident, he or she will usually call the insurance company's reporting phone number immediately to file a claim, and the car owner will describe in detail the specific circumstances of the car accident on the phone, including the time, location, cause of the accident, vehicle damage, and possible casualties. This information can be recognized as text data through automatic speech recognition technology.

[0092] Specifically, the preliminary text data can be processed by word embedding through the language network layer to obtain preliminary text data represented by vectors, and the attention weights (i.e., query Q-key K-value V) between words can be calculated using the Attention Head on the preliminary text data represented by the vectors. The word representation is updated according to the attention weights to finally obtain preliminary text features.

[0093] In step S104 of some embodiments, specifically, the candidate attention network layers may be attention network layers with directed connection relationships, that is, the directed connection relationship is an input and output relationship between the candidate attention network layers.

[0094] For example, in the financial field, if the Lth candidate attention network layer is the input attention network layer, and if the Rth candidate attention network layer is the preliminary text features or preliminary image features received from the output of the Lth candidate attention network layer, then the Rth candidate attention network layer is the output attention network layer.

[0095] Specifically, the network layer weight refers to the weight value from the input attention network layer to the output attention network layer, which indicates the importance of the preliminary text features or preliminary image features between the candidate attention network layers to the speech quality inspection. The larger the network layer weight value, the greater the influence of the candidate attention network layer on the speech quality inspection results.

[0096] Specifically, the network layer weights can be determined through expert experience or directly obtained from the network layer parameters.

[0097] See also Figure 2 In some embodiments, step S105 includes but is not limited to steps S201 to S205:

[0098] Step S201, select any network layer from the candidate attention network layers as the input attention network layer, and obtain the candidate attention network layer that has a connection relationship with the input attention network layer as the output attention network layer.

[0099] Step S202, obtaining the number of output features of the input attention network layer; obtaining the number of input features of the output attention network layer.

[0100] Step S203, obtaining the input activation value of the input attention network layer, and determining the layer feature contribution of the input attention network layer according to the input activation value, the number of output features and the network layer weight, to obtain the input network layer contribution.

[0101] Step S204, obtaining the output activation value of the output attention network layer, and identifying the layer feature contribution of the output attention network layer according to the output activation value, the number of input features and the network layer weight, to obtain the output network layer contribution.

[0102] Step S205, based on the input network layer contribution, the output network layer contribution and the network layer weight, the candidate attention network layer is pruned to obtain a pruned attention network layer.

[0103] In step S201 of some embodiments, specifically, the Lth layer can be selected from the candidate attention network layers as the input attention network layer, and the Rth candidate attention network layer connected to the Lth layer can be selected as the output attention network layer.

[0104] In step S202 of some embodiments, specifically, the number of output features represents the number of preliminary text features or the number of preliminary image features in the Rth candidate attention network layer, that is, the sum of all features connected to the output preliminary text features or preliminary image features of the Lth candidate attention network layer.

[0105] Specifically, the number of input features represents the number of preliminary text features or preliminary image features in the Lth candidate attention network layer, that is, the sum of all features connected to the preliminary text features or preliminary image features input to the Rth candidate attention network layer.

[0106] In step S203 of some embodiments, specifically, the input network layer contribution refers to the importance of the input attention network layer in the entire network layer represented by a quantified network layer indicator, that is, the influence of all preliminary text features or preliminary image features in the input attention network layer on the speech quality inspection results.

[0107] Specifically, the contribution of the input network layer can be determined by the following formula:

[0108]

[0109] Among them, S(L) represents the input network layer contribution of the Lth input attention network layer, R represents the number of output features, and a L represents the input activation value vector of the Lth input attention network layer, ||a L ||2 represents the bi-norm of the input activation value vector, that is, the input activation value, θ LR Represents the network layer weights from the Lth input attention network layer to the Rth output attention network layer.

[0110] In step S204 of some embodiments, specifically, the output network layer contribution refers to the importance of the output attention network layer in the entire network layer represented by a quantified network layer indicator, that is, the influence of all preliminary text features or preliminary image features in the output attention network layer on the speech quality inspection results.

[0111] Specifically, the contribution of the output network layer can be determined by the following formula:

[0112]

[0113] Among them, S(R) represents the output network layer contribution of the R-th output attention network layer, L represents the number of input features, and a R represents the input activation value vector of the Rth output attention network layer, ||a R ||2 represents the bi-norm of the output activation value vector, that is, the output activation value, θ LRRepresents the network layer weights from the Lth input attention network layer to the Rth output attention network layer.

[0114] In this embodiment, the layer feature contribution of the output attention network layer is identified according to the output activation value, the number of input features and the network layer weight, which can quantify the contribution of each output attention network layer to the target network layer, and help to intuitively extract whether the features extracted by different output attention network layers are key features, thereby facilitating the improvement of subsequent speech quality inspection efficiency.

[0115] In step S205 of some embodiments, specifically, by determining the network layer contribution by the output network layer contribution and the input network layer contribution, and multiplying the network layer contribution by the network layer weight, the importance of the features from the Lth input attention network layer to the Rth output attention network layer to the speech quality inspection result can be determined.

[0116] For example, in the financial field, when making a vehicle insurance claim, it is necessary to calculate the claim amount. If the Lth input attention network layer identifies the vehicle cracks, dents and vehicle model characteristics, the input network layer contribution is 0.8, and the Rth output attention network layer receives the vehicle cracks, dents and vehicle model characteristics, and the output network layer contribution is 0.85, then the degree of damage to the vehicle is determined to be 0.825. Combined with the network layer weight of 0.7 from the L layer to the R layer, it can be determined that the importance of the characteristics of the Lth candidate attention network layer in affecting the calculation of the claim amount is 0.58. Since 0.58 is greater than the preset network layer contribution of 0.5, there is no need to prune the L layer; if the The vehicle features of the L input attention network layers are vehicle model, license plate, vehicle logo, etc., and the obtained input network layer contribution is 0.5. The Rth output attention network layer receives the vehicle model, license plate and vehicle logo features, and the obtained output network layer contribution is 0.5. The vehicle damage degree is determined to be 0.5. Combined with the network layer weight of 0.7 from the L layer to the R layer, it can be determined that the importance of the features of the Lth candidate attention network layer in affecting the calculation of the claim amount is 0.35. Since 0.35 is less than the preset network layer contribution of 0.5, it means that the vehicle model, license plate and vehicle logo of the L layer have little effect on the claim amount, so the Lth input attention network layer needs to be pruned to avoid irrelevant redundant features occupying the network computing memory.

[0117] In this embodiment, network layer pruning is performed on candidate attention network layers based on the input network layer contribution, the output network layer contribution and the network layer weight, so as to remove those network layers that have the least impact on the speech quality inspection effect, thereby reducing the redundancy and computational burden of the network layer and facilitating the subsequent improvement of speech quality inspection efficiency.

[0118] In step S106 of some embodiments, the unit weights are automatically learned through the training and optimization process of the network layer, and these weights reflect the importance that the network layer attaches to the output information of different candidate attention units when processing different speech quality inspection tasks.

[0119] See also Figure 3 In some embodiments, step S106 includes but is not limited to steps S301 to S304:

[0120] Step S301, select the unit included in the pruned attention network layer from the candidate attention units as the target candidate attention unit.

[0121] Step S302, obtaining candidate features of the target candidate attention unit; the candidate features are preliminary text features or preliminary image features obtained by the target candidate attention unit.

[0122] Step S303, screening the candidate features according to their contribution levels according to the network layer weights to obtain target features.

[0123] Step S304, mask the candidate attention units according to the unit weights to obtain pruned attention units.

[0124] In step S301 of some embodiments, specifically, the target candidate attention unit is a unit included in the pruned attention network layer.

[0125] For example, the candidate attention network layers include 1, 2, 3, and 4. After pruning, the attention network layers are 1, 2, and 4. When pruning the third candidate attention network layer, the attention units in the attention layer are also pruned, so the remaining attention units are the units contained in the pruned attention layer.

[0126] In step S302 of some embodiments, specifically, the candidate feature is a preliminary text feature or a preliminary image feature obtained by the target candidate attention unit.

[0127] For example, in the financial field, when making a vehicle insurance claim, preliminary text features (such as the cause of the vehicle accident, tire damage, paint peeling features, etc.) obtained through multiple Attention Heads (i.e., target candidate attention units) are used as candidate features, and preliminary image features (such as car body crack feature maps, bumper dent feature maps, and tire damage feature maps, etc.) are obtained through different Attention Heads as candidate features.

[0128] In step S303 of some embodiments, the pruned attention network layer is firstly obtained to extract preliminary text features or preliminary image features, and the feature weights of the features in each network layer are obtained. The feature weights are compared with the network layer weights, and the features that are greater than the network layer weights are sorted from large to small according to the comparison results, and the top k most important features are selected as target features based on the sorting; wherein the feature weights are determined based on the layer parameter weights in the pruned attention network layer.

[0129] In this embodiment, by screening the degree of feature contribution of candidate features according to the network layer weights, the distribution of key parameters of the pruned attention network layer when extracting features can be reflected, which helps to retain these important parameter information in the subsequent unit pruning process.

[0130] In step S304 of some embodiments, specifically, by comparing the unit weight with the target feature weight, if the target feature weight is greater than or equal to the unit weight, it means that the attention unit parameter of the target feature is also a key parameter and is retained during the masking process; if the target feature weight is less than the unit weight, it means that the attention unit parameter of the target feature is not a key parameter and the target feature is masked to obtain a pruned attention unit.

[0131] In this embodiment, the candidate attention units are masked according to the unit weights to obtain pruned attention units, which can ensure that only the attention units that are most sensitive to the preliminary text and preliminary image features are retained, and the redundant attention units in the network layer are further removed, thereby improving the efficiency of speech quality inspection while still ensuring the network layer's ability to capture key information.

[0132] In step S107 of some embodiments, specifically, by fusing the parameters of the pruned attention units and the pruned attention network layer, a target attention network layer can be generated.

[0133] Specifically, after generating the target attention network layer, the target attention network layer can also be fine-tuned or verified to ensure that the speech quality inspection performance of the generated target attention network layer can achieve the expected effect.

[0134] See also Figure 4 In some embodiments, after step S107, the voice quality inspection method may include but is not limited to steps S401 to S406:

[0135] Step S401, obtaining the unit gradient value of the candidate attention unit according to the preliminary text features and the preliminary image features.

[0136] Step S402, averaging all unit gradient values ​​to obtain the network layer gradient value of the candidate attention network layer;

[0137] Step S403, determining the network gradient variance of the candidate attention network layer according to the unit gradient value and the network layer gradient value.

[0138] Step S404, performing gradient network layer pruning on the candidate attention network layer according to the network layer gradient value and the network layer gradient variance to obtain a gradient pruned attention network layer.

[0139] Step S405, performing gradient unit pruning on the candidate attention units in the gradient pruned attention network layer according to the unit gradient value and the network layer gradient variance to obtain a gradient pruned attention unit.

[0140] Step S406, updating the target attention network layer according to the gradient pruned attention network layer and the gradient pruned attention unit to obtain an updated attention network layer, and using the updated attention network layer as the target attention network layer.

[0141] In step S401 of some embodiments, specifically, the gradient is the partial derivative of the loss function with respect to the attention unit parameters, which indicates the direction in which the loss function increases fastest in the parameter space. The learning rate determines the magnitude of the update of the attention unit parameters. Combining the gradient with the learning rate can represent the actual amount of change in the attention unit parameters at the current learning rate.

[0142] Specifically, the unit gradient value can be represented by the average value of the ratio of Δy to Δx, where Δx is used to represent the learning rate, which is used to control the step size of the attention unit parameter update, and Δy is used to represent the output change caused by the update step size. The unit gradient value is used to indicate how the attention unit parameters should be adjusted to minimize the loss at the current learning rate.

[0143] For example, in the financial field, when making a vehicle insurance claim, the network layer calculates the loss by comparing the actual claim amount with the predicted claim amount. The unit gradient value represents the gradient of the loss and is used to indicate how to adjust the attention unit parameters to reduce the prediction error.

[0144] In step S402 of some embodiments, specifically, the network layer gradient value can be represented by the average value of the unit gradient values ​​of all the Attention Heads it contains, and the network layer gradient value is used to indicate the parameter adjustment of the candidate attention network layer.

[0145] In step S403 of some embodiments, specifically, for the unit gradient value of each attention head, the network gradient variance is obtained by calculating the sum of the squares of the differences between the unit gradient value and the network layer gradient mean of the entire attention network layer, and dividing the sum by the number of attention heads in the attention network layer.

[0146] Specifically, the network gradient variance can be used to measure the discreteness of the unit gradient values ​​of the attention heads in the attention network layer. The larger the network gradient variance, the greater the difference in the unit gradient values ​​of different attention heads. Conversely, the smaller the difference in the unit gradient values ​​of different attention heads.

[0147] See also Figure 5 In some embodiments, step S404 includes but is not limited to steps S501 to S503:

[0148] Step S501, performing a gradient variance importance test on the network layer gradient variance to obtain the network layer gradient variance importance.

[0149] Step S502: Perform a gradient importance test on the network layer gradient value to obtain the network layer gradient importance.

[0150] Step S503, based on the network layer gradient importance and the network layer gradient importance, the candidate attention network layer is gradient pruned to obtain a gradient pruned attention network layer.

[0151] In step S501 of some embodiments, specifically, the gradient variance importance detection of the network layer gradient variance is achieved by comparing the network layer gradient variance of the selected candidate attention network layer with a preset network layer gradient variance threshold.

[0152] Specifically, the preset network layer gradient variance threshold can systematically find the optimal network layer gradient variance threshold based on hyperparameter search technology (such as grid search, random search, Bayesian optimization, etc.).

[0153] In step S502 of some embodiments, specifically, the gradient importance detection of the network layer gradient value is achieved by comparing the network layer gradient value of the selected candidate attention network layer with a preset network layer gradient threshold.

[0154] Specifically, the preset network layer gradient threshold is also based on hyperparameter search technology (such as grid search, random search, Bayesian optimization, etc.) to systematically find the optimal network layer gradient threshold.

[0155] In step S503 of some embodiments, specifically, the pruning evaluation basis of the gradient network layer is that if the network layer gradient variance of the selected candidate attention network layer is less than a preset network layer gradient variance threshold, and the network layer gradient value of the candidate attention network layer is also less than the preset network layer gradient threshold, it means that the contribution of the candidate attention network layer to the target network layer is low, and the gradient pruned attention network layer is obtained by removing the candidate attention network layer and all network connections of the layer.

[0156] For example, in the financial field, when conducting vehicle claims assessment, if the image features extracted by one of the candidate attention network layers are non-vehicle loss parts in the image (such as pedestrians, license plates, etc.), and the contribution of this candidate attention network layer to the target network layer is low, then the candidate attention network layer and all network connections of this layer are removed.

[0157] In this embodiment, gradient network layer pruning is performed on the candidate attention network layer according to the network layer gradient value and the network layer gradient variance, which can remove those layers that contribute less to the gradient of the final output of the target network layer, significantly reduce the computational complexity of the target network layer, and improve the computational efficiency of the target network layer.

[0158] In step S405 of some embodiments, specifically, gradient unit pruning is performed on candidate attention units in the gradient pruned attention network layer according to unit gradient values ​​and network layer gradient variances to obtain gradient pruned attention units, including: performing gradient importance detection on unit gradient values ​​to obtain unit gradient importance; gradient unit pruning is performed on candidate attention units in the gradient pruned attention network layer based on unit gradient importance, network layer gradient method importance and network layer gradient importance to obtain gradient pruned attention units.

[0159] Specifically, the gradient importance detection of the unit gradient value is achieved by comparing the unit gradient values ​​of all candidate attention units contained in the selected candidate attention network layer with the network layer gradient value of the candidate attention network layer.

[0160] Specifically, the pruning evaluation basis of the gradient unit is that if the network layer gradient variance of the selected candidate attention network layer is greater than the preset network layer gradient variance threshold, the network layer gradient value of the candidate attention network layer is less than the preset network layer gradient threshold, and the unit gradient value is less than the network layer gradient value, it means that the contribution of the candidate attention unit to the target network layer is low. By removing the unit parameters of the candidate attention unit, a gradient pruned attention unit is obtained.

[0161] For example, in the financial field, when conducting vehicle claims assessment, if an Attention Head extracts preliminary text features that are irrelevant to the degree of vehicle loss (such as repeated vehicle loss description information, the structure of the vehicle wheel hub, etc.), and the contribution of the Attention Head to the target network layer is low, all parameters of the Attention Head are removed.

[0162] In this embodiment, by performing gradient unit pruning on the candidate attention units in the gradient pruning attention network layer according to the unit gradient value and the network layer gradient variance, the redundant features of the candidate attention units can be further identified based on the size and direction of the gradient, and the redundant features of the attention unit can be removed. In addition, unit gradient pruning can also help the target network layer to focus more on the feature parts related to speech quality inspection, further improving the computational efficiency of the target network layer.

[0163] In step S406 of some embodiments, specifically, all parameters of the gradient pruned attention network layer and the gradient pruned attention unit are fused with all parameters of the target attention network layer to obtain a fused attention network layer, and the fused attention network layer is used as the target attention network layer.

[0164] In this embodiment, by combining the gradient pruning of the attention network layer and the gradient pruning of the attention unit, the direction and size of the gradient can be combined to further identify the parameters that have the least impact on the output of the target network layer, and these parameters can be removed, thereby retaining the important characteristics of the target network layer. While ensuring the accuracy of the target network layer in speech quality inspection, the redundant parameters of the target network layer can be further reduced, thereby improving the computational efficiency of the target network layer.

[0165] See also Figure 6 In some embodiments, step S108 includes but is not limited to steps S601 to S604:

[0166] Step S601, perform speech recognition on the target speech data to obtain target text data, and use the target attention network layer to perform feature extraction on the target text data to obtain target text features.

[0167] Step S602, using the target attention network layer to extract features of the target image data to obtain target image features.

[0168] Step S603: Fusing the target text feature with the target image feature to obtain the target feature.

[0169] Step S604: Perform voice category quality inspection on the target feature to obtain a voice quality inspection category.

[0170] In step S601 of some embodiments, the target speech data refers to speech data to be subjected to speech quality inspection, and the target text data refers to semantic content information contained in the target speech data described in text.

[0171] For example, in the financial field, if the target voice is the car owner's voice reporting a vehicle accident, then the target text is the car owner's description of the specific circumstances of the vehicle accident, including time, location, cause of the accident, vehicle damage, and possible casualties. This information can be recognized as text data through automatic speech recognition technology.

[0172] Furthermore, the method of performing speech recognition on the target speech data to obtain the target text data is consistent with the method of performing speech recognition on the original speech data to obtain the preliminary text data, which will not be described in detail here.

[0173] Specifically, the target text data is feature extracted through the target attention unit in the target attention network layer. This feature extraction method is consistent with the method of extracting text features from preliminary text data using the original attention unit, and will not be repeated here.

[0174] In step S602 of some embodiments, specifically, the target image data refers to image data to be subjected to voice quality inspection.

[0175] For example, in the financial field, if the target image data is the recorded video of the driving recorder of the accident vehicle or the photos of the vehicle accident taken by the car owner, the vehicle loss features (such as damaged tires, paint peeling, bumper dents, etc.) are extracted from the accident video recorded by the driving recorder and the photos of the vehicle accident taken by the car owner to evaluate the vehicle loss and determine the vehicle claim amount.

[0176] Specifically, the method of using the target attention unit in the target attention network layer to extract features of the target image data is consistent with the method of using the original attention unit in the original attention network layer to extract image features of the original image data, which will not be repeated here.

[0177] In step S603 of some embodiments, specifically, the target feature refers to the result of concatenating the target text feature and the target image feature, and the target feature can fully represent all features of the target text feature and the target image feature.

[0178] In step S604 of some embodiments, specifically, the speech quality inspection category refers to the speech content category when the speech quality inspection is performed.

[0179] For example, in the financial field, voice quality inspection refers to the need to perform category analysis on different voice data in order to determine whether the voice quality inspection task category is insurance product recommendation or insurance claim assessment; if the target voice is the owner's vehicle accident report voice, the vehicle loss is assessed based on the report voice and the vehicle accident video to determine the vehicle claim amount.

[0180] In this embodiment, by performing speech category quality inspection on the target features, speech quality inspection of the target attention network layer combined with network layer pruning and unit pruning is achieved, thereby avoiding the extraction of redundant text or image features and improving the efficiency of speech quality inspection.

[0181] The embodiment of the present application obtains original multimodal data; wherein the original multimodal data includes original image data and original voice data; uses the original attention unit of the original attention network layer to extract image features from the original image data to obtain preliminary image features; performs voice recognition on the original voice data to obtain preliminary text data, and uses the original attention unit to extract text features from the preliminary text data to obtain preliminary text features; obtains the network layer weight of the candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is an original attention network layer with a connection relationship; performs network layer pruning on the candidate attention network layer according to the network layer weight to obtain a pruned attention network layer; obtains the unit weight of the candidate attention unit in each layer of the candidate attention network layer according to the preliminary text features and the preliminary image features, and performs unit pruning on the candidate attention unit according to the unit weight to obtain a pruned attention unit; determines the target attention network layer according to the pruned attention network layer and the pruned attention unit; performs voice category quality inspection on the target multimodal data using the target attention network layer to obtain a voice quality inspection category; wherein the target multimodal data includes target voice data and target image data. The present application first performs network layer pruning on candidate attention network layers according to network layer weights to obtain pruned attention network layers, which can remove those network layers that have the least impact on speech quality inspection effects, so as to reduce the redundancy and computational burden of the network layer, and facilitate the subsequent improvement of speech quality inspection efficiency; secondly, unit pruning is performed on candidate attention units according to unit weights to obtain pruned attention units, ensuring that only attention units that are most sensitive to preliminary text and preliminary image features are retained, and redundant attention units in the network layer are further removed, thereby improving the speech quality inspection efficiency while still ensuring the network layer's ability to capture key information; finally, the target attention network layer is used to perform speech category quality inspection on the target multimodal data to obtain a speech quality inspection category, thereby realizing speech quality inspection of the target attention network layer that combines network layer pruning and unit pruning, avoiding the extraction of redundant text or image features, and further improving the efficiency of speech quality inspection.

[0182] See also Figure 7 The embodiment of the present application also provides a speech quality inspection device, which can implement the above-mentioned speech quality inspection method, and the device includes:

[0183] A multimodal data acquisition module, used to acquire original multimodal data; wherein the original multimodal data includes original image data and original voice data;

[0184] An image feature extraction module, used to extract image features from original image data using the original attention unit of the original attention network layer to obtain preliminary image features;

[0185] A text feature extraction module is used to perform speech recognition on the original speech data to obtain preliminary text data, and to extract text features from the preliminary text data using the original attention unit to obtain preliminary text features;

[0186] A network layer weight acquisition module, used to acquire the network layer weight of the candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is the original attention network layer with a connection relationship;

[0187] A network layer pruning module is used to prune the candidate attention network layer according to the network layer weight to obtain a pruned attention network layer;

[0188] A unit gradient pruning module is used to obtain the unit weights of the candidate attention units in each candidate attention network layer according to the preliminary text features and the preliminary image features, and prune the candidate attention units according to the unit weights to obtain pruned attention units;

[0189] A target network layer acquisition module is used to determine a target attention network layer according to the pruned attention network layer and the pruned attention unit;

[0190] The speech quality inspection module is used to perform speech category quality inspection on the target multimodal data using the target attention network layer to obtain the speech quality inspection category; wherein the target multimodal data includes target speech data and target image data.

[0191] The specific implementation of the speech quality inspection device is basically the same as the specific implementation of the above-mentioned speech quality inspection method, and will not be repeated here.

[0192] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned voice quality inspection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0193] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0194] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0195] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the processing system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the speech quality inspection method of the embodiment of this application;

[0196] Input / output interface 803, used to implement information input and output;

[0197] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0198] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0199] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0200] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned voice quality inspection method is implemented.

[0201] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0202] The speech quality inspection method, speech quality inspection device, electronic device and storage medium provided in the embodiments of the present application obtain original multimodal data; wherein the original multimodal data includes original image data and original speech data; the original attention unit of the original attention network layer is used to extract image features from the original image data to obtain preliminary image features; speech recognition is performed on the original speech data to obtain preliminary text data, and the original attention unit is used to extract text features from the preliminary text data to obtain preliminary text features; the network layer weights of the candidate attention network layer are obtained according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer It is an original attention network layer with a connection relationship; the candidate attention network layer is pruned according to the network layer weight to obtain a pruned attention network layer; the unit weights of the candidate attention units in each layer of the candidate attention network layer are obtained according to the preliminary text features and the preliminary image features, and the candidate attention units are pruned according to the unit weights to obtain pruned attention units; the target attention network layer is determined according to the pruned attention network layer and the pruned attention units; the target attention network layer is used to perform speech category quality inspection on the target multimodal data to obtain a speech quality inspection category; wherein the target multimodal data includes target speech data and target image data. The present application first performs network layer pruning on candidate attention network layers according to network layer weights to obtain pruned attention network layers, which can remove those network layers that have the least impact on speech quality inspection effects, so as to reduce the redundancy and computational burden of the network layer, and facilitate the subsequent improvement of speech quality inspection efficiency; secondly, unit pruning is performed on candidate attention units according to unit weights to obtain pruned attention units, ensuring that only attention units that are most sensitive to preliminary text and preliminary image features are retained, and redundant attention units in the network layer are further removed, thereby improving the speech quality inspection efficiency while still ensuring the network layer's ability to capture key information; finally, the target attention network layer is used to perform speech category quality inspection on the target multimodal data to obtain a speech quality inspection category, thereby realizing speech quality inspection of the target attention network layer that combines network layer pruning and unit pruning, avoiding the extraction of redundant text or image features, and further improving the efficiency of speech quality inspection.

[0203] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0204] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0205] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0207] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0208] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-OnLy Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0213] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech quality inspection method, characterized in that: The method comprises: Acquire original multimodal data; wherein the original multimodal data includes original image data and original voice data; Using the original attention unit of the original attention network layer to extract image features from the original image data, to obtain preliminary image features; Performing speech recognition on the original speech data to obtain preliminary text data, and performing text feature extraction on the preliminary text data using the original attention unit to obtain preliminary text features; Obtaining a network layer weight of a candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is an original attention network layer with a connection relationship; Performing network layer pruning on the candidate attention network layer according to the network layer weight to obtain a pruned attention network layer; Obtaining unit weights of candidate attention units in each layer of the candidate attention network layer according to the preliminary text features and the preliminary image features, and performing unit pruning on the candidate attention units according to the unit weights to obtain pruned attention units; Determine a target attention network layer according to the pruned attention network layer and the pruned attention unit; The target attention network layer is used to perform speech category quality inspection on the target multimodal data to obtain a speech quality inspection category; wherein the target multimodal data includes target speech data and target image data.

2. The method according to claim 1, characterized in that The performing network layer pruning on the candidate attention network layer according to the network layer weight to obtain a pruned attention network layer includes: Select any network layer from the candidate attention network layers as the input attention network layer, and obtain the candidate attention network layer having a connection relationship with the input attention network layer as the output attention network layer; Obtain the number of output features of the input attention network layer; Obtain the number of input features of the output attention network layer; Obtaining an input activation value of the input attention network layer, and determining layer feature contribution recognition of the input attention network layer according to the input activation value, the number of output features and the network layer weight, to obtain an input network layer contribution; Obtaining an output activation value of the output attention network layer, and identifying the layer feature contribution of the output attention network layer according to the output activation value, the number of input features and the network layer weight, to obtain the output network layer contribution; The candidate attention network layer is pruned based on the input network layer contribution, the output network layer contribution and the network layer weight to obtain the pruned attention network layer.

3. The method according to claim 1, characterized in that The performing unit pruning on the candidate attention unit according to the unit weight to obtain a pruned attention unit includes: Selecting the unit included in the pruned attention network layer from the candidate attention units as the target candidate attention unit; Acquire candidate features of the target candidate attention unit; the candidate features are the preliminary text features or the preliminary image features acquired by the target candidate attention unit; Screening the candidate features according to the network layer weights to determine their contribution to obtain target features; The candidate attention unit is masked according to the unit weight to obtain the pruned attention unit.

4. The method according to any one of claims 1 to 3, characterized in that The method of using the target attention network layer to perform speech category quality inspection on the target multimodal data to obtain the speech quality inspection category includes: Performing speech recognition on the target speech data to obtain target text data, and performing feature extraction on the target text data using the target attention network layer to obtain target text features; Using the target attention network layer to extract features of the target image data to obtain target image features; Fusing the target text feature with the target image feature to obtain a target feature; Performing a speech category quality check on the target feature to obtain the speech quality check category.

5. The method according to claim 1, characterized in that After determining the target attention network layer according to the pruned attention network layer and the pruned attention unit, the method includes: Obtaining a unit gradient value of the candidate attention unit according to the preliminary text feature and the preliminary image feature; Averaging all the unit gradient values ​​to obtain the network layer gradient value of the candidate attention network layer; Determine the network gradient variance of the candidate attention network layer according to the unit gradient value and the network layer gradient value; Performing gradient network layer pruning on the candidate attention network layer according to the network layer gradient value and the network layer gradient variance to obtain a gradient pruned attention network layer; Perform gradient unit pruning on the candidate attention units in the gradient pruning attention network layer according to the unit gradient value and the network layer gradient variance to obtain a gradient pruned attention unit; The target attention network layer is updated according to the gradient pruning attention network layer and the gradient pruning attention unit to obtain an updated attention network layer, and the updated attention network layer is used as the target attention network layer.

6. The method according to claim 5, characterized in that The step of performing gradient network layer pruning on the candidate attention network layer according to the network layer gradient value and the network layer gradient variance to obtain a gradient pruned attention network layer comprises: Performing a gradient variance importance test on the network layer gradient variance to obtain the network layer gradient variance importance; Performing a gradient importance detection on the network layer gradient value to obtain the network layer gradient importance; Based on the network layer gradient importance and the network layer gradient importance, the candidate attention network layer is subjected to gradient network layer pruning to obtain the gradient pruned attention network layer.

7. The method according to claim 6, characterized in that The step of performing gradient unit pruning on the candidate attention unit in the gradient pruning attention network layer according to the unit gradient value and the network layer gradient variance to obtain a gradient pruning attention unit includes: Performing a gradient importance test on the unit gradient value to obtain the unit gradient importance; Based on the unit gradient importance, the network layer gradient method importance and the network layer gradient importance, gradient unit pruning is performed on the candidate attention units in the gradient pruned attention network layer to obtain the gradient pruned attention unit.

8. A speech quality inspection device, characterized in that: The device comprises: A multimodal data acquisition module, used to acquire original multimodal data; wherein the original multimodal data includes original image data and original voice data; An image feature extraction module, used to extract image features from the original image data using the original attention unit of the original attention network layer to obtain preliminary image features; A text feature extraction module, configured to perform speech recognition on the original speech data to obtain preliminary text data, and to perform text feature extraction on the preliminary text data using the original attention unit to obtain preliminary text features; A network layer weight acquisition module, used to acquire the network layer weight of the candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is an original attention network layer with a connection relationship; A network layer pruning module, used for pruning the candidate attention network layer according to the network layer weight to obtain a pruned attention network layer; A unit gradient pruning module, used for obtaining the unit weights of the candidate attention units in each layer of the candidate attention network layer according to the preliminary text features and the preliminary image features, and performing unit pruning on the candidate attention units according to the unit weights to obtain pruned attention units; A target network layer acquisition module, used to determine a target attention network layer according to the pruned attention network layer and the pruned attention unit; A speech quality inspection module is used to perform speech category quality inspection on target multimodal data using the target attention network layer to obtain a speech quality inspection category; wherein the target multimodal data includes target speech data and target image data.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the speech quality inspection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech quality inspection method according to any one of claims 1 to 7 is implemented.

Images