Voice quality inspection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a voice quality inspection method and device, electronic equipment and storage medium, belonging to the field of financial technology and artificial intelligence. The method comprises: performing network layer pruning on a candidate attention network layer according to a network layer weight, and performing unit pruning on a candidate attention unit according to a unit weight, determining a target attention network layer according to the pruned attention network layer and the pruned attention unit; and performing voice category quality inspection on target multi-modal data by using the target attention network layer. The embodiments of the present application first perform network layer pruning on the candidate attention network layer according to the network layer weight, which can remove the network layers that have the least influence on the voice quality inspection effect; secondly, the unit pruning is performed on the candidate attention unit according to the unit weight, which further removes the redundant attention units in the network layer; finally, the voice category quality inspection is performed on the target multi-modal data by using the target attention network layer, thereby improving the efficiency of the voice quality inspection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial technology and artificial intelligence, and particularly relates to a voice quality inspection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] At present, voice quality inspection is usually performed on labeled voice data through a neural network (such as a convolutional neural network (CNN) or the like). For example, in a financial technology application scenario, when the voice data is vehicle accident voice, the vehicle accident voice is analyzed through a convolutional neural network, it is identified that the vehicle needs to be appraised, and the vehicle damage degree is associated with the vehicle appraisal clause corresponding thereto, thereby outputting a voice quality inspection report that needs to be handled for vehicle claim. However, this method still excessively relies on the neural network, and a lot of redundant information is extracted between neural network channels, so that the efficiency of voice quality inspection is low. Therefore, how to improve the efficiency of voice quality inspection has become a problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a voice quality inspection method and device, electronic equipment and a storage medium, which aims to improve the efficiency of voice quality inspection.

[0004] To achieve the above purpose, a first aspect of the embodiments of the present application provides a voice quality inspection method, which comprises:

[0005] obtaining original multi-modal data; wherein the original multi-modal data comprises original image data and original voice data;

[0006] performing image feature extraction on the original image data by using an original attention unit of an original attention network layer, to obtain preliminary image features;

[0007] performing voice recognition on the original voice data to obtain preliminary text data, and performing text feature extraction on the preliminary text data by using the original attention unit, to obtain preliminary text features;

[0008] obtaining network layer weights of a candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is an original attention network layer having a connection relationship;

[0009] performing network layer pruning on the candidate attention network layer according to the network layer weights, to obtain a pruned attention network layer;

[0010] obtaining unit weights of a candidate attention unit in each layer of the candidate attention network layer according to the preliminary text features and the preliminary image features, and performing unit pruning on the candidate attention unit according to the unit weights, to obtain a pruned attention unit;

[0011] determining a target attention network layer according to the pruned attention network layer and the pruned attention unit;

[0012] performing speech category quality inspection on target multi-modal data by using the target attention network layer to obtain a speech quality inspection category; wherein the target multi-modal data comprises target speech data and target image data.

[0013] In some embodiments, the network layer pruning of the candidate attention network layer according to the network layer weight to obtain the pruned attention network layer comprises:

[0014] selecting any network layer from the candidate attention network layer as an input attention network layer, and obtaining a candidate attention network layer having a connection relationship with the input attention network layer as an output attention network layer;

[0015] obtaining the number of output features of the input attention network layer; and obtaining the number of input features of the output attention network layer;

[0016] obtaining the input activation value of the input attention network layer, and determining the layer feature contribution degree identification of the input attention network layer according to the input activation value, the number of output features and the network layer weight to obtain an input network layer contribution degree;

[0017] obtaining the output activation value of the output attention network layer, and performing layer feature contribution degree identification on the output attention network layer according to the output activation value, the number of input features and the network layer weight to obtain an output network layer contribution degree;

[0018] pruning the candidate attention network layer according to the input network layer contribution degree, the output network layer contribution degree and the network layer weight to obtain the pruned attention network layer.

[0019] In some embodiments, the unit pruning of the candidate attention unit according to the unit weight to obtain the pruned attention unit comprises:

[0020] selecting a unit included in the pruned attention network layer from the candidate attention unit as a target candidate attention unit;

[0021] obtaining a candidate feature of the target candidate attention unit; the candidate feature is the preliminary text feature or the preliminary image feature obtained by the target candidate attention unit;

[0022] performing feature contribution degree screening on the candidate feature according to the network layer weight to obtain a target feature;

[0023] The candidate attention unit is masked according to the unit weight, to obtain the pruned attention unit.

[0024] In some embodiments, the performing speech category quality inspection on the target multi-modal data by using the target attention network layer to obtain a speech quality inspection category comprises:

[0025] The target speech data is subjected to speech recognition to obtain target text data, and the target text data is subjected to feature extraction by using the target attention network layer to obtain target text features;

[0026] The target image data is subjected to feature extraction by using the target attention network layer to obtain target image features;

[0027] The target text features and the target image features are fused to obtain target features;

[0028] The target features are subjected to speech category quality inspection to obtain the speech quality inspection category.

[0029] In some embodiments, after determining the target attention network layer according to the pruned attention network layer and the pruned attention unit, the method comprises:

[0030] The unit gradient values of the candidate attention unit are obtained according to the preliminary text features and the preliminary image features;

[0031] The network layer gradient values of the candidate attention network layer are obtained by averaging all the unit gradient values;

[0032] The network gradient variance of the candidate attention network layer is determined according to the unit gradient values and the network layer gradient values;

[0033] The gradient network layer pruning is performed on the candidate attention network layer according to the network layer gradient values and the network layer gradient variance to obtain a gradient pruned attention network layer;

[0034] The gradient unit pruning is performed on the candidate attention unit in the gradient pruned attention network layer according to the unit gradient values and the network layer gradient variance to obtain a gradient pruned attention unit;

[0035] The target attention network layer is updated according to the gradient pruned attention network layer and the gradient pruned attention unit to obtain an updated attention network layer, and the updated attention network layer is used as the target attention network layer.

[0036] In some embodiments, the gradient network layer pruning of the candidate attention network layer according to the network layer gradient value and the network layer gradient variance includes:

[0037] The network layer gradient variance is subjected to gradient variance importance detection to obtain a network layer gradient variance importance;

[0038] The network layer gradient value is subjected to gradient importance detection to obtain a network layer gradient importance;

[0039] The candidate attention network layer is subjected to gradient network layer pruning based on the network layer gradient importance and the network layer gradient importance to obtain the gradient pruning attention network layer.

[0040] In some embodiments, the gradient unit pruning of the candidate attention unit in the gradient pruning attention network layer according to the unit gradient value and the network layer gradient variance includes:

[0041] The unit gradient value is subjected to gradient importance detection to obtain a unit gradient importance;

[0042] The candidate attention unit in the gradient pruning attention network layer is subjected to gradient unit pruning based on the unit gradient importance, the network layer gradient method importance and the network layer gradient importance to obtain the gradient pruning attention unit.

[0043] To achieve the above object, a second aspect of the embodiment of the present application proposes a speech quality inspection device, which comprises:

[0044] A multi-modal data acquisition module is configured to acquire original multi-modal data; wherein the original multi-modal data comprises original image data and original speech data;

[0045] An image feature extraction module is configured to perform image feature extraction on the original image data by using original attention units of an original attention network layer to obtain preliminary image features;

[0046] A text feature extraction module is configured to perform speech recognition on the original speech data to obtain preliminary text data, and perform text feature extraction on the preliminary text data by using the original attention units to obtain preliminary text features;

[0047] A network layer weight acquisition module is configured to acquire network layer weights of a candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is the original attention network layer having a connection relationship;

[0048] a network layer pruning module configured to prune the candidate attention network layer according to the network layer weights to obtain a pruned attention network layer;

[0049] a unit gradient pruning module configured to obtain unit weights of a candidate attention unit in each of the candidate attention network layers according to the preliminary text features and the preliminary image features, and prune the candidate attention unit according to the unit weights to obtain a pruned attention unit;

[0050] a target network layer obtaining module configured to determine a target attention network layer according to the pruned attention network layer and the pruned attention unit;

[0051] a voice quality inspection module configured to perform voice category quality inspection on target multi-modal data by using the target attention network layer to obtain a voice quality inspection category, wherein the target multi-modal data comprises target voice data and target image data.

[0052] To achieve the above object, a third aspect of embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0053] To achieve the above object, a fourth aspect of embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0054] The voice quality inspection method and device, the electronic device and the storage medium provided by the present application can obtain original multi-modal data, wherein the original multi-modal data comprises original image data and original voice data; image feature extraction is performed on the original image data by using an original attention unit of an original attention network layer to obtain preliminary image features; voice recognition is performed on the original voice data to obtain preliminary text data, and text feature extraction is performed on the preliminary text data by using the original attention unit to obtain preliminary text features; network layer weights of a candidate attention network layer are obtained according to the preliminary text features and the preliminary image features, wherein the candidate attention network layer is the original attention network layer having a connection relationship; network layer pruning is performed on the candidate attention network layer according to the network layer weights to obtain a pruned attention network layer; unit weights of a candidate attention unit in each layer of the candidate attention network layer are obtained according to the preliminary text features and the preliminary image features, and unit pruning is performed on the candidate attention unit according to the unit weights to obtain a pruned attention unit; a target attention network layer is determined according to the pruned attention network layer and the pruned attention unit; voice quality inspection categories are obtained by performing voice quality inspection on target multi-modal data by using the target attention network layer, wherein the target multi-modal data comprises target voice data and target image data. The voice quality inspection method and device, the electronic device and the storage medium provided by the present application can first perform network layer pruning on the candidate attention network layer according to the network layer weights to obtain the pruned attention network layer, which can remove the network layers having the least influence on the voice quality inspection effect, so as to reduce the redundancy and the calculation burden of the network layer, facilitate subsequent improvement of the voice quality inspection efficiency, secondly, perform unit pruning on the candidate attention unit according to the unit weights to obtain the pruned attention unit, which ensures that only the attention units most sensitive to the preliminary text and the preliminary image features are retained, further removes the redundant attention units in the network layer, and improves the voice quality inspection efficiency while still ensuring the capturing ability of the network layer to the key information, and finally performs voice quality inspection on the target multi-modal data by using the target attention network layer to obtain the voice quality inspection categories, which realizes voice quality inspection of the target attention network layer combined with network layer pruning and unit pruning, avoids extraction of redundant text or image features, and further improves the voice quality inspection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart of the voice quality inspection method provided by the present application;

[0056] Figure 2 is a flowchart of step S105 in Figure 1

[0057] Figure 3 is a flowchart of step S106 in Figure 1

[0058] Figure 4 is another flowchart of the voice quality inspection method provided by the present application;​​

[0059] Figure 5 is Figure 4 a flowchart of step S404 in

[0060] Figure 6 is Figure 1 a flowchart of step S108 in

[0061] Figure 7 is a structural schematic diagram of a voice quality inspection device provided by an embodiment of the present application;

[0062] Figure 8 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0064] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0066] First, several terms involved in the present application are analyzed:

[0067] Artificial intelligence (AI): is a new technical science of studying, developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0068] Based on this, the embodiments of the present application provide a voice quality inspection method and device, electronic equipment and storage medium, aiming to improve the efficiency of voice quality inspection.

[0069] The voice quality inspection method and device, electronic equipment and storage medium provided by the embodiments of the present application are specifically explained by the following embodiments, first, the voice quality inspection method in the embodiments of the present application is described.

[0070] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0071] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, processing / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0072] The voice quality inspection method provided in the embodiments of the present application relates to the technical field of financial technology and artificial intelligence. The voice quality inspection method provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can further be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer or the like; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system formed by a plurality of physical servers, and can further be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application for implementing the voice quality inspection method, but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0074] Figure 1 is an optional flowchart of the voice quality inspection method provided in the embodiments of the present application, Figure 1 The method in the flowchart can include, but is not limited to, steps S101 to S108.

[0075] Step S101, obtaining original multi-modal data; wherein the original multi-modal data includes original image data and original voice data.

[0076] Step S102, performing image feature extraction on the original image data by using an original attention unit of an original attention network layer to obtain preliminary image features.

[0077] Step S103, performing voice recognition on the original voice data to obtain preliminary text data, and performing text feature extraction on the preliminary text data by using the original attention unit to obtain preliminary text features.

[0078] In step S104, the network layer weight of the candidate attention network layer is obtained according to the preliminary text feature and the preliminary image feature; the candidate attention network layer is the original attention network layer having a connection relationship.

[0079] In step S105, the network layer pruning is performed on the candidate attention network layer according to the network layer weight, and the pruned attention network layer is obtained.

[0080] In step S106, the unit weight of the candidate attention unit in each layer of the candidate attention network layer is obtained according to the preliminary text feature and the preliminary image feature, and the unit pruning is performed on the candidate attention unit according to the unit weight, and the pruned attention unit is obtained.

[0081] In step S107, the target attention network layer is determined according to the pruned attention network layer and the pruned attention unit.

[0082] In step S108, the speech category quality inspection is performed on the target multi-modal data by using the target attention network layer, and the speech quality inspection category is obtained; the target multi-modal data includes target speech data and target image data.

[0083] The steps S101 to S108 shown in the embodiments of the present application are as follows: original multi-modal data is obtained; the original multi-modal data includes original image data and original speech data; an original attention unit of an original attention network layer is used to perform image feature extraction on the original image data to obtain preliminary image features; speech recognition is performed on the original speech data to obtain preliminary text data, and the original attention unit is used to perform text feature extraction on the preliminary text data to obtain preliminary text features; network layer weights of a candidate attention network layer are obtained according to the preliminary text features and the preliminary image features; the candidate attention network layer is the original attention network layer having a connection relationship; network layer pruning is performed on the candidate attention network layer according to the network layer weights to obtain a pruned attention network layer; unit weights of a candidate attention unit in each layer of the candidate attention network layer are obtained according to the preliminary text features and the preliminary image features, and unit pruning is performed on the candidate attention unit according to the unit weights to obtain a pruned attention unit; a target attention network layer is determined according to the pruned attention network layer and the pruned attention unit; speech category quality inspection is performed on target multi-modal data by using the target attention network layer to obtain a speech quality inspection category; the target multi-modal data includes target speech data and target image data. First, the network layer pruning is performed on the candidate attention network layer according to the network layer weights to obtain the pruned attention network layer, which can remove the network layers that have the least impact on the speech quality inspection effect, so as to reduce the redundancy and computational burden of the network layer, facilitate subsequent improvement of the speech quality inspection efficiency, and the like. Second, the unit pruning is performed on the candidate attention unit according to the unit weights to obtain the pruned attention unit, which ensures that only the attention units that are most sensitive to the preliminary text and preliminary image features are retained, further removes the redundant attention units in the network layer, improves the speech quality inspection efficiency, and still guarantees the capturing ability of the network layer to the key information. Finally, the speech category quality inspection is performed on the target multi-modal data by using the target attention network layer to obtain the speech quality inspection category, which realizes the speech quality inspection of the target attention network layer combined with the network layer pruning and the unit pruning, avoids extracting redundant text or image features, and further improves the efficiency of the speech quality inspection.

[0084] In step S101 of some embodiments, specifically, the original multi-modal data includes original image data and original speech data.

[0085] For example, in a financial and technological application scenario, when vehicle insurance claims settlement needs to be performed, the original speech data is vehicle accident speech data, and the original image data is a vehicle damage photo of a vehicle accident scene and a video frame image of the vehicle accident.

[0086] Specifically, the original speech data can be obtained from a report telephone recording or a vehicle accident scene recording, and the original image data can be obtained from a photo taken at the vehicle accident scene and an accident video recorded by a vehicle event data recorder.

[0087] In step S102 of some embodiments, the original attention network layer is a Mult iway Transformer model, which includes a visual network layer and a language network layer, and each network layer contains at least two original attention units, which can be Attention Heads.

[0088] Specifically, the visual network layer is used for image feature extraction of the original image data, and the language network layer is used for extracting semantic information of the preliminary text data in the original speech data.

[0089] Specifically, the original image data can be subjected to convolution operation by the visual network layer to obtain a feature map, and the Attention Head is used to perform attention weight (i.e., query Q-key K-value V) operation on the feature map to obtain a high-dimensional vector representation of the preliminary image feature.

[0090] In step S103 of some embodiments, specifically, the original speech data is first preprocessed by automatic speech recognition technology (including removing background noise, echo cancellation, etc.) to obtain a preprocessed speech signal, so as to ensure the clarity of the original speech data; secondly, the preprocessed speech signal is converted into a series of digital features, which can reflect the acoustic characteristics (such as pitch, intensity, duration, etc.) of the original speech data; finally, the possible pronunciation unit sequence is predicted according to the acoustic characteristics, and the pronunciation unit sequence is combined into meaningful words and sentences according to the grammar rules and context information to generate the preliminary text data.

[0091] For example, in the financial and technological scene, when the vehicle owner encounters a vehicle accident, he / she usually calls the insurance company's report phone number for claim application immediately, and the vehicle owner will describe the specific circumstances of the vehicle accident in the phone, including time, place, cause of the accident, vehicle loss, and possible personnel casualties, etc. Information can be recognized as text data by automatic speech recognition technology.

[0092] Specifically, the preliminary text data can be subjected to word embedding processing by the language network layer to obtain vector representation of the preliminary text data, and the Attention Head is used to perform attention weight (i.e., query Q-key K-value V) operation between words of the vector representation of the preliminary text data, and the word representation is updated according to the attention weight, and finally the preliminary text feature is obtained.

[0093] In step S104 of some embodiments, specifically, the candidate attention network layer can be an attention network layer with a directed connection relationship, i.e., the directed connection relationship is the input and output relationship between the candidate attention network layers.

[0094] For example, in the financial field, if the Lth candidate attention network layer is an input attention network layer, and if the Rth candidate attention network layer receives the preliminary text feature or the preliminary image feature output by the Lth candidate attention network layer, the Rth candidate attention network layer is an output attention network layer.

[0095] Specifically, the network layer weight refers to the weight value from the input attention network layer to the output attention network layer, indicating the importance of the preliminary text feature or the preliminary image feature between the candidate attention network layers to the voice quality inspection, and the greater the network layer weight value, the greater the influence of the candidate attention network layer on the voice quality inspection result.

[0096] Specifically, the network layer weight can be determined by expert experience or directly obtained from the network layer parameters.

[0097] Please refer to Figure 2 In some embodiments, step S105 includes but is not limited to steps S201 to S205:

[0098] Step S201, selecting any network layer from the candidate attention network layers as an input attention network layer, and obtaining the candidate attention network layer having a connection relationship with the input attention network layer as an output attention network layer.

[0099] Step S202, obtaining the output feature quantity of the input attention network layer; obtaining the input feature quantity of the output attention network layer.

[0100] Step S203, obtaining the input activation value of the input attention network layer, and determining the layer feature contribution degree identification of the input attention network layer according to the input activation value, the output feature quantity and the network layer weight, to obtain the input network layer contribution degree.

[0101] Step S204, obtaining the output activation value of the output attention network layer, and performing layer feature contribution degree identification on the output attention network layer according to the output activation value, the input feature quantity and the network layer weight, to obtain the output network layer contribution degree.

[0102] Step S205, performing network layer pruning on the candidate attention network layer based on the input network layer contribution degree, the output network layer contribution degree and the network layer weight, to obtain a pruned attention network layer.

[0103] In step S201 of some embodiments, specifically, the Lth layer can be selected from the candidate attention network layers as an input attention network layer, and the Rth layer candidate attention network layer connected to the Lth layer can be selected as an output attention network layer.

[0104] In step S202 of some embodiments, specifically, the output feature quantity represents the number of preliminary text features or preliminary image features in the Rth candidate attention network layer, i.e., the sum of all features connected to the output preliminary text features or preliminary image features of the Lth candidate attention network layer.

[0105] Specifically, the input feature quantity represents the number of preliminary text features or preliminary image features in the Lth candidate attention network layer, i.e., the sum of all features connected to the input preliminary text features or preliminary image features of the Rth candidate attention network layer.

[0106] In step S203 of some embodiments, specifically, the input network layer contribution degree refers to the importance of the input attention network layer in the entire network layer represented by the quantified network layer index, i.e., the influence degree of all preliminary text features or preliminary image features in the input attention network layer on the voice quality inspection result.

[0107] Specifically, the input network layer contribution degree can be determined by the following formula:

[0108]

[0109] wherein S(L) represents the input network layer contribution degree of the Lth input attention network layer, R represents the output feature quantity, a L represents the input activation value vector of the Lth input attention network layer, and ||a L ||2represents the two-norm of the input activation value vector, i.e., the input activation value, and θ LR represents the network layer weight from the Lth input attention network layer to the Rth output attention network layer.

[0110] In step S204 of some embodiments, specifically, the output network layer contribution degree refers to the importance of the output attention network layer in the entire network layer represented by the quantified network layer index, i.e., the influence degree of all preliminary text features or preliminary image features in the output attention network layer on the voice quality inspection result.

[0111] Specifically, the output network layer contribution degree can be determined by the following formula:

[0112]

[0113] wherein S(R) represents the output network layer contribution degree of the Rth output attention network layer, L represents the input feature quantity, a R represents the input activation value vector of the Rth output attention network layer, and ||a R ||2represents the two-norm of the output activation value vector, i.e., the output activation value, and θ LRdenote the network layer weights from the Lth input attention network layer to the Rth output attention network layer.

[0114] In this embodiment, the layer feature contribution degree of the output attention network layer is identified according to the output activation value, the number of input features and the network layer weights, which can quantify the contribution degree of each output attention network layer to the target network layer, and help to intuitively extract whether the features extracted by different output attention network layers are key features, thereby facilitating the improvement of subsequent voice quality inspection efficiency.

[0115] In step S205 of some embodiments, specifically, the network layer contribution degree is determined by multiplying the output network layer contribution degree and the input network layer contribution degree, and the network layer contribution degree is multiplied by the network layer weight, so as to determine the importance of the features from the Lth input attention network layer to the Rth output attention network layer to the voice quality inspection result.

[0116] For example, in the financial field, when performing vehicle insurance claim settlement, the calculation of the claim settlement amount is required. If the Lth input attention network layer performs vehicle feature recognition of vehicle cracks, depressions and vehicle types, the input network layer contribution degree obtained is 0.8, and the Rth output attention network layer receives the vehicle crack, depression and vehicle type features, and the output network layer contribution degree obtained is 0.85, then the vehicle damage degree is determined as 0.825, and combined with the network layer weight from the Lth layer to the Rth layer being 0.7, the importance of the features of the Lth candidate attention network layer to the calculation of the claim settlement amount is determined as 0.58. Since 0.58 is greater than the preset network layer contribution degree 0.5, the Lth layer does not need to be pruned. If the vehicle features of the Lth input attention network layer are vehicle type, license plate and logo, the input network layer contribution degree obtained is 0.5, and the Rth output attention network layer receives the vehicle type, license plate and logo features, and the output network layer contribution degree obtained is 0.5, then the vehicle damage degree is determined as 0.5, and combined with the network layer weight from the Lth layer to the Rth layer being 0.7, the importance of the features of the Lth candidate attention network layer to the calculation of the claim settlement amount is determined as 0.35. Since 0.35 is less than the preset network layer contribution degree 0.5, it indicates that the vehicle type, license plate and logo of the Lth layer have less influence on the claim settlement amount, and therefore the Lth input attention network layer needs to be pruned to avoid irrelevant redundant features occupying network operation memory.

[0117] In this embodiment, the network layer pruning of the candidate attention network layer is performed based on the input network layer contribution degree, the output network layer contribution degree and the network layer weight, which can remove the network layers that have the least influence on the voice quality inspection effect, so as to reduce the redundancy and calculation burden of the network layers, and facilitate the subsequent improvement of the voice quality inspection efficiency.

[0118] In step S106 of some embodiments, the unit weights are automatically learned through the training and optimization process of the network layer, and these weights reflect the importance of different candidate attention unit output information when the network layer processes different speech quality inspection tasks.

[0119] Referring to Figure 3 In some embodiments, step S106 includes, but is not limited to, steps S301 to S304:

[0120] In step S301, the target candidate attention unit is selected from the candidate attention units included in the pruned attention network layer.

[0121] In step S302, the candidate feature of the target candidate attention unit is obtained; the candidate feature is the preliminary text feature or the preliminary image feature obtained by the target candidate attention unit.

[0122] In step S303, the candidate feature is filtered according to the feature contribution degree of the network layer weight, and the target feature is obtained.

[0123] In step S304, the candidate attention unit is masked according to the unit weight, and the pruned attention unit is obtained.

[0124] In step S301 of some embodiments, specifically, the target candidate attention unit is the unit included in the pruned attention network layer.

[0125] For example, the candidate attention network layer includes 1, 2, 3, and 4 layers, and the pruned attention network layer includes 1, 2, and 4 layers. When pruning the third layer of the candidate attention network layer, the attention units in the third layer are also pruned, so the remaining attention units are the units included in the pruned attention layer.

[0126] In step S302 of some embodiments, specifically, the candidate feature is the preliminary text feature or the preliminary image feature obtained by the target candidate attention unit.

[0127] For example, in the financial field, when handling vehicle insurance claims, the preliminary text features (such as vehicle accident cause, tire damage, and paint drop features) obtained by multiple Attention Heads (i.e., target candidate attention units) are used as candidate features, and the preliminary image features (such as vehicle body crack feature map, bumper indentation feature map, and tire damage feature map) obtained by different Attention Heads are used as candidate features.

[0128] In step S303 of some embodiments, first, the pruning attention network layer extracts preliminary text features or preliminary image features, obtains feature weights of the features in each network layer, compares the feature weights with network layer weights, sorts the features greater than the network layer weights in descending order according to the comparison results, and selects the top k most important features as target features based on the sorting; wherein the feature weights are determined based on the layer parameter weights in the pruning attention network layer.

[0129] In this embodiment, by screening the feature contribution degree of the candidate features according to the network layer weights, the key parameter distribution of the pruning attention network layer when extracting features can be reflected, which helps to retain important parameter information in the subsequent unit pruning process.

[0130] In step S304 of some embodiments, specifically, by comparing the unit weight with the target feature weight, if the target feature weight is greater than or equal to the unit weight, it means that the attention unit parameter of the target feature also belongs to the key parameter, which is retained in the mask process; if the target feature weight is less than the unit weight, it means that the attention unit parameter of the target feature does not belong to the key parameter, and the target feature is subjected to mask processing to obtain a pruning attention unit.

[0131] In this embodiment, the candidate attention unit is subjected to mask processing according to the unit weight to obtain a pruning attention unit, which can ensure that only the attention unit most sensitive to the preliminary text and preliminary image features is retained, further removing redundant attention units in the network layer, improving the efficiency of voice quality inspection while still ensuring the ability of the network layer to capture key information.

[0132] In step S107 of some embodiments, specifically, by fusing the parameters of the pruned attention unit and the pruned attention network layer, a target attention network layer can be generated.

[0133] Specifically, after generating the target attention network layer, the target attention network layer can also be fine-tuned or verified to ensure that the voice quality inspection performance of the generated target attention network layer can achieve the expected effect.

[0134] Please refer to Figure 4 In some embodiments, after step S107, the voice quality inspection method can include but is not limited to steps S401 to S406:

[0135] Step S401: obtaining a unit gradient value of a candidate attention unit according to the preliminary text features and the preliminary image features.

[0136] Step S402: performing average processing on all unit gradient values to obtain a network layer gradient value of a candidate attention network layer;

[0137] In step S403, the network gradient variance of the candidate attention network layer is determined according to the unit gradient value and the network layer gradient value.

[0138] In step S404, the gradient network layer pruning is performed on the candidate attention network layer according to the network layer gradient value and the network layer gradient variance, to obtain a gradient-pruned attention network layer.

[0139] In step S405, the gradient unit pruning is performed on the candidate attention unit in the gradient-pruned attention network layer according to the unit gradient value and the network layer gradient variance, to obtain a gradient-pruned attention unit.

[0140] In step S406, the target attention network layer is updated according to the gradient-pruned attention network layer and the gradient-pruned attention unit, to obtain an updated attention network layer, and the updated attention network layer is taken as the target attention network layer.

[0141] In step S401 of some embodiments, specifically, the gradient is the partial derivative of the loss function with respect to the attention unit parameter, which indicates the direction in which the loss function increases fastest in the parameter space, and the learning rate determines the magnitude of the attention unit parameter update. Combining the gradient with the learning rate can represent the actual number of changes of the attention unit parameter at the current learning rate.

[0142] Specifically, the unit gradient value can be represented by the average of the ratio of Δy to Δx, Δx is used to represent the learning rate for controlling the step size of the attention unit parameter update, and Δy is used to represent the output change caused by the updated step size. The unit gradient value is used to represent how the attention unit parameter should be adjusted to minimize the loss at the current learning rate.

[0143] For example, in the financial field, when performing vehicle insurance claim settlement, the network layer calculates the loss by comparing the actual claim settlement amount with the predicted claim settlement amount, and the unit gradient value represents the gradient of the loss, which is used to indicate how to adjust the attention unit parameter to reduce the prediction error.

[0144] In step S402 of some embodiments, specifically, the network layer gradient value can be represented by the average of the unit gradient values of all Attention Heads contained therein, and the network layer gradient value is used to indicate the parameter adjustment of the candidate attention network layer.

[0145] In step S403 of some embodiments, specifically, for the unit gradient value of each attention head, the sum of squares of the difference between the unit gradient value and the average of the network layer gradient values of the entire attention network layer is calculated, and then divided by the number of attention heads in the attention network layer, to obtain the network gradient variance.

[0146] Specifically, the network gradient variance can be used to measure the dispersion degree of the unit gradient values of the attention heads in the attention network layer. The greater the network gradient variance is, the greater the difference between the unit gradient values of different attention heads is, and vice versa.

[0147] Referring to Figure 5 In some embodiments, step S404 includes, but is not limited to, steps S501 to S503.

[0148] Step S501, gradient variance importance detection is performed on the network layer gradient variance to obtain the network layer gradient variance importance.

[0149] Step S502, gradient importance detection is performed on the network layer gradient value to obtain the network layer gradient importance.

[0150] Step S503, gradient network layer pruning is performed on the candidate attention network layer based on the network layer gradient importance and the network layer gradient importance to obtain the gradient-pruned attention network layer.

[0151] In step S501 of some embodiments, specifically, the gradient variance importance detection on the network layer gradient variance is achieved by comparing the network layer gradient variance of the selected candidate attention network layer with a preset network layer gradient variance threshold.

[0152] Specifically, the preset network layer gradient variance threshold can be systematically searched for the optimal network layer gradient variance threshold based on a hyperparameter search technique (such as grid search, random search, Bayesian optimization, etc.).

[0153] In step S502 of some embodiments, specifically, the gradient importance detection on the network layer gradient value is achieved by comparing the network layer gradient value of the selected candidate attention network layer with a preset network layer gradient threshold.

[0154] Specifically, the preset network layer gradient threshold is also systematically searched for the optimal network layer gradient threshold based on a hyperparameter search technique (such as grid search, random search, Bayesian optimization, etc.).

[0155] In step S503 of some embodiments, specifically, the pruning evaluation of the gradient network layer is based on that if the network layer gradient variance of the selected candidate attention network layer is less than the preset network layer gradient variance threshold, and the network layer gradient value of the candidate attention network layer is also less than the preset network layer gradient threshold, it indicates that the contribution of the candidate attention network layer to the target network layer is low. By removing the candidate attention network layer and all network connections of the layer, the gradient-pruned attention network layer is obtained.

[0156] For example, in the field of finance, when performing vehicle claim assessment, if the image features extracted by one of the candidate attention network layers are non-vehicle loss parts in the image (such as pedestrians, license plates, etc.), and the contribution of the candidate attention network layer to the target network layer is low, the candidate attention network layer and all network connections of the layer are removed.

[0157] In this embodiment, gradient network layer pruning is performed on the candidate attention network layer according to the network layer gradient value and the network layer gradient variance, which can remove layers that have a small contribution to the gradient of the final output of the target network layer, significantly reduce the computational complexity of the target network layer, and improve the operation efficiency of the target network layer.

[0158] In step S405 of some embodiments, specifically, gradient unit pruning is performed on the candidate attention unit in the gradient pruning attention network layer according to the unit gradient value and the network layer gradient variance to obtain a gradient pruning attention unit, including: performing gradient importance detection on the unit gradient value to obtain a unit gradient importance; and performing gradient unit pruning on the candidate attention unit in the gradient pruning attention network layer based on the unit gradient importance, the network layer gradient method importance, and the network layer gradient importance to obtain a gradient pruning attention unit.

[0159] Specifically, the gradient importance detection on the unit gradient value is achieved by comparing the unit gradient value of all candidate attention units included in the selected candidate attention network layer with the network layer gradient value of the candidate attention network layer.

[0160] Specifically, the gradient unit pruning evaluation is based on that if the network layer gradient variance of the selected candidate attention network layer is greater than the preset network layer gradient variance threshold, the network layer gradient value of the candidate attention network layer is less than the preset network layer gradient threshold, and the unit gradient value is less than the network layer gradient value, it indicates that the contribution of the candidate attention unit to the target network layer is low, and by removing the unit parameters of the candidate attention unit, a gradient pruning attention unit is obtained.

[0161] For example, in the field of finance, when performing vehicle claim assessment, if the preliminary text features extracted by one of the Attention Heads are features irrelevant to the degree of vehicle loss (such as repeated vehicle loss description information, the structure of vehicle hubs, etc.), and the contribution of the Attention Head to the target network layer is low, all parameters of the Attention Head are removed.

[0162] In this embodiment, by pruning the gradient units in the candidate attention units in the gradient pruning attention network layer according to the unit gradient value and the network layer gradient variance, the redundant features of the candidate attention units can be further identified based on the size and direction of the gradient, and the redundant features of the attention units are removed, and the unit gradient pruning can also help the target network layer to focus more on the feature part related to the voice quality inspection, further improving the operation efficiency of the target network layer.

[0163] In step S406 of some embodiments, specifically, all parameters of the gradient pruning attention network layer and the gradient pruning attention unit are fused with all parameters of the target attention network layer to obtain a fused attention network layer, and the fused attention network layer is used as the target attention network layer.

[0164] In this embodiment, by combining the gradient pruning of the attention network layer and the gradient pruning of the attention unit, the parameters that have the least influence on the output of the target network layer can be further identified in combination with the direction and size of the gradient, and these parameters are removed, the important features of the target network layer are retained, the redundant parameters of the target network layer are further reduced while ensuring the accuracy of the target network layer for voice quality inspection, and the operation efficiency of the target network layer is improved.

[0165] Please refer to Figure 6 In some embodiments, step S108 includes but is not limited to steps S601 to S604:

[0166] Step S601, performing voice recognition on the target voice data to obtain target text data, and performing feature extraction on the target text data using the target attention network layer to obtain target text features.

[0167] Step S602, performing feature extraction on the target image data using the target attention network layer to obtain target image features.

[0168] Step S603, fusing the target text features and the target image features to obtain target features.

[0169] Step S604, performing voice category quality inspection on the target features to obtain a voice quality inspection category.

[0170] In step S601 of some embodiments, the target voice data refers to voice data to be subjected to voice quality inspection, and the target text data refers to text describing semantic content information contained in the target voice data.

[0171] For example, in the financial field, if the target voice is the vehicle accident report voice of the vehicle owner, the target text is the specific situation of the vehicle accident described by the vehicle owner, including time, location, accident cause, vehicle loss, and possible personnel casualties, etc. Through automatic speech recognition technology, these information can be recognized as text data.

[0172] Further, the method of performing speech recognition on the target voice data to obtain the target text data is consistent with the method of performing speech recognition on the original voice data to obtain the preliminary text data, which will not be repeated here.

[0173] Specifically, the target text data is extracted by the target attention unit in the target attention network layer, and the feature extraction method is consistent with the method of extracting text features from the preliminary text data using the original attention unit, which will not be repeated here.

[0174] In step S602 of some embodiments, specifically, the target image data refers to the image data to be subjected to voice quality inspection.

[0175] For example, in the financial field, if the target image data is the recording video of the driving recorder of the accident vehicle or the vehicle accident photo taken by the vehicle owner, the vehicle loss features (such as vehicle damage tire, vehicle paint loss place, bumper indentation, etc.) are extracted from the accident occurrence video recorded by the driving recorder and the vehicle accident photo taken by the vehicle owner to evaluate the vehicle loss, so as to determine the vehicle claim amount.

[0176] Specifically, the method of extracting features from the target image data using the target attention unit in the target attention network layer is consistent with the method of extracting image features from the original image data using the original attention unit in the original attention network layer, which will not be repeated here.

[0177] In step S603 of some embodiments, specifically, the target feature refers to the target text feature and the target image feature spliced, and the target feature can comprehensively represent all features in the target text feature and the target image feature.

[0178] In step S604 of some embodiments, specifically, the voice quality inspection category refers to the voice content category of the voice during quality inspection.

[0179] For example, in the financial field, voice quality inspection refers to the need to analyze different voice data, so as to determine the voice quality inspection task category as insurance product recommendation or insurance claim evaluation; if the target voice is the vehicle accident report voice of the vehicle owner, the vehicle loss is evaluated in combination with the report voice and the vehicle accident video, so as to determine the vehicle claim amount.

[0180] In this embodiment, by performing speech category quality inspection on target features, speech quality inspection of the target attention network layer, which combines network layer pruning and unit pruning, is achieved, avoiding the extraction of redundant text or image features and improving the efficiency of speech quality inspection.

[0181] This application embodiment acquires raw multimodal data, including raw image data and raw speech data. Image features are extracted from the raw image data using the original attention units of the original attention network layer to obtain preliminary image features. Speech recognition is performed on the raw speech data to obtain preliminary text data, and text features are extracted from the preliminary text data using the original attention units to obtain preliminary text features. Network layer weights for candidate attention network layers are obtained based on the preliminary text features and preliminary image features. Candidate attention network layers are raw attention network layers with connectivity relationships. Network layer pruning is performed on the candidate attention network layers based on the network layer weights to obtain pruned attention network layers. Unit weights of candidate attention units in each candidate attention network layer are obtained based on the preliminary text features and preliminary image features, and unit pruning is performed on the candidate attention units based on the unit weights to obtain pruned attention units. A target attention network layer is determined based on the pruned attention network layer and pruned attention units. Speech category quality inspection is performed on the target multimodal data using the target attention network layer to obtain speech quality inspection categories. The target multimodal data includes target speech data and target image data. This application first prunes candidate attention network layers based on their weights to obtain pruned attention network layers. This removes network layers that have the least impact on speech quality inspection, reducing redundancy and computational burden, and facilitating subsequent improvements in speech quality inspection efficiency. Second, it prunes candidate attention units based on their weights to obtain pruned attention units. This ensures that only attention units most sensitive to initial text and image features are retained, further removing redundant attention units. This improves speech quality inspection efficiency while maintaining the network layers' ability to capture key information. Finally, it uses a target attention network layer to perform speech category quality inspection on target multimodal data, obtaining the speech quality inspection category. This application achieves speech quality inspection using a target attention network layer that combines network layer pruning and unit pruning, avoiding the extraction of redundant text or image features and further improving the efficiency of speech quality inspection.

[0182] Please see Figure 7 This application also provides a voice quality inspection device that can implement the above-mentioned voice quality inspection method. The device includes:

[0183] The multimodal data acquisition module is used to acquire raw multimodal data, which includes raw image data and raw speech data.

[0184] The image feature extraction module is configured to perform image feature extraction on the original image data by using the original attention units of the original attention network layer to obtain preliminary image features.

[0185] The text feature extraction module is configured to perform speech recognition on the original speech data to obtain preliminary text data, and perform text feature extraction on the preliminary text data by using the original attention units to obtain preliminary text features.

[0186] The network layer weight acquisition module is configured to acquire network layer weights of a candidate attention network layer according to the preliminary text features and the preliminary image features; the candidate attention network layer is the original attention network layer having a connection relationship.

[0187] The network layer pruning module is configured to perform network layer pruning on the candidate attention network layer according to the network layer weights to obtain a pruned attention network layer.

[0188] The unit gradient pruning module is configured to acquire unit weights of the candidate attention units in each layer of the candidate attention network layer according to the preliminary text features and the preliminary image features, and perform unit pruning on the candidate attention units according to the unit weights to obtain pruned attention units.

[0189] The target network layer acquisition module is configured to determine a target attention network layer according to the pruned attention network layer and the pruned attention units.

[0190] The speech quality inspection module is configured to perform speech category quality inspection on target multi-modal data by using the target attention network layer to obtain a speech quality inspection category; the target multi-modal data includes target speech data and target image data.

[0191] The specific implementation of the speech quality inspection device is basically the same as that of the above-mentioned speech quality inspection method, and will not be described here.

[0192] Embodiments of the present application also provide an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned speech quality inspection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0193] Please refer to Figure 8 , Figure 8 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0194] The processor 801 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0195] The memory 802 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 802 can store processing systems and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 802 and are invoked and executed by the processor 801 to implement the voice quality inspection method of the embodiments of the present application.

[0196] The input / output interface 803 is configured to implement information input and output.

[0197] The communication interface 804 is configured to implement the communication interaction between the device and other devices. The communication can be implemented by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, a WI-FI, a Bluetooth, etc.).

[0198] The bus 805 is configured to transmit information between various components (for example, the processor 801, the memory 802, the input / output interface 803, and the communication interface 804) of the device.

[0199] The processor 801, the memory 802, the input / output interface 803, and the communication interface 804 are connected to each other by the bus 805 to realize the communication connection between the devices.

[0200] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the voice quality inspection method.

[0201] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0202] The voice quality inspection method and device, the electronic equipment and the storage medium provided by the embodiments of the present application can obtain original multi-modal data, wherein the original multi-modal data comprises original image data and original voice data; image feature extraction is performed on the original image data by using an original attention unit of an original attention network layer to obtain preliminary image features; voice recognition is performed on the original voice data to obtain preliminary text data, and text feature extraction is performed on the preliminary text data by using the original attention unit to obtain preliminary text features; network layer weights of a candidate attention network layer are obtained according to the preliminary text features and the preliminary image features, wherein the candidate attention network layer is the original attention network layer having a connection relationship; network layer pruning is performed on the candidate attention network layer according to the network layer weights to obtain a pruned attention network layer; unit weights of a candidate attention unit in each layer of the candidate attention network layer are obtained according to the preliminary text features and the preliminary image features, and unit pruning is performed on the candidate attention unit according to the unit weights to obtain a pruned attention unit; a target attention network layer is determined according to the pruned attention network layer and the pruned attention unit; voice category quality inspection is performed on target multi-modal data by using the target attention network layer to obtain a voice quality inspection category, wherein the target multi-modal data comprises target voice data and target image data. The voice quality inspection method and device, the electronic equipment and the storage medium provided by the embodiments of the present application can first perform network layer pruning on the candidate attention network layer according to the network layer weights to obtain the pruned attention network layer, which can remove the network layers that have the least influence on the voice quality inspection effect, so as to reduce the redundancy and the calculation burden of the network layer, facilitate subsequent improvement of the voice quality inspection efficiency, secondly, perform unit pruning on the candidate attention unit according to the unit weights to obtain the pruned attention unit, which ensures that only the attention units that are most sensitive to the preliminary text and the preliminary image features are retained, further removes the redundant attention units in the network layer, and improves the voice quality inspection efficiency while still ensuring the capturing ability of the network layer to the key information, and finally, perform voice category quality inspection on the target multi-modal data by using the target attention network layer to obtain the voice quality inspection category, which realizes voice quality inspection of the target attention network layer combined with network layer pruning and unit pruning, avoids extraction of redundant text or image features, and further improves the voice quality inspection efficiency.

[0203] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0204] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0205] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0206] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0207] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0208] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0209] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0210] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0211] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0212] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0213] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A voice quality inspection method, characterized by, The method comprises: obtaining original multi-modal data; wherein the original multi-modal data comprises original image data and original speech data; performing image feature extraction on the original image data by using an original attention unit of an original attention network layer to obtain preliminary image features; performing speech recognition on the original speech data to obtain preliminary text data, and performing text feature extraction on the preliminary text data by using the original attention unit to obtain preliminary text features; obtaining network layer weights of a candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is an original attention network layer having a connection relationship; performing network layer pruning on the candidate attention network layer according to the network layer weights to obtain a pruned attention network layer; obtaining unit weights of a candidate attention unit in each layer of the candidate attention network layer according to the preliminary text features and the preliminary image features, and performing unit pruning on the candidate attention unit according to the unit weights to obtain a pruned attention unit; determining a target attention network layer according to the pruned attention network layer and the pruned attention unit; performing speech category quality inspection on target multi-modal data by using the target attention network layer to obtain a speech quality inspection category; wherein the target multi-modal data comprises target speech data and target image data.

2. The method of claim 1, wherein, The method comprises: selecting any network layer from the candidate attention network layer as an input attention network layer, and obtaining a candidate attention network layer having a connection relationship with the input attention network layer as an output attention network layer; obtaining the number of output features of the input attention network layer; obtaining the number of input features of the output attention network layer; obtaining input activation values of the input attention network layer, and determining layer feature contribution degree identification of the input attention network layer according to the input activation values, the number of output features and the network layer weights to obtain an input network layer contribution degree; obtaining output activation values of the output attention network layer, and performing layer feature contribution degree identification of the output attention network layer according to the output activation values, the number of input features and the network layer weights to obtain an output network layer contribution degree; performing network layer pruning on the candidate attention network layer based on the input network layer contribution degree, the output network layer contribution degree and the network layer weights to obtain the pruned attention network layer.

3. The method of claim 1, wherein, The method comprises: selecting units included in the pruned attention network layer from the candidate attention units as target candidate attention units; obtaining candidate features of the target candidate attention units; the candidate features are the preliminary text features or the preliminary image features obtained by the target candidate attention units; performing feature contribution degree screening on the candidate features according to the network layer weights to obtain target features; The candidate attention units are masked according to the unit weights, to obtain the pruned attention units.

4. The method according to any one of claims 1 to 3, characterized in that, The target multi-modal data is input into the target attention network layer, to obtain target text data and target image data. The target text data is subjected to speech recognition to obtain the target text data, and the target text data is subjected to feature extraction by using the target attention network layer, to obtain target text features. The target text features and the target image features are fused to obtain target features. The target features are subjected to speech category quality inspection, to obtain the speech quality inspection category. After the target attention network layer is determined according to the pruned attention network layer and the pruned attention units, the method comprises:

5. The method of claim 1, wherein, The unit gradient values of the candidate attention units are obtained according to the preliminary text features and the preliminary image features. The network layer gradient values of the candidate attention network layer are obtained by averaging all the unit gradient values. The network layer gradient variance of the candidate attention network layer is determined according to the unit gradient values and the network layer gradient values. The gradient network layer pruning is performed on the candidate attention network layer according to the network layer gradient values and the network layer gradient variance, to obtain a gradient pruned attention network layer. The gradient unit pruning is performed on the candidate attention units in the gradient pruned attention network layer according to the unit gradient values and the network layer gradient variance, to obtain gradient pruned attention units. The target attention network layer is updated according to the gradient pruned attention network layer and the gradient pruned attention units, to obtain an updated attention network layer, and the updated attention network layer is taken as the target attention network layer. The gradient network layer pruning is performed on the candidate attention network layer according to the network layer gradient values and the network layer gradient variance, to obtain a gradient pruned attention network layer, which comprises:

6. The method of claim 5, wherein, The network layer gradient variance importance degree is obtained by gradient variance importance degree detection on the network layer gradient variance. The network layer gradient importance degree is obtained by gradient importance degree detection on the network layer gradient values. The gradient network layer pruning is performed on the candidate attention network layer based on the network layer gradient importance degree and the network layer gradient importance degree, to obtain the gradient pruned attention network layer. The gradient unit pruning is performed on the candidate attention units in the gradient pruned attention network layer according to the unit gradient values and the network layer gradient variance, to obtain gradient pruned attention units, which comprises:

7. The method of claim 6, wherein, The unit gradient importance degree is obtained by gradient importance degree detection on the unit gradient values. The gradient unit pruning is performed on the candidate attention units in the gradient pruned attention network layer based on the unit gradient importance degree, the network layer gradient variance importance degree and the network layer gradient importance degree, to obtain the gradient pruned attention units. The device comprises:

8. A voice quality monitoring device, characterized by ​ The multi-modal data acquisition module is configured to acquire original multi-modal data; wherein the original multi-modal data comprises original image data and original speech data; The image feature extraction module is configured to perform image feature extraction on the original image data by using an original attention unit of an original attention network layer, to obtain preliminary image features; The text feature extraction module is configured to perform speech recognition on the original speech data to obtain preliminary text data, and perform text feature extraction on the preliminary text data by using the original attention unit, to obtain preliminary text features; The network layer weight acquisition module is configured to acquire network layer weights of a candidate attention network layer according to the preliminary text features and the preliminary image features; wherein the candidate attention network layer is the original attention network layer having a connection relationship; The network layer pruning module is configured to perform network layer pruning on the candidate attention network layer according to the network layer weights, to obtain a pruned attention network layer; The unit gradient pruning module is configured to acquire unit weights of a candidate attention unit in each layer of the candidate attention network layer according to the preliminary text features and the preliminary image features, and perform unit pruning on the candidate attention unit according to the unit weights, to obtain a pruned attention unit; The target network layer acquisition module is configured to determine a target attention network layer according to the pruned attention network layer and the pruned attention unit; The speech quality inspection module is configured to perform speech category quality inspection on target multi-modal data by using the target attention network layer, to obtain a speech quality inspection category; wherein the target multi-modal data comprises target speech data and target image data.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech quality inspection method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the speech quality inspection method of any one of claims 1 to 7.

Images