Pruned multi-language recognition model conference recording method and device, equipment and medium

Abstract

The invention relates to the field of intelligent decision making and the field of financial science and technology, and discloses a pruning multi-language recognition model conference recording method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a trained model, carrying out the importance analysis of neurons of the trained model, obtaining the importance of the neurons, and carrying out the analysis of the importance of the neurons; performing importance analysis on weight parameters of the trained multi-language recognition model to obtain parameter importance; performing neuron pruning on the trained model to obtain a pruned neuron model, and performing weight parameter pruning on the pruned neuron model according to parameter importance to obtain a pruned parameter model; outputting text data corresponding to the voice to be recognized through the trained pruning model; and identifying text semantics in the text data by utilizing a natural language processing model, performing semantic summarization on the text semantics, and taking summarized semantics as an intelligent conference recording result of the trained pruning model. According to the method, the repetition phenomenon in the multi-language recognition model can be fully dealt with.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent decision-making and financial technology, and in particular to a pruned multi-language recognition model meeting recording method, device, computer equipment and computer-readable storage medium. Background Art

[0002] As a leading technology in the field of speech recognition, the Whisper model has been widely used in various speech transcription and natural language processing tasks. However, in the processing of long texts or complex speech inputs, the recognition results of the Whisper model may contain repeated words or sentences, which seriously affects the recognition efficiency and accuracy of the model. In addition, this repetition phenomenon not only affects the coherence and accuracy of the recognized text, but also increases the complexity of subsequent processing, affecting the user experience. The Whisper model can be applied to the financial technology industry. For example, in the smart conference scenario of the insurance industry, the text data of the meeting can be analyzed based on the user's historical voice conversation records of insurance products, and the text data can be summarized to automatically generate summary data of the meeting, thereby counting all voice conference data in the insurance industry.

[0003] Existing optimization methods mainly hope to optimize model performance through parameter tuning or data enhancement, but the effect of adjusting hyperparameters to optimize model performance is limited, and it does not target the duplication problem, so the effect is not good. Improving the generalization ability of the model by expanding and enriching the training data set is inefficient and cannot fundamentally avoid the fundamental problem of redundancy in the model's internal structure. Therefore, the existing technology cannot fully deal with the duplication phenomenon in the Whisper model, either in terms of the performance of the recognition results or in terms of optimizing the efficiency of the recognition process.

[0004] In the prior art, there is a lack of an effective method to fully deal with the duplication phenomenon occurring in the Whisper model. Summary of the invention

[0005] The present invention provides a pruned multi-language recognition model conference recording method, device, computer equipment and computer-readable storage medium, the main purpose of which is to fully deal with the repetition phenomenon occurring in the Whisper model.

[0006] In a first aspect, the present invention provides a pruned multi-language recognition model conference recording method, comprising:

[0007] Obtaining a trained multilingual recognition model, performing importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, and performing importance analysis on weight parameters in the trained multilingual recognition model to obtain parameter importance;

[0008] According to the neuron importance, the trained multi-language recognition model is pruned to obtain a pruned neuron model; according to the parameter importance, the weight parameter of the pruned neuron model is pruned to obtain a pruned parameter model;

[0009] Acquire the training data of the trained multilingual recognition model, and use the training data to perform model training on the pruning parameter model to obtain a trained pruning model;

[0010] Collecting speech to be recognized from the user terminal corresponding to the trained pruning model, inputting the speech to be recognized into the trained pruning model, and outputting text data corresponding to the speech to be recognized through the trained pruning model;

[0011] The text semantics in the text data are identified using a preset natural language processing model, the text semantics are semantically summarized to obtain the summarized semantics corresponding to the speech to be identified, and the summarized semantics are used as the intelligent meeting record result of the trained pruning model.

[0012] In a second aspect, the present invention further provides a pruned multi-language recognition model conference recording device, the device comprising:

[0013] A parameter analysis module, used to obtain a trained multilingual recognition model, perform importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, and perform importance analysis on weight parameters in the trained multilingual recognition model to obtain parameter importance;

[0014] A parameter pruning module, used for performing neuron pruning on the trained multilingual recognition model according to the neuron importance to obtain a pruned neuron model, and performing weight parameter pruning on the pruned neuron model according to the parameter importance to obtain a pruned parameter model;

[0015] A model training module, used to obtain training data of the trained multilingual recognition model, and use the training data to perform model training on the pruning parameter model to obtain a trained pruning model;

[0016] A text output module, used to collect speech to be recognized from the user terminal corresponding to the trained pruning model, input the speech to be recognized into the trained pruning model, and output text data corresponding to the speech to be recognized through the trained pruning model;

[0017] The semantic summarization module is used to use a preset natural language processing model to identify the text semantics in the text data, perform semantic summarization on the text semantics, obtain the summary semantics corresponding to the speech to be recognized, and use the summary semantics as the intelligent meeting record result of the trained pruning model.

[0018] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the pruned multilingual recognition model meeting recording method are implemented.

[0019] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the pruned multilingual recognition model meeting recording method are implemented.

[0020] It can be seen that the pruned multilingual recognition model meeting recording method, device, computer equipment and computer-readable storage medium proposed in the embodiments of the present invention can fully deal with the repetition phenomenon occurring in the Whisper model.

[0021] Furthermore, the embodiment of the present invention performs an importance analysis on the neurons in the trained multilingual recognition model and performs an importance analysis on the weight parameters in the trained multilingual recognition model to obtain parameter importance, which is used to subsequently assign a value of 0 to the weight parameters with smaller weight values. Furthermore, the embodiment of the present invention performs neuron pruning on the trained multilingual recognition model according to the neuron importance to remove redundant neurons and connections in the model, reduce the internal structural redundancy of the model, and thus reduce duplication in the recognition results. Furthermore, the embodiment of the present invention performs weight parameter pruning on the pruned neuron model according to the parameter importance to obtain a pruned parameter model for selecting some neurons with lower importance. The weight parameters are assigned a value of 0, thereby optimizing the structure of the neural network model. Furthermore, the embodiment of the present invention outputs the text data corresponding to the speech to be recognized through the trained pruning model, so as to convert the speech data into text data. Furthermore, the embodiment of the present invention recognizes the text semantics in the text data by utilizing a preset natural language processing model, so as to transcribe the content. The system can further generate a concise meeting summary based on the text semantics, thereby providing clear meeting points for the participants. Furthermore, the duplication of recognition results can be significantly reduced, the accuracy and readability of the meeting minutes can be ensured, and the user experience can be further improved. The user does not need to manually proofread and delete duplicate content, thereby reducing the workload of post-processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A schematic diagram of an application environment of a pruned multi-language recognition model conference recording method provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a process flow of a pruned multi-language recognition model conference recording method provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a module of a pruned multi-language recognition model conference recording device provided by an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0026] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention.

[0027] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The embodiment of the present invention provides a pruned multi-language recognition model conference recording method, which can be applied in Figure 1In an application environment, wherein the client communicates with the server through a network, the server can obtain a trained multilingual recognition model through the client, perform an importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, and perform an importance analysis on weight parameters in the trained multilingual recognition model to obtain parameter importance; perform neuron pruning on the trained multilingual recognition model according to the neuron importance to obtain a pruned neuron model; perform weight parameter pruning on the pruned neuron model according to the parameter importance to obtain a pruned parameter model; obtain the trained multilingual recognition model; The training data of the language recognition model is used to train the pruning parameter model using the training data to obtain a trained pruning model; the speech to be recognized is collected from the user terminal corresponding to the trained pruning model, and the speech to be recognized is input into the trained pruning model, so as to output the text data corresponding to the speech to be recognized through the trained pruning model; the text semantics in the text data are recognized using a preset natural language processing model, the text semantics are semantically summarized to obtain the summarized semantics corresponding to the speech to be recognized, and the summarized semantics are used as the intelligent meeting record result of the trained pruning model.

[0030] Reference Figure 2 As shown, it is a flowchart of a pruned multi-language recognition model conference recording method provided by an embodiment of the present invention. In the embodiment of the present invention, the pruned multi-language recognition model conference recording method includes the following steps S1-S5:

[0031] S1. Obtain a trained multilingual recognition model, perform importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, and perform importance analysis on weight parameters in the trained multilingual recognition model to obtain parameter importance.

[0032] The embodiment of the present invention obtains a trained multilingual recognition model to perform importance analysis when pruning the trained multilingual recognition model.

[0033] The trained multilingual recognition model refers to a trained Whisper model.

[0034] Furthermore, the embodiment of the present invention performs importance analysis on neurons in the trained multilingual recognition model, so as to subsequently assign a value of 0 to neurons with smaller weight values.

[0035] In one embodiment of the present invention, the importance analysis of the neurons in the trained multilingual recognition model to obtain the neuron importance includes: obtaining the weight matrix in the trained multilingual recognition model; determining whether to perform column pruning on the neuron; when the neuron is not pruned, calculating the row sum of squares of the row weights in the weight matrix; using the row sum of squares as the neuron importance of the neuron; when the neuron is pruned, calculating the column sum of squares of the column weights in the weight matrix; and using the column sum of squares as the neuron importance of the neuron.

[0036] Among them, the neuron importance refers to the root of the sum of squares of rows or columns. It should be noted that the neuron is composed of a weight matrix. Therefore, the weight matrix in the aforementioned trained multilingual recognition model represents the neuron. When pruning the neuron, it is only necessary to delete or assign 0 to the weight parameter of a row or a column of the weight matrix.

[0037] Optionally, the process of determining whether to perform column pruning on the neuron refers to pre-setting whether to use row pruning or column pruning according to pre-analyzed requirements of neural network prediction accuracy or prediction complexity.

[0038] Optionally, the process of calculating the row sum of squares of row weights in the weight matrix refers to the process of calculating the sum of squares of weight parameters of each row in the weight matrix.

[0039] Optionally, the process of determining the neuron importance of the neuron using the row sum of squares refers to using the row sum of squares as the neuron importance of the neuron.

[0040] Optionally, the principle of calculating the column sum of squares of the column weights in the weight matrix is ​​similar to the aforementioned process of calculating the row sum of squares of the row weights in the weight matrix, and is not further elaborated here.

[0041] Optionally, the principle of using the column sum of squares to determine the neuron importance of the neuron is similar to the aforementioned process of using the row sum of squares to determine the neuron importance of the neuron, and is not further elaborated here.

[0042] Furthermore, the embodiment of the present invention performs importance analysis on the weight parameters in the trained multilingual recognition model to obtain parameter importance, so as to subsequently assign zero values ​​to weight parameters with smaller weight values.

[0043] In one embodiment of the present invention, performing importance analysis on the weight parameters in the trained multilingual recognition model to obtain parameter importance includes: obtaining a weight matrix in the trained multilingual recognition model; determining the absolute value of the weight of the weight parameter in the weight matrix; and using the absolute value of the weight as the parameter importance of the weight parameter.

[0044] The absolute value of the weight refers to the absolute value of each weight parameter in the weight matrix.

[0045] S2. According to the neuron importance, prune the trained multilingual recognition model to obtain a pruned neuron model; according to the parameter importance, prune the weight parameters of the pruned neuron model to obtain a pruned parameter model.

[0046] The embodiment of the present invention performs neuron pruning on the trained multilingual recognition model according to the neuron importance, so as to remove redundant neurons and connections in the model, reduce the redundancy of the internal structure of the model, and thus reduce the duplication in the recognition results.

[0047] In one embodiment of the present invention, the neuron pruning of the trained multilingual recognition model according to the neuron importance to obtain a pruned neuron model includes: sorting the neuron importance in ascending order to obtain a neuron sequence; obtaining a neuron pruning ratio of the trained multilingual recognition model; calculating the number of neuron pruning in the trained multilingual recognition model using the neuron pruning ratio; selecting a target neuron importance that meets the neuron pruning number from the neuron sequence in ascending order; performing parameter assignment processing on a weight parameter corresponding to the target neuron importance to obtain an assignment parameter; and determining the pruned neuron model of the trained multilingual recognition model using the assignment parameter.

[0048] Among them, the neuron pruning ratio refers to the ratio between the preset number of neurons that need to be pruned and the number of neurons that do not need to be pruned. It should be noted that there are multiple minimum importances. This is because each neuron has a minimum importance, and multiple neurons have multiple minimum importances. The target neuron importance refers to the minimum importance among multiple minimum importances.

[0049] Exemplarily, the process of calculating the number of neuron pruning in the trained multilingual recognition model using the neuron pruning ratio is, for example: when the neuron ratio is 1:3, it means that when the total number of neurons is 4, the number of neurons that need to be pruned is 1 neuron, and the number of neurons that do not need to be pruned is 3 neurons.

[0050] Exemplarily, the process of selecting the target neuron importance that meets the number of neuron pruning from the neuron sequence is, for example: one neuron corresponds to one weight matrix, and the number of neurons that need to be pruned is 1 neuron, then the minimum row importance or the minimum column importance is selected from any weight matrix, a weight matrix selects a minimum neuron importance, and then multiple minimum neuron importances of multiple neurons can be obtained, and then the minimum neuron importance that meets the number of neuron pruning is selected from these multiple minimum neuron importances. For example, the multiple minimum neuron importances are A, B, and C respectively. If A is the minimum value among the minimum neuron importances, A is used as the target neuron importance.

[0051] Optionally, the process of selecting the target neuron importance that meets the neuron pruning number from the neuron sequence in order from small to large includes: obtaining the initial neuron importance belonging to the same neuron in the neuron sequence; selecting the minimum importance from the initial neuron importances; and selecting the target neuron importance from multiple minimum importances among the minimum importances.

[0052] Optionally, the process of performing parameter assignment processing on the weight parameters corresponding to the importance of the target neuron and obtaining the assigned parameters refers to assigning a value of 0 to the row weight parameters or column weight parameters corresponding to the importance of the target neuron.

[0053] Furthermore, the embodiment of the present invention prunes the weight parameters of the pruned neuron model according to the importance of the parameters to obtain a pruned parameter model for selecting some less important weight parameters to be assigned a value of 0, thereby optimizing the structure of the neural network model.

[0054] In one embodiment of the present invention, the weight parameters of the pruned neuron model are pruned according to the parameter importance to obtain a pruned parameter model, including: sorting the parameter importance in ascending order to obtain a parameter sequence; obtaining the parameter pruning ratio of the pruned neuron model; calculating the number of parameter pruning in the pruned neuron model using the parameter pruning ratio; selecting, in ascending order, a target parameter importance that meets the parameter pruning number from the parameter sequence; performing parameter pruning processing on the weight parameters corresponding to the target parameter importance to obtain pruning parameters; and determining the pruning parameter model of the pruned neuron model using the pruning parameters.

[0055] Among them, the parameter pruning ratio refers to the ratio of pruned weight parameters in the weight matrix of a neuron. For example, the number of weight parameters in the weight matrix of a neuron is 4 weight parameters, and the parameter pruning ratio is 1:3, which means that one weight parameter is selected from the four weight parameters for pruning, and the selected weight parameter is used as the target parameter importance.

[0056] Optionally, the process of performing parameter pruning on the weight parameter corresponding to the importance of the target parameter to obtain the pruned parameter refers to: assigning a value of 0 to the weight parameter corresponding to the importance of the target parameter.

[0057] S3. Acquire training data of the trained multilingual recognition model, and use the training data to perform model training on the pruning parameter model to obtain a trained pruning model.

[0058] The embodiment of the present invention obtains the training data of the trained multilingual recognition model to train the pruned trained multilingual recognition model using the training data, so that the weight parameters of the pruned parameter model can be adjusted to ensure that the output result of the pruned parameter model remains similar to the output result of the trained multilingual recognition model, thereby avoiding output errors after the trained multilingual recognition model is pruned.

[0059] Among them, the training data refers to data used to input into an untrained multilingual recognition model. It should be noted that after the untrained multilingual recognition model is trained with the training data, the untrained multilingual recognition model becomes a trained multilingual recognition model, and the output result of the trained multilingual recognition model meets the expected output result corresponding to the training data. If, after the above-mentioned neuron pruning and weight parameter pruning, the output result of the pruned parameter model cannot output the expected output result corresponding to the training data like the trained multilingual recognition model, it is necessary to train the pruned parameter model again, and the process of the pruned parameter model is to calculate the loss function value between the output result of the pruned parameter model and the actual result, and based on the loss function value, the pruned parameter model is trained by using an optimization algorithm such as a gradient descent algorithm.

[0060] Furthermore, the embodiment of the present invention uses the training data to perform model training on the pruned parameter model so as to use the training data to train the pruned trained multilingual recognition model. In this way, the weight parameters of the pruned parameter model can be adjusted to ensure that the output result of the pruned parameter model remains similar to the output result of the trained multilingual recognition model, thereby avoiding output errors in the trained multilingual recognition model after pruning.

[0061] Optionally, the process of performing model training on the pruned parameter model using the training data refers to a process of fine-tuning weight parameters in the pruned whisper model.

[0062] S4. Collecting speech to be recognized from the user terminal corresponding to the trained pruning model, inputting the speech to be recognized into the trained pruning model, and outputting text data corresponding to the speech to be recognized through the trained pruning model.

[0063] The embodiment of the present invention collects the speech to be recognized from the user terminal corresponding to the trained pruning model, so as to perform speech recognition on the speech to be recognized by using the trained pruning model, and converts the speech to be recognized into text data.

[0064] Furthermore, in an embodiment of the present invention, the speech to be recognized refers to the speech data of a meeting currently being recorded in the intelligent conference system.

[0065] Furthermore, the embodiment of the present invention inputs the speech to be recognized into the trained pruning model so as to perform speech-to-text conversion on the speech to be recognized using the trained pruning model.

[0066] Furthermore, the embodiment of the present invention outputs text data corresponding to the speech to be recognized through the trained pruning model, so as to convert the speech data into text data.

[0067] The trained pruning model includes an encoder and a decoder.

[0068] In one embodiment of the present invention, outputting the text data corresponding to the speech to be recognized through the trained pruning model includes: using a preset feature extractor to convert the speech to be recognized into a spectrogram; using an encoder in the trained pruning model to extract the hidden state in the spectrogram; and using a decoder in the trained pruning model to output the text data corresponding to the hidden state.

[0069] The feature extractor is used to perform preprocessing and format conversion on speech data, for example, pre-emphasis, frame windowing and SIFT algorithm on speech signals.

[0070] S5. Use a preset natural language processing model to identify the text semantics in the text data, perform semantic summary on the text semantics, obtain the summary semantics corresponding to the speech to be recognized, and use the summary semantics as the intelligent meeting record result of the trained pruning model.

[0071] The embodiment of the present invention utilizes a preset natural language processing model to identify the text semantics in the text data, so that after the content is transcribed, the system can further generate a concise meeting summary based on the text semantics, thereby providing participants with clear meeting points. Furthermore, the duplication of recognition results can be significantly reduced, ensuring the accuracy and readability of meeting minutes, further improving the user experience, and users do not need to manually proofread and delete duplicate content, thereby reducing the workload of post-processing.

[0072] Among them, the text semantics refers to the intention information of important keywords remaining after removing unimportant keywords in the text data, and the natural language processing model includes a feature extraction layer, a bidirectional recurrent neural network, a splicing layer and a convolutional neural network.

[0073] In one embodiment of the present invention, the method of identifying the text semantics in the text data using a preset natural language processing model includes: performing data preprocessing on the text data to obtain preprocessed data; using a feature extraction layer in the natural language processing model to identify a text vector of the preprocessed data; using a bidirectional recurrent neural network in the natural language processing model to extract a context representation of the text vector; concatenating the text vector and the context representation to obtain a concatenated vector; using a convolutional neural network in the natural language processing model to output a semantic probability value of the concatenated vector; and determining the text semantics in the text data using the semantic probability value.

[0074] Among them, the feature extraction layer refers to a feature extraction model, such as word2vec, the bidirectional recurrent neural network includes a forward RNN and a backward RNN, and the convolutional neural network refers to a CNN network.

[0075] Optionally, the process of performing data preprocessing on the text data to obtain preprocessed data refers to the process of segmenting the text data, and the process of determining the text semantics in the text data using the semantic probability value refers to taking the intention category corresponding to the probability value with the largest semantic probability value as the text semantics.

[0076] Optionally, the process of semantically summarizing the text semantics refers to the process of summarizing the text semantics.

[0077] It can be seen that the pruned multilingual recognition model meeting recording method, device, computer equipment and computer-readable storage medium proposed in the embodiments of the present invention can fully deal with the repetition phenomenon occurring in the Whisper model.

[0078] Furthermore, the embodiment of the present invention performs an importance analysis on the neurons in the trained multilingual recognition model and performs an importance analysis on the weight parameters in the trained multilingual recognition model to obtain parameter importance, which is used to subsequently assign a value of 0 to the weight parameters with smaller weight values. Furthermore, the embodiment of the present invention performs neuron pruning on the trained multilingual recognition model according to the neuron importance to remove redundant neurons and connections in the model, reduce the internal structural redundancy of the model, and thus reduce duplication in the recognition results. Furthermore, the embodiment of the present invention performs weight parameter pruning on the pruned neuron model according to the parameter importance to obtain a pruned parameter model for selecting some neurons with lower importance. The weight parameters are assigned a value of 0, thereby optimizing the structure of the neural network model. Furthermore, the embodiment of the present invention outputs the text data corresponding to the speech to be recognized through the trained pruning model, so as to convert the speech data into text data. Furthermore, the embodiment of the present invention recognizes the text semantics in the text data by utilizing a preset natural language processing model, so as to transcribe the content. The system can further generate a concise meeting summary based on the text semantics, thereby providing clear meeting points for the participants. Furthermore, the duplication of recognition results can be significantly reduced, the accuracy and readability of the meeting minutes can be ensured, and the user experience can be further improved. The user does not need to manually proofread and delete duplicate content, thereby reducing the workload of post-processing.

[0079] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0080] In one embodiment, a pruned multi-language recognition model conference recording device is provided, and the pruned multi-language recognition model conference recording device corresponds one-to-one to the pruned multi-language recognition model conference recording method in the above embodiment. Figure 3 As shown, the pruned multi-language recognition model conference recording device includes a parameter analysis module 101, a parameter pruning module 102, a model training module 103, a text output module 104 and a semantic summary module 105. The functional modules are described in detail as follows:

[0081] The parameter analysis module 101 is used to obtain a trained multilingual recognition model, perform importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, and perform importance analysis on weight parameters in the trained multilingual recognition model to obtain parameter importance;

[0082] The parameter pruning module 102 is used to perform neuron pruning on the trained multi-language recognition model according to the neuron importance to obtain a pruned neuron model, and perform weight parameter pruning on the pruned neuron model according to the parameter importance to obtain a pruned parameter model;

[0083] The model training module 103 is used to obtain the training data of the trained multi-language recognition model, and use the training data to perform model training on the pruning parameter model to obtain a trained pruning model;

[0084] The text output module 104 is used to collect the speech to be recognized from the user terminal corresponding to the trained pruning model, input the speech to be recognized into the trained pruning model, and output the text data corresponding to the speech to be recognized through the trained pruning model;

[0085] The semantic summarization module 105 is used to use a preset natural language processing model to identify the text semantics in the text data, perform semantic summary on the text semantics, obtain the summary semantics corresponding to the speech to be recognized, and use the summary semantics as the intelligent meeting record result of the trained pruning model.

[0086] In one embodiment, the parameter analysis module 101 performs importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, which is used to: obtain a weight matrix in the trained multilingual recognition model; determine whether to perform column pruning on the neuron; when performing column pruning on the neuron, calculate the row sum of squares of row weights in the weight matrix; determine the neuron importance of the neuron using the row sum of squares; when performing column pruning on the neuron, calculate the column sum of squares of column weights in the weight matrix; and determine the neuron importance of the neuron using the column sum of squares.

[0087] In one embodiment, the parameter analysis module 101 performs importance analysis on the weight parameters in the trained multilingual recognition model to obtain parameter importance, which is used to: obtain a weight matrix in the trained multilingual recognition model; determine the absolute value of the weight of the weight parameter in the weight matrix; and use the absolute value of the weight as the parameter importance of the weight parameter.

[0088] In one embodiment, the parameter pruning module 102 performs neuron pruning on the trained multilingual recognition model according to the neuron importance to obtain a pruned neuron model, and is used to: sort the neuron importance in ascending order to obtain a neuron sequence; obtain a neuron pruning ratio of the trained multilingual recognition model; calculate the number of neuron pruning in the trained multilingual recognition model using the neuron pruning ratio; select a target neuron importance that meets the neuron pruning number from the neuron sequence in ascending order; perform parameter assignment processing on a weight parameter corresponding to the target neuron importance to obtain an assignment parameter; and determine the pruned neuron model of the trained multilingual recognition model using the assignment parameter; wherein the process of selecting a target neuron importance that meets the neuron pruning number from the neuron sequence in ascending order includes: obtaining an initial neuron importance belonging to the same neuron in the neuron sequence; selecting a minimum importance from the initial neuron importance; and selecting the target neuron importance from multiple minimum importances among the minimum importances.

[0089] In one embodiment, when the parameter pruning module 103 performs weight parameter pruning on the pruned neuron model according to the parameter importance to obtain a pruned parameter model, it is used to: sort the parameter importance in ascending order to obtain a parameter sequence; obtain the parameter pruning ratio of the trained multilingual recognition model; calculate the number of parameter pruning in the trained multilingual recognition model using the parameter pruning ratio; select a target parameter importance that meets the parameter pruning number from the parameter sequence in ascending order; perform parameter pruning processing on the weight parameter corresponding to the target parameter importance to obtain a pruning parameter; and determine the pruning parameter model of the trained multilingual recognition model using the pruning parameter.

[0090] In one embodiment, the text output module 104 outputs the text data corresponding to the speech to be recognized through the trained pruning model, and is used to: convert the speech to be recognized into a spectrogram using a preset feature extractor; extract the hidden state in the spectrogram using the encoder in the trained pruning model; and output the text data corresponding to the hidden state using the decoder in the trained pruning model.

[0091] In one embodiment, the semantic summary module 105 uses a preset natural language processing model to identify the text semantics in the text data, and is used to: perform data preprocessing on the text data to obtain preprocessed data; use the feature extraction layer in the natural language processing model to identify the text vector of the preprocessed data; use the bidirectional recurrent neural network in the natural language processing model to extract the context representation of the text vector; concatenate the text vector and the context representation to obtain a concatenated vector; use the convolutional neural network in the natural language processing model to output the semantic probability value of the concatenated vector; and use the semantic probability value to determine the text semantics in the text data.

[0092] In detail, the modules in the pruned multi-language recognition model conference recording device 100 in the embodiment of the present invention are used in the same manner as described above. Figure 1 The pruned multi-language recognition model conference record method described in the present invention has the same technical means and can produce the same technical effect, so it will not be repeated here.

[0093] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a pruned multi-language recognition model meeting recording method.

[0094] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a pruned multi-language recognition model meeting recording method.

[0095] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0096] Obtaining a trained multilingual recognition model, performing importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, and performing importance analysis on weight parameters in the trained multilingual recognition model to obtain parameter importance;

[0097] According to the neuron importance, the trained multilingual recognition model is pruned to obtain a pruned neuron model; according to the parameter importance, the trained multilingual recognition model is pruned to obtain a pruned parameter model;

[0098] Acquire the training data of the trained multilingual recognition model, and use the training data to perform model training on the pruning parameter model to obtain a trained pruning model;

[0099] Collecting speech to be recognized from the user terminal corresponding to the trained pruning model, inputting the speech to be recognized into the trained pruning model, and outputting text data corresponding to the speech to be recognized through the trained pruning model;

[0100] The text semantics in the text data are identified using a preset natural language processing model, the text semantics are semantically summarized to obtain the summarized semantics corresponding to the speech to be identified, and the summarized semantics are used as the intelligent meeting record result of the trained pruning model.

[0101] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0102] Obtaining a trained multilingual recognition model, performing importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, and performing importance analysis on weight parameters in the trained multilingual recognition model to obtain parameter importance;

[0103] According to the neuron importance, the trained multilingual recognition model is pruned to obtain a pruned neuron model; according to the parameter importance, the trained multilingual recognition model is pruned to obtain a pruned parameter model;

[0104] Acquire the training data of the trained multilingual recognition model, and use the training data to perform model training on the pruning parameter model to obtain a trained pruning model;

[0105] Collecting speech to be recognized from the user terminal corresponding to the trained pruning model, inputting the speech to be recognized into the trained pruning model, and outputting text data corresponding to the speech to be recognized through the trained pruning model;

[0106] The text semantics in the text data are identified using a preset natural language processing model, the text semantics are semantically summarized to obtain the summarized semantics corresponding to the speech to be identified, and the summarized semantics are used as the intelligent meeting record result of the trained pruning model.

[0107] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0109] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0110] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should be included in the protection scope of the present invention. It should be noted that if software tools or components other than those of the company appear in the embodiments of this application, they are only used for illustration and do not represent actual use.

Claims

1. A pruned multi-language recognition model conference recording method, characterized in that: The method comprises: Obtaining a trained multilingual recognition model, performing importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, and performing importance analysis on weight parameters in the trained multilingual recognition model to obtain parameter importance; According to the neuron importance, the trained multi-language recognition model is pruned to obtain a pruned neuron model; according to the parameter importance, the weight parameter of the pruned neuron model is pruned to obtain a pruned parameter model; Acquire the training data of the trained multilingual recognition model, and use the training data to perform model training on the pruning parameter model to obtain a trained pruning model; Collecting speech to be recognized from the user terminal corresponding to the trained pruning model, inputting the speech to be recognized into the trained pruning model, and outputting text data corresponding to the speech to be recognized through the trained pruning model; The text semantics in the text data are identified using a preset natural language processing model, the text semantics are semantically summarized to obtain the summarized semantics corresponding to the speech to be identified, and the summarized semantics are used as the intelligent meeting record result of the trained pruning model.

2. The pruned multi-language recognition model conference recording method according to claim 1, characterized in that: The performing importance analysis on the neurons in the trained multilingual recognition model to obtain the importance of the neurons includes: Obtaining a weight matrix in the trained multilingual recognition model; Determine whether to perform column pruning on the neuron; When the neuron is not subjected to column pruning, calculating the row sum of squares of the row weights in the weight matrix; Taking the row sum of squares as the neuron importance of the neuron; When performing column pruning on the neuron, calculating the column sum of squares of the column weights in the weight matrix; The column sum of squares is taken as the neuron importance of the neuron.

3. The pruned multi-language recognition model conference recording method according to claim 1, characterized in that: The performing importance analysis on the weight parameters in the trained multi-language recognition model to obtain parameter importance includes: Obtaining a weight matrix in the trained multilingual recognition model; Determining the absolute values ​​of weights of weight parameters in the weight matrix; The absolute value of the weight is used as the parameter importance of the weight parameter.

4. The pruned multi-language recognition model conference recording method according to claim 1, characterized in that: The step of performing neuron pruning on the trained multilingual recognition model according to the neuron importance to obtain a pruned neuron model includes: Sorting the importance of the neurons in ascending order to obtain a neuron sequence; Obtaining the neuron pruning ratio of the trained multilingual recognition model; Calculating the number of neuron pruning in the trained multilingual recognition model using the neuron pruning ratio; Selecting, from the neuron sequence, the importance of target neurons that meet the number of neuron pruning in ascending order; Performing parameter assignment processing on the weight parameters corresponding to the importance of the target neuron to obtain assignment parameters; The assigned parameters are used to determine the pruned neuron model of the trained multilingual recognition model.

5. The pruned multi-language recognition model conference recording method according to claim 1, characterized in that: The step of pruning the weight parameters of the pruned neuron model according to the parameter importance to obtain a pruned parameter model includes: Sorting the importance of the parameters in ascending order to obtain a parameter sequence; Obtaining a parameter pruning ratio of the pruned neuron model; Calculating the number of parameter pruning in the pruned neuron model using the parameter pruning ratio; Selecting, from the parameter sequence, target parameter importances that meet the parameter pruning number in ascending order; Performing parameter pruning processing on the weight parameters corresponding to the importance of the target parameters to obtain pruning parameters; The pruning parameters are used to determine a pruning parameter model of the pruning neuron model.

6. The pruned multi-language recognition model conference recording method according to claim 1, characterized in that: Outputting text data corresponding to the speech to be recognized through the trained pruning model includes: Converting the speech to be recognized into a spectrogram using a preset feature extractor; Extracting hidden states in the spectrogram using an encoder in the trained pruning model; The decoder in the trained pruning model is used to output the text data corresponding to the hidden state.

7. The pruned multi-language recognition model conference recording method according to claim 1, characterized in that: The identifying text semantics in the text data by using a preset natural language processing model includes: Performing data preprocessing on the text data to obtain preprocessed data; Identifying text vectors of the preprocessed data using a feature extraction layer in the natural language processing model; Extracting a contextual representation of the text vector using a bidirectional recurrent neural network in the natural language processing model; Concatenating the text vector and the context representation to obtain a concatenated vector; Outputting the semantic probability value of the concatenated vector using a convolutional neural network in the natural language processing model; The semantic probability value is used to determine the text semantics in the text data.

8. A pruned multi-language recognition model conference recording device, characterized in that: The device comprises: A parameter analysis module, used to obtain a trained multilingual recognition model, perform importance analysis on neurons in the trained multilingual recognition model to obtain neuron importance, and perform importance analysis on weight parameters in the trained multilingual recognition model to obtain parameter importance; A parameter pruning module, used for performing neuron pruning on the trained multilingual recognition model according to the neuron importance to obtain a pruned neuron model, and performing weight parameter pruning on the pruned neuron model according to the parameter importance to obtain a pruned parameter model; A model training module, used to obtain training data of the trained multilingual recognition model, and use the training data to perform model training on the pruning parameter model to obtain a trained pruning model; A text output module, used to collect speech to be recognized from the user terminal corresponding to the trained pruning model, input the speech to be recognized into the trained pruning model, and output text data corresponding to the speech to be recognized through the trained pruning model; The semantic summarization module is used to use a preset natural language processing model to identify the text semantics in the text data, perform semantic summarization on the text semantics, obtain the summary semantics corresponding to the speech to be recognized, and use the summary semantics as the intelligent meeting record result of the trained pruning model.

9. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the pruned multi-language recognition model conference recording method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the pruned multi-language recognition model conference recording method according to any one of claims 1 to 7 is implemented.

Images