Speech processing method and apparatus, and device and storage medium

Abstract

A speech processing method and apparatus, and a device and a storage medium. The speech processing method comprises: performing feature extraction on speech data to be processed, so as to obtain target speech representation information of said speech data (S101); acquiring a prompt word regarding said speech data, and performing fusion processing on a speech content vector, a paralanguage vector and the prompt word, so as to obtain a speech fusion feature (S102); and performing speech conversion processing on the speech fusion feature, so as to obtain text information corresponding to said speech data (S103). The speech processing method can improve the accuracy of speech recognition.

Description

Voice processing method, device, equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 12, 2023, with application number 2023111711595 and application name “Speech Processing Method, Device, Equipment and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a speech processing method, apparatus, device and storage medium. Background Art

[0003] Speech recognition technology is used in a variety of scenarios. For example, in intelligent conversations, by recognizing and understanding the interlocutor's speech data, we can understand the interlocutor's intended meaning and select the appropriate response data for accurate response. However, in addition to the text content, the interlocutor's speech data generally contains auxiliary information that assists in identifying the interlocutor's speech data. However, current speech recognition and understanding technologies only convert the interlocutor's speech data into text information. This auxiliary information in the speech data cannot be reflected in the text information, resulting in low speech recognition accuracy.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a speech processing method, apparatus, device, and storage medium that can improve the accuracy of speech recognition.

[0006] In a first aspect, the present application provides a speech processing method, comprising:

[0007] Performing feature extraction on the speech data to be processed to obtain target speech representation information of the speech data to be processed; the target speech representation information includes a speech content vector and a paralanguage vector corresponding to the speech data to be processed, the paralanguage vector being used to assist in identifying text information corresponding to the speech data to be processed;

[0008] Obtaining a prompt word for the speech data to be processed, and fusing the speech content vector, the paralanguage vector, and the prompt word to obtain a speech fusion feature;

[0009] The speech fusion feature is subjected to speech conversion processing to obtain text information corresponding to the speech data to be processed.

[0010] In a second aspect, the present application provides a speech processing device, comprising:

[0011] a feature extraction unit configured to extract features from the speech data to be processed to obtain target speech representation information of the speech data to be processed; the target speech representation information includes a speech content vector and a paralanguage vector corresponding to the speech data to be processed, the paralanguage vector being used to assist in identifying text information corresponding to the speech data to be processed;

[0012] an information fusion unit, configured to obtain a prompt word related to the speech data to be processed, and fuse the speech content vector, the paralanguage vector, and the prompt word to obtain a speech fusion feature;

[0013] The speech conversion unit is used to perform speech conversion processing on the speech fusion feature to obtain text information corresponding to the speech data to be processed.

[0014] In a third aspect, the present application provides a computer device comprising a processor, a memory, and a network interface, wherein the processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store a computer program, and the computer program includes program instructions. The processor is configured to call the program instructions so that the computer device including the processor executes the speech processing method.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor performs the above-mentioned speech processing method.

[0016] In a fifth aspect, the present application provides a computer program product or a computer program, which includes computer instructions, and when the computer instructions are executed by a processor, the above-mentioned speech processing method can be implemented.

[0017] In an embodiment of the present application, feature extraction is performed on the speech data to be processed to obtain target speech representation information of the speech data to be processed. Prompt words about the speech data to be processed are obtained, and the target speech representation information and the prompt words are fused to obtain speech fusion features; the speech fusion features are subjected to speech conversion processing to obtain text information corresponding to the speech data to be processed. Since the target speech representation information includes the speech content vector and the paralanguage vector corresponding to the speech data to be processed, and the paralanguage vector is used to assist in identifying the text information corresponding to the speech data to be processed, when performing speech recognition on the speech data to be processed, it is possible to combine the speech content information of the speech data to be processed, the paralanguage information in the speech data to be processed, and the text content corresponding to the prompt words for speech recognition. Since more comprehensive and rich speech representation information is used for speech recognition processing, the present application can achieve in-depth speech recognition and understanding of the speech data to be processed, thereby improving the accuracy of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG1 is a schematic diagram of a network architecture of a speech processing system provided in an embodiment of the present application;

[0019] FIG2 is a schematic diagram of an application scenario of a speech processing method provided in an embodiment of the present application;

[0020] FIG3 is a flow chart of a speech processing method provided in an embodiment of the present application;

[0021] FIG4 is a schematic diagram of the architecture of a speech feature extraction model provided in an embodiment of the present application;

[0022] FIG5 is a flow chart of a method for training a speech feature extraction model provided in an embodiment of the present application;

[0023] FIG6 is a flow chart of a method for training a speech conversion model provided in an embodiment of the present application;

[0024] FIG7 is a schematic diagram of parameter adjustment in a speech conversion model provided in an embodiment of the present application;

[0025] FIG8 is a schematic diagram of the structure of a speech processing device provided in an embodiment of the present application;

[0026] FIG9 is a schematic diagram of the composition structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In some speech processing tasks, speech recognition and speech comprehension are performed on speech data to generate text data, which is then processed to achieve speech comprehension. Since speech comprehension is based on the text data generated by speech recognition, and text data only contains textual content, speech comprehension can only be based on the textual content and cannot utilize the paralinguistic information contained in the speech data. Paralinguistic information, for example, can reflect emotions, demeanor, intonation, and other hidden information within the speech data. Therefore, in complex scenarios where speech recognition performance is poor, using speech recognition results as a prerequisite for speech comprehension can lead to accumulated errors in speech comprehension, making it impossible to correct errors and resulting in low speech recognition accuracy.

[0028] In view of this, the present application provides a voice processing solution that is applicable to any voice interaction scenario such as: conference scenarios, interview scenarios, intelligent dialogue scenarios, game scenarios, etc., and can convert voice data into text information in the above-mentioned voice interaction scenarios, which can improve the accuracy of voice recognition, so as to facilitate the execution of related business processing operations in the voice interaction scenarios. Specifically, the voice processing solution does not directly recognize the voice data as text data and understand the text data, but extracts the voice representation information containing voice content vectors and paralanguage vectors in the voice data, and obtains the final text information by further processing the voice content vectors, paralanguage vectors and prompt words. Since the voice data contains paralanguage information, the accuracy of voice understanding can be improved by combining the paralanguage information to perform voice understanding on the voice data. Specifically, the principle of the voice processing solution is roughly as follows:

[0029] (1) Acquiring the voice data to be processed in the business scenario, wherein the business scenario here includes but is not limited to any voice interaction scenario such as a conference scenario, an interview scenario, an intelligent dialogue scenario, and a game scenario; the voice data to be processed can be generated by any object in the business scenario, such as a conference host in a conference scenario, a game control object or a game character in a game scenario, and so on.

[0030] (2) Feature extraction is performed on the speech data to be processed to obtain target speech representation information of the speech data to be processed. The target speech representation information includes a speech content vector and a paralanguage vector corresponding to the speech data to be processed. The paralanguage vector is used to assist in identifying the text information corresponding to the speech data to be processed.

[0031] (3) Obtain prompt words for the speech data to be processed, and fuse the speech content vector, paralanguage vector, and prompt words to obtain speech fusion features. Perform speech conversion on the speech fusion features to obtain text information corresponding to the speech data to be processed.

[0032] As can be seen from the above, in the embodiments of the present application, the target speech representation information includes the speech content vector and paralinguistic vector corresponding to the speech data to be processed, and the paralinguistic vector is used to assist in identifying the text information corresponding to the speech data to be processed. Therefore, when performing speech recognition on the speech data to be processed, speech recognition can be performed in conjunction with both the speech content information of the speech data to be processed, the paralinguistic information in the speech data to be processed, and the text content corresponding to the prompt word. By utilizing more comprehensive and rich speech representation information, the present application can achieve in-depth speech recognition and understanding of the speech data to be processed, thereby improving speech recognition accuracy.

[0033] The technical solution of this application can be applied to scenarios where voice data is converted into text information through voice recognition. For example, it can be applied to scenarios such as voice transcription in online meetings, voice input in social applications, voice-to-text conversion from recording devices in interview scenarios, and voice conversations in intelligent conversations. For example, in online conferencing scenarios, by recording the voice in the meeting to obtain voice data and then performing voice transcription on the voice data to obtain the corresponding meeting minutes, the efficiency of obtaining the key content of the meeting can be improved. For another example, in social applications, by obtaining text information from the user's voice data through voice recognition, the efficiency of text information input can be improved. For another example, in interview scenarios, by converting voice data recorded by a recording device into text, the efficiency of obtaining interview text can be improved. For another example, in intelligent conversation scenarios, by performing voice recognition on the interlocutor's voice data to obtain text information, accurate text conversations can be achieved. Optionally, the technical solution of this application can also be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0034] It should be noted that when the embodiments of this application involve data related to subject information (such as unprocessed voice data, sample voice data, prompt words, etc.), when the embodiments of this application are applied to specific products or technologies, the subject's permission or consent must be obtained, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant region. For example, the subject may refer to the user of a terminal device or computer device.

[0035] The following is a detailed introduction to the system architecture provided in the embodiments of the present application.

[0036] Please refer to Figure 1, which is a network architecture diagram of a speech processing system provided in an embodiment of the present application. As shown in Figure 1, a computer device can exchange data with a terminal device, and the number of terminal devices can be one or at least two. For example, when the number of terminal devices is multiple, the terminal devices can include terminal device 101a, terminal device 101b, and terminal device 101c in Figure 1. Taking terminal device 101a as an example, the computer device 102 can perform feature extraction on the speech data to be processed to obtain target speech representation information of the speech data to be processed. Furthermore, the computer device 102 can obtain prompt words about the speech data to be processed, and fuse the speech content vector, paralanguage vector, and prompt words to obtain speech fusion features. Furthermore, the computer device 102 can also perform speech conversion processing on the speech fusion features to obtain text information corresponding to the speech data to be processed. Optionally, the computer device 102 can send text information corresponding to the voice data to be processed to the terminal device 101a to display the text data on the terminal device 101a, or the computer device 102 can also determine reply text information based on the text information corresponding to the voice data to be processed, and send the reply text information to the terminal device 101a, and so on.

[0037] It is understood that the computer devices mentioned in the embodiments of the present application include but are not limited to terminal devices or servers. In other words, the computer device can be a server or a terminal device, or it can be a system composed of a server and a terminal device. Among them, the terminal device mentioned above can be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, laptop computers, PDAs, vehicle-mounted devices, intelligent voice interaction devices, augmented reality / virtual reality (AR / VR) devices, helmet displays, wearable devices, smart speakers, smart home appliances, aircraft, digital cameras, cameras and other mobile Internet devices (mobile internet device, MID) with network access capabilities. Among them, the server mentioned above can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0038] Please refer to Figure 2, which is a schematic diagram of an application scenario of a speech processing method provided by an embodiment of the present application; as shown in Figure 2, the speech data 21 to be processed can be input into a speech feature extraction model 22, and the speech feature extraction model 22 is used to perform feature encoding on the speech data 21 to be processed to obtain a speech vector matrix of the speech data 21 to be processed. For example, the speech data 21 to be processed is "Well, yes, well, I agree, well". The speech vector matrix of the speech data 21 to be processed is subjected to feature conversion by the speech representation fully connected layer in the speech feature extraction model 22 to obtain the target speech representation information 23 of the speech data 21 to be processed. Among them, the target speech representation information includes the speech content vector and the paralanguage vector corresponding to the speech data 21 to be processed. Furthermore, by obtaining a prompt word 24 about the speech data 21 to be processed (such as a prompt word that is a de-colloquialized prompt word), the target speech representation information 23 (i.e., the speech content vector and the paralanguage vector) and the prompt word 24 can be input into the speech conversion model 25. The speech conversion model 25 performs feature processing on the prompt word 24, such as feature encoding, to obtain a feature vector matrix 26 corresponding to the prompt word 24. The speech conversion model 25 then performs fusion processing on the feature vector matrix 26 corresponding to the prompt word and the target speech representation information 23 of the speech data 21 to be processed to obtain speech fusion features. The speech conversion model 25 can then convert the speech fusion features into text information 27 corresponding to the speech data 21 to be processed, and finally output the text information 27. For example, the de-colloquialized text information 27 is "Yes, I agree."

[0039] Based on this, in the aforementioned application scenario, since the target speech representation information includes the speech content vector and paralinguistic vector corresponding to the speech data to be processed, and the paralinguistic vector is used to assist in identifying the text information corresponding to the speech data to be processed, speech recognition of the speech data to be processed can be combined with both the speech content information of the speech data to be processed, the paralinguistic information in the speech data to be processed, and the text content corresponding to the prompt word. This allows for in-depth speech recognition and understanding of the speech data to be processed, thereby improving speech recognition accuracy.

[0040] The specific embodiments involved in this application are described in detail below.

[0041] Please refer to FIG3 , which is a flowchart of a voice processing method provided in an embodiment of the present application. As shown in FIG3 , the voice processing method can be applied to a computer device, and the voice processing method includes but is not limited to the following steps S101-S103:

[0042] S101 , performing feature extraction on speech data to be processed to obtain target speech representation information of the speech data to be processed.

[0043] Specifically, the method for obtaining the voice data to be processed includes any of the following methods:

[0044] Method 1: Collect voice data in real time from business scenarios as voice data to be processed. In one possible implementation method, voice data can be collected in real time from business scenarios (for example, any type of business scenarios such as conference scenarios, interview scenarios, intelligent dialogue scenarios, game scenarios, etc.), for example, game voice data can be obtained from game scenarios through a recording device (such as a mobile phone's microphone), and conference voice data can be obtained from conference scenarios through a recording device (such as a mobile phone's microphone), and so on; and the collected voice data can be used as voice data to be processed. Optionally, after the voice data is collected in real time, the real-time collected voice data can be preprocessed, and the preprocessed voice data can be used as voice data to be processed. The preprocessing operations here include but are not limited to: voice noise reduction, voice alignment, data normalization and other processing methods. The preprocessed voice data is more accurate. Under this implementation method, voice data can be collected in real time in business scenarios as voice data to be processed. This method of collecting data in real time has good real-time performance and is conducive to timely processing of data.

[0045] Method 2: Acquire the collected voice data from a preset database as the voice data to be processed. In one possible implementation method, the preset database stores voice data collected from various types of business scenarios, and different types of business scenarios are associated with the corresponding voice data. For example, game voice data is stored in a game scenario, and meeting voice data is stored in a meeting scenario. Then, the collected voice data in any business scenario can be directly obtained from the preset database as the voice data to be processed according to business needs. In this implementation method, the voice data to be processed can be directly acquired from the preset database without real-time acquisition, thereby improving the efficiency of data acquisition.

[0046] Among them, the target speech representation information includes the speech content vector and paralanguage vector corresponding to the speech data to be processed. Specifically, (1) the speech content vector can reflect the speech content in the speech data to be processed; for example, the speech content can refer to the specific content contained in the speech frame corresponding to each character in the speech data to be processed. For example, if the speech data to be processed is "Have you eaten?", then the speech data to be processed includes the characters "you", "eat", "meal", "already", and "maybe". Furthermore, the speech content refers to the speech obtained after pronouncing the above characters, such as "you", "eat", "meal", "already", and "maybe". (2) The paralanguage vector is used to assist in identifying the text information corresponding to the speech data to be processed. The paralanguage vector can reflect the paralanguage information in the data to be processed. The so-called paralanguage information can be used to assist in identifying the speech data to be processed; the paralanguage information can include, for example, the volume, timbre, speed, and other information of the pronunciation of each character in the speech data to be processed, or can also include the emotional information corresponding to the pronunciation; it can be seen that paralanguage information is a kind of data used to reflect hidden information such as emotions and intonation in the speech data to be processed. The so-called hidden information refers to information that cannot be directly discovered from the surface of the data.

[0047] It should be noted that the paralinguistic vectors and speech content vectors in the processed speech data are both vectors corresponding to information about the speech modality. Paralinguistic information and speech content are difficult to separate. Directly converting the processed speech data into text data only converts the speech content into text data, but not the paralinguistic information. Furthermore, because the converted text data only contains text, information such as the speaker's emotions when pronouncing the text and the speaker's volume cannot be reflected in the text data. In the embodiments of the present application, since the paralinguistic information in the speech data reflects this information about the speaker through speech, the speech content and paralinguistic information can fully convey information such as the speaker's speech content.

[0048] The specific process of how to extract the target speech representation information of the speech data to be processed is described in detail below.

[0049] Method 1: Extract target speech representation information of the speech data to be processed through prompt words.

[0050] In one embodiment, the process of extracting target speech representation information from speech data to be processed includes the following steps (1)-(3):

[0051] (1) Obtaining feature conversion parameters for performing feature conversion on the prompt word. The feature conversion parameters can be used to perform feature conversion on the prompt word to obtain a feature vector matrix corresponding to the prompt word.

[0052] (2) Feature encoding is performed on the speech data to be processed to obtain a speech vector matrix of the speech data to be processed. By feature encoding the speech data to be processed, each character in the speech data to be processed can be encoded into a speech vector. Feature encoding of the speech data to be processed means embedding the speech signal into a vector space of fixed dimension to obtain a speech vector. Since the speech data to be processed includes multiple characters, it can be encoded into speech vectors corresponding to the multiple characters to obtain a speech vector matrix, thereby obtaining a speech vector matrix corresponding to the speech data to be processed. For example, the dimension of the speech vector obtained by feature encoding a character is m-dimensional, and the speech data to be processed containing n characters is feature encoded to obtain an m*n-dimensional matrix, that is, the speech vector matrix can be an m*n-dimensional matrix.

[0053] In one embodiment, the process of obtaining the speech vector matrix of the speech data to be processed may include the following steps ①-②:

[0054] ① Divide the voice data to be processed to obtain N audio frames (N is a positive integer); for example, when the voice data to be processed is obtained, the voice data to be processed can be framed to obtain N audio frames. Framing can refer to dividing the voice data to be processed according to the frame length to obtain N audio frames. For example, the frame length can take any value, such as 10 milliseconds to 30 milliseconds. Generally speaking, one character in the voice content of the voice signal to be processed can correspond to N audio frames. For example, if the voice signal corresponding to one character in the voice content is 1 second and the frame length is 30 milliseconds, the number of audio frames corresponding to the character is approximately 33.

[0055] ② Perform feature encoding on each audio frame to obtain the speech vector matrix of each audio frame. The feature encoding here is to convert the audio frame from voice data into a feature vector. By converting the voice data into a speech vector matrix, subsequent calculations are facilitated. In an optional implementation, feature encoding can be performed in combination with the position features of each audio frame to obtain the speech vector matrix of the audio frame. For example, based on the division order of N audio frames, the position features of each audio frame can be determined, and the position features are used to indicate the position of the corresponding audio frame in the voice data to be processed; feature encoding is performed on each audio frame to obtain the encoding features of each audio frame; for audio frame i in the N audio frames, the position features of audio frame i and the encoding features of audio frame i are feature spliced ​​to obtain the speech vector matrix of audio frame i. The position features of each audio frame can be used to indicate the position of each audio frame in the voice data to be processed. Since the voice data to be processed is generally divided according to the order of pronunciation in the voice data to be processed, the division order of the audio frames corresponding to the voice data with earlier pronunciation in the voice data to be processed is earlier, and the division order of the audio frames corresponding to the voice data with later pronunciation in the voice data to be processed is later, it is possible to determine the position features of each audio frame based on the division order of the N audio frames. Then, when splicing the coding features of the N audio frames, the feature splicing of each audio frame can be combined with the position features of each audio frame to obtain the voice vector matrix of the N audio frames. By introducing the position features during feature encoding, the splicing can be combined with the position features of the audio frames during subsequent splicing of the coding features of the audio frames, thereby ensuring the accuracy of the order of the characters in the text information and making the speech recognition results more accurate.

[0056] In an embodiment of the present application, if the probability of the currently traversed audio frame being mapped to a certain text in the voice content is the largest and is greater than the probability threshold, it means that the currently traversed audio frame can be mapped to the text in the voice content. If the maximum probability among the probabilities of the currently traversed audio frame being mapped to various texts in the voice content is less than the probability threshold, it means that the currently traversed audio frame cannot be mapped to various texts in the voice content, that is, the voice information in the audio frame is less, and the candidate voice representation information of the currently traversed audio frame can be deleted from the candidate voice representation information of the N audio frames. By deleting the candidate voice representation information of the audio frames whose probabilities of being mapped to various texts in the voice content of the voice data to be processed are all less than the probability threshold from the N audio frames, after the traversal is completed, the target voice representation information of the voice data to be processed can be obtained based on the remaining candidate voice representation information. For example, the remaining candidate voice representation information can be spliced ​​or combined to obtain the target voice representation information of the voice data to be processed.

[0057] In the embodiment of the present application, the probability may refer to the posterior probability output by the speech feature extraction model, for example. When the probability corresponding to an audio frame is less than a probability threshold, the corresponding audio frame may be omitted. By dividing the speech data to be processed into N audio frames, the speech data corresponding to each text contains N audio frames. Therefore, among the N audio frames corresponding to a text, there are audio frames with less speech information. Deleting these audio frames with less speech information has little effect on the overall speech recognition result. Therefore, by deleting the audio frames with less information in the speech data to be processed, the amount of calculation can be reduced and the calculation efficiency can be improved.

[0058] (3) Using feature conversion parameters to perform feature conversion on the speech vector matrix of the speech data to be processed to obtain target speech representation information of the speech data to be processed; the dimension of the feature vector matrix represented by the target speech representation information is the same as the dimension of the feature vector matrix corresponding to the prompt word. The purpose of using feature conversion parameters to perform feature conversion on the speech vector matrix of the speech data to be processed is to make the dimension of the speech vector matrix of the speech data to be processed the same as the dimension of the feature vector matrix corresponding to the prompt word, so that the speech vector matrix of the speech data to be processed and the feature vector matrix corresponding to the prompt word can be spliced ​​later.

[0059] In one embodiment, when performing feature conversion on the speech vector matrix of the speech data to be processed to obtain target speech representation information for the speech data to be processed, meaningless audio frames in the speech data to be processed can also be removed, thereby reducing the amount of computation. Specifically, feature conversion parameters can be used to perform feature conversion on the speech vector matrix of each audio frame to obtain candidate speech representation information for each audio frame. N audio frames can be traversed, and based on the candidate speech representation information of the currently traversed audio frame, the probability of the currently traversed audio frame being mapped to each text in the speech content is predicted; the speech content refers to the content indicated by the speech content vector. If the maximum probability among the probabilities of the currently traversed audio frame being mapped to each text in the speech content is less than a probability threshold, the candidate speech representation information of the currently traversed audio frame is deleted from the candidate speech representation information of each audio frame. After the traversal is completed, the target speech representation information of the speech data to be processed is obtained based on the remaining candidate speech representation information. In this embodiment, by traversing N audio frames of the speech data to be processed, the probability of the N audio frames being mapped to each text in the speech content of the speech data to be processed can be predicted based on the candidate speech representation information of the N audio frames. The probability that each audio frame is mapped to each character in the voice content of the voice data to be processed can be used to reflect the possibility that each audio frame is mapped to each character in the voice content of the voice data to be processed. That is, the greater the probability, the greater the possibility that the characters corresponding to the audio frame are the characters in the voice content of the voice data to be processed, and the smaller the probability, the smaller the possibility that the audio frame is the characters in the voice content of the voice data to be processed. The characters corresponding to the audio frame may refer to the audio frame obtained by pronouncing the characters. If the probability that the audio frame is mapped to each character in the voice content of the voice data to be processed is less than the probability threshold, it means that the audio frame is a meaningless audio frame, that is, the audio frame contains less voice information, and in order to speed up the calculation efficiency, the meaningless audio frame can be omitted.

[0060] From the above steps (1)-(3), it can be seen that in the process of extracting the target speech representation information, since the dimension of the feature vector matrix represented by the target speech representation information is the same as the dimension of the feature vector matrix corresponding to the prompt word, and the target speech representation information includes the speech content vector and the paralanguage vector, the essence of the present application of fusing the target speech representation information with the feature vector matrix corresponding to the prompt word is to fuse the speech content vector, the paralanguage vector and the prompt word. By using the feature conversion parameters to perform feature conversion on the speech vector matrix of the speech data to be processed, the target speech representation information of the speech data to be processed is obtained, which facilitates the subsequent better fusion of the speech representation information and the text information corresponding to the prompt word, thereby improving the accuracy of speech recognition.

[0061] Method 2: Use the trained speech feature extraction model to extract the target speech representation information of the speech data to be processed.

[0062] Specifically, feature extraction can be performed by using a trained speech feature extraction model to improve the efficiency of feature extraction. Among them, the speech feature extraction model may include, but is not limited to, an automatic speech recognition model (Automatic Speech Recognition, ASR), a Transformer model based on a self-attention mechanism, a convolution-enhanced Transformer model, a connectionist temporal classification model based on a neural network (Connectionist temporal classification, CTC), and the like. Optionally, the speech feature extraction model may include: a speech vector matrix extraction layer and a speech representation fully connected layer, the speech vector matrix extraction layer may be used to extract the speech vector matrix of the speech data to be processed, and the speech representation fully connected layer may be used to convert the speech vector matrix of the speech data to be processed into the target speech representation information of the speech data to be processed.

[0063] For example, please refer to Figure 4, which is a schematic diagram of the architecture of a speech feature extraction model provided by an embodiment of the present application, wherein the speech feature extraction model may include a speech vector matrix extraction layer and a speech representation fully connected layer, the speech vector matrix extraction layer may be used to output the speech vector matrix of the speech data to be processed, and the speech representation fully connected layer may be used to output the target speech representation information of the speech data to be processed. Further optionally, the speech vector matrix extraction layer may also include a coding layer, a multi-head attention layer, and a normalization layer. Specifically, the speech data to be processed is input into the speech feature extraction model, and the speech vector matrix extraction layer in the speech feature extraction model processes the speech data to be processed. For example, the coding layer in the speech vector matrix extraction layer may encode each audio frame in the speech data to be processed to obtain coding features. The position features of each audio frame in the speech data to be processed are obtained through the coding layer, and the coding features and position features of each audio frame are spliced ​​to obtain spliced ​​coding features of each audio frame. The spliced ​​coding features are speech vectors with position information. A multi-head attention layer calculates the similarity between the concatenated coded features of each audio frame, determining a similarity score between each pair of audio frames. A normalization layer then normalizes these similarity scores to a range of 0 to 1. For each pair of audio frames, the higher the similarity score, the greater the weight between them; the lower the similarity score, the smaller the weight between them. By combining the weights between the audio frames and performing a weighted summation of each audio frame with the other audio frames, the resulting audio frame contains both the speech information of its own audio frame and the speech information of the other audio frames. This incorporates the contextual speech information of the processed speech data, ensuring that each audio frame contains information about the entire processed speech data. The normalization layer outputs a matrix with the same dimensions as the concatenated coded features, namely the speech vector matrix of the processed speech data. Furthermore, the feature conversion parameters in the speech representation fully-connected layer can be pre-fixed. Therefore, by using these feature conversion parameters in the speech representation fully-connected layer to transform the speech vector matrix of the processed speech data, the target speech representation information for the processed speech data can be output.

[0064] Optionally, the speech feature extraction model may also include a text output layer. By inputting the target speech representation information of each audio frame in the speech data to be processed into the text output layer, the text output layer can predict the probability of mapping the target speech representation information of each audio frame in the speech data to be processed to each text in the speech content of the speech data to be processed, that is, predicting the probability that the target speech representation information of each audio frame is multiple texts, thereby determining the text data corresponding to the speech data to be processed and outputting the text data, such as "Have you eaten?"

[0065] For example, when the speech feature extraction model is an ASR model, the previous fully connected layer of the CTC fully connected layer can be determined as the speech representation fully connected layer. For example, feature conversion parameters can be obtained and used to replace the original parameters in the fully connected layer. The fully connected layer after the replacement parameters is called the speech representation fully connected layer. The purpose of the speech representation fully connected layer is to share parameters with the feature conversion parameters used to convert the prompt word, so as to align the latent space representation distribution of speech and text, thereby achieving subsequent information fusion between the two different modalities of speech and text.

[0066] S102 , obtaining prompt words related to the speech data to be processed, and fusing the speech content vector, the paralanguage vector, and the prompt words to obtain speech fusion features.

[0067] In an embodiment of the present application, a speech fusion feature is obtained by obtaining a prompt word for the speech data to be processed and fusing the speech content vector, paralinguistic vector, and prompt word. The speech fusion feature contains both the prompt word of the speech data to be processed and the speech content and paralinguistic information. Therefore, the text information obtained by subsequently processing the speech fusion feature can reflect both the speech content of the speech data to be processed and the paralinguistic information in the speech data to be processed, as well as the text content corresponding to the prompt word. This enables in-depth speech understanding of the speech data and improves speech recognition accuracy. Since the target speech representation information includes the speech content vector and the paralinguistic vector, fusing the speech content vector, paralinguistic vector, and prompt word is essentially fusing the target speech representation information with the prompt word.

[0068] In one embodiment, prompt words about the voice data to be processed can be obtained in the following manner: based on the display interface, a plurality of preset prompt words are output, and in response to a selection operation on the display interface, the selection operation includes prompt words; based on the selection operation, prompt words about the voice data to be processed are selected from the plurality of preset prompt words output from the display interface. Among them, the prompt words can be used to reflect the voice understanding method of the voice data to be processed. The prompt words may include but are not limited to re-identification prompt words, emotion recognition prompt words, de-colloquialization prompt words, punctuation prompt words, text smoothing prompt words, error correction prompt words, etc. Among them, the emotion categories may include but are not limited to categories such as joy, sadness, fear, anger, surprise and disgust. Error correction prompt words may include professional terms in various fields. By selecting the prompt words about the voice data to be processed, the corresponding text information can be output in combination with the prompt words.

[0069] For example, if the selected prompt word is a re-identification prompt word, the voice data to be processed can be re-identified, and the text information after the re-identification of the voice data to be processed can be output. Or the selected prompt word is an emotion recognition prompt word, and the output text information can include the emotion category corresponding to the voice data to be processed, that is, the emotion category of the speaker when speaking, and the text information corresponding to the voice data to be processed can also be output at the same time. Or the selected prompt word is a de-colloquialization prompt word, and the output text information can be the text information after the voice data to be processed is de-colloquialized. Or the selected prompt word is a punctuation prompt word, and the output text information can be the text information after punctuation is added to the text content corresponding to the voice data to be processed. Or the selected prompt word is a text smoothing prompt word, and the output text information can be the text information after the voice data to be processed is smoothed, that is, the text information is more fluent. Or the selected prompt word is an error correction prompt word, and the output text information can be the text information after the text content corresponding to the voice data to be processed is corrected.

[0070] In an optional implementation, a prompt word corresponding to the voice data to be processed can be selected, so that text information corresponding to the selected prompt word can be output based on the target voice representation information of the voice data to be processed obtained. For example, if the selected prompt word is a re-identification prompt word, text information can be output based on the target voice representation information of the voice data to be processed obtained, and the output text information is as smooth and coherent as possible. For example, if the selected prompt word is an emotion recognition prompt word, the speaker's emotion category can be judged based on the target voice representation information of the voice data to be processed obtained, and the emotion category corresponding to the voice data to be processed is selected from the following multiple emotion categories (such as joy, sadness, fear, anger, surprise and disgust) based on the speaker's emotion category, and the selected emotion category is output. For example, if the selected prompt word is a de-colloquialization prompt word, text information can be obtained based on the target voice representation information of the voice data to be processed obtained, and the colloquial words in the text information are removed, so that the text information is as smooth and easy to read as possible, thereby outputting text information with the colloquial words removed from the text information. For example, if the selected prompt word is a punctuation prompt word, text information can be obtained based on the target speech representation information of the acquired speech data to be processed, and punctuation can be added to the text information to output the text information after punctuation.

[0071] In the embodiment of the present application, by selecting prompt words in combination with the requirements corresponding to the voice data to be processed, voice understanding can be performed while the voice data to be processed is recognized, thereby improving the accuracy of voice recognition and obtaining more accurate text information.

[0072] In one embodiment, the prompt word and the target speech representation information can be fused in the following manner: feature conversion parameters are used to perform feature conversion on the prompt word to obtain a feature vector matrix corresponding to the prompt word; and feature concatenation is performed on the speech content vector, the paralanguage vector, and the feature vector matrix corresponding to the prompt word to obtain speech fusion features.

[0073] The essence of feature concatenation of the speech content vector, paralinguistic vector, and feature vector matrices corresponding to the prompt word is to concatenate the target speech representation information with the feature vector matrices corresponding to the prompt word. The two concatenation methods yield consistent speech fusion features. Since the dimension of the feature vector matrix corresponding to the prompt word is the same as the dimension of the feature vector matrix representing the target speech representation information, the feature vector matrices corresponding to the target speech representation information and the prompt word can be concatenated to yield speech fusion features. The speech fusion features reflect both the speech content of the speech data to be processed, the paralinguistic information in the speech data to be processed, and the text content corresponding to the prompt word. Therefore, subsequent processing of the speech fusion features yields text information that includes both the speech content of the speech data to be processed, the paralinguistic information in the speech data to be processed, and the text content corresponding to the prompt word, thereby improving speech recognition accuracy.

[0074] Optionally, the feature conversion parameters may refer to the parameters of the word embedding layer, which may perform feature conversion on input text data such as prompt words, and convert the text data into a feature vector matrix. The essence of performing feature conversion on prompt words through the parameters in the word embedding layer is feature encoding, that is, encoding the data of the text dimension into a feature vector. Word embedding refers to the process of encoding the divided words into dense vectors, that is, mapping the words to a mathematical space. For example, the parameters of the word embedding layer, i.e., the feature conversion parameters, may be set in advance, and by inputting the prompt words into the word embedding layer, the parameters of the word embedding layer, i.e., the feature conversion parameters, may convert the prompt words into a feature vector matrix. By using the word embedding layer to perform feature conversion on the prompt words, it is convenient to use the feature vector matrix for feature fusion and speech understanding later, thereby improving the accuracy of speech recognition.

[0075] In an embodiment of the present application, by converting the speech data to be processed into speech representation information having a dimension equal to the dimension of the feature vector matrix corresponding to the prompt word, the speech representation information and the text semantics can be aligned, that is, the latent space representation of the speech of the speech data to be processed is consistent with the latent space representation of the text corresponding to the speech data to be processed, thereby achieving the fusion of speech representation information and text semantic information, thereby improving the accuracy of speech recognition.

[0076] S103: Perform voice conversion processing on the voice fusion features to obtain text information corresponding to the voice data to be processed.

[0077] In the embodiment of the present application, since the speech fusion feature includes the text content corresponding to the prompt word, the speech content and paralinguistic information of the speech data to be processed, the speech fusion feature is subjected to speech conversion processing to obtain text information corresponding to the speech data to be processed. The text information corresponding to the speech data to be processed can include the speech content of the speech data to be processed, reflect the paralinguistic information in the speech data to be processed, and also reflect the text content corresponding to the prompt word, that is, deep speech understanding can be achieved, thereby improving the accuracy of speech recognition.

[0078] Optionally, a trained speech conversion model can be used to perform speech conversion processing on the speech fusion features to obtain text information corresponding to the speech data to be processed. Examples of speech conversion models include, but are not limited to, large language models (LLMs), generative general language models (GLMs), open source conversational language models (MOSSs), and generative pre-training models (GPTs).

[0079] For example, the process of using the trained speech conversion model to perform speech conversion processing on the speech fusion feature to obtain the text information corresponding to the speech data to be processed can be as follows: dividing the speech fusion feature into multiple feature units, predicting the next feature unit based on the input feature unit sequence of the speech conversion model, adding the input feature unit and the predicted feature unit to the feature unit sequence, and continuing to predict the next feature unit until multiple feature units corresponding to the speech fusion feature are predicted. Whenever a feature unit is predicted, the feature unit and the feature unit before the feature unit are added to the feature unit sequence, and the next feature unit of the feature unit is predicted. Among them, the feature unit can refer to the basic unit of text, for example, it can refer to a Chinese character, or a word, etc. Optionally, the BPE (Byte pair encoding) method can be used to divide the word into smaller units, for example, substrings or characters as basic units. In an optional implementation method, the basic component units of the text can be trained based on the text corpus as feature units.

[0080] In a specific implementation, since the feature vector matrix represented by the target speech representation information is a multidimensional matrix, and the feature vector matrix corresponding to the prompt word is a multidimensional matrix, and the dimensions of the two matrices are the same, the speech fusion feature obtained by fusion is also a multidimensional matrix, and the dimensions of the three matrices are the same. When the multidimensional matrix corresponding to the speech fusion feature is input into the trained speech conversion model, a column of the matrix in the multidimensional matrix corresponding to the speech fusion feature can be input as a feature unit. A column of the matrix can contain the speech corresponding to a text in the speech data to be processed, so that the next column of feature units can be predicted. When predicting the next column of feature units, the previously predicted feature units are input into the speech conversion model as a feature unit sequence, thereby realizing the prediction of the text information corresponding to the speech fusion feature.

[0081] In the embodiment of the present application, since speech recognition belongs to a perceptual task, by combining the LLM model, the cognitive ability of speech data can be improved, thereby combining speech and text modal information to improve the ability to understand speech data and enhance the performance on more speech and semantic related tasks. Since the LLM model can handle any form of text tasks, the use of speech and semantic related tasks can be expanded on the technical solution of the present application. For example, more modalities such as visual information can be further integrated on the basis of speech and text. For example, visual information can be converted into text representation information, and further combined with text information and speech representation information to input into the LLM model for processing, thereby enriching the speech understanding content and improving the accuracy of speech recognition.

[0082] In an embodiment of the present application, feature extraction is performed on the speech data to be processed to obtain target speech representation information of the speech data to be processed. Prompt words about the speech data to be processed are obtained, and the target speech representation information and the prompt words are fused to obtain speech fusion features; the speech fusion features are subjected to speech conversion processing to obtain text information corresponding to the speech data to be processed. Since the target speech representation information includes the speech content vector and the paralanguage vector corresponding to the speech data to be processed, and the paralanguage vector is used to assist in identifying the text information corresponding to the speech data to be processed, when performing speech recognition on the speech data to be processed, it is possible to combine the speech content information of the speech data to be processed, the paralanguage information in the speech data to be processed, and the text content corresponding to the prompt words for speech recognition. Since more comprehensive and rich speech representation information is used for speech recognition processing, the present application can achieve in-depth speech recognition and understanding of the speech data to be processed, thereby improving the accuracy of speech recognition.

[0083] Further, please refer to Figure 5, which is a flowchart of a method for training a speech feature extraction model provided in an embodiment of the present application. The method can be applied to a computer device; as shown in Figure 5, the method includes but is not limited to the following steps:

[0084] S201 , obtaining sample speech data, and performing feature extraction on the sample speech data using a speech feature extraction model to obtain sample speech representation information of the sample speech data.

[0085] In the embodiments of the present application, the sample speech data may be pre-acquired, for example, by downloading from a speech data storage website, uploading from a terminal device, or obtaining from locally stored speech data. To increase the amount of training data, the sample speech data may be further processed by cropping, rotating, tuning, adding noise, and so on, thereby expanding the amount of sample speech data. By using a large amount of sample audio data as training data for the speech feature extraction model, the accuracy of the speech feature extraction model can be improved.

[0086] In embodiments of the present application, for example, a speech feature extraction model may be used to perform feature encoding on sample speech data to obtain a speech vector matrix of the speech data to be processed. Feature conversion parameters in the speech feature extraction model may then be used to perform feature conversion on the speech vector matrix of the speech data to obtain sample speech representation information of the sample speech data. The sample speech representation information of the sample speech data may include a sample speech content vector and a sample paralanguage vector corresponding to the sample speech data.

[0087] In one embodiment, the speech feature extraction model may include a speech vector matrix extraction layer and a speech representation fully connected layer. The speech vector matrix extraction layer and the speech representation fully connected layer may then be combined to determine the sample speech representation information of the sample speech data. For example, the sample speech data may be input into the speech vector matrix extraction layer, feature encoded on the sample speech data by the speech vector matrix extraction layer to obtain a speech vector matrix for the sample speech data, and the speech vector matrix for the sample speech data may be input into the speech representation fully connected layer. The speech vector matrix for the sample speech data may be feature transformed using the feature transformation parameters in the speech representation fully connected layer to obtain the sample speech representation information for the sample speech data.

[0088] Further optionally, the speech vector matrix extraction layer may also include an encoding layer, a multi-head attention layer, and a normalization layer. Specifically, the sample speech data is input into the speech feature extraction model, and the encoding layer encodes each audio frame in the sample speech data to obtain encoding features, obtains the position features of each audio frame in the sample speech data, and splices the encoding features and position features of each audio frame to obtain the spliced ​​encoding features of each audio frame. The spliced ​​encoding features are speech vectors with position information. The multi-head attention layer calculates the similarity between the spliced ​​encoding features of each audio frame in the sample speech data, determines the similarity score between each two audio frames, and the normalization layer normalizes the similarity score to a range of 0 to 1. For each two audio frames in the sample speech data, the higher the similarity score between the two audio frames, the greater the weight between the two audio frames, and the lower the similarity score between the two audio frames, the smaller the weight between the two audio frames. By combining the weights between each audio frame and other audio frames in the sample voice data and performing weighted summation, the obtained audio frame contains the voice information of its own audio frame and the voice information of other audio frames in the sample voice data, thereby introducing the contextual voice information in the sample voice data, and thus making each audio frame in the sample voice data contain the information of the entire sample voice data.

[0089] S202: Obtain a sample speech representation label of the sample speech data.

[0090] The sample speech representation labels can be pre-acquired user speech representation labels that reflect the true value of the sample speech data. By obtaining the sample speech representation labels for the sample speech data, the speech feature extraction model can be adjusted during subsequent training by combining the sample speech representation labels with the sample speech representation information output by the speech feature extraction model.

[0091] S203 , training a speech feature extraction model based on the sample speech representation label and the sample speech representation information to obtain a trained speech feature extraction model.

[0092] Here, the sample speech representation information refers to the model output value of the speech feature extraction model, and the sample speech representation label refers to the true value of the sample. The purpose of training the speech feature extraction model is to make the model output value and the true value of the sample as consistent as possible. If the model output value and the true value of the sample are inconsistent, the model parameters in the speech feature extraction model are further adjusted to make the model output value consistent with the true value of the sample. When the model output value and the true value of the sample are consistent, the speech feature extraction model at this time is regarded as the trained speech feature extraction model.

[0093] Training the speech feature extraction model involves comparing the difference between the sample speech representation label and the sample speech representation information, and determining a loss function for the speech feature extraction model based on the difference between the sample speech representation label and the sample speech representation information. The difference between the sample speech representation label and the sample speech representation information can be calculated based on a similarity calculation method, i.e., the greater the similarity between the sample speech representation label and the sample speech representation information, the smaller the difference between the sample speech representation label and the sample speech representation information. The smaller the similarity between the sample speech representation label and the sample speech representation information, the greater the difference between the sample speech representation label and the sample speech representation information. If the difference between the sample speech representation label and the sample speech representation information is greater than a difference threshold, the loss function of the speech feature extraction model is greater than a first loss threshold, and the model parameters of the speech feature extraction model are further adjusted to reduce the loss function of the speech feature extraction model. When the difference between the sample speech representation label and the sample speech representation information is less than or equal to the difference threshold, the loss function of the speech feature extraction model is less than or equal to the first loss threshold, and the current speech feature extraction model can be saved as the trained speech feature extraction model.

[0094] Optionally, when the number of iterative training of the speech feature extraction model is greater than a threshold number or the speech feature extraction model reaches a convergence condition, the model parameters in the speech feature extraction model can be stopped from being adjusted to obtain a trained speech feature extraction model.

[0095] In an optional implementation, the speech feature extraction model can also be trained in the following way: using the speech feature extraction model to extract features from the sample speech data to obtain sample speech representation information of the sample speech data; using the speech feature extraction model to predict the sample text data corresponding to the sample speech representation information of the sample speech data; obtaining the sample text label of the sample speech data, and training the speech feature extraction model based on the sample text label and the sample text data to obtain the trained speech feature extraction model.

[0096] Among them, by predicting the text data of the sample speech representation information of the sample speech data, it is equivalent to converting the sample speech representation information into information in text mode, so as to train the speech feature extraction model based on the difference between the two texts. The sample text label can refer to the real text of the sample speech data, and the text data of the sample speech representation information can refer to the text predicted by the speech feature extraction model, that is, the text output by the model. By comparing the difference between the sample text label and the text data of the sample speech representation information, the speech feature extraction model is trained based on the difference between the sample text label and the text data of the sample speech representation information. The difference between the sample text label and the text data of the sample speech representation information can be calculated based on the text similarity calculation method, which is not limited in the embodiment of the present application. By converting the sample speech representation information into information in text mode for comparison, text comparison can be performed to determine the difference between the texts, so as to adjust the speech feature extraction model.

[0097] In one embodiment, when the speech feature extraction model includes a speech vector matrix extraction layer and a speech representation fully connected layer, the speech feature extraction model can be trained in the following manner:

[0098] The parameters of the speech vector matrix extraction layer are adjusted based on the sample speech representation label and the sample speech representation information to obtain a trained speech feature extraction model.

[0099] In the embodiment of the present application, the parameters in the speech representation fully connected layer are fixed when training the speech feature extraction model. That is, the parameters in the speech representation fully connected layer are fixed to the feature conversion parameters of the word embedding layer. By fixing the parameters in the speech representation fully connected layer, the speech feature extraction model can perform speech recognition while its latent space representation is consistent with the speech conversion model, so that the target speech representation information of the speech data to be processed output by the speech feature extraction model can be input into the speech conversion model.

[0100] In an optional implementation, when training the speech feature extraction model, meaningless frames in the sample speech data can be deleted separately, thereby reducing the amount of calculation and improving the training efficiency of the speech feature extraction model.

[0101] In an embodiment of the present application, the target speech representation information of the speech data to be processed is obtained by removing meaningless frames from the speech data to be processed, thereby reducing the amount of computation. By weight-sharing the parameters of the speech representation fully connected layer and the word embedding layer in the speech conversion model, the speech representation information output by the speech representation fully connected layer can be aligned with the encoding features of the prompt word output by the word embedding layer, thereby achieving fusion of information between the two modalities. The parameters of the speech representation fully connected layer come from the word embedding layer in the speech conversion model.

[0102] In the embodiments of the present application, by training a speech feature extraction model, the trained speech feature extraction model can be used to extract features from the speech data to be processed, thereby obtaining target speech representation information for the speech data to be processed, thereby improving speech data processing efficiency. Because a large amount of sample speech data is used to train the speech feature extraction model, the accuracy of the speech feature extraction model can be improved.

[0103] Alternatively, please refer to Figure 6, which is a flowchart of a method for training a speech conversion model provided in an embodiment of the present application. The method can be applied to a computer device; as shown in Figure 6, the method includes but is not limited to the following steps:

[0104] S301: Obtain sample speech representation information and sample prompt words corresponding to sample speech data.

[0105] In an embodiment of the present application, the sample speech representation information can be obtained by extracting features from the sample speech data. For example, the sample speech data can be divided into N audio frames (N is a positive integer), the N audio frames in the sample speech data are feature-encoded to obtain a speech vector matrix of the N audio frames, and the feature conversion parameters in the speech feature extraction model are used to perform feature conversion on the speech vector matrix of the N audio frames to obtain candidate speech representation information of the N audio frames. The sample speech representation information is obtained by deleting meaningless audio frames from the candidate speech representation information of the N audio frames in the sample audio data. In an embodiment of the present application, a variety of sample prompt words may be included, and each sample prompt word may correspond to a different scenario. When preparing training data, corresponding sample prompt words can be selected based on actual needs. For example, sample prompt words may include prompt words such as text smoothing, de-colloquialization, error correction, re-identification, emotion recognition, and punctuation. For example, error correction prompt words may include professional terms in various fields. De-colloquialization prompt words may include some colloquial words. Text smoothing prompt words may include words composed of repeated characters, and so on.

[0106] It is understandable that since the prompt words have no fixed format, for example, the prompt words in scenarios such as de-colloquialization and emotion recognition are text, while the prompt words in scenarios with punctuation are punctuation marks, or the prompt words in re-identification scenarios are prompt information indicating the re-identification, therefore, in actual usage scenarios, you only need to keep the selected prompt words consistent with those during model training.

[0107] S302: Using a speech conversion model, a sample speech content vector, a sample paralanguage vector, and a sample prompt word are fused to obtain a sample speech fusion feature.

[0108] Here, since the sample speech representation information of the sample speech data includes the sample speech content vector and sample paralanguage vector corresponding to the sample speech data, fusing the sample speech content vector, the sample paralanguage vector, and the sample prompt word essentially involves fusing the sample speech representation information and the sample prompt word. Since the speech representation information is represented by a feature vector matrix, while the prompt word is represented by text, before fusing the two, the prompt word can be converted into a feature vector matrix to facilitate feature fusion, such as feature concatenation.

[0109] S303: Perform speech conversion processing on the sample speech fusion features using a speech conversion model to obtain text information corresponding to the sample speech data.

[0110] For example, the process of using a speech conversion model to perform speech conversion processing on the sample speech fusion features to obtain text information corresponding to the sample speech data can be as follows: dividing the sample speech fusion features into multiple feature units, predicting the next feature unit based on the feature unit sequence input to the speech conversion model, adding the input feature unit and the predicted feature unit to the feature unit sequence, and continuing to predict the next feature unit until multiple feature units corresponding to the sample speech fusion features are predicted.

[0111] In an optional implementation, since the feature vector matrix represented by the sample speech representation information is a multidimensional matrix, and the feature vector matrix corresponding to the sample prompt word is a multidimensional matrix, and the dimensions of the two matrices are the same, the sample speech fusion feature obtained by fusion is also a multidimensional matrix, and the dimensions of the three matrices are the same. When the multidimensional matrix corresponding to the sample speech fusion feature is input into the trained speech conversion model, a column of the matrix in the multidimensional matrix corresponding to the sample speech fusion feature can be input as a feature unit. A column of the matrix can represent the speech data corresponding to a text in the sample speech data, so that the next column of feature units can be predicted. When predicting the next column of feature units, the previously predicted feature units are input into the speech conversion model as a feature unit sequence, thereby realizing the prediction of the text information corresponding to the sample speech fusion feature.

[0112] Optionally, the speech conversion model can use a currently open source large-scale language model, such as the widely used model based on the Transformer structure, which adopts an autoregressive form, that is, based on the input token (feature unit) sequence, predict the next token, and then based on the input and the predicted token, predict the next token, and so on to predict the text information corresponding to the speech data to be processed.

[0113] S304: Obtain a sample text label corresponding to the sample speech data, and train a speech conversion model based on the sample text label and text information corresponding to the sample speech data to obtain a trained speech conversion model.

[0114] In the embodiments of the present application, the sample text label may refer to the actual text label of the sample speech data, and the text information corresponding to the sample speech data may refer to the model output value based on the output of the speech conversion model. The purpose of training the speech conversion model is to make the sample text label and the text information corresponding to the sample speech data as consistent as possible. When the sample text label and the text information corresponding to the sample speech data are consistent, the speech conversion model at this time can be determined as the trained speech conversion model. The text information corresponding to the sample text label and the sample speech data can be calculated using a text similarity calculation method.

[0115] Training the speech conversion model based on the sample text labels and the text information corresponding to the sample speech data refers to determining a loss function for the speech conversion model based on the difference between the sample text labels and the text information corresponding to the sample speech data. When the loss function of the speech conversion model is greater than a second loss threshold, further adjusting model parameters of the speech conversion model to reduce the loss function of the speech conversion model. When the loss function of the speech conversion model is less than or equal to the second loss threshold, determining the current speech conversion model as the trained speech conversion model.

[0116] Optionally, the process of training the speech conversion model is actually a process of adjusting the parameters in the speech conversion model. Since the speech conversion model includes a large number of parameters, adjusting all the parameters in the speech conversion model during training takes a lot of time and reduces the training efficiency. Therefore, some parameters in the speech conversion model can be adjusted to achieve the purpose of improving the efficiency of the speech conversion model.

[0117] Please refer to Figure 7, which is a schematic diagram of parameter adjustment in a speech conversion model provided in an embodiment of the present application. The left part of Figure 7 is the pre-trained model parameter W (i.e., pre-trained weight) in the speech conversion model (such as the LLM model). A branch is added next to the pre-trained model structure. This branch contains two structures A and B. The two parameters A and B are initialized to Gaussian distribution and 0, respectively. At the beginning of training, the additional parameters are 0, the input dimension of A and the output dimension of B are the same as the input and output dimensions of the original model, respectively, while the output dimension of A and the input dimension of B are a value much smaller than the input and output dimensions of the original model. This can greatly reduce the parameters to be trained in the LLM model. When training the LLM model, only the parameters of A and B are updated, and the pre-trained model parameters W are fixed. AB is merged with the original model parameter matrix W, so that no additional calculations are introduced in the inference. For different downstream tasks, only A and B need to be retrained based on the pre-trained model. After training the new parameters, they are merged with the old ones. Using re-parameterization, this approach achieves fine-tuning results for the new task without increasing model inference time, thereby improving model training efficiency. Because the LLM model itself has a large number of parameters, fine-tuning only requires adding a small number of parameters for training, which improves training efficiency.

[0118] When training a speech conversion model, the key to predicting its output data from its input data lies in preparing the training set. For a given task (such as text smoothing, de-colloquialization, or error correction), the following training set can be prepared: Sample speech data is passed through the aforementioned speech feature conversion model to remove meaningless frames, resulting in speech representation information for the sample speech data. This information, along with the corresponding prompt word, is then fed into the speech conversion model, which outputs text information for the corresponding task.

[0119] Example 1 is a text smoothing scenario:

[0120] Input: Have you eaten yet? (The input is the speech representation information of the speech modality), plus the corresponding prompt words such as "you you, me me" and other repeated words;

[0121] Output: Have you eaten?

[0122] Example 2 is a de-colloquialization scenario:

[0123] Input: Yeah, yes, yeah, I agree, yeah (the input is the speech representation information of the speech modality), plus corresponding prompt words such as "hmm, yes, yes" and other colloquial words;

[0124] Output: Yes, I agree.

[0125] Example 3 is an error correction scenario:

[0126] Input: The original system's performance is not very good (the input is speech representation information in the speech mode), plus corresponding prompt words such as "far field" and other professional terms for the corresponding scenario;

[0127] Output: The far field performance of this system is not very good.

[0128] Example 4 is about adding punctuation marks:

[0129] Input: This is a long sentence. Does it need punctuation? I think it does (the input is the speech representation information of the speech modality), plus the corresponding prompt words such as "comma, question mark, period" and other punctuation marks;

[0130] Output: This is a long sentence. Does it need punctuation? I think it does.

[0131] In an embodiment of the present application, a speech feature extraction model such as an ASR model aligns speech representation information to the text semantic space of the LLM model by reusing the word embedding mechanism of the LLM model, so that the speech representation information can be directly used as the input of the LLM model. Since the speech representation information contains text content and paralinguistic information, the LLM model can make full use of the information of the speech modality and further improve the ability of speech recognition and understanding. The LLM model can make full use of other information in the speech data besides the text content, thereby enhancing the speech recognition and speech understanding capabilities. By setting different prompt words during speech recognition, the speech recognition capabilities of de-colloquialization, hot word replacement, emotion recognition, and punctuation are directly improved in the speech recognition link, forming an end-to-end model to output the corresponding text information. Instead of recognizing it as text first in the speech recognition link and then giving the text to the LLM model for processing. The technical solution of the present application can also be extended to information of other modalities, such as visual information, which can be used as the input of the LLM model to further enhance the speech recognition and speech understanding capabilities of the model.

[0132] In the embodiments of this application, by training a speech conversion model, the trained speech conversion model can be used to perform speech conversion processing on speech fusion features, obtaining text information corresponding to the speech data to be processed, thereby improving speech data processing efficiency. Because a large amount of sample speech data is used to train the speech conversion model, the accuracy of the speech conversion model can be improved.

[0133] The method of the embodiment of the present application is introduced above, and the device of the embodiment of the present application is introduced below.

[0134] Referring to FIG8 , FIG8 is a schematic diagram of the structure of a speech processing device provided in an embodiment of the present application. The speech processing device can be deployed on a computer device; the speech processing device can be used to perform the corresponding steps of the speech processing method provided in an embodiment of the present application. The speech processing device 80 includes:

[0135] A feature extraction unit 801 is configured to extract features from the speech data to be processed to obtain target speech representation information of the speech data to be processed; the target speech representation information includes a speech content vector and a paralanguage vector corresponding to the speech data to be processed, the paralanguage vector being used to assist in identifying text information corresponding to the speech data to be processed;

[0136] An information fusion unit 802 is configured to obtain a prompt word related to the speech data to be processed, and fuse the speech content vector, the paralanguage vector, and the prompt word to obtain a speech fusion feature;

[0137] The speech conversion unit 803 is configured to perform speech conversion processing on the speech fusion feature to obtain text information corresponding to the speech data to be processed.

[0138] Optionally, the information fusion unit 802 is specifically configured to:

[0139] The feature conversion parameters are used to perform feature conversion on the prompt word to obtain the feature vector matrix corresponding to the prompt word;

[0140] The speech content vector, the paralanguage vector, and the feature vector matrix corresponding to the prompt word are concatenated to obtain the speech fusion feature.

[0141] Optionally, the feature extraction unit 801 is specifically configured to:

[0142] Obtaining feature conversion parameters for performing feature conversion on the prompt word;

[0143] Performing feature encoding on the speech data to be processed to obtain a speech vector matrix of the speech data to be processed;

[0144] The feature conversion parameters are used to perform feature conversion on the speech vector matrix of the speech data to be processed to obtain target speech representation information of the speech data to be processed; the dimension of the feature vector matrix represented by the target speech representation information is the same as the dimension of the feature vector matrix corresponding to the prompt word.

[0145] Optionally, the feature extraction unit 801 is specifically configured to:

[0146] Dividing the speech data to be processed to obtain N audio frames;

[0147] Perform feature encoding on each audio frame to obtain a speech vector matrix for each audio frame;

[0148] Performing feature conversion on the speech vector matrix of each audio frame using the feature conversion parameter to obtain candidate speech representation information of each audio frame;

[0149] Traversing the N audio frames, and based on the candidate speech representation information of the currently traversed audio frame, predicting the probability that the currently traversed audio frame is mapped to each word in the speech content; the speech content refers to the content indicated by the speech content vector;

[0150] If the maximum probability of the currently traversed audio frame being mapped to each character in the speech content is less than a probability threshold, deleting the candidate speech representation information of the currently traversed audio frame from the candidate speech representation information of each audio frame;

[0151] After the traversal is completed, the target speech representation information of the speech data to be processed is obtained based on the remaining candidate speech representation information.

[0152] Optionally, the feature extraction unit 801 is further configured to:

[0153] Determining a position feature of each audio frame based on a division order of the N audio frames, where the position feature is used to indicate a position of the corresponding audio frame in the speech data to be processed;

[0154] The feature extraction unit 801 is specifically configured to:

[0155] Perform feature encoding on each audio frame to obtain encoding features of N audio frames;

[0156] For audio frame i among N audio frames, feature splicing processing is performed on the position feature of audio frame i and the encoding feature of audio frame i to obtain a speech vector matrix of audio frame i, where i is a positive integer and 1≤i≤N.

[0157] Optionally, the text information corresponding to the speech data to be processed is obtained through a trained speech conversion model. The speech processing device 80 further includes: a first training unit 804, wherein the first training unit 804 is configured to:

[0158] Obtaining sample speech representation information and sample prompt words corresponding to the sample speech data; the sample speech representation information includes a sample speech content vector and a sample paralanguage vector corresponding to the sample speech data;

[0159] The speech content vector of the sample, the paralanguage vector of the sample and the prompt word of the sample are fused using a speech conversion model to obtain a fusion feature of the sample speech;

[0160] Using the speech conversion model to perform speech conversion processing on the sample speech fusion feature to obtain text information corresponding to the sample speech data;

[0161] A sample text label corresponding to the sample speech data is obtained, and the speech conversion model is trained based on the sample text label and text information corresponding to the sample speech data to obtain the trained speech conversion model.

[0162] Optionally, the target speech representation information of the speech data to be processed is obtained through a trained speech feature extraction model. The speech processing device 80 further includes: a second training unit 805, wherein the second training unit 805 is configured to:

[0163] Acquire sample speech data, and perform feature extraction on the sample speech data using a speech feature extraction model to obtain sample speech representation information of the sample speech data;

[0164] A sample speech representation label of the sample speech data is obtained, and the speech feature extraction model is trained based on the sample speech representation label and the sample speech representation information to obtain the trained speech feature extraction model.

[0165] Optionally, the speech feature extraction model includes a speech vector matrix extraction layer and a speech representation fully connected layer; the second training unit 805 is specifically used to:

[0166] Performing feature encoding on the sample speech data through the speech vector matrix extraction layer to obtain a speech vector matrix of the sample speech data;

[0167] Performing feature conversion on the speech vector matrix of the sample speech data using the feature conversion parameters in the speech representation fully connected layer to obtain sample speech representation information of the sample speech data;

[0168] The parameters of the speech vector matrix extraction layer are adjusted based on the sample speech representation label and the sample speech representation information to obtain the trained speech feature extraction model.

[0169] It should be noted that for the contents not mentioned in the embodiment corresponding to FIG8 , reference can be made to the description of the method embodiment, which will not be repeated here.

[0170] In an embodiment of the present application, feature extraction is performed on the speech data to be processed to obtain target speech representation information of the speech data to be processed. Prompt words about the speech data to be processed are obtained, and the target speech representation information and the prompt words are fused to obtain speech fusion features; the speech fusion features are subjected to speech conversion processing to obtain text information corresponding to the speech data to be processed. Since the target speech representation information includes the speech content vector and the paralanguage vector corresponding to the speech data to be processed, and the paralanguage vector is used to assist in identifying the text information corresponding to the speech data to be processed, when performing speech recognition on the speech data to be processed, it is possible to combine the speech content information of the speech data to be processed, the paralanguage information in the speech data to be processed, and the text content corresponding to the prompt words for speech recognition. Since more comprehensive and rich speech representation information is used for speech recognition processing, the present application can achieve in-depth speech recognition and understanding of the speech data to be processed, thereby improving the accuracy of speech recognition.

[0171] Referring to Figure 9 , Figure 9 is a schematic diagram illustrating the structure of a computer device provided in an embodiment of the present application. As shown in Figure 9 , the computer device 90 may include a processor 901, a memory 902, and a network interface 903. The processor 901 is connected to the memory 902 and the network interface 903. For example, the processor 901 may be connected to the memory 902 and the network interface 903 via a bus. The computer device may be a terminal device or a server.

[0172] The processor 901 is configured to support the speech processing device in executing the corresponding functions in the above-mentioned speech processing method. The processor 901 can be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The above-mentioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0173] Memory 902 is used to store program instructions and data. Memory 902 may include volatile memory (VM), such as random access memory (RAM); non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the aforementioned types of memory.

[0174] The network interface 903 is used to provide network communication functions.

[0175] The processor 901 may call the program code to perform the following operations:

[0176] Performing feature extraction on the speech data to be processed to obtain target speech representation information of the speech data to be processed; the target speech representation information includes a speech content vector and a paralanguage vector corresponding to the speech data to be processed, the paralanguage vector being used to assist in identifying text information corresponding to the speech data to be processed;

[0177] Obtaining a prompt word for the speech data to be processed, and fusing the speech content vector, the paralanguage vector, and the prompt word to obtain a speech fusion feature;

[0178] The speech fusion feature is subjected to speech conversion processing to obtain text information corresponding to the speech data to be processed.

[0179] It should be understood that the computer device 90 described in the embodiments of the present application can execute the methods described in the embodiments corresponding to Figures 3, 5, and 6 above, and can also execute the speech processing device described in the embodiment corresponding to Figure 8 above, and thus will not be repeated here. In addition, the description of the beneficial effects of using the same methods will not be repeated here.

[0180] In an embodiment of the present application, feature extraction is performed on the speech data to be processed to obtain target speech representation information of the speech data to be processed. Prompt words about the speech data to be processed are obtained, and the target speech representation information and the prompt words are fused to obtain speech fusion features; the speech fusion features are subjected to speech conversion processing to obtain text information corresponding to the speech data to be processed. Since the target speech representation information includes the speech content vector and the paralanguage vector corresponding to the speech data to be processed, and the paralanguage vector is used to assist in identifying the text information corresponding to the speech data to be processed, when performing speech recognition on the speech data to be processed, it is possible to combine the speech content information of the speech data to be processed, the paralanguage information in the speech data to be processed, and the text content corresponding to the prompt words for speech recognition. Since more comprehensive and rich speech representation information is used for speech recognition processing, the present application can achieve in-depth speech recognition and understanding of the speech data to be processed, thereby improving the accuracy of speech recognition.

[0181] Optionally, when the program instructions are executed by the processor, other steps of the method in the above embodiment may also be implemented, which will not be repeated here.

[0182] The present application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, causes the computer to perform the method of the aforementioned embodiment. The computer may be part of the aforementioned computer device. As an example, the program instructions may be deployed on a single computer device for execution, or deployed on multiple computer devices located at a single location for execution, or executed on multiple computer devices distributed at multiple locations and interconnected via a communication network. Multiple computer devices distributed at multiple locations and interconnected via a communication network may constitute a blockchain network.

[0183] The present application also provides a computer program product or computer program, which includes computer instructions that, when executed by a processor, implement some or all of the steps in the above-described method. For example, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps performed in the above-described method embodiments.

[0184] Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0185] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A speech processing method, characterized in that: The method comprises: Performing feature extraction on the speech data to be processed to obtain target speech representation information of the speech data to be processed; the target speech representation information includes a speech content vector and a paralanguage vector corresponding to the speech data to be processed, and the paralanguage vector is used to assist in identifying text information corresponding to the speech data to be processed; Acquire prompt words about the speech data to be processed, and fuse the speech content vector, the paralanguage vector and the prompt words to obtain speech fusion features; The speech fusion feature is subjected to speech conversion processing to obtain text information corresponding to the speech data to be processed.

2. The method according to claim 1, characterized in that The fusing the speech content vector, the paralanguage vector and the prompt word to obtain a speech fusion feature includes: Performing feature conversion on the prompt word using a feature conversion parameter to obtain a feature vector matrix corresponding to the prompt word; The speech content vector, the paralanguage vector, and the feature vector matrix corresponding to the prompt word are concatenated to obtain the speech fusion feature.

3. The method according to claim 1 or 2, characterized in that: The feature extraction of the speech data to be processed to obtain target speech representation information of the speech data to be processed includes: Acquire feature conversion parameters for performing feature conversion on the prompt word; Performing feature encoding on the speech data to be processed to obtain a speech vector matrix of the speech data to be processed; The feature conversion parameters are used to perform feature conversion on the speech vector matrix of the speech data to be processed to obtain target speech representation information of the speech data to be processed; the dimension of the feature vector matrix represented by the target speech representation information is the same as the dimension of the feature vector matrix corresponding to the prompt word.

4. The method according to any one of claims 1 to 3, characterized in that: The step of performing feature encoding on the speech data to be processed to obtain a speech vector matrix of the speech data to be processed includes: Dividing the speech data to be processed to obtain N audio frames, where N is a positive integer; Performing feature encoding on N audio frames to obtain speech vector matrices of the N audio frames; The step of using the feature conversion parameters to perform feature conversion on the speech vector matrix of the speech data to be processed to obtain target speech representation information of the speech data to be processed includes: Performing feature conversion on the speech vector matrices of the N audio frames using the feature conversion parameters to obtain candidate speech representation information of the N audio frames; Traversing the N audio frames, and predicting the probability of the currently traversed audio frame being mapped to each word in the speech content based on the candidate speech representation information of the currently traversed audio frame; the speech content refers to the content indicated by the speech content vector; If the maximum probability of the currently traversed audio frame being mapped to each character in the speech content is less than a probability threshold, deleting the candidate speech representation information of the currently traversed audio frame from the candidate speech representation information of each audio frame; After the traversal is completed, the target speech representation information of the speech data to be processed is obtained based on the remaining candidate speech representation information.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determine, based on the division order of the N audio frames, a position feature of each audio frame, wherein the position feature is used to indicate a position of the corresponding audio frame in the speech data to be processed; The feature encoding of each audio frame to obtain a speech vector matrix of each audio frame includes: Performing feature encoding on each audio frame to obtain encoding features of each audio frame; For audio frame i among N audio frames, feature splicing processing is performed on the position features of the audio frame i and the encoding features of the audio frame i to obtain a speech vector matrix of the audio frame i, where i is a positive integer and 1≤i≤N.

6. The method according to any one of claims 1 to 5, characterized in that: The text information corresponding to the speech data to be processed is obtained through a trained speech conversion model, and the training method of the trained speech conversion model includes: Acquire sample speech representation information and sample prompt words corresponding to the sample speech data; the sample speech representation information includes a sample speech content vector and a sample paralanguage vector corresponding to the sample speech data; Using a speech conversion model to fuse the sample speech content vector, the sample paralanguage vector and the sample prompt word to obtain a sample speech fusion feature; Using the speech conversion model to perform speech conversion processing on the sample speech fusion features to obtain text information corresponding to the sample speech data; A sample text label corresponding to the sample voice data is obtained, and the voice conversion model is trained based on the sample text label and text information corresponding to the sample voice data to obtain the trained voice conversion model.

7. The method according to any one of claims 1 to 6, characterized in that: The target speech representation information of the speech data to be processed is obtained by a trained speech feature extraction model, and the training method of the trained speech feature extraction model includes: Acquire sample speech data, and use a speech feature extraction model to perform feature extraction on the sample speech data to obtain sample speech representation information of the sample speech data; A sample speech representation label of the sample speech data is obtained, and the speech feature extraction model is trained based on the sample speech representation label and the sample speech representation information to obtain the trained speech feature extraction model.

8. The method according to any one of claims 1 to 7, characterized in that: The speech feature extraction model includes a speech vector matrix extraction layer and a speech representation fully connected layer; The step of extracting features from the sample speech data using the speech feature extraction model to obtain sample speech representation information of the sample speech data includes: Performing feature encoding on the sample speech data through the speech vector matrix extraction layer to obtain a speech vector matrix of the sample speech data; Performing feature conversion on the speech vector matrix of the sample speech data by using the feature conversion parameters in the speech representation fully connected layer to obtain sample speech representation information of the sample speech data; The step of training the speech feature extraction model based on the sample speech representation label and the sample speech representation information to obtain the trained speech feature extraction model includes: The parameters of the speech vector matrix extraction layer are adjusted based on the sample speech representation label and the sample speech representation information to obtain the trained speech feature extraction model.

9. A speech processing device, characterized in that: The device comprises: A feature extraction unit, configured to extract features from the speech data to be processed to obtain target speech representation information of the speech data to be processed; the target speech representation information includes a speech content vector and a paralanguage vector corresponding to the speech data to be processed, and the paralanguage vector is used to assist in identifying text information corresponding to the speech data to be processed; An information fusion unit, used for obtaining a prompt word about the speech data to be processed, and fusing the speech content vector, the paralanguage vector and the prompt word to obtain a speech fusion feature; The speech conversion unit is used to perform speech conversion processing on the speech fusion feature to obtain text information corresponding to the speech data to be processed.

10. A computer device, characterized in that: The method comprises a processor, a memory and a network interface, wherein the processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions so that the computer device executes the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is performed.

Images