Phonetic-lexical parallel data-based Chinese text error detection method and device and related equipment

Abstract

The application discloses a kind of audio-text parallel data Chinese text error character detection method, device and related equipment, the character modality feature of the text to be detected in audio-text parallel data is extracted, the audio modality feature of the audio parallel with the text to be detected is extracted, the audio modality feature and character modality feature are fused, the real text corresponding to audio is determined based on fusion feature, compare real text and text to be detected, to obtain error character detection result.The application considers the character modality feature of text to be detected in error character detection, further fuses the audio modality feature of parallel audio, so that the prediction result is more accurate, on this basis, by comparing real text and text to be detected, determine error character detection result, greatly improve the accuracy of error character detection.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and more specifically, to a method, apparatus, device, and storage medium for detecting typos in audio-related text. Background Technology

[0002] With the development of information technology and the continuous emergence of media platforms, an era characterized by diversified information transmission forms and multiple transmission sources has arrived. In addition to various programs launched by conventional media organizations, more and more ordinary individuals are joining the ranks of self-media, including a large number of self-media personalities who appear on camera to disseminate professional knowledge or spread social hot topics. Beyond the public media sphere, due to the continuous development of the social economy and the increasing cross-regional exchanges across the country and even the world, the demand for remote communication methods such as video conferencing is growing. Real-time subtitles generated during online meetings have played a crucial role in improving communication efficiency.

[0003] Due to carelessness by subtitle creators or the immaturity of related subtitle generation technology, numerous video subtitles on short video platforms contain typos; typos are also frequently seen in subtitles generated in real-time by video conferencing software. This phenomenon poses a serious threat to the accuracy of information transmission and the breadth of cultural dissemination. Relying solely on manual proofreading and correction of these texts would consume a significant amount of manpower and time.

[0004] In today's era of booming artificial intelligence, especially thanks to advancements in natural language processing technology, various text error detection and correction systems have emerged to help people efficiently check and correct textual errors. Taking video subtitles as an example, existing error correction systems generally identify video subtitles, perform error correction processing based on the context of the subtitle text information, locate potential errors, and return the results to the user.

[0005] Existing error correction methods only utilize plain text information for error correction, resulting in low accuracy in typo detection. Summary of the Invention

[0006] In view of the above problems, this application is proposed to provide a method, apparatus, device, and storage medium for detecting typos in audio-related text, so as to improve the accuracy of typo detection in audio-text parallel data. The specific solution is as follows:

[0007] Firstly, a method for detecting typos in parallel audio-text data is provided, including:

[0008] Obtain input audio-text parallel data, wherein the audio-text parallel data includes the text to be detected and the audio that matches the text to be detected;

[0009] Extract the audio modal features of the audio, and extract the text modal features of the text to be detected;

[0010] The audio modal features and the text modal features are fused to obtain fused features;

[0011] The real text corresponding to the audio is determined based on the fusion features;

[0012] By comparing the real text and the text to be detected, the typo detection results in the text to be detected are obtained.

[0013] Secondly, a device for detecting misspellings in parallel audio-text data is provided, including:

[0014] The data acquisition unit is used to acquire input audio-text parallel data, which includes the text to be detected and the audio that matches the text to be detected;

[0015] The feature extraction unit is used to extract the audio modal features of the audio and the text modal features of the text to be detected.

[0016] The feature fusion unit is used to fuse the audio modal features and the text modal features to obtain fused features;

[0017] The real text determination unit is used to determine the real text corresponding to the audio based on the fusion features;

[0018] The misspelling detection unit is used to compare the real text and the text to be detected to obtain the misspelling detection result in the text to be detected.

[0019] Thirdly, a device for detecting misspellings in parallel audio-text data is provided, including: a memory and a processor;

[0020] The memory is used to store programs;

[0021] The processor is used to execute the program to implement the various steps of the above-described method for detecting misspelled words in parallel audio-text data.

[0022] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the various steps of the above-described method for detecting misspelled words in parallel audio-text data.

[0023] By employing the aforementioned technical solution, this application extracts the text modal features of the text to be detected and the audio modal features of the audio parallel to the text from the input audio-text parallel data. The audio modal features and text modal features are then fused, and the corresponding real text is determined based on the fused features. The real text and the text to be detected are then compared to obtain the misspelling detection result. Therefore, this application, when detecting misspellings in the text to be detected, not only considers the text modal features of the text to be detected but also further fuses the audio modal features of the audio parallel to the text. That is, it fully utilizes the audio modal features of the audio corresponding to the text to assist in the prediction of the real text. Thus, this application can use audio modal features to assist in the prediction of the real text, making the prediction result more accurate compared to simply predicting the real text based on the context of the text to be detected. Based on this, by comparing the real text and the text to be detected, the misspelling detection result is determined, greatly improving the accuracy of misspelling detection. Attached Figure Description

[0024] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0025] Figure 1 A flowchart illustrating a method for detecting misspelled words in parallel audio-text data provided in this application embodiment;

[0026] Figure 2 This example illustrates the process of marking typos in the text to be detected.

[0027] Figure 3 This example illustrates the structure of an audio text recognition model.

[0028] Figure 4 An example is a schematic diagram of the structure of an audio processing module;

[0029] Figure 5 This example illustrates the structure of a text processing module;

[0030] Figure 6 An example is a schematic diagram of the structure of a multimodal fusion module;

[0031] Figure 7 A schematic diagram illustrating the processing flow of a multimodal fusion module is provided.

[0032] Figure 8 A schematic diagram of a text misspelling detection device for parallel audio-text data provided in this application embodiment;

[0033] Figure 9 A schematic diagram of the structure of the audio-text parallel data Chinese text misspelling detection device provided in the embodiments of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] This application provides a method for detecting typos in parallel audio-text data, which can be applied to the task of detecting typos in parallel audio-text data. The parallel audio-text data includes the text to be detected and the audio that is parallel to the text to be detected. The parallel audio matches the text to be detected. For example, if the text to be detected is a multimedia subtitle text, then the parallel audio is the audio data in the multimedia data that matches the subtitle.

[0036] The proposed solution can be implemented based on a terminal with data processing capabilities, such as a mobile phone, computer, server, or cloud platform.

[0037] Next, combined Figure 1 The method for detecting misspelled words in parallel audio-text data described in this application may include the following steps:

[0038] Step S100: Obtain the input audio-text parallel data, wherein the audio-text parallel data includes the text to be detected and the audio that matches the text to be detected.

[0039] Specifically, the text to be detected in the audio-text parallel data is the text information that needs to be checked for typos. The text to be detected can be text information entered by the user, or text information identified from images, video screenshots, etc. The text information contained in the text to be detected can include Chinese characters and non-Chinese characters, such as English letters, special symbols, numbers, etc.

[0040] The audio in the audio-text parallel data is the audio information that matches the text to be detected. It can be user-recorded or extracted from multimedia data. For example, when the text to be detected is subtitle text, the audio can be multimedia audio that matches the subtitle.

[0041] For example Figure 2 The text to be detected is subtitle text: The story of this little water sprite was absolutely amazing!

[0042] At the same time, audio data corresponding to the subtitle text is extracted from the multimedia data matched with the subtitles.

[0043] It can be known that the characters "lai" and "miao" in the subtitle "The story of that little water otter just now is really wonderful!" are misspelled, and the correct ones should be "獭" and "妙".

[0044] Step S110: Extract the audio modality features of the audio, and extract the text modality features of the text to be detected.

[0045] Specifically, when extracting text modality features, it can be extracted by using a set text feature extraction algorithm, or by using a pre-trained natural language processing model.

[0046] Therefore, in order to assist in detecting misspelled words in the text to be detected, the audio modality features of the input audio related to the text to be detected are extracted in this step.

[0047] Among them, various different algorithms can be used when extracting audio modality features. For example, the audio modality features of the input audio can be extracted through a pre-trained neural network model, etc.

[0048] Step S120: Fuse the audio modality features and the text modality features to obtain fused features.

[0049] Specifically, the audio modality features and the text modality features respectively describe relevant information from the two perspectives of audio and text. In order to more accurately predict the true text corresponding to the audio, the audio modality features and the text modality features are fused in this step, and the information of the obtained fused features is more abundant and the expression ability is stronger.

[0050] Step S130: Determine the true text corresponding to the audio based on the fused features.

[0051] Specifically, after obtaining the fused features in the above step, the true text corresponding to the audio can be predicted based on the fused features. In this step, a pre-trained neural network model can be used to predict the true text.

[0052] The true text predicted through this step is the correct text corresponding to the text to be detected identified in this application.

[0053] Step S140: Compare the true text and the text to be detected to obtain the misspelled word detection result in the text to be detected.

[0054] Specifically, in this step, the true text can be used as a benchmark to compare the text to be detected and the true text, determine whether the text to be detected contains misspelled words, and the specific content of the misspelled words included, so as to obtain the misspelled word detection result in the text to be detected.

[0055] Exemplarily, in this step, it is possible to match whether there are characters in the text to be detected that are inconsistent with the real text. If so, the inconsistent characters in the text to be detected are regarded as typos in the text to be detected.

[0056] The method for detecting typos in text in the parallel audio-visual data provided by the embodiments of the present application extracts the text modality features of the text to be detected and the audio modality features of the audio parallel to the text to be detected from the input parallel audio-visual data, fuses the audio modality features and the text modality features, and determines the real text corresponding to the audio based on the fused features. By comparing the real text and the text to be detected, the typo detection result is obtained. It can be seen that when detecting typos in the text to be detected, the present application further fuses the audio modality features of the audio parallel to the text to be detected on the basis of considering the text modality features of the text to be detected, that is, fully utilizes the audio modality features of the audio corresponding to the text to assist in predicting the real text. Therefore, the present application can use the audio modality features to assist in predicting the real text, making the prediction result more accurate compared to simply predicting the real text based on the context of the text to be detected. On this basis, by comparing the real text and the text to be detected, the typo detection result is determined, greatly improving the accuracy of typo detection.

[0057] Optionally, after obtaining the typo detection result in the above step S140, if it is confirmed that the text to be detected contains typos, the position of the typos in the text to be detected can be further determined, and then this position is output.

[0058] Reference Figure 2 , for the typos "lai" and "miao" identified in the text to be detected, they are located at the 7th and 13th characters in the text to be detected respectively.

[0059] Furthermore, in order to more intuitively display the typos in the text to be detected, the typos can also be marked in the text to be detected according to the positions of the detected typos. For example, the marking is carried out in the form of a rectangular box.

[0060] Of course, the marking form of the typos is not limited to the rectangular box marking, and other various marking methods can also be used, such as highlighting, underlining, etc.

[0061] In some embodiments of the present application, the process of fusing the audio modality features and the text modality features in the above step S120 to obtain the fused features is described.

[0062] Optionally, the audio modal features and text modal features extracted in step S110 can be in vector form. The vector dimensions of the audio modal features and text modal features can be the same or different. Based on this, when performing feature fusion in this step, the two vector features can be fused to obtain fused features.

[0063] When performing vector fusion, various fusion methods can be used. This embodiment provides a gated fusion method that fuses audio modal features and text modal features in vector form to obtain fused features.

[0064] By employing a gated fusion method, using audio modal features as the gate, a portion of the text modal features are extracted to obtain fused features. In other words, from the perspective of audio modal features, the most important part of the text modal features is extracted as the feature representation of audio modality and character modality fusion.

[0065] Optionally, this application provides several different gating fusion methods, such as: bitwise multiplication gating fusion, bitwise addition or division gating fusion, etc. For ease of description, the following embodiments only use bitwise multiplication gating fusion as an example.

[0066] Furthermore, to avoid the loss of global features at the text language level, in this embodiment, the above-mentioned fusion features can be added to the text modal features to obtain residual fusion features, which are used as the final fusion features.

[0067] To enhance the richness of audio modal feature representation, before feature fusion in step S120, representation shift and nonlinear transformation processing can be added to the audio modal features to obtain processed audio modal features, which can then be fused with text modal features in step S120.

[0068] In some embodiments of this application, a preprocessing process for audio-text parallel data can be added before extracting the audio modal features of the audio in step S110 and the text modal features of the text to be detected, so that the features extracted later are more accurate and reliable.

[0069] This preprocessing procedure may include the following operations:

[0070] 1. Perform format checks on the input audio and text parallel data. If the data does not conform to the standard format, the check can be rejected and the user will be notified.

[0071] 2. Perform noise reduction processing on the audio in the input audio-text parallel data.

[0072] 3. Segment the text to be detected in the input audio-text parallel data to obtain a sequence of text segments. Segment the audio according to the length of each text segment in the text segment sequence to obtain audio segments that match the length and semantics of each text segment.

[0073] Specifically, to ensure that the lengths of text and audio segments do not exceed a set threshold, the text and audio to be detected can be segmented. When segmenting the text, entire sentences can be segmented based on punctuation marks. Further, the audio can be initially segmented based on sound wave interval information. Then, a pre-trained speech length determination model is used to identify the number of characters in each audio segment. The identified number is compared with the length of the corresponding text segment. If they match, the matching of the current text and audio segments is complete. If they do not match, the length of the audio segment is adjusted according to the length of the text segment to ensure that the number of characters in the adjusted audio segment is equal to the length of the text segment.

[0074] In some embodiments of this application, steps S110-S130 described in the foregoing embodiments can be obtained by processing a pre-trained audio-text recognition model.

[0075] For audio-text recognition models, they can be configured as follows: extract the audio modal features of the input audio, extract the text modal features of the input text to be detected, fuse the audio modal features and the text modal features, and predict the internal state representation of the real text corresponding to the input audio based on the fused features.

[0076] The input to the audio text recognition model can include audio-text parallel data, that is, the text information to be detected, and the audio parallel to the text to be detected.

[0077] In this embodiment, by pre-training the audio-text recognition model, the powerful learning ability of the neural network model can be utilized to extract the audio modal features of the input audio and the text modal features of the text to be detected. Based on this, the real text is predicted after fusion.

[0078] Next, combined Figure 3 As shown, this embodiment provides an optional component structure for the audio text recognition model.

[0079] An audio-to-text recognition model can include an audio processing module, a text processing module, a multimodal fusion module, and an output module. Among them:

[0080] The audio processing module is used to extract the audio modal features of the input audio.

[0081] Specifically, the input to the audio processing module can be audio features, such as Fbank features. The audio processing module encodes the audio features to obtain the audio modal features of the input audio.

[0082] The text processing module is used to extract the text modal features of the text to be detected.

[0083] The multimodal fusion module is used to fuse the audio modal features and the text modal features to obtain fused features.

[0084] The output module is used to determine the real text corresponding to the audio based on the fusion features.

[0085] The output module can be trained using the MLM (Masked Language Model) method. Based on the fusion features output by the multimodal fusion module, the real text corresponding to the audio is predicted.

[0086] Next, each of the above modules will be explained in detail.

[0087] 1. Audio processing module

[0088] This embodiment describes an optional structure for the audio processing module, such as... Figure 4 As shown, it may include:

[0089] The audio standardization module is used to standardize the input audio to obtain the processed audio.

[0090] The audio standardization module can take as input audio segments that have been segmented.

[0091] Considering the potential complexity of the input audio, the varying sizes and distributions of the segmented audio clips, and inherent issues such as aliasing, high-order harmonic distortion, and high-frequency distortion, the audio processing module can first perform standardization on the input audio. For example, this can involve using digital filters to pre-emphasize the input audio, performing frame segmentation and windowing to adapt to short-time analysis techniques, endpoint detection, randomly adding Gaussian noise, and adjusting pitch and time axis. This standardization process makes the processed audio signal more uniform and smoother, improving audio data quality and facilitating the extraction of higher-quality audio modal features.

[0092] The audio feature extraction module is used to extract audio modal features from the processed audio.

[0093] like Figure 4For example, an audio feature extraction module can be composed of several audio feature recognition blocks connected in series. Each audio feature recognition block can include several convolutional layers, Transformer layers, and nonlinear layers. The size and number of convolutional kernels in the convolutional layers within different audio feature recognition blocks can vary to enrich the perspectives of audio modal feature extraction, thereby resulting in a richer and more accurate final audio modal feature representation.

[0094] The Transformer layer can employ an audio pre-trained model structure, trained using phoneme prediction or character prediction as the pre-training task. By using such a task as the pre-training task, the Transformer layer gains the ability to extract features related to the real text corresponding to the input audio; that is, the audio feature extraction module gains the ability to extract audio modal features related to the real text corresponding to the input audio.

[0095] The linear transformation module is used to perform a linear transformation on the dimensions of the audio modal features to output audio modal features with the same dimensions as the text modal features.

[0096] Specifically, the number of channels of the audio modal features extracted by the audio feature extraction module may not be directly matched with the dimension of the text modal features extracted by the text processing module. Therefore, it is necessary to perform a linear transformation on the dimension of the audio modal features through the linear transformation module to output audio modal features with the same dimension as the text modal features.

[0097] 2. Text Processing Module

[0098] This embodiment describes an optional structural configuration for the text processing module, such as... Figure 5 As shown, it may include:

[0099] The text preprocessing module is used to edit the text to be detected to a set length by padding with specified characters, and to determine the feature representation of the edited text.

[0100] Specifically, to standardize the length of different texts to be detected, this embodiment uses a text preprocessing module to edit the text to be detected to a set length using padding. For texts shorter than the set length, a set padding character, such as [PAD], can be added to the end of the text to supplement it to the set length. For texts longer than the set length, the set length can be truncated from the first character as a single edited text. If the remaining length still exceeds the set length, the truncating operation is repeated. If the remaining length does not exceed the set length, the remaining portion is used as another edited text.

[0101] For each edited text to be detected, a pre-trained tokenizer can be used to encode the text into a feature representation that the model can recognize. Specifically, the edited text to be detected is segmented into words, and each word is encoded to obtain the token feature representation corresponding to the word.

[0102] Among them, the pre-trained tokenizer can adopt pre-trained model structures such as BERT tokenizer.

[0103] The text modality feature extraction module is used to encode the feature representation of the text to be detected to obtain the text modality features of the text to be detected.

[0104] Specifically, the text modality feature extraction module can use a pre-trained model (such as BERT, Transformer, etc.) to encode the feature representation of the text to be detected after the text preprocessing module, so as to obtain the text modality features of the text to be detected.

[0105] 3. Multimodal fusion module

[0106] This embodiment describes an optional structural composition of the multimodal fusion module, such as... Figure 6 As shown, it may include: a feature editing module, a gating fusion module, and a residual connection module.

[0107] The processing flow of each module is combined Figure 7 Explanation:

[0108] The feature editing module is used to perform representation shifts and nonlinear transformations on audio modal features to obtain processed audio modal features.

[0109] To enhance the representation of audio modal features, representation shift and nonlinear transformation can be applied. Representation shift involves adding a learnable bias parameter to each position of the audio modal feature. Nonlinear transformation uses nonlinear function layers, such as ReLU, sigmoid, or tanh layers, to nonlinearly transform the represented audio modal features to a relatively small range near 0. For example, the range of the transformed sigmoid function is (0, 1), and the range of the transformed tanh function is (-1, 1).

[0110] The gated fusion module is used to fuse the processed audio modal features and the text modal features using a gated fusion method to obtain fused features.

[0111] Specifically, in this embodiment, a gating fusion module is designed to perform bitwise multiplication, bitwise addition, or bitwise division to fuse the processed audio modal features and text modal features to obtain fused features.

[0112] Figure 7 Taking the bitwise multiplication gated fusion method as an example, by using the bitwise multiplication gated fusion method, the audio modal features are used as the gate to extract some features from the text modal features to obtain the fused features. That is, from the perspective of the audio modal features, the most important part of the text modal features is extracted as the feature representation of the audio modality and character modality fusion.

[0113] After the feature editing module processes the audio modal features, the audio modal features, compared to the text modal features, have additional representation offsets and nonlinear transformations, mapping the audio modal features to a relatively small range near 0, such as the sigmoid function having a range of (0,1). The range and distribution of the text modal features remain unchanged. To put it figuratively, each position in the audio modal features after editing is like a faucet (fully open corresponds to the upper bound of the nonlinear function's range, fully closed corresponds to the lower bound), used to control the information at the corresponding position in the text modal features. The more the faucet is open at this position in the audio modal features, the more information is retained in the corresponding position in the text modal features, and vice versa. Clearly, this bitwise multiplication yields the text modal feature portion controlled by the degree of retention from the audio perspective; that is, from the perspective of the audio modal features, the most important part of the text modal features is extracted as the feature representation for the fusion of audio and text modal features.

[0114] The residual connection module is used to add the fused feature to the text modal feature to obtain the residual fused feature, which is used as the final fused feature.

[0115] Furthermore, to avoid the loss of global features at the text language level, this embodiment can also add the above-mentioned fused features to the text modal features through the residual connection module to obtain residual fused features, which are used as the final fused features.

[0116] In some embodiments of this application, in order to further improve the accuracy of typo detection, after step S140, comparing the real text and the text to be detected to obtain the typo detection results in the text to be detected, a post-processing operation for typo verification can be further added.

[0117] In this embodiment, the post-processing for misspelling verification can be performed from the perspective of sentence semantic fluency, specifically including:

[0118] S1. Delete the typos identified in the text to be detected to obtain the edited text with the typos removed.

[0119] S2. Using a pre-trained language model, calculate the perplexity of the text to be detected and the edited text after deleting typos.

[0120] Specifically, perplexity is an indicator that measures the semantic fluency of a sentence; the more fluent a sentence is semantically, the lower its perplexity.

[0121] A language model is a probabilistic model used to calculate the probability that a sentence is a semantically correct sentence. Perplexity is a sentence-length-normalized metric related to the probability that a language model predicts a sentence. For a perfectly correct sentence, the lower the perplexity of the language model, the better the language model. Conversely, if a very good language model has been selected, then for a given sentence, if the perplexity of the language model is very low, it means that the sentence is highly likely to be correct.

[0122] In this step, to verify whether the previously identified typos are truly typos, the perplexity of the text to be detected and the edited text after deleting the typos are calculated respectively.

[0123] S3. If the perplexity of the edited text after deleting the typos is less than the perplexity of the text to be detected, and the absolute value of the difference between the two is greater than a set threshold, then the typos are taken as the final typo detection result; otherwise, the typos are removed from the final typo detection result.

[0124] Understandably, if the perplexity of the edited text after deleting the typo is less than that of the text to be detected, and the absolute value of the difference between the two is greater than a set threshold, it means that the semantics of the edited text after deleting the typo is more fluent than the semantics of the identified text before deletion. In other words, the deleted typo was indeed a typo, and therefore the deleted typo can be added to the final typo detection result. Conversely, if the perplexity is less than a set threshold, it indicates that the typo identified in the previous steps is a pseudo-typo, and it can be removed from the final typo detection result, meaning it will not be identified as a typo in the end.

[0125] In this embodiment, the accuracy of misspelling recognition is further improved by adding a post-processing operation that performs secondary verification of misspellings from the perspective of sentence semantic fluency.

[0126] The following describes the typo detection device for parallel audio-text data provided in the embodiments of this application. The typo detection device for parallel audio-text data described below can be referred to in correspondence with the typo detection method for parallel audio-text data described above.

[0127] See Figure 8 , Figure 8This is a schematic diagram of the structure of a text misspelling detection device for parallel audio-text data disclosed in an embodiment of this application.

[0128] like Figure 8 As shown, the device may include:

[0129] The data acquisition unit 11 is used to acquire input audio-text parallel data, wherein the audio-text parallel data includes the text to be detected and the audio that matches the text to be detected;

[0130] The feature extraction unit 12 is used to extract the audio modal features of the audio and the text modal features of the text to be detected.

[0131] The feature fusion unit 13 is used to fuse the audio modal features and the text modal features to obtain fused features;

[0132] The real text determination unit 14 is used to determine the real text corresponding to the audio based on the fusion features;

[0133] The misspelling detection unit 15 is used to compare the real text and the text to be detected to obtain the misspelling detection result in the text to be detected.

[0134] Optionally, if the audio modal features and the text modal features are both in vector form, then the process by which the feature fusion unit fuses the audio modal features and the text modal features to obtain fused features may include:

[0135] A gated fusion method is used to fuse audio modal features and text modal features in vector form to obtain fused features.

[0136] Optionally, the above gating fusion methods may include gating fusion methods of bitwise multiplication, bitwise addition or division, etc.

[0137] Optionally, after fusing the vector-based audio modal features and text modal features using a gated fusion method, the aforementioned feature fusion unit may further include:

[0138] The fused features are added to the text modal features to obtain residual fused features, which are used as the final fused features.

[0139] Optionally, before fusing the vector-based audio modal features and text modal features using a gated fusion method, the aforementioned feature fusion unit may further include:

[0140] The audio modal features are subjected to representation shift and nonlinear transformation to obtain the processed audio modal features.

[0141] Optionally, the apparatus of this application may further include:

[0142] A data preprocessing unit is configured to segment the text to be detected into a text segment sequence before the feature extraction unit; and to segment the audio according to the length of each text segment in the text segment sequence to obtain audio segments that match the length and semantics of each text segment. Based on this, the process by which the feature extraction unit extracts the audio modal features of the audio and the text modal features of the text to be detected may include:

[0143] For each combination of text segment and matching audio segment, extract the audio modal features of the audio segment in the combination, and extract the text modal features of the text segment in the combination.

[0144] Optionally, the processing of the above-mentioned feature extraction unit 12, feature fusion unit 13 and real text determination unit 14 can be implemented by a pre-trained audio text recognition model. The audio text recognition model is configured to extract the audio modal features of the input audio, extract the text modal features of the text to be detected, fuse the audio modal features and the text modal features, and predict the internal state representation of the real text corresponding to the input audio based on the fused features.

[0145] The audio text recognition model may include: an audio processing module, a text processing module, a multimodal fusion module, and an output module;

[0146] The audio processing module is used to extract the audio modal features of the input audio.

[0147] The text processing module is used to extract the text modal features of the text to be detected;

[0148] A multimodal fusion module is used to fuse the audio modal features and the text modal features to obtain fused features;

[0149] The output module is used to determine the real text corresponding to the input audio based on the fusion features.

[0150] Optionally, the above-mentioned multimodal fusion module may further include:

[0151] The feature editing module is used to perform representation shift and nonlinear transformation on the audio modal features to obtain the processed audio modal features;

[0152] The gated fusion module is used to fuse the processed audio modal features and the text modal features using a gated fusion method to obtain fused features;

[0153] The residual connection module is used to add the fused feature to the text modal feature to obtain the residual fused feature, which is used as the final fused feature.

[0154] Optionally, the above audio processing module may further include:

[0155] The audio standardization module is used to standardize the input audio to obtain the processed audio.

[0156] An audio feature extraction module is used to extract audio modal features from the processed audio.

[0157] The linear transformation module is used to perform a linear transformation on the dimensions of the audio modal features to output audio modal features with the same dimensions as the text modal features.

[0158] Optionally, the above text processing module may further include:

[0159] The text preprocessing module is used to edit the text to be detected to a set length by filling in set characters, and to determine the feature representation of the edited text to be detected.

[0160] The text modality feature extraction module is used to encode the feature representation of the text to be detected to obtain the text modality features of the text to be detected.

[0161] Optionally, the process by which the above-mentioned misspelling determination unit compares the real text and the text to be detected to obtain the misspelling detection result in the text to be detected may include:

[0162] The system checks whether there are any characters in the text to be detected that are inconsistent with the actual text. If such characters are found, they are treated as typos.

[0163] Optionally, the apparatus of this application may further include: a misspelling verification unit, configured to: after comparing the real text and the text to be detected to obtain a misspelling detection result in the text to be detected, delete the misspellings identified in the text to be detected to obtain an edited text with the misspellings removed; use a pre-trained language model to calculate the perplexity of the text to be detected and the edited text with the misspellings removed respectively; if the perplexity of the edited text with the misspellings removed is less than the perplexity of the text to be detected, and the absolute value of the difference between the two is greater than a set threshold, then the misspelling is taken as the final misspelling detection result; otherwise, the misspelling is removed from the final misspelling detection result.

[0164] Optionally, the apparatus of this application may further include: a misspelling location determination unit, used to: after comparing the real text and the text to be detected to obtain the misspelling detection result in the text to be detected, determine the location of the misspelling in the text to be detected and output the location.

[0165] Optionally, the apparatus of this application may further include: a misspelling location marking unit, used to mark the misspelling in the text to be detected according to the misspelling location.

[0166] The text misspelling detection device for audio-text parallel data provided in this application embodiment can be applied to text misspelling detection devices for audio-text parallel data, such as terminals: mobile phones, computers, etc. Optionally, Figure 9 The diagram shows the hardware structure of a text misspelling detection device in parallel audio-text data. (Refer to...) Figure 9 The hardware structure of the device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0167] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0168] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0169] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0170] The memory stores a program, which the processor can call. The program is used for:

[0171] Obtain input audio-text parallel data, wherein the audio-text parallel data includes the text to be detected and the audio that matches the text to be detected;

[0172] Extract the audio modal features of the audio, and extract the text modal features of the text to be detected;

[0173] The audio modal features and the text modal features are fused to obtain fused features;

[0174] The real text corresponding to the audio is determined based on the fusion features;

[0175] By comparing the real text and the text to be detected, the typo detection results in the text to be detected are obtained.

[0176] Optionally, the refined and extended functions of the program can be found in the description above.

[0177] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0178] Obtain input audio-text parallel data, wherein the audio-text parallel data includes the text to be detected and the audio that matches the text to be detected;

[0179] Extract the audio modal features of the audio, and extract the text modal features of the text to be detected;

[0180] The audio modal features and the text modal features are fused to obtain fused features;

[0181] The real text corresponding to the audio is determined based on the fusion features;

[0182] By comparing the real text and the text to be detected, the typo detection results in the text to be detected are obtained.

[0183] Optionally, the refined and extended functions of the program can be found in the description above.

[0184] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0185] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0186] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting misspellings in parallel audio-text data, characterized in that, include: Obtain input audio-text parallel data, wherein the audio-text parallel data includes the text to be detected and the audio that matches the text to be detected; Extract the audio modal features of the audio, and extract the text modal features of the text to be detected; The audio modal features and the text modal features are fused to obtain fused features; The real text corresponding to the audio is determined based on the fusion features; By comparing the real text and the text to be detected, the typo detection results in the text to be detected are obtained; Wherein, the audio modal features and the text modal features are both in vector form, the process of fusing the audio modal features and the text modal features to obtain the fused features includes: A gated fusion method is used to fuse audio modal features and text modal features in vector form to obtain fused features.

2. The method according to claim 1, characterized in that, After fusing the vector-based audio modal features and text modal features using a gated fusion method, the process also includes: The fused features are added to the text modal features to obtain residual fused features, which are used as the final fused features.

3. The method according to claim 1, characterized in that, Before fusing the vector-based audio modal features and text modal features using a gated fusion method, the following steps are also included: The audio modal features are subjected to representation shift and nonlinear transformation to obtain the processed audio modal features.

4. The method according to claim 1, characterized in that, Before extracting the audio modal features of the audio and the text modal features of the text to be detected, the method further includes: The text to be detected is segmented to obtain a sequence of text fragments; The audio is segmented according to the length of each text segment in the text segment sequence to obtain audio segments that match the length and semantics of each text segment; The process of extracting the audio modal features of the audio and the text modal features of the text to be detected includes: For each combination of text segment and matching audio segment, extract the audio modal features of the audio segment in the combination, and extract the text modal features of the text segment in the combination.

5. The method according to claim 1, characterized in that, The process of extracting the audio modal features and text modal features, fusing the two, and determining the real text corresponding to the audio based on the fused features is obtained through a pre-trained audio-text recognition model. The audio-text recognition model is configured to extract audio modal features of the input audio, extract text modal features of the input text to be detected, fuse the audio modal features and text modal features, and predict the internal state representation of the real text corresponding to the input audio based on the fused features.

6. The method according to claim 5, characterized in that, The audio-text recognition model includes: an audio processing module, a text processing module, a multimodal fusion module, and an output module; The audio processing module is used to extract the audio modal features of the input audio. The text processing module is used to extract the text modal features of the text to be detected; A multimodal fusion module is used to fuse the audio modal features and the text modal features to obtain fused features; The output module is used to determine the real text corresponding to the input audio based on the fusion features.

7. The method according to claim 6, characterized in that, The multimodal fusion module includes: The feature editing module is used to perform representation shift and nonlinear transformation on the audio modal features to obtain the processed audio modal features; The gated fusion module is used to fuse the processed audio modal features and the text modal features using a gated fusion method to obtain fused features; The residual connection module is used to add the fused feature to the text modal feature to obtain the residual fused feature, which is used as the final fused feature.

8. The method according to claim 6, characterized in that, The audio processing module includes: The audio standardization module is used to standardize the input audio to obtain the processed audio. An audio feature extraction module is used to extract audio modal features from the processed audio. The linear transformation module is used to perform a linear transformation on the dimensions of the audio modal features to output audio modal features with the same dimensions as the text modal features.

9. The method according to claim 6, characterized in that, The text processing module includes: The text preprocessing module is used to edit the text to be detected to a set length by filling in set characters, and to determine the feature representation of the edited text to be detected. The text modality feature extraction module is used to encode the feature representation of the text to be detected to obtain the text modality features of the text to be detected.

10. The method according to any one of claims 1-9, characterized in that, The process of comparing the real text and the text to be detected to obtain the typo detection results in the text to be detected includes: The system checks whether there are any characters in the text to be detected that are inconsistent with the actual text. If such characters are found, they are treated as typos.

11. The method according to any one of claims 1-9, characterized in that, After comparing the real text and the text to be detected to obtain the typo detection results in the text to be detected, the method further includes: The identified typos in the text to be detected are deleted to obtain the edited text after the typos are removed; Using a pre-trained language model, the perplexity of the text to be detected and the edited text after deleting typos are calculated respectively; If the perplexity of the edited text after deleting the typos is less than the perplexity of the text to be detected, and the absolute value of the difference between the two is greater than a set threshold, then the typo is taken as the final typo detection result; otherwise, the typo is removed from the final typo detection result.

12. The method according to any one of claims 1-9, characterized in that, After comparing the real text and the text to be detected to obtain the typo detection results in the text to be detected, the method further includes: Determine the location of the misspelled word in the text to be detected, and output the location.

13. A device for detecting misspelled words in parallel audio-text data, characterized in that, include: The data acquisition unit is used to acquire input audio-text parallel data, which includes the text to be detected and the audio that matches the text to be detected; The feature extraction unit is used to extract the audio modal features of the audio and the text modal features of the text to be detected. The feature fusion unit is used to fuse the audio modal features and the text modal features to obtain fused features; The real text determination unit is used to determine the real text corresponding to the audio based on the fusion features; The misspelling detection unit is used to compare the real text and the text to be detected to obtain the misspelling detection result in the text to be detected. Wherein, the audio modal features and the text modal features are both in vector form, the feature fusion unit fuses the audio modal features and the text modal features to obtain fused features, the process of which includes: A gated fusion method is used to fuse audio modal features and text modal features in vector form to obtain fused features.

14. A device for detecting misspelled words in parallel audio-text data, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the method for detecting misspelled words in parallel audio-text data as described in any one of claims 1 to 12.

15. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the method for detecting typos in parallel audio-text data as described in any one of claims 1 to 12.

Images