Methods, devices, electronic equipment, storage media, and program products for acquiring corpora

Abstract

This invention discloses a method, apparatus, electronic device, storage medium, and program product for acquiring corpus data. The method includes: acquiring audio and video data, wherein the audio and video data includes audio data representing variant languages ​​and image data; determining first text content corresponding to the audio data and second text content corresponding to the image data, wherein the first text content describes the audio data and the second text content describes the image data; matching the first text content and the second text content to obtain matching information, wherein the matching information indicates the degree of matching between the first text content and the second text content; based on the matching information, determining target text content that matches the audio data from the first text content and the second text content; and combining the audio data and the target text content to obtain target corpus data. This invention solves the technical problem of low efficiency in acquiring corpus data.

Description

Technical Field

[0001] This invention relates to the field of large models, and more specifically, to a method, apparatus, electronic device, storage medium, and program product for acquiring corpora. Background Technology

[0002] Currently, variant languages ​​(such as dialects) are variations of natural language, typically used within specific geographical regions or social groups. Traditional speech recognition technologies often face the challenge of dialect differences because they are primarily trained and optimized for the standard language.

[0003] In related technologies, dialect data is often difficult to obtain, especially some relatively niche dialects. Relying solely on traditional recording and annotation methods results in high costs and long training times for models, leading to low efficiency in acquiring corpora.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, storage medium, and program product for acquiring corpus data, in order to at least solve the technical problem of low efficiency in acquiring corpus data.

[0006] According to one aspect of the present invention, a method for acquiring corpus is provided. The method may include: acquiring audio and video data, wherein the audio and video data includes audio data for representing variant languages ​​and image data; determining first text content corresponding to the audio data and second text content corresponding to the image data, wherein the first text content describes the audio data and the second text content describes the image data; matching the first text content and the second text content to obtain matching information, wherein the matching information indicates the degree of matching between the first text content and the second text content; based on the matching information, determining target text content that matches the audio data from the first text content and the second text content; and combining the audio data and the target text content to obtain target corpus.

[0007] Optionally, determining the first text content corresponding to the audio data in the audio and video data includes: using a speech recognition model to recognize human voices appearing in the audio data to obtain the first text content, wherein the speech recognition model is used to recognize variant languages.

[0008] Optionally, determining the second text content corresponding to the image data in the audio / video data includes: determining the text region in the image data and the location information corresponding to the text region; recognizing the text region to obtain the third text content in the text region and the initial recognition confidence level corresponding to the third text content; and obtaining the second text content based on the location information, the third text content, and the initial recognition confidence level.

[0009] Optionally, determining a text region in the image data includes: determining at least one sub-text region in the image data, wherein the sub-text region contains text whose deployment angle satisfies a horizontal threshold; and clustering and merging the at least one sub-text region to obtain a text region.

[0010] Optionally, the second text content is obtained based on location information, third text content, and initial recognition confidence, including: filtering subtitle text from the image data from the third text content based on location information; merging the subtitle text in the image data to obtain merged text; determining the target recognition confidence corresponding to the merged text based on the initial recognition confidence; and obtaining the second text content based on the target recognition confidence, the time information corresponding to the image data, and the merged text.

[0011] Optionally, the second text content is obtained based on the target recognition confidence score, the time information corresponding to the image data, and the merged text, including: determining the adjacent frame images of the image data; determining the first similarity between the merged text in the image data and the merged text in the adjacent frame images; in response to the first similarity score being greater than a similarity threshold, determining the merged text in the merged text and the merged text corresponding to the adjacent frame images whose target recognition confidence score is greater than the confidence threshold as the second text content; and updating the text content of the merged text in the image data and the text content of the merged text in the adjacent frame images as the second text content.

[0012] Optionally, matching the first text content and the second text content to obtain matching information includes: performing voice localization on the audio data to obtain the first start and end times of the first text content corresponding to at least one voice in the audio data; determining the second start and end times of the second text content; and matching the first text content and the second text content based on the first start and end times to obtain matching information.

[0013] Optionally, determining the start and end times of the second text content to obtain the second start and end times includes: determining the first image data in the multi-frame image data of the video data where the text content is the second text content, and the last image data where the text content is the second text content; and determining the second start and end times based on the time when the first image data appears and the time when the last image data appears.

[0014] Optionally, based on the first start and end times and the second start and end times, the first text content and the second text content are matched to obtain matching information, including: extending the first start and end times to obtain a third start and end times; determining at least one matching text content among multiple second text contents whose second start and end times are located in the third start and end times; and matching the at least one matching text content with the first text content respectively to obtain matching information.

[0015] According to another aspect of the present invention, a corpus acquisition apparatus is also provided. The apparatus may include: an acquisition unit for acquiring audio and video data, wherein the audio and video data includes audio data representing a variant language and image data; a first determining unit for determining first text content corresponding to the audio data in the audio and video data, and second text content corresponding to the image data in the audio and video data, wherein the first text content describes the audio data and the second text content describes the image data; a processing unit for matching the first text content and the second text content to obtain matching information, wherein the matching information indicates the degree of matching between the first text content and the second text content; a second determining unit for determining, based on the matching information, target text content matching the audio data from the first text content and the second text content; and a combining unit for combining the audio data and the target text content to obtain target corpus.

[0016] According to another aspect of the present invention, a non-volatile storage medium is also provided, wherein the non-volatile storage medium stores a plurality of instructions, the instructions being adapted to be loaded by a processor and executed any one of the above-described methods for acquiring corpus.

[0017] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-described methods for acquiring corpus.

[0018] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described methods for acquiring corpora.

[0019] In this embodiment of the invention, audio and video data are acquired, including audio data representing variant languages ​​and image data; first text content corresponding to the audio data and second text content corresponding to the image data are determined, wherein the first text content describes the audio data and the second text content describes the image data; the first text content and the second text content are matched to obtain matching information, wherein the matching information indicates the degree of matching between the first text content and the second text content; based on the matching information, target text content matching the audio data is determined from the first text content and the second text content; the audio data and the target text content are combined to obtain target corpus. In other words, this embodiment of the invention acquires audio and video data with variant languages ​​and text content, processes the audio data in the audio and video data to obtain first text information, processes multi-frame image data in the audio and video data to obtain second text information, and matches the first text information and the second text information to determine the target text content matching the audio data. The audio data and the target text content can be combined to obtain dialect corpus that can be used as training data, thereby achieving the technical effect of improving the efficiency of corpus acquisition and solving the technical problem of low efficiency in corpus acquisition. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0021] Figure 1 This is a flowchart of a corpus acquisition method according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart of a dialectal speech recognition corpus processing method based on OCR and VAD according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of a corpus acquisition device according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0028] Optical Character Recognition (OCR) is used to convert images or scanned text into machine-editable text data. Through image processing and character recognition technology, printed or handwritten characters can be extracted from paper documents or images for further processing, editing or storage.

[0029] Voice Activity Detection (VAD) is used to find the start and end points of a human voice in a speech signal containing silence, noise, etc.

[0030] Automatic speech recognition (ASR) is a technology that converts human speech into text or instructions. It can include voice assistants (such as Siri, Alexa, and Google Assistant), speech-to-text, telephone automation systems, and voice command control.

[0031] According to an embodiment of the present invention, an embodiment of a method for acquiring corpus is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] Figure 1 This is a flowchart of a corpus acquisition method according to an embodiment of the present invention, such as... Figure 1 As shown, the method may include the following steps:

[0033] Step S102: Obtain audio and video data, wherein the audio and video data includes audio data for representing variant languages ​​and image data.

[0034] In this embodiment, when corpus needs to be acquired, audio and video data can be obtained. The audio and video data can be the corpus used to train the speech recognition model, and may include audio data and image data representing variant languages. For example, it can be audio and video data with subtitles and dialect audio. Variant languages ​​can be dialects, slang, accents, jargon, etc. It should be noted that this is only an example and does not impose specific limitations on the types of variant languages. The audio data can be a speech signal that contains the pronunciation of the variant language. The image data may contain subtitles, which can be used to further verify the text content corresponding to the pronunciation in the audio data.

[0035] Optionally, the aforementioned audio and video data may include multiple frames of image data. Audio and video data with subtitles and containing dialect audio can be downloaded, wherein the audio data within the audio and video data may include dialect audio.

[0036] For example, various audio and video data with variant speech and subtitles can be obtained from variety shows, talk shows, movies, TV dramas, and the internet. This audio and video data can include audio data and multi-frame image data. The audio data can include dialects. The image data can include subtitles corresponding to the dialects. It should be noted that this is merely an example and does not impose specific restrictions on the sources of the audio and video data.

[0037] Step S104: Determine the first text content corresponding to the audio data and the second text content corresponding to the image data, wherein the first text content is used to describe the audio data and the second text content is used to describe the image data.

[0038] In this embodiment, a first text content corresponding to the audio data and a second text content corresponding to the image data can be determined. The first text content can be the result of Automatic Speech Recognition (ASR), used to determine the text content corresponding to the audio, representing the language spoken by a human in the audio data, and can be initial text information. The second text content can be text extracted from the image data, used to describe the subtitle content in the image data.

[0039] Optionally, when human voices are present in the audio data, the content of the voices can be determined to obtain the first text content. The audio / video data can consist of multiple image frames. Therefore, the text displayed in each frame can be extracted to obtain the second text content corresponding to the image data.

[0040] Step S106: Match the first text content and the second text content to obtain matching information, wherein the matching information is used to indicate the degree of matching between the first text content and the second text content.

[0041] In this embodiment, the first text content and the second text content can be matched to obtain matching information, which can then be used to determine the degree of matching between the first text content and the second text content. The matching information can be used to characterize the similarity between the text in the first text content and the second text content.

[0042] Optionally, matching the first text content and the second text content can determine the locations where the same text exists in the first text content and the second text content, thereby determining the matching information.

[0043] Step S108: Based on the matching information, determine the target text content that matches the audio data from the first text content and the second text content.

[0044] In this embodiment, target text content matching the audio data can be determined from the first text content and the second text content based on matching information. The target text content can be text corresponding to a variant language.

[0045] Step S110: Combine the audio data and the target text content to obtain the target corpus.

[0046] In this embodiment, audio data and target text content can be combined to obtain target corpus. The target corpus can be a dialect corpus or slang corpus, etc., which can be used to train an acoustic model to improve the accuracy of the acoustic model in recognizing variant languages. It should be noted that this is only an example and does not impose specific limitations on the type of target corpus. The acoustic model can be used to recognize variant languages.

[0047] Because variant languages ​​(e.g., dialects) are often difficult to obtain, especially some relatively niche dialects, relying solely on traditional recording and annotation methods results in high training costs and long training times for acoustic models, leading to low accuracy in variant language recognition. To address this issue, this embodiment acquires audio and video data, identifies the first text content emitted by the audio data, and the subtitle content (second text content) from multiple frames of images within the audio and video data. The first and second text contents are matched to obtain matching information. Based on this matching information, target text content with a high degree of matching with the audio data is determined. The audio data and target text content are then combined to obtain target corpus containing the pronunciation of the variant language and its corresponding text. This approach enriches the training data for the acoustic model, thereby improving the accuracy of the acoustic model in recognizing title languages.

[0048] Through steps S102 to S106 above, in response to a data transmission request, at least one first identification information is determined from the data transmission request, wherein the first identification information is used to identify the user equipment to which data is to be transmitted; a second identification information associated with the first identification information is determined; the first identification information and the data to be transmitted are transmitted to the session management function identified by the second identification information, so that the session management function transmits the data to be transmitted to a target user equipment group in the user equipment, wherein the first identification information of different target user equipments in the target user equipment group is the same. That is, in this embodiment of the invention, the second identification information associated with the first identification information of the user equipment is determined, and based on the second identification information, the session management function is directly called to transmit data to user equipments with the same first identification, so as to avoid the need to repeatedly call the same session management function to transmit data to multiple user equipments, thereby achieving the technical effect of avoiding resource waste during data transmission and solving the technical problem of low efficiency in acquiring corpora.

[0049] The method described in this embodiment will be further described below.

[0050] As an optional embodiment, step S104, determining the first text content corresponding to the audio data in the audio and video data, includes: using a speech recognition model to recognize human voices appearing in the audio data to obtain the first text content, wherein the speech recognition model is used to recognize variant languages.

[0051] In this embodiment, a speech recognition model can be used to identify the voices appearing in the audio data to obtain the first text content. The speech recognition model can be used to convert variant languages ​​in the audio data into text content. For example, it can be a dialect speech recognition model based on an attention-based neural network structure (conformer). It should be noted that this is only an example and does not impose specific limitations on the type of dialect speech recognition model.

[0052] Optionally, this embodiment can use a small amount of collected dialect data (i.e., dialect corpus) to first train a Conformer-based dialect speech recognition model. This model can be constructed by combining a Convolutional Neural Network (CNN) structure for acoustic feature extraction with a Transformer attention mechanism structure for context modeling. This construction method enables the model to effectively capture the temporal and contextual information of the audio data, thereby improving recognition accuracy. The acoustic features extracted by the trained dialect speech recognition model can be Mel-Frequency Cepstral Coefficients (MFCCs), which can employ an encoder-decoder structure. The encoder encodes the acoustic features of the audio data into a contextual representation, while the decoder converts the contextual representation into text output to obtain the first text content.

[0053] Optionally, after acquiring the audio and video data, the audio data can be extracted from the audio and video data, and the human voice appearing in the audio can be identified using a dialect speech recognition model to obtain the first text content. Here, the human voice can be the sound of someone speaking in the video, and the human voice can include variant languages, such as Mandarin, dialects, slang, etc. It should be noted that this is only an example and does not impose specific restrictions on the type of human voice.

[0054] Since the dialect speech recognition model at this point was trained using only a small amount of corpus, the first text content obtained using this model will not be very accurate. In subsequent steps, the first text content will be corrected based on the second text content.

[0055] As an optional embodiment, step S104, determining the second text content corresponding to the image data in the audio and video data, includes: determining the text region in the image data and the location information corresponding to the text region; recognizing the text region to obtain the third text content in the text region and the initial recognition confidence level corresponding to the third text content; and obtaining the second text content based on the location information, the third text content, and the initial recognition confidence level.

[0056] In this embodiment, text may appear in multiple locations within the image data. However, some of this text may not be the text we need. The following steps can be used to process the multiple text regions where text appears to ensure that the second text content is the content we require:

[0057] Optionally, determining the text region in the image data and its corresponding location information can be achieved by using a text recognition model to identify the text region, obtaining at least one third text content within the text region, and an initial recognition confidence score corresponding to the third text content. Based on the location information and the initial recognition confidence score, the third text content is processed to obtain the second text content. The text region can be any area in the image data containing text, such as a region containing subtitles or subtitles with text. The location information can be used to determine the region where the text region is located in the image data, represented by (X, Y), and can include the deployment position and angle of the text region. The initial recognition confidence score can be used to characterize the probability that the third text content in the text region contains real text. The text recognition model can be constructed based on a Semantic Visual Textual Relevance (SVTR) algorithm and can be used to identify the text content within the text region.

[0058] Optionally, text may appear in multiple regions of the image data. Therefore, the text regions and their corresponding location information can be determined. A text recognition model can be used to identify the text regions to determine at least one third text content within a text region, and the initial recognition confidence level corresponding to the third text content can be determined using the text recognition model. Based on the location information and the initial recognition confidence level, the third text content is filtered and processed to obtain the second text content.

[0059] Optionally, since much of the audio and video data containing dialect audio is quite old and lacks high clarity, it's possible to extract multiple frames of image data per second for optical character recognition (OCR). To balance batch processing efficiency, a frequency of 4 frames per second (4 frames per second, with a 0.25-second time difference between adjacent frames) can be selected to extract segments from the audio and video data. For example, a 30-minute video episode can yield 7200 images in this way, resulting in multiple frames of image data. It should be noted that the data size mentioned above is merely illustrative and not a specific limitation; the frame rate can be selected based on actual needs.

[0060] For example, Chinese character recognition and detection models can be used to perform OCR on the captured image data. The Chinese detection model can be used to determine the text regions and their corresponding location information within the image data. Furthermore, the character recognition model can be used to identify the text regions, determining the third text content within those regions and the initial recognition confidence level for that third text region. Based on the initial recognition confidence level and location information, the third text content is adjusted to obtain the second text content. The Chinese detection model can employ a database algorithm (DB), which can be a DB module.

[0061] Optionally, the Chinese detection model can be used for text detection. It can locate and extract text regions from image data, along with their corresponding location information. This Chinese detection model can be a lightweight deep learning model, such as the Efficient and Accurate SceneText Detector (EAST). It should be noted that this is merely an example and does not impose specific limitations on the type of text detection model.

[0062] Optionally, the character recognition model can be used to complete the text recognition task, that is, to convert the characters in the detected text region into a readable text string in order to obtain the third text content corresponding to the text region.

[0063] For example, a text recognition model can be built using a variational autoencoder and a Transformer model. It can employ a sequence-to-sequence algorithm, where each text region is first segmented into individual characters, and these characters are then fed into a variational autoencoder for feature extraction and encoding. The Transformer model is then used to decode and generate the corresponding text sequence, yielding accurate third-party text content. Furthermore, to improve robustness, the text recognition model employs an attention mechanism to model the dependencies between different characters, thereby enhancing recognition accuracy.

[0064] Optionally, the Chinese detection model can determine the text region and location information of the text appearing in the image data, and the character recognition model can identify the specific content (third text content) in the text region and the initial recognition confidence.

[0065] As an optional embodiment, determining a text region in image data includes: determining at least one sub-text region in the image data, wherein the sub-text region contains text whose deployment angle satisfies a horizontal threshold; and clustering and merging the at least one sub-text region to obtain a text region.

[0066] In this embodiment, text may exist in multiple locations within the image data. Therefore, at least one sub-text region in the image data whose deployment angle meets a horizontal threshold can be identified. This at least one sub-text region is then clustered and merged to obtain the final text region. The horizontal threshold can be a pre-set value, such as 180 degrees. It should be noted that this is merely an example and no specific limitation is made on the value of the horizontal threshold. The deployment angle can be the angle at which the sub-text region is placed. The sub-text region may include text lines or text segments.

[0067] Since some image data may contain irrelevant text such as logos and advertisements, multiple text instances may be identified. However, only the subtitle text is needed. Therefore, text regions with a horizontal or near-horizontal deployment angle can be detected to obtain at least one text region. By clustering and merging the detected text lines, they are combined into a larger text region to obtain the final text detection result, i.e., the text region.

[0068] As an optional embodiment, step S104, based on location information, third text content, and initial recognition confidence, obtains second text content, including: filtering subtitle text from the image data based on location information; merging the subtitle text in the image data to obtain merged text; determining the target recognition confidence corresponding to the merged text based on the initial recognition confidence; and obtaining the second text content based on the target recognition confidence, the time information corresponding to the image data, and the merged text.

[0069] In this embodiment, since the subtitle text in the image data can be used to determine the actual content that the title language in the audio data intends to express, the subtitle text of the image data can be filtered out from the third text content. Furthermore, since there may be spaces between subtitles within the same image data, the subtitle text can be merged to obtain merged text.

[0070] Optionally, based on location information, subtitle text can be filtered from the third text content to extract subtitle text from the image data. This subtitle text can then be merged to obtain merged text. Based on the initial recognition confidence level, the target recognition confidence level corresponding to the merged text can be determined. Based on the target confidence level, the time information corresponding to the image data, and the merged text, the second text content is obtained. The time information corresponding to the image data can be used to determine the time when the image data appears in the audio / video data.

[0071] Because some images may contain irrelevant text such as logos or advertisements, multiple text regions may be identified. Therefore, after batch identifying the third text content in all frames of image data, the positions of the third text content can be statistically analyzed. Since it can be assumed that subtitles in the same video appear in roughly the same position, the approximate position of the most frequently occurring third text content in multiple frames of image data can be determined statistically. Furthermore, because subtitles are horizontal, the X-axis positions of the four endpoints of each subtitle are not fixed, but the Y-axis is relatively fixed. Therefore, we can statistically determine a fixed interval on the Y-axis of the subtitle appearance area to filter out non-subtitle text. Using the above method, subtitle text in image data can be filtered out from third text content based on positional information.

[0072] Optionally, since there may be spaces between the subtitle text in the same image data, there will be multiple recognition results for the text region. In this case, multiple text regions can be merged to obtain merged text. The initial recognition confidence of each third text content in the merged text is determined, and the average of the initial recognition confidence of multiple third text contents is determined to obtain the target recognition confidence corresponding to the merged text. Through this step, triplet data containing time information, third text content, and target recognition confidence can be obtained for each frame of image data.

[0073] Optionally, the content of the second text can be further determined based on the target recognition confidence level, the time information corresponding to the image data, and the merged text.

[0074] As an optional embodiment, the second text content is obtained based on the target recognition confidence level, the time information corresponding to the image data, and the merged text, including: determining adjacent frame images of the image data; determining a first similarity between the merged text in the image data and the merged text in the adjacent frame images; in response to the first similarity being greater than a similarity threshold, determining the merged text in the merged text and the merged text corresponding to the adjacent frame images whose target recognition confidence level is greater than the confidence threshold as the second text content; and updating the text content of the merged text in the image data and the text content of the merged text in the adjacent frame images to the second text content.

[0075] In this embodiment, several adjacent image data may correspond to the same subtitle text. However, due to factors such as low image data resolution and errors in the OCR model (text detection model + text recognition model), image data that should have the same subtitle text may be recognized as different subtitle texts. Therefore, we need to determine whether adjacent image data correspond to the same subtitle text. If adjacent image data correspond to the same subtitle text, the subtitle text in multiple frames of image data can be unified.

[0076] Optionally, adjacent frame images of the image data are determined. A first similarity is determined between the merged text in the image data and the merged text in the adjacent frame images. If the first similarity is greater than a similarity threshold, the merged text in the merged text corresponding to the adjacent frame images with a target confidence score greater than the confidence threshold can be determined as the second text content. The text content of the merged text in the image data and the text content of the merged text in the adjacent frame images can be updated as the second text content.

[0077] For example, identify adjacent frames 2 and 3 of image data 1, where adjacent frame 2 plays before image data 1, and adjacent frame 3 plays before image data 2. Determine the first similarity between the merged text in image data 1 and the merged text in adjacent frame 2, and the first similarity between the merged text in image data 1 and the merged text in adjacent frame 3. If the first similarity is greater than a similarity threshold, it can be determined that the subtitles corresponding to image data 1, image data 2, and image data 3 are the same. Therefore, the merged text with the highest target confidence among the merged texts of image data 1, image data 2, and image data 3 can be determined as the second text content. Update the merged texts in image data 1, image data 2, and image data 3 as the second text content.

[0078] Optionally, determine whether the merged text (i.e., the final displayed subtitle) in adjacent image data is the same. If so, merge them. Determine the start time and content text of each subtitle and filter out subtitles that are too short or have low OCR confidence.

[0079] For example, a method can be adopted to calculate the similarity between adjacent texts. If the first similarity between the merged texts of adjacent image data is greater than the similarity threshold, then the adjacent image data can be considered to correspond to the same subtitle. Experiments have shown that the similarity threshold can be 0.5. The comparison can start from the first frame image data where the subtitle (which can be merged text) appears and proceed backwards. If the first similarity obtained is greater than the threshold, the merged text with the higher confidence in the recognition result is updated to the correct text corresponding to the subtitles of the two image data (i.e., the second text content). The start time and end time of the second text content are also updated. If the OCR fails to recognize the merged text, it is skipped, and the similarity comparison continues until the first similarity with the correct text is less than the similarity threshold. At this point, the current frame is considered the starting point of a new subtitle, and the previous frame is considered the end of the previous subtitle. The merging operation for the next subtitle then begins from the current frame. After the above processing, a triplet including the start time, end time, second text content, and recognition confidence can be obtained.

[0080] Optionally, triplets with a recognition confidence score of less than 0.8 for the second text content can be filtered out. This is because if the highest confidence score of all frames of image data corresponding to a certain subtitle does not reach 0.8, the recognition result of that image data can be considered poor. At the same time, excessively short recognition results are likely to be incorrect. Therefore, triplets with excessively short durations, i.e., second text content appearing for only 2 frames or less, can be filtered out. A large portion of erroneous text can be filtered out through the above steps. It should be noted that the numbers mentioned above are only illustrative and can be selected according to actual circumstances; no specific restrictions are imposed here.

[0081] As an optional embodiment, step S106, matching the first text content and the second text content to obtain matching information, includes: performing voice localization on the audio data to obtain the first start and end times of the first text content corresponding to at least one voice in the audio data; determining the second start and end times of the second text content; and matching the first text content and the second text content based on the first start and end times to obtain matching information.

[0082] In this embodiment, voice localization can be performed on the audio data to obtain the first start and end times of the first text content corresponding to at least one voice in the audio data. The audio data may include at least one segment of voice corresponding to the first start and end times. The first start and end times may include the start and end times of the voice.

[0083] Optionally, in multi-frame image data, there may be cases where the second text content is the same. The time when the second text content first appears and the time when it ends in the video data are determined in order to obtain the second start and end time corresponding to the second text content.

[0084] For example, audio information can be extracted from video data. Simultaneously, a Voice Activity Detection (VAD) model based on an attention-based neural network structure can be used to batch process the extracted audio information. That is, the VAD model is used to determine the start and end points of human voices in the audio information to obtain the first start and end times.

[0085] Optionally, the input features of the speech endpoint detection model can be Mel-frequency cepstral coefficients (MFCCs), and the speech endpoint detection model can consist of linear layers and multiple core modules. Each core module consists of a feedforward network module, a convolutional module, a multi-head attention mechanism module, and a layer normalization layer, and each module has residual connections.

[0086] Optionally, the speech endpoint detection model can classify noise (including silence) and human voice into two categories. It can use VAD to locate each human voice in the audio and obtain the start and end times of the human voice, thus obtaining the first start and end time. The first start and end time includes the start and end times of the human voice.

[0087] As an optional embodiment, determining the start and end times of the second text content to obtain the second start and end times includes: determining the first image data in the multi-frame image data of the video data where the text content is the second text content, and the last image data where the text content is the second text content; and determining the second start and end times based on the time when the first image data appears and the time when the last image data appears.

[0088] In this embodiment, the first image data in the multi-frame image data of the video data whose text content is the second text content, and the last image data whose text content is the second text content, can be determined. The second start and end time can be determined based on the time when the first image data appears and the time when the last image data appears.

[0089] It should be noted that no other second text content can exist between the times of the first and last image data. For example, if the second text content in multiple frames of image data is Text Content 1, Text Content 1, Text Content 2, and Text Content 1 respectively, then the start time of Text Content 1 can be determined as the first second, and the stop time as the second second, not the fourth second. If the second text content in multiple frames of image data is Text Content 1, Text Content 1, Text Content 1, and Text Content 2 respectively, then the start time of Text Content 1 can be determined as the first second, and the stop time as the third second.

[0090] As an optional embodiment, matching the first text content and the second text content based on the first start and end times and the second start and end times to obtain matching information includes: extending the first start and end times to obtain a third start and end time; identifying at least one second target text content among a plurality of second text contents whose second start and end times are located within the third start and end times; and matching the at least one second target text content with the first text content to obtain matching information.

[0091] In this embodiment, since the audio data and multi-frame image data are extracted from the same audio and video data, the first text content and the second text content should be the same. Therefore, the first text content and the second text content can be matched based on the first start time and the second start time to obtain matching information. Based on the matching information, the target text content that matches the audio data can be determined from the first text content and the second text content.

[0092] Optionally, ASR results (i.e., the first text content) and OCR results (i.e., the second text content) from similar times are matched. If a match is found, the OCR result is used to replace the ASR result. That is, the VAD result is used as a timestamp and the OCR text is used as an annotation for the speech recognition dialect corpus.

[0093] Simply aligning the OCR and ASR results using timestamps results in a low accuracy rate for dialect recognition; furthermore, relying solely on manual judgment and filtering leads to inefficient data processing. To address these issues, this embodiment uses the obtained OCR results to correct the VAD and ASR results.

[0094] Optionally, since the first start and end times of the first text content in the video data are only around the time of human speech in the video, while the second start and end times of the second text content are the start and end times of each sentence's pronunciation obtained from the acoustic level, the timestamp given by the first start and end times is more accurate, while the timestamp of the second start and end times can only be used as a reference to correct the first text content with the second text content. Therefore, among multiple second text contents, at least one matching text content whose second start and end times are located within the third start and end times is identified; the at least one matching text content is then matched with the first text content to obtain matching information.

[0095] Optionally, based on the matching information, at least one target matching text content that matches the first text content with a high degree of matching can be selected, and the first text content can be replaced with the target matching text content.

[0096] For example, because excessively short text content may affect the accuracy of ASR and OCR text matching, short ASR results can be merged, such as merging text corresponding to speech lengths between 7-15 seconds. For each ASR recognition text, the corresponding subtitle should generally appear three seconds before the start time obtained from VAD and three seconds after the end time obtained from VAD. Therefore, the first start and end time can be extended to obtain a third start and end time [start, end]. Since the start and end times of each second text content have been determined, if any part of the second start and end time corresponding to the second text content falls between [start, end], then the OCR result subtitle (second text content) can be considered as potentially matching the ASR result (first text content). Then, all these possible subtitles are merged into a long text, and the corresponding OCR text of the current ASR text can be considered to exist within this long text. Finally, a matching algorithm is used to accurately locate the specific OCR text corresponding to the ASR text. The first five characters and last five characters of the ASR text are matched against the long text to find positions with a match score of 0.8 or higher. These two positions, including the middle part, are then used as the corresponding OCR text. Simultaneously, the last five characters of the previous ASR text and the first five characters of the next ASR text are matched against the long text. If a match is found, this is also considered a possible beginning or end position for the current sentence. If multiple positions meet the matching requirements, the similarity between the multiple OCR texts obtained through the above rules and the original ASR text is calculated. The OCR text with the highest similarity and a length ratio of less than 1.5 to the ASR text is considered the correct text. If no matching position is found, the matching is abandoned, and the ASR text is discarded. The original ASR text is then replaced with the OCR result to obtain the corrected target text content.

[0097] In this embodiment of the invention, audio and video data with variant languages ​​and text content are acquired. The audio data in the audio and video data is processed to obtain first text information, and the multi-frame image data in the audio and video data is processed to obtain second text information. The first text information and the second text information are matched to determine the target text content that matches the audio data. The audio data and the target text content can be combined to obtain dialect corpus that can be used as training data, thereby achieving the technical effect of improving the efficiency of corpus acquisition and solving the technical problem of low efficiency in corpus acquisition.

[0098] The following section provides a further description of an embodiment of the present invention, focusing on a terminal capability and network service type indication and its usage.

[0099] Speech recognition technology converts sound signals into readable text. The core of this technology lies in acoustic models, which can identify phonemes, syllables, and sound features in speech. Typically, the development of acoustic models relies on deep learning techniques, such as deep neural networks (DNNs) and convolutional neural networks, to improve accuracy and robustness.

[0100] Simultaneously, speech recognition systems also need to incorporate language models to better understand and interpret the meaning of spoken language. Language models consider the relationships between words and phrases, helping the system more accurately convert speech input into text output. Furthermore, large-scale training data is crucial for improving the performance of speech recognition systems. This training data includes speech samples and corresponding text transcriptions, which are then used by machine learning models to learn and continuously optimize.

[0101] Dialects are variations of natural language, typically used within specific geographical regions or social groups. Traditional speech recognition technologies often face the challenge of dialectal differences because they are primarily trained and optimized for the standard language. To address this challenge and enable speech recognition technologies to accurately identify and understand diverse dialects, a large dialect corpus is needed, encompassing extensive coverage of dialect pronunciation, vocabulary, and grammatical rules. The richness of the dialect corpus is crucial for the training and performance of language models, requiring samples from various communities or geographical areas, covering diverse dialect variations and pronunciation differences, to train the language model. Simultaneously, the dialect corpus should include representative samples from different ages, genders, social groups, and usage scenarios to ensure the robustness and universality of the language model. A rich dialect corpus also helps improve the acoustic and linguistic models, thereby enhancing the accuracy and usability of dialect speech recognition.

[0102] However, dialect data is often difficult to obtain in related technologies, especially for some relatively niche dialects. Training models using only traditional recording and annotation methods is costly and time-consuming. Furthermore, achieving a balance between accuracy and retention when extracting subtitles using OCR requires specific methods.

[0103] To address the aforementioned problems in the language recognition process, this embodiment proposes a dialect speech recognition corpus processing method based on OCR and VAD. This method can obtain speech recognition corpus from dialect videos with embedded subtitles that can be used to train a dialect speech recognition model, and use the results of OCR to correct the text of the corpus to improve the quality of the corpus, thereby further improving the quality of the dialect speech recognition model. This achieves the technical effect of improving the accuracy of dialect recognition and solves the technical problem of low accuracy in dialect recognition.

[0104] Figure 2 This is a flowchart of a dialectal speech recognition corpus processing method based on OCR and VAD according to an embodiment of the present invention, such as... Figure 2 As shown, the method may include the following steps:

[0105] Step S201: Obtain audio and video data with subtitles.

[0106] In this embodiment, audio and video data with subtitles and dialect audio can be downloaded, wherein the audio data in the audio and video data may contain dialect audio.

[0107] Step S202: Prompt the audio data of the audio and video data, and determine the first start and end time of the human voice in the audio information.

[0108] In this embodiment, audio information is extracted from audio and video data. Simultaneously, a speech endpoint detection model based on an attention mechanism neural network structure can be used to process the extracted audio information in batches. That is, the speech endpoint detection model is used to determine the start and end points of human voices in the audio information, thereby improving the fluency of voice interaction dialogue and user experience.

[0109] Optionally, the input features of the speech endpoint detection model can be Mel-frequency cepstral coefficients (MFCCs), and the speech endpoint detection model can consist of linear layers and multiple core modules. Each core module consists of a feedforward network module, a convolutional module, a multi-head attention mechanism module, and a layer normalization layer, and each module has residual connections.

[0110] Optionally, the speech endpoint detection model can classify noise (including silence) and human voice into two categories. Using VAD (Voice Endpoint Detection), it can locate each human voice segment in the audio and obtain its start and end times. The start and end times include the initial start time and the final end time.

[0111] Step S203: Perform speech recognition on the human voice to determine the first text content.

[0112] In this embodiment, a Transformer-based dialect speech recognition model can be trained using a small amount of collected dialect data (i.e., dialect corpus). This model can be constructed by combining a CNN structure for acoustic feature extraction with a Transformer structure for context modeling. This construction method enables the model to effectively capture the temporal and contextual information of the audio data, thereby improving recognition accuracy. The acoustic features extracted by the trained dialect speech recognition model can be Mel-Frequency Cepstral Coefficients (MFCCs), which can employ an encoder-decoder structure. The encoder encodes the acoustic features of the audio data into a contextual representation, while the decoder converts the contextual representation into text output to obtain the first text content.

[0113] Optionally, the dialect speech recognition model trained on the site can be used to convert the speech information with determined start and end times to obtain the initial text information.

[0114] Optionally, since the dialect speech recognition model at this time is only trained using a small amount of corpus, the first text content obtained using this model will not be very accurate. In subsequent steps, the first text content will be corrected based on the second text content.

[0115] Step S204: Extract audio and video data to obtain multiple frames of image data.

[0116] In this embodiment, since much of the audio and video data containing dialect audio is quite old and lacks high clarity, multiple frames of image data can be extracted per second for optical character recognition. To balance the efficiency of batch OCR processing, a frequency of 4 frames per second (4 frames per second, with a time difference of 0.25 seconds between adjacent frames) can be selected to extract audio and video data. For example, a 30-minute video episode can yield 7200 images in this way, resulting in multiple frames of image data. It should be noted that the above data size is only an example and is not a specific limitation; the frame rate can be selected according to the actual situation.

[0117] Step S205: Identify the text location and second text content of the image data.

[0118] In this embodiment, a Chinese detection model and a Chinese recognition model can be used to perform OCR on the captured image data. The Chinese detection model can employ a database algorithm (DB), and the Chinese recognition model can use the SVTR algorithm.

[0119] Optionally, the Chinese detection model can be used for text detection. It can locate and extract text regions from image data, along with their corresponding location information. This Chinese detection model can be a lightweight deep learning model, such as the Efficient and Accurate SceneText Detector (EAST). It should be noted that this is merely an example and does not impose specific limitations on the type of text detection model.

[0120] Optionally, the character recognition model can be used to complete the text recognition task, that is, to convert the characters in the detected text region into a readable text string in order to obtain the third text content corresponding to the text region.

[0121] For example, a text recognition model can be built using a variational autoencoder and a Transformer model. It can employ a sequence-to-sequence algorithm, where each text region is first segmented into individual characters, and these characters are then fed into a variational autoencoder for feature extraction and encoding. The Transformer model is then used to decode and generate the corresponding text sequence, yielding accurate third-party text content. Furthermore, to improve robustness, the text recognition model employs an attention mechanism to model the dependencies between different characters, thereby enhancing recognition accuracy.

[0122] Optionally, the Chinese detection model can determine the text region and location information of the text appearing in the image data, and the character recognition model can identify the specific content (third text content) in the text region and the initial recognition confidence.

[0123] Step S206: Select subtitle text from the second text content.

[0124] In this embodiment, since some images may contain irrelevant text such as logos or advertisements, multiple text regions may be identified. Therefore, after batch identifying the third text content in all frames of image data, the positions of the third text content can be statistically analyzed. Since it can be assumed that subtitles in the same video appear in roughly the same position, the approximate position of the most frequently occurring third text content across multiple frames of image data can be statistically determined. Furthermore, because subtitles are horizontal, the X-axis positions of the four endpoints of each subtitle are not fixed, but the Y-axis is relatively fixed. Therefore, we can statistically determine a fixed interval along the Y-axis of the subtitle appearance area to filter out non-subtitle text. Using the above method, subtitle text in image data can be filtered from the third text content based on positional information.

[0125] Optionally, since there may be spaces between the subtitle text in the same image data, there will be multiple recognition results for the text region. In this case, multiple text regions can be merged to obtain merged text. The initial recognition confidence of each third text content in the merged text is determined, and the average of the initial recognition confidence of multiple third text contents is determined to obtain the target recognition confidence corresponding to the merged text. Through this step, triplet data containing time information, third text content, and target recognition confidence can be obtained for each frame of image data.

[0126] Step S207: Process the captions of adjacent image data.

[0127] In this embodiment, it is determined whether the subtitles of adjacent images are the same. If so, they are merged to obtain the start time and content of each subtitle, and subtitles that are too short or have low OCR confidence are filtered out.

[0128] Optionally, the resulting series of triplet data can be further processed.

[0129] Optionally, since several adjacent images actually correspond to the same subtitle, they need to be merged together. However, due to low resolution and errors in the OCR model results, different images of the same subtitle often recognize different text. Therefore, it is necessary to first determine whether adjacent images correspond to the same subtitle.

[0130] Optionally, we can calculate the similarity between adjacent texts. If the similarity is greater than a threshold, they are considered to correspond to the same subtitle. Here, we set the threshold to 0.5 after experimentation. We compare from the first frame where the subtitle appears to the next. If the similarity is greater than the threshold, we update the text with the higher confidence score as the correct text for the current subtitle, and update the start and end times. If we encounter a case where the OCR fails to recognize the text, we skip it and continue comparing similarities until a similarity to the correct text is less than the threshold. At this point, we consider the current frame as the start of a new subtitle, and the previous frame as the end of the previous subtitle. Then, we start merging the next subtitle from the current frame. After the above processing, we obtain a triplet including the start time, end time, text, and recognition confidence score. Then, we remove the Arabic numerals, because 10 9 8 7 will be OCRed as 10987, which will sound different depending on the sentence structure.

[0131] Optionally, triples with a recognition confidence score below 0.8 can be filtered out. If the highest confidence score across all frames corresponding to a given subtitle is less than 0.8, the recognition is considered poor. Similarly, triples with excessively short durations (i.e., OCR text appearing in only 2 frames or less) can be filtered out, as such short recognition results are likely incorrect. These steps can filter out a large portion of erroneous text. It should be noted that the numbers mentioned above are merely illustrative and can be adjusted based on actual circumstances; no specific limitations are imposed here.

[0132] Step S208: Match ASR results and OCR results from similar time periods.

[0133] In this embodiment, ASR results (i.e., the first text content) and OCR results (i.e., the second text content) from similar times are matched. If a match is found, the OCR result is used to replace the ASR result. That is, the VAD result is used as a timestamp and the OCR text is used as an annotation for the speech recognition dialect corpus.

[0134] Simply aligning OCR and ASR results using timestamps results in low dialect recognition accuracy; furthermore, relying solely on manual judgment and filtering leads to inefficient data processing. To address these issues, this embodiment uses the obtained OCR results to correct the VAD and ASR results.

[0135] Alternatively, since the subtitles in the video only appear around the time when the person in the video is speaking, while VAD obtains the start and end times of each sentence's sound from an acoustic level, the timestamps given by the VAD results are more accurate, while the timestamps of OCR can only be used as a reference to help correct the ASR text with the OCR results.

[0136] For example, because excessively short texts may affect the accuracy of ASR and OCR text matching, excessively short ASR results can be merged, such as merging texts corresponding to speech with a length between 7 and 15 seconds. For each ASR recognition text, the corresponding subtitle should generally appear three seconds before the start time obtained from VAD and three seconds after the end time obtained from VAD. For ease of subsequent description, the start and end times can be represented as [start, end]. Since the start and end times of each OCR result have been determined in step S205, as long as a portion of the time appears between [start, end], the OCR result subtitle is considered to be a possible subtitle that can match the ASR result. Then, all these possible subtitles are merged into a long text, and the corresponding OCR text of the current ASR text can be considered to exist within this long text. Then, the matching algorithm is used to accurately locate the specific OCR text corresponding to the ASR text. The first 5 characters and the last 5 characters of the ASR text are matched with the long text respectively, and positions with a matching degree of 0.8 or higher are found. Then, these two positions, including the middle part, are taken as the corresponding OCR text. Simultaneously, the last five characters of the previous ASR text and the first five characters of the next ASR text are matched against the long text. If a match is found, it is considered a possible beginning or end of the current sentence. If multiple matching positions are found, the similarity between the multiple OCR texts obtained through the above rules and the original ASR text is calculated. The text with the highest similarity and a length ratio of the OCR text to the ASR text within 1.5 is considered the correct text. If no matching position is found, the matching is abandoned, and the ASR text is discarded. The original ASR text is then replaced with the OCR result to obtain the corrected dialectal speech recognition corpus.

[0137] In this embodiment, audio and video data with variant languages ​​and text content are acquired. The audio data in the audio and video data is processed to obtain first text information, and the multi-frame image data in the audio and video data is processed to obtain second text information. The first text information and the second text information are matched to determine the target text content that matches the audio data. The audio data and the target text content can be combined to obtain dialect corpus that can be used as training data, thereby achieving the technical effect of improving the efficiency of corpus acquisition and solving the technical problem of low efficiency in corpus acquisition.

[0138] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0140] This embodiment also provides a corpus acquisition device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "unit" and "device" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0141] According to an embodiment of the present invention, an embodiment of an apparatus for acquiring the above-described corpus is also provided. Figure 3 This is a schematic diagram of the structure of a corpus acquisition device according to an embodiment of the present invention, as shown below. Figure 3 As shown, the above-mentioned corpus acquisition device includes: acquisition unit 302, first determination unit 304, processing unit 306, second determination unit 308 and combination unit 310.

[0142] The aforementioned acquisition unit 302 is used to acquire audio and video data, wherein the audio and video data includes audio data for representing variant languages ​​and image data.

[0143] The first determining unit 304 is used to determine the first text content corresponding to the audio data and the second text content corresponding to the image data, wherein the first text content is used to describe the audio data and the second text content is used to describe the image data.

[0144] The aforementioned processing unit 306 is used to match the first text content and the second text content to obtain matching information, wherein the matching information is used to indicate the degree of matching between the first text content and the second text content.

[0145] The second determining unit 308 is used to determine the target text content that matches the audio data from the first text content and the second text content based on the matching information.

[0146] The aforementioned combination unit 310 is used to combine audio data and target text content to obtain target corpus.

[0147] In this embodiment of the invention, the acquisition unit 302 acquires audio and video data, which includes audio data representing variant languages ​​and image data. The first determining unit 304 determines first text content corresponding to the audio data and second text content corresponding to the image data, where the first text content describes the audio data and the second text content describes the image data. The processing unit 306 matches the first and second text content to obtain matching information, which indicates the degree of matching between the first and second text content. The second determining unit 308, based on the matching information, determines target text content that matches the audio data from the first and second text content. The combining unit 310 combines the audio data and the target text content to obtain target corpus, thereby improving the efficiency of corpus acquisition and solving the problem of low corpus acquisition efficiency.

[0148] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0149] It should be noted that the acquisition unit 302, the first determination unit 304, the processing unit 306, the second determination unit 308, and the combination unit 310 correspond to steps S102 to S110 in the embodiments. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can run on a computer terminal.

[0150] It should be noted that the optional or preferred implementation methods of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0151] The aforementioned corpus acquisition device may also include a processor and a memory. The acquisition unit 302, the first determining unit 304, the processing unit 306, the second determining unit 308, and the combination unit 310 are all stored in the memory as program modules. The processor executes the aforementioned program modules stored in the memory to realize the corresponding functions.

[0152] The processor contains a core that retrieves the corresponding program modules from memory. One or more cores may be configured. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory includes at least one memory chip.

[0153] According to an embodiment of the present invention, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium includes a stored program, wherein, when the program is running, it controls the device where the non-volatile storage medium is located to execute any of the above-mentioned corpus acquisition methods.

[0154] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals, and the non-volatile storage medium includes stored programs.

[0155] Optionally, during program execution, the device containing the non-volatile storage medium performs the following functions: in response to a data transmission request, determining at least one first identification information from the data transmission request, wherein the first identification information is used to identify the user equipment to which data is to be transmitted; determining a second identification information associated with the first identification information; and transmitting the first identification information and the data to be transmitted to a session management function identified by the second identification information, so that the session management function transmits the data to be transmitted to a target user equipment group in the user equipment, wherein the first identification information of different target user equipments in the target user equipment group is the same.

[0156] According to an embodiment of the present invention, an embodiment of a processor is also provided. Optionally, in this embodiment, the processor is used to run a program, wherein the program executes any of the above-described methods for acquiring corpora.

[0157] According to an embodiment of the present invention, an embodiment of a computer program product is also provided, which, when executed on a data processing device, is adapted to execute a program that initializes the acquisition method steps of any of the above-described corpora.

[0158] Optionally, when the above-mentioned computer program product is executed on a data processing device, it is suitable to execute an initialization program having the following method steps: acquiring audio and video data, wherein the audio and video data includes audio data for representing variant languages ​​and image data; determining a first text content corresponding to the audio data and a second text content corresponding to the image data, wherein the first text content is used to describe the audio data and the second text content is used to describe the image data; matching the first text content and the second text content to obtain matching information, wherein the matching information is used to indicate the degree of matching between the first text content and the second text content; based on the matching information, determining a target text content that matches the audio data from the first text content and the second text content; and combining the audio data and the target text content to obtain a target corpus.

[0159] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention, such as... Figure 4 As shown, this embodiment of the invention provides an electronic device 40, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring audio and video data, wherein the audio and video data includes audio data for representing variant languages ​​and image data; determining a first text content corresponding to the audio data and a second text content corresponding to the image data, wherein the first text content is used to describe the audio data and the second text content is used to describe the image data; matching the first text content and the second text content to obtain matching information, wherein the matching information is used to indicate the degree of matching between the first text content and the second text content; based on the matching information, determining a target text content that matches the audio data from the first text content and the second text content; and combining the audio data and the target text content to obtain a target corpus.

[0160] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0161] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules described above can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between modules, and may be electrical or other forms.

[0163] The modules described above as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0164] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0165] If the aforementioned integrated modules are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a non-volatile storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned non-volatile storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0166] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for acquiring a corpus, characterized in that, include: Acquire audio and video data, wherein the audio and video data includes audio data for representing variant languages ​​and image data, the audio and video data consists of multiple frames of images, the audio data includes dialect audio, and the image data contains subtitles corresponding to the dialect audio; Determine the first text content corresponding to the audio data and the second text content corresponding to the image data, wherein the first text content is used to describe the audio data and the second text content is used to describe the image data; The first text content and the second text content are matched to obtain matching information, wherein the matching information is used to indicate the degree of matching between the first text content and the second text content; Based on the matching information, target text content that matches the audio data is determined from the first text content and the second text content; The audio data and the target text content are combined to obtain the target corpus; Based on the matching information, determining the target text content that matches the audio data from the first text content and the second text content includes: in response to the matching information indicating that the first text content and the second text content match, modifying the first text content based on the second text content to obtain the target text content; Matching the first text content and the second text content to obtain the matching information includes: using a speech endpoint detection model to locate human voices in the audio data, obtaining a first start and end time of the first text content corresponding to at least one human voice in the audio data, wherein the speech endpoint detection model is used to determine the start and end points of the human voice in the audio data; extending the first start and end time to obtain a third start and end time; determining at least one matching text content whose start and end time is located in the third start and end time from a plurality of second text contents; and determining the matching degree between the at least one matching text content and the first text content to obtain the matching information. The step of determining the first text content corresponding to the audio data includes: in response to the appearance of a human voice in the audio data, determining the content of the human voice to obtain the first text content; Determining the second text content corresponding to the image data includes: determining a text region in the image data and the location information corresponding to the text region; using a character recognition model to recognize the text region to obtain at least one third text content in the text region, and an initial recognition confidence score corresponding to the third text content, wherein the text region is a region in the image data containing text, and the location information is used to determine the region where the text region is located in the image data, the location information including the deployment position and deployment angle of the text region; the character recognition model is constructed based on a semantic visual text relevance algorithm and is used to recognize the text content in the text region; based on the location information and the initial recognition confidence score, the third text content is processed to obtain the second text content.

2. The method according to claim 1, characterized in that, Determining the first text content corresponding to the audio data in the audio and video data includes: The first text content is obtained by recognizing human voices in the audio data using a speech recognition model, wherein the speech recognition model is used to recognize the variant language.

3. The method according to claim 1, characterized in that, Determining the text region in the image data includes: Identify at least one sub-text region in the image data, wherein the sub-text region contains text whose deployment angle satisfies a horizontal threshold; The text region is obtained by clustering and merging the at least one sub-text region.

4. The method according to claim 1, characterized in that, Based on the location information and the initial identification confidence level, the third text content is processed to obtain the second text content, including: Based on the location information, subtitle text is filtered out from the image data from the third text content; The subtitle text in the image data is merged to obtain merged text; Based on the initial recognition confidence level, determine the target recognition confidence level corresponding to the merged text; The second text content is obtained based on the target recognition confidence level, the time information corresponding to the image data, and the merged text.

5. The method according to claim 4, characterized in that, Based on the target recognition confidence level, the time information corresponding to the image data, and the merged text, the second text content is obtained, including: Determine the adjacent frame images of the image data; Determine the first similarity between the merged text in the image data and the merged text in the adjacent frame images; In response to the first similarity being greater than a similarity threshold, the merged text in which the target recognition confidence is greater than a confidence threshold is determined as the second text content among the merged text and the merged text corresponding to the adjacent frame images; The text content of the merged text in the image data, and the text content of the merged text in the adjacent frame images, are updated to the second text content.

6. The method according to claim 1, characterized in that, The method further includes: Determine the second start and end times for the second text content.

7. The method according to claim 6, characterized in that, Determine the start and end times of the second text content to obtain the second start and end times, including: In the multi-frame image data of the video data, the first image data in which the text content is the second text content, and the last image data in which the text content is the second text content; The second start and end times are determined based on the time when the first image data appears and the time when the last image data appears.

8. A corpus acquisition device, characterized in that, include: An acquisition unit is used to acquire audio and video data, wherein the audio and video data includes audio data for representing variant languages ​​and image data, the audio and video data is composed of multiple frames of images, the audio data includes dialect audio, and the image data contains subtitles corresponding to the dialect audio; The first determining unit is configured to determine the first text content corresponding to the audio data and the second text content corresponding to the image data, wherein the first text content is used to describe the audio data and the second text content is used to describe the image data. A processing unit is configured to match the first text content and the second text content to obtain matching information, wherein the matching information is used to indicate the degree of matching between the first text content and the second text content; The second determining unit is used to determine, based on the matching information, the target text content that matches the audio data from the first text content and the second text content; A combination unit is used to combine the audio data and the target text content to obtain the target corpus; The second determining unit is configured to determine the target text content that matches the audio data from the first text content and the second text content based on the matching information through the following steps: in response to the matching information indicating that the first text content and the second text content match, the first text content is modified based on the second text content to obtain the target text content; The processing unit is configured to match the first text content and the second text content through the following steps to obtain the matching information: using a speech endpoint detection model to locate human voices in the audio data, obtaining a first start and end time for the first text content corresponding to at least one human voice in the audio data, wherein the speech endpoint detection model is used to determine the start and end points of the human voice in the audio data; extending the first start and end time to obtain a third start and end time; determining at least one matching text content whose start and end time is located within the third start and end time from a plurality of second text contents; and determining the matching degree between the at least one matching text content and the first text content to obtain the matching information. The first determining unit is further configured to perform the following steps: in response to the appearance of a human voice in the audio data, determining the content of the human voice to obtain the first text content; determining a text region in the image data and the location information corresponding to the text region; using a character recognition model to recognize the text region to obtain at least one third text content in the text region and an initial recognition confidence level corresponding to the third text content, wherein the text region is a region in the image data containing text, and the location information is used to determine the region where the text region is located in the image data, the location information including the deployment position and deployment angle of the text region; the character recognition model is constructed based on a semantic visual text relevance algorithm and is used to recognize the text content in the text region; based on the location information and the initial recognition confidence level, the third text content is processed to obtain the second text content.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-volatile storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images