Corpus quality evaluation method and device

Abstract

The invention provides a corpus quality evaluation method and device. The method comprises the following steps: acquiring a corpus text; quality evaluation results of the corpus text are determined, and the quality evaluation results comprise a first quality evaluation result of sentences in the corpus text, a second quality evaluation result of paragraphs and a third quality evaluation result of the corpus text; wherein the first quality evaluation result is obtained by analyzing sentences based on a pre-trained syntactic structure model, the paragraph is obtained by analyzing a semantic relationship among a plurality of sentences based on a pre-trained semantic fragmentation model, and the second quality evaluation result is obtained by analyzing the paragraph through a pre-trained logic association model; the third quality evaluation result is obtained by analyzing the corpus text through a pre-trained information density model. In this way, the model training speed and the model performance are improved through the corpus training model with higher quality.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for corpus quality assessment. Background Technology

[0002] With the development of artificial intelligence technology, corpora are being used more and more widely in business scenarios such as large model training and digital human model training.

[0003] The quality of the corpus usually determines the performance of the trained model. However, high-quality corpora are scarce, and the quality of a large amount of corpus data varies greatly, which limits the upper limit of the model's performance.

[0004] Therefore, evaluating the quality of the corpus and using high-quality corpus for model training is crucial for improving model performance. Summary of the Invention

[0005] This application provides a corpus quality assessment method and apparatus that can evaluate the quality of corpus from multiple dimensions, thereby enabling users to select high-quality corpus to train models and improve the performance of trained models.

[0006] In a first aspect, embodiments of this application provide a corpus quality assessment method, including:

[0007] Obtain the corpus text;

[0008] Determine the quality assessment results of the corpus text, which include the first quality assessment results of sentences, the second quality assessment results of paragraphs, and the third quality assessment results of the corpus text.

[0009] The first quality assessment result is obtained by analyzing sentences based on a pre-trained syntactic structure model, the second quality assessment result is obtained by analyzing paragraphs through a pre-trained semantic segmentation model, and the third quality assessment result is obtained by analyzing corpus text through a pre-trained information density model.

[0010] According to this scheme, sentences in the corpus text are analyzed using a syntactic structure model to obtain the first quality assessment result for each sentence. Semantic segmentation model is used to analyze the semantic relationships between sentences, resulting in multiple paragraphs. Then, a logical association model is used to analyze the paragraphs, yielding the second quality assessment result for each paragraph. Finally, an information density model is used to analyze the information density of the corpus text, resulting in the third quality assessment result for the corpus text. In this way, multi-dimensional quality assessment of the corpus text is possible, with comprehensive assessment dimensions and fine-grained assessment levels. This allows for training the model with higher-quality corpus text, improving model training speed and model performance.

[0011] In one possible implementation, the method further includes:

[0012] The sentences are refined based on the initial quality assessment results for each sentence;

[0013] And / or, refine the paragraphs based on the results of the second quality assessment for each paragraph;

[0014] The purified sentences and / or purified paragraphs are used as training samples.

[0015] In this way, not only can the quality of the corpus text be improved, but the efficiency of model training and the performance of the trained model can also be improved by using the purified text as training samples.

[0016] In one possible implementation, the method further includes:

[0017] Generate metadata for the corpus text, including the quality assessment results of the corpus text.

[0018] In this way, during subsequent model training, high-quality corpus texts can be selected directly based on the quality assessment results of the corpus texts, thereby improving the efficiency of model training and the performance of the trained model.

[0019] In one possible implementation, the method further includes:

[0020] Determine the domain to which the corpus text belongs;

[0021] Metadata can also include domains.

[0022] In this way, the model can be trained based on the domain of the corpus text, making the model's learning of knowledge more accurate.

[0023] In one possible implementation, the domain to which the corpus text belongs is obtained based on a pre-trained domain identification model and the corpus text.

[0024] In one possible implementation, determining the domain to which the corpus text belongs includes:

[0025] Each paragraph is analyzed using a domain-specific identification model to determine the domain to which each paragraph belongs.

[0026] The domain to which the corpus text belongs is determined based on the domain to which each paragraph belongs.

[0027] In this way, the model can be trained based on the domain of the corpus text, making the model's learning of knowledge more accurate.

[0028] In one possible implementation, the method further includes:

[0029] Paragraphs belonging to the same domain are grouped into training texts for model training.

[0030] In this way, the model can be trained based on the domain of the corpus text, making the model's learning of knowledge more accurate.

[0031] Secondly, embodiments of this application provide a corpus quality assessment device, comprising:

[0032] The acquisition module is used to acquire the text from the corpus.

[0033] The determination module is used to determine the quality assessment results of the corpus text, wherein the quality assessment results include the first quality assessment results of sentences, the second quality assessment results of paragraphs, and the third quality assessment results of the corpus text.

[0034] The first quality assessment result is obtained by analyzing sentences based on a pre-trained syntactic structure model, the second quality assessment result is obtained by analyzing paragraphs through a pre-trained semantic segmentation model, and the third quality assessment result is obtained by analyzing corpus text through a pre-trained information density model.

[0035] According to this scheme, sentences in the corpus text are analyzed using a syntactic structure model to obtain the first quality assessment result for each sentence. Semantic segmentation is then used to analyze the semantic relationships between sentences, resulting in multiple paragraphs. Furthermore, a logical association model is used to analyze these paragraphs, yielding the second quality assessment result. Finally, an information density model is used to analyze the information density of the corpus text, resulting in the third quality assessment result. This multi-dimensional quality assessment of the corpus text is comprehensive and refined, allowing users to select higher-quality corpora for model training, thus improving model training speed and performance.

[0036] In one possible implementation, the device further includes:

[0037] The processing module is used to refine sentences based on the first quality assessment result of each sentence;

[0038] And / or, refine the paragraphs based on the results of the second quality assessment for each paragraph;

[0039] The purified sentences and / or purified paragraphs are used as training samples.

[0040] In this way, not only can the quality of the corpus text be improved, but the efficiency of model training and the performance of the trained model can also be improved by using the purified text as training samples.

[0041] In one possible implementation, the device further includes:

[0042] The generation module is used to generate metadata for the corpus text, which includes the quality assessment results of the corpus text.

[0043] In this way, during subsequent model training, high-quality corpus texts can be selected directly based on the quality assessment results of the corpus texts, thereby improving the efficiency of model training and the performance of the trained model.

[0044] In one possible implementation, the determining module is also used to determine the domain to which the corpus text belongs;

[0045] Metadata can also include domains.

[0046] In this way, the model can be trained based on the domain of the corpus text, making the model's learning of knowledge more accurate.

[0047] In one possible implementation, the domain to which the corpus text belongs is obtained based on a pre-trained domain identification model and the corpus text.

[0048] In one possible implementation, the determining module is used for:

[0049] Each paragraph is analyzed using a domain-specific identification model to determine the domain to which each paragraph belongs.

[0050] The domain to which the corpus text belongs is determined based on the domain to which each paragraph belongs.

[0051] In this way, the model can be trained based on the domain of the corpus text, making the model's learning of knowledge more accurate.

[0052] In one possible implementation, the device further includes:

[0053] The classification module is used to classify paragraphs with the same domain as training text for model training.

[0054] In this way, the model can be trained based on the domain of the corpus text, making the model's learning of knowledge more accurate.

[0055] Thirdly, embodiments of this application provide a computing device, including: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is used to execute the method provided in the first aspect.

[0056] Fourthly, embodiments of this application provide a corpus quality assessment apparatus, characterized in that the apparatus executes computer program instructions to perform the method provided in the first aspect. Exemplarily, the apparatus may be a chip or a processor.

[0057] In one example, the device may include a processor that can be coupled to memory, read instructions from the memory, and execute the methods provided in the first aspect according to those instructions. The memory may be integrated into the chip or processor, or it may be independent of the chip or processor.

[0058] Fifthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect.

[0059] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the architecture of a corpus quality assessment system provided in an embodiment of this application;

[0061] Figure 2 This is a flowchart illustrating a corpus quality assessment method provided in an embodiment of this application;

[0062] Figure 3 This is a schematic flowchart of an exemplary corpus quality assessment method provided in an embodiment of this application;

[0063] Figure 4 This is a flowchart illustrating a method for obtaining multi-dimensional quality assessment results of corpus text provided in an embodiment of this application.

[0064] Figure 5 This is a schematic diagram of the structure of a corpus quality assessment device provided in an embodiment of this application;

[0065] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0066] Figure 7 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0067] Figure 8 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0069] In the training of large-scale models and digital human models, a large amount of corpus data is used to enable the models to learn more knowledge. The quality of the corpus data has a significant impact on the training speed and the performance of the trained model. Therefore, how to evaluate the quality of the corpus data so that users can select higher-quality corpus data for model training is crucial to the training speed and the performance of the trained model.

[0070] Based on this, embodiments of this application provide a method, apparatus, device, and storage medium for corpus quality assessment. In this method, sentences in the corpus text are analyzed using a syntactic structure model to obtain a first quality assessment result for each sentence. Semantic segmentation models are used to analyze the semantic relationships between sentences, resulting in multiple paragraphs. Furthermore, logical association models are used to analyze the paragraphs, yielding a second quality assessment result for each paragraph. Finally, information density models are used to analyze the information density of the corpus text, resulting in a third quality assessment result for the corpus text. This allows for multi-dimensional quality assessment of the corpus text, providing comprehensive evaluation dimensions and fine-grained evaluation levels. This enables users to select higher-quality corpus data for model training based on the final quality assessment results, improving model training speed and model performance.

[0071] Figure 1 This is a schematic diagram of the architecture of the corpus quality assessment system provided in the embodiments of this application. For example... Figure 1 As shown, the corpus evaluation system provided in this embodiment includes a computing device cluster 11, a client 12, and a network 13. The client 12 can communicate with the computing device cluster 11 via the network 13.

[0072] In this embodiment, a user can transmit documents, images with text information, voice data, and other information to a computing device cluster 11 via a client 12. In this way, the computing device cluster 11 can perform quality assessment on the corpus corresponding to the documents, images with text information, and voice data transmitted by the user, and transmit the quality assessment results of the corpus to the client 12 for the user to use.

[0073] In this embodiment, the computing device cluster 11 includes at least one computing device. The computing device may be a server, computer, etc. The computing device cluster 11 may be a cloud platform, thereby providing cloud services to implement the corpus quality assessment method provided in this embodiment.

[0074] Figure 2 This is a flowchart illustrating a corpus quality assessment method provided in an embodiment of this application. Figure 2 As shown, the corpus quality assessment method provided in this application includes the following steps S201 and S202.

[0075] S201, Obtain the corpus text.

[0076] The corpus text includes textual information used to record relevant information from different fields. Here, the corpus text is a plain text file.

[0077] In some embodiments, the corpus text is obtained by converting different types of files. For example, the files can be documents in Portable Document Format, Document (Doc) format documents, text (TXT) format documents, etc.

[0078] Specifically, when files of different formats are obtained, information extraction methods corresponding to the file format type are used to parse and extract the information recorded in the file, thereby obtaining the text corpus. For example, if the file is a PDF document, PDF parsing and text recognition algorithms are called to parse and extract all the text information in the PDF document. The extracted text information is then saved as a plain text file. For example, such as... Figure 3 As shown, after receiving document 1, the text information in document 1 is extracted to obtain text 1.

[0079] In some embodiments, the corpus text is extracted from other multimodal data. This multimodal data includes images, speech data, and other data containing textual information. For example, corpus text is obtained by extracting textual information from images. Yet another example is that corpus text is obtained by performing text recognition on speech data.

[0080] In some embodiments, after obtaining the corpus text, operations such as language filtering, garbled character filtering, simple text deduplication, and sensitive word filtering can be performed on the corpus text to achieve preliminary cleaning of the corpus text.

[0081] For example, when uploading data such as documents, images, or audio, users can choose through the client whether preliminary text cleaning of the corpus is required. When the computing device cluster receives an instruction to perform preliminary text cleaning, it performs the cleaning. This reduces the processing burden on the model.

[0082] S202, determine the quality assessment results of the corpus text, wherein the quality assessment results include the first quality assessment results of sentences, the second quality assessment results of paragraphs, and the third quality assessment results of the corpus text; the first quality assessment results are obtained by analyzing sentences based on a pre-trained syntactic structure model, the paragraphs are obtained by analyzing the semantic relationships between multiple sentences based on a pre-trained semantic segmentation model, the second quality assessment results are obtained by analyzing paragraphs through a pre-trained logical association model, and the third quality assessment results are obtained by analyzing the corpus text through a pre-trained information density model.

[0083] The quality assessment results of the corpus text include the quality assessment results of the corpus text in multiple dimensions, namely the first quality assessment result of sentences, the second quality assessment result of paragraphs, and the third quality assessment result of the corpus text.

[0084] In some embodiments, such as Figure 4 As shown, steps S401 to S404 may be included in S202.

[0085] S401: The sentences in the corpus text are sequentially input into a pre-trained syntactic structure model. The syntactic structure model analyzes the sentences and obtains the first quality assessment result of the sentences in the corpus text.

[0086] The first quality assessment result refers to the quality assessment result of the sentence. For example, the first quality assessment result includes the quality score of the sentence.

[0087] Syntactic structure models can analyze and evaluate the correctness and appropriateness of a sentence in terms of its syntax, semantics, and expressive logic. Thus, syntactic structure models can analyze sentences and evaluate their first-order quality assessment based on the correctness and appropriateness of their syntax, semantics, and expressive logic.

[0088] For example, such as Figure 3 As shown, Text 1 contains multiple sentences, namely Sentence 1, Sentence 2, Sentence 3, Sentence 4, etc. These sentences are input into a syntactic structure model, which analyzes them to obtain quality scores. Sentence 1 received a quality score of 86, Sentence 2 received 90, Sentence 3 received 80, and Sentence 4 received 72.

[0089] In this embodiment of the application, after obtaining the first quality assessment result of the sentence, the first quality assessment result of the sentence can be sent to the client.

[0090] It should be noted that multiple sentences can be input into the syntactic structure model sequentially, allowing the model to analyze them sequentially. Alternatively, multiple sentences can be input into the syntactic structure model simultaneously, enabling multi-threaded operation and concurrent analysis of the sentences.

[0091] In this application embodiment, as one possible implementation, the syntactic structure model can analyze each sentence. As another possible implementation, the syntactic structure model can evaluate partial sentences or typical sentences, thereby improving sentence analysis efficiency and saving computational resources. In practical applications, the model can be configured according to actual needs; this application embodiment does not impose specific limitations.

[0092] S402, the sentence is input into the pre-trained semantic segmentation model, which analyzes the semantic relationships between sentences to obtain paragraphs.

[0093] Semantic segmentation models can analyze multiple sentences to identify semantic relationships between them and locate the beginning and end of paragraphs. This enables semantic-level paragraph division, ensuring the semantic integrity of paragraphs and making the content more focused.

[0094] For example, such as Figure 3 As shown, sentences 1, 2, 3, and 4 are input into the semantic segmentation model. The semantic segmentation model analyzes the semantic relationships between sentences to obtain paragraph 1, paragraph 2, paragraph 3, etc.

[0095] S403: Input the paragraph into a pre-trained logical association model. The logical association model performs quality analysis on the paragraph and obtains the second quality assessment result of the paragraph.

[0096] The logical association model can analyze the logic of paragraphs to evaluate their quality. Paragraphs segmented by the semantic fragmentation model are input into the logical association model to obtain a second quality assessment result.

[0097] For example, such as Figure 3 As shown, paragraphs 1, 2, and 3 are input into the logical association model. The logical association model analyzes the quality of paragraphs 1, 2, and 3. For example, some paragraphs may contain advertising information, which would make the semantics of the paragraphs incoherent. The logical association model can evaluate the quality of paragraphs through semantic analysis.

[0098] In this embodiment of the application, as one possible implementation, the logical association model can analyze each paragraph. As another possible implementation, the logical association model can evaluate some paragraphs or typical paragraphs, thereby improving paragraph analysis efficiency and saving computational resources. In practical applications, the model can be configured according to actual needs, and this embodiment of the application does not impose specific limitations.

[0099] S404: Input the corpus text into a pre-trained information density model. The information density model performs quality analysis on the corpus text and obtains the third quality assessment result of the corpus text.

[0100] Information density models can analyze the knowledge-based information density of corpus texts and determine their knowledge content. Thus, by analyzing corpus texts using information density models, an information density score, or third-order quality assessment result, can be obtained. It's understandable that since corpus texts are extracted from documents, the third-order quality assessment result of the corpus texts is also the third-order quality assessment result of the documents.

[0101] In S202, each document can be analyzed across different dimensions, or a subset of documents can be analyzed across different dimensions. For example, if multiple corpus texts originate from the same source, at least one corpus text from the same source can be evaluated. For instance, documents, images, and audio data can carry their own source identifiers; based on these source identifiers, corpus texts from the same source can be identified. This reduces the number of corpus texts that need to be evaluated, thereby improving the efficiency of corpus text evaluation and saving computational resources.

[0102] In some embodiments, the corpus text required for model training is relatively large. Therefore, the data uploaded by users, such as documents, may include various types, such as journals, textbooks, and papers. After receiving the document data, it can be analyzed to filter out documents such as journals, textbooks, and papers. For documents of this type, it can be directly determined that the quality assessment results are high across different dimensions, meaning that these documents are high-quality documents. This reduces the amount of corpus text that needs to be evaluated, thereby improving the efficiency of corpus text evaluation and saving computational resources.

[0103] In some embodiments, the computing device cluster can also transmit quality assessment results to the client, thereby displaying the quality assessment results of the corpus text in multiple dimensions to the user through the client.

[0104] The first quality assessment result for sentences, the second quality assessment result for paragraphs, and the third quality assessment result for the corpus text are transmitted to the client. This allows for a multi-dimensional presentation of the document's assessment results to the user. In some embodiments, the first, second, and third quality assessment results can be visualized to enhance user perception and facilitate a clear understanding of the quality distribution of the corpus text across various dimensions.

[0105] In some embodiments, multiple paragraphs can also be input into a pre-trained domain identification model. The domain identification model analyzes each paragraph to determine the domain to which the corpus text belongs. The domain to which the corpus text belongs is used to indicate the academic field, technical field, etc., to which the corpus text belongs. Specifically, the domain identification model determines the domain to which each paragraph belongs. Based on the domain to which each paragraph belongs, the domain to which the corpus text belongs is determined. For example, as... Figure 3 As shown, the domain identification model performs domain analysis on paragraphs 1, 2, and 3, determining that paragraph 1 belongs to the medical domain, paragraph 2 belongs to the medical domain, and paragraph 3 belongs to the biological domain. Based on the domains to which paragraphs 1 to 3 belong, the domain of the corpus text can be determined to be the medical domain.

[0106] In this embodiment of the application, the computing device cluster can classify paragraphs with the same domain, so that the classified paragraphs can be used as training samples for the model, thereby improving the efficiency of model training.

[0107] In some embodiments, to further improve the speed of model training, the corpus text can be purified and stored as training text for subsequent model training. This not only improves the quality of the text but also allows the purified documents to be directly used as training samples during subsequent model training, thereby saving training data processing time and increasing the iteration rate of model training.

[0108] One possible approach is to refine the sentences based on the initial quality assessment results, thereby obtaining higher-quality sentences. The refined sentences are then stored. This process might involve removing garbled characters or correcting the sentences according to their logical structure. In this way, the goal of data quality refinement can be achieved.

[0109] As another possible approach, each paragraph is refined based on its second quality assessment result, resulting in high-quality paragraphs. The refined paragraphs are then stored. For example, useless information such as embedded advertisements is removed from the paragraphs. This achieves the goal of data quality refinement.

[0110] In some embodiments, the computing device cluster can also generate metadata for the corpus text. The metadata for the corpus text is used to indicate the quality assessment results of the corpus text. Specifically, the metadata for the corpus text includes the quality assessment results of the corpus text in various dimensions. In this way, users can select text from the stored corpus text from different dimensions as training samples for the model based on the metadata of the corpus text. This improves the training efficiency of the model.

[0111] In other embodiments, the metadata of the corpus text may also include different paragraphs and / or the domain to which the corpus text belongs. Thus, it is possible to select paragraphs or corpus text belonging to the desired domain as training text based on the metadata of the corpus text.

[0112] In some embodiments, the syntactic structure model, semantic segmentation model, logical association model, information density model, and domain-specific discrimination model are obtained through pre-training. Specifically, sample corpus text and tag information are acquired, wherein the tag information includes the quality assessment results of each sentence sample, each paragraph sample, and the quality assessment results of the paragraph samples in the sample corpus text, as well as the quality assessment results of the sample corpus text itself.

[0113] Specifically, each sentence sample is analyzed using a syntactic structure model to obtain a quality prediction result for each sentence sample; a first loss value is calculated between the quality prediction result and the quality assessment result of the sentence sample. The syntactic structure model is then trained based on the first loss value.

[0114] The semantic segmentation model analyzes each sentence sample to obtain prediction results for multiple paragraph samples. Based on the prediction results of the paragraph samples and a second loss value between the paragraph samples, the semantic segmentation model is trained.

[0115] Each paragraph sample is analyzed using a logistic regression model to obtain a quality prediction result for each paragraph sample. The logistic regression model is then trained based on a third loss value between the paragraph sample's quality prediction result and its quality assessment result.

[0116] The sample text is analyzed using an information density model to obtain a quality prediction result. The information density model is then trained based on the quality prediction and quality assessment results of the sample text.

[0117] Based on the same concept as the method embodiments of this application, this application also provides a corpus quality assessment device. The corpus quality assessment device includes several modules, each module being used to execute various steps in the corpus quality assessment method provided in this application. The division of modules is not limited here. Those skilled in the art will clearly understand that in practical applications, the various steps in the corpus quality assessment method provided in this application can be assigned to different modules as needed, that is, the internal structure of the device can be divided into different modules to complete all or part of the functions described above. The modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0118] For example, the corpus quality assessment device is used to perform the corpus quality assessment method provided in the embodiments of this application. Figure 5 This is a schematic diagram of the corpus quality assessment device provided in the embodiments of this application. Figure 5 As shown, the corpus quality assessment device provided in this application embodiment includes:

[0119] Module 501 is used to acquire corpus text;

[0120] The determination module 502 is used to determine the quality assessment results of the corpus text, wherein the quality assessment results include the first quality assessment results of sentences, the second quality assessment results of paragraphs, and the third quality assessment results of the corpus text.

[0121] The first quality assessment result is obtained by analyzing sentences based on a pre-trained syntactic structure model, the second quality assessment result is obtained by analyzing paragraphs through a pre-trained semantic segmentation model, and the third quality assessment result is obtained by analyzing corpus text through a pre-trained information density model.

[0122] According to this scheme, sentences in the corpus text are analyzed using a syntactic structure model to obtain the first quality assessment result for each sentence. Semantic segmentation is then used to analyze the semantic relationships between sentences, resulting in multiple paragraphs. Furthermore, a logical association model is used to analyze these paragraphs, yielding the second quality assessment result. Finally, an information density model is used to analyze the information density of the corpus text, resulting in the third quality assessment result. This multi-dimensional quality assessment of the corpus text is comprehensive and refined, allowing users to select higher-quality corpora for model training, thus improving model training speed and performance.

[0123] In one possible implementation, the device further includes:

[0124] The processing module is used to refine sentences based on the first quality assessment result of each sentence;

[0125] And / or, refine the paragraphs based on the results of the second quality assessment for each paragraph;

[0126] The purified sentences and / or purified paragraphs are used as training samples.

[0127] In this way, not only can the quality of the corpus text be improved, but the efficiency of model training and the performance of the trained model can also be improved by using the purified text as training samples.

[0128] In one possible implementation, the device further includes:

[0129] The generation module is used to generate metadata for the corpus text, which includes the quality assessment results of the corpus text.

[0130] In this way, during subsequent model training, high-quality corpus texts can be selected directly based on the quality assessment results of the corpus texts, thereby improving the efficiency of model training and the performance of the trained model.

[0131] In one possible implementation, the determining module is also used to determine the domain to which the corpus text belongs;

[0132] Metadata can also include domains.

[0133] In this way, the model can be trained based on the domain of the corpus text, making the model's learning of knowledge more accurate.

[0134] In one possible implementation, the domain to which the corpus text belongs is obtained based on a pre-trained domain identification model and the corpus text.

[0135] In one possible implementation, the determining module is used for:

[0136] Each paragraph is analyzed using a domain-specific identification model to determine the domain to which each paragraph belongs.

[0137] The domain to which the corpus text belongs is determined based on the domain to which each paragraph belongs.

[0138] In this way, the model can be trained based on the domain of the corpus text, making the model's learning of knowledge more accurate.

[0139] In one possible implementation, the device further includes:

[0140] The classification module is used to classify paragraphs with the same domain as training text for model training.

[0141] In this way, the model can be trained based on the domain of the corpus text, making the model's learning of knowledge more accurate.

[0142] Based on the same concept as the method embodiments of this application, this application also provides a computing device. This computing device is also known as a corpus quality assessment device. The computing device can be a server or a terminal device.

[0143] Figure 6 This is a schematic diagram of the hardware structure of a computing device 600 provided in an embodiment of this application.

[0144] See Figure 6 The computing device 600 includes a chip 601, a memory 602, a communication interface 603, and a bus 604. The chip 601, memory 602, and communication interface 603 are connected to each other via the bus 604. The chip 601, memory 602, and communication interface 603 can also be connected using other methods besides the bus 604.

[0145] The memory 602 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, hard disk, etc.

[0146] Chip 601 can be the aforementioned System-on-a-Chip (SOC), which may include a processor and a connector. The processor can be a general-purpose processor. A general-purpose processor can be a processor that performs specific steps and / or operations by reading and executing contents stored in memory (e.g., memory 602). For example, a general-purpose processor can be a central processing unit (CPU). Chip 601 may include at least one circuit to perform... Figure 3 or Figure 6 All or part of the steps of the method provided in the illustrated embodiments.

[0147] The communication interface 603 includes input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the computing device 600, as well as interfaces for interconnecting the computing device 600 with other devices (such as other computing devices or user equipment). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.

[0148] The bus 604 can be any type of communication bus used to interconnect the chip 601, memory 602 and communication interface 603, such as a system bus.

[0149] The aforementioned devices can be disposed on separate chips, or at least partially or entirely on the same chip. Whether to dispose of the devices independently on different chips or integrate them on one or more chips often depends on the needs of the product design. This application does not limit the specific implementation of the aforementioned devices.

[0150] Figure 6 The computing device 600 shown is merely exemplary. In its implementation, the computing device 600 may also include other components, which will not be listed here.

[0151] based on Figure 2 In addition to the method shown, this application also provides a computing device cluster.

[0152] Figure 7 This is a computing device cluster 700 provided in an embodiment of this application. For example... Figure 7 As shown, the computing device cluster includes at least one computing device 600. The memory 602 of one or more computing devices 600 in the computing device cluster may store the same memory for executing... Figure 2The instructions for the method are shown. The acquisition module 501 and determination module 502 in the corpus quality assessment device can be deployed on a single computing device 600 or distributed across multiple computing devices 600. In the distributed deployment scenario, the acquisition module 501 can be deployed on a single computing device or distributed across multiple computing devices 600; similarly, the determination module 502 can also be deployed on a single computing device 600 or distributed across multiple computing devices 600.

[0153] The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0154] In some possible implementations, the memory 602 of one or more computing devices 600 in the computing device cluster may also store data for execution. Figure 2 The instructions of the method shown are partial. In other words, a combination of one or more computing devices 600 can coexist for executing... Figure 2 The instructions for the method shown.

[0155] It should be noted that the memory 602 in different computing devices 600 within the computing device cluster can store different instructions, each for execution. Figure 5 The illustrated device performs some of its functions. Specifically, the instructions stored in the memory 602 of different computing devices 600 can implement... Figure 5 The function of one or more modules in the device shown.

[0156] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 8 One possible implementation is shown. For example... Figure 8 As shown, two computing devices 600A and 600B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 602 in computing device 600A stores instructions for implementing the function of the acquisition module 501. Simultaneously, the memory 602 in computing device 600B stores instructions for implementing the function of the determination module 502.

[0157] It should be understood that Figure 8 The functions of computing device 600A shown can also be performed by multiple computing devices 600. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600.

[0158] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0159] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. It should be understood that in the embodiments of this application, the order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0160] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

[0161] In addition to the methods, apparatus, and computing devices described above, embodiments of this application may also provide a computer program product, comprising computer program instructions. When executed by a processor, these computer program instructions cause the processor to perform the steps of the methods in the various embodiments of this application described in the "Method" section of this specification. The computer program product may be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The computer program code may be in source code form, object code form, executable file, or some intermediate form. The computer program code may be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0162] Furthermore, embodiments of this application may also provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the display control method according to various embodiments of this disclosure as described in the "Method" section above. The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, a computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0163] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0164] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0165] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0166] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0167] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of the various embodiments of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0168] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0169] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0170] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

[0171] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A corpus quality assessment method, characterized in that, The method includes: Obtain the corpus text; Determine the quality assessment results of the corpus text, wherein the quality assessment results include a first quality assessment result of sentences, a second quality assessment result of paragraphs, and a third quality assessment result of the corpus text; The first quality assessment result is obtained by analyzing the sentence based on a pre-trained syntactic structure model, the paragraph is obtained by analyzing the semantic relationships between multiple sentences based on a pre-trained semantic segmentation model, the second quality assessment result is obtained by analyzing the paragraph through a pre-trained logical association model, and the third quality assessment result is obtained by analyzing the corpus text through a pre-trained information density model.

2. The method according to claim 1, characterized in that, The method further includes: The sentences are refined based on the first quality assessment results for each sentence; And / or, based on the second quality assessment results of each paragraph, the paragraphs are purified; The purified sentences and / or purified paragraphs are used as training samples.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Generate metadata for the corpus text, the metadata including the quality assessment results of the corpus text.

4. The method according to claim 3, characterized in that, The method further includes: Determine the domain to which the corpus text belongs; The metadata may also include the domain.

5. The method according to claim 4, characterized in that, The domain to which the corpus text belongs is obtained based on a pre-trained domain identification model and the corpus text.

6. The method according to claim 5, characterized in that, Determining the domain to which the corpus text belongs includes: Each paragraph is analyzed using the domain-specific identification model to determine the domain to which each paragraph belongs. The domain to which the corpus text belongs is determined based on the domain to which each paragraph belongs.

7. The method according to claim 6, characterized in that, The method further includes: Paragraphs belonging to the same domain are categorized as training texts for model training.

8. A corpus quality assessment device, characterized in that, The device includes: The acquisition module is used to acquire the text from the corpus. A determining module is used to determine the quality assessment result of the corpus text, wherein the quality assessment result includes a first quality assessment result of sentences, a second quality assessment result of paragraphs, and a third quality assessment result of the corpus text; The first quality assessment result is obtained by analyzing the sentence based on a pre-trained syntactic structure model, the paragraph is obtained by analyzing the semantic relationships between multiple sentences based on a pre-trained semantic segmentation model, the second quality assessment result is obtained by analyzing the paragraph through a pre-trained logical association model, and the third quality assessment result is obtained by analyzing the corpus text through a pre-trained information density model.

9. The apparatus according to claim 8, characterized in that, The device further includes: The processing module is used to refine the sentences based on the first quality assessment result of each sentence; And / or, based on the second quality assessment results of each paragraph, the paragraphs are purified; The purified sentences and / or purified paragraphs are used as training samples.

10. The apparatus according to claim 8 or 9, characterized in that, The device further includes: A generation module is used to generate metadata for the corpus text, the metadata including the quality assessment results of the corpus text.

11. The apparatus according to claim 10, characterized in that, The determining module is also used to determine the domain to which the corpus text belongs; The metadata may also include the domain.

12. The apparatus according to claim 11, characterized in that, The domain to which the corpus text belongs is obtained based on a pre-trained domain identification model and the corpus text.

13. The apparatus according to claim 12, characterized in that, The determining module is used for: Each paragraph is analyzed using the domain-specific identification model to determine the domain to which each paragraph belongs. The domain to which the corpus text belongs is determined based on the domain to which each paragraph belongs.

14. The apparatus according to claim 13, characterized in that, The device further includes: The classification module is used to classify paragraphs in the same domain as training texts for model training.

15. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-7.

16. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-7.

17. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1-7.

Images