Method, apparatus, device and storage medium for constructing dataset

Abstract

The disclosure provides a method, device and equipment for constructing a dataset, and a storage medium, relates to the technical field of computers, in particular to the fields of natural language processing, cloud computing, deep learning and the like. The specific implementation scheme is as follows: determining a target topic according to a topic-word distribution matrix output by a topic model based on a text set; determining a target text according to a text-topic distribution matrix output by the topic model based on the text set by using the target topic, wherein the target text is derived from the text set; and constructing a dataset based on the target text. According to the scheme of the disclosure, the target text required for constructing the dataset can be quickly and accurately screened from the text set.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to the fields of natural language processing, cloud computing, deep learning, etc. BACKGROUND

[0002] During model training, a pre-labeled data set needs to be relied on as model input, so as to further realize optimization of the model. The quality of the pre-labeled data set will directly affect the efficiency of model training and the effect of the optimized model. The quality of the pre-labeled data set mainly depends on the quality of data in a to-be-labeled data set constructed, and therefore, it is very important to construct the to-be-labeled data set. SUMMARY

[0003] The present disclosure provides a method, device and equipment for constructing a data set and a storage medium.

[0004] According to an aspect of the present disclosure, a method for constructing a data set is provided, comprising:

[0005] determining a target topic according to a topic-word distribution matrix output by a topic model based on a text set;

[0006] determining a target text according to a text-topic distribution matrix output by the topic model based on the text set by using the target topic, wherein the target text is derived from the text set;

[0007] constructing a data set based on the target text.

[0008] According to another aspect of the present disclosure, a device for constructing a data set is provided, comprising:

[0009] a first determining module configured to determine a target topic according to a topic-word distribution matrix output by a topic model based on a text set;

[0010] a second determining module configured to determine a target text according to a text-topic distribution matrix output by the topic model based on the text set by using the target topic, wherein the target text is derived from the text set;

[0011] a constructing module configured to construct a data set based on the target text.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0013] at least one processor; and

[0014] a memory in communication with the at least one processor; wherein

[0015] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method in any embodiment of the present disclosure.

[0016] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method in any embodiment of the present disclosure.

[0017] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method in any embodiment of the present disclosure.

[0018] According to the scheme of the present disclosure, the target text required for constructing the data set can be quickly and accurately screened from the text set.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0021] Figure 1 is a schematic diagram of the method for constructing the data set according to the embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of the subject-word distribution matrix according to the embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of the hardware scene application of the method for constructing the data set according to the embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of the method for constructing the data set according to another embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of the method for constructing the data set according to another embodiment of the present disclosure;

[0026] Figure 6 is a schematic diagram of the method for constructing the data set according to another embodiment of the present disclosure;

[0027] Figure 7 is a schematic diagram of the method for constructing the data set according to another embodiment of the present disclosure;

[0028] Figure 8 is a schematic diagram of the method for constructing the data set according to another embodiment of the present disclosure;

[0029] Figure 9 is a schematic diagram of a method for constructing a data set according to another embodiment of the present disclosure;

[0030] Figure 10 is a schematic diagram of an apparatus for constructing a data set according to an embodiment of the present disclosure;

[0031] Figure 11 is a block diagram of an electronic device for implementing the method for constructing a data set according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, which should be considered in a descriptive sense only. Thus, it will be apparent to those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0033] Embodiments of the present disclosure provide a method for constructing a data set, as shown in Figure 1 is a flowchart of a method for constructing a data set according to an embodiment of the present disclosure, which can include the following steps:

[0034] Step S101: determining a target topic based on a topic-word distribution matrix output by a topic model based on a text set.

[0035] Step S102: determining a target text based on a text-topic distribution matrix output by a topic model based on a text set using the target topic, wherein the target text is derived from the text set.

[0036] Step S103: constructing a data set based on the target text.

[0037] The topic model of the present disclosure can adopt any topic model, which is not specifically limited herein. Algorithms of the topic model include, but are not limited to, LFM (Latent Factor Model), LSI (Latent Semantic Analysis), PLSI (Probabilistic Latent Semantic Analysis), and LDA (Latent Dirichlet Allocation). In the field of machine learning and natural language processing, the topic model is a statistical model that can be used to find abstract topics in a series of documents. If an article has a central idea, some specific words will appear more frequently. For example, if an article is about dogs, words such as "dog" and "bone" will appear more frequently. If an article is about cats, words such as "cat" and "fish" will appear more frequently. Some words such as "this" and "and" will appear approximately equally in two articles. However, the reality is that an article usually contains multiple topics, and each topic has a different proportion. Therefore, if an article is 10% about cats and 90% about dogs, the number of times the keywords related to dogs will be approximately 9 times the number of times the keywords related to cats. The topic model attempts to reflect this feature of the document in a mathematical framework. The topic model can automatically analyze the statistics of each word in the document, determine the topic-word distribution and the doc-topic distribution, and determine which words each topic has, which topics the document contains, and the proportion of each topic according to the statistical information.

[0038] The documents contained in the document set of the present disclosure can be articles or sentences, and the documents in the document set are adjusted according to the granularity needs of the constructed data set. For example, if the data granularity of the data set to be constructed is an article, each document in the document set can be an article. If the data granularity of the data set to be constructed is a sentence, each document in the document set can be a sentence.

[0039] The topic-word distribution matrix and the text-topic distribution matrix are two results output by the topic model based on the input model of the document set. The topic-word distribution matrix corresponds to the topic-word distribution, and the text-topic distribution matrix corresponds to the doc-topic distribution. As shown in the following table, the topic-word distribution matrix and the text-topic distribution matrix are two results output by the topic model based on the input model of the document set. Figure 2An example of the topic-word distribution matrix is shown, each row in the matrix represents a topic, and according to the topic-word distribution matrix, which words constitute a topic can be determined. Similarly, an example of the text-topic distribution matrix is shown, each row in the matrix can represent a text, and each column can represent a topic, and according to the text-topic distribution matrix, which topics each text includes and the proportion of each topic (for example, document A includes three topics of art, sports and entertainment, and the proportion of art-related description is 50%, the proportion of sports-related description is 40%, and the proportion of entertainment-related description is 10%) can be determined.

[0040] The target topic of the present disclosure can understand that part of the data in the required constructed data set needs to contain the topic. Step S101 can be understood as: according to the words corresponding to each topic in the topic-word distribution matrix, determine which topics are the required target topics.

[0041] Step S102 of the present disclosure can be understood as: based on the topics contained in each text in the text-topic distribution matrix, finding the topics matching the target topic, and determining the texts corresponding to the topics matching the target topic as target texts. Finding the topics matching the target topic can be understood as: as long as the text in the text-topic distribution matrix contains the target topic, it is considered that the topic matching the target topic is found. It can also be understood that: after each topic in the text-topic distribution matrix is filtered in a predetermined manner, only when the target topic is contained in the topic filtered from each text, it is considered that the topic matching the target topic is found.

[0042] The number of target texts contained in the data set constructed by the present disclosure can be selected and adjusted as needed. Each target text in the constructed data set can contain different target topics. That is to say, multiple target topics can be determined based on the topic-word distribution matrix of step S101, and then based on each target topic, one or more target texts corresponding to each target topic can be determined from the text-topic distribution matrix through step S102.

[0043] According to the scheme of the present disclosure, the target text required for constructing the data set can be quickly and accurately screened from the text set. The scheme of the present disclosure can quickly explore the high-value business topics and possible required topics by using each word in the topic-word distribution matrix, and accurately extract the required text to be labeled from the document set based on these topics, thereby realizing the preliminary screening of the document set using the topic model, reducing the noise influence of low-value useless text in the constructed data set, generating a higher-quality data set, and reducing the cost of subsequent labeling of the data set. At the same time, the use of each word in the topic-word distribution matrix quickly explores the high-value business topics, ensuring the diversity of the words used when screening topics and the diversity of the topics screened, and avoiding the problems of limited divergence ability and limited vocabulary when manually setting keywords for screening. In the current diverse Internet scene, the scheme of the present disclosure can quickly and accurately understand the main content topics of the text set, and guide the division of reasonable data sets to be labeled according to the actual data distribution, and further define accurate label ranges based on topics. The data set constructed by the scheme of the present disclosure can be used to train various models, such as text classification supervised machine learning models and deep learning models.

[0044] Generally, there are two methods for constructing a data set. One is to predefine labels that meet the scene text classification according to artificial historical experience or to set labels according to task definition, and then randomly sample a specified number of documents from the existing text set for manual labeling. However, the random sampling method cannot guarantee the balanced distribution of each category topic, i.e., it may result in a data set that is mostly useless topics or labels, rather than high-value business topics or labels. Using the random sampling method to construct a data set will waste a lot of manpower and time in the subsequent manual labeling phase to deal with useless data in the data set. The other is to construct keywords based on background knowledge, and use the text containing the keywords as the data set to be labeled. For example, in the text sentiment classification task, keywords expressing positive emotions (haha, happy, delighted, and go) or keywords expressing negative emotions (hate, cry, and dislike) can be used as seeds to recall documents containing these keywords, and then manually label them accurately. However, using keywords to recall text may result in a large number of false positives. For example, using "like" as a keyword to recall positive sentiment text may result in the recall of text containing "dislike" because they both contain the same keyword, but the sentiment expressions of these two texts are opposite. In addition, simply using keywords as a way to recall different categories of text may result in the loss of document diversity in each category because this method depends on the number and richness of the vocabulary that can be exhausted. The scheme of the present disclosure can completely overcome the drawbacks of the above two methods for constructing a data set.

[0045] In one example, the method for constructing a data set provided by the embodiments of the present disclosure can be applied to the scenario framework as shown in Figure 3 In Figure 3 which, 10 represents a first client, 20 represents a server, 30 represents a distributed computer system, and 40 represents a second client. The method for constructing a data set provided by the embodiments of the present disclosure can be performed by the server 20 or the distributed computer system 30, and the first client 10 is configured to send a text set to the server 20 or the distributed computer system 30. After the server 20 or the distributed computer system 30 completes the construction of the data set, the server 20 or the distributed computer system 30 can feed back the data set to the first client 10, so that the first client 10 performs subsequent processing based on the constructed data set, such as labeling the data set, inputting the labeled data set into a to-be-trained model, and the like. After the server 20 or the distributed computer system 30 completes the construction of the data set, the server 20 or the distributed computer system 30 can also send the data set to the second client 40, so that the second client 40 performs subsequent processing based on the constructed data set, such as labeling the data set, inputting the labeled data set into a to-be-trained model, and the like.

[0046] In one embodiment, as shown in Figure 4 The method for constructing a data set provided by the embodiments of the present disclosure includes the above steps S101-S103, wherein before step S101: determining a target topic based on a topic-word distribution matrix output by a topic model based on a text set, the method can further include the following steps:

[0047] Step S401: performing word segmentation preprocessing on a plurality of texts to generate a text set.

[0048] Step S402: inputting the text set into a topic model.

[0049] The specific word segmentation manner of the word segmentation preprocessing of the present disclosure can be selected and adjusted as needed, for example, word segmentation processing according to words, word segmentation processing according to phrases, and the like. The source and quantity of the text that needs to be segmented can be selected and adjusted as needed, which is not specifically limited herein.

[0050] According to the scheme of the present disclosure, by preprocessing the texts in the text set, the topic model can more efficiently and accurately calculate the text set, and ensure the accuracy of the output topic-word distribution matrix and text-topic distribution matrix.

[0051] In one embodiment, as shown in Figure 5 The method for constructing a data set provided by the embodiments of the present disclosure includes the above steps S101-S103, wherein step S101: determining a target topic based on a topic-word distribution matrix output by a topic model based on a text set, can further include:

[0052] Step S501: According to the topic-word distribution matrix output by the topic model based on the text set, determine the word set corresponding to each topic in the topic-word distribution matrix.

[0053] Step S502: If the word set contains a threshold number of target words, determine that the topic corresponding to the word set is the target topic, wherein the target words are the words corresponding to the preset topic.

[0054] The word set corresponding to each topic of the present disclosure can be understood with reference to Figure 2 As shown in the figure, each row can be understood as a word set corresponding to a topic.

[0055] The preset topic of the present disclosure can be understood as a specific topic type or a general topic type direction that needs to be determined in advance. Based on this, the words that can be contained in the preset topic are further determined. For example, if the preset topic is a specific entertainment topic, according to historical experience and existing data, the entertainment topic can contain words such as stars, variety shows, red carpets, singing, TV dramas, etc. If the preset topic is a general topic type direction "education", according to historical experience and existing data, the education topic can contain different disciplines, so the words can include words such as civil servant examination, college entrance examination, Chinese, composition, English, oral English, and Olympiad.

[0056] The threshold number of target words of the present disclosure can be set as needed, and the specific number is not limited herein. For example, if the threshold number is 5 words, if the word set corresponding to the topic contains 5 target words, it is determined that the topic is the target topic.

[0057] The determination of the topic corresponding to the word set as the target topic can be understood as the determination of the topic number (topic_index) corresponding to the position of the word set in the topic-word distribution matrix.

[0058] For the convenience of understanding, the step S502 of the present disclosure is exemplarily described. For example, if the preset topic is sports, the target words corresponding to the sports topic can include: skiing, basketball, athletes, opening ceremony, skating, gymnastics, fencing, football, table tennis, badminton, etc. If a word set in the topic-word distribution matrix includes "skiing, basketball, skating, gymnastics" 5 target words (threshold number is 4), it is considered that the topic corresponding to the word set may be the preset topic, and the topic is determined as the target topic.

[0059] For example, if the preset theme is set to be education, the target words corresponding to the education direction can include: civil servant examination, college entrance examination, Chinese, composition, English, oral English, Olympiad, and study abroad. If a word set in the theme-word distribution matrix includes four target words (threshold number is four) of "composition, English, oral English, and Olympiad", it is considered that the theme corresponding to the word set can be the preset theme, and the theme is determined as the target theme.

[0060] According to the scheme of the present disclosure, the words contained in the word set in the theme-word distribution matrix can be used to quickly and accurately find the target theme matching the preset theme from the theme-word distribution matrix, and then the corresponding documents can be found by using the target theme for data set construction.

[0061] In an embodiment, as shown in Figure 6 The method for constructing a data set provided by the embodiment of the present disclosure includes the above steps S101 to S103, and step S102 of determining the target text according to the text-theme distribution matrix output by the text set based on the theme model using the target theme can further include:

[0062] Step S601: determining the theme matching the target theme in the text-theme distribution matrix according to the text-theme distribution matrix output by the text set based on the theme model.

[0063] Step S602: determining the text corresponding to the theme matching the target theme in the text-theme distribution matrix as the target text.

[0064] In the present disclosure, the theme matching the target theme can be understood as a theme in the text-theme distribution matrix being consistent or approximate with the target theme.

[0065] The text-theme distribution matrix is a matrix composed of themes and texts. If the row is defined as the text number and the column is defined as different themes, each row in the matrix represents the themes contained in a text, and each column in the matrix represents the texts in which each theme appears.

[0066] For ease of understanding, the steps S601 and S602 of the present disclosure are exemplarily described. If the target theme is sports, the theme matching the sports in the text-theme distribution matrix is indexed, and if the theme matching the sports (for example, the themes of sports, swimming, and basketball) is indexed, the text corresponding to the matched theme is further determined, for example, the theme appears in the first row, the third row, and the fifth row in the text-theme distribution matrix, and the text corresponding to the first row, the third row, and the fifth row is screened out as the target text.

[0067] According to the scheme of the present disclosure, by utilizing the matching relationship between the topics corresponding to each text in the text-topic distribution matrix and the target topic, the text containing the target topic or the preset topic required for constructing the data set can be quickly and accurately found from the text-topic distribution matrix.

[0068] In an embodiment, as shown in FIG. 1, the method for constructing a data set provided by the embodiment of the present disclosure comprises the above steps S101-S103, wherein step S102 of determining the target text according to the text-topic distribution matrix output by the text set based on the topic model by utilizing the target topic can further comprise: Figure 7

[0069] Step S701: determining the topic proportion of each topic contained in each text according to the text-topic distribution matrix output by the text set based on the topic model.

[0070] Step S702: determining the top N topics with large topic proportions in each text, wherein N ≥ 1, and N is an integer.

[0071] Step S703: in the case where the top N topics include the topic matching the target topic, determining the text corresponding to the top N topics as the target text.

[0072] In the present disclosure, the topic matching the target topic can be understood as the topic in the text-topic distribution matrix being consistent or approximate with the target topic.

[0073] In the present disclosure, the value of N can be selected and adjusted as needed. For example, when N = 1, the topic with the largest proportion in each text is matched with the target topic, and then the text is screened. For another example, when N = 2, the top two topics with large proportions in each text are matched with the target topic, and then the text is screened.

[0074] The text-topic distribution matrix of the present disclosure not only shows the topics contained in each text, but also further shows how much proportion each topic occupies in the text. For example, a text contains three topics, A topic occupies 30%, B topic occupies 50%, and C topic occupies 20%. When N = 1, the topic with the largest proportion B is matched with the target topic to determine whether the text can be used as the target text.

[0075] ​According to the scheme of the present disclosure, by using the proportion of the topic corresponding to each text in the text-topic distribution matrix, some noise texts, i.e. texts containing a topic matching the target topic but with too small a proportion of the topic, can be pre-screened before matching the target topic. These screened texts indicate that the focus is not on describing the content related to the target topic. The scheme of the present disclosure can quickly and accurately find the text containing the target topic or the preset topic from the text-topic distribution matrix for constructing the data set.

[0076] In one example, in the case of determining the target topic according to the word set, the topic index corresponding to the position of the word set in the topic-word distribution matrix is determined.

[0077] According to the topic probability of each document in the text-topic distribution matrix and the target topic, the argmax of the topic probability vector of each document is determined, i.e. the topic with the largest topic probability of each document is determined.

[0078] According to the topic_index of the target topic, the topic corresponding to the topic_index of the target topic is determined as the target text in the topic with the largest topic probability of each document.

[0079] The present disclosure determines that the topic corresponding to the word set is the target topic, which can be understood as determining the topic index corresponding to the position of the word set in the topic-word distribution matrix.

[0080] In one embodiment, as shown in Figure 8 The method for constructing a data set provided by the embodiment of the present disclosure includes the above steps S101 to S103, wherein step S103: constructing a data set based on the target text, can further include:

[0081] Step S801: determining the label category corresponding to the target topic according to the word set corresponding to the target topic.

[0082] Step S802: associating the label category with the target text.

[0083] Step S803: constructing a data set based on the target text associated with the label category.

[0084] The label of the present disclosure can be understood as a higher concept of the topic. For example, the label is education, and the topic is civil service examination, judicial examination, college entrance examination, mathematics, Chinese, English writing, etc.

[0085] According to the scheme of the present disclosure, the specific topic of the target topic and the label corresponding to the target topic can be determined through the words in the word set corresponding to the target topic. By identifying the label of the target text, the information content contained in the constructed data set can be more comprehensive. This facilitates quick determination of the division and labeling method of the data in the data set when labeling the data set subsequently.

[0086] In one application example, as shown in Figure 9 The method for constructing a data set provided by the embodiments of the present disclosure includes:

[0087] segmenting a plurality of documents and constructing a data set;

[0088] inputting the data set into a topic model, the topic model outputting a topic-word distribution matrix and a text-topic distribution matrix based on the document set;

[0089] determining a target topic and the label type (category) of the topic through the word set in the topic-word distribution matrix, and obtaining the topic_index (topic number) of the target topic;

[0090] determining the argmax (arguments of the maxima) of the topic probability vector of each document, i.e., determining the topic with the maximum topic probability of each document, according to the topic probability of each document in the text-topic distribution matrix and the target topic;

[0091] indexing the target document according to the topic_index of the target topic and the argmax of the topic probability vector of each document;

[0092] generating a data set to be labeled based on the target document.

[0093] The embodiments of the present disclosure provide a device for constructing a data set, as shown in Figure 10 The device for constructing a data set of the present embodiment has a structure block diagram, which can include:

[0094] The first determination module 1001 is configured to determine a target topic according to the topic-word distribution matrix output by the topic model based on the text set.

[0095] The second determination module 1002 is configured to determine a target text according to the text-topic distribution matrix output by the topic model based on the text set by using the target topic, wherein the target text is derived from the text set.

[0096] The construction module 1003 is configured to construct a data set based on the target text.

[0097] In an implementation, the apparatus for constructing a data set further comprises:

[0098] a preprocessing module configured to perform word segmentation preprocessing on the plurality of texts to generate a text collection.

[0099] an input module configured to input the text collection into a topic model.

[0100] In an implementation, the first determining module comprises:

[0101] a first determining submodule configured to determine, according to a topic-word distribution matrix output by the topic model based on the text collection, a word set corresponding to each topic in the topic-word distribution matrix.

[0102] a second determining submodule configured to determine, in a case where the word set contains a threshold number of target words, that a topic corresponding to the word set is a target topic, wherein the target words are words corresponding to a preset topic.

[0103] In an implementation, the second determining module comprises:

[0104] a third determining submodule configured to determine, according to a text-topic distribution matrix output by the topic model based on the text collection, a topic matching the target topic in the text-topic distribution matrix.

[0105] a fourth determining submodule configured to determine, as a target text, a text corresponding to the topic matching the target topic in the text-topic distribution matrix.

[0106] In an implementation, the second determining module comprises:

[0107] a fifth determining submodule configured to determine, according to the text-topic distribution matrix output by the topic model based on the text collection, a topic proportion of each topic contained in each text.

[0108] a sixth determining submodule configured to determine a top N topics with a large topic proportion in each text, wherein N≥1, and N is an integer.

[0109] a seventh determining submodule configured to determine, as the target text, a text corresponding to the top N topics in a case where the top N topics include a topic matching the target topic.

[0110] In an implementation, the constructing module comprises:

[0111] an eighth determining submodule configured to determine, according to a word set corresponding to the target topic, a label category corresponding to the target topic.

[0112] a label association submodule configured to associate the label category with the target text.

[0113] The constructing submodule is configured to construct a data set based on the target text associated with the label category.

[0114] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information are in line with relevant laws and regulations and do not violate public order and good customs.

[0115] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0116] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0117] As shown in Figure 11 The electronic device 1100 includes a computing unit 1101 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the electronic device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0118] Various components in the electronic device 1100 are connected to the I / O interface 1105, including an input unit 1106, such as a keyboard, a mouse, etc., an output unit 1107, such as various types of displays, a speaker, etc., a storage unit 1108, such as a magnetic disk, an optical disk, etc., and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0119] The computing unit 1101 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs various methods and processes described above, such as the method of constructing a dataset. For example, in some embodiments, the method of constructing a dataset can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded onto the RAM 1103 and executed by the computing unit 1101, one or more steps of the method of constructing a dataset described above can be performed. Alternatively, in other embodiments, the computing unit 1101 can be configured to perform the method of constructing a dataset by any other suitable means, such as by means of firmware.

[0120] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0121] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable computing apparatus to produce a machine, such that the program code, when executed by the processor or controller, implements the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0122] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0124] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0125] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0126] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0127] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for constructing a dataset, comprising: Based on the topic-word distribution matrix output by the topic model based on the text set, the word set corresponding to each topic in the topic-word distribution matrix is ​​determined, and the topic corresponding to the word set is determined as the target topic. The rows and columns of the topic-word distribution matrix correspond to different topics and the words included in each topic, respectively. Using the target topic, the target text is determined based on the text-topic distribution matrix output by the topic model based on the text set, wherein the target text originates from the text set, and the rows and columns of the text-topic distribution matrix correspond to different texts and the topics included in each text, respectively. Based on the target text, construct a dataset; The step of determining the target text using the target topic and based on the text-topic distribution matrix output by the topic model based on the text set includes: Based on the text-topic distribution matrix output by the topic model based on the text set, determine the topics in the text-topic distribution matrix that match the target topic; and determine the texts corresponding to the topics that match the target topic in the text-topic distribution matrix as the target texts. or, Based on the text-topic distribution matrix output by the topic model based on the text set, determine the topic proportion of each topic contained in each text; determine the top N topics with the largest topic proportion in each text, where N≥1, and N is an integer; if the top N topics include topics that match the target topic, determine the text corresponding to the top N topics as the target text.

2. The method according to claim 1, further comprising, before determining the target topic based on the topic-word distribution matrix output by the topic model based on the text set, the method includes: Perform word segmentation preprocessing on multiple texts to generate a text set; Input the text set into the topic model.

3. The method according to claim 1, wherein determining the topic corresponding to the word set as the target topic includes: If the word set contains a threshold number of target words, the topic corresponding to the word set is determined as the target topic, wherein the target words are words corresponding to the preset topic.

4. The method according to claim 1, wherein constructing the dataset based on the target text includes: Based on the set of words corresponding to the target topic, determine the tag category corresponding to the target topic; Associate the tag category with the target text; A dataset is constructed based on the target text associated with the aforementioned label categories.

5. An apparatus for constructing a dataset, comprising: The first determining module is used to determine the word set corresponding to each topic in the topic-word distribution matrix based on the topic model output by the text set, and to determine the topic corresponding to the word set as the target topic, wherein the rows and columns of the topic-word distribution matrix correspond to different topics and the words included in each topic, respectively; The second determining module is used to determine the target text by utilizing the target topic and based on the text-topic distribution matrix output by the topic model based on the text set, wherein the target text originates from the text set, and the rows and columns of the text-topic distribution matrix correspond to different texts and the topics included in each text; A building module is used to construct a dataset based on the target text; The second determining module includes: The third determining submodule is used to determine the topic that matches the target topic in the text-topic distribution matrix based on the text set output by the topic model; the fourth determining submodule is used to determine the text corresponding to the topic that matches the target topic in the text-topic distribution matrix as the target text. or, The second determining module includes: The fifth determining submodule is used to determine the proportion of each topic contained in each text based on the text-topic distribution matrix output by the topic model based on the text set; the sixth determining submodule is used to determine the top N topics with the largest proportion in each text, where N≥1 and N is an integer; the seventh determining submodule is used to determine the text corresponding to the top N topics as the target text if the top N topics include topics that match the target topic.

6. The apparatus according to claim 5, further comprising: The preprocessing module is used to perform word segmentation preprocessing on multiple texts to generate a text set; The input module is used to input the text set into the topic model.

7. The apparatus according to claim 5, wherein the first determining module comprises: The second determining submodule is used to determine the topic corresponding to the word set as the target topic when the word set contains a threshold number of target words, wherein the target words are words corresponding to a preset topic.

8. The apparatus according to claim 5, wherein the building module comprises: The eighth determining submodule is used to determine the tag category corresponding to the target topic based on the set of words corresponding to the target topic; The tag association submodule is used to associate tag categories with the target text; The construction submodule is used to build a dataset based on target text associated with the stated label categories.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 4.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images