Title deduplication method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a question deduplication method, device, electronic equipment and storage medium. The method comprises: obtaining question information corresponding to each question in a to-be-deduplicated question bank; dividing the to-be-deduplicated question bank into a seed question bank and a non-seed question bank according to the question information, wherein the question information of any two seed questions in the seed question bank is different; calculating a first similarity between a first non-seed question in the non-seed question bank and each seed question in the seed question bank, the first non-seed question being any non-seed question in the non-seed question bank; in response to a maximum value in the first similarity being less than a first threshold, adding the first non-seed question to the seed question bank; and in response to all non-seed questions in the non-seed question bank being traversed, determining the seed question bank as a question bank after deduplication of the to-be-deduplicated question bank. The present scheme improves the question deduplication quality and ensures the reliability of the question bank.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a question deduplication method and device, electronic equipment and storage medium. BACKGROUND

[0002] In a question search and recommendation system, questions are obtained from a question bank for recommendation to users. If there are many duplicate questions in the question bank, the questions searched may be the same, which makes the questions recommended to users duplicate, resulting in a poor experience for question recommendation. Therefore, it is very important to deduplicate the question bank.

[0003] In related technologies, deduplication of the question bank is usually achieved by calculating the text similarity. Questions with a text similarity higher than a preset value are determined as duplicate questions and removed. The reliability of the deduplicated question bank obtained by this method is not high. SUMMARY

[0004] To solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a question deduplication method, device, electronic equipment and storage medium.

[0005] According to an aspect of the present disclosure, a question deduplication method is provided, comprising:

[0006] obtaining question information corresponding to each question in a question bank to be deduplicated;

[0007] dividing the question bank to be deduplicated into a seed question bank and a non-seed question bank according to the question information, wherein the question information of any two seed questions in the seed question bank is different;

[0008] calculating a first similarity between a first non-seed question in the non-seed question bank and each seed question in the seed question bank, the first non-seed question being any non-seed question in the non-seed question bank;

[0009] in response to a maximum value in the first similarity being less than a first threshold, adding the first non-seed question to the seed question bank;

[0010] in response to all non-seed questions in the non-seed question bank being traversed, determining the seed question bank as a question bank after deduplication of the question bank to be deduplicated.

[0011] According to another aspect of the present disclosure, a question deduplication device is provided, comprising:

[0012] an information obtaining module configured to obtain question information corresponding to each question in a question bank to be deduplicated;

[0013] The question bank division module is configured to divide the to-be-deduplicated question bank into a seed question bank and a non-seed question bank according to the question information, wherein the question information of any two seed questions in the seed question bank is different;

[0014] The first calculation module is configured to calculate a first similarity between a first non-seed question in the non-seed question bank and each seed question in the seed question bank, wherein the first non-seed question is any non-seed question in the non-seed question bank;

[0015] The first addition module is configured to add the first non-seed question to the seed question bank in response to a maximum value in the first similarity being less than a first threshold value.

[0016] The question bank determination module is configured to determine the seed question bank as a question bank after deduplication of the to-be-deduplicated question bank in response to all non-seed questions in the non-seed question bank being traversed.

[0017] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0018] a processor; and

[0019] a memory storing a program,

[0020] wherein the program includes instructions that, when executed by the processor, cause the processor to perform the question deduplication method according to the foregoing aspect.

[0021] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the question deduplication method according to the foregoing aspect.

[0022] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the question deduplication method according to the foregoing aspect.

[0023] One or more technical solutions provided in this disclosure involve obtaining the question information corresponding to each question in the question bank to be deduplicated, and dividing the question bank into a seed question bank and a non-seed question bank according to the question information. The question information of any two seed questions in the seed question bank is different. Then, the first similarity between the first non-seed question in the non-seed question bank and each seed question in the seed question bank is calculated. The first non-seed question is any non-seed question in the non-seed question bank. In response to the maximum value of the first similarity being less than a first threshold, the first non-seed question is added to the seed question bank. Finally, in response to traversing all non-seed questions in the non-seed question bank, the seed question bank is determined as the target question bank after deduplication of the question bank to be deduplicated. The proposed solution first selects questions with different question information to build a seed question bank. Then, the seed question bank is used to further filter non-seed questions in the non-seed question bank, and non-seed questions that are not similar to the seed questions are added to the seed question bank. This process is repeated until all non-seed questions have been traversed, resulting in a deduplicated target question bank. Therefore, the target question bank contains questions with inconsistent question information as well as questions with low similarity, which improves the deduplication quality and ensures the reliability of the target question bank. Attached Figure Description

[0024] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a title deduplication method according to an exemplary embodiment of the present disclosure is shown;

[0026] Figure 2 A flowchart of a title deduplication method according to another exemplary embodiment of this disclosure is shown;

[0027] Figure 3 A flowchart of a title deduplication method according to yet another exemplary embodiment of this disclosure is shown;

[0028] Figure 4 A schematic block diagram of a title deduplication apparatus according to an exemplary embodiment of the present disclosure is shown;

[0029] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0030] Embodiments of the present disclosure will be described herein below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments set forth herein, but rather these embodiments are provided so as to more thoroughly and completely understand the present disclosure. It is understood that the drawings and embodiments of the present disclosure are merely for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that each step recited in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "comprising" and variations thereof as used herein are open-ended, that is "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related terms are defined as follows. It should be noted that the concepts mentioned in the present disclosure are merely for distinguishing different apparatuses, modules or units, and are not intended to limit the functions of these apparatuses, modules or units.

[0033] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0034] The names of the messages or information exchanged between the plurality of apparatuses in the embodiments of the present disclosure are merely for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0035] The title deduplication method, device, electronic equipment and storage medium provided by the present disclosure are described below with reference to the accompanying drawings.

[0036] In a title search and recommendation system, there are many duplicate titles in the title library, and many of the searched titles are the same. This brings a very bad experience to title search and recommendation and similar title search, so it is particularly important to remove duplicate titles. In addition, removing duplicate titles makes the title library more standardized and clear, which can avoid users from repeatedly practicing the same title when doing the title, and helps to save the user's time and effort. At the same time, removing duplicate titles can also improve the quality and reliability of the title library, and avoid reducing the user's trust in the title library due to duplicate titles.

[0037] Currently, the common way of removing duplicate questions is to calculate the text similarity of the questions in the question bank, and determine the questions with high text similarity as duplicate questions for removal, and only keep the questions with low text similarity. However, considering the diversity of question description methods, different text contents may represent the same question meaning, but the calculated text similarity may be small, resulting in the same meaning questions being misjudged as non-duplicate questions and not being removed. In addition, considering the numerical difference of mathematical questions, only the questions with large numerical difference may be misjudged as non-duplicate questions and cannot be removed due to low similarity. Therefore, the reliability of the question bank after removing duplicates by the existing method of removing duplicates by text similarity is not high.

[0038] To solve the above problems, the present disclosure provides a question removing duplicate method, which obtains the question information corresponding to each question in the question bank to be removed, and divides the question bank to be removed into a seed question bank and a non-seed question bank according to the question information, wherein the question information of any two seed questions in the seed question bank is different, then calculates the first similarity between a first non-seed question in the non-seed question bank and each seed question in the seed question bank, the first non-seed question is any non-seed question in the non-seed question bank, and in response to the maximum value in the first similarity being less than a first threshold, the first non-seed question is added to the seed question bank, finally, in response to all non-seed questions in the non-seed question bank being traversed, the seed question bank is determined as the question bank after removing duplicates from the question bank to be removed. By using the method of the present disclosure, the seed question bank is constructed by screening questions with different question information according to the question information of the questions, and the non-seed questions in the non-seed question bank are further screened by using the constructed seed question bank, and the non-seed questions that are not similar to the seed questions are added to the seed question bank until all non-seed questions are traversed, and the question bank after removing duplicates is obtained. Therefore, the questions in the question bank after removing duplicates include questions with inconsistent question information and questions with low similarity, which improves the quality of question removing duplicates and ensures the reliability of the question bank.

[0039] Figure 1 A flowchart of a question removing duplicate method according to an example embodiment of the present disclosure is shown, which can be executed by a question removing duplicate device provided by the embodiment of the present disclosure, wherein the device can be implemented by software and / or hardware, and can be integrated in an electronic device including a computer, a tablet computer, a server and the like.

[0040] As shown in Figure 1 the question removing duplicate method can include the following steps:

[0041] Step 101, obtaining the question information corresponding to each question in the question bank to be removed.

[0042] In the embodiments of the present disclosure, the to-be-de-duplicated question bank can contain a large number of questions, which can be questions of the same subject, for example, the questions in the to-be-de-duplicated question bank are all mathematical questions, or the questions can be questions of different subjects, for example, the questions in the to-be-de-duplicated question bank are mathematical questions, Chinese questions, English questions and questions of multiple subjects. For each question in the to-be-de-duplicated question bank, question information of the question can be obtained.

[0043] The question information can include, but is not limited to, one or more of the subject, the number of question characters, the number of Chinese characters, the number of English characters, the number of numerical characters, the question identifier, and the question type (image question or text question). In the embodiments of the present disclosure, characters other than Chinese characters and numerical characters are regarded as English characters; when counting the number of numerical characters, consecutive numbers are counted as one numerical character, for example, "100" in "100 apples" is counted as one numerical character, that is, the number of numerical characters refers to the number of numerical values in the question, for example, the numerical value "100" is one numerical character, and the numerical value "15" is one numerical character; the number of question characters is equal to the sum of the number of Chinese characters, the number of English characters and the number of numerical characters. When counting the number of each character, the question text can be first processed to remove stop words and punctuation, and then the number of each character is counted. When obtaining the subject to which the question belongs, if the subject cannot be directly obtained from the text, the subject of the question can be recorded as no subject, or a lightweight model can be used to pre-train a subject classifier, and the subject classifier is used to classify the subject of the question without a subject, and the classification result output by the subject classifier is taken as the subject of the question.

[0044] For each question in the to-be-de-duplicated question bank, the question type field value of the question in the to-be-de-duplicated question bank can be used to determine whether the question is an image question or a text question. If it is a text question, the corresponding question information can be directly counted according to the text information of the question; if it is an image question, a commonly used high-precision optical character recognition (OCR) interface can be called first to convert the image question into a text question through the high-precision OCR interface, and then the question information corresponding to the question is counted based on the recognized text content.

[0045] In an optional embodiment of the present disclosure, if the number of questions in the to-be-de-duplicated question library is huge, such as containing hundreds of millions or even tens of billions of questions, it is very time-consuming to traverse the to-be-de-duplicated question library once, and therefore the entire to-be-de-duplicated question library can be divided into several parts. For example, if the to-be-de-duplicated question library contains 1 billion questions, it can be divided into 1000 groups, each group containing 1 million questions. In this way, 1000 threads can be started to simultaneously traverse the to-be-de-duplicated question library to perform question information statistics. The 1000 threads can be distributed on several machines, for example, 10 machines, and each machine can start 100 threads to process simultaneously. It can be understood that in actual use, the number of questions in the to-be-de-duplicated question library and the number of machines can be dynamically adjusted. Alternatively, question information statistics can be performed on the questions in a part or a sub-library of the full to-be-de-duplicated question library, and the seed question library can be screened based on the statistical question information.

[0046] In step 102, the to-be-de-duplicated question library is divided into a seed question library and a non-seed question library according to the question information, wherein the question information of any two seed questions in the seed question library is different.

[0047] In the embodiment of the present disclosure, after obtaining the question information of each question in the to-be-de-duplicated question library, the questions in the to-be-de-duplicated question library can be split according to the question information to obtain the seed question library and the non-seed question library.

[0048] As an example, the question information of each question in the to-be-de-duplicated question library can be compared, and questions with different question information can be screened out to construct the seed question library. If the question information of multiple questions is the same, a question with the same question information can be randomly selected from the multiple questions as a seed question for constructing the seed question library. The remaining questions in the to-be-de-duplicated question library after removing the seed questions are non-seed questions, and the non-seed question library can be constructed using the non-seed questions.

[0049] As another example, the questions in the to-be-de-duplicated question library can be grouped according to the question information, and questions with the same question information can be divided into a group. If the question information of a question is different from the question information of other questions, the question is determined as a seed question for constructing the seed question library. For each group of questions, seed questions can be selected from each group according to a preset rule to construct the seed question library, and the remaining non-seed questions in each group are used to construct the non-seed question library.

[0050] In step 103, the first similarity between a first non-seed question in the non-seed question library and each seed question in the seed question library is calculated, and the first non-seed question is any non-seed question in the non-seed question library.

[0051] In the embodiments of the present disclosure, after the questions in the to-be-de-duplication question bank are split into the seed questions in the seed question bank and the non-seed questions in the non-seed question bank, the non-seed questions in the non-seed question bank can be screened based on the seed question bank to find the non-duplicate questions in the non-seed question bank and add them to the seed question bank, so as to ensure the comprehensiveness of the screened non-duplicate questions. The non-duplicate questions in the non-seed question bank can be represented by similarity, and the higher the similarity is, the greater the probability of duplication is. Therefore, in the embodiments of the present disclosure, for each non-seed question in the non-seed question bank, each non-seed question can be traversed in turn, and the similarity (for the sake of distinction and description, referred to as first similarity) between the first non-seed question and each seed question in the seed question bank is calculated, where the first non-seed question is the currently traversed non-seed question in the non-seed question bank, and can be any non-seed question in the non-seed question bank. It can be understood that the number of first similarities is multiple, and specifically consistent with the number of seed questions included in the seed question bank.

[0052] Exemplarily, the text similarity between the first non-seed question and each seed question can be calculated as the first similarity by performing a text search on the seed question bank according to the text content corresponding to the first non-seed question.

[0053] In step 104, in response to the maximum value in the first similarities being less than a first threshold, the first non-seed question is added to the seed question bank.

[0054] The first threshold can be pre-set according to actual needs, or can be obtained by experimental statistics, and the present disclosure does not limit this.

[0055] In the embodiments of the present disclosure, for the first non-seed question, after the first similarities between the first non-seed question and each seed question are calculated, the maximum value can be determined from the first similarities, and the maximum value is compared with the first threshold. If the maximum value is less than the first threshold, it can be determined that the first non-seed question is not similar to each seed question in the seed question bank, that is, the first non-seed question is not a duplicate question, and then the first non-seed question is added to the seed question bank, and the next non-seed question is obtained as a new first non-seed question to be searched in the seed question bank and the similarity between the new first non-seed question and each seed question is calculated.

[0056] Further, in an optional embodiment of the present disclosure, if the maximum value in the first similarity is not less than the first threshold value, the target seed question corresponding to the maximum value in the first similarity can be further obtained, for example, the first similarity between the first non-seed question and the seed question a is the maximum value in all the first similarities, and the seed question a is the target seed question obtained. Then, the question information corresponding to the target seed question (referred to as target question information) can be obtained, and the target question information is compared with the question information of the first non-seed question. If the target question information is inconsistent with the question information of the first non-seed question, it is determined that the first non-seed question and the target seed question are not duplicate questions, and then the first non-seed question is added to the seed question library. Alternatively, when comparing the question information of the first non-seed question with the target question information of the target seed question, only part of the information in the question information can be compared to determine whether they are consistent. If they are inconsistent, it is determined that the first non-seed question and the target seed question are not duplicate questions. For example, the question information includes subject, number of question characters, number of Chinese characters, number of English characters and number of digital characters. The subject and the number of digital characters in the target question information can be compared with the subject and the number of digital characters in the question information of the first non-seed question. If they are consistent, it is determined that the target seed question and the first non-seed question are duplicate questions, and the first non-seed question is filtered out. If they are inconsistent, the first non-seed question is added to the seed question library.

[0057] In the embodiment of the present disclosure, when the maximum value in the first similarity is not less than the first threshold value, the target question information corresponding to the most similar target seed question is further compared with the question information of the first non-seed question to determine whether they are consistent. If they are inconsistent, the first non-seed question is added to the seed question library. In this way, the non-duplicate questions in the non-seed question library can be screened as much as possible, and the quality of removing duplicate questions is guaranteed.

[0058] In step 105, in response to traversing all non-seed questions in the non-seed question library, the seed question library is determined as the target question library after the to-be-removed question library is removed.

[0059] In the embodiment of the present disclosure, the similarity between each non-seed question in the non-seed question library and each seed question in the seed question library is compared to screen the non-seed questions that are not duplicated with the seed questions from the non-seed question library and add them to the seed question library. When all non-seed questions in the non-seed question library are traversed, that is, the first non-seed question is the last question in the non-seed question library, the comparison ends. At this time, the seed question library contains not only the questions screened according to the question information, but also the questions screened from the non-seed question library. The seed question library at this time can be determined as the target question library after the to-be-removed question library is removed.

[0060] The subject deduplication method of the embodiments of the present disclosure, by obtaining the subject information corresponding to each subject in the to-be-deduplicated subject library, and according to the subject information, dividing the to-be-deduplicated subject library into a seed subject library and a non-seed subject library, wherein the subject information of any two seed subjects in the seed subject library is different, then calculating the first similarity between a first non-seed subject in the non-seed subject library and each seed subject in the seed subject library, the first non-seed subject being any non-seed subject in the non-seed subject library, and in response to the maximum value in the first similarity being less than a first threshold, adding the first non-seed subject to the seed subject library, and finally, in response to all non-seed subjects in the non-seed subject library being traversed, determining the seed subject library as the subject library after deduplication of the to-be-deduplicated subject library. By using the scheme of the present disclosure, the seed subject library is constructed by screening out subjects with different subject information according to the subject information of the subjects, and the non-seed subjects in the non-seed subject library are further screened by using the constructed seed subject library, and the non-seed subjects that are not similar to the seed subjects are added to the seed subject library, until all non-seed subjects are traversed, and the deduplicated subject library is obtained. Therefore, the subjects in the subject library include subjects with inconsistent subject information and subjects with low similarity, which improves the subject deduplication quality and ensures the reliability of the subject library.

[0061] In an optional embodiment of the present disclosure, the subject information includes a subject and a subject character number, wherein the subject character number is the sum of the number of Chinese characters, the number of English characters and the number of numerical characters included in the same subject, and the seed subjects can be screened according to the subject and the subject character number, so that, as shown in the embodiment shown in Figure 2 , on the basis of the embodiment shown in Figure 1 , the step 102 can include the following sub-steps:

[0062] Step 201, clustering the subjects in the to-be-deduplicated subject library according to the subject and the subject character number, to obtain a plurality of subject groups, wherein the subjects in the same subject group have the same subject and subject character number.

[0063] In the embodiments of the present disclosure, for each subject in the to-be-deduplicated subject library, the obtained subject information includes a subject and a subject character number, and the subjects in the to-be-deduplicated subject library can be clustered according to the subject and the subject character number, and the subjects with the same subject and subject character number are clustered into a class, to obtain a plurality of subject groups.

[0064] For example, the subject and the subject character number can be spliced together as a category, and the subjects belonging to the category are screened to form a subject group, for example, "mathematics 30", and the subject of each subject in the subject group corresponding to the category is mathematics, and the subject character number is 30.

[0065] In an optional embodiment of the present disclosure, if the number of questions in the to-be-de-duplicated question bank is huge, the questions in the to-be-de-duplicated question bank can be divided into several parts, for example, 1 million questions as a stack, and then the questions in each stack are clustered in a multi-threaded manner, and then each question group obtained by clustering in each thread is merged according to categories, for example, the question group of the category "mathematics 30" obtained by clustering in thread 1 is merged with the question group of the category "mathematics 30" obtained by clustering in thread 2, and the final question group corresponding to the category "mathematics 30" is obtained to screen out the seed question.

[0066] Step 202, according to the preset screening rule, screening out the seed question from the plurality of question groups.

[0067] In the embodiments of the present disclosure, for the plurality of question groups obtained by clustering, the seed question can be screened out from each question group according to the preset screening rule.

[0068] As an optional embodiment, the preset screening rule includes: if a question group includes only one question, the question is determined as the seed question screened out from the question group; if a question group includes at least two questions, the seed question is screened out according to the number of Chinese characters, the number of English characters and the number of numerical characters of the questions, and the questions with at least two different items of the number of Chinese characters, the number of English characters and the number of numerical characters in the same question group are determined as the seed questions. In this embodiment, the obtained question information further includes the number of Chinese characters, the number of English characters and the number of numerical characters, the number of numerical characters refers to the number of numerical values appearing in the question, and when the seed question is screened out from the plurality of question groups, if it is a question group including only one question, the question is directly determined as the seed question; if it is a question group including at least two questions, the number of Chinese characters, the number of English characters and the number of numerical characters of each question in the question group are compared, and if at least two of the number of Chinese characters, the number of English characters and the number of numerical characters of a question are different from those of other questions in the question group, the question is determined as a seed question. That is, for any question group, the number of Chinese characters, the number of English characters and the number of numerical characters of any two seed questions screened out from the question group are different, which satisfy one of the following situations: (1) the number of Chinese characters is the same, the number of numerical characters is different, and the number of English characters is different; (2) the number of Chinese characters is different, the number of numerical characters is the same, and the number of English characters is different; (3) the number of Chinese characters is different, the number of numerical characters is different, and the number of English characters is the same; (4) the number of Chinese characters is different, the number of numerical characters is different, and the number of English characters is different.

[0069] In the embodiments of the present disclosure, the seed questions are screened from the multiple question groups according to at least two different screening rules of the number of Chinese characters, the number of English characters and the number of numerical characters, so as to ensure the diversity of the seed questions and help improve the coverage range and quality of the questions after deduplication. In addition, the number of numerical characters is limited to the number of numerical values in the questions, which can avoid the problem that the questions are not determined as repeated questions only due to the difference in numerical value, and help improve the accuracy of question deduplication.

[0070] As another optional implementation, the preset screening rule includes: randomly selecting one from each question group as a seed question. After obtaining the multiple question groups, one can be randomly selected from each question group as a seed question. If a question group contains only one question, the question is the seed question.

[0071] In step 203, a seed question bank is constructed using the seed questions.

[0072] In the embodiments of the present disclosure, after the seed questions are screened, the seed question bank can be constructed using the screened seed questions.

[0073] In an optional implementation of the present disclosure, in order to ensure the quality and reliability of the seed question bank, the seed questions can also be deduplicated to obtain a more refined and accurate seed question bank. Specifically, an initial seed question bank can be generated using the seed questions, and then each seed question in the initial seed question bank is traversed. The currently traversed seed question can be referred to as a first seed question. The similarity between the first seed question and a second seed question (referred to as a second similarity for the sake of distinction and description) is calculated, where the first seed question can be any question in the initial seed question bank, and the second seed question is each seed question in the initial seed question bank except the first seed question. The number of second seed questions is at least one. The second similarity corresponding to each second seed question is calculated. It can be understood that the number of second similarities is consistent with the number of second seed questions. After the second similarity is calculated, the maximum value can be determined from the second similarity, and the maximum value is compared with a second threshold. If the maximum value is less than the second threshold, it is determined that the first seed question is not repeated with other seed questions in the initial seed question bank, the first seed question is retained, and the next seed question is continued to be traversed. If the maximum value is not less than the second threshold, it is determined that the first seed question is repeated with other seed questions in the initial seed question bank, the first seed question is deleted from the initial seed question bank, and the next seed question is continued to be traversed. The second threshold can be set according to the implementation requirements. When all seed questions in the initial seed question bank are traversed, the similarity between the seed questions retained in the initial seed question bank at this time is less than the second threshold, and they are all non-repeated questions. The initial seed question bank at this time can be determined as the seed question bank.

[0074] Step 204, using the remaining questions in the plurality of question groups except the seed questions, to construct a non-seed question bank.

[0075] In the embodiments of the present disclosure, after the seed questions are selected from each question group, the remaining questions in each question group are non-seed questions. The non-seed question bank can be constructed by using the remaining non-seed questions in all question groups. The set of questions in the seed question bank and the non-seed question bank is all the questions in the initial question bank to be deduplicated.

[0076] The question deduplication method in the embodiments of the present disclosure clusters the questions in the question bank to be deduplicated according to subjects and question character numbers, to obtain a plurality of question groups, wherein the subjects and question character numbers of the questions in the same question group are the same. Then, the seed questions are selected from the plurality of question groups according to a preset selection rule. Further, the seed question bank is constructed by using the seed questions, and the non-seed question bank is constructed by using the remaining questions in the plurality of question groups except the seed questions. Thus, the seed question bank obtained contains seed questions of different subjects or question character numbers, which helps to improve the coverage range and quality of the seed question bank.

[0077] In an optional embodiment of the present disclosure, the first threshold value can be determined by experimental statistics to ensure the accuracy of the first threshold value, thereby improving the accuracy of the repeated question screening. Thus, as shown in Figure 3 On the basis of the foregoing embodiments, the question deduplication method provided in the embodiments of the present disclosure can further include the following steps:

[0078] Step 301, obtaining a target non-seed question from the non-seed question bank.

[0079] In the embodiments of the present disclosure, after the seed question bank and the non-seed question bank are determined, a target non-seed question can be obtained from the non-seed question bank.

[0080] Exemplarily, 1000 non-seed questions can be randomly obtained from the non-seed question bank as the target non-seed questions.

[0081] Step 302, calculating a third similarity between each target non-seed question and each seed question in the seed question bank.

[0082] In the embodiments of the present disclosure, for each target non-seed question obtained, the similarity (for the sake of distinction and description, referred to as the third similarity) between the target non-seed question and each seed question in the seed question bank can be calculated.

[0083] Exemplarily, assuming that there are 100 seed questions in the seed question library, the third similarity between the target non-seed question 1 and each of the 100 seed questions is calculated to obtain 100 third similarities corresponding to the target non-seed question 1; the third similarity between the target non-seed question 2 and each of the 100 seed questions is calculated to obtain 100 third similarities corresponding to the target non-seed question 2, and the like.

[0084] In step 303, the maximum third similarity corresponding to each of the target non-seed questions is determined according to the third similarity between each of the target non-seed questions and each of the seed questions.

[0085] In the embodiments of the present disclosure, after the third similarity between each target non-seed question and each seed question is calculated, a maximum value can be determined from the calculated third similarities, and the maximum value is determined as the maximum third similarity.

[0086] Exemplarily, assuming that there are 100 seed questions in the seed question library, after the 100 third similarities corresponding to the target non-seed question 1 are calculated, a maximum value is determined from the 100 third similarities as the maximum third similarity corresponding to the target non-seed question 1; after the 100 third similarities corresponding to the target non-seed question 2 are calculated, a maximum value is determined from the 100 third similarities as the maximum third similarity corresponding to the target non-seed question 2; and the like, and the maximum third similarity corresponding to each target non-seed question can be obtained.

[0087] In step 304, the number of repeated questions in the target non-seed questions whose maximum third similarity is not less than the candidate threshold value is counted according to the maximum third similarity corresponding to each of the target non-seed questions for different candidate threshold values.

[0088] In the embodiments of the present disclosure, after the maximum third similarity corresponding to each target non-seed question is determined, the number of repeated questions screened under different candidate threshold values can be counted for different candidate threshold values. It can be understood that when the maximum third similarity of a target non-seed question is not less than a candidate threshold value, the target non-seed question is determined as a repeated question with the seed question, and the smaller the value of the candidate threshold value, the more the number of target non-seed questions determined as repeated questions. Therefore, the number of repeated questions screened under different candidate threshold values, that is, the number of repeated questions in the target non-seed questions whose maximum third similarity is not less than the candidate threshold value, is counted.

[0089] Exemplarily, the candidate threshold value can be 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, for each candidate threshold value, the number of target non-seed questions with the maximum third similarity not less than the candidate threshold value is counted, that is, the number of screened duplicate questions under the candidate threshold value is obtained. For example, assuming that the number of target non-seed questions is 1000, when the candidate threshold value is 0.2, the number of target non-seed questions with the maximum third similarity not less than 0.2 is 800, that is, the number of screened duplicate questions when the candidate threshold value is 0.2 is 800; when the candidate threshold value is 0.3, the number of target non-seed questions with the maximum third similarity not less than 0.3 is 760, that is, the number of screened duplicate questions when the candidate threshold value is 0.3 is 760; when the candidate threshold value is 0.4, the number of target non-seed questions with the maximum third similarity not less than 0.4 is 650, that is, the number of screened duplicate questions when the candidate threshold value is 0.4 is 650; the number of screened duplicate questions under each candidate threshold value is counted in turn, when the candidate threshold value is 0.8, the number of target non-seed questions with the maximum third similarity not less than 0.8 is 210, that is, the number of screened duplicate questions when the candidate threshold value is 0.8 is 210; when the candidate threshold value is 0.9, the number of target non-seed questions with the maximum third similarity not less than 0.9 is 80, that is, the number of screened duplicate questions when the candidate threshold value is 0.9 is 80.

[0090] In step 305, the candidate threshold value with the closest number of duplicate questions to the actual number of duplicate questions is determined as the first threshold value.

[0091] The actual number of duplicate questions can be counted by a user based on the obtained target non-seed questions. After the electronic device determines the seed question library and obtains the target non-seed questions from the non-seed question library, the obtained target non-seed questions and the seed question library can be displayed to the user, and the user can count the duplicate questions in the target non-seed questions based on the seed question library, that is, the actual number of duplicate questions in the target non-seed questions.

[0092] In the embodiments of the present disclosure, after the number of duplicate questions corresponding to each candidate threshold value is counted, each number of duplicate questions can be compared with the actual number of duplicate questions, the target number of duplicate questions closest to the actual number of duplicate questions is found from the multiple numbers of duplicate questions, and the candidate threshold value corresponding to the target number of duplicate questions is determined as the first threshold value.

[0093] Exemplarily, assuming that the number of real duplicate questions is 200, the number of duplicate questions screened out when the candidate threshold is 0.2 is 800, the number of duplicate questions screened out when the candidate threshold is 0.3 is 760, the number of duplicate questions screened out when the candidate threshold is 0.4 is 650, the number of duplicate questions screened out when the candidate threshold is 0.5 is 550, the number of duplicate questions screened out when the candidate threshold is 0.6 is 380, the number of duplicate questions screened out when the candidate threshold is 0.7 is 290, the number of duplicate questions screened out when the candidate threshold is 0.8 is 210, and the number of duplicate questions screened out when the candidate threshold is 0.9 is 80, by comparing the number of duplicate questions with the number of real duplicate questions, it can be determined that the number of duplicate questions (i.e., 210) screened out when the candidate threshold is 0.8 is the closest to the number of real duplicate questions (i.e., 200), and therefore the candidate threshold 0.8 is determined as the first threshold. Alternatively, assuming that the number of duplicate questions screened out when the candidate threshold is 0.8 is 210, and the number of duplicate questions screened out when the candidate threshold is 0.9 is also 210, by comparing the number of duplicate questions with the number of real duplicate questions, it can be determined that the number of duplicate questions (i.e., 210) screened out when the candidate threshold is 0.8 and the number of duplicate questions (i.e., 210) screened out when the candidate threshold is 0.9 are both the closest to the number of real duplicate questions (i.e., 200), and therefore the larger candidate threshold can be selected as the first threshold, i.e., the candidate threshold 0.9 is determined as the first threshold.

[0094] The question deduplication method of the embodiment of the present disclosure, by obtaining target non-seed questions from the non-seed question bank, and for each target non-seed question, calculating the third similarity between the target non-seed question and each seed question in the seed question bank, determining the maximum third similarity corresponding to each target non-seed question according to the third similarity between each target non-seed question and each seed question, and then for different candidate thresholds, according to the maximum third similarity corresponding to each target non-seed question, counting the number of duplicate questions in the target non-seed questions whose maximum third similarity is not less than the candidate threshold, and then determining the candidate threshold whose counted number of duplicate questions is the closest to the number of real duplicate questions as the first threshold, thereby realizing the flexible determination of the first threshold through test statistics, making the determined first threshold more accurate, and thus being beneficial to improving the accuracy of duplicate question screening in the non-seed question bank.

[0095] In an actual application scenario, the number of questions in the to-be-de-duplicated question library can be very large, and the number of non-seed questions screened out is also very large. If only one thread is used to screen out the questions that are duplicated with the seed questions in the non-seed questions, it is very time-consuming and the screening efficiency is very low. Therefore, in an optional embodiment of the present disclosure, after the seed question library and the non-seed question library are screened out, the number of non-seed questions in the non-seed question library can be compared with a preset value, wherein the preset value can be set according to actual needs. If the number of non-seed questions in the non-seed question library is greater than the preset value, the non-seed questions in the non-seed question library can be grouped to obtain a plurality of non-seed question groups. For example, the non-seed questions are grouped according to the rule of 100,000 questions per group to obtain a plurality of non-seed question groups. Then, a plurality of same index libraries can be constructed based on the seed question library, the number of constructed index libraries is the same as the number of non-seed question groups, and one non-seed question group corresponds to one index library. For example, the text content of each seed question in the seed question library can be used as the index corresponding to the seed question to construct the index library, that is, the text content of each seed question is stored in the index library. For another example, when the index library is constructed, each index can be uniquely numbered.

[0096] Then, in the present embodiment, a plurality of threads can be created to perform de-duplication processing on the plurality of non-seed question groups, the number of created threads can be the same as the number of non-seed question groups, one thread is used to perform de-duplication processing on one non-seed question group, and when performing de-duplication processing, one index library is used to perform de-duplication processing on the non-seed question group corresponding to the index library, and the index of the non-seed question with the maximum similarity less than the first threshold value in the non-seed question group is added to the index library. That is, one index library and one non-seed question group are loaded in one thread, and in the thread, the non-seed questions that are duplicated with the seed questions in the index library are screened out in the non-seed question group, and the index of the non-duplicated non-seed question is added to the index library. For example, the text content of the non-duplicated non-seed question can be added to the index library as an index.

[0097] It can be understood that the process of de-duplication processing of the non-seed question group by using the thread can refer to the description of the repeated question screening of the non-seed question based on the seed question library in the foregoing embodiments, and the implementation principle is similar, which will not be described here.

[0098] Since the non-seed questions screened out without duplication with the seed questions are dispersed in each index library, it is necessary to combine each index library to obtain a large index library, and then obtain the final question title library after deduplication based on the large index library. In order to avoid the existence of indexes corresponding to repeated questions in the index library obtained by direct combination from affecting the reliability and quality of the question title library, the index library can be deduplicated. Therefore, in an optional embodiment of the present disclosure, after creating multiple threads to respectively deduplicate multiple non-seed question groups, in response to traversing all non-seed questions in the non-seed question library, the seed question library is determined as the question title library after deduplication of the to-be-deduplicated question library, comprising:

[0099] In response to the multiple threads completing the deduplication processing of the non-seed question groups, the current multiple index libraries are combined to obtain a candidate index library; the fourth similarity between the first index and the second index in the candidate index library is calculated, wherein the first index is any index in the candidate index library, and the second index is each index in the candidate index library except the first index; in response to the maximum value in the fourth similarity being not less than the third threshold, the first index is deleted from the candidate index library; in response to traversing all indexes in the candidate index library, a deduplicated target index library is obtained; based on the questions corresponding to each index in the target index library, a question title library is constructed.

[0100] In the embodiment, after each thread in the multiple threads completes the deduplication processing of the non-seed question groups, each index library currently contains indexes corresponding to the original seed questions and indexes corresponding to the non-duplicated non-seed questions screened out, and the current each index library can be combined to obtain a candidate index library.

[0101] As an example, when each index library obtained after deduplication processing is combined to obtain a candidate index library, each index library can be directly combined into a large index library to obtain the candidate index library.

[0102] As another example, when constructing the index library based on the seed question library, the indexes corresponding to the seed questions in the index library can be uniquely numbered, and the indexes corresponding to the non-seed questions added in the index library are not numbered, so that when each index library obtained after deduplication processing is combined to obtain a candidate index library, only one index with the same number is retained and added to the candidate index library, and other unnumbered indexes are all added to the candidate index library. Therefore, the operation amount of deduplicating the candidate index library can be reduced.

[0103] In this embodiment, after obtaining the candidate index library, each index in the candidate index library can be traversed, and the similarity (referred to as the fourth similarity for the sake of distinction and description) between the first index and the second index is calculated, wherein the first index is the index currently traversed, which can be any index in the index library, and the second index is each index in the candidate index library except the first index. The number of the fourth similarity is consistent with the number of the second index. If the maximum value in the fourth similarity is not less than the third threshold value, it is determined that the first index and the second index corresponding to the maximum value in the fourth similarity are repeated, and the first index is deleted from the candidate index library, wherein the third threshold value can be set according to actual requirements. After traversing all the indexes in the candidate index library, the indexes remaining in the current candidate index library are not repeated, thereby obtaining the target index library after deduplication. Since the indexes in the target index library are not repeated, the questions corresponding to the indexes in the target index library are also not repeated. Therefore, the question library is constructed based on the questions corresponding to the indexes in the target index library.

[0104] In the embodiments of the present disclosure, when the number of non-seed questions is greater than the preset value, the non-seed questions are divided into a plurality of non-seed question groups, and a plurality of threads are used to perform deduplication processing on the non-seed question groups, thereby improving the processing efficiency. In addition, the candidate index library obtained by merging each index library is subjected to deduplication processing, and the questions corresponding to the indexes in the target index library after deduplication are determined as the questions after deduplication to obtain the question library, thereby ensuring the reliability of the question library.

[0105] The exemplary embodiments of the present disclosure also provide a question deduplication device. Figure 4 A schematic block diagram of a question deduplication device according to an exemplary embodiment of the present disclosure is shown, as shown in Figure 4 The question deduplication device 40 includes an information acquisition module 410, a question library division module 420, a first calculation module 430, a first addition module 440, and a question library determination module 450.

[0106] The information acquisition module 410 is configured to acquire question information corresponding to each question in a question library to be deduplicated.

[0107] The question library division module 420 is configured to divide the question library to be deduplicated into a seed question library and a non-seed question library according to the question information, wherein the question information of any two seed questions in the seed question library is different.

[0108] The first calculation module 430 is configured to calculate a first similarity between a first non-seed question in the non-seed question library and each seed question in the seed question library, wherein the first non-seed question is any non-seed question in the non-seed question library.

[0109] The first adding module 440 is configured to add the first non-seed question into the seed question library in response to the maximum value in the first similarity being less than a first threshold value.

[0110] The question library determining module 450 is configured to determine the seed question library as a question library after deduplication is performed on the to-be-deduplicated question library in response to all non-seed questions in the non-seed question library being traversed.

[0111] Optionally, the question information comprises a subject and a question character number; and the question library dividing module 420 comprises:

[0112] A clustering unit is configured to cluster questions in the to-be-deduplicated question library according to the subject and the question character number, to obtain a plurality of question groups, wherein the questions in a same question group have the same subject and the same question character number.

[0113] A seed question screening unit is configured to screen seed questions from the plurality of question groups according to a preset screening rule.

[0114] A first constructing unit is configured to construct a seed question library by using the seed questions.

[0115] A second constructing unit is configured to construct a non-seed question library by using the remaining questions in the plurality of question groups except the seed questions.

[0116] Optionally, the question information further comprises a Chinese character number, an English character number and a number character number, the number character number referring to a number of numerical values appearing in a question; and the seed question screening unit is further configured to:

[0117] For each question group, screen seed questions according to the Chinese character number, the English character number and the number character number of the questions in the same question group, wherein at least two of the Chinese character number, the English character number and the number character number of any two of the seed questions screened from the same question group are different.

[0118] Optionally, the first constructing unit is further configured to:

[0119] generate an initial seed question library by using the seed questions;

[0120] calculate a second similarity between a first seed question and a second seed question, wherein the first seed question is any question in the initial seed question library, and the second seed question is each seed question in the initial seed question library except the first seed question;

[0121] in response to a maximum value in the second similarity not being less than a second threshold value, delete the first seed question from the initial seed question library;

[0122] In response to traversing all the seed questions in the initial seed question library, a seed question library is obtained.

[0123] Optionally, the question deduplication apparatus 40 further comprises:

[0124] The first obtaining module is configured to, in response to the maximum value in the first similarity not being less than the first threshold value, obtain a target seed question corresponding to the maximum value in the first similarity.

[0125] The second obtaining module is configured to obtain question title information of the target seed question.

[0126] The second adding module is configured to, in response to the question title information of the target seed question being inconsistent with the question information of the first non-seed question, add the first non-seed question to the seed question library.

[0127] Optionally, the question deduplication apparatus 40 further comprises:

[0128] The third obtaining module is configured to obtain a target non-seed question from the non-seed question library.

[0129] The second calculating module is configured to, for each of the target non-seed questions, calculate a third similarity between the target non-seed question and each seed question in the seed question library.

[0130] The maximum value determining module is configured to determine a maximum third similarity corresponding to each of the target non-seed questions according to the third similarity between the target non-seed question and each seed question.

[0131] The counting module is configured to, for different candidate threshold values, count a number of duplicate questions in the target non-seed questions according to the maximum third similarity corresponding to each of the target non-seed questions, the number of duplicate questions being not less than the candidate threshold value.

[0132] The threshold value determining module is configured to determine, as the first threshold value, a candidate threshold value closest to an actual number of duplicate questions in the target non-seed questions among the counted numbers of duplicate questions.

[0133] Optionally, the question deduplication apparatus 40 further comprises:

[0134] The grouping module is configured to, in response to a number of non-seed questions in the non-seed question library being greater than a preset value, group the non-seed questions in the non-seed question library to obtain a plurality of non-seed question groups.

[0135] The index library constructing module is configured to construct a plurality of same index libraries based on the seed question library, a number of the index libraries being same as a number of the non-seed question groups, and one of the non-seed question groups corresponding to one of the index libraries.

[0136] The creating module is configured to create a plurality of threads to respectively perform deduplication on the plurality of non-seed question groups, and one thread performs deduplication on a non-seed question group corresponding to an index library based on the index library, and adds an index of a non-seed question with a maximum similarity less than the first threshold to the index library.

[0137] And the question library determining module 450 is further configured to:

[0138] In response to the plurality of threads all completing the deduplication on the non-seed question groups, merging the current plurality of index libraries to obtain a candidate index library;

[0139] calculating a fourth similarity between a first index and a second index in the candidate index library, wherein the first index is any index in the candidate index library, and the second index is each index in the candidate index library except the first index;

[0140] In response to a maximum value in the fourth similarity being not less than a third threshold, deleting the first index from the candidate index library;

[0141] In response to traversing all the indexes in the candidate index library, obtaining a target index library after deduplication;

[0142] Based on the questions corresponding to the indexes in the target index library, constructing the question title library.

[0143] The question deduplication apparatus provided in the embodiments of the present disclosure can execute any question deduplication method applicable to the electronic device provided in the embodiments of the present disclosure, and has the function modules and beneficial effects corresponding to the execution method. The contents not described in detail in the device embodiments of the present disclosure can be referred to the description in any method embodiment of the present disclosure.

[0144] The exemplary embodiments of the present disclosure further provide an electronic device, which includes at least one processor, and a memory connected with the at least one processor in communication. The memory stores a computer program capable of being executed by the at least one processor, and the computer program, when executed by the at least one processor, is configured to cause the electronic device to execute the question deduplication method according to the embodiments of the present disclosure.

[0145] The exemplary embodiments of the present disclosure further provide a non-transitory computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to execute the question deduplication method according to the embodiments of the present disclosure.

[0146] The example embodiments of the present disclosure further provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to perform the question deduplication method according to the embodiments of the present disclosure.

[0147] Reference Figure 5 The structure block diagram of the electronic device 1100 which can be applied to various aspects of the present disclosure will now be described, which is an example of a hardware device that can be applied to the present disclosure. The electronic device is intended to represent a variety of forms of digital electronic computing devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computing devices. The electronic device can also represent a variety of forms of mobile devices such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0148] As Figure 5 shown, the electronic device 1100 includes a computing unit 1101 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded into a random access memory (RAM) 1103 from a storage unit 1108. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0149] The various components in the electronic device 1100 are connected to the I / O interface 1105, including an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 can be any type of device capable of inputting information to the electronic device 1100, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 1107 can be any type of device capable of presenting information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0150] The computing unit 1101 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs various methods and processes described above. For example, in some embodiments, the topic deduplication method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. In some embodiments, the computing unit 1101 can be configured to perform the topic deduplication method by any other appropriate means, such as by means of firmware.

[0151] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0152] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0153] As used in this disclosure, the terms "machine-readable medium" and "computer- readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0154] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0155] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0156] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A method of title deduplication, wherein, The method comprises: obtaining question information corresponding to each question in a to-be-deduplicated question bank; dividing the to-be-deduplicated question bank into a seed question bank and a non-seed question bank according to the question information, wherein questions with the same question information in the to-be-deduplicated question bank are divided into a group, the seed question bank is constructed based on the questions in each group after grouping, and the non-seed question bank is constructed using the questions remaining after the seed question bank is constructed, and the question information of any two seed questions in the seed question bank is different; calculating a first similarity between a first non-seed question in the non-seed question bank and each seed question in the seed question bank, wherein the first non-seed question is any non-seed question in the non-seed question bank; in response to a maximum value in the first similarity being less than a first threshold, adding the first non-seed question to the seed question bank; in response to all non-seed questions in the non-seed question bank being traversed, determining the seed question bank as a question bank after deduplication is performed on the to-be-deduplicated question bank.

2. The title deduplication method of claim 1, wherein, The question information comprises a subject and a question character number; and the dividing the to-be-deduplicated question bank into a seed question bank and a non-seed question bank according to the question information comprises: clustering the questions in the to-be-deduplicated question bank according to the subject and the question character number to obtain a plurality of question groups, wherein the subject and the question character number of the questions in a same question group are the same; selecting seed questions from the plurality of question groups according to a preset selection rule; constructing a seed question bank using the seed questions; constructing a non-seed question bank using the remaining questions in the plurality of question groups other than the seed questions.

3. The title deduplication method of claim 2, wherein, The question information further comprises a Chinese character number, an English character number, and a number character number, wherein the number character number refers to the number of numerical values appearing in a question; and wherein the selecting seed questions from the plurality of question groups according to a preset selection rule comprises: for each question group, selecting seed questions according to the Chinese character number, the English character number, and the number character number of the questions in the same question group, wherein at least two of the Chinese character number, the English character number, and the number character number of any two seed questions selected from the same question group are different.

4. The title deduplication method of claim 2, wherein, The constructing a seed question bank using the seed questions comprises: generating an initial seed question bank using the seed questions; calculating a second similarity between a first seed question and a second seed question, wherein the first seed question is any question in the initial seed question bank, and the second seed question is each seed question in the initial seed question bank other than the first seed question; in response to a maximum value in the second similarity being not less than a second threshold, deleting the first seed question from the initial seed question bank; in response to all seed questions in the initial seed question bank being traversed, obtaining a seed question bank.

5. The title deduplication method of any of claims 1-4, wherein, The method further comprises: in response to the maximum value in the first similarity being not less than the first threshold, obtaining a target seed question corresponding to the maximum value in the first similarity; obtaining question information of the target seed question; in response to the subject information of the subject question being inconsistent with the subject information of the first non-seed question, adding the first non-seed question to the seed question bank.

6. The title deduplication method of any of claims 1-4, wherein, The method further comprises: obtaining a target non-seed question from the non-seed question bank; for each of the target non-seed questions, calculating a third similarity between the target non-seed question and each seed question in the seed question bank; determining a maximum third similarity corresponding to each of the target non-seed questions according to the third similarity between each of the target non-seed questions and each seed question; for different candidate thresholds, counting the number of duplicate questions in the target non-seed questions according to the maximum third similarity corresponding to each of the target non-seed questions, wherein the number of duplicate questions is not less than the candidate threshold; determining the candidate threshold closest to the actual number of duplicate questions as the first threshold.

7. The title deduplication method of any of claims 1-4, wherein, The method further comprises: in response to the number of non-seed questions in the non-seed question bank being greater than a preset value, grouping the non-seed questions in the non-seed question bank to obtain a plurality of non-seed question groups; constructing a plurality of identical index banks based on the seed question bank, wherein the number of index banks is the same as the number of non-seed question groups, and one non-seed question group corresponds to one index bank; creating a plurality of threads to perform deduplication processing on the plurality of non-seed question groups respectively, wherein one thread performs deduplication processing on the non-seed question group corresponding to the index bank based on the index bank, and adds the index of the non-seed question with a maximum similarity less than the first threshold to the index bank; and wherein, in response to traversing all non-seed questions in the non-seed question bank, determining the seed question bank as the subject question bank after deduplication of the to-be-deduplicated question bank, comprising: in response to the plurality of threads completing the deduplication processing of the non-seed question groups, merging the current plurality of index banks to obtain a candidate index bank; calculating a fourth similarity between a first index and a second index in the candidate index bank, wherein the first index is any index in the candidate index bank, and the second index is each index in the candidate index bank except the first index; in response to the maximum value in the fourth similarity being not less than a third threshold, deleting the first index from the candidate index bank; in response to traversing all indexes in the candidate index bank, obtaining a deduplicated target index bank; constructing the subject question bank based on the questions corresponding to the indexes in the target index bank.

8. A title deduplication apparatus, wherein, The device comprises: an information acquisition module configured to acquire subject information corresponding to each question in a to-be-deduplicated question bank; a question bank division module configured to divide the to-be-deduplicated question bank into a seed question bank and a non-seed question bank according to the subject information, wherein questions with the same subject information in the to-be-deduplicated question bank are divided into a group, the seed question bank is constructed based on the questions in each group after grouping, and the non-seed question bank is constructed using the questions remaining after constructing the seed question bank, and the subject information of any two seed questions in the seed question bank is different. The first computing module is configured to calculate a first similarity between each non-seed question in the non-seed question bank and each seed question in the seed question bank. The first adding module is configured to add the first non-seed question to the seed question bank in response to a maximum value in the first similarities being less than a first threshold. The question bank determining module is configured to determine the seed question bank as a question bank after the to-be-de-duplicated question bank is de-duplicated in response to all non-seed questions in the non-seed question bank being traversed.

9. An electronic device, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the question de-duplication method according to any one of claims 1-7.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the question de-duplication method according to any one of claims 1-7.

Images