A document detection method and device, and a storage medium

Abstract

The embodiment of the application provides a document detection method and device, and a storage medium, the method comprises the following steps: extracting at least one text feature from a target document, and determining at least one weight value corresponding to the at least one text feature according to the number of times that the at least one text feature appears in the target document; in the case that the target text feature is found from a preset text feature library, determining a first weight value corresponding to the target text feature from the at least one weight value, and obtaining a second weight value corresponding to the target text feature and a target level corresponding to the target text feature from the preset text feature library; processing the first weight value and the second weight value to obtain a first numerical value; in the case that the first numerical value is greater than a first preset numerical value, responding to the target document based on a response scheme corresponding to the target level. Through the above technical solution, the purpose of improving the real-time performance of event detection is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a document detection method, apparatus, and storage medium. Background Technology

[0002] In existing technologies, when an event occurs, target hot events are discovered through frequent item monitoring deterministic algorithms and similarity clustering algorithms of different indicators. After the event is discovered, corresponding measures are prepared to deal with it. However, this method can only determine whether an event is a target hot event after it has occurred, based on indicators such as comments and reposts. It cannot detect the event as soon as it occurs. Therefore, the real-time performance of existing technologies for event detection is low. Summary of the Invention

[0003] This application provides a document detection method, apparatus, and storage medium, which can improve the real-time performance of event detection.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a document detection method, the method comprising:

[0006] Extract at least one text feature from the target document, and determine at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document;

[0007] If a target text feature is found in a preset text feature library, a first weight value corresponding to the target text feature is determined from the at least one weight value, and a second weight value corresponding to the target text feature and a target level corresponding to the target text feature are obtained from the preset text feature library; the preset text feature library stores preset text features, preset weight values ​​corresponding to the preset text features, and preset levels; the target text feature is a text feature among the at least one text feature.

[0008] The first weight value and the second weight value are processed to obtain the first value;

[0009] If the first value is greater than the first preset value, the target document is processed based on the response scheme corresponding to the target level.

[0010] In the above document detection method, before extracting at least one text feature from the target document, the method further includes:

[0011] Acquire and analyze historical documents to obtain at least one initial feature in the historical documents, and determine at least one initial weight value corresponding to the at least one initial feature based on the number of times the at least one initial feature appears in the historical documents;

[0012] The preset text feature library is generated based on the at least one initial feature and the at least one initial weight value.

[0013] In the above document detection method, generating the preset text feature library based on the at least one initial feature and the at least one initial weight value includes:

[0014] The at least one initial weight value is compared with the second preset value in sequence;

[0015] If the first weight value among the at least one initial weight value is greater than the second preset value, the first initial feature corresponding to the first weight value is determined from the at least one initial feature;

[0016] The preset text feature library is generated based on the first initial feature and the first weight value.

[0017] In the above document detection method, after determining at least one weight value corresponding to the at least one text feature based on the frequency of its occurrence in the target document, and before determining the first weight value corresponding to the target text feature from the at least one weight value when the target text feature is found from a preset text feature library, the method further includes:

[0018] Obtain and analyze the second document to obtain a set of weight values ​​corresponding to the text features in the second document;

[0019] The similarity between the target document and the second document is determined based on the at least one weight value and the set of weight values.

[0020] When the similarity is greater than a preset similarity threshold, document detection is prohibited for the second document, which is a document whose preset discrimination standard value is less than that of the target document.

[0021] In the above document detection method, determining the similarity between the target document and the second document based on the at least one weight value and the set of weight values ​​includes:

[0022] The at least one weight value and the set of weight values ​​are respectively input into a preset hash function to obtain a first set of hash values ​​and a second set of hash values;

[0023] The first set of real number vectors is determined based on the first set of hash values, and the second set of real number vectors is determined based on the second set of hash values.

[0024] The first semantic fingerprint corresponding to the target document is determined based on the first set of real number vectors, and the second semantic fingerprint corresponding to the second document is determined based on the second set of real number vectors;

[0025] The Hamming distance between the target document and the second document is determined based on the first semantic fingerprint and the second semantic fingerprint;

[0026] If the Hamming distance is less than or equal to a third preset value, the similarity is determined to be greater than the preset similarity threshold.

[0027] In the above document detection method, after processing the target document based on the response scheme corresponding to the target level when the first value is greater than the first preset value, the method further includes:

[0028] The second weight value in the preset text feature library is replaced by the first weight value according to a preset replacement rule, thereby updating the preset text feature library.

[0029] In the above document detection method, after processing the target document based on the response scheme corresponding to the target level when the first value is greater than the first preset value, the method further includes:

[0030] Find a first text feature that is different from the target text feature from the at least one text feature, and obtain a third weight value corresponding to the first text feature from the at least one weight value;

[0031] The first text feature and the third weight value are added to the preset text feature library according to the target level, thereby updating the preset text feature library.

[0032] Secondly, embodiments of this application provide a document detection device, the device comprising:

[0033] A determining unit is configured to extract at least one text feature from a target document and determine at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document;

[0034] The determining unit is further configured to, when a target text feature is found in a preset text feature library, determine a first weight value corresponding to the target text feature from the at least one weight value, and obtain a second weight value corresponding to the target text feature and a target level corresponding to the target text feature from the preset text feature library; the preset text feature library stores preset text features, preset weight values ​​corresponding to the preset text features, and preset levels; the target text feature is a text feature among the at least one text feature;

[0035] A data processing unit is used to process the first weight value and the second weight value to obtain a first value;

[0036] The response unit is used to process the target document based on the response scheme corresponding to the target level when the first value is greater than the first preset value.

[0037] Thirdly, embodiments of this application provide a document detection device, characterized in that the device includes: a processor, a memory, and a communication bus; when the processor executes a running program stored in the memory, it implements the document detection method as described in any of the above claims.

[0038] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the document detection method as described in any of the preceding claims.

[0039] This application provides a document detection method, apparatus, and storage medium. The method includes: extracting at least one text feature from a target document, and determining at least one weight value corresponding to the at least one text feature based on the frequency of occurrence of the at least one text feature in the target document; if the target text feature is found in a preset text feature library, determining a first weight value corresponding to the target text feature from the at least one weight value, and obtaining a second weight value corresponding to the target text feature and a target level corresponding to the target text feature from the preset text feature library; processing the first weight value and the second weight value to obtain a first value; if the first value is greater than a first preset value, performing response processing on the target document based on a response scheme corresponding to the target level. By adopting the above implementation scheme, events are abstracted as documents, and the matching of events with the preset text feature library is achieved by matching the text features of the document with the preset text feature library. This allows the level corresponding to the event to be determined as soon as the event occurs by matching with the preset text feature library, thereby responding to the event in a timely manner according to the response scheme corresponding to the level, and achieving the purpose of improving the real-time performance of event detection. Attached Figure Description

[0040] Figure 1A flowchart of a document detection method provided in this application embodiment;

[0041] Figure 2 An exemplary flowchart for determining the similarity between documents is provided for embodiments of this application;

[0042] Figure 3 An exemplary vector transformation diagram provided for an embodiment of this application;

[0043] Figure 4 An exemplary semantic fingerprint generation diagram is provided for an embodiment of this application;

[0044] Figure 5 A schematic diagram of the composition structure of a document detection device 1 provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of the composition structure of a document inspection device 1 provided in an embodiment of this application. Detailed Implementation

[0046] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the scope of this application.

[0047] This application provides a document detection method, applied to a document detection device. Figure 1 A flowchart of a document detection method provided in this application embodiment is shown below. Figure 1 As shown, document detection methods may include:

[0048] S101. Extract at least one text feature from the target document, and determine at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document.

[0049] In this embodiment of the application, after obtaining the target document, the document detection device extracts at least one text feature from the target document and determines at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document.

[0050] It should be noted that before obtaining the target document, this application needs to use the parser Jsoup to crawl the news list of the day from major news websites such as China News Network and People's Daily Online, and perform webpage parsing, treating each news item as an event, and generating a target document based on each event.

[0051] It should be noted that when filtering text features, at least one text feature is extracted from the target document using the term frequency–inverse document frequency (TF-IDF) index. Then, the weight value of each text feature is calculated to obtain a set of weighted text features.

[0052] It should be noted that TF-IDF is a combination of TF (Term Frequency) and IDF (Inverse Text Frequency Index), indicating that the more times a word appears in a document, the less times it appears in all documents, the more representative it is of that article and the more distinguishable it is from other documents.

[0053] For example, this application provides a method for calculating TF-IDF, and the calculation method of TF is as follows:

[0054]

[0055] In equation (1) above, Nw is the number of times word W appears in a document, and N is the total number of words in the document. TF is the weight of each feature.

[0056] The calculation method for IDF is as follows:

[0057]

[0058] In the above formula (2), Y is the total number of documents in the corpus, Yw is the number of documents containing the term w, and the denominator is added to avoid the case where w does not appear in any document, resulting in a denominator of 0.

[0059] The TF-IDF is calculated as follows:

[0060] TF-IDF w =TF w *IDF w (3)

[0061] The TF-IDF value obtained from the above formula (3) is the weight value of the text feature.

[0062] In this embodiment of the application, after determining the weight values ​​of text features, the document detection device needs to detect the similarity between different target documents and perform deduplication.

[0063] Specifically, the second document is acquired and analyzed to obtain a set of weight values ​​corresponding to the text features in the second document; based on at least one weight value and a set of weight values, the similarity between the target document and the second document is determined.

[0064] In this embodiment of the application, the similarity between the target document and the second document is determined by the Hamming distance between them. Specifically, Figure 2 An exemplary flowchart for determining the similarity between documents is provided for this application, such as... Figure 2 As shown, the method includes:

[0065] S201. Input at least one weight value and a set of weight values ​​into a preset hash function to obtain the first set of hash values ​​and the second set of hash values.

[0066] In this embodiment of the application, after obtaining at least one weight value and a set of weight values, the document detection device inputs the at least one weight value and the set of weight values ​​into a preset hash function to obtain a first set of hash values ​​and a second set of hash values.

[0067] For example, a hash function is used to map at least one weight value and a set of weight values ​​into binary values ​​of bit_count bits. Assuming bit_count is 6, each text feature will be mapped to a 6-bit hash value. Finally, a first set of hash values ​​and a second set of hash values ​​are obtained, and both the first set of hash values ​​and the second set of hash values ​​are weighted hash values.

[0068] S202. Determine the first group of real number vectors based on the first group of hash values, and determine the second group of real number vectors based on the second group of hash values.

[0069] In this embodiment of the application, after obtaining the first set of hash values ​​and the second set of hash values, the document detection device determines the first set of real number vectors based on the first set of hash values ​​and the second set of real number vectors based on the second set of hash values.

[0070] For example, Figure 3 An exemplary vector transformation diagram provided for an embodiment of this application, such as... Figure 3 As shown, the weight value is incorporated into each binary hash value, that is, each bit of the hash value is used as the sign bit of the weight, 1 is positive and 0 is negative. In this way, the binary hash value of bit_count bits is rewritten into bit_count real numbers, forming a real number vector W(w,-w,…,w), and finally a set of real number vectors is obtained.

[0071] S203. Determine the first semantic fingerprint corresponding to the target document based on the first set of real number vectors, and determine the second semantic fingerprint corresponding to the second document based on the second set of real number vectors.

[0072] In this embodiment of the application, after obtaining the first set of real number vectors and the second set of real number vectors, the document detection device determines the first semantic fingerprint corresponding to the target document based on the first set of real number vectors, and determines the second semantic fingerprint corresponding to the second document based on the second set of real number vectors.

[0073] Specifically, Figure 4 An exemplary semantic fingerprint generation diagram is provided for this application, such as... Figure 4 As shown, it is necessary to accumulate each dimension of the generated first set of real number vectors and the second set of real number vectors, that is, to add the values ​​of the corresponding subscripts, to obtain a real number vector that can represent the target document and a real number vector that can represent the second document. Then, based on the real number vector that can represent the target document and the real number vector that can represent the second document, the first semantic fingerprint corresponding to the target document and the second semantic fingerprint corresponding to the second document are determined.

[0074] For example, after obtaining a real number vector that can represent the target document, it is converted into a hash value of bit_count, which is the final semantic fingerprint.

[0075] S204. Determine the Hamming distance between the target document and the second document based on the first semantic fingerprint and the second semantic fingerprint.

[0076] In this embodiment of the application, after determining the first semantic fingerprint and the second semantic fingerprint, the document detection device determines the Hamming distance between the target document and the second document based on the first semantic fingerprint and the second semantic fingerprint.

[0077] S205. If the Hamming distance is less than or equal to the third preset value, determine that the similarity is greater than the preset similarity threshold.

[0078] In this embodiment of the application, after determining the Hamming distance between the target document and the second document, if the Hamming distance is less than or equal to a third preset value, the similarity is determined to be greater than a preset similarity threshold.

[0079] In this embodiment of the application, when the similarity is greater than a preset similarity threshold, document detection of the second document is prohibited. The second document is a document whose preset discrimination standard value is less than that of the target document.

[0080] After determining that there are no documents similar to the target document, document detection is performed on the target document.

[0081] S102. When a target text feature is found in a preset text feature library, a first weight value corresponding to the target text feature is determined from at least one weight value, and a second weight value corresponding to the target text feature and a target level corresponding to the target text feature are obtained from the preset text feature library; the preset text feature library stores preset text features, preset weight values ​​corresponding to preset text features, and preset levels; the target text feature is a text feature among at least one text feature.

[0082] In this embodiment, after determining the weight value of at least one text feature, the document detection device sequentially searches for at least one text feature in a preset text feature library. If a target text feature is found in the preset text feature library, the device determines the first weight value corresponding to the target text feature from the at least one weight value, and obtains the second weight value corresponding to the target text feature and the target level corresponding to the target text feature from the preset text feature library. The preset text feature library stores preset text features, preset weight values ​​corresponding to preset text features, and preset levels. The target text feature is a text feature among at least one text feature.

[0083] For example, suppose the target document has 5 text features. When searching for these 5 text features in the preset text feature library, it is found that there are 3 identical text features in the preset text feature library. At this time, the first weight value of these 3 text features is obtained from at least one weight value determined before. Then, the second weight value of these 3 text features stored in the preset text feature library is obtained from the preset text feature library. The first weight value and the second weight value of these 3 text features are then processed.

[0084] In this embodiment, the preset text feature library is obtained based on historical documents corresponding to historical events.

[0085] Specifically, historical documents are acquired and analyzed to obtain at least one initial feature from the historical documents, and at least one initial weight value corresponding to the at least one initial feature is determined based on the number of times the at least one initial feature appears in the historical documents; a preset text feature library is generated based on the at least one initial feature and the at least one initial weight value.

[0086] It should be noted that at this time, the calculation of the weight value only needs to use the calculation method of TF, that is, formula (1).

[0087] It should be noted that when analyzing historical documents to obtain initial features, the initial features of historical documents can be determined by the Jieba word segmentation algorithm. The specific analysis algorithm can be determined according to the actual situation, and this application embodiment does not limit it here.

[0088] It should be noted that when generating a preset text feature library based on at least one initial feature and at least one initial weight value, it is necessary to determine whether the initial feature is a representative initial feature.

[0089] Specifically, at least one initial weight value is compared with a second preset value in sequence; if the first weight value among the at least one initial weight value is greater than the second preset value, a first initial feature corresponding to the first weight value is determined from at least one initial feature; and a preset text feature library is generated based on the first initial feature and the first weight value.

[0090] Understandably, since some initial features appear less frequently in a document, they can be removed, leaving only representative initial features. This simplifies the preset text feature library and speeds up document detection.

[0091] It should be noted that before generating the preset text feature library, there is a blank text feature library. Then, different categories and levels of features are manually added to this blank text feature library to obtain the initial text feature library.

[0092] It should be noted that since historical events have their own corresponding categories and levels, the initial features of historical events are stored in the initial text feature library according to the category and level corresponding to the initial features.

[0093] It should be noted that the preset text feature library stores preset text features, preset weight values ​​and preset levels corresponding to the preset text features, and can also store the categories corresponding to the preset text features.

[0094] S103. Process the first weight value and the second weight value to obtain the first value.

[0095] In this embodiment of the application, after obtaining the first weight value and the second weight value, the document detection device processes the first weight value and the second weight value to obtain the first value.

[0096] For example, assuming that the first weight value obtained at this time is 0.5 and the second weight value is 1, the first weight value and the second weight value are multiplied to obtain a weight value of 0.5, which is the first value; the specific processing method is determined according to the actual situation, and this application embodiment does not limit it here.

[0097] For example, suppose that three target text features are obtained at this time, and the first weight values ​​of the three target text features are 0.5, 1 and 2 respectively; the second weight values ​​corresponding to these three target text features are obtained from the preset text feature library, which are 1, 0.5 and 3 respectively. First, the first weight value and the second weight value are multiplied by each other, and finally accumulated to obtain the first value, that is, 0.5×1+1×0.5+2×3, and the processing result is 7. At this time, 7 is the first value; the specific processing method is determined according to the actual situation, and this application embodiment does not limit it here.

[0098] S104. If the first value is greater than the first preset value, the target document is processed according to the response scheme corresponding to the target level.

[0099] In this embodiment of the application, after processing the first weight value and the second weight value to obtain the first value, the document detection device determines the size of the first value and the first preset value. If the first value is greater than the first preset value, the target document is processed based on the response scheme corresponding to the target level.

[0100] It should be noted that since the target document is obtained after parsing the event, by abstracting the event into a document and comparing it with the preset text feature library, it is possible to determine whether the event is related to the preset text feature library. Furthermore, since the events are crawled from the webpage in real time, the latest events related to the preset text feature library can be obtained. Based on the target level corresponding to the target document in the preset text feature library, the response scheme corresponding to the target level is determined, and the target document is processed accordingly.

[0101] It should be noted that when determining the target level of a target document, the determination is based on the level of the text features corresponding to the target document in the preset text feature library. Suppose that the target document and the preset text feature library match 8 text features, but 2 of these text features are at level 1, 2 are at level 2, and 4 are at level 3. The target level of the target document needs to be determined based on the level with the largest proportion. If there are equal proportions, then any target level can be selected.

[0102] In this embodiment of the application, after the document detection device performs response processing on the target document, it uses the first weight value to replace the second weight value in the preset text feature library according to the preset replacement rule, thereby updating the preset text feature library.

[0103] It should be noted that the preset replacement rule can be to compare the size of the first weight value and the second weight value. If the first weight value is greater than the second weight value, the second weight value is replaced by the first weight value; if the first weight value is less than the second weight value, no replacement is performed; if the first weight value is equal to the second weight value, no replacement is performed.

[0104] For example, suppose the target document's text features and the preset text feature library have three identical text features, and the weight values ​​of the target document's text features are: text feature 1 has a weight value of 0.5, text feature 2 has a weight value of 1, and text feature 3 has a weight value of 2. Meanwhile, in the preset text feature library, text feature 1 has a weight value of 1, text feature 2 has a weight value of 0.5, and text feature 3 has a weight value of 2. It can be seen that the weight value of text feature 1 in the preset text feature library is greater than the weight value of text feature 1 in the target document; therefore, the weight value of text feature 1 in the preset text feature library remains unchanged. The weight value of text feature 2 in the preset text feature library is less than the weight value of text feature 2 in the target document; therefore, the weight value of text feature 2 in the preset text feature library is changed from 0.5 to 1. The weight value of text feature 3 in the preset text feature library is equal to the weight value of text feature 3 in the target document; therefore, the weight value of text feature 3 in the preset text feature library remains unchanged.

[0105] In this embodiment of the application, after the document detection device performs response processing on the target document, it needs to update not only the weight values ​​of the text features in the preset text feature library, but also the text features in the preset text feature library.

[0106] Find a first text feature that is different from the target text feature from at least one text feature, and obtain a third weight value corresponding to the first text feature from at least one weight value; add the first text feature and the third weight value to the preset text feature library according to the target level to realize the update of the preset text feature library.

[0107] For example, suppose the target document has 3 identical text features to the preset text feature library, and the target document has a total of 5 text features. This means that 2 text features of the target document are not stored in the preset text feature library. Therefore, the remaining 2 text features and their corresponding weight values ​​need to be stored in the preset text feature library according to the target level to update the preset text feature library.

[0108] In this embodiment of the application, if the first value is less than or equal to a first preset value, the target document is discarded and the matching process for the next document is initiated.

[0109] It should be noted that if the first value is less than or equal to the first preset value, it means that the target document does not match the preset text feature library well enough. In other words, the event corresponding to the target document is not related to the preset text feature library. Therefore, the target document is discarded and the matching process for the next document begins.

[0110] This application provides a document detection method, which includes: extracting at least one text feature from a target document, and determining at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document; if the target text feature is found in a preset text feature library, determining a first weight value corresponding to the target text feature from the at least one weight value, and obtaining a second weight value corresponding to the target text feature and a target level corresponding to the target text feature from the preset text feature library; processing the first weight value and the second weight value to obtain a first value; if the first value is greater than a first preset value, responding to the target document based on the response scheme corresponding to the target level; by adopting the above implementation scheme, events are abstracted into documents, and the matching of events with the preset text feature library is achieved by matching the text features of the document with the preset text feature library, so that the level corresponding to the event can be determined by matching with the preset text feature library as soon as the event occurs, and the event can be responded to in a timely manner according to the response scheme corresponding to the level, thereby improving the real-time performance of event detection.

[0111] Based on the above embodiments, in another embodiment of this application, a document detection device 1 is provided. Figure 5 A schematic diagram of the composition structure of a document detection device 1 provided in this application. Figure 1 ,like Figure 5 As shown, the document detection device 1 includes:

[0112] The determining unit 10 is configured to extract at least one text feature from the target document and determine at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document;

[0113] The determining unit 10 is further configured to, when a target text feature is found in a preset text feature library, determine a first weight value corresponding to the target text feature from the at least one weight value, and obtain a second weight value corresponding to the target text feature and a target level corresponding to the target text feature from the preset text feature library; the preset text feature library stores preset text features, preset weight values ​​corresponding to the preset text features, and preset levels; the target text feature is a text feature among the at least one text feature;

[0114] Data processing unit 11 is used to process the first weight value and the second weight value to obtain a first value;

[0115] The response unit 12 is used to perform response processing on the target document based on the response scheme corresponding to the target level when the first value is greater than the first preset value.

[0116] Optionally, the document detection device 1 further includes: an acquisition unit and a generation unit;

[0117] The acquisition unit is used to acquire and analyze historical documents, obtain at least one initial feature in the historical documents, and determine at least one initial weight value corresponding to the at least one initial feature based on the number of times the at least one initial feature appears in the historical documents.

[0118] The generation unit is used to generate the preset text feature library based on the at least one initial feature and the at least one initial weight value.

[0119] Optionally, the document detection device 1 further includes: a comparison unit;

[0120] The comparison unit is used to sequentially compare the at least one initial weight value with a second preset value;

[0121] The determining unit 10 is further configured to determine, when the first weight value in the at least one initial weight value is greater than the second preset value, a first initial feature corresponding to the first weight value from the at least one initial feature;

[0122] The generation unit is further configured to generate the preset text feature library based on the first initial feature and the first weight value.

[0123] Optionally, the document detection device 1 further includes: a blocking unit;

[0124] The acquisition unit is further configured to acquire and analyze the second document to obtain a set of weight values ​​corresponding to the text features in the second document;

[0125] The determining unit 10 is further configured to determine the similarity between the target document and the second document based on the at least one weight value and the set of weight values;

[0126] The prohibition unit is used to prohibit document detection of the second document when the similarity is greater than a preset similarity threshold, wherein the second document is a document whose preset discrimination standard value is less than that of the target document.

[0127] Optionally, the data processing unit 11 is further configured to input the at least one weight value and the set of weight values ​​into a preset hash function to obtain a first set of hash values ​​and a second set of hash values.

[0128] The determining unit 10 is further configured to determine a first group of real number vectors based on the first group of hash values, and to determine a second group of real number vectors based on the second group of hash values;

[0129] The determining unit 10 is further configured to determine the first semantic fingerprint corresponding to the target document based on the first set of real number vectors, and to determine the second semantic fingerprint corresponding to the second document based on the second set of real number vectors;

[0130] The determining unit 10 is further configured to determine the Hamming distance between the target document and the second document based on the first semantic fingerprint and the second semantic fingerprint;

[0131] The determining unit 10 is further configured to determine that the similarity is greater than the preset similarity threshold when the Hamming distance is less than or equal to a third preset value.

[0132] Optionally, the document detection device 1 further includes: an update unit;

[0133] The update unit is used to replace the second weight value in the preset text feature library with the first weight value according to a preset replacement rule, thereby updating the preset text feature library.

[0134] Optionally, the document detection device 1 further includes: a search unit;

[0135] The search unit is configured to search for a first text feature that is different from the target text feature from the at least one text feature, and to obtain a third weight value corresponding to the first text feature from the at least one weight value;

[0136] The updating unit is further configured to add the first text feature and the third weight value to the preset text feature library according to the target level, thereby updating the preset text feature library.

[0137] This application provides a document detection device, which includes: extracting at least one text feature from a target document, and determining at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document; when a target text feature is found in a preset text feature library, determining a first weight value corresponding to the target text feature from the at least one weight value, and obtaining a second weight value corresponding to the target text feature and a target level corresponding to the target text feature from the preset text feature library; processing the first weight value and the second weight value to obtain a first value; when the first value is greater than a first preset value, performing response processing on the target document based on a response scheme corresponding to the target level; by adopting the above implementation scheme, events are abstracted as documents, and the matching of events with the preset text feature library is achieved by matching the text features of the document with the preset text feature library, so that the level corresponding to the event can be determined by matching with the preset text feature library as soon as the event occurs, thereby responding to the event in a timely manner according to the response scheme corresponding to the level, and achieving the purpose of improving the real-time performance of event detection.

[0138] Figure 6 A schematic diagram of the composition structure of a document inspection device 1 provided in this application embodiment. Figure 2 In practical applications, based on the same disclosed concept of the above embodiments, such as Figure 6 As shown, the document detection device 1 in this embodiment includes: a processor 13, a memory 14, and a communication bus 15.

[0139] In a specific embodiment, the aforementioned determining unit 10, searching unit 11, data processing unit 12, acquiring unit, generating unit, comparing unit, updating unit, and adding unit can be implemented by a processor 13 located on the document detection device 1. The processor 13 can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, or microprocessor. It is understood that for different document detection devices, the electronic device used to implement the above processor functions can also be other types; this embodiment does not specifically limit this.

[0140] In this embodiment, the communication bus 14 is used to realize the connection communication between the processor 13 and the memory 13; when the processor 13 executes the running program stored in the memory 13, it implements the following document detection method:

[0141] Extract at least one text feature from the target document, and determine at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document;

[0142] If a target text feature is found in a preset text feature library, a first weight value corresponding to the target text feature is determined from the at least one weight value, and a second weight value corresponding to the target text feature and a target level corresponding to the target text feature are obtained from the preset text feature library; the preset text feature library stores preset text features, preset weight values ​​corresponding to the preset text features, and preset levels; the target text feature is a text feature among the at least one text feature.

[0143] The first weight value and the second weight value are processed to obtain the first value;

[0144] If the first value is greater than the first preset value, the target document is processed based on the response scheme corresponding to the target level.

[0145] Optionally, the processor 13 is further configured to acquire and analyze historical documents, obtain at least one initial feature in the historical documents, determine at least one initial weight value corresponding to the at least one initial feature based on the number of times the at least one initial feature appears in the historical documents, and generate the preset text feature library based on the at least one initial feature and the at least one initial weight value.

[0146] Optionally, the processor 13 is further configured to sequentially compare the at least one initial weight value with a second preset value; if the first weight value among the at least one initial weight value is greater than the second preset value, determine a first initial feature corresponding to the first weight value from the at least one initial feature; and generate the preset text feature library based on the first initial feature and the first weight value.

[0147] Optionally, the processor 13 is further configured to acquire and analyze the second document to obtain a set of weight values ​​corresponding to the text features in the second document; determine the similarity between the target document and the second document based on the at least one weight value and the set of weight values; and prohibit document detection of the second document when the similarity is greater than a preset similarity threshold, wherein the second document is a document whose preset discrimination standard value is less than that of the target document.

[0148] Optionally, the processor 13 is further configured to input the at least one weight value and the set of weight values ​​into a preset hash function to obtain a first set of hash values ​​and a second set of hash values; determine a first set of real number vectors based on the first set of hash values, and determine a second set of real number vectors based on the second set of hash values; determine a first semantic fingerprint corresponding to the target document based on the first set of real number vectors, and determine a second semantic fingerprint corresponding to the second document based on the second set of real number vectors; determine the Hamming distance between the target document and the second document based on the first semantic fingerprint and the second semantic fingerprint; and determine that the similarity is greater than the preset similarity threshold if the Hamming distance is less than or equal to a third preset value.

[0149] Optionally, the processor 13 is further configured to replace the second weight value in the preset text feature library with the first weight value according to a preset replacement rule, thereby updating the preset text feature library.

[0150] Optionally, the processor 13 is further configured to find a first text feature that is different from the target text feature from the at least one text feature, and obtain a third weight value corresponding to the first text feature from the at least one weight value; add the first text feature and the third weight value to the preset text feature library according to the target level, thereby updating the preset text feature library.

[0151] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors and applied in a document detection device. The computer program implements the document detection method described above.

[0152] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the document detection methods described in the various embodiments of this disclosure.

[0154] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A document detection method, characterized in that, The method includes: Extract at least one text feature from the target document, and determine at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document; Obtain and analyze the second document to obtain a set of weight values ​​corresponding to the text features in the second document; The similarity between the target document and the second document is determined based on the at least one weight value and the set of weight values. When the similarity is greater than a preset similarity threshold, document detection is prohibited for the second document, which is a document whose preset discrimination standard value is less than that of the target document; If a target text feature is found in a preset text feature library, a first weight value corresponding to the target text feature is determined from the at least one weight value, and a second weight value corresponding to the target text feature and a target level corresponding to the target text feature are obtained from the preset text feature library; the preset text feature library stores preset text features, preset weight values ​​corresponding to the preset text features, and preset levels; the target text feature is a text feature among the at least one text feature. The first weight value and the second weight value are processed to obtain the first value; If the first value is greater than the first preset value, the target document is processed based on the response scheme corresponding to the target level.

2. The method according to claim 1, characterized in that, Before extracting at least one text feature from the target document, the method further includes: Acquire and analyze historical documents to obtain at least one initial feature in the historical documents, and determine at least one initial weight value corresponding to the at least one initial feature based on the number of times the at least one initial feature appears in the historical documents; The preset text feature library is generated based on the at least one initial feature and the at least one initial weight value.

3. The method according to claim 2, characterized in that, The step of generating the preset text feature library based on the at least one initial feature and the at least one initial weight value includes: The at least one initial weight value is compared with the second preset value in sequence; If the first weight value among the at least one initial weight value is greater than the second preset value, the first initial feature corresponding to the first weight value is determined from the at least one initial feature; The preset text feature library is generated based on the first initial feature and the first weight value.

4. The method according to claim 1, characterized in that, Determining the similarity between the target document and the second document based on the at least one weight value and the set of weight values ​​includes: The at least one weight value and the set of weight values ​​are respectively input into a preset hash function to obtain a first set of hash values ​​and a second set of hash values; The first set of real number vectors is determined based on the first set of hash values, and the second set of real number vectors is determined based on the second set of hash values. The first semantic fingerprint corresponding to the target document is determined based on the first set of real number vectors, and the second semantic fingerprint corresponding to the second document is determined based on the second set of real number vectors; The Hamming distance between the target document and the second document is determined based on the first semantic fingerprint and the second semantic fingerprint; If the Hamming distance is less than or equal to a third preset value, the similarity is determined to be greater than the preset similarity threshold.

5. The method according to claim 1, characterized in that, When the first value is greater than the first preset value, after processing the target document according to the response scheme corresponding to the target level, the method further includes: The second weight value in the preset text feature library is replaced by the first weight value according to a preset replacement rule, thereby updating the preset text feature library.

6. The method according to claim 1, characterized in that, When the first value is greater than the first preset value, after processing the target document according to the response scheme corresponding to the target level, the method further includes: Find a first text feature that is different from the target text feature from the at least one text feature, and obtain a third weight value corresponding to the first text feature from the at least one weight value; The first text feature and the third weight value are added to the preset text feature library according to the target level, thereby updating the preset text feature library.

7. A document detection device, characterized in that, The device includes: A determining unit is configured to extract at least one text feature from a target document and determine at least one weight value corresponding to the at least one text feature based on the number of times the at least one text feature appears in the target document; The determining unit is further configured to acquire and analyze a second document to obtain a set of weight values ​​corresponding to text features in the second document; determine the similarity between the target document and the second document based on the at least one weight value and the set of weight values; prohibit document detection of the second document when the similarity is greater than a preset similarity threshold, wherein the second document is a document whose preset discrimination standard value is less than that of the target document; when a target text feature is found in a preset text feature library, determine a first weight value corresponding to the target text feature from the at least one weight value, and acquire a second weight value corresponding to the target text feature and a target level corresponding to the target text feature from the preset text feature library; the preset text feature library stores preset text features, preset weight values ​​corresponding to the preset text features, and preset levels; the target text feature is a text feature among the at least one text feature; A data processing unit is used to process the first weight value and the second weight value to obtain a first value; The response unit is used to process the target document based on the response scheme corresponding to the target level when the first value is greater than the first preset value.

8. A document inspection device, characterized in that, The device includes: a processor, a memory, and a communication bus; when the processor executes the running program stored in the memory, it implements the method as described in any one of claims 1-6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images