Deduplication method, device, equipment and storage medium based on perceptual neural network

Abstract

The present application provides a deduplication method, device, equipment and storage medium based on a perceptual neural network, which inputs the text to be stored into the overall perception layer to determine the text macro-features of the text to be stored, and determines the first comparison library based on the text macro-features and the text library; inputs the text to be stored into the keyword matching layer to determine the keyword matching result, and determines the second comparison library based on the keyword matching result and the first comparison library; performs a full-text search on the second comparison library based on the text to be stored to determine the third comparison library; determines whether there is a text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and enters the text to be stored into the text library if it does not exist. Thus, the hierarchical processing of the perceptual neural network simulates the human perception mechanism, and the text to be stored and the text library are screened and compared layer by layer to determine whether to enter the text to be stored into the text library, which effectively improves the efficiency and accuracy of deduplication.

Description

Technical Field

[0001] The present application relates to the field of computer data processing technology, and in particular to a deduplication method, device, equipment and storage medium based on a perceptual neural network. Background Art

[0002] In scenarios involving text retrieval, the construction and maintenance of text libraries are key links. However, when entering new texts into text libraries, data redundancy is often encountered, that is, repeated or highly similar texts are entered multiple times. This not only takes up additional storage space, but also leads to a decline in user experience when searching, because the search engine will return a large number of repeated or similar results. Most of the existing deduplication methods are based on one-by-one comparison. As the amount of data increases, the efficiency of this method becomes very low, and the accuracy of deduplication is not high. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present application provides a deduplication method, device, equipment and storage medium based on a perceptual neural network, aiming to solve the technical problems of low deduplication efficiency and low deduplication accuracy when entering new text in the prior art. Specifically:

[0004] In a first aspect, an embodiment of the present application provides a deduplication method based on a perceptual neural network, the method comprising: inputting the text to be stored into the overall perception layer to determine the text macro features of the text to be stored, and determining a first comparison library based on the text macro features and the text library; inputting the text to be stored into the keyword matching layer to determine the keyword matching result, and determining a second comparison library based on the keyword matching result and the first comparison library; performing a full-text search on the second comparison library based on the text to be stored to determine a third comparison library; determining whether there is text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and if not, entering the text to be stored into the text library.

[0005] In the second aspect, an embodiment of the present application provides a deduplication device based on a perceptual neural network, which includes: an overall perception module, which is used to input the text to be stored into the overall perception layer to determine the text macro-features of the text to be stored, and determine a first comparison library based on the text macro-features and the text library; a keyword matching module, which is used to input the text to be stored into the keyword matching layer to determine the keyword matching result, and determine the second comparison library based on the keyword matching result and the first comparison library; a full-text retrieval module, which is used to perform a full-text search on the second comparison library based on the text to be stored to determine a third comparison library; a library entry determination module, which is used to determine whether there is a text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and if not, enter the text to be stored into the text library.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: one or more processors, a memory, and one or more applications. The one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method of the first aspect.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method of the first aspect.

[0008] In the technical solution provided by the present application, the text macro features of the text to be stored are determined by inputting the text to be stored into the overall perception layer, and the first comparison library is determined based on the text macro features and the text library; the text to be stored is input into the keyword matching layer to determine the keyword matching result, and the second comparison library is determined based on the keyword matching result and the first comparison library; the third comparison library is determined by performing a full-text search on the second comparison library based on the text to be stored; based on a preset similarity algorithm, it is determined whether there is a text matching the text to be stored in the third comparison library, and if not, the text to be stored is entered into the text library. Thus, the hierarchical processing of the perceptual neural network simulates the human perception mechanism, and the text to be stored is screened and compared with the text library layer by layer to determine whether the text to be stored is entered into the text library, which effectively improves the efficiency and accuracy of deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments and drawings obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0010] Figure 1 A flow chart of a deduplication method based on a perceptual neural network proposed in an embodiment of the present application is shown.

[0011] Figure 2 A flow chart of another deduplication method based on a perceptual neural network proposed in an embodiment of the present application is shown.

[0012] Figure 3 A structural block diagram of a deduplication device based on a perceptual neural network proposed in an embodiment of the present application is shown.

[0013] Figure 4 A structural block diagram of an electronic device proposed in an embodiment of the present application is shown.

[0014] Figure 5 A structural block diagram of a computer-readable storage medium proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0017] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0018] It should be further understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0019] In scenarios involving text retrieval, the construction and maintenance of text libraries (such as question banks) are key links. However, when entering new texts (such as new test questions), text libraries often encounter data redundancy problems, that is, repeated or highly similar texts are entered multiple times. This not only takes up additional storage space, but also leads to a decline in user experience when searching, because search engines return a large number of repeated or similar results. To solve this problem, it is usually necessary to perform deduplication before entering new text. Traditional deduplication methods are mostly based on one-by-one comparison. As the amount of data increases, this method becomes very inefficient and difficult to adapt to the growing size of the text library. In addition, when dealing with texts with similar semantics but different expressions, the accuracy of traditional methods is also unsatisfactory, and it is easy to make misjudgments or omissions.

[0020] Therefore, in order to solve these problems, this application introduces the concept of perceptual neural network, which simulates the human perception mechanism through the hierarchical processing of perceptual neural network, screens and compares the text to be stored with the text library layer by layer, and uses the hierarchical neural network structure to efficiently identify and filter repeated or similar texts to be stored, effectively improving the efficiency and accuracy of deduplication, thereby providing users with a better retrieval experience, while saving the space resources and maintenance costs of the text library.

[0021] The following describes it in conjunction with specific embodiments.

[0022] See also Figure 1 , Figure 1 The present application provides a deduplication method based on a perceptual neural network, which is applied to a deduplication system based on a perceptual neural network. The deduplication method includes at least steps S110 to S140, specifically:

[0023] Step S110: input the text to be stored into the overall perception layer to determine the text macro features of the text to be stored, and determine a first comparison library based on the text macro features and the text library.

[0024] In an embodiment of the present application, the text to be stored is a new text to be entered into the text library, and the text content in the text to be stored can include at least one of Chinese characters, numeric characters, English characters and special characters. Among them, special characters include: α, β, γ, δ, ɛ, ζ, η, θ, ι, κ, λ, μ, ν, ξ, π, ρ, σ, τ, υ, φ, χ, ψ, ω, Γ, Δ, ∑, △, ▽, ∫, ∮, ≠, ≤, ≥, ≈, ∈, {, 〈, \\, ‖, ∠, ◇, ∞, ←, →, ~, ! , =, +, -, *, ), @, #, ￥, %, &. Further, the text to be stored can also include illustrations, options, large and small questions, tables and formulas.

[0025] In the present application, the overall perception layer is used to simulate the overall perception of human beings towards the stored text, and the text macro-features are used to characterize the specific overall perception information; the first comparison library is a collection of texts in the text library that match the text macro-features of the text to be stored.

[0026] In some embodiments, macro features may include text length information, illustration information, option information, question information, table information, and formula information; text macro features may be used to indicate the text length of the text to be stored, whether the text to be stored contains illustrations, whether the text to be stored has options, whether the text to be stored is divided into questions, whether the text to be stored contains tables, and whether the text to be stored has embedded formulas.

[0027] Specifically, after obtaining the text to be stored, the text to be stored is first input into the overall perception layer, the overall perception layer extracts the macro features in the text to be stored, and then determines the text macro features of the text to be stored, and then obtains the text matching the text macro features of the text to be stored from the text library as the first comparison library, so as to filter out the text similar to it from the text library according to the overall perception of the text to be stored, which provides a basis for subsequent fine screening.

[0028] In some implementations, the text length of the text to be stored can be determined based on the number of Chinese characters, numeric characters, English characters, and special characters matched after removing the Html tags.

[0029] In some implementations, regular expressions can be used to match img tags, option features, question features, table tags, and text features of formulas, and then determine based on the matching results whether the text to be stored contains illustrations, whether there are options, whether it is divided into questions, whether it contains tables, and whether it contains formulas.

[0030] Furthermore, the option features are: [AF][\s\.、]+.*(?=\s[AF]|<br[\s / ]*> );The characteristics of the big and small questions are: (?<=<br[\s / ]*> )\([1-5]\).*(?<=<br[\s / ]*> ); The text features of the formula are pairs of "$" marks or pairs of "$$" marks.

[0031] For example, the text to be stored is "As shown in the figure, vertices A and B of △ABC are on ⊙O, point C is inside ⊙O (O and C are on the same side of AB), ∠AOB=66°, then the degree of ∠C may be () A. 33°B. 43°C. 24°D. 23", input the text to be stored into the overall perception layer, and the macro features of the text are "text length: Chinese characters 45, numeric characters 10, English characters 17, special characters 5; whether it contains illustrations: yes; whether there are options: yes; whether it is divided into large and small questions: no; whether it contains tables: no; whether it is embedded with formulas: no".

[0032] Step S120: input the text to be stored into the keyword matching layer to determine the keyword matching result, and determine the second comparison library based on the keyword matching result and the first comparison library.

[0033] In the embodiment of the present application, the keyword matching result is the keyword in the text to be stored in the library; the keyword matching layer is used to determine the keywords in the text to be stored in the library, and then select the text matching the keywords from the first comparison library as the second comparison library.

[0034] It can be understood that the keywords are keywords stored in the keyword library, and the number of keywords in the keyword library is limited. For example, when the text library is a question library and the text to be stored is a new test question, the keywords can be subject keywords pre-stored in the keyword library. Further, the keywords can be stored in the keyword library according to preset rules. For example, when the text library is a question library and the text to be stored is a new test question, the keywords can be stored according to subject partitions, such as junior high school mathematics (rational numbers, irrational numbers, integers, decimals, fractions, number axes, opposite numbers, absolute values, triangles, parallelograms, circles, scientific notation, square roots, functions, etc.), junior high school Chinese (viewing the sea, classical Chinese, Tang poetry, Song lyrics, lotus pond moonlight, etc.), junior high school biology, etc.

[0035] In some implementations, the keyword library can be continuously optimized through machine learning algorithms to adapt to updates and changes in related text content. This customized keyword matching mechanism improves the ability to recognize professional terms and concepts, thereby enhancing the accuracy of semantic analysis.

[0036] In some implementations, the keywords in the text to be stored can be directly obtained by searching for a character string. Specifically, after the text to be stored is input into the keyword matching layer, the keyword matching layer performs a character string search on the text to be stored, and then uses the keywords found in the keyword library as the keyword matching result, and then obtains the text containing the keyword matching result from the first comparison library as the second comparison library.

[0037] In some implementations, the keywords in the text to be stored can be obtained based on the semantic recognition results of the text to be stored. Specifically, after the text to be stored is input into the keyword matching layer, the keyword matching layer performs semantic recognition on the text to be stored to determine the semantic recognition result, and then performs a string search on the semantic recognition result, extracts the keywords belonging to the keyword library as the keyword matching result, and then obtains the text containing the keyword matching result from the first comparison library as the second comparison library.

[0038] In some implementations, the keywords in the text to be stored can also be determined according to the above two keyword acquisition methods at the same time. According to actual needs, the intersection or union of the keywords obtained by the above two keyword acquisition methods can be used as the keyword matching result of the text to be stored.

[0039] It is understandable that since the number of keywords in the text to be stored is limited, and the number of texts containing a certain keyword is also limited, performing semantic analysis through step S120 can further reduce the amount of data that needs to be compared in detail later on the basis of step S110, thereby improving the pertinence and efficiency of subsequent deduplication processing.

[0040] Step S130: Perform a full-text search on the second comparison library based on the text to be stored to determine a third comparison library.

[0041] In an embodiment of the present application, the third comparison library is a collection of texts in the second comparison library that match the text to be stored. Since the previous two layers of screening have greatly reduced the amount of text data to be compared, full-text retrieval will become more efficient at this stage compared to the existing technology.

[0042] It should be noted that the prior art mentioned here refers to the scenario of using full-text search without data screening. If full-text search is used without data screening, when professional terms, new concepts or spelling errors appear, the full-text search may not accurately match the relevant text, resulting in missed detection or false detection.

[0043] Specifically, the text to be stored is segmented, and then based on the segmentation result, a search is performed in the inverted index established according to the text in the second comparison library, and then a collection of texts that meet the matching requirements is used as the third comparison library.

[0044] Step S140: determining whether there is text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and if not, entering the text to be stored into the text library.

[0045] In an embodiment of the present application, when there is no text matching the text to be entered in the third comparison library, it means that there is no text in the text library that is repeated or highly similar to the text to be entered, and the text to be entered meets the entry conditions, so that the text to be entered can be entered into the text library.

[0046] Specifically, the text to be stored and each text in the third comparison library may be matched for string similarity based on a preset similarity algorithm, and then it is determined whether there is a text in the third comparison library that matches the text to be stored according to the string similarity matching result.

[0047] In some implementations, the preset similarity algorithm may be a cosine similarity algorithm or a Jaccard similarity algorithm.

[0048] Furthermore, if there is text matching the text to be entered in the third comparison library, it means that there is text in the text library that is repeated or highly similar to the text to be entered, and it is confirmed that the text to be entered does not meet the entry conditions, thereby prohibiting the text to be entered from being entered into the text library.

[0049] In the embodiment of the present application, when each step determines the matching degree (or similarity) between the text to be stored and the text in the text library for screening, a corresponding preset threshold is set. If the preset threshold is exceeded, the two texts are considered similar. The preset threshold in each specific step can be set according to actual needs, and the present application does not impose any restrictions on this.

[0050] It should be noted that as the size of the text library grows, the computing resources required for full-text retrieval and similarity calculation also increase, which may lead to slower processing speed and higher costs. Therefore, the embodiment of the present application first performs screening through a perceptual neural network before performing full-text retrieval and similarity calculation, which can effectively reduce the data required to be processed during full-text retrieval and similarity calculation, and can set a more reasonable preset threshold for screening to avoid missed detection due to high threshold settings, and also avoid large amounts of calculation and slow processing speeds due to low threshold settings. This can greatly improve the efficiency and accuracy of deduplication.

[0051] The embodiment of the present application provides a deduplication method based on a perceptual neural network, which determines the text macro-features of the text to be stored in the overall perception layer by inputting the text to be stored in the library, and determines the first comparison library based on the text macro-features and the text library; inputs the text to be stored in the library into the keyword matching layer to determine the keyword matching result, and determines the second comparison library based on the keyword matching result and the first comparison library; performs a full-text search on the second comparison library based on the text to be stored in the library to determine the third comparison library; determines whether there is a text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and enters the text to be stored into the text library if it does not exist. Thus, the hierarchical processing of the perceptual neural network simulates the human perception mechanism from shallow to deep, and the text to be stored in the library is screened and compared layer by layer with the text library to determine whether to enter the text to be stored in the text library, while ensuring the efficiency and accuracy of deduplication, the demand for computing resources is minimized, and the consumption of computing resources and processing efficiency are effectively balanced.

[0052] See also Figure 2 , Figure 2 Another deduplication method based on a perceptual neural network provided in an embodiment of the present application is shown. The method is applied to a deduplication system based on a perceptual neural network. The method includes at least steps S210 to S250. Specifically:

[0053] Step S210: input the text to be stored into the overall perception layer to determine the text macro features of the text to be stored, and determine a first comparison library based on the text macro features and the text library.

[0054] Step S220: input the text to be stored into the keyword matching layer to determine the keyword matching result, and determine the second comparison library based on the keyword matching result and the first comparison library.

[0055] In the embodiment of the present application, the detailed description of step S210 to step S220 can refer to the aforementioned step S110 to step S120, which will not be repeated here.

[0056] Step S230, if the illustration information and / or table information in the text macro information meets the preset conditions, the text to be stored is input into the detail perception layer to determine the text micro features of the text to be stored, and the fourth comparison library is determined based on the text micro features and the second comparison library.

[0057] Specifically, if it is determined based on the text macro information that the text to be stored contains illustrations and / or tables, the text to be stored needs to be input into the detail perception layer for a more in-depth analysis of the illustrations and / or tables, so as to determine the text micro features of the text to be stored, and finally obtain the text that matches the text micro features from the second comparison library as the fourth comparison library.

[0058] In some embodiments, since the detail perception layer performs a deeper perception of illustrations and tables, after determining that the text to be stored contains illustrations and / or tables, the illustrations and / or tables of the text to be stored can be used as detail perception information. Specifically, if the text to be stored contains illustrations and / or tables, the text to be stored is input into the detail perception layer, the detail perception layer obtains the detail perception information of the text to be stored, and then extracts microscopic features of the detail perception information, and finally determines the text microscopic features of the text to be stored.

[0059] Furthermore, the micro-feature extraction of detail perception information is specifically performed by first extracting the illustration size and determining the illustration size extraction result; and / or extracting the table row and column features and determining the table row and column feature extraction result, and then inserting the size extraction result and / or the table row and column feature extraction result as the text micro-features of the text to be stored.

[0060] As an embodiment, when the text to be stored contains only illustrations, the detail perception information is the illustrations, and the illustration size of the illustrations is extracted to obtain illustration size extraction results, and then the illustration size extraction results are used as text micro features of the text to be stored.

[0061] As another embodiment, when the text to be stored contains only a table, the detail perception information is a table, and the table row and column features of the table are extracted to obtain the table row and column feature extraction results, and then the table row and column feature extraction results are used as the text micro features of the text to be stored.

[0062] As another embodiment, when the text to be stored contains both illustrations and tables, the detail perception information is the illustrations and tables, the illustration size is extracted to obtain the illustration size extraction result, and the table row and column features are extracted to obtain the table row and column feature extraction result, and then the illustration size extraction result and the table row and column feature extraction result are used as the text micro features of the text to be stored.

[0063] In some embodiments, a specific method for extracting the size of an illustration is to obtain images of the illustration projected on 8*8 pixel, 16*16 pixel, and 32*32 pixel canvases, respectively, and then calculate the normalized values ​​respectively. It is understandable that since the actual illustrations of the text are relatively large, it will consume a lot of performance if the original pixels are used for comparison; at the same time, for two similar images, the 8*8 pixel, 16*16 pixel, and 32*32 pixel images projected on the canvas will theoretically be very similar; therefore, in order to reduce the amount of calculation, the illustration thumbnails of the text to be stored can be compared with the illustration thumbnails of the text in the second comparison library. Furthermore, the text whose normalized values ​​of each thumbnail corresponding to the illustration in the second comparison library are within the preset floating range of the normalized values ​​of each thumbnail corresponding to the illustration in the text to be stored can be obtained as the fourth comparison library.

[0064] In some implementations, table row and column features can be extracted by regular expressions, that is, the number of table rows and columns can be calculated. Further, texts in the second comparison library whose table row and column numbers are within a preset floating range of the table row and column numbers of the text to be stored can be obtained as the fourth comparison library.

[0065] It should be noted that when matching at the overall perception layer, keyword matching layer, and detail perception layer, attribute-level deduplication is performed based on the extracted features (including keywords), which is very efficient.

[0066] Step S240: Perform a full-text search on the fourth comparison library based on the text to be stored to determine the third comparison library.

[0067] Specifically, the text to be stored is segmented, and then based on the segmentation result, a search is performed in the inverted index established according to the text in the fourth comparison library, and then a collection of texts that meet the matching requirements is used as the third comparison library.

[0068] In the embodiment of the present application, the rest of the detailed description of step S240 can be found in the aforementioned step S140, which will not be repeated here.

[0069] Step S250: determining whether there is text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and if not, entering the text to be stored into the text library.

[0070] In the embodiment of the present application, the detailed description of step S250 can be found in the aforementioned step S140, which will not be repeated here.

[0071] The embodiment of the present application provides a deduplication method based on a perceptual neural network, which determines the text macro-features of the text to be stored in the overall perception layer by inputting the text to be stored in the library, and determines the first comparison library based on the text macro-features and the text library; inputs the text to be stored in the library into the keyword matching layer to determine the keyword matching result, and determines the second comparison library based on the keyword matching result and the first comparison library; performs a full-text search on the second comparison library based on the text to be stored in the library to determine the third comparison library; determines whether there is a text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and enters the text to be stored into the text library if it does not exist. Thus, the hierarchical processing of the perceptual neural network simulates the human perception mechanism, and the text to be stored in the library is screened and compared layer by layer with the text library to determine whether to enter the text to be stored in the text library, which can efficiently identify and filter repeated or similar texts, greatly reducing the amount of data to be processed during full-text retrieval and similarity calculation, thereby effectively improving the efficiency and accuracy of deduplication.

[0072] See also Figure 3 , Figure 3 The present invention shows a deduplication device 100 based on a perceptual neural network, which includes an overall perception module 110, a keyword matching module 120, a full-text search module 130 and a storage determination module 140. Specifically:

[0073] The overall perception module 110 is used to input the text to be stored into the overall perception layer to determine the text macro features of the text to be stored, and determine the first comparison library based on the text macro features and the text library;

[0074] The keyword matching module 120 is used to input the text to be stored into the keyword matching layer to determine the keyword matching result, and determine the second comparison library based on the keyword matching result and the first comparison library;

[0075] A full-text search module 130, used for performing a full-text search on the second comparison library based on the text to be stored to determine a third comparison library;

[0076] The storage determination module 140 is used to determine whether there is a text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and if not, enter the text to be stored into the text library.

[0077] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated one by one in the device embodiment.

[0078] See also Figure 4 , Figure 4 An electronic device 200 proposed in an embodiment of the present application is shown, and the electronic device 200 may be a computer, a tablet, a mobile phone, etc. The electronic device 200 in the present application may include one or more of the following components: a processor 210, a memory 220, and one or more applications, wherein the one or more applications may be stored in the memory 220 and configured to be executed by one or more processors 210, and the one or more applications are configured to execute the method described in the foregoing method embodiment.

[0079] The processor 210 may include one or more processing cores. The processor 210 uses various interfaces and lines to connect various parts of the entire electronic device 200, and executes various functions and processes data of the electronic device 200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 220, and calling data stored in the memory 220. Optionally, the processor 210 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 210 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 210, but may be implemented separately through a communication chip.

[0080] The memory 220 may include a random access memory (RAM) or a read-only memory (ROM). The memory 220 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as an overall perception function, a detail perception function, a keyword matching function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as text macro features, keyword matching results, text micro features, etc.) created by the electronic device 200 during use.

[0081] See also Figure 5 , Figure 5 The structure block diagram of a computer-readable storage medium proposed in the embodiment of the present application is shown. The computer-readable storage medium 300 stores program codes, and the program codes can be called by a processor to execute the method described in the above method embodiment.

[0082] The computer-readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. Optionally, the computer-readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 300 has storage space for program code 310 for executing any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in a suitable form.

[0083] The present application provides a deduplication method, device, equipment and storage medium based on a perceptual neural network, which inputs the text to be stored into the overall perception layer to determine the text macro-features of the text to be stored, and determines the first comparison library based on the text macro-features and the text library; inputs the text to be stored into the keyword matching layer to determine the keyword matching result, and determines the second comparison library based on the keyword matching result and the first comparison library; performs a full-text search on the second comparison library based on the text to be stored to determine the third comparison library; determines whether there is a text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and enters the text to be stored into the text library if it does not exist. Thus, the hierarchical processing of the perceptual neural network simulates the human perception mechanism, and the text to be stored is screened and compared with the text library layer by layer, which effectively improves the efficiency and accuracy of deduplication.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A deduplication method based on perceptual neural network, characterized in that: The method comprises: Input the text to be stored into the overall perception layer to determine the text macro features of the text to be stored, and determine the first comparison library based on the text macro features and the text library; wherein the text macro features are used to indicate the text length of the text to be stored, whether the text to be stored contains illustrations, whether the text to be stored contains options, whether the text to be stored is divided into large and small questions, whether the text to be stored contains tables, and whether the text to be stored is embedded with formulas. The text length of the text to be stored is determined according to the number of Chinese characters, numeric characters, English characters, and special characters matched after removing the Html tag. Whether the text to be stored contains illustrations, whether the text to be stored contains options, whether the text to be stored is divided into large and small questions, whether the text to be stored contains tables, and whether the text to be stored is embedded with formulas are all determined according to the matching results of regular expressions; Inputting the text to be stored into a keyword matching layer to determine a keyword matching result, and determining a second comparison library based on the keyword matching result and the first comparison library; If the illustration information and table information in the text macro-features meet the preset conditions, the text to be stored is input into the detail perception layer to obtain the detail perception information of the text to be stored; wherein the detail perception information is the illustrations and tables contained in the text to be stored; Projecting the illustrations onto different pixel canvases respectively to extract the illustration size and determine the illustration size extraction result; extracting the table row and column features of the table and determining the table row and column feature extraction result; Using the illustration size extraction result and the table row and column feature extraction result as text micro features of the text to be stored, and determining a fourth comparison library based on the text micro features and the second comparison library; Performing a full-text search on the fourth comparison library based on the text to be stored to determine a third comparison library; Based on a preset similarity algorithm, it is determined whether there is a text matching the text to be stored in the third comparison library, and if not, the text to be stored is entered into the text library.

2. The method according to claim 1, characterized in that The step of inputting the text to be stored into the overall perception layer to determine the text macro features of the text to be stored, and determining the first comparison library based on the text macro features and the text library, includes: Input the text to be stored into the overall perception layer for macro-feature extraction, and determine the text macro-features of the text to be stored; wherein the macro-features include text length information, illustration information, option information, question information, table information and formula information; The texts matching the text macro features are obtained from the text library as the first comparison library.

3. The method according to claim 1, characterized in that The step of inputting the text to be stored into a keyword matching layer to determine a keyword matching result, and determining a second comparison library based on the keyword matching result and the first comparison library includes: Input the text to be stored into the keyword matching layer to perform a string search on the text to be stored, use the found keywords belonging to the keyword library as keyword matching results, and obtain the text containing the keyword matching results from the first comparison library as the second comparison library; or Inputting the text to be stored into a keyword matching layer to perform semantic recognition on the text to be stored; Keywords belonging to the keyword library in the semantic recognition result are extracted as keyword matching results, and text containing the keyword matching results is obtained from the first comparison library as a second comparison library.

4. The method according to claim 1, characterized in that The determining whether there is a text matching the text to be stored in the third comparison library based on a preset similarity algorithm includes: Based on a preset similarity algorithm, character string similarity matching is performed on the text to be stored and each text in the third comparison library respectively; wherein the preset similarity algorithm is a cosine similarity algorithm or a Jaccard similarity algorithm; Determine whether there is text matching the text to be stored in the third comparison library according to the string similarity matching result.

5. A deduplication device based on perceptual neural network, characterized in that: The device comprises: The overall perception module is used to input the text to be stored into the overall perception layer to determine the text macro features of the text to be stored, and determine the first comparison library based on the text macro features and the text library; wherein the text macro features are used to indicate the text length of the text to be stored, whether the text to be stored contains illustrations, whether the text to be stored contains options, whether the text to be stored is divided into large and small questions, whether the text to be stored contains tables, and whether the text to be stored is embedded with formulas. The text length of the text to be stored is determined according to the number of Chinese characters, numeric characters, English characters, and special characters matched after removing the Html tag. Whether the text to be stored contains illustrations, whether the text to be stored contains options, whether the text to be stored is divided into large and small questions, whether the text to be stored contains tables, and whether the text to be stored is embedded with formulas are all determined according to the matching results of regular expressions; A keyword matching module, used for inputting the text to be stored into the keyword matching layer to determine a keyword matching result, and determining a second comparison library based on the keyword matching result and the first comparison library; A full-text retrieval module is used to input the text to be stored into a detail perception layer to obtain detail perception information of the text to be stored if the illustration information and table information in the text macro-features meet preset conditions; wherein the detail perception information is the illustrations and tables contained in the text to be stored; the illustrations are respectively projected onto different pixel canvases to extract the illustration size and determine the illustration size extraction result; the table row and column feature extraction is performed on the table to determine the table row and column feature extraction result; the illustration size extraction result and the table row and column feature extraction result are used as the text micro-features of the text to be stored, and a fourth comparison library is determined based on the text micro-features and the second comparison library; a full-text search is performed on the fourth comparison library based on the text to be stored to determine a third comparison library; The storage determination module is used to determine whether there is a text matching the text to be stored in the third comparison library based on a preset similarity algorithm, and if not, enter the text to be stored into the text library.

6. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 4.

Images