Document duplicate checking method based on language processing and related equipment

Abstract

The invention provides a document duplicate checking method based on language processing and related equipment. The method comprises the following steps: acquiring a to-be-searched document; performing feature extraction on the to-be-searched document to generate to-be-searched text features; comparing the to-be-checked text feature with a target feature vector to obtain a duplicate checking result; wherein the target feature vector comprises a preset text feature corresponding to a preset document library or a specified text feature corresponding to a specified document; and displaying the duplicate checking result.

Description

Technical Field

[0001] The present disclosure relates to the field of language processing, and in particular to a document duplication checking method based on language processing and related equipment. Background Art

[0002] Among the current document duplicate checking technologies, the keyword matching method is simple to implement and is suitable for checking duplicates of identical texts, but it is not able to cope with situations such as word order changes, synonym replacements, or minor text modifications. The N-Gram method can identify simple word order changes or text modifications by comparing continuous substrings in the document, which is more flexible than keyword matching, but it is still difficult to handle synonym replacements or more complex semantic changes. The hash function-based duplicate checking method detects duplication by comparing the hash values ​​converted from the document content. Although it has a fast calculation speed and is suitable for checking duplicates of large-scale data sets, it may cause false positives or false negatives due to the possibility of hash conflicts, and can only detect directly copied content and cannot handle semantic similarity issues. Therefore, the effectiveness of existing document duplicate checking technologies is not high. Summary of the invention

[0003] The present disclosure proposes a document duplication checking method based on language processing and related equipment, so as to solve the technical problem that the effectiveness of document duplication checking technology is not high to a certain extent.

[0004] In a first aspect, the present disclosure provides a document duplication checking method based on language processing, comprising:

[0005] Get the document to be checked;

[0006] Extracting features of the document to be checked to generate features of the text to be checked;

[0007] Comparing the features of the text to be checked with the target feature vector to obtain a duplicate checking result; wherein the target feature vector includes a preset text feature corresponding to a preset document library, or a specified text feature corresponding to a specified document;

[0008] The duplicate checking results are displayed.

[0009] In a second aspect of the present disclosure, a document duplication checking device based on language processing is provided, comprising:

[0010] The acquisition module is used to obtain the document to be searched;

[0011] A feature module is used to extract features of the document to be checked and generate features of the text to be checked;

[0012] A comparison module, used for comparing the to-be-checked text features with a target feature vector to obtain a duplicate checking result; wherein the target feature vector includes a preset text feature corresponding to a preset document library, or a specified text feature corresponding to a specified document;

[0013] A display module is used to display the duplicate checking results.

[0014] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method described in the first aspect.

[0015] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0016] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0017] As can be seen from the above, the document duplication checking method and related equipment based on language processing provided by the present disclosure, by using the existing database to load the sentence embedding model training model, the existing database, feature vector generation, to form a feature vector library of all projects. Then, for the content input by the user, the sentence embedding model training model and the database are used to extract the feature vector, and the cosine similarity algorithm of the feature vector value is used to calculate the similarity between the user input content and the existing items in the database. For the input content, it can be checked and compared in the form of sentences, paragraphs or full texts, and finally the duplication checking results are displayed in a multi-dimensional visual form, so as to achieve accurate similarity evaluation, comprehensive duplication checking coverage, efficient feature extraction and comparison, intuitive duplication checking result display and good flexibility and scalability, which significantly improves the accuracy of duplication checking and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of a document duplication checking architecture based on language processing according to an embodiment of the present disclosure.

[0020] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0021] Figure 3 The present invention is a flowchart of a document duplication checking method based on language processing according to an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of a document duplication checking method based on language processing according to an embodiment of the present disclosure.

[0023] Figure 5 A schematic diagram of the duplicate checking result of an embodiment of the present disclosure.

[0024] Figure 6 Schematic diagram of a document duplication checking device based on language processing according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0028] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0029] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0030] Figure 1 A schematic diagram of a document duplication checking architecture based on language processing according to an embodiment of the present disclosure is shown. Figure 1 , the document duplication checking architecture 100 based on language processing may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0031] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0032] It should be noted that the document duplication checking method based on language processing provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.

[0033] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.

[0034] Processor 202 may be a central processing unit (CPU), a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, processor 202 may include a plurality of processors 202a, 202b, and 202c.

[0035] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2 As shown, the data stored in the memory 204 may include program instructions (for example, program instructions for implementing the document duplicate checking method based on language processing of the embodiment of the present disclosure) and data to be processed (for example, the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0036] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC) etc.), a cellular network, the Internet or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 206 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.

[0037] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0038] The bus 210 can be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0039] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal execution. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0040] Currently, most document duplication checks use the following methods: Duplicate check based on keyword matching. The keyword matching duplicate check method splits the document into several keywords and then matches the keywords. This method is simple to implement and works well for identical texts. However, it cannot handle word order changes, synonym replacements, or minor text modifications. Duplicate check based on N-Gram. The N-Gram method divides the text in the document into continuous substrings (n-grams) of length N and then compares these substrings. This method can identify simple word order changes or text modifications and is more flexible than simple keyword matching, but it cannot effectively handle synonym replacements or more complex semantic changes. Duplicate check based on hash functions. The document content is converted into hash values ​​using hash functions, and the similarity of the hash values ​​is compared to check for duplicates. This method has a fast calculation speed and is suitable for checking for duplicates in large data sets, but this method may have hash conflicts, resulting in false positives or false negatives in duplicate checks. It cannot handle semantic similarities and can only detect direct duplication of content. It can be seen that the existing document duplication detection methods are difficult to deal with word order changes, synonym replacements, and subtle semantic modifications in the text, resulting in limited effectiveness in detecting disguised plagiarized content and prone to false positives or false negatives. These limitations limit their effectiveness in dealing with complex document plagiarism detection scenarios. Therefore, how to improve the accuracy and timeliness of document duplication detection has become a technical problem that needs to be solved urgently.

[0041] In view of this, the embodiments of the present disclosure provide a document duplication checking method and related equipment based on language processing. By using the existing database to load the sentence embedding model training model, feature vectors are generated for the existing database to form a feature vector library for all projects. Then, the sentence embedding model training model and the database are used to extract feature vectors for the content input by the user, and the cosine similarity algorithm of the feature vector value is used to calculate the similarity between the user input content and the existing items in the database. For the input content, it can be checked for duplicates and compared in the form of sentences, paragraphs or full texts, and finally the results of the duplicate checking can be displayed in a multi-dimensional visual form, thereby achieving accurate similarity evaluation, comprehensive duplication checking coverage, efficient feature extraction and comparison, intuitive display of duplicate checking results, and good flexibility and scalability, which significantly improves the accuracy of the duplicate checking and user experience.

[0042] See also Figure 3 , Figure 3 A schematic flow chart of a document duplicate checking method based on language processing according to an embodiment of the present disclosure is shown. The document duplicate checking method based on language processing according to an embodiment of the present disclosure can be deployed on a server or a terminal. Figure 3 In the document duplication checking method 300 based on language processing may further include the following steps.

[0043] In step S310, the document to be searched is obtained.

[0044] The document to be checked may refer to a document or text content that needs to be checked for duplicate content. The document to be checked may be any form of written material such as a paper, report, contract text, email, project description, etc. For example, the document to be checked may be a project document about the establishment of a project.

[0045] In step S320, feature extraction is performed on the document to be searched to generate text features to be searched.

[0046] Feature extraction refers to the process of extracting useful features or information from raw data. For example, to extract a vector representation that can reflect the content characteristics of a document to be checked, a pre-trained CoSENT model can be used to convert a research report into a set of numerical vectors that can represent its semantic information. The feature of the text to be checked can refer to the feature vector extracted from the document to be checked for subsequent duplication checking and comparison. It can capture the semantic information of the document so that it can be compared with other documents for similarity.

[0047] In some embodiments, extracting features from the document to be checked to generate text features to be checked includes: extracting features from the document to be checked based on a trained text feature model to obtain the text features to be checked.

[0048] Among them, by extracting features from the document to be checked and generating text features to be checked, and using the trained text feature model, the document can be converted into a vector representation that can reflect its semantic information, thereby providing accurate basic data for subsequent duplicate checking and comparison, ensuring the accuracy and reliability of the duplicate checking results.

[0049] In some embodiments, the trained text feature model is obtained by training a language processing model based on training samples, including:

[0050] Acquire the training sample, wherein the training sample includes project content and corresponding assessment indicators;

[0051] Performing data enhancement processing on the training samples to obtain a first enhanced sample and a second enhanced sample;

[0052] The first enhanced sample and the second enhanced sample are determined as positive samples, and other training samples are determined as negative samples, and the language processing model is trained to obtain the trained text feature model.

[0053] Among them, by obtaining training samples containing project content and its corresponding assessment indicators, and performing data enhancement processing on them to generate the first enhanced sample and the second enhanced sample, these enhanced samples are used as positive samples, and other training samples are used as negative samples, and then the language processing model is trained, and finally a trained text feature model is obtained. This method enhances the generalization ability and robustness of the model, so that the generated text features can more accurately reflect the real content of the document, thereby improving the accuracy and reliability of the duplicate checking results.

[0054] In some embodiments, performing data enhancement processing on the training sample to obtain a first enhanced sample and a second enhanced sample includes:

[0055] Synonym replacement and / or noise addition are performed on the training samples to obtain different first enhanced samples and second enhanced samples.

[0056] The training samples can be processed by synonym replacement and / or noise addition to generate different first enhanced samples and second enhanced samples, so as to increase the diversity and richness of the training data. In this way, the trained text feature model can better capture the intrinsic semantics of the text, improve the robustness and generalization ability of the model, and thus provide more accurate similarity evaluation results in the duplicate checking process.

[0057] In some embodiments, the first enhanced sample and the second enhanced sample are determined as positive samples, and other training samples are determined as negative samples, and the language processing model is trained to obtain the trained text feature model, including:

[0058] The parameters of the language processing model are adjusted so that the distance between the first enhanced sample and the second enhanced sample in the vector space is smaller, and the distance between the first enhanced sample and the second enhanced sample and the other training samples in the vector space is larger, so as to obtain the trained text feature model.

[0059] Among them, by adjusting the parameters of the language processing model, the distance between the first enhanced sample and the second enhanced sample in the vector space becomes smaller, and the distance between these enhanced samples and other training samples becomes larger, thereby obtaining a trained text feature model. This method enhances the model's ability to recognize semantically similar texts and improves the accuracy and reliability of duplicate checking results.

[0060] In step S330, a comparison is performed based on the text features to be checked and the target feature vector to obtain a duplicate checking result; wherein the target feature vector includes a preset text feature corresponding to a preset document library, or a specified text feature corresponding to a specified document.

[0061] Among them, by comparing the feature vector of the document to be checked with the preset text features in the preset document library or the specified text features of the specified document, the similarity between the document to be checked and the existing documents can be efficiently calculated, thereby obtaining accurate duplicate checking results. It uses the spatial distance of feature vectors to measure the content similarity between documents, ensuring the accuracy and reliability of the duplicate checking process, while improving the efficiency of duplicate checking and user experience.

[0062] In some embodiments, the comparison between the feature of the text to be checked and the target feature vector is performed to obtain a duplicate checking result, including:

[0063] Calculating a first similarity between the to-be-queried text feature and the preset text feature;

[0064] sorting the preset text features from high to low based on the first similarity;

[0065] The preset documents corresponding to the first n preset text features in the ranking and the corresponding first similarities are determined as the duplicate checking results, where n is a positive integer.

[0066] Among them, by calculating the first similarity between the text feature to be checked and the preset text feature, and sorting the preset text features from high to low according to the first similarity, finally the preset documents corresponding to the first n preset text features in the sorting and their similarities are selected as the duplicate checking results, so as to efficiently identify the existing documents most similar to the document to be checked and their similarity levels, ensure the accuracy and reliability of the duplicate checking results, and provide a clear similarity reference.

[0067] In some embodiments, the preset text features are obtained by extracting features from preset documents in the preset document library based on a trained text feature model.

[0068] Among them, the trained text feature model is used to extract features of preset documents in the preset document library to obtain preset text features, thereby ensuring the consistency and accuracy of the feature representation of the document to be checked and the preset document during the duplicate checking process, thereby improving the reliability and accuracy of the duplicate checking results.

[0069] In some embodiments, the comparison between the feature of the text to be checked and the target feature vector is performed to obtain a duplicate checking result, including:

[0070] Performing feature extraction on the designated document to obtain designated text features of multiple dimensions, wherein the multiple dimensions include a single sentence, a single paragraph, and an entire document;

[0071] Calculating a second similarity between the to-be-queried text feature and the specified text feature;

[0072] The duplicate checking result is determined based on the second similarity, and the duplicate checking result includes at least one of the second similarities in multiple dimensions or a weighted sum based on the second similarities.

[0073] Among them, the specified text features of multiple dimensions (including single sentences, single paragraphs, and entire documents) are obtained by extracting features from the specified document, and the second similarity between the text features to be checked and these specified text features is calculated. Finally, the duplicate checking result is determined based on the second similarity. The duplicate checking result can include the second similarity under each dimension or its weighted sum. This method can comprehensively evaluate the similarity between the document to be checked and the specified document at different granularities, thereby providing more detailed and accurate duplicate checking results, and enhancing the practicality and reliability of the duplicate checking system.

[0074] In step S340, the duplicate checking result is displayed.

[0075] Among them, a variety of display methods are used to allow users to intuitively understand the duplicate checking results.

[0076] In some embodiments, displaying the duplicate checking result includes:

[0077] Displaying the duplicate checking results in the form of a result list, wherein the result list includes the duplicate checking results sorted from high to low based on the degree of duplication;

[0078] Alternatively, the duplicate checking results are displayed based on different dimensions.

[0079] Among them, users can input the documents to be checked through the interface. The input documents to be checked are first sent to the duplicate checking service, and the feature vector of the input content is generated through feature vector extraction (such as CoSENT model). The duplicate checking service uses the feature vector value to compare with the project vector library to calculate the similarity between the input content and the existing projects. The duplicate checking results can be displayed in a list format, and the items in the list are sorted from high to low according to the degree of duplication. Each item is followed by a similarity value, which intuitively shows the similarity between the item and the user's input content. The user can click on any similarity data in the list to view the comprehensive similarity between the input content and the specific item. In addition, you can also choose to view specific similarity details according to three dimensions: single sentence, single paragraph and full text. For example, when the user clicks on any item, such as "Project A", you can see that the comprehensive similarity between the item and the user's input content is 85%. The user can also choose to view specific similarity details, and the system will display: Single sentence dimension: List the sentences in the research report that are more similar to Project A, and mark the similarity score. Single paragraph dimension: List the paragraphs in the research report that are more similar to Project A, and mark the similarity score. Full-text dimension: Provides an overall similarity analysis between the research report and Project A.

[0080] See also Figure 4 , Figure 4 A flow chart of a document duplication checking method based on language processing according to an embodiment of the present disclosure is shown. Figure 4 First, you can build a service framework based on the AI.DJL library. The AI.DJL library is an open source deep learning framework that provides powerful deep learning-related API interfaces and supports multiple underlying engines including MXNet, TensorFlow, and PyTorch. When building the service framework, this device uses the Java microservice framework. The AI.DJL class library is introduced into the framework, and on the basis of the open source framework, it encapsulates capabilities including model loading, training, and reasoning. At the same time, for duplicate checking services, it encapsulates related interfaces such as text similarity comparison, text retrieval, and vector generation. The framework integrates ES components. For a large number of structured and unstructured scientific and technological project samples and project databases, ES is used as the basic library for storage and retrieval, so that the duplicate checking service has a large amount of data processing and storage capabilities.

[0081] Collect information on relevant science and technology projects in the power sector, extract research content and assessment indicators from science and technology projects as sample data for model training, train the model on these sample data, and obtain the Text2Vec model related to science and technology projects in the power sector. In the implementation process, firstly, supervised training is performed based on the basic text vector model using the CoSENT model training method. After the data enhancement module, the input content is converted into two different versions of token embeddings, and then the different vector representations of the two versions can be obtained through the encoder. The two different versions of the vector representation are mutually positive samples, and the other samples are negative samples. The ultimate goal of the training is to make the representations that originally belong to the same sample closer in space and away from other negative samples. Through this training method, the text vector model is obtained, ensuring the consistency of the training goal and the reasoning goal. Evaluation tests were carried out on relevant data sets such as science and technology projects, and good model evaluation results were obtained. The discrimination of short texts has been significantly improved. The accuracy of the representation of science and technology project texts in the power sector can reach more than 95%, and the accuracy of the text similarity obtained by the model can reach more than 90%.

[0082] Then, construct the text vector. You can first collect information about science and technology projects related to the power sector and store the information in the project library. Then, based on the duplicate checking service, load the trained text vector model, and generate corresponding vector data for all projects in the project library through the vector generation interface, and store it in the project vector library.

[0083] Next, multi-item similarity calculation is performed. For the content to be checked for duplicate content input by the user, the duplicate checking service is used to generate the feature vector value of the content to be checked for duplicate content. Then, the cosine similarity algorithm interface in the duplicate checking service is used to compare the feature vector value of the input content with the project vector library, and the data is returned in descending order of similarity.

[0084] Then, the single-item similarity calculation is performed. For the content to be checked for duplicate content input by the user, the feature vector value of the content to be checked for duplicate content is generated. When selecting the duplicate information of a certain item, the feature vector value of the item can be generated according to the three dimensions of single sentence, single paragraph, and whole article, and then the cosine similarity calculation is performed according to the feature vector values ​​of the three dimensions to finally obtain the similarity result.

[0085] The user inputs the information to be checked for duplicates, and the device returns multiple analysis results based on the calculation of the duplicate checking service. After the user enters the duplicate checking information, it is first displayed in the form of a project list. The list results are displayed in descending order of repetition, and the similarity results are displayed on the back end of each project. When the similarity data is clicked, the comprehensive similarity between the input result and the project is displayed. At the same time, the similarity can be displayed in three dimensions: single sentence, single paragraph and full text, such as Figure 5 shown. Figure 5A schematic diagram of a duplicate checking result according to an embodiment of the present disclosure is shown.

[0086] It can be seen that according to the method of the embodiment of the present disclosure, based on the Natural Language Processing (NLP) technology, combined with the AI.DJL framework and the semantic similarity algorithm, the word feature vector value is extracted, and the cosine similarity algorithm is used to calculate the feature vector of the input content and the vector in the text feature vector library, and finally the duplicate check result is obtained. The duplicate check results are displayed in the duplicate check result list according to the similarity from high to low, and the project name and project research content are displayed. According to the needs, the repetition degree of the input content and a certain project can be checked, and the repetition degree can be checked by sentence, paragraph, and full text. The present invention adopts natural language technology, CoSENT training model, ES (Elasticsearch) search engine, AI.DJL deep learning framework to develop java application services to realize the retrieval and duplicate check of information such as project establishment research content and achievement indicators.

[0087] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0088] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a document duplication checking device based on language processing, see Figure 6 , the document duplication checking device based on language processing, the device comprising:

[0090] The acquisition module is used to obtain the document to be searched;

[0091] A feature module is used to extract features of the document to be checked and generate features of the text to be checked;

[0092] A comparison module, used for comparing the to-be-checked text features with a target feature vector to obtain a duplicate checking result; wherein the target feature vector includes a preset text feature corresponding to a preset document library, or a specified text feature corresponding to a specified document;

[0093] A display module is used to display the duplicate checking results.

[0094] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0095] The device of the above embodiment is used to implement the corresponding document duplication checking method based on language processing in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0096] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the document duplication checking method based on language processing as described in any of the above embodiments.

[0097] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0098] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the document duplication checking method based on language processing as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0099] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0100] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0101] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0102] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A document duplication checking method based on language processing, comprising: Get the document to be checked; Extracting features of the document to be checked to generate features of the text to be checked; Comparing the to-be-checked text features with the target feature vector to obtain a duplicate checking result; wherein the target feature vector includes a preset text feature corresponding to a preset document library, or a specified text feature corresponding to a specified document; The duplicate checking results are displayed.

2. The method according to claim 1, wherein: Based on the comparison between the features of the text to be checked and the target feature vector, a duplicate checking result is obtained, including: Calculating a first similarity between the to-be-queried text feature and the preset text feature; sorting the preset text features from high to low based on the first similarity; The preset documents corresponding to the first n preset text features in the ranking and the corresponding first similarities are determined as the duplicate checking results, where n is a positive integer.

3. The method according to claim 1, wherein: Based on the comparison between the features of the text to be checked and the target feature vector, a duplicate checking result is obtained, including: Performing feature extraction on the designated document to obtain designated text features of multiple dimensions, wherein the multiple dimensions include a single sentence, a single paragraph, and an entire document; Calculating a second similarity between the to-be-queried text feature and the specified text feature; The duplicate checking result is determined based on the second similarity, and the duplicate checking result includes at least one of the second similarities in multiple dimensions or a weighted sum based on the second similarities.

4. The method according to claim 1, wherein The preset text features are obtained by extracting features of preset documents in the preset document library based on a trained text feature model; And / or, extracting features from the document to be checked to generate text features to be checked, including: extracting features from the document to be checked based on a trained text feature model to obtain the text features to be checked; The trained text feature model is obtained by training a language processing model based on training samples, including: Acquire the training sample, wherein the training sample includes project content and corresponding assessment indicators; Performing data enhancement processing on the training samples to obtain a first enhanced sample and a second enhanced sample; The first enhanced sample and the second enhanced sample are determined as positive samples, and other training samples are determined as negative samples, and the language processing model is trained to obtain the trained text feature model.

5. The method according to claim 4, wherein the first enhanced sample and the second enhanced sample are determined as positive samples, and other training samples are determined as negative samples, and the language processing model is trained to obtain the trained text feature model, comprising: The parameters of the language processing model are adjusted so that the distance between the first enhanced sample and the second enhanced sample in the vector space is smaller, and the distance between the first enhanced sample and the second enhanced sample and the other training samples in the vector space is larger, so as to obtain the trained text feature model.

6. The method according to claim 4, wherein: Performing data enhancement processing on the training sample to obtain a first enhanced sample and a second enhanced sample includes: Synonym replacement and / or noise addition are performed on the training samples to obtain different first enhanced samples and second enhanced samples.

7. The method according to claim 1, wherein: The display of the duplicate checking results includes: Displaying the duplicate checking results in the form of a result list, wherein the result list includes the duplicate checking results sorted from high to low based on the degree of duplication; Alternatively, the duplicate checking results are displayed based on different dimensions.

8. A document duplication checking device based on language processing, comprising: The acquisition module is used to obtain the document to be searched; A feature module is used to extract features of the document to be checked and generate features of the text to be checked; A comparison module, used for comparing the to-be-checked text features with a target feature vector to obtain a duplicate checking result; wherein the target feature vector includes a preset text feature corresponding to a preset document library, or a specified text feature corresponding to a specified document; A display module is used to display the duplicate checking results.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images