Duplicate checking method and device, electronic equipment and storage medium

Abstract

The invention provides a duplicate checking method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and data processing, in particular to the fields of natural language processing, big data, intelligent search, deep learning and the like. According to the specific implementation scheme, similarity matching is conducted on a first text feature of information to be subjected to duplicate checking and a second text feature of existing information in a duplicate checking database, and a first similarity value is obtained; in response to determining that the first similarity value is greater than or equal to a preset similarity threshold value, performing similarity matching on the first emotional feature of the to-be-duplicated information and a second emotional feature of the existing information to obtain a second similarity matching result; and according to the second similarity matching result, determining a duplicate checking result of the information to be subjected to duplicate checking relative to the existing information.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and data processing technology, and in particular to the fields of natural language processing, big data, intelligent search, deep learning, and more specifically to a duplicate checking method, device, electronic device, and storage medium. Background Art

[0002] With the development of computer and network technology, people are getting more and more convenient to obtain content. In some fields, there are higher requirements for the originality of content. Therefore, "checking for duplicate content" has emerged as an effective tool. Summary of the invention

[0003] The present invention provides a duplicate checking method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, a duplicate checking method is provided, including: performing similarity matching on a first text feature of information to be checked for duplicates and a second text feature of existing information in a duplicate checking database to obtain a first similarity value; in response to determining that the first similarity value is greater than or equal to a preset similarity threshold, performing similarity matching on a first emotional feature of the information to be checked for duplicates and a second emotional feature of the existing information to obtain a second similarity matching result; and determining, based on the second similarity matching result, a duplicate checking result of the information to be checked for duplicates relative to the existing information.

[0005] According to another aspect of the present disclosure, a duplicate checking device is provided, including: a text similarity matching module, used to perform similarity matching on a first text feature of information to be checked for duplicates and a second text feature of existing information in a duplicate checking database to obtain a first similarity value; an emotional similarity matching module, used to perform similarity matching on a first emotional feature of the information to be checked for duplicates and a second emotional feature of the existing information in response to determining that the first similarity value is greater than or equal to a preset similarity threshold to obtain a second similarity matching result; and a duplicate checking result determination module, used to determine, based on the second similarity matching result, a duplicate checking result of the information to be checked for duplicates relative to the existing information.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the duplicate checking method of the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the duplicate checking method of the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program stored on at least one of a readable storage medium and an electronic device, and the computer program implements the duplicate checking method of the present disclosure when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 An exemplary system architecture to which the duplicate checking method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0012] Figure 2 A flowchart of a duplicate checking method according to an embodiment of the present disclosure is schematically shown;

[0013] Figure 3 The following schematically shows an architecture diagram of a multimodal intelligent duplicate checking system according to an embodiment of the present disclosure;

[0014] Figure 4 The overall flow chart of the duplicate checking method according to the embodiment of the present disclosure is schematically shown;

[0015] Figure 5 A block diagram of a duplicate checking device according to an embodiment of the present disclosure is schematically shown; and

[0016] Figure 6 A schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0019] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0020] Related duplicate checking technologies are mainly based on string matching, keyword comparison and simple fuzzy matching. Most duplicate checking tools generate similarity reports by comparing the text submitted by the user with the documents or web pages stored in the database word by word. These tools usually rely on fixed databases, such as academic paper libraries, news media libraries, etc., and the content they process is mostly in plain text format.

[0021] In the process of implementing the concept disclosed herein, the inventors discovered that the duplicate checking results of existing tools are not complete enough, and it is difficult for users to understand the duplicate checking results and analyze the duplicate checking situation.

[0022] Figure 1 An exemplary system architecture to which the duplicate checking method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0023] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the duplicate checking method and apparatus can be applied may include a terminal device, but the terminal device may implement the duplicate checking method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0024] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0025] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0026] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0027] The server 105 may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0028] It should be noted that the duplicate checking method provided in the embodiment of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103. Correspondingly, the duplicate checking device provided in the embodiment of the present disclosure can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0029] Alternatively, the duplicate checking method provided in the embodiment of the present disclosure may also be generally performed by the server 105. Accordingly, the duplicate checking device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The duplicate checking method provided in the embodiment of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the duplicate checking device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0030] For example, when performing information duplication check, the first terminal device 101, the second terminal device 102, and the third terminal device 103 can obtain the information to be checked for duplication, and then send the obtained information to be checked for duplication to the server 105, and the server 105 performs similarity matching on the first text feature of the information to be checked for duplication and the second text feature of the existing information in the duplication check database to obtain a first similarity value; in response to determining that the first similarity value is greater than or equal to a preset similarity threshold, the first emotional feature of the information to be checked for duplication and the second emotional feature of the existing information are similarly matched to obtain a second similarity matching result; and according to the second similarity matching result, the duplication checking result of the information to be checked for duplication relative to the existing information is determined. Alternatively, a server or server cluster that can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105 analyzes the information to be checked for duplication, and determines the duplication checking result of the information to be checked for duplication relative to the existing information.

[0031] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0032] Figure 2 The flowchart of the duplicate checking method according to an embodiment of the present disclosure is schematically shown.

[0033] like Figure 2 As shown, the method includes operations S210 to S230.

[0034] In operation S210, a first text feature of the information to be checked for duplicates is matched with a second text feature of the existing information in the duplicate checking database to obtain a first similarity value.

[0035] According to an embodiment of the present disclosure, both the information to be checked for duplicates and the existing information may include text information. By performing text feature extraction on each text information, a first text feature and a second text feature may be obtained. The second text feature may be pre-stored in a third-party database that is connected to the duplicate checking database, and has an associated relationship with the original text information for subsequent similarity matching. By extracting text features, the information to be checked for duplicates and the existing information may be literally matched for similarity.

[0036] According to an embodiment of the present disclosure, the preprocessed text can be efficiently compared with the content in the multi-source database based on parallel computing and an optimized fuzzy matching algorithm to obtain a first similarity value.

[0037] In operation S220, in response to determining that the first similarity value is greater than or equal to a preset similarity threshold, a similarity match is performed between the first emotional feature of the information to be checked for duplicates and the second emotional feature of the existing information to obtain a second similarity matching result.

[0038] According to an embodiment of the present disclosure, the emotion feature may represent other features in a text file in addition to the text feature, such as sentiment feature, style feature, tense feature, voice feature, word order feature, etc., and may not be limited thereto.

[0039] According to an embodiment of the present disclosure, when it is determined that the first similarity value is greater than or equal to a preset similarity threshold, it can be determined that the information to be checked for duplicates and the existing information have similarities in text features. For the information to be checked for duplicates and the existing information that have similarities in text features, the similarities between the two in emotional features can be further matched to obtain a second similarity matching result.

[0040] In operation S230, a duplicate checking result of the information to be checked for duplicates relative to the existing information is determined according to the second similarity matching result.

[0041] According to an embodiment of the present disclosure, the second similarity matching result may include similarity matching values ​​corresponding to various types of emotional features. In addition to recording the information to be checked for duplicate content and the existing information with similar text features, the duplicate checking result may also record the second similarity matching result of the information to be checked for duplicate content and the existing information in terms of emotional features, so as to achieve further duplicate checking analysis.

[0042] According to an embodiment of the present disclosure, the above-mentioned duplicate checking database can be updated in real time so that the duplicate checking results can cover the latest content.

[0043] Through the above-mentioned embodiments of the present disclosure, the matching objects with similar text features can be further analyzed in combination with the emotional features, and the emotional features other than the text features in the matching objects can be automatically analyzed to obtain the duplicate checking results containing richer information, which is conducive to the user to analyze the duplicate checking situation more intuitively by understanding the duplicate checking results.

[0044] In conjunction with specific embodiments, Figure 2 The method shown is further explained.

[0045] In the process of implementing the concept of the present disclosure, the inventors also found that related duplicate checking tools can only process pure text, and it is difficult to deal with text content in images or files mixed in multiple formats. It is difficult to check for duplicate content when facing content in multiple formats. Related duplicate checking tools mainly rely on keyword matching or string comparison, which is difficult to accurately identify content that is semantically similar but expressed differently. Traditional duplicate checking tools mainly rely on keywords, resulting in a lot of content being missed.

[0046] According to an embodiment of the present disclosure, the information to be checked for duplicates may include information in multiple formats. The above operation S210 may include: performing text conversion on the information in multiple formats to obtain the text information to be checked for duplicates. Performing semantic feature extraction on the text information to be checked for duplicates to obtain a semantic vector of the text to be checked for duplicates. Performing similarity matching between the semantic vector of the text to be checked for duplicates and the existing text semantic vector of the existing information to obtain a first similarity value.

[0047] According to an embodiment of the present disclosure, the information to be checked for duplicates may include information in a variety of text formats, such as information in PDF, Word, text, image and other formats, and may not be limited thereto. Text feature extraction may include semantic feature extraction. In the case where the information to be checked for duplicates is a text file, semantic feature extraction may be performed on the text file to obtain a semantic vector of the text to be checked for duplicates. In the case where the information to be checked for duplicates is a non-text file, texts in different formats may first be uniformly converted into a standard format that the system can handle, such as a plain text format, and then semantic feature extraction may be performed. For example, if the information to be checked for duplicates is a PDF or an image, OCR (optical character recognition) technology may be used to extract the text content therein, and then semantic feature extraction may be performed to obtain a semantic vector of the text to be checked for duplicates.

[0048] According to the embodiments of the present disclosure, before semantic feature extraction, the text file may also be preprocessed, including text extraction, grammatical analysis, sentence segmentation, keyword extraction, etc., and may not be limited thereto. For example, the text may be first segmented into several sentences or paragraphs, and then semantic feature extraction may be performed on the sentences or paragraphs. For unstructured text, natural language processing may be performed to extract key semantic information.

[0049] According to an embodiment of the present disclosure, the duplicate checking database may be a multi-source database, and the existing information may also include information in multiple text formats. For the existing information in the duplicate checking database, the above semantic feature extraction method may be performed to obtain the existing text semantic vector.

[0050] According to the above-mentioned embodiments of the present disclosure, a multimodal intelligent duplicate checking system is provided, which can combine natural language processing (NLP), deep learning, parallel computing, and multimodal data analysis to achieve comprehensive and efficient duplicate checking of multi-format and multi-source texts.

[0051] Figure 3 The schematic diagram shows an architecture diagram of a multimodal intelligent duplicate checking system according to an embodiment of the present disclosure.

[0052] like Figure 3 As shown, the duplicate checking system 300 includes a text input module 310 , a preprocessing module 320 , a multimodal analysis module 330 , and a duplicate checking and comparison module 340 .

[0053] According to an embodiment of the present disclosure, after the user uploads text through the front-end interface, the text input module 310 performs operations S311 to S312.

[0054] In operation S311, texts in multiple formats are received.

[0055] In operation S312, the format is identified and converted to a standard format. This includes: for text files, the system directly extracts the text content. For PDF and Word text, the embedded text is parsed. For image files, OCR technology is used to convert the text in the image into recognizable text.

[0056] According to an embodiment of the present disclosure, the preprocessing module 320 may perform operations S321 - S322 .

[0057] In operation S321, a standardized text is received.

[0058] In operation S322, text extraction, grammatical analysis, sentence segmentation, keyword extraction, OCR text extraction, and NLP processing of unstructured text are performed.

[0059] According to an embodiment of the present disclosure, the preprocessed text is transmitted to the multimodal analysis module 330, the NLP 331 and the deep learning model 332 are integrated, and operation S333 is performed.

[0060] In operation S333, deep semantic analysis is performed to generate a semantic vector of the text to be checked for duplicates.

[0061] According to an embodiment of the present disclosure, the duplicate checking and comparison module 340 calls the multi-threaded parallel computing technology 341 and the fuzzy matching algorithm 442 to perform operations S343~S344.

[0062] In operation S343, the semantic vector of the text to be checked for duplicates is quickly matched with the semantic vectors of existing texts in the multi-source database.

[0063] According to an embodiment of the present disclosure, the duplicate checking system 300 can connect to multiple external databases, and obtain and update the latest document and web page data in real time through an API (Application Programming Interface), dynamically update the database according to user needs, ensure that the duplicate checking scope covers the most extensive and latest information, and improve the timeliness of the duplicate checking and the reliability of the duplicate checking results.

[0064] In operation S344, the contents with higher similarity are recorded and the repeated parts are marked.

[0065] Through the optimized fuzzy matching algorithm, not only can the literally similar content be identified, but also the rewritten content can be detected, thereby realizing the identification of semantically similar content under different expressions.

[0066] According to an embodiment of the present disclosure, the duplicate checking system 300 may also include a visual report generation module. The comparison results obtained based on the above-mentioned duplicate checking system 300 may also be passed to the visual report generation module to generate a detailed duplicate checking report, including displaying similarity scores, color-coding paragraphs with high similarity, and attaching a link to the comparison source. The user can view the report through the visual interface and click on the link to view the specific comparison content, so that the duplicate content can be quickly located and analyzed according to the report.

[0067] According to an embodiment of the present disclosure, the duplicate checking system 300 may further include a privacy protection module. The privacy protection module may use data encryption technology to protect the security of the user's text content during transmission and storage, and ensure that the text content will not be leaked or abused. The privacy protection module is responsible for the transmission and storage of text data throughout the entire process to ensure data security. For example, the text content uploaded by the user may be encrypted using AES (Advanced Encryption Standard), and the SSL (Secure Sockets Layer) / TLS (Transport Layer Security) encryption protocol may be used during data transmission to prevent data from being stolen during transmission, and after the duplicate checking is completed, the stored plaintext data may be automatically deleted immediately to ensure privacy and security.

[0068] Through the above-mentioned embodiments of the present disclosure, it is possible to process information in various formats and realize application in multimodal scenarios. In addition, by performing similarity matching between the semantic vectors of the text to be checked for duplicate content and the semantic vectors of the existing text of the existing information, semantic matching can be achieved on the basis of literal matching, thereby improving the accuracy of the duplicate checking results.

[0069] According to an embodiment of the present disclosure, the text information to be checked for duplicates may include information in multiple languages, and the semantic vector of the existing text may be obtained by extracting semantic features from the target language form of the existing information. The above-mentioned extraction of semantic features from the text information to be checked for duplicates to obtain the semantic vector of the text to be checked for duplicates includes: converting the information in multiple languages ​​into target language information in the target language form. Extracting semantic features from the target language information to obtain the semantic vector of the text to be checked for duplicates.

[0070] According to an embodiment of the present disclosure, after obtaining text content in a non-target language, the duplicate checking database can first convert the text content into text content in a target language, then perform preprocessing, and then perform semantic feature extraction on the preprocessed text content in the target language to obtain a semantic vector of the existing text.

[0071] According to an embodiment of the present disclosure, when the information to be checked for duplicate content is obtained in a non-target language, the information to be checked for duplicate content may be sequentially subjected to language conversion, preprocessing, and semantic feature extraction to obtain a semantic vector of the text to be checked for duplicate content.

[0072] According to an embodiment of the present disclosure, corresponding to the above-mentioned operation S220, in response to determining that the first similarity value is greater than or equal to a preset similarity threshold, it may also include: in response to determining that the language form of the information to be checked for duplicates is different from the language form of the existing information, determining that the duplicate checking result of the information to be checked for duplicates relative to the existing information includes language change type overlap.

[0073] According to an embodiment of the present disclosure, based on the above method, language detection, semantic vectorization, cross-language semantic matching and other methods can be combined to achieve cross-language duplication checking.

[0074] In the language detection stage, the language detection model can be used to automatically detect the language type of the text uploaded by the user. This includes: segmenting the uploaded text. Using the language detection model to analyze each segment to determine the main language of the text or a combination of multiple languages. The language detection model can use, for example, Compact Language Detector (language identification), Langid (a language identification service), etc., but is not limited to these.

[0075] In the semantic vectorization stage, a multilingual pre-trained language model can be used to semantically vectorize the text. The model can map the semantic representations of different languages ​​into a shared semantic space. Including: inputting the detected texts in each language into the multilingual model. Extracting the semantic vector of each sentence in the shared semantic space and storing it. Pre-trained language models can use mBERT (Multilingual BERT), XLM-R (Large-Scale Unsupervised Cross-Lingual Representation Model), etc., but are not limited to these.

[0076] In the cross-language semantic matching stage, similarity matching can be performed in the shared semantic space, and the similarity of the cross-language text can be determined by calculating the cosine similarity of the two semantic vectors. This includes: extracting the semantic vectors of existing texts in the target language from the duplicate checking database. Calculating the semantic similarity between the semantic vectors of the text to be checked for duplicates and the semantic vectors of the existing texts in the duplicate checking database, and identifying highly similar content. Performing language comparison on highly similar content to confirm whether there is any translation overlap.

[0077] According to an embodiment of the present disclosure, after obtaining a duplicate check result indicating whether there is translation overlap, a report can be generated to display the results of cross-language matching. This includes: displaying the duplicate check result including language change overlap in a table or graphic form. Marking possible translation overlaps, including similarity scores and potential overlap paragraphs, and attaching a comparison of the original text and the translated content.

[0078] Through the above-mentioned embodiments of the present disclosure, cross-language duplication checking can be implemented, and duplication checking of multi-language texts can be supported. By automatically detecting the language type of the text and performing cross-language semantic comparison, it is helpful to identify overlapping content after translation.

[0079] According to an embodiment of the present disclosure, the above operation S220 may include: performing sentiment classification on the information to be checked for duplicates to obtain a sentiment tag of the information to be checked for duplicates, wherein the sentiment tag has a sentiment evaluation value. Performing similarity matching on the sentiment tag of the information to be checked for duplicates and the sentiment tag of the existing information to obtain a second similarity value. The above operation S230 may include: in response to determining that the second similarity value is less than a preset sentiment threshold, determining that the result of checking for duplicates of the information to be checked for duplicates relative to the existing information includes a sentiment change type overlap.

[0080] According to the embodiments of the present disclosure, the sentiment tags of the information to be checked for duplicates and the sentiment tags of the existing information can be similarly calculated to obtain a second similarity value. The sentiment evaluation value can also be used as the weight assigned to the sentiment tag, and the sentiment tags of the information to be checked for duplicates that have been weighted and the sentiment tags of the existing information that have been weighted can be similarly calculated to obtain a second similarity value. This is not limited here.

[0081] According to an embodiment of the present disclosure, based on the above method, sentiment analysis can be implemented by combining sentiment analysis models, sentiment similarity matching and other methods.

[0082] According to an embodiment of the present disclosure, a sentiment analysis model can be used to classify the sentiment tendency of a text. Including: segmenting the text. Using a pre-trained sentiment analysis model to classify the sentiment of each paragraph, such as positive, neutral, negative, etc. Assign a sentiment score to each paragraph and store it. The sentiment analysis model can use BERT-based Sentiment Analysis (bert-based sentiment analysis model), etc., but is not limited to this.

[0083] In the sentiment similarity matching stage, for example, by comparing sentiment scores, it is possible to detect whether the information to be checked for duplicate content is similar or changed with the existing information in the duplicate checking database. This includes: extracting sentiment features of the text to be checked for duplicate content. Searching for text paragraphs with similar sentiments in the duplicate checking database. Identifying content with similar semantic features but obvious sentiment changes, and marking them as potential sentiment change type overlaps.

[0084] According to an embodiment of the present disclosure, the above operation S220 may include: performing style classification on the information to be checked for duplicates to obtain a style classification result of the information to be checked for duplicates. Performing feature extraction on the style classification result to obtain a style feature vector. Performing similarity matching on the style feature vector of the information to be checked for duplicates and the style feature vector of the existing information to obtain a third similarity value. The above operation S230 may include: in response to determining that the third similarity value is less than a preset style threshold, determining that the duplicate checking result of the information to be checked for duplicates relative to the existing information includes a style change type overlap.

[0085] According to an embodiment of the present disclosure, based on the above method, writing style analysis, style similarity matching and other methods can be combined to achieve style analysis.

[0086] In the writing style analysis stage, the NLP model can be used to analyze the writing style of the text and extract style features such as grammar, sentence structure, and vocabulary usage. Including: Using text analysis tools to extract writing style features. Using style classification models to classify text styles, such as formal, informal, and scientific writing. Encode the style features of each paragraph and store them. NLP models can use GPT (Generative pre-trained transformers, a generative pre-trained model based on transformers) style analysis models, etc., but are not limited to them. Text analysis tools can use N-gram (a phrase sequence prediction and evaluation model in natural language processing) analysis, TF-IDF (term frequency–inverse document frequency), etc., but are not limited to them.

[0087] In the style similarity matching stage, for example, by comparing style codes, it is possible to detect whether the information to be checked for duplicate content is similar or changed with the existing information in the duplicate checking database. This includes: extracting the style features of the text to be checked for duplicate content. Searching for text paragraphs with similar styles in the duplicate checking database. Identifying content with similar semantic features but obvious style changes, and marking them as potential sentiment change overlaps.

[0088] According to an embodiment of the present disclosure, when the sentiment or style duplication check result is obtained, a sentiment or style analysis report can be generated to display the detected sentiment or style changes. This includes: visualizing the sentiment or style analysis result and highlighting the changed part. Generating a report with comparison information of similar content.

[0089] Through the above-mentioned embodiments of the present disclosure, by analyzing the sentiment or writing style of the text, it is possible to detect situations in which attempts are made to change the sentiment or style of the original text during the rewriting process, so that users can further analyze the duplicate checking results in combination with the sentiment or style dimensions.

[0090] According to an embodiment of the present disclosure, the above operation S220 may include: performing grammatical analysis on the information to be checked for duplicates to obtain the tense tag and voice tag of the information to be checked for duplicates. Performing consistency comparison on the tense tag of the information to be checked for duplicates and the tense tag of the existing information to obtain a first comparison result. Performing similarity matching on the voice tag of the information to be checked for duplicates and the voice tag of the existing information to obtain a second comparison result. The above operation S230 may include: in response to determining that at least one of the first comparison result and the second comparison result represents inconsistency, determining that the duplication result of the information to be checked for duplicates relative to the existing information includes grammatical change type overlap.

[0091] According to the embodiments of the present disclosure, based on the above method, grammatical analysis such as tense and voice can be implemented by combining grammatical analysis and parsing, tense and voice conversion recognition, and rewriting detection models.

[0092] In the grammatical analysis and parsing stage, you can use syntactic analysis tools to perform grammatical analysis on sentences and identify the tense and voice of sentences. This includes: tagging each sentence with parts of speech and building a syntactic tree. Extract the tense of verbs, such as past tense, present tense, etc., and voice, such as active voice, passive voice, etc., from the syntactic tree. Syntactic analysis tools can be used, such as SpaCy (an open source natural language processing library), NLP Parser (NLP parser), etc., but are not limited to these.

[0093] In the tense and voice conversion identification stage, the parsed tense and voice information can be compared with the existing information in the duplicate checking database using a text similarity model to detect whether there is a rewriting behavior of tense or voice conversion. Including: comparing the tense and voice features of the information to be checked for duplicates and the existing information. Comparing the semantic similarity of sentences before and after tense or voice conversion. Identifying content with similar text features but inconsistent tense or voice, marking it as a potential grammatical change overlap. The text similarity model can use, for example, edit distance or a similarity model based on BERT (Bidirectional Encoder Representation from Transformers), but is not limited to this.

[0094] According to an embodiment of the present disclosure, when a grammar duplicate check result is obtained, a grammar analysis report can be generated to display the tense and voice conversion parts detected, and to indicate possible rewriting overlaps. This includes: displaying the duplicate check results including grammar change type overlaps in a table, marking the sentences with tense and voice conversions, and providing comparison information of the original text and the rewritten content.

[0095] Through the above-mentioned embodiments of the present disclosure, by identifying the conversion of tense and voice in the text, the situation of masking the overlap by rewriting the tense or voice can be detected.

[0096] According to an embodiment of the present disclosure, in response to determining that the first similarity value is greater than or equal to a preset similarity threshold, it may also include: obtaining existing similar text information from existing information whose first similarity value with the text information to be checked for duplicates is greater than or equal to the preset similarity threshold. Determine first text position information of the first text in the text information to be checked for duplicates in the text information to be checked for duplicates. Determine second text position information of the second text in the existing similar text information in the existing similar text information, the second text representing text in the existing similar text information whose first similarity value with the first text is greater than or equal to the preset similarity threshold. In response to determining that the first text position information and the second text position information are different, determine that the duplicate checking result of the information to be checked for duplicates relative to the existing information includes context order change type overlap.

[0097] According to an embodiment of the present disclosure, based on the above method, context modeling, context similarity matching, rewriting and reorganization identification and other methods can be combined to implement context-based duplication checking.

[0098] In the context modeling stage, context-aware NLP models (such as Transformer models and GPT series models) can be used to model the entire text, taking into account the context information of each sentence or paragraph. This includes: dividing the text into sentence or paragraph units and inputting them into the context-aware model. The model encodes each unit together with its context to generate a context-aware semantic representation.

[0099] In the context similarity matching stage, the context-aware semantic representation can be globally compared with the existing information in the duplicate checking database to detect parts with similar semantics but changed contexts. This includes: calculating the context similarity between the text information to be checked for duplicates and the existing similar text information in the duplicate checking database. Identifying similar combinations of the first text in the text information to be checked for duplicates and the second text in the existing similar text information, and marking possible overlap behaviors.

[0100] In the rewriting and reorganization identification stage, the paragraph or sentence reorganization behavior in the text can be detected to determine whether these reorganizations attempt to cover up overlapping content. This includes: analyzing the order of paragraphs and sentences in the text information to be checked for duplicate content, and comparing them with the order of paragraphs and sentences in similar text information already in the duplicate checking database. Identifying parts where the position of paragraphs or sentences has changed but the text features are similar.

[0101] According to an embodiment of the present disclosure, when a part of a paragraph or sentence with a changed position but similar text features is obtained, a context-aware duplicate checking report can be generated to display the detected context similarity and rewritten parts. This includes: visually displaying the context similarity results, displaying the parts of content reorganization or context changes. Outputting a report with detailed context comparison information.

[0102] Through the above-mentioned embodiments of the present disclosure, by combining the contextual information of the text for duplication checking, it is possible to identify overlapping content that may be ignored when viewing separately, thereby further improving the duplication checking effect.

[0103] Figure 4 The overall flow chart of the duplicate checking method according to an embodiment of the present disclosure is schematically shown.

[0104] like Figure 4 As shown, the method includes operations S401 to S419.

[0105] In operation S401, a multi-format text is acquired.

[0106] In operation S402, the multi-format text is converted into a standardized text.

[0107] In operation S403, it is determined whether the text is in the target language. If so, operation S405 is executed; if not, operations S404 to S405 are executed.

[0108] In operation S404, the standardized text is converted into a standardized text in a target language.

[0109] In operation S405, the standardized text is preprocessed to obtain segmented text.

[0110] In operation S406, based on the context information of the segmented text, similarity matching is performed between the segmented text and existing text in the duplicate checking database to obtain a first similarity value.

[0111] In operation S407, it is determined whether the first similarity value is greater than or equal to a preset similarity threshold. If yes, operations S408 to S417 are performed; if no, operation S419 is performed.

[0112] In operation S408, it is determined whether the original language form of the segmented text is consistent with the original language form of the existing text. If so, operation S418 is executed; if not, operation S409 is executed.

[0113] In operation S409, a language change type overlap is output.

[0114] In operation S410, it is determined whether the sentiment feature of the segmented text is consistent with the sentiment feature of the existing text. If so, operation S418 is executed; if not, operation S411 is executed.

[0115] In operation S411, the emotion change type overlap is output.

[0116] In operation S412, it is determined whether the style feature of the segmented text is consistent with the style feature of the existing text. If so, operation S418 is executed; if not, operation S413 is executed.

[0117] In operation S413, the style-changed type overlap is output.

[0118] In operation S414, it is determined whether the grammatical features of the segmented text are consistent with the grammatical features of the existing text. If so, operation S418 is executed; if not, operation S415 is executed.

[0119] In operation S415, the grammar-changed overlap is output.

[0120] In operation S416, it is determined whether the text order of the segmented text is consistent with the text order of the existing text. If so, operation S418 is executed; if not, operation S417 is executed.

[0121] In operation S417, the context order change type overlap is output.

[0122] In operation S418, a normal coincidence is output.

[0123] In operation S419, no overlap is output.

[0124] According to the embodiments of the present disclosure, the above-mentioned duplicate checking method can be widely used in the fields of academic papers, patent documents, media manuscripts, contract documents, etc., helping users to effectively prevent overlaps, improve originality, and ensure the originality and legality of documents. In addition, the multimodal analysis function can process multiple data types such as text, images, and audio, making its application scope even wider.

[0125] Through the above-mentioned embodiments of the present disclosure, semantic similarity, partial overlap and rewritten content can be accurately identified, and the comprehensiveness, accuracy and efficiency of duplicate checking can be improved, which fully solves the shortcomings of existing duplicate checking technologies in multi-format document processing, semantic similarity identification, real-time, processing efficiency and privacy protection, and has significant innovation and broad application prospects. In addition, it also has functions such as real-time update, privacy protection and visual report generation to meet the diverse needs of users and solve the shortcomings of existing duplicate checking tools in processing multi-format documents, complex semantics, real-time and privacy protection. It can not only improve the accuracy and efficiency of duplicate checking, but also provide users with a safer and more intuitive duplicate checking experience. It can effectively improve the ability to check for duplicates and accurately identify complex overlaps and rewriting behaviors.

[0126] Figure 5 A block diagram of a duplicate checking device according to an embodiment of the present disclosure is schematically shown.

[0127] like Figure 5 As shown, the duplicate checking device 500 includes a text similarity matching module 510 , a sentiment similarity matching module 520 and a duplicate checking result determination module 530 .

[0128] The text similarity matching module 510 is used to perform similarity matching on the first text feature of the information to be checked for duplicates and the second text feature of the existing information in the duplicate checking database to obtain a first similarity value.

[0129] The emotion similarity matching module 520 is used to perform similarity matching on the first emotion feature of the information to be checked for duplicates and the second emotion feature of the existing information in response to determining that the first similarity value is greater than or equal to a preset similarity threshold, to obtain a second similarity matching result.

[0130] The duplicate checking result determination module 530 is used to determine the duplicate checking result of the information to be checked for duplicates relative to the existing information according to the second similarity matching result.

[0131] According to an embodiment of the present disclosure, the duplicate checking apparatus further includes a language-changing overlap determination module.

[0132] The language change type overlap determination module is used to determine that the language form of the information to be checked for duplicates is different from the language form of the existing information, and determine that the duplicate checking result of the information to be checked for duplicates relative to the existing information includes language change type overlap.

[0133] According to an embodiment of the present disclosure, the emotion similarity matching module includes an emotion classification unit and an emotion matching unit. The duplicate checking result determination module includes an emotion change type overlap determination unit.

[0134] The sentiment classification unit is used to perform sentiment classification on the information to be checked for duplicates, and obtain a sentiment label of the information to be checked for duplicates, wherein the sentiment label has a sentiment evaluation value.

[0135] The sentiment matching unit is used to perform similarity matching on the sentiment tags of the information to be checked for duplicates and the sentiment tags of the existing information to obtain a second similarity value.

[0136] The emotion change type overlap determination unit is used to determine that the duplicate checking result of the to-be-checked duplicate information relative to the existing information includes emotion change type overlap in response to determining that the second similarity value is less than a preset emotion threshold.

[0137] According to an embodiment of the present disclosure, the emotion similarity matching module includes a style classification unit, a style feature extraction unit and a style matching unit. The duplicate checking result determination module includes a style change type overlap determination unit.

[0138] The style classification unit is used to classify the style of the information to be checked for duplicates and obtain the style classification result of the information to be checked for duplicates.

[0139] The style feature extraction unit is used to extract features from the style classification results to obtain a style feature vector.

[0140] The style matching unit is used to perform similarity matching on the style feature vector of the information to be checked for duplicates and the style feature vector of the existing information to obtain a third similarity value.

[0141] The style change type overlap determination unit is used to determine that the duplicate checking result of the to-be-checked duplicate information relative to the existing information includes style change type overlap in response to determining that the third similarity value is less than a preset style threshold.

[0142] According to an embodiment of the present disclosure, the emotion similarity matching module includes a grammar analysis unit, a tense matching unit and a voice matching unit. The duplicate checking result determination module includes a grammar change type overlap determination unit.

[0143] The grammatical analysis unit is used to perform grammatical analysis on the information to be checked for duplicates, and obtain the tense label and voice label of the information to be checked for duplicates.

[0144] The temporal matching unit is used to compare the temporal labels of the information to be checked for duplicates with the temporal labels of the existing information to obtain a first comparison result.

[0145] The voice matching unit is used to perform similarity matching on the voice tags of the information to be checked for duplicates and the voice tags of the existing information to obtain a second comparison result.

[0146] The grammatical change type overlap determination unit is used to determine that the duplicate checking result of the to-be-checked duplicate information relative to the existing information includes grammatical change type overlap in response to determining that at least one of the first comparison result and the second comparison result represents inconsistency.

[0147] According to an embodiment of the present disclosure, the information to be checked for duplicate content includes information in multiple formats. The text similarity matching module includes a text conversion unit, a semantic feature extraction unit and a semantic matching unit.

[0148] The text conversion unit is used to perform text conversion on information in various formats to obtain text information to be checked for duplicates.

[0149] The semantic feature extraction unit is used to extract semantic features from the text information to be checked for duplicates, and obtain a semantic vector of the text to be checked for duplicates.

[0150] The semantic matching unit is used to perform similarity matching between the semantic vector of the text to be checked for duplicate content and the semantic vector of the existing text in the existing information to obtain a first similarity value.

[0151] According to an embodiment of the present disclosure, the text information to be checked for duplicates includes information in multiple languages, and the existing text semantic vector is obtained by extracting semantic features from the target language form of the existing information. The semantic feature extraction unit includes a language conversion subunit and a semantic feature extraction subunit.

[0152] The language conversion subunit is used to convert information in multiple languages ​​into target language information in the target language form.

[0153] The semantic feature extraction subunit is used to extract semantic features of the target language information to obtain the semantic vector of the text to be checked for duplicates.

[0154] According to an embodiment of the present disclosure, the duplicate checking device also includes an existing similar text information acquisition module, a first text position information determination module, a second text position information determination module and a context order change overlap determination module.

[0155] The existing similar text information acquisition module is used to acquire, from the existing information, existing similar text information whose first similarity value with the text information to be checked for duplicates is greater than or equal to a preset similarity threshold.

[0156] The first text position information determining module is used to determine the first text position information of the first text in the text information to be checked for duplicates.

[0157] The second text position information determination module is used to determine the second text position information of the second text in the existing similar text information, where the second text represents the text in the existing similar text information whose first similarity value with the first text is greater than or equal to a preset similarity threshold.

[0158] The context order change type overlap determination module is used to determine that the duplicate checking result of the information to be checked for duplicates relative to the existing information includes context order change type overlap in response to determining that the first text position information and the second text position information are different.

[0159] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0160] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the duplicate checking method of the present disclosure.

[0161] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the duplicate checking method of the present disclosure.

[0162] According to an embodiment of the present disclosure, a computer program product includes a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and the computer program implements the duplicate checking method of the present disclosure when executed by a processor.

[0163] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0164] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0165] A number of components in the device 600 are connected to an input / output (I / O) interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0166] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a duplicate checking method. For example, in some embodiments, the duplicate checking method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the duplicate checking method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the duplicate checking method in any other appropriate manner (e.g., by means of firmware).

[0167] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0169] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0171] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0172] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0173] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0174] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for checking for duplicate content, comprising: Performing similarity matching on the first text feature of the information to be checked for duplicate content and the second text feature of the existing information in the duplicate checking database to obtain a first similarity value; In response to determining that the first similarity value is greater than or equal to a preset similarity threshold, performing similarity matching on the first emotional feature of the information to be checked for duplicates and the second emotional feature of the existing information to obtain a second similarity matching result; as well as According to the second similarity matching result, a duplicate checking result of the information to be checked for duplicates relative to the existing information is determined.

2. The method according to claim 1, wherein: In response to determining that the first similarity value is greater than or equal to a preset similarity threshold, the method further includes: In response to determining that the language form of the information to be checked for duplicates is different from the language form of the existing information, it is determined that the duplicate checking result of the information to be checked for duplicates relative to the existing information includes language change type overlap.

3. The method according to claim 1 or 2, wherein: The performing similarity matching on the first emotional feature of the information to be checked for duplicates and the second emotional feature of the existing information to obtain a second similarity matching result includes: Performing sentiment classification on the information to be checked for duplicates to obtain a sentiment label of the information to be checked for duplicates, wherein the sentiment label has a sentiment evaluation value; and Performing similarity matching on the sentiment tag of the information to be checked for duplicates and the sentiment tag of the existing information to obtain a second similarity value; Wherein, determining the duplicate checking result of the information to be checked for duplicates relative to the existing information according to the second similarity matching result includes: In response to determining that the second similarity value is less than a preset emotion threshold, it is determined that the result of checking the information for duplication to be checked for duplication relative to the existing information includes a coincidence of emotion change type.

4. The method according to any one of claims 1 to 3, wherein: The performing similarity matching on the first emotional feature of the information to be checked for duplicates and the second emotional feature of the existing information to obtain a second similarity matching result includes: Performing style classification on the information to be checked for duplicates, and obtaining a style classification result of the information to be checked for duplicates; Performing feature extraction on the style classification results to obtain a style feature vector; and Performing similarity matching on the style feature vector of the information to be checked for duplicates and the style feature vector of the existing information to obtain a third similarity value; Wherein, determining the duplicate checking result of the information to be checked for duplicates relative to the existing information according to the second similarity matching result includes: In response to determining that the third similarity value is less than a preset style threshold, it is determined that the result of checking the information to be checked for duplicates relative to the existing information includes a style change type overlap.

5. The method according to any one of claims 1 to 4, wherein: The performing similarity matching on the first emotional feature of the information to be checked for duplicates and the second emotional feature of the existing information to obtain a second similarity matching result includes: Performing grammatical analysis on the information to be checked for duplicates to obtain a tense tag and a voice tag of the information to be checked for duplicates; Comparing the temporal label of the information to be checked for duplicates with the temporal label of the existing information for consistency, to obtain a first comparison result; and Performing similarity matching on the voice tag of the information to be checked for duplicates and the voice tag of the existing information to obtain a second comparison result; Wherein, determining the duplicate checking result of the information to be checked for duplicates relative to the existing information according to the second similarity matching result includes: In response to determining that at least one of the first comparison result and the second comparison result represents inconsistency, it is determined that the result of checking for duplicate information relative to the existing information includes grammatical change type overlap.

6. The method according to any one of claims 1 to 5, wherein: The information to be checked for duplicates includes information in various formats; The similarity matching between the first text feature of the information to be checked for duplicates and the second text feature of the information already in the duplicate checking database to obtain the first similarity value includes: Performing text conversion on the information in the multiple formats to obtain the text information to be checked for duplicates; Extracting semantic features from the text to be checked for duplicates to obtain a semantic vector of the text to be checked for duplicates; and A similarity match is performed between the semantic vector of the text to be checked for duplicates and the semantic vector of the existing text of the existing information to obtain the first similarity value.

7. The method according to claim 6, wherein: The text information to be checked for duplicates includes information in multiple languages, and the existing text semantic vector is obtained by extracting semantic features of information in the target language form of the existing information; extracting semantic features from the text information to be checked for duplicates to obtain the semantic vector of the text to be checked for duplicates includes: converting the information in the multiple languages ​​into target language information in the target language; and Semantic features are extracted from the target language information to obtain a semantic vector of the text to be checked for duplicates.

8. The method according to claim 6 or 7, wherein: In response to determining that the first similarity value is greater than or equal to a preset similarity threshold, the method further includes: Acquire existing similar text information from the existing information, the similarity value of which to the text information to be checked for duplicates is greater than or equal to the preset similarity threshold; Determine first text position information of the first text in the text information to be checked for duplicates in the text information to be checked for duplicates; Determining second text position information of a second text in the existing similar text information, wherein the second text represents text in the existing similar text information whose first similarity value with the first text is greater than or equal to the preset similarity threshold; and In response to determining that the first text position information and the second text position information are different, it is determined that the result of checking the information for duplicates with respect to the existing information includes a context order change type overlap.

9. A duplicate checking device, comprising: A text similarity matching module, used for performing similarity matching between a first text feature of the information to be checked for duplicate content and a second text feature of the existing information in the duplicate checking database, to obtain a first similarity value; an emotion similarity matching module, configured to, in response to determining that the first similarity value is greater than or equal to a preset similarity threshold, perform similarity matching on the first emotion feature of the information to be checked for duplicates and the second emotion feature of the existing information to obtain a second similarity matching result; as well as A duplicate checking result determination module is used to determine the duplicate checking result of the information to be checked for duplicates relative to the existing information based on the second similarity matching result.

10. The apparatus according to claim 9, further comprising: The language change type overlap determination module is used to determine that the duplicate checking result of the information to be checked for duplicates relative to the existing information includes language change type overlap in response to determining that the language form of the information to be checked for duplicates is different from the language form of the existing information.

11. The device according to claim 9 or 10, wherein: The emotion similarity matching module comprises: A sentiment classification unit, used for performing sentiment classification on the information to be checked for duplicates, and obtaining a sentiment label of the information to be checked for duplicates, wherein the sentiment label has a sentiment evaluation value; and The emotion matching unit is used to perform similarity matching on the emotion label of the information to be checked for duplicates and the emotion label of the existing information to obtain a second similarity value; Wherein, the duplicate checking result determination module includes: The emotion change type overlap determination unit is used to determine that the duplicate checking result of the to-be-checked duplicate information relative to the existing information includes emotion change type overlap in response to determining that the second similarity value is less than a preset emotion threshold.

12. The device according to any one of claims 9 to 11, wherein: The emotion similarity matching module comprises: A style classification unit, used for classifying the style of the information to be checked for duplicates, and obtaining a style classification result of the information to be checked for duplicates; A style feature extraction unit, used to extract features from the style classification results to obtain a style feature vector; and A style matching unit, configured to perform similarity matching on the style feature vector of the information to be checked for duplicates and the style feature vector of the existing information to obtain a third similarity value; Wherein, the duplicate checking result determination module includes: The style-changed overlap determining unit is configured to determine, in response to determining that the third similarity value is less than a preset style threshold, that the result of checking the duplicate information for the existing information includes a style-changed overlap.

13. The device according to any one of claims 9 to 12, wherein: The emotion similarity matching module comprises: A grammatical analysis unit, used for performing grammatical analysis on the information to be checked for duplicates, to obtain a tense tag and a voice tag of the information to be checked for duplicates; a temporal matching unit, configured to compare the temporal label of the information to be checked for duplicates with the temporal label of the existing information for consistency, and obtain a first comparison result; and A voice matching unit, used for performing similarity matching between the voice tag of the information to be checked for duplicates and the voice tag of the existing information to obtain a second comparison result; Wherein, the duplicate checking result determination module includes: A grammatical change type overlap determination unit is used to determine that the duplicate checking result of the to-be-checked duplicate information relative to the existing information includes grammatical change type overlap in response to determining that at least one of the first comparison result and the second comparison result represents inconsistency.

14. The device according to any one of claims 9 to 13, wherein: The information to be checked for duplicates includes information in various formats; The text similarity matching module comprises: A text conversion unit, used for performing text conversion on the information in the plurality of formats to obtain the text information to be checked for duplicates; A semantic feature extraction unit, used for extracting semantic features from the text information to be checked for duplicates, to obtain a semantic vector of the text to be checked for duplicates; and The semantic matching unit is used to perform similarity matching on the semantic vector of the text to be checked for duplicates and the semantic vector of the existing text of the existing information to obtain the first similarity value.

15. The device according to claim 14, wherein: The text information to be checked for duplicates includes information in multiple languages, and the existing text semantic vector is obtained by extracting semantic features of information in the target language form of the existing information; The semantic feature extraction unit comprises: A language conversion subunit, configured to convert the information in the multiple languages ​​into target language information in the target language; as well as The semantic feature extraction subunit is used to extract semantic features from the target language information to obtain the semantic vector of the text to be checked for duplicates.

16. The device according to claim 14 or 15, further comprising: An existing similar text information acquisition module, used to acquire, from the existing information, existing similar text information whose first similarity value with the text information to be checked for duplicates is greater than or equal to the preset similarity threshold; A first text position information determination module, used to determine first text position information of the first text in the text information to be checked for duplicates in the text information to be checked for duplicates; A second text position information determination module, used to determine second text position information of a second text in the existing similar text information in the existing similar text information, wherein the second text represents text in the existing similar text information whose first similarity value with the first text is greater than or equal to the preset similarity threshold; as well as A context order change type overlap determination module is used to determine that the duplicate checking result of the information to be checked for duplicates relative to the existing information includes context order change type overlap in response to determining that the first text position information and the second text position information are different.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

19. A computer program product, comprising a computer program, wherein the computer program is stored in at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images