Text duplicate checking method, vehicle, computer readable storage medium and computer program product

Abstract

The invention discloses a text duplicate checking method, a vehicle, a computer readable storage medium and a computer program product, and relates to the technical field of information processing. The method comprises the following steps: carrying out standardized preprocessing on an input text to be subjected to duplicate checking to obtain a current text fragment; performing semantic coding processing on the current text fragment by utilizing the target coding model to obtain a query vector; a similarity retrieval interface corresponding to the target search engine is called, a plurality of candidate vectors corresponding to the query vector are recalled from a semantic vector index, and the semantic vector index is constructed based on a historical text set and a graph index algorithm in the target search engine; sorting analysis is conducted on the multiple candidate vectors based on the multi-level duplicate judgment threshold values, a duplicate checking result is obtained, and the duplicate checking result is used for representing the duplicate situation between the historical texts corresponding to the multiple candidate vectors and the input text. The technical problems of low accuracy and low efficiency of a text duplicate checking method in related technologies are solved.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and more specifically, to a text deduplication method, a vehicle, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In applications that process user feedback text, identifying repetitive feedback text can help developers locate and resolve problems more quickly in order to meet user needs.

[0003] In related technologies, text comparison analysis is commonly used for text plagiarism detection. However, this method is prone to false positives and false negatives when dealing with semantically complex and flexibly expressed text, resulting in low accuracy. Furthermore, this method is computationally complex, leading to low efficiency.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a text plagiarism detection method, a vehicle, a computer-readable storage medium, and a computer program product to at least solve the technical problems of low accuracy and low efficiency in related technologies.

[0006] According to one aspect of the embodiments of this application, a text deduplication method is provided, comprising: performing normalization preprocessing on the input text to be deduplicated to obtain a current text segment; performing semantic encoding processing on the current text segment using a target encoding model to obtain a query vector; calling the similarity retrieval interface corresponding to the target search engine to recall multiple candidate vectors corresponding to the query vector from the semantic vector index, wherein the semantic vector index is constructed based on a historical text set and a graph index algorithm in the target search engine; and performing sorting analysis on the multiple candidate vectors based on a multi-level deduplication threshold to obtain a deduplication result, wherein the deduplication result is used to characterize the duplication situation between the historical text corresponding to the multiple candidate vectors and the input text.

[0007] Optionally, the text deduplication method further includes: performing normalization preprocessing on the historical text set to be indexed to obtain historical text corpus; using the target encoding model and the preset encoding dimension to perform semantic encoding processing on the historical text corpus to obtain a dense semantic vector set; and using the graph indexing algorithm and the dense semantic vector set to construct a semantic vector index, wherein the graph indexing algorithm is used to perform similar nearest neighbor search in the feature space of the preset encoding dimension according to the target graph structure parameters.

[0008] Optionally, the text deduplication method further includes: performing data feature analysis on the historical text set to determine the corpus size features and vector distribution features; and updating the initial graph structure parameters in the graph indexing algorithm to target graph structure parameters based on the corpus size features, vector distribution features, and service optimization objectives. The service optimization objectives include recall rate objectives and query latency objectives, and the target graph structure parameters are used to constrain the upper limit of the number of graph node connections and the upper limit of the search depth during the similar nearest neighbor search process.

[0009] Optionally, the text deduplication method further includes: performing mean pooling on the initial encoding model based on a preset encoding dimension to obtain an intermediate encoding model, wherein the output dimension of the intermediate encoding model is determined by the preset encoding dimension; and fine-tuning the parameters of the intermediate encoding model using a historical text set and a preset text enhancement algorithm to obtain a target encoding model, wherein the preset text enhancement algorithm includes at least one of the following: semantic similarity enhancement algorithm, contrastive learning algorithm, and knowledge distillation algorithm.

[0010] Optionally, the similarity retrieval interface includes a first-level call interface built based on standard cosine similarity and a second-level filtering interface built based on target similarity. Calling the similarity retrieval interface to retrieve multiple candidate vectors corresponding to the query vector from the semantic vector index includes: calling the first-level call interface to query the semantic vector index to obtain a preliminary number of semantic vectors corresponding to the query vector, wherein the preliminary number is determined based on the distribution density of the query vector in the feature space of a preset encoding dimension; and calling the second-level filtering interface to perform similarity calculation and filtering on the preliminary number of semantic vectors to obtain multiple candidate vectors.

[0011] Optionally, the semantic vectors in the initial screening are subjected to similarity fine-tuning to obtain multiple candidate vectors, including: calculating the standard cosine similarity between the query vector and each semantic vector in the initial screening; calculating the weighted vector inner product between the query vector and each semantic vector based on the word segmentation semantic weight of the query vector and the word segmentation semantic weight of each semantic vector; performing matching analysis on the query vector and each semantic vector using a preset language model to obtain a matching score; performing a weighted sum of the standard cosine similarity, the weighted vector inner product, and the matching score to determine the target similarity between the query vector and each semantic vector; and selecting multiple candidate vectors from the semantic vectors in the initial screening based on the target similarity.

[0012] Optionally, the multi-level deduplication threshold includes multiple similarity thresholds set based on target similarity. These multiple similarity thresholds correspond to multiple repetition levels. The deduplication result is obtained by sorting and analyzing multiple candidate vectors based on the multi-level deduplication thresholds, including: sorting the multiple candidate vectors based on the target similarity between the multiple candidate vectors and the query vector to obtain a sorting result; traversing the multiple candidate vectors according to the sorting result, comparing the target similarity corresponding to the current candidate vector with the multi-level deduplication threshold to determine the repetition level corresponding to the current candidate vector; and generating deduplication results based on the repetition levels corresponding to the multiple candidate vectors respectively.

[0013] According to another aspect of the embodiments of this application, a vehicle is also provided, including: an on-board memory storing an executable program; and an on-board processor for running the program, wherein the program executes any of the above-mentioned text deduplication methods during runtime.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the storage medium is located to execute any of the above-mentioned text deduplication methods.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the text deduplication method according to any one of the above.

[0016] In this embodiment, by performing normalization preprocessing on the input text to be checked, the format differences of the input text to be checked can be reduced, and the input text to be checked can be transformed into a normalized current text fragment. The normalization of the current text fragment is better, which helps the subsequent system to perform semantic encoding processing on the current text fragment more accurately. Furthermore, by using a target encoding model to perform semantic encoding processing on the current text fragment, the current text fragment in natural language form is transformed into a series of more easily understood query vectors, which can improve the accuracy of the system in recalling multiple candidate vectors corresponding to the query vector from the semantic vector index, thereby improving the accuracy of the text deduplication method. Furthermore, a semantic vector index is constructed based on historical text sets and graph indexing algorithms, so that multiple candidate vectors corresponding to the query vector can be quickly recalled from the semantic vector index by calling the similarity retrieval interface. Furthermore, by setting multi-level deduplication thresholds, the retrieved multiple candidate vectors are sorted and analyzed to obtain duplicate results, which helps users to perform subsequent processing and analysis of the deduplication results and quickly realize text deduplication. Therefore, this application achieves the goal of transforming the input text to be checked and the historical text set into vectorized representations, constructing semantic vector indexes, and performing text plagiarism checks efficiently and accurately. This achieves the technical effect of improving the accuracy and efficiency of text plagiarism checking methods, and solves the technical problems of low accuracy and low efficiency of text plagiarism checking methods in related technologies. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and provide related descriptions of those embodiments to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of an optional computing terminal for implementing a text deduplication method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a text deduplication method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of an optional text deduplication method according to an embodiment of this application;

[0021] Figure 4 This is a structural block diagram of a text plagiarism detection device according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments only include a part of the embodiments of the present application, and not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] First, the operating environment of the above method embodiments will be described by way of example. Figure 1 This is a hardware structure block diagram of an optional computing terminal for implementing a text plagiarism detection method according to an embodiment of this application, such as... Figure 1 As shown, the computing terminal 10 (e.g., a computer terminal, a mobile smart terminal, a vehicle terminal, or a cloud computing virtual terminal, etc.) may include: one or more processors 102 (e.g., processors 102a, 102b, ..., 102n), a memory 104 for storing data, and a transmission device 106 for implementing communication functions. The processor 102 may include, but is not limited to, processing components such as microprocessors (MCUs) or field-programmable gate arrays (FPGAs).

[0025] The aforementioned computing terminal 10 may further include: a display device 110, an input / output interface 108, a Universal Serial Bus (USB) port (which can be used as one of the ports of a computer bus, not shown in the figure), a network interface (not shown in the figure), a power supply (not shown in the figure), and a camera (not shown in the figure).

[0026] It should be noted that one or more processors 102 and / or other data processing circuits in the aforementioned computing terminal 10 may be wholly or partially embodied in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computing terminal 10 (or mobile device).

[0027] The memory 104 can be used to store software programs and modules of application software, such as the program instructions and data storage devices corresponding to the text plagiarism detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned text plagiarism detection method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the vehicle terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the vehicle terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC) and a network interface, which can be connected to other network devices via a base station to communicate with the Internet. The transmission device 106 can use wired and / or wireless network connections for data communication. In one example, the transmission device 106 can be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] The input / output interface can be connected to the corresponding input / output device of the computing terminal 10 to realize input / output functions. This input / output device may include, but is not limited to, a cursor control device, a keyboard, and a display. The aforementioned input / output device may be built into the computing terminal 10 or be an external device connected to the computing terminal 10.

[0030] Those skilled in the art will understand that Figure 1 The structure of the computing terminal 10 shown is for illustrative purposes only and does not impose strict limitations on the structure of the computing terminal 10 described above. For example, the computing terminal 10 may also include components that are larger than... Figure 1 The more or fewer components shown, or the computing terminal 10 may have the same Figure 1The components are shown in different categories.

[0031] This application provides a text deduplication method. This method can be used to provide information retrieval and processing functions for preset application scenarios. These preset application scenarios may include the following scenarios.

[0032] For example, in customer service and support scenarios, such as in the automotive and electronics industries, by utilizing the technical solutions of the embodiments of this application, repeated questions can be quickly identified when processing user feedback, avoiding repeated answers and improving customer service efficiency and user satisfaction.

[0033] For example, in social media platform scenarios, the technical solutions of this application can detect and filter duplicate or similar content, optimize the content recommendation system, and ensure the diversity and novelty of recommended content.

[0034] For example, in online forums and Q&A communities, the technical solutions of this application can effectively help users and administrators quickly identify duplicate questions with existing answers, reduce redundant answers, and improve the quality of the community knowledge base.

[0035] The text deduplication method for vehicles provided in this application embodiment can be applied to, but is not limited to, the application scenarios listed above. With the continuous evolution of technology, the above method can also be applied to a wider range of scenarios, such as information record management and work assistance correction. By performing text deduplication efficiently and accurately, it supports a variety of advanced functions and applications.

[0036] Under the above operating environment, the embodiments of this application provide the following: Figure 2 The text plagiarism detection method shown is as follows: Figure 2 This is a flowchart of a text plagiarism detection method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following implementation steps.

[0037] Step S201: Perform normalization preprocessing on the input text to be checked for plagiarism to obtain the current text fragment.

[0038] The input text to be checked for plagiarism can represent user feedback information (e.g., feedback submitted by users through the in-vehicle screen system, feedback submitted by users through the customer service platform, etc.). This input text can be extracted from customer service dialogue records, user log files, or, in particular, real-time input by the user in the text input boxes displayed on the system's interactive interface. The input text to be checked for plagiarism can include, but is not limited to: fault description text, function suggestion text, and function experience report text data. The input text to be checked for plagiarism can be structured text data; specifically, the input text to be queried can be read from the system database, storing the original user-input feedback information in a structured format within the system database.

[0039] In an exemplary application scenario, before performing standardized preprocessing on the input text to be checked for plagiarism, the input text can be read from the database using a unified encoding method. This converts the original user input feedback information according to a unified character encoding, resulting in the input text to be queried. This reduces parsing errors caused by inconsistent character encodings of the input text and improves compatibility with processing input text using different character encodings. Specifically, the unified encoding method can be a variable-length character encoding method (e.g., UTF-8). During the process of reading the input text to be checked for plagiarism, preliminary preprocessing can also be performed on the original user input feedback information stored in the database. For example, invalid information (such as illegal characters, spaces, garbled characters, invalid tags, emoticons, special graphic symbols, etc.) can be filtered out, and the punctuation format in the original user input feedback information can be unified (e.g., converting full-width characters to half-width characters), thereby obtaining the input text to be checked for plagiarism.

[0040] The above-mentioned standardized preprocessing may include, but is not limited to: word segmentation, stop word filtering, noise filtering, stemming, and part-of-speech tagging (for example, analyzing the semantic roles of words in the input text to be deduplicated helps the system to understand the text structure more accurately; these semantic roles may include subject, predicate, object, etc.).

[0041] Understandably, since the input text to be checked is user feedback, different users have significantly different descriptive habits. Directly using this input text would increase the difficulty of subsequent text plagiarism detection, thus reducing the system's efficiency. In this application, by performing normalization preprocessing on the input text to be checked, it can be transformed into normalized text, resulting in a current text fragment. Compared to the input text, this current text fragment is clearer and more standardized, helping the system to quickly understand and process it.

[0042] The aforementioned current text fragment can be normalized text data. Specifically, the aforementioned current text fragment can be obtained by performing word segmentation, stop word filtering, and noise filtering on the input text to be checked for plagiarism. Using word segmentation technology, the input text to be checked for plagiarism can be divided into finer-grained text fragments. Using stop word filtering technology, redundant data in the input text to be checked for plagiarism can be removed. Using noise filtering, noise interference can be reduced. The resulting current text fragment can more accurately and clearly represent the semantics of the user's input feedback information.

[0043] It is easy to understand that by performing normalization preprocessing on the input text to be checked, the format differences of the input text to be checked can be reduced, and the input text to be checked can be transformed into a normalized current text fragment. The normalization of the current text fragment is better, which helps the subsequent system to perform semantic encoding processing on the current text fragment.

[0044] Step S202: Use the target encoding model to perform semantic encoding on the current text fragment to obtain the query vector.

[0045] The aforementioned target encoding model can be used to convert the current text fragment into a numerical vector representation. This target encoding model can be pre-trained or loaded from a pre-downloaded model library. The target encoding model can include, but is not limited to: text vectorization models (e.g., Text to Vector, or Text2Vec model), and sentence bidirectional encoder representation models (e.g., Sentence Bidirectional Encoder Representations from Transformers, or Sentence-BERT model).

[0046] Target encoding models can identify the inherent semantic structure of the current text fragment, mapping the natural language text to points in a semantic vector space to obtain a query vector. In particular, they can map the natural language text to a fixed-dimensional (e.g., 768-dimensional) semantic vector space.

[0047] The query vector mentioned above can be a vector represented in numerical form. This query vector can be used to characterize the semantic features of the current text fragment (e.g., the topic, sentiment, entities, etc. of the current text fragment). It is understood that by using a target encoding model to semantically encode the current text fragment, converting the natural language form of the current text fragment into a structured numerical vector, the semantic information of the current text fragment can be preserved while simplifying the data format. This transforms the text comparison problem into mathematical operations between vectors, laying the foundation for efficiently retrieving multiple candidate vectors corresponding to the query vector from the semantic vector index.

[0048] It is easy to understand that by using the target encoding model to transform the current text fragment into a query vector, a mapping from the natural language space to the numerical space is achieved, and the text comparison problem is transformed into mathematical operations between vectors, which helps to efficiently and accurately find data similar to the current text fragment.

[0049] Step S203: Call the similarity retrieval interface corresponding to the target search engine to retrieve multiple candidate vectors corresponding to the query vector from the semantic vector index. The semantic vector index is constructed based on the historical text set and the graph index algorithm in the target search engine.

[0050] The aforementioned target search engine can refer to a similarity search library (also known as an approximate nearest neighbor search library). This target search engine can provide a similarity retrieval interface and can incorporate algorithms for various search vector sets. The aforementioned similarity retrieval interface may include, but is not limited to: a retrieval interface built on standard cosine similarity, a retrieval interface built on target similarity, a retrieval interface built on inner product, a retrieval interface built on Euclidean distance, and a retrieval interface built on Manhattan distance.

[0051] The aforementioned historical text set can be historical input text, or it can refer to user input feedback information in historical records. This historical text set can be obtained through continuous data collection and storage, or it can be loaded from a pre-defined historical database. In the process of constructing a semantic vector index, by building the historical text set into a set of vector data, the corresponding semantic vector space of the historical text set is obtained, which can help identify whether there is content in the historical text set that is repeated or similar to the current text fragment.

[0052] The aforementioned graph indexing algorithm refers to an algorithm used to construct and optimize vector index structures. This graph indexing algorithm can be used to determine similarity relationships in a vector space. During the construction of a semantic vector index, this graph indexing algorithm can store similar data from historical text sets in nearby positions within the semantic vector space corresponding to those historical text sets. This eliminates the need to compare each query vector with vectors in the semantic vector space corresponding to the historical text sets during subsequent searches. It allows for rapid location and retrieval of multiple candidate vectors similar to the current text fragment within the semantic vector space corresponding to the historical text sets, thereby improving search efficiency.

[0053] It's important to note that graph indexing constructs a multi-layered graph structure, where each layer contains all vector nodes, but the number of connections between vector nodes decreases with each layer. During the search process, the graph indexing algorithm first performs rapid location at the top level of the graph index structure, then gradually delves into lower layers until reaching the bottom layer, reducing the number of vector nodes that need to be compared and thus improving search speed.

[0054] The aforementioned semantic vector index can refer to an index structure based on machine learning technology. It represents the semantic features of a document by converting each document in the historical text set into vector form, and stores these vector-form historical text sets in a semantic vector space (which can also be understood as a vector index library). Furthermore, based on graph indexing algorithms, multiple candidate vectors similar to the current text fragment are efficiently retrieved from the semantic vector space.

[0055] By calling the similarity retrieval interface of the target search engine, multiple candidate vectors corresponding to the query vector can be retrieved in the semantic space (i.e., multiple candidate vectors corresponding to the query vector are recalled from the semantic vector index). In particular, multiple vectors that are close to the query vector (high similarity) can be recalled from the semantic vector index as candidate vectors. When the distance between a vector in the semantic space and the query vector is less than a distance threshold, it can be considered that the vector is close to the query vector (i.e., the vector can be considered to have high similarity to the query vector).

[0056] It is easy to understand that a semantic vector index is constructed based on the historical text set and the graph indexing algorithm. By constructing this semantic vector index, the historical text set can be converted into a vector representation. Then, in the subsequent process of searching for vectors similar to the query vector from the historical text set, multiple candidate vectors similar to the query vector can be quickly retrieved based on the graph indexing algorithm by calling the similarity retrieval interface. Multiple candidate vectors corresponding to the query vector can be quickly recalled from the semantic vector index.

[0057] Step S204: Based on the multi-level deduplication threshold, sort and analyze multiple candidate vectors to obtain the deduplication result. The deduplication result is used to characterize the duplication between the historical text and the input text corresponding to multiple candidate vectors.

[0058] The aforementioned multi-level deduplication thresholds can be used to determine the level of repetition between historical text and input text. This repetition level can characterize the degree of similarity between the historical text corresponding to the candidate vector and the input text. The levels of repetition can include, but are not limited to: Level 1 (complete repetition), Level 2 (semantic similarity), Level 3 (structural similarity), Level 4 (slight similarity), and Level 5 (low similarity).

[0059] The aforementioned multi-level deduplication thresholds may include, but are not limited to: thresholds set based on cosine similarity, thresholds set based on vector inner product, and thresholds set based on a custom similarity metric (e.g., similarity determined by comprehensively considering multiple analysis results obtained through multiple analysis methods).

[0060] The values ​​of the aforementioned multi-level plagiarism detection thresholds can be determined using the following methods: First, a data-driven approach, which involves analyzing the distribution of similar texts in a historical text set and combining statistical analysis methods to set the values ​​of the multi-level plagiarism detection thresholds, or automatically generating the values ​​of the multi-level plagiarism detection thresholds using a machine learning model to match the characteristics and distribution patterns of the dataset; Second, a dynamic adjustment method based on user ratings, which dynamically adjusts the values ​​of the multi-level plagiarism detection thresholds based on user ratings of the plagiarism detection results to better meet user expectations and plagiarism detection needs. It is understood that by setting multi-level plagiarism detection thresholds in this application, the system can classify the duplication between historical texts corresponding to multiple candidate vectors and input texts into multiple levels, providing more refined plagiarism detection results.

[0061] The plagiarism detection results described above can be a quantitative representation of the degree of repetition between the input text and documents in a historical text set. These results may include, but are not limited to: the level of repetition, similarity scores corresponding to multiple candidate vectors, and the historical texts corresponding to multiple candidate vectors.

[0062] In an exemplary application scenario, a similarity score is calculated between each candidate vector and the query vector from a pool of candidate vectors. The similarity score of each candidate vector is compared to a set multi-level plagiarism threshold. Based on the threshold range into which each candidate vector's similarity score falls, the candidate vectors are classified into the corresponding plagiarism level. Further, within each plagiarism level, the candidate vectors are sorted according to their similarity scores. The sorted candidate vectors are then integrated to obtain the plagiarism detection result. It should be noted that different sorting methods can be used when sorting within different plagiarism levels. For example, for the first level, sorting can be based solely on the similarity score, while for the second level, sorting can be combined with the similarity score and other features of the candidate vectors in the second level. It is understandable that sorting and analyzing multiple candidate vectors yields clearer and more accurate plagiarism detection results, helping users to further process and analyze the results and quickly achieve text plagiarism detection.

[0063] In this embodiment, by performing normalization preprocessing on the input text to be checked, the format differences of the input text to be checked can be reduced, and the input text to be checked can be transformed into a normalized current text fragment. The normalization of the current text fragment is better, which helps the subsequent system to perform semantic encoding processing on the current text fragment more accurately. Furthermore, by using a target encoding model to perform semantic encoding processing on the current text fragment, the current text fragment in natural language form is transformed into a series of more easily understood query vectors, which can improve the accuracy of the system in recalling multiple candidate vectors corresponding to the query vector from the semantic vector index, thereby improving the accuracy of the text deduplication method. Furthermore, a semantic vector index is constructed based on historical text sets and graph indexing algorithms, so that multiple candidate vectors corresponding to the query vector can be quickly recalled from the semantic vector index by calling the similarity retrieval interface. Furthermore, by setting multi-level deduplication thresholds, the retrieved multiple candidate vectors are sorted and analyzed to obtain duplicate results, which helps users to perform subsequent processing and analysis of the deduplication results and quickly realize text deduplication. Therefore, this application achieves the goal of transforming the input text to be checked and the historical text set into vectorized representations, constructing semantic vector indexes, and performing text plagiarism checks efficiently and accurately. This achieves the technical effect of improving the accuracy and efficiency of text plagiarism checking methods, and solves the technical problems of low accuracy and low efficiency of text plagiarism checking methods in related technologies.

[0064] Optionally, the above text plagiarism detection method also includes the following implementation steps:

[0065] Step S205: Perform normalization preprocessing on the historical text set to be indexed to obtain historical text corpus;

[0066] Step S206: Using the target encoding model and preset encoding dimensions, perform semantic encoding processing on the historical text corpus to obtain a dense semantic vector set;

[0067] Step S207: Construct a semantic vector index using a graph indexing algorithm and a dense semantic vector set. The graph indexing algorithm is used to perform similar nearest neighbor search in the feature space of a preset encoding dimension according to the target graph structure parameters.

[0068] The aforementioned set of historical texts to be indexed can be understood as the historical input texts needed in the text deduplication process. This historical text corpus may include multiple historical texts and instances of repetition between them.

[0069] In an exemplary application scenario, such as Figure 3 As shown, the normalization preprocessing process can include word segmentation, stop word filtering, and noise filtering. During word segmentation, the system calls a Chinese word segmentation tool to identify word boundaries in the historical input text set according to a preset segmentation algorithm. This historical text set is then segmented into a series of meaningful basic units (e.g., independent words), and these basic units are output as the segmentation result. The aforementioned word segmentation algorithm can be selected based on the needs of the text plagiarism detection task. This algorithm should focus on extracting key semantic words from the text to avoid over-segmentation affecting semantic accuracy. Specifically, a custom dictionary can be imported into the system. This custom dictionary includes, but is not limited to, industry terms, professional terms, and common abbreviations. During word segmentation, this custom dictionary enhances the word segmentation algorithm's ability to recognize text in specific fields (e.g., vehicle technology).

[0070] Continuing in the aforementioned application scenario, during the stop word filtering process, the system loads a pre-defined domain-specific stop word list from memory. It then uses a hash algorithm to match words in the segmentation results against this list, extracting words that successfully match the stop word list and removing these matched words to obtain the initial filtering result. Further, during noise filtering, the system loads a pre-defined noise word list from memory and filters the initial filtering result based on this list, obtaining an intermediate filtering result. Finally, it analyzes the word frequency of each word in the intermediate filtering result. Based on a set word frequency threshold (denoted as TF), the word frequency of each word in the intermediate filtering result is compared with the threshold. Words with frequencies below the threshold are removed (i.e., low-frequency invalid words are removed from the intermediate filtering result), resulting in a historical text corpus (e.g., ultimately obtaining 10,000 historical texts and the repetition patterns among them). The aforementioned word frequency threshold can be set according to actual application requirements. In some applications, during the noise filtering process described above, special character cleanup and removal of standard markup language (e.g., HTML) tags for webpage structure and layout can also be performed on the initial filtering results. Specifically, the normalization preprocessing performed on the historical text set to be indexed is the same as the normalization preprocessing performed on the input text to be deduplicated. It is understood that performing normalization preprocessing on the historical text set to be indexed according to the above-described process yields the historical text corpus; performing normalization preprocessing on the input text to be deduplicated according to the above-described process yields the current text fragment (which can be considered as normalized plain text corpus).

[0071] The aforementioned preset encoding dimension can be set according to actual application requirements, for example, 768 dimensions. In some applications, this preset encoding dimension can also be set in conjunction with the performance of the target encoding model. This preset encoding dimension can refer to the dimensional standard for semantic information encoding. This preset encoding dimension can characterize the representation granularity of vectors (including query vectors and dense vectors in a dense semantic vector set) in a multidimensional feature space. Using the target encoding model and the preset encoding dimension, semantic encoding processing is performed on historical text corpora. The target encoding model can transform the historical text corpora into dense vectors with a vector length (equal to the preset encoding dimension), and these dense vectors are used to construct a dense semantic vector set.

[0072] The main difference between dense vectors and ordinary semantic vectors lies in their density and information carrying capacity. The dense vectors in the aforementioned dense semantic vector set are uniformly distributed in a pre-defined high-dimensional space, with each dimension carrying certain semantic features of the text. This allows the dense semantic vectors in the dense vector set to represent text semantics more comprehensively and precisely. In contrast, ordinary semantic vectors are usually sparse (i.e., ordinary semantic vectors have many zero-valued dimensions and only a few non-zero dimensions), which limits their ability to capture semantic information. Therefore, dense vector sets can more accurately reflect semantic differences in vector space and are more suitable for advanced natural language understanding tasks, especially text deduplication, text similarity calculation, and semantic retrieval.

[0073] In an exemplary application scenario, such as Figure 3 As shown, the Text2Vec model is selected as the target encoding model, and the preset encoding dimension is set to 768 dimensions. The vector generation process can be as follows: by setting the preset encoding dimension (e.g., 768 dimensions), the length of the vector output by the target encoding model can be determined. The target encoding model, through pre-training, can recognize the inherent semantic structure of the current text segment and map the semantic information in the historical text corpus to a high-dimensional vector space (i.e., map the natural language text to the numerical semantic vector space), thereby converting the historical text corpus into a dense vector of 768 dimensions, thus constructing a dense semantic vector set.

[0074] Continuing in the aforementioned application scenario, the semantic encoding process can be as follows: The normalized preprocessed historical text corpus is used as input to the target encoding model. This model transforms each word in the historical text corpus into a low-dimensional dense vector (i.e., word embedding), initially capturing the meaning of words in context and laying the foundation for subsequent deeper semantic feature understanding. Furthermore, in the encoder part of Text2Vec, a multi-layered neural network structure (which may include self-attention mechanisms, fully connected layers, recurrent neural network units, etc., capable of comprehensively analyzing the grammatical structure and semantic content of the text) processes the word embeddings, extracting the text's contextual dependencies and deep semantic features. Finally, the last layer of the Text2Vec encoder can perform vector normalization, outputting a vector of a preset encoding dimension (i.e., a dense vector within a dense semantic vector set). By utilizing the target encoding model for semantic encoding, natural language text can be transformed into structured numerical vectors, simplifying the data format and laying the foundation for subsequent high-dimensional vector index construction. In particular, the process of semantic encoding of historical text corpora described above is the same as the process of semantic encoding of the current text fragment described earlier.

[0075] The aforementioned target graph structure parameters may include, but are not limited to: the upper limit of the number of graph node connections during the similar nearest neighbor search process, the upper limit of the search depth, and the upper limit of the query search.

[0076] In an exemplary application scenario, the target search engine mentioned above is a similarity search engine (e.g., Facebook AI Similarity Search, or FAISS for short). The Hierarchical Navigable Small World Graph (HNSW) algorithm from FAISS is selected as the graph indexing algorithm. The process of constructing the semantic vector index can be as follows: using the HNSW algorithm, nodes and connecting edges are created in a feature space of a preset encoding dimension based on vectors in a dense semantic vector set, forming a network graph structure capable of quickly retrieving similar neighbors. This results in the construction of a semantic vector index in the feature space of the preset encoding dimension. Specifically, when a new vector (e.g., a query vector) is added, the semantic vector index inserts this new vector near its nearest neighbor node based on its position in the feature space, thus maintaining the orderliness of the index structure and query performance. Therefore, when a user submits new feedback text (i.e., the input text to be checked for plagiarism), the system can quickly locate the historical text that is semantically closest to the input text to be checked, achieving efficient plagiarism detection and similar content retrieval.

[0077] It is easy to understand that, through the above steps S205 to S207, in this embodiment of the application, by performing normalization preprocessing and semantic encoding on the historical text set, the historical text corpus can be converted into a dense vector with a preset encoding dimension. By using the graph indexing algorithm, similarity search can be performed efficiently in the high-dimensional feature space, and the candidate vector most similar to the input text to be deduplicated can be quickly found. Thus, by constructing a semantic vector index, this application can achieve fast retrieval of a large-scale historical text set, improving the efficiency and response speed of text deduplication.

[0078] Optionally, the above text plagiarism detection method also includes the following implementation steps:

[0079] Step S208: Perform data feature analysis on the historical text set to determine the corpus size features and vector distribution features;

[0080] Step S209: Based on the corpus size features, vector distribution features, and service optimization objectives, update the initial graph structure parameters in the graph indexing algorithm to target graph structure parameters. The service optimization objectives include recall and query latency objectives. The target graph structure parameters are used to constrain the upper limit of the number of graph node connections and the upper limit of the search depth during the similar nearest neighbor search process.

[0081] The aforementioned corpus size characteristics refer to the data volume characteristics that constitute the historical text set. These characteristics may include, but are not limited to: the total number of texts in the historical text set, and the growth trend of the historical text set (e.g., the number of new texts added daily, monthly, and annually). These corpus size characteristics can be used to characterize the size and growth dynamics of the historical text set. Utilizing corpus size characteristics helps in assessing the required computational resources (e.g., memory, processor time) and can guide the optimization of parameters in graph indexing algorithms to better balance search performance and resource consumption. Especially in large-scale data environments, utilizing corpus size characteristics can improve the efficiency of graph index construction and query response speed.

[0082] The aforementioned vector distribution features can refer to the geometric characteristics of the dense semantic vector set corresponding to the historical text set. Specifically, they can refer to the geometric characteristics of the dense vector set corresponding to the historical text set after semantic encoding, under a preset encoding dimension (e.g., 768 dimensions). These vector distribution features may include, but are not limited to: clustering density (average intra-class distance), separation degree (average inter-class distance), and variance distribution (mean variance of each dimension). These vector distribution features can be obtained by calculating the Euclidean distance or cosine similarity between vectors, or by using vector dimensionality reduction analysis methods (including nonlinear dimensionality reduction methods, such as t-SNE, and linear dimensionality reduction methods, such as PCA). Utilizing vector distribution features helps to more accurately adjust the parameters in the graph indexing algorithm, so as to more quickly update the initial graph structure parameters to the target graph structure parameters that meet the service optimization objectives.

[0083] By utilizing data feature analysis methods to analyze historical text sets, the corpus size and vector distribution characteristics can be determined. These data feature analysis methods may include, but are not limited to, dimensionality reduction, statistical analysis, clustering, and vector density estimation. Specifically, multiple data feature analysis methods can be combined during the analysis of historical text sets. For example, statistical analysis can be used to determine the corpus size, while dimensionality reduction can be used to determine the vector distribution. By selecting data feature analysis methods that combine the characteristics of both corpus size and vector distribution, the specificity of the analysis can be enhanced, leading to more accurate determination of these characteristics.

[0084] The aforementioned service optimization goals refer to the expected standards of system performance and user experience when building and maintaining graph indexes. These goals can be extracted from online Service Level Agreements (SLAs). They may include recall and query latency goals. The recall goal characterizes the system's ability to correctly retrieve all relevant historical text. The query latency goal characterizes the system's real-time processing capabilities.

[0085] By parsing the online Service Level Agreement (SLA), the constraints (i.e., service optimization goals) can be extracted, including recall targets (e.g., minimum recall greater than or equal to 98%) and query latency targets (e.g., maximum allowed query latency less than or equal to 50 milliseconds). During the process of updating the initial graph structure parameters in the graph indexing algorithm to the target graph structure parameters, the service optimization goals can be considered in conjunction with the needs of the actual application's business scenarios to balance index quality and performance.

[0086] The initial graph structure parameters mentioned above may include, but are not limited to: the upper limit of the number of graph node connections during the similar nearest neighbor search process, the upper limit of the search depth, and the upper limit of the query search. The upper limit of the number of graph node connections during the similar nearest neighbor search process can refer to the maximum number of connections between a graph node and its neighbors. This number of graph node connections will affect the local density and connectivity of the graph, directly impacting the neighborhood coverage during the query phase.

[0087] The aforementioned upper limit on search depth can be used to determine the size of the candidate set for insertion points during the construction phase. This upper limit will affect the search depth and accuracy of the inserted nodes, thus impacting the graph construction speed. It is understandable that the upper limit on the number of graph node connections and the upper limit on search depth in the similar nearest neighbor search process directly affect the graph construction speed, memory consumption, and recall and latency during queries. Therefore, during the update of the initial graph structure parameters, optimization is primarily focused on these two parameters.

[0088] The aforementioned query search upper limit can be used to control the size of the search candidate set during the query phase. This query search upper limit can adapt to the requirements of deduplication accuracy and latency during the query phase. In particular, during the process of updating the initial graph structure parameters, this query search upper limit can be set to a fixed value, eliminating the need to update the query search upper limit.

[0089] In an exemplary application scenario, such as Figure 3As shown, the HNSW algorithm is selected as the graph indexing algorithm mentioned above. The initial graph structure parameters in the HNSW algorithm include the upper limit of the number of graph node connections and the upper limit of the search depth during the similar nearest neighbor search process. The number of graph node connections during the similar nearest neighbor search process is defined as parameter "M", and the upper limit of the search depth is defined as parameter "efConstruction". The larger the "M" value, the better the connectivity of the graph and the higher the recall rate, but the construction time and memory consumption will also be greater, which will reduce the search speed. In particular, if the "M" value is high (e.g., M≥48), the number of edges connected to each graph node increases, which can more completely cover its feature semantic space. At this time, similar or duplicate content will fall into the same semantic cluster, forming a high-density subgraph in the graph. When performing deduplication, the entire subgraph can be accessed with fewer jumps. The larger the "efConstruction" value, the higher the quality of the constructed index (higher recall rate), but it will reduce the construction speed. In particular, a larger "efConstruction" value (e.g., 400 to 800) allows more paths to be searched, ensuring that the insertion position is close to the real semantic neighbors, which can effectively reduce orphans and index fragmentation, thereby improving the accuracy and recall stability during deduplication. In practical applications, the values ​​of parameter "M" and parameter "efConstruction" can be determined by combining the data volume of the historical text set (i.e., corpus size characteristics), vector distribution characteristics, and service optimization objectives. For example, the value of parameter "M" can be determined to be 16, and the value of parameter "efConstruction" can be determined to be 200.

[0090] In another exemplary application scenario, a parameter setting and optimization process was designed to further optimize the performance of the text deduplication method. This process consists of three steps: offline performance sampling and grid evaluation, multi-objective Bayesian optimization strategy, and real-traffic gray-scale verification and finalization.

[0091] In offline performance sampling and grid evaluation, 10,000 historical texts with duplicate annotations were extracted from the historical text set to evaluate the performance of the HNSW index under different parameter combinations. Furthermore, the historical text corpus was converted into a 768-dimensional dense vector using a target encoding model to construct a dense semantic vector set. Using the graph indexing algorithm and the dense semantic vector set, multiple sets of semantic vector indices were constructed. Parameter combinations were set for each set of semantic vector indices. The parameters of each set of semantic vector indices include "M" (the upper limit of the number of graph node connections in the similar nearest neighbor search process) and "efConstruction" (the upper limit of the search depth), where the value of "M" can be 16, 32, 48, or 64, and the value of "efConstruction" can be 100, 200, 400, or 800. By evaluating the similarity of the top 10 recalled vectors in each index group (denoted as recall@10), the index building time (denoted as T_build), the peak memory requirement (denoted as Mem_peak), and the peak query latency of the query text (denoted as p99), further, parameter combinations that meet specific performance indicators (e.g., similarity of recalled vectors greater than or equal to 95%, peak query latency less than or equal to 50 milliseconds, and peak memory requirement less than or equal to 8GB) are selected, and performance surfaces are plotted to identify the optimal region. In the multi-objective Bayesian optimization strategy, an objective function is defined, denoted as J(M; efC), specifically, this objective function is expressed as:

[0092] J=α(1-racall@10)+β(T_bulid / T_max)+γ(Mem_peak / M_max),

[0093] Where T_max represents the maximum time taken to build the index, and M_max represents the maximum value of the peak memory requirement; further, Gaussian process regression modeling is used, combined with the expected improvement function (denoted as EI) for iterative optimization sampling. In the initial stage, 5 to 10 parameter points are randomly evaluated, followed by 20 to 30 iterations until the objective function converges. Through the Bayesian optimization strategy, a better set of parameter combinations for the semantic vector index can be determined (i.e., the set of parameter combinations that achieves the best balance in the case of multi-objective conflict). In the gray-scale verification and finalization of real traffic, gray-scale clusters (e.g., 2 clusters) can be divided in the real deployment environment, and different parameter combinations can be deployed to these two clusters respectively. Real-time monitoring and collection of operation indicators (including similarity (recall@10), peak memory requirement (Mem_peak), and peak query latency of query text (p99)) are performed. Based on the collected operation indicators, a comprehensive scoring mechanism is applied, as shown in Equation (1), to score the parameter combinations. 4. According to the ranking of the scores, the highest one is selected as the final index building parameter configuration in the production environment.

[0094] Score1=0.5·recall@10-0.3(p99-t1)-0.2(Mem_peak / Mem_max) Formula (1)

[0095] In equation (1), t1 represents the peak value of the target query latency, and Mem_max represents the upper limit of the peak memory requirement.

[0096] It is easy to understand that, through the above steps S208 to S209, in this embodiment of the application, by performing data feature analysis on the historical text set, the corpus size characteristics and vector distribution characteristics are determined, providing data support for updating the initial graph structure parameters in the graph indexing algorithm; furthermore, based on the corpus size characteristics, vector distribution characteristics, and service optimization objectives, the graph structure parameters in the graph indexing algorithm are dynamically adjusted, which can realize dynamic parameter adaptive optimization, thereby achieving fully automated operation and maintenance.

[0097] Optionally, the above text plagiarism detection method also includes the following implementation steps:

[0098] Step S210: Based on the preset encoding dimension, the initial encoding model is subjected to mean pooling to obtain an intermediate encoding model, wherein the output dimension of the intermediate encoding model is determined by the preset encoding dimension.

[0099] Step S211: Using the historical text set and the preset text enhancement algorithm, fine-tune the parameters of the intermediate encoding model to obtain the target encoding model. The preset text enhancement algorithm includes at least one of the following: semantic similarity enhancement algorithm, contrastive learning algorithm, and knowledge distillation algorithm.

[0100] The output dimension of the above intermediate encoding model can be equal to the above preset encoding dimension.

[0101] In one exemplary application scenario, based on a preset encoding dimension, the vector matrix output by the initial encoding model is subjected to mean pooling along the preset encoding dimension. That is, the average value of each local region in the vector matrix output by the initial encoding model is calculated to generate a feature vector of a fixed size. Specifically, a mean pooling layer can be set after the output layer of the initial encoding model to obtain an intermediate encoding model. This intermediate encoding model can output results with a preset encoding dimension. For example, if the preset encoding dimension is set to 768 dimensions, the output of the intermediate encoding model will be a 768-dimensional vector. By performing mean pooling, regardless of the length of the input text, the vector dimension output by the intermediate encoding model can be ensured to remain consistent. Compared to the initial encoding model, the intermediate encoding model can output a more stable encoded vector.

[0102] The aforementioned pre-defined text enhancement algorithms can be used to optimize the parameters of intermediate encoding models to enhance their performance on specific tasks (e.g., text deduplication). These pre-defined text enhancement algorithms include at least one of the following: semantic similarity enhancement algorithms, contrastive learning algorithms, and knowledge distillation algorithms. The semantic similarity enhancement algorithm can adjust the parameters of the intermediate encoding model to more accurately capture semantic information by comparing the similarity between text before and after encoding. The contrastive learning algorithm can adjust the parameters of the intermediate encoding model to more accurately distinguish between similar and dissimilar text vector representations by introducing additional positive and negative samples for comparison. The knowledge distillation algorithm can extract knowledge from a high-performance large model and transfer the extracted knowledge to the intermediate encoding model, enabling the intermediate encoding model to exhibit high encoding quality even with a smaller training set.

[0103] The aforementioned historical text set can be used to provide text data for specific domains. During the process of fine-tuning the parameters of the intermediate coding model, the performance of the intermediate coding model in that specific domain (e.g., the vehicle domain) can be enhanced by using the historical text set to provide text data for that specific domain.

[0104] In an exemplary application scenario, the aforementioned historical text set originates from the field of vehicle technology. This historical text set is used in conjunction with a pre-defined text enhancement algorithm to fine-tune the parameters of the intermediate encoding model. The pre-defined text enhancement algorithm may include at least one of the following: a semantic similarity enhancement algorithm, a contrastive learning algorithm, or a knowledge distillation algorithm. Specifically, the semantic similarity enhancement algorithm may include a contrastive loss function, a dual-tower model, etc. When selecting a semantic similarity enhancement algorithm to fine-tune the parameters of the intermediate encoding model, text pairs with similarity and differences are selected as training data. The parameters of the intermediate encoding model are adjusted to minimize the distance between the encoding vectors of similar text pairs while maximizing the vector distance between different text pairs. Thus, after fine-tuning the parameters of the intermediate encoding model, the vector distance between similar texts decreases, and the vector distance between dissimilar texts increases. This allows the output of the target encoding model to more accurately reflect the semantic relationships between texts, effectively improving the recall and precision in the plagiarism detection task.

[0105] Continuing in the aforementioned application scenario, when using a contrastive learning algorithm to fine-tune the parameters of the intermediate encoding model, positive and negative sample pairs are constructed. Positive sample pairs typically consist of two versions of the same text obtained through different transformations, while negative sample pairs are instances from different texts. By maximizing the similarity score of the positive sample pairs and minimizing the similarity score of the negative sample pairs, the parameters of the intermediate encoding model are optimized, resulting in a target encoding model that more accurately represents the semantics of the text. This contrastive learning algorithm is well-suited for unsupervised applications, leveraging large amounts of unlabeled data to improve the performance of the intermediate encoding model.

[0106] Still in the above application scenario, when the knowledge distillation algorithm is selected to fine-tune the parameters of the intermediate encoding model, the knowledge extracted by the pre-trained large model (teacher model) is transferred to the small model (student model), that is, transferred to the intermediate encoding model. By imitating the decision-making process of the teacher model, the intermediate encoding model can be fine-tuned on the historical text set, and finally the target encoding model is obtained.

[0107] It is easy to understand that, through the above steps S210 to S211, in this embodiment of the application, by utilizing the above-mentioned mechanism of pre-training the target encoding model, the target encoding model can more accurately understand the terms of a specific domain, making the target encoding model more suitable for text deduplication in a specific domain corresponding to a historical text set, thereby enabling the output of the target encoding model to more accurately reflect the semantic relationship between texts, thus effectively improving the recall and precision in the text deduplication task.

[0108] Optionally, the similarity retrieval interface includes a first-level calling interface built based on standard cosine similarity and a second-level filtering interface built based on target similarity. In step S203 above, calling the similarity retrieval interface to retrieve multiple candidate vectors corresponding to the query vector from the semantic vector index includes the following implementation steps:

[0109] Step S231: Call the first-level call interface to query the semantic vector corresponding to the initial screening quantity from the semantic vector index. The initial screening quantity is determined based on the distribution density of the query vector in the feature space of the preset encoding dimension.

[0110] Step S232: Call the second-level filtering interface to perform similarity calculation and filtering on the semantic vectors of the initial screening quantity to obtain multiple candidate vectors.

[0111] The aforementioned first-level API can be designed based on standard cosine similarity. By calling this API, the cosine similarity between each vector in the sense vector index and the query vector can be calculated, quickly finding vectors in the sense vector index that are similar in direction to the query vector. The first-level API, built on standard cosine similarity, can improve deduplication efficiency while maintaining a certain level of comprehensiveness in deduplication, making it suitable for retrieval in high-dimensional vector spaces.

[0112] The distribution density of the query vector in the feature space of the preset encoding dimension refers to the density of other vectors within the multidimensional vector space region where the query vector is located. By determining the distribution density of the query vector in the feature space of the preset encoding dimension, the number of semantically similar texts around the query vector can be indirectly determined. The more vectors near the query vector in the feature space of the preset encoding dimension, the higher the distribution density, which also means that there may be more potential duplicate or related texts in this region.

[0113] The initial screening count is determined based on the distribution density of the query vector in the feature space of the preset encoding dimension. When the query vector is located in a sparse region, the system automatically increases the initial screening count to improve the recall rate; conversely, it decreases the count to optimize the allocation of computing resources. This achieves dynamic adjustment of the initial screening count. Based on this mechanism of dynamically adjusting the initial screening count, the system can quickly identify candidate vectors within a certain range, laying a preliminary foundation for further screening and deduplication.

[0114] The aforementioned second-level filtering interface can be constructed based on target similarity. By calling the second-level filtering interface, the semantic vectors retrieved from the initial screening can be further filtered, eliminating semantic vectors that do not semantically match the query vector. Specifically, a target filtering threshold can be set. For any semantic vector in the initial screening, the target similarity between that semantic vector and the query vector can be calculated. When the target similarity between the semantic vector and the query vector is less than the target filtering threshold, the semantic vector is considered not to match the query vector semantically and is thus removed, thereby improving the accuracy of candidate vectors and providing users with more precise result matching.

[0115] In an exemplary application scenario, such as Figure 3 As shown, during the plagiarism detection process, the aforementioned distribution density can be determined by statistically analyzing the number and distance of vectors within the neighborhood of the query vector. Specifically, the K-nearest neighbor algorithm can be used to quantify the vector distribution across the entire feature space and construct a density estimation model. This allows for the rapid determination of the distribution density of the query vector in the feature space of the preset encoding dimension during text plagiarism detection. The initial screening quantity is dynamically adjusted based on the distribution density of the query vector in the feature space of the preset encoding dimension; that is, the K value is adaptively adjusted. When the distribution density is high, the initial screening quantity is reduced to avoid too many irrelevant vectors entering the subsequent fine-tuning stage, reducing computational resource consumption. When the distribution density is low, the initial screening quantity is increased to ensure that a sufficient number of candidate vectors are recalled, thereby improving the plagiarism detection recall rate.

[0116] Still in the above application scenario, by calling the first-level call interface provided by FAISS (e.g., the Top-K query interface), the semantic vector corresponding to the initial screening quantity (in particular, the minimum and maximum values ​​of the initial screening quantity can be set to provide a dynamic adjustment range, for example, if the minimum value of the initial screening quantity is set to 10 and the maximum value of the initial screening quantity is set to 30, then the dynamic adjustment range of the initial screening quantity is 10 to 30) can be obtained from the semantic vector index.

[0117] It should be noted that before calling the similarity retrieval interface to retrieve multiple candidate vectors corresponding to the query vector from the semantic vector index, Euclidean norm normalization (i.e., L2 normalization) can be performed on the query vector output by the target encoding model to ensure the consistency of target similarity calculation between the semantic vector and the query vector. Furthermore, Term Frequency-Inverse Document Frequency (TF-IDF) vectors or entity word vectors corresponding to the keywords can be added to the query vector. Concatenating the query vector with the TF-IDF vectors or entity word vectors forms a concatenated vector structure, improving semantic coverage.

[0118] It is readily understood that, through steps S231 to S232 described above, this embodiment of the application designs a two-layer retrieval interface. The first-level calling interface performs rapid cosine similarity calculation, quickly filtering out semantic vectors irrelevant to the query vector. The second-level filtering interface, through more refined similarity calculation, ensures the accuracy of the plagiarism detection results. This layered retrieval strategy guarantees both the comprehensiveness and efficiency of the text plagiarism detection method. Furthermore, by setting a mechanism to dynamically adjust the number of initial screenings, the number of initial screenings can be automatically increased to improve recall when the query vector is located in a sparse region; and automatically decreased to optimize computational resource allocation when the query vector is located in a dense region.

[0119] Optionally, in step S232 above, performing similarity calculations on the semantic vectors initially screened to obtain multiple candidate vectors includes the following implementation steps:

[0120] Step S2321: Calculate the standard cosine similarity between the query vector and each semantic vector in the semantic vector of the initial screening quantity;

[0121] Step S2322: Calculate the weighted vector inner product between the query vector and each semantic vector based on the word segmentation semantic weight of the query vector and the word segmentation semantic weight of each semantic vector.

[0122] Step S2323: Using a preset language model, perform matching analysis on the query vector and each semantic vector to obtain a matching score;

[0123] Step S2324: Perform a weighted summation of the standard cosine similarity, weighted vector inner product, and matching score to determine the target similarity between the query vector and each semantic vector;

[0124] Step S2325: Select multiple candidate vectors from the semantic vectors of the initial screening based on the target similarity.

[0125] The standard cosine similarity mentioned above can be used as an indicator to measure the angular similarity between two vectors. This standard cosine similarity can be used to characterize their directional consistency in multidimensional space. For any semantic vector in the initial screening of semantic vectors, the standard cosine similarity is equal to the dot product of the two vectors (i.e., the query vector and the semantic vector) divided by the product of their magnitudes. It can be understood that the range of the standard cosine similarity is [-1, 1]. The closer the standard cosine similarity value is to 1, the more similar the two vectors are; a value close to 0 indicates that the two vectors are almost orthogonal, i.e., dissimilar; a value close to -1 indicates that the two vectors are in opposite directions.

[0126] The aforementioned semantic weights for word segmentation refer to the numerical values ​​assigned to each word during text vectorization, reflecting its semantic importance within the document content. These semantic weights can be determined based on the domain lexicon corresponding to the input text or automatically set by the attention network. The aforementioned weighted vector inner product can be used to characterize the contribution of different words or features in text similarity judgment. By calculating the weighted vector inner product, it is possible to highlight those words that are more crucial and distinctive in the text, and more accurately determine the semantic strength difference between the query vector and each semantic vector. In calculating the weighted vector inner product, each dimension of the query vector and the initial screening vector can be multiplied by the corresponding weight coefficient before performing the inner product operation.

[0127] The matching score mentioned above can be used to characterize the semantic and structural matching degree between the query vector and each semantic vector. This matching score may include, but is not limited to: the similarity probability predicted by the pre-defined language model, the semantic alignment in a specific context, and the contextual consistency score.

[0128] In one exemplary application scenario, query vectors and semantic vectors are input into a pre-defined language model, which then performs matching analysis on the query vectors and initial screening vectors. Specifically, the query vector can be concatenated with each semantic vector and input into the pre-defined language model. The model then extracts scores or probabilities related to matching from the input data, using these scores or probabilities as the matching score. Alternatively, a local vector pooling method can be used. The query vector and each semantic vector are input into the pre-defined language model, which divides the query vector and each semantic vector into windows. The maximum matching score within each local window is calculated, and then weighted and fused to obtain the matching score. Calculating the matching score provides a more comprehensive semantic comparison dimension for the plagiarism detection algorithm. Especially when dealing with complex contexts and long texts, calculating the matching score between the query vector and each semantic vector helps to more accurately distinguish and identify duplicate content, making the text plagiarism detection method more robust to duplicate parts in long text fragments.

[0129] The aforementioned target similarity can be obtained using a weighted mathematical operation method or a machine learning weighted calculation model. The aforementioned weighted mathematical operation method may include, but is not limited to: direct addition, arithmetic mean, and weighted summation. Specifically, the aforementioned target similarity can be calculated using equation (2).

[0130] Score(q,d)=α·Cossin(q,d)+β·WIP(q,d)+γ·NLPBoost(q,d) Formula (2)

[0131] In Equation (2), Score represents target similarity, Cossin represents standard cosine similarity, WIP represents weighted vector inner product, NLPBoost represents matching score, q represents query vector, d represents semantic vector, and α, β, and γ represent adjustable weight parameters.

[0132] In an exemplary application scenario, after calculating the target similarity between the query vector and each semantic vector, multiple candidate vectors can be selected from the initially screened number of semantic vectors. The number of candidate vectors selected can be set based on the actual application (e.g., selecting 5 candidate vectors). It should be noted that during the selection of candidate vectors, 5 semantic vectors with a target similarity greater than the target selection threshold can be randomly selected from the initially screened number of semantic vectors as multiple candidate vectors. When the number of semantic vectors with a target similarity greater than the target selection threshold between the query vector and the semantic vectors in the initially screened number of semantic vectors is less than 5, only the semantic vectors with a target similarity greater than the target selection threshold between the query vector and the semantic vectors in the initially screened number of semantic vectors can be selected as multiple candidate vectors. Alternatively, the top 5 semantic vectors with the highest target similarity values ​​(in descending order of target similarity value) from the initially screened number of semantic vectors can be selected as multiple candidate vectors.

[0133] It is easy to understand that, through the above steps S2321 to S2325, in this embodiment of the application, by calculating the standard cosine similarity, weighted vector inner product, and matching score between the query vector and each semantic vector respectively, the distance between the query vector and each semantic vector can be determined at multiple scales (e.g., vectors at the overall, sentence, and word levels). Furthermore, by performing a weighted summation of the standard cosine similarity, weighted vector inner product, and matching score, the target similarity between the query vector and each semantic vector is obtained, which enables distance fusion at multiple scales, improves the adaptability of duplicate detection in complex text structures, and thus, based on the target similarity, multiple candidate vectors can be selected more accurately from the initial number of semantic vectors.

[0134] Optionally, the multi-level deduplication threshold includes multiple similarity thresholds set based on target similarity, with each similarity threshold corresponding to multiple levels of repetition. In step S204 above, the process of sorting and analyzing multiple candidate vectors based on the multi-level deduplication threshold to obtain the deduplication result includes the following implementation steps:

[0135] Step S241: Based on the target similarity between multiple candidate vectors and the query vector, sort the multiple candidate vectors to obtain the sorting result;

[0136] Step S242: Based on the sorting results, traverse multiple candidate vectors, compare the target similarity corresponding to the current candidate vector in the traversal with the multi-level deduplication threshold, and determine the duplication level corresponding to the current candidate vector.

[0137] Step S243: Generate plagiarism detection results based on the repetition levels corresponding to the multiple candidate vectors.

[0138] The aforementioned repetition levels can include Level 1 (complete repetition), Level 2 (semantic similarity), Level 3 (structural similarity), and Level 5 (slight similarity). It is understood that these multiple similarity thresholds correspond to multiple repetition levels. In some application scenarios, these multiple similarity thresholds may include a first threshold, a second threshold, a third threshold, a fourth threshold, and a fifth threshold, which correspond to Level 1 (complete repetition), Level 2 (semantic similarity), Level 3 (structural similarity), and Level 4 (slight similarity), respectively. Specifically, the aforementioned multiple similarity thresholds may also include a standard cosine similarity threshold.

[0139] The sorting result can be an ordered list of vectors. It can be understood that each candidate vector corresponds to a target similarity. The sorting result is obtained by ranking the candidate vectors in descending order of their target similarity to the query vector, based on the target similarity between the candidate vectors and the query vector.

[0140] In an exemplary application scenario, such as Figure 3 As shown, after obtaining the sorting results, multiple candidate vectors are traversed based on the sorting results. The target similarity corresponding to the current candidate vector is compared with the multi-level deduplication threshold to determine the duplication level of the current candidate vector. The specific process of comparing the target similarity corresponding to the current candidate vector with the multi-level deduplication threshold can be as follows: the standard cosine similarity corresponding to the current candidate vector is compared with the standard cosine similarity threshold (for example, set to 0.090). If the standard cosine similarity corresponding to the current candidate vector is less than or equal to the standard cosine similarity threshold, it is considered that the current candidate vector does not duplicate the query vector; if the standard cosine similarity corresponding to the current candidate vector is greater than the standard cosine similarity threshold, it is considered that the current candidate vector duplicates the query vector, and the duplication level corresponding to the current candidate vector is further determined.

[0141] Continuing with the aforementioned application scenario, the first, second, third, fourth, and fifth thresholds can be set to 0.90, 0.80, 0.60, and 0.30 respectively. If the target similarity corresponding to the current candidate vector is greater than or equal to the first threshold (0.90), the repetition level of the current candidate vector can be determined to be the first level (complete repetition level). If the target similarity corresponding to the current candidate vector is greater than or equal to the second threshold (0.80) and less than the first threshold (0.90), the repetition level of the current candidate vector can be determined to be the second level (semantic similarity level). If the target similarity corresponding to the current candidate vector is greater than or equal to the third threshold (0.60) and less than the second threshold (0.80), then the repetition level corresponding to the current candidate vector can be determined to be the third level (structural similarity level). If the target similarity corresponding to the current candidate vector is greater than or equal to the fourth threshold (0.30) and less than the third threshold (0.60), then the repetition level corresponding to the current candidate vector can be determined to be the fourth level (slight similarity level). If the target similarity corresponding to the current candidate vector is less than the fourth threshold (0.30), then it can be considered that the current candidate vector and the query vector do not overlap.

[0142] In another exemplary application scenario, the above-mentioned multiple similarity thresholds may include a first cosine threshold, an entity repetition threshold, a matching threshold, a structural matching threshold, a soft loop threshold, and a structural vector distance threshold. The above-mentioned complete repetition level can indicate that the text corresponding to the current candidate vector and the query vector is completely consistent. The above-mentioned semantic similarity level can indicate that the text corresponding to the current candidate vector and the query vector differs in literal content but describes the same function (i.e., the meaning is similar but the details of the problem description are slightly different). The above-mentioned structural similarity level can indicate that the structural patterns corresponding to the current candidate vector and the query vector are similar but the semantics are not similar. Specifically, if the standard cosine similarity of the current candidate vector is greater than or equal to the first cosine threshold (e.g., 0.99) and the entity repetition of the current candidate vector is greater than the entity repetition threshold (e.g., 0.9), then the repetition level of the current candidate vector can be considered to be the first level (complete repetition level). The aforementioned entity repetition can be determined using a preset language model. If the matching score of the current candidate vector is greater than or equal to the matching threshold (e.g., 0.8) and the structural matching degree of the current candidate vector is greater than the structural matching degree threshold (e.g., 0.75), then the repetition level of the current candidate vector can be considered to be the second level (semantic similarity level). In this case, the target similarity of the current candidate vector can also be considered to be greater than or equal to the second threshold. The aforementioned structural matching degree can be obtained using a preset structural matching model. If the soft loop value of the current candidate vector (which can be obtained using a preset structural matching model) is greater than or equal to the soft loop threshold (e.g., 0.65), then the structural vector distance of the current candidate vector is considered to be less than the structural vector distance threshold (denoted as τ). In this case, the repetition level of the current candidate vector can be considered to be the third level (structural similarity level).

[0143] The aforementioned repetition results may include the text corresponding to multiple candidate vectors, as well as the repetition levels corresponding to multiple candidate vectors. In some application scenarios, the aforementioned repetition results may include, but are not limited to: repetition level labels, target similarity, standard cosine similarity, structural matching degree, matching score, and entity repetition degree.

[0144] In another exemplary application scenario, the process of ranking and analyzing multiple candidate vectors based on multi-level redundancy thresholds to obtain the plagiarism detection results can be implemented by the system's plagiarism detection module. This plagiarism detection module can include four levels of components, each with independent logic and incremental learning capabilities. Specifically, the plagiarism detection module can include a Multi-Level Redundancy Decision Engine layer to provide multi-level redundancy classification logic; a Semantic Feature Fusion layer to extract multi-dimensional structural and semantic features beyond text similarity; an Explainable Output Generator layer to output the reasons for the judgment, weight distribution, and suggested corrections; and an Online Label Loop & Error Injection layer to update the judgment logic based on user evaluations of the plagiarism detection results.

[0145] It is readily understood that, through steps S241 to S243 described above, in this embodiment of the application, by utilizing a multi-level deduplication threshold design, the system can distinguish different degrees of repetition based on the text similarity level, thereby enhancing the interpretability and practicality of the deduplication results. Furthermore, multiple multi-level deduplication thresholds corresponding to various similarity indicators can be set. By comprehensively considering multiple similarity indicators, the system can more comprehensively evaluate the similarity between texts, improving the accuracy of the deduplication results.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0147] It should be noted that, for the sake of simplicity, the technical solutions in the above method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the order of actions in the described action combination, because according to this application, some of the above steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this application specification are preferred embodiments, and the actions and modules involved are not necessarily essential for implementing the technical solutions of this application.

[0148] According to embodiments of the present invention, a device embodiment for a text plagiarism detection apparatus is also provided. This text plagiarism detection apparatus is used to implement the above-described method embodiments and various optional implementations of the method embodiments. The technical content already described above will not be repeated in the device embodiment. It should be noted that, in the following related descriptions of the device embodiment, "module" can be software, hardware, or a combination of software and hardware used to implement a specified function.

[0149] Please refer to Figure 4 The aforementioned text plagiarism detection device may include a preprocessing module 401, used to perform normalization preprocessing on the input text to be checked for plagiarism, to obtain the current text segment; an encoding module 402, used to perform semantic encoding processing on the current text segment using a target encoding model, to obtain a query vector; a calling module 403, used to call the similarity retrieval interface corresponding to the target search engine, to recall multiple candidate vectors corresponding to the query vector from the semantic vector index, wherein the semantic vector index is constructed based on the historical text set and the graph index algorithm in the target search engine; and an analysis module 404, used to sort and analyze the multiple candidate vectors based on a multi-level plagiarism threshold, to obtain plagiarism detection results, wherein the plagiarism detection results are used to characterize the duplication between the historical text and the input text corresponding to the multiple candidate vectors.

[0150] It should be noted that the preprocessing module 401, encoding module 402, calling module 403 and analysis module 404 mentioned above correspond to steps S201 to S204 in the method embodiment. The four modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above method embodiment.

[0151] It should be noted that the modules mentioned in the above device embodiments can be implemented by software, hardware, or a combination of both. For example, when the modules are implemented by hardware, they can be placed in the same processor, or they can be placed in different processors in any combination. As another example, the modules can be hardware or software components stored in memory and processed by one or more processors; they can also run as part of a computing terminal.

[0152] According to an embodiment of the present invention, a vehicle embodiment is also provided, the vehicle including an on-board memory and an on-board processor, the on-board memory storing a computer program, and the on-board processor being configured to run the computer program to implement the above-described text plagiarism detection method.

[0153] According to an embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to implement the above-described text deduplication method.

[0154] Optionally, the aforementioned computer storage media may include, but are not limited to: hard disk drives (HDDs), solid state drives (SSDs), USB flash drives, optical discs, memory cards, cloud storage media, and network-attached storage (NAS) devices.

[0155] Optionally, the aforementioned computer-readable storage medium may be configured to store a computer program for performing the following steps: performing normalization preprocessing on the input text to be deduplicated to obtain a current text fragment; performing semantic encoding processing on the current text fragment using a target encoding model to obtain a query vector; calling the similarity retrieval interface corresponding to the target search engine to recall multiple candidate vectors corresponding to the query vector from the semantic vector index, wherein the semantic vector index is constructed based on the historical text set and the graph index algorithm in the target search engine; and performing sorting analysis on the multiple candidate vectors based on a multi-level deduplication threshold to obtain a deduplication result, wherein the deduplication result is used to characterize the duplication situation between the historical text corresponding to the multiple candidate vectors and the input text.

[0156] According to embodiments of the present invention, a computer program product is also provided. This computer program product includes a computer program that, when executed by a processor, can implement the aforementioned text deduplication method.

[0157] Optionally, the aforementioned computer program product can provide text plagiarism detection services based on the aforementioned text plagiarism detection method.

[0158] Optionally, in this embodiment, the computer program product described above may be a set of instructions and code pre-written according to the text plagiarism detection method described above. This computer program product can run on various computer platforms, including personal computers, servers, mobile devices, etc.

[0159] Optionally, in this embodiment, the instructions and code corresponding to the computer program product are used to implement the following method steps: performing normalization preprocessing on the input text to be deduplicated to obtain the current text segment; performing semantic encoding processing on the current text segment using the target encoding model to obtain the query vector; calling the similarity retrieval interface corresponding to the target search engine to recall multiple candidate vectors corresponding to the query vector from the semantic vector index, wherein the semantic vector index is constructed based on the historical text set and the graph index algorithm in the target search engine; and performing sorting analysis on the multiple candidate vectors based on the multi-level deduplication threshold to obtain the deduplication result, wherein the deduplication result is used to characterize the duplication situation between the historical text corresponding to the multiple candidate vectors and the input text.

[0160] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of multiple modules can be a logical functional division, and in actual implementation in application scenarios, there can be any other possible division methods. Furthermore, multiple modules (or units or components within modules) can be combined with each other and integrated into another system. For example, some features in the method embodiments described above can be ignored or skipped during execution.

[0162] It should be noted that in the above embodiments, the modules, components, or units described as separate parts can be physically separated or physically integrated. The components shown as modules or units can be physical modules or units, or virtual modules or units. That is, multiple modules or units can be located in the same position or distributed across multiple positions or spaces. In application scenarios, depending on the actual needs of the scenario, some or all of the multiple modules or units can be selected to implement the technical solutions of the embodiments of this application, thereby achieving the corresponding technical objectives.

[0163] Specifically, for integrated functional modules or units, if they are implemented as software functional units and sold or used as independent products, the module or functional unit can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0164] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A text plagiarism detection method, characterized in that, include: The input text to be checked for plagiarism is preprocessed to obtain the current text fragment; The current text fragment is semantically encoded using a target encoding model to obtain a query vector; The similarity retrieval interface corresponding to the target search engine is invoked to retrieve multiple candidate vectors corresponding to the query vector from the semantic vector index, wherein the semantic vector index is constructed based on the historical text set and the graph indexing algorithm in the target search engine; The multiple candidate vectors are sorted and analyzed based on multi-level deduplication thresholds to obtain deduplication results, wherein the deduplication results are used to characterize the duplication between the historical text corresponding to the multiple candidate vectors and the input text.

2. The text plagiarism detection method according to claim 1, characterized in that, The text deduplication method also includes: The historical text set to be indexed is preprocessed to obtain the historical text corpus; Using the target encoding model and the preset encoding dimension, the historical text corpus is semantically encoded to obtain a dense semantic vector set; The semantic vector index is constructed using the graph indexing algorithm and the dense semantic vector set, wherein the graph indexing algorithm is used to perform similar nearest neighbor search in the feature space of the preset encoding dimension according to the target graph structure parameters.

3. The text plagiarism detection method according to claim 2, characterized in that, The text deduplication method also includes: Data feature analysis is performed on the historical text set to determine the corpus size features and vector distribution features; Based on the corpus size features, the vector distribution features, and the service optimization objective, the initial graph structure parameters in the graph indexing algorithm are updated to the target graph structure parameters. The service optimization objective includes a recall target and a query latency target. The target graph structure parameters are used to constrain the upper limit of the number of graph node connections and the upper limit of the search depth in the similar nearest neighbor search process.

4. The text plagiarism detection method according to claim 1, characterized in that, The text deduplication method also includes: Based on a preset encoding dimension, the initial encoding model is subjected to mean pooling to obtain an intermediate encoding model, wherein the output dimension of the intermediate encoding model is determined by the preset encoding dimension. Using the historical text set and a preset text enhancement algorithm, the intermediate encoding model is fine-tuned to obtain the target encoding model. The preset text enhancement algorithm includes at least one of the following: semantic similarity enhancement algorithm, contrastive learning algorithm, and knowledge distillation algorithm.

5. The text plagiarism detection method according to claim 1, characterized in that, The similarity retrieval interface includes a first-level calling interface built based on standard cosine similarity and a second-level filtering interface built based on target similarity. Calling the similarity retrieval interface retrieves the plurality of candidate vectors corresponding to the query vector from the semantic vector index, including: The first-level call interface is invoked to query the semantic vector index to obtain the semantic vector of the initial screening quantity corresponding to the query vector, wherein the initial screening quantity is determined based on the distribution density of the query vector in the feature space of the preset encoding dimension. The second-level filtering interface is called to perform similarity calculation and filtering on the semantic vectors of the initial screening quantity to obtain the multiple candidate vectors.

6. The text deduplication method according to claim 5, characterized in that, The semantic vectors of the initial screening quantity are subjected to similarity calculation and further screening to obtain the multiple candidate vectors, including: Calculate the standard cosine similarity between the query vector and each semantic vector in the initial screening quantity; Based on the word segmentation semantic weights of the query vector and the word segmentation semantic weights of each semantic vector, the weighted vector inner product between the query vector and each semantic vector is calculated. Using a pre-defined language model, a matching analysis is performed on the query vector and each semantic vector to obtain a matching score; The target similarity between the query vector and each semantic vector is determined by weighted summation of the standard cosine similarity, the weighted vector inner product, and the matching score. Based on the target similarity, the plurality of candidate vectors are selected from the semantic vectors of the initial screening quantity.

7. The text plagiarism detection method according to claim 1, characterized in that, The multi-level deduplication thresholds include multiple similarity thresholds set based on target similarity. These multiple similarity thresholds correspond to multiple levels of repetition. Based on the multi-level deduplication thresholds, the multiple candidate vectors are sorted and analyzed to obtain the deduplication results, including: Based on the target similarity between the multiple candidate vectors and the query vector, the multiple candidate vectors are sorted to obtain a sorting result; Based on the sorting result, the multiple candidate vectors are traversed, and the target similarity corresponding to the current candidate vector in the traversal is compared with the multi-level deduplication threshold to determine the duplication level corresponding to the current candidate vector; The plagiarism detection result is generated based on the repetition level corresponding to the multiple candidate vectors.

8. A vehicle, characterized in that, include: Onboard storage device, containing executable programs; An onboard processor is used to run the program, wherein the program executes the text deduplication method according to any one of claims 1 to 7 when it runs.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the text deduplication method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the text deduplication method according to any one of claims 1 to 7.

Images