Method for checking duplicate of mass resource data of mobile terminal and background

By adopting a multi-source data plagiarism checking model based on semantic matching and multi-layer perceptron optimization technology in the mobile terminal and backend massive resource data plagiarism checking technology, the problem of insufficient scalability and adaptability in the existing technology is solved, and the problem of "insufficient" is realized. Efficient and accurate text matching and plagiarism checking process, supporting different application scenarios and text data types.

CN119025928AInactive Publication Date: 2024-11-26GUANGDONG YOUTI INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410817358.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-24
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing mobile and backend data plagiarism checking technology has shortcomings in terms of scalability and adaptability, which affects the efficiency of plagiarism checking and cannot meet the needs of different application scenarios.

Method used

SDSMM, a multi-source data plagiarism check model based on semantic matching, combines BERT and XLNet models for data preprocessing and feature extraction, uses a multi-layer perceptron to optimize the matching layer, outputs fixed-length matching vectors, and improves unsupervision through network crawling technology. Corpus crawling efficiency realizes modular design and automated plagiarism checking process.

Benefits of technology

It improves the accuracy and efficiency of plagiarism checking, enhances the scalability and adaptability of the system, can quickly generate matching results, reduces computational complexity and manual intervention, supports text data of different sizes and types, and ensures long-term stable operation.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a method for checking duplicate of mass resource data of a mobile terminal and a background. According to the method, the semantic relationship between the texts can be better captured through the multi-layer perceptron, so that the matching accuracy is improved. And through nonlinear transformation and dimension reduction operation, deeper semantic information of the text can be mined. And by outputting a matching vector with the length of 768, subsequent matching and analysis are facilitated. Due to the fixed length, the calculation complexity is reduced, and the matching efficiency is improved. Selection of the number of neurons of the hidden layer can be adjusted according to actual requirements so as to optimize the performance of the model. The selection of the activation function can influence the output of the model, and the tanh function can provide good nonlinear expression ability. And expandability and adaptability are achieved, the requirements of different application scenes are considered, and high expandability and adaptability are achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data duplication checking, and specifically relates to a method for checking duplicate data of a mobile terminal and backend massive resources. Background Art

[0002] Mobile and backend massive resource data duplication checking refers to the process of quickly checking large amounts of resource data for duplication on mobile devices. This technology is widely used in content review, copyright protection, anti-cheating and other fields, aiming to ensure the originality and compliance of data. The main challenges faced by massive resource data duplication checking include large data volume, high duplication checking speed requirements, and performance limitations of mobile devices. In order to meet these challenges, mobile and backend massive resource data duplication checking technology needs to be efficient, accurate, and responsive in real time. Mobile and backend massive resource data duplication checking technology is of great significance in protecting original content, combating infringement, and improving content quality. With the rapid development of mobile Internet, this technology will be more widely used in content review, copyright protection, anti-cheating and other fields.

[0003] However, common methods cannot take into account the needs of different application scenarios, which makes the overall scalability and adaptability of the method insufficient, thus affecting the overall efficiency of duplicate checking. Summary of the invention

[0004] The purpose of the present invention is to provide a method for checking duplicate data of massive resources on a mobile terminal and in the background in order to solve the above-mentioned problems.

[0005] The technical solution adopted by the present invention is as follows: a method for checking duplicate data of a mobile terminal and a backend massive resource, the method comprising the following steps:

[0006] S1: front-end data collection: The mobile front-end collects the data that users need to check for duplicates through various means, including text, pictures, etc., and pre-processes and converts them into comparable data formats;

[0007] S2: Data transmission: the collected data is transmitted to the backend server through the network;

[0008] S3: Data storage: After receiving the data, the backend server stores it in a database or file system;

[0009] S4: Data preprocessing: Preprocess the stored data, including data segmentation, data part-of-speech tagging, data feature extraction, etc., to prepare for subsequent duplicate checking;

[0010] S5: Conduct duplicate checking model training: Use the multi-source data duplicate checking model SDSMM based on semantic matching to train the duplicate checking model for massive data and establish a duplicate checking model;

[0011] S6: Perform duplicate checking and matching: input the data to be checked for duplicates into the duplicate checking model, perform similarity calculation, and determine whether it is duplicate data;

[0012] S7: Display the duplicate checking result, return the duplicate checking result to the mobile terminal, and display it to the user, after which the method of checking duplicates on the entire mobile terminal and the background massive multi-source data can be completed.

[0013] In a preferred embodiment, in step S4, data segmentation is performed using a pre-trained model such as BERT. The algorithm steps are:

[0014] Feed the input text sequence into the BERT model.

[0015] The BERT model outputs an embedding vector for each word.

[0016] According to the output of BERT, the continuous word embedding vector is split into words.

[0017] In a preferred embodiment, in step S4, feature extraction uses a BERT model to extract text features.

[0018] The algorithm steps are:

[0019] Input the segmented text sequence into the BERT model.

[0020] The BERT model outputs an embedding vector for each word.

[0021] In a preferred embodiment, in step S4, data part-of-speech tagging is performed using a NER model.

[0022] The algorithm steps are:

[0023] The segmented text sequence is input into the NER model.

[0024] The NER model outputs a part-of-speech tag for each word.

[0025] Label the words according to their part-of-speech tags.

[0026] In a preferred embodiment, in step S5, a document data duplication detection model based on semantic matching is used;

[0027] The operation steps of the document data duplication detection model based on semantic matching are as follows:

[0028] a. Representation layer: Use two weight-sharing Document-Encoder models to perform semantic extraction on the input document to obtain a deep semantic representation of the document.

[0029] b. Matching layer: A multi-layer perceptron is used to interact the two semantic representations output by the representation layer to generate an interaction vector of fixed length.

[0030] c. Fully connected layer: The interaction vector output by the matching layer is used as input, and binary classification is performed to output a two-dimensional probability distribution, indicating the probability of text similarity or dissimilarity.

[0031] d. Output: The model’s final output label has two values: 0 and 1, which represent the text similarity and dissimilarity matching results respectively.

[0032] e. Evaluation: Use accuracy and F1 score to evaluate the performance of the model.

[0033] f. Training: Use the SDSMC dataset to perform supervised training on the SDSMM model, freeze the low-level parameters, and only fine-tune the high-level parameters.

[0034] g. Test: Verify the document duplication detection effect of the model on the SDSMC test set.

[0035] h. Optimization: Adopt a pre-training fine-tuning strategy, use unsupervised data for pre-training, and then use a small amount of labeled data for supervised fine-tuning to avoid overfitting.

[0036] In a preferred embodiment, in step S5, the representation layer of the SDSMM model is the first layer of the model, and Document-Encoder is used as a feature extractor to perform semantic extraction on the input document. The Document-Encoder model is based on the XLNet pre-trained model and has a strong ability to process long texts. The Document-Encoder model uses a hierarchical attention mechanism to pool the output of the XLNet model, and thus ultimately outputs a 768-dimensional semantic vector. The innovation of SDSMM in the representation layer is mainly the proposal of a document semantic representation model that not only completes the task of duplicate detection of short texts, but can also be effectively applied to the task of duplicate detection of long texts, thereby improving the model's ability to detect duplicates of texts of different lengths.

[0037] In a preferred embodiment, in step S5, the matching layer of the SDSMM model is the second layer of the model. The main task of the matching layer is to match the semantic representation of the text to be matched output by the representation layer, and output a semantic vector with a fixed length. The matching layer of Sentence-BERT directly concatenates the semantic representations u, v and lu-v of the text and inputs them into the softmax classifier for classification, without allowing the semantic representation of the text to fully interact with each other. This method uses a multi-layer perceptron in the matching layer to optimize the matching layer of the model, improve the degree of interaction between the semantic representations of the text, and output a matching vector with a fixed length of 768 dimensions.

[0038] The matching layer first subtracts the absolute value of the semantic representations u and v of the text to obtain |uv. Then u, v and |uv| are concatenated into a 2304-dimensional vector. The concatenated vector is then input into the multi-layer perceptron for information interaction. The multi-layer perceptron performs nonlinear transformation and dimensionality reduction operations on the input concatenated vector, and outputs a 768-dimensional matching vector to represent the interactive information of the text semantic representation.

[0039] The perceptron used in this method contains three hidden layers, and the number of neurons in each hidden layer is 2304, 1536, and 768. After the input vector is calculated by the three hidden layers, a matching vector with a dimension of 768 will be output. The three hidden layers of the perceptron all use the tanh function as the activation function.

[0040] In a preferred embodiment, the output calculation formula of the hidden layer I is:

[0041] a l =σ(W l a l-1 +b l ), where a l W represents the output vector of the lth layer. l and b l Represent the weight matrix and bias vector of the lth layer respectively. l-1 represents the output vector of the l-1th layer. σ represents the activation function, and this method uses the tanh function.

[0042] The hidden layer calculation formula is as follows:

[0043] First hidden layer:

[0044] a l =σ(W l a0+b l );

[0045] Where a_0 = [u, v, |uv|] (u, v are the semantic representations of the text to be matched);

[0046] Second hidden layer:

[0047] a2=σ(W2a1+b2);

[0048] The third hidden layer:

[0049] a3=σ(W2a1+b3);

[0050] Finally, the matching vector s output by the matching layer is:

[0051] s=W3a2+b3

[0052] In this structure, a0 is a concatenated vector containing u, v and |uv|. After the concatenated vector is calculated through three hidden layers, a matching vector s with a length of 768 is finally obtained.

[0053] In a preferred embodiment, in step S6, the SDSMC data set has been constructed, but large-scale corpus in the field has not been obtained. Web crawler technology is used in the duplicate matching process. By using the crawler toolkit of the python programming language, a web crawler applet is implemented to crawl some public corpus related to computer-related courses and save the crawled data locally. Considering that crawling data is an I / O intensive task, multi-threading technology can be used to improve execution efficiency. This method is based on python's multiprocessing multi-threaded library, and uses a thread pool to improve the single-threaded crawler technology, which greatly improves the crawling efficiency of unsupervised corpus.

[0054] In a preferred embodiment, after the duplicate checking in step S7 is completed, it is necessary to perform the following steps:

[0055] Key step records: Record key steps in the duplicate checking process, such as text collection, data transmission, preprocessing, duplicate checking and matching, etc. Record key information such as timestamps, input and output data, and execution results.

[0056] Error and exception handling: Record the errors and exceptions that occur during the duplicate checking process. Record the error type, occurrence time, error information, and handling measures.

[0057] Performance monitoring: Record the system's operating status and performance indicators, such as CPU and memory usage, query response time, etc. Monitor the performance trend of the system during long-term operation.

[0058] Log storage and query: Store log data in a reliable log system, such as the ELK stack. Provide log query and analysis functions to facilitate troubleshooting and performance optimization.

[0059] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0060] 1. In the present invention, the semantic relationship between texts can be better captured by the multi-layer perceptron, thereby improving the accuracy of matching. Through nonlinear transformation and dimensionality reduction operations, deeper semantic information of the text can be mined. By outputting a matching vector with a length of 768, subsequent matching and analysis are facilitated. The fixed length reduces the computational complexity and improves the matching efficiency. The number of hidden layer neurons can be adjusted according to actual needs to optimize the model performance. The choice of activation function can affect the output of the model, and the tanh function can provide good nonlinear expression capabilities. Scalability and adaptability: Taking into account the needs of different application scenarios, it has strong scalability and adaptability.

[0061] 2. In the present invention, the design of the hidden layer can be modified and optimized to cope with text data of different sizes and types. Automation and efficiency: The entire matching process is automated, reducing manual intervention and improving the efficiency of duplicate checking. The calculation process of the matching layer is efficient and can quickly generate matching results. Modular design: As part of the model, the matching layer can be trained and optimized independently of other modules. Modular design is conducive to the expansion and maintenance of the model. Long-term stable operation: The design of the matching layer takes into account the long-term stable operation of the model, which is conducive to the continuous optimization and upgrading of the system. The fixed matching vector length is conducive to the long-term stable operation of the model. The above method improves the accuracy, efficiency and adaptability of text matching by introducing a multi-layer perceptron and a fixed-length matching vector, and provides effective technical support for duplicate checking of mobile terminals and back-end massive resource data. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] Example:

[0064] A method for checking duplicate data between a mobile terminal and a backend massive resource, the method comprising the following steps:

[0065] S1: front-end data collection: The mobile front-end collects the data that users need to check for duplicates through various means, including text, pictures, etc., and pre-processes and converts them into comparable data formats;

[0066] S2: Data transmission: the collected data is transmitted to the backend server through the network;

[0067] S3: Data storage: After receiving the data, the backend server stores it in a database or file system;

[0068] S4: Data preprocessing: Preprocess the stored data, including data segmentation, data part-of-speech tagging, data feature extraction, etc., to prepare for subsequent duplicate checking;

[0069] S5: Conduct duplicate checking model training: Use the multi-source data duplicate checking model SDSMM based on semantic matching to train the duplicate checking model for massive data and establish a duplicate checking model;

[0070] S6: Perform duplicate checking and matching: input the data to be checked for duplicates into the duplicate checking model, perform similarity calculation, and determine whether it is duplicate data;

[0071] S7: Display the duplicate checking result, return the duplicate checking result to the mobile terminal, and display it to the user, after which the method of checking duplicates on the entire mobile terminal and the background massive multi-source data can be completed.

[0072] In step S4, data segmentation is performed using a pre-trained model such as BERT. The algorithm steps are:

[0073] Feed the input text sequence into the BERT model.

[0074] The BERT model outputs an embedding vector for each word.

[0075] According to the output of BERT, the continuous word embedding vector is split into words

[0076] In step S4, feature extraction uses the BERT model to extract text features.

[0077] The algorithm steps are:

[0078] Input the segmented text sequence into the BERT model.

[0079] The BERT model outputs an embedding vector for each word.

[0080] Concatenate the embedding vectors of all words together to get the vector representation of the entire text.

[0081] In step S4, data part-of-speech tagging is performed using a NER model.

[0082] The algorithm steps are:

[0083] The segmented text sequence is input into the NER model.

[0084] The NER model outputs a part-of-speech tag for each word.

[0085] Label the words according to their part-of-speech tags.

[0086] In step S5, a document data duplication detection model based on semantic matching is used;

[0087] The operation steps of the document data duplication detection model based on semantic matching are as follows:

[0088] a. Representation layer: Use two weight-sharing Document-Encoder models to perform semantic extraction on the input document to obtain a deep semantic representation of the document.

[0089] b. Matching layer: A multi-layer perceptron is used to interact the two semantic representations output by the representation layer to generate an interaction vector of fixed length.

[0090] c. Fully connected layer: The interaction vector output by the matching layer is used as input, and binary classification is performed to output a two-dimensional probability distribution, indicating the probability of text similarity or dissimilarity.

[0091] d. Output: The model’s final output label has two values: 0 and 1, which represent the text similarity and dissimilarity matching results respectively.

[0092] e. Evaluation: Use accuracy and F1 score to evaluate the performance of the model.

[0093] f. Training: Use the SDSMC dataset to perform supervised training on the SDSMM model, freeze the low-level parameters, and only fine-tune the high-level parameters.

[0094] g. Test: Verify the document duplication detection effect of the model on the SDSMC test set.

[0095] h. Optimization: Adopt a pre-training fine-tuning strategy, use unsupervised data for pre-training, and then use a small amount of labeled data for supervised fine-tuning to avoid overfitting.

[0096] In step S5, the representation layer of the SDSMM model is the first layer of the model, and Document-Encoder is used as a feature extractor to perform semantic extraction on the input document. The Document-Encoder model is based on the XLNet pre-trained model and has a strong ability to process long texts. The Document-Encoder model uses a hierarchical attention mechanism to pool the output of the XLNet model, and ultimately outputs a 768-dimensional semantic vector. The innovation of SDSMM in the representation layer is mainly the proposal of a document semantic representation model that can not only complete the task of duplicate detection of short texts, but can also be effectively applied to the task of duplicate detection of long texts, thereby improving the model's ability to detect duplicates of texts of different lengths.

[0097] In step S5, the matching layer of the SDSMM model is the second layer of the model. The main task of the matching layer is to match the semantic representation of the text to be matched output by the representation layer and output a semantic vector with a fixed length. The matching layer of Sentence-BERT directly concatenates the semantic representations u, v and lu-v of the text and inputs them into the softmax classifier for classification, without allowing the semantic representation of the text to fully interact with each other. This method uses a multi-layer perceptron in the matching layer to optimize the matching layer of the model, improve the degree of interaction between the semantic representations of the text, and output a matching vector with a fixed length of 768 dimensions.

[0098] The matching layer first subtracts the absolute value of the semantic representations u and v of the text to obtain |uv. Then u, v and |uv| are concatenated into a 2304-dimensional vector. The concatenated vector is then input into the multi-layer perceptron for information interaction. The multi-layer perceptron performs nonlinear transformation and dimensionality reduction operations on the input concatenated vector, and outputs a 768-dimensional matching vector to represent the interactive information of the text semantic representation.

[0099] The perceptron used in this method contains three hidden layers, and the number of neurons in each hidden layer is 2304, 1536, and 768. After the input vector is calculated by the three hidden layers, a matching vector with a dimension of 768 will be output. The three hidden layers of the perceptron all use the tanh function as the activation function.

[0100] The output calculation formula of the hidden layer I is:

[0101] a l =σ(W l a l-1 +b l ), where a l W represents the output vector of the lth layer. l and b l Represent the weight matrix and bias vector of the lth layer respectively. l-1 represents the output vector of the l-1th layer. σ represents the activation function, and this method uses the tanh function.

[0102] The hidden layer calculation formula is as follows:

[0103] First hidden layer:

[0104] a l =σ(W l a0+b l );

[0105] Where a_0 = [u, v, |uv|] (u, v are the semantic representations of the text to be matched);

[0106] Second hidden layer:

[0107] a2=σ(W2a1+b2);

[0108] The third hidden layer:

[0109] a3=σ(W2a1+b3);

[0110] Finally, the matching vector s output by the matching layer is:

[0111] s=W3a2+b3

[0112] In this structure, a0 is a concatenated vector containing u, v and |uv|. After the concatenated vector is calculated through three hidden layers, a matching vector s with a length of 768 is finally obtained.

[0113] In step S6, the SDSMC dataset has been constructed, but large-scale corpus in the field has not yet been obtained. Web crawler technology is used in the duplicate matching process. By using the crawler toolkit of the python programming language, a web crawler applet is implemented to crawl some public corpus related to computer-related courses and save the crawled data locally. Considering that crawling data is an I / O intensive task, multi-threading technology can be used to improve execution efficiency. This method is based on python's multiprocessing multi-threaded library, and uses a thread pool to improve the single-threaded crawler technology, which greatly improves the crawling efficiency of unsupervised corpus.

[0114] After the duplicate checking in step S7 is completed, it is necessary to:

[0115] Key step records: Record key steps in the duplicate checking process, such as text collection, data transmission, preprocessing, duplicate checking and matching, etc. Record key information such as timestamps, input and output data, and execution results.

[0116] Error and exception handling: Record the errors and exceptions that occur during the duplicate checking process. Record the error type, occurrence time, error information, and handling measures.

[0117] Performance monitoring: Record the system's operating status and performance indicators, such as CPU and memory usage, query response time, etc. Monitor the performance trend of the system during long-term operation.

[0118] Log storage and query: Store log data in a reliable log system, such as the ELK stack. Provide log query and analysis functions to facilitate troubleshooting and performance optimization.

[0119] In the present invention, the semantic relationship between texts can be better captured by the multi-layer perceptron, thereby improving the accuracy of matching. Through nonlinear transformation and dimensionality reduction operations, deeper semantic information of the text can be mined. By outputting a matching vector with a length of 768, subsequent matching and analysis are facilitated. The fixed length reduces the computational complexity and improves the matching efficiency. The number of hidden layer neurons can be adjusted according to actual needs to optimize the model performance. The choice of activation function can affect the output of the model, and the tanh function can provide good nonlinear expression capabilities. Scalability and adaptability: The needs of different application scenarios are taken into consideration, and it has strong scalability and adaptability.

[0120] 2. In the present invention, the design of the hidden layer can be modified and optimized to cope with text data of different sizes and types. Automation and efficiency: The entire matching process is automated, reducing manual intervention and improving the efficiency of duplicate checking. The calculation process of the matching layer is efficient and can quickly generate matching results. Modular design: As part of the model, the matching layer can be trained and optimized independently of other modules. Modular design is conducive to the expansion and maintenance of the model. Long-term stable operation: The design of the matching layer takes into account the long-term stable operation of the model, which is conducive to the continuous optimization and upgrading of the system. The fixed matching vector length is conducive to the long-term stable operation of the model. The above method improves the accuracy, efficiency and adaptability of text matching by introducing a multi-layer perceptron and a fixed-length matching vector, and provides effective technical support for duplicate checking of mobile terminals and back-end massive resource data.

[0121] It should be noted that, in the present method, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or still includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0122] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for checking duplicate data between a mobile terminal and a backend massive resource, characterized in that: The method comprises the following steps: S1: front-end data collection: The mobile front-end collects the data that users need to check for duplicates, including text and pictures, through various means, and performs pre-processing and converts them into comparable data formats; S2: Data transmission: the collected data is transmitted to the backend server through the network; S3: Data storage: After receiving the data, the backend server stores it in a database or file system; S4: Data preprocessing: Preprocess the stored data, including data segmentation, data part-of-speech tagging, and data feature extraction, in preparation for subsequent duplicate checking; S5: Conduct duplicate checking model training: Use the multi-source data duplicate checking model SDSMM based on semantic matching to train the duplicate checking model for massive data and establish a duplicate checking model; S6: Perform duplicate checking and matching: input the data to be checked for duplicates into the duplicate checking model, perform similarity calculation, and determine whether it is duplicate data; S7: Display the duplicate checking result, return the duplicate checking result to the mobile terminal, and display it to the user, after which the method of checking duplicates on the entire mobile terminal and the background massive multi-source data can be completed.

2. A method for checking duplicate data between a mobile terminal and a backend massive resource as claimed in claim 1, characterized in that: In step S4, data segmentation is performed using a pre-trained model such as BERT; the algorithm steps are: Input the input text sequence to the BERT model; The BERT model outputs an embedding vector for each word; According to the output of BERT, the continuous word embedding vector is split into words.

3. A method for checking duplicate data between a mobile terminal and a backend massive resource as claimed in claim 1, characterized in that: In step S4, feature extraction uses a BERT model to extract text features; The algorithm steps are: Input the segmented text sequence into the BERT model; The BERT model outputs an embedding vector for each word; Concatenate the embedding vectors of all words together to get the vector representation of the entire text.

4. A method for checking duplicate data between a mobile terminal and a backend massive resource as claimed in claim 1, characterized in that: In step S4, data part-of-speech tagging is performed using a NER model; The algorithm steps are: Input the segmented text sequence into the NER model; The NER model outputs the part-of-speech tag for each word; Label the words according to their part-of-speech tags.

5. A method for checking duplicate data between a mobile terminal and a backend massive resource as claimed in claim 1, characterized in that: In the step S5, a document data duplication detection model based on semantic matching is used; The operation steps of the document data duplication detection model based on semantic matching are as follows: a. Representation layer: Use two weight-sharing Document-Encoder models to perform semantic extraction on the input document to obtain a deep semantic representation of the document; b. Matching layer: A multi-layer perceptron is used to interact the two semantic representations output by the representation layer to generate an interaction vector of fixed length; c. Fully connected layer: takes the interaction vector output by the matching layer as input, performs binary classification, and outputs a two-dimensional probability distribution, indicating the probability of text similarity or dissimilarity; d. Output: The model outputs labels with two values: 0 and 1, which represent the text similarity and dissimilarity matching results respectively. e. Evaluation: Use accuracy and F1 score to evaluate the performance of the model; f. Training: Use the SDSMC dataset to perform supervised training on the SDSMM model, freeze the low-level parameters, and only fine-tune the high-level parameters; g. Test: Verify the document duplication detection effect of the model on the SDSMC test set; h. Optimization: Adopt a pre-training fine-tuning strategy, use unsupervised data for pre-training, and then use a small amount of labeled data for supervised fine-tuning to avoid overfitting.

6. A method for checking duplicate data between a mobile terminal and a backend massive resource as claimed in claim 1, characterized in that: In the step S5, the representation layer of the SDSMM model is the first layer of the model, and Document-Encoder is used as a feature extractor to perform semantic extraction on the input document; the Document-Encoder model is based on the XLNet pre-trained model and has a strong ability to process long texts; the Document-Encoder model uses a hierarchical attention mechanism to pool the output of the XLNet model, and thus ultimately outputs a 768-dimensional semantic vector; the innovation of SDSMM in the representation layer is mainly to propose a document semantic representation model so that it can not only complete the repetition detection task of short texts, but also be effectively applied to the repetition detection task of long texts, thereby improving the model's ability to detect repetitions in texts of different lengths.

7. A method for checking duplicate data between a mobile terminal and a backend massive resource as claimed in claim 1, characterized in that: In step S5, the matching layer of the SDSMM model is the second layer of the model. The main function of the matching layer is to match the semantic representation of the text to be matched output by the representation layer and output a semantic vector with a fixed length. The matching layer of Sentence-BERT directly concatenates the semantic representations u, v and lu-v of the text and inputs them into the softmax classifier for classification, without making the semantic representation of the text fully interactive. The method uses a multi-layer perceptron in the matching layer to optimize the matching layer of the model, improve the degree of interaction between the semantic representations of the text, and output a matching vector with a fixed length of 768 dimensions. The matching layer first subtracts the absolute value of the semantic representations u and v of the text to obtain |uv; then concatenates u, v and |uv| into a 2304-dimensional vector; then the concatenated vector is input into the multi-layer perceptron for information interaction; the multi-layer perceptron performs nonlinear transformation and dimensionality reduction operations on the input concatenated vector, and outputs a 768-dimensional matching vector to represent the interactive information of the text semantic representation; The perceptron used in this method contains three hidden layers, and the number of neurons in each hidden layer is 2304, 1536, and 768 respectively; the input vector is calculated by the three hidden layers and finally outputs a matching vector with a dimension of 768; the three hidden layers of the perceptron all use the tanh function as the activation function.

8. A method for checking duplicate data between a mobile terminal and a backend massive resource as claimed in claim 7, characterized in that: The output calculation formula of the hidden layer I is: a l =σ(W l a l-1 +b l ), where a l represents the output vector of the lth layer; W l and b l Represent the weight matrix and bias vector of the lth layer respectively; a l-1 represents the output vector of the l-1th layer; σ represents the activation function, and this method uses the tanh function; The hidden layer calculation formula is as follows: First hidden layer: a l =σ(W l a0+b l ); Where a_0 = [u, v, |uv|] (u, v are the semantic representations of the text to be matched); Second hidden layer: a2=σ(W2a1+b2); The third hidden layer: a3=σ(W2a1+b3); Finally, the matching vector s output by the matching layer is: s=W3a2+b3 In this structure, a0 is a concatenated vector containing u, v and |uv|; after the concatenated vector is calculated through three hidden layers, a matching vector s with a length of 768 is finally obtained.

9. A method for checking duplicate data between a mobile terminal and a backend massive resource as claimed in claim 1, characterized in that: In step S6, the SDSMC data set has been constructed, but large-scale corpus in the field has not yet been obtained; web crawler technology is used in the duplicate checking and matching process, and a web crawler applet is implemented by using the crawler toolkit of the python programming language to crawl some public corpus related to computer-related courses, and the crawled data is saved locally; considering that crawling data is an I / O intensive task, multi-threading technology can be used to improve execution efficiency; this method is based on python's multiprocessing multi-threaded library, and uses a thread pool to improve the single-threaded crawler technology, which greatly improves the crawling efficiency of unsupervised corpus.

10. A method for checking duplicate data between a mobile terminal and a backend massive resource as claimed in claim 1, characterized in that: After the duplicate checking in step S7 is completed, it is necessary to perform the following: Key step records: record key steps in the duplicate checking process, such as text collection, data transmission, preprocessing, duplicate checking and matching; record timestamps, input and output data, and key information of execution results; Error and exception handling: record the errors and exceptions that occur during the duplicate checking process; Record the error type, occurrence time, error message, and handling measures; Performance monitoring: records the system's operating status and performance indicators, such as CPU and memory usage, and query response time; Monitor the performance trend of the system during long-term operation; Log storage and query: Store log data in a reliable log system, such as the ELK stack; provide log query and analysis functions to facilitate troubleshooting and performance optimization.