Project sentence duplicate removal method
By combining online APIs and hash algorithms, automatic deduplication of manuscripts within the project was achieved, solving the problem of translators translating repeatedly and improving translation efficiency and consistency.
Patent Information
- Application Number
- CN202511798651.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-01-20
AI Technical Summary
In the traditional translation industry, each task within a project requires separate analysis and deduplication of the manuscript, resulting in translators performing repetitive translation work and lacking consistency.
A sentence deduplication method is adopted, which inputs the task manuscript through an online API, calculates the hash value of the sentence using a hash algorithm, stores it in a dictionary for deduplication judgment, and outputs the deduplication result.
It automatically removes duplicates from the manuscripts for each task within the project, reducing repetitive translations, improving response speed, and ensuring the consistency and reliability of translations.
Smart Images
Figure CN121365660A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of translation, in particular to a project sentence deduplication method. BACKGROUND
[0002] In the traditional translation industry, each manuscript in each task in a project is analyzed and deduplicated individually, or not deduplicated. Here, the translator needs to invest in repetitive translation work. If a method can store all manuscripts in all tasks in a project for deduplication, it will reduce the repetitive translation of the translator to some extent and maintain the consistency of translation. SUMMARY
[0003] The present application aims at the deficiencies of the prior art, and provides a project sentence deduplication method, which can deduplicate each manuscript in each task in a project.
[0004] To achieve the above object, the present application adopts the following technical scheme: The present application provides a project sentence deduplication method, comprising the following steps: S1, inputting task manuscripts of a project; S2, storing sentences; S3, deduplication judgment; S4, outputting deduplication results.
[0005] Further, the S1 specifically comprises: inputting task manuscripts of a project through an online API, which can deliver multiple task manuscripts.
[0006] Further, the S2 specifically comprises: sequentially reading each task manuscript to obtain the content of each manuscript, and the delivered manuscript has been processed by sentence breaking in advance, and each sentence has a corresponding sentence id and original text.
[0007] Further, the S2 further comprises: reading sentences line by line, using a hash algorithm to calculate the hash value of each sentence, and using the hash of each sentence as the key of a dictionary dict, and the dictionary value is a dictionary containing the information of the sentence itself.
[0008] Further, the S3 comprises: S301, calculating the hash value of the sentence by using a hash algorithm; S302, requesting the sentence dictionary dict of the stored sentence by using the hash value, if there is a corresponding value, it indicates that the sentence has a repeated sentence stored in the dictionary dict, and taking the sentence id in the stored sentence information of the sentence dictionary dict as the deduplicated sentence id of the input sentence; if there is no corresponding value, it indicates that the sentence has no repeated sentence at present, and the sentence is stored in the sentence dictionary dict.
[0009] Further, the S4 is specifically: according to the deduplication judgment of the S3, the sentence is stored once according to each sentence, and the deduplication is performed synchronously; if the sentence has a repeated sentence id, the repeated sentence id is saved to a separate deduplication result dictionary, and the request sentence id is used as the key, and the value is the repeated sentence id, finally the deduplication result dictionary is converted into json and returned to the api request result.
[0010] The method uses a set of deduplication logic to deduplicate each manuscript in each task in the project, can timely deduplicate newly added manuscripts, has fast response speed, and guarantees reliability by relying on the project manuscripts. BRIEF DESCRIPTION OF DRAWINGS
[0011] Figure 1 A flowchart of a project sentence deduplication method. DETAILED DESCRIPTION
[0012] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.
[0013] Please refer to Figure 1 A project sentence deduplication method, comprising the following steps: S1, inputting task manuscripts of a project; S2, storing sentences; S3, deduplication judgment; S4, outputting deduplication results.
[0014] The S1 is specifically: inputting task manuscripts of a project through an online API, and multiple task manuscripts can be transmitted.
[0015] The S2 is specifically: reading each task manuscript to obtain the content of each manuscript in sequence, the transmitted manuscript has been processed by sentence breaking, and each sentence has a corresponding sentence id and sentence original text.
[0016] The S2 further comprises: reading the sentence line by line, calculating the hash value of each sentence using a hash algorithm, using the hash of each sentence as the key of the dictionary dict, and the dictionary value is a dictionary containing the information of the sentence itself.
[0017] The S3 comprises: S301, calculating the hash value of the sentence by using a hash algorithm; S302, request the sentence dictionary dict of the previous step to store the sentence with the hash value, if there is a corresponding value, it means that the sentence has a repeated sentence stored in the sentence dictionary dict, take the sentence id in the sentence information stored in the sentence dictionary dict as the de-duplication sentence id of the input sentence and return; if there is no corresponding value, it means that the sentence has no repeated sentence at present, and store the sentence to the sentence dictionary dict.
[0018] The S4 is specifically: according to the de-duplication judgment of the S3, store the sentence according to each sentence once, and synchronize the de-duplication; if the sentence has a repeated sentence id, save it to a separate de-duplication result dictionary, take the request sentence id as the key, and the value is the repeated sentence id, finally convert the de-duplication result dictionary into json and return the api request result.
[0019] The above-mentioned embodiments only express the implementation of the present application, and the description is more specific and detailed, but it cannot be understood as the limitation of the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be the appended claims.
Claims
1. A method for project sentence deduplication, the method comprising: The method comprises the following steps: S1, inputting a task manuscript of a project; S2, storing a sentence; S3, judging duplication; S4, outputting a result of deduplication.
2. The method of claim 1, wherein: The S1 is specifically: inputting a task manuscript of a project through an online API, and multiple task manuscripts can be transmitted.
3. The method of claim 1, wherein: The S2 is specifically: reading each task manuscript in sequence to obtain the content of each manuscript, the transmitted manuscript is a manuscript after being processed by a sentence breaking, and each sentence has a corresponding sentence id and a sentence original text.
4. The method of claim 3, wherein: The S2 further comprises: reading the sentence in a row, calculating a hash value of each sentence by using a hash algorithm, taking the hash of each sentence as a key of a dictionary dict, and taking a dictionary value as a dictionary containing information of the sentence itself.
5. The method of claim 1, wherein: The S3 comprises: S301, calculating a hash value of a sentence by using a hash algorithm; S302, requesting a sentence dictionary dict of the step of storing a sentence by using the hash value, if there is a corresponding value, it indicates that there is a repeated sentence stored in the dictionary dict, and taking a sentence id in the stored sentence information of the sentence dictionary dict as a deduplication sentence id of an input sentence is returned; if there is no corresponding value, it indicates that there is no repeated sentence at present, and the sentence is stored in the sentence dictionary dict.
6. The method of claim 1, wherein: The S4 is specifically: according to the deduplication judgment of the S3, storing the sentence once according to each sentence, and synchronously performing deduplication; if the sentence has a repeated sentence id, it is saved to a separate deduplication result dictionary, taking a request sentence id as a key, and a repeated sentence id as a value, finally, the deduplication result dictionary is converted into a json to return an api request result.