Batch content duplication detection method, system, device, terminal equipment and storage medium

Abstract

The present invention discloses a batch content duplication detection method, system, device, terminal device and storage medium, which obtains each detail of the current batch; generates a corresponding character string according to each detail, and calculates the information summary feature code of the character string; obtains the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; and queries and calls the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code, so as to obtain the duplication detection result. The present invention improves the detection efficiency of batch content duplication detection.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a batch content duplication detection method, system, device, terminal equipment and storage medium. Background Art

[0002] With the development of Internet technology, many industries need to process data in large quantities, such as education, public security, finance, taxation and other industries; large-scale product data and transaction data managed by large enterprises; large-scale common data generated in social management, urban construction and other aspects. As data is distributed in a discrete, non-uniform and non-standardized manner from different platforms, it is shifting towards large-scale centralization or integration of data. There is a wide range of practical needs to extract large data sets from massive data for rapid batch processing. In the scenario of distributed cluster batch processing, how to detect suspicious batches of large-scale and high-concurrency batch content has become an increasingly prominent issue in data batch processing projects.

[0003] Currently, the main method for duplicate detection of batch content is to match by file content. This method requires comparing the new batch content with the content of each historical batch. Therefore, each time a new batch is generated, it is necessary to perform a comparison operation with each historical file, which takes a long time to detect, thus affecting the user experience.

[0004] Therefore, it is necessary to propose a solution to improve the detection efficiency of batch content duplication detection. Summary of the invention

[0005] The main purpose of the present invention is to provide a batch content duplication detection method, system, device, terminal equipment and storage medium, aiming to improve the detection efficiency of batch content duplication detection.

[0006] To achieve the above object, the present invention provides a batch processing content duplication detection method, the batch processing content duplication detection method comprising:

[0007] Get the details of each item in the current batch;

[0008] Generate a corresponding character string according to each of the details, and calculate an information summary feature code of the character string;

[0009] Obtain the distributed lock corresponding to the information summary feature code in the remote dictionary cache service;

[0010] The remote dictionary cache service is queried and called according to the distributed lock to determine whether there is cache data with the value of the information digest feature code to obtain a duplicate detection result.

[0011] Optionally, after the step of obtaining the distributed lock corresponding to the information summary feature code in the remote dictionary cache service, the step further includes:

[0012] Determine whether a distributed lock corresponding to the information summary feature code in the remote dictionary cache service is obtained;

[0013] If the distributed lock corresponding to the information digest feature code in the remote dictionary cache service is not obtained, after waiting for a preset time, return to execute the step of obtaining the distributed lock corresponding to the information digest feature code in the remote dictionary cache service and subsequent steps.

[0014] Optionally, the step of querying and calling the remote dictionary cache service according to the distributed lock to determine whether there is cache data with a value of the information digest feature code to obtain a duplicate detection result includes:

[0015] Performing a query call on the remote dictionary cache service according to the distributed lock to obtain query data;

[0016] Determine whether there is cache data with the value of the information summary feature code in the query data;

[0017] If there is cache data with the value of the information summary feature code in the query data, the current batch is determined to be suspicious;

[0018] If the cache data whose value is the information digest feature code does not exist in the query data, the current batch is determined to be compliant, and the cache data whose value is the information digest feature code is inserted into the remote dictionary cache service;

[0019] The result of judging whether the current batch is suspicious or compliant is used as the weight determination result.

[0020] Optionally, after the step of querying and calling the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information digest feature code to obtain a duplicate detection result, the step further includes:

[0021] Releasing the distributed lock corresponding to the information summary feature code;

[0022] Return the duplicate determination result.

[0023] Optionally, the step of generating a corresponding character string according to each of the details and calculating an information summary feature code of the character string includes:

[0024] Extract the key elements of each of the details;

[0025] splicing the key elements in the same order to obtain a spliced ​​string;

[0026] Put the concatenated string into a concatenated string list;

[0027] Sorting the concatenated string list, and concatenating the sorted concatenated string list into a string;

[0028] Calculate the information summary feature code of the character string.

[0029] Optionally, the step of sorting the concatenated string list and concatenating the sorted concatenated string list into a string includes:

[0030] Sorting the concatenated string list according to a low-order-first string sorting algorithm;

[0031] Concatenate the sorted concatenated string list into one string one by one.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a batch processing content duplication detection system, the batch processing content duplication detection system comprising:

[0033] The external application is used to send a batch request to the batch processing service cluster, wherein the batch request includes details of the current batch;

[0034] The batch processing service cluster is used to obtain each detail of the current batch in the batch request sent by the external application; generate a corresponding character string according to each detail, and calculate the information summary feature code of the character string; obtain the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; query and call the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code to obtain a duplicate detection result;

[0035] The remote dictionary cache service is used to provide the batch processing service cluster with a distributed lock corresponding to the information digest feature code; provide query data according to the distributed lock corresponding to the information digest feature code, and store cache data with the value of the information digest feature code inserted by the batch processing service cluster.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a batch processing content duplication detection device, the batch processing content duplication detection device comprising:

[0037] The acquisition module is used to obtain the details of the current batch;

[0038] A generating and calculating module, used to generate a corresponding character string according to each of the details, and calculate an information summary feature code of the character string;

[0039] A lock acquisition module is used to acquire a distributed lock corresponding to the information summary feature code in the remote dictionary cache service;

[0040] The duplicate detection module is used to query and call the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code to obtain a duplicate detection result.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal device, which includes a memory, a processor, and a batch content duplication detection program stored in the memory and executable on the processor, and the batch content duplication detection program, when executed by the processor, implements the steps of the batch content duplication detection method as described above.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a batch content duplication detection program is stored. When the batch content duplication detection program is executed by a processor, the steps of the batch content duplication detection method as described above are implemented.

[0043] The embodiments of the present invention propose a batch content duplication detection method, device, terminal device and storage medium, which obtains each detail of the current batch; generates a corresponding character string according to each detail, and calculates the information summary feature code of the character string; obtains the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; and queries and calls the remote dictionary cache service according to the distributed lock to determine whether there is cached data with the value of the information summary feature code to obtain the duplication detection result. By generating a character string according to the current batch and calculating the information summary feature code, and then querying the remote dictionary cache service to obtain the duplication detection result, there is no need to match the content of the current batch with the content in the cache service one by one, thereby greatly improving the detection efficiency and thus improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a functional module diagram of a terminal device to which the batch content duplication detection device of the present invention belongs;

[0045] Figure 2 It is a flowchart of an exemplary embodiment of a method for batch content duplication detection of the present invention;

[0046] Figure 3 for Figure 2 A schematic diagram of a specific process of step S40 in the embodiment;

[0047] Figure 4 Schematic diagram of the working principle of the batch content duplication detection system in an embodiment of the present invention.

[0048] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0050] The main solution of the embodiment of the present invention is: by obtaining each detail of the current batch; generating a corresponding string according to each detail, and calculating the information summary feature code of the string; obtaining the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; querying and calling the remote dictionary cache service according to the distributed lock to determine whether there is cached data with the value of the information summary feature code to obtain the duplicate detection result. By generating a string according to the current batch and calculating the information summary feature code, and then querying the remote dictionary cache service to obtain the duplicate detection result, there is no need to match the content of the current batch with the content in the cache service one by one, thereby greatly improving the detection efficiency and thus improving the user experience.

[0051] Technical terms involved in the embodiments of the present invention:

[0052] MD5 (Message-Digest Algorithm): A widely used cryptographic hash function that produces a 128-bit (16-byte) hash value to ensure that information is transmitted intact and consistent.

[0053] Redis (Remote Dictionary Server): is an open source log-based, key-value database written in ANSI C, supporting the network, memory-based or persistent, and providing APIs in multiple languages.

[0054] Hashmap: It is a hash table that stores key-value mappings.

[0055] Since the main method for batch content duplication detection in the existing technology is to match according to file content, this method requires comparing the new batch content with the content of each historical batch. Therefore, each new batch needs to be compared with each historical file, which requires a longer detection time, thereby affecting the user experience.

[0056] The present invention provides a solution, which calculates the information digest of the key information of the current batch and then saves it to the distributed cache. The subsequent batch anti-duplicate inspection only needs to calculate the corresponding information digest, and then compare whether there is the same information digest in the cache to quickly detect whether the current file is suspicious. Since there is no need to match the current file content with the existing file content one by one, especially in the scenario of large data volume and high concurrency, the batch detection efficiency is greatly improved, and the user experience of merchants is enhanced.

[0057] The present invention solves the problem of detecting suspicious batches of large-scale, high-concurrency batch contents in the scenario of distributed cluster batch processing. The processing speed is greatly improved compared with the file content matching and duplicate checking. Originally, the new batch contents needed to be compared with each historical transaction content. Thus, each time a new transaction came, a comparison operation needed to be performed with each file. The improved solution performs MD5 on the historical transaction contents and stores them in a Hashmap data structure (Redis). It is necessary to perform an MD5 data summary on the new file contents, and then search for the new file contents in the Hashmap-like data through the summary. If the new file contents are found, they are considered suspicious. Thus, the entire matching operation only needs to compare the MD5 codes for new transactions. Since the search speed of HashMap is O(1), the speed of transaction duplicate checking remains unchanged regardless of the transaction volume.

[0058] Specifically, refer to Figure 1 , Figure 1 The functional module diagram of the terminal device to which the batch content duplication detection device of the present invention belongs is shown. The batch content duplication detection device can be a device that is independent of the terminal device and can perform batch content duplication detection, and can be carried on the terminal device in the form of hardware or software. The terminal device can be a smart mobile terminal with data processing function such as a mobile phone or a tablet computer, and can also be a fixed terminal device or server with data processing function.

[0059] In this embodiment, the terminal device to which the apparatus for batch content duplication determination belongs includes at least an output module 110 , a processor 120 , a memory 130 and a communication module 140 .

[0060] The memory 130 stores an operating system and a batch content duplication detection program. The batch content duplication detection device can obtain the details of the current batch, the corresponding character strings generated according to the details, the information summary feature code of the character string calculated, the distributed lock corresponding to the information summary feature code in the remote dictionary cache service, and query and call the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code, and store the obtained duplication detection results and other information in the memory 130; the output module 110 can be a display screen, etc. The communication module 140 can include a WIFI module, a mobile communication module, a Bluetooth module, etc., and communicate with an external device or server through the communication module 140.

[0061] The batch content duplication detection program in the memory 130 implements the following steps when executed by the processor:

[0062] Get the details of each item in the current batch;

[0063] Generate a corresponding character string according to each of the details, and calculate an information summary feature code of the character string;

[0064] Obtain the distributed lock corresponding to the information summary feature code in the remote dictionary cache service;

[0065] The remote dictionary cache service is queried and called according to the distributed lock to determine whether there is cache data with the value of the information digest feature code to obtain a duplicate detection result.

[0066] Furthermore, when the batch content duplication detection program in the memory 130 is executed by the processor, the following steps are also implemented:

[0067] Determine whether a distributed lock corresponding to the information summary feature code in the remote dictionary cache service is obtained;

[0068] If the distributed lock corresponding to the information digest feature code in the remote dictionary cache service is not obtained, after waiting for a preset time, return to execute the step of obtaining the distributed lock corresponding to the information digest feature code in the remote dictionary cache service and subsequent steps.

[0069] Furthermore, when the batch content duplication detection program in the memory 130 is executed by the processor, the following steps are also implemented:

[0070] Performing a query call on the remote dictionary cache service according to the distributed lock to obtain query data;

[0071] Determine whether there is cache data with the value of the information summary feature code in the query data;

[0072] If there is cache data with the value of the information summary feature code in the query data, the current batch is determined to be suspicious;

[0073] If the cache data whose value is the information digest feature code does not exist in the query data, the current batch is determined to be compliant, and the cache data whose value is the information digest feature code is inserted into the remote dictionary cache service;

[0074] The result of judging whether the current batch is suspicious or compliant is used as the weight determination result.

[0075] Furthermore, when the batch content duplication detection program in the memory 130 is executed by the processor, the following steps are also implemented:

[0076] Releasing the distributed lock corresponding to the information summary feature code;

[0077] Return the duplicate determination result.

[0078] Furthermore, when the batch content duplication detection program in the memory 130 is executed by the processor, the following steps are also implemented:

[0079] Extract the key elements of each of the details;

[0080] splicing the key elements in the same order to obtain a spliced ​​string;

[0081] Put the concatenated string into a concatenated string list;

[0082] Sorting the concatenated string list, and concatenating the sorted concatenated string list into a string;

[0083] Calculate the information summary feature code of the character string.

[0084] Furthermore, when the batch content duplication detection program in the memory 130 is executed by the processor, the following steps are also implemented:

[0085] Sorting the concatenated string list according to a low-order-first string sorting algorithm;

[0086] Concatenate the sorted concatenated string list into one string one by one.

[0087] This embodiment adopts the above scheme, specifically by obtaining each detail of the current batch; generating a corresponding character string according to each detail, and calculating the information summary feature code of the character string; obtaining the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; querying and calling the remote dictionary cache service according to the distributed lock, and judging whether there is cache data with the value of the information summary feature code to obtain the duplicate detection result. By generating a character string according to the current batch and calculating the information summary feature code, and then querying the remote dictionary cache service to obtain the duplicate detection result, there is no need to match the content of the current batch with the content in the cache service one by one, thereby greatly improving the detection speed and thus improving the user experience.

[0088] Based on the above terminal device architecture but not limited to the above architecture, an embodiment of the method of the present invention is proposed.

[0089] The execution subject of the method of this embodiment may be a batch processing content duplication detection device or terminal equipment, etc. This embodiment takes a batch processing content duplication detection device as an example.

[0090] Reference Figure 2 , Figure 2 The flowchart of an exemplary embodiment of a batch processing content duplication detection method of the present invention is shown in FIG. The batch processing content duplication detection method comprises:

[0091] Step S10, obtaining the details of each item of the current batch;

[0092] In the actual application of data warehouses and large and medium-sized databases, there are many industries that need to process data in large batches. For example, in general, institutions are set up according to administrative regions, and businesses are divided according to service types, but the development trend and direction is to constantly adapt to the requirements of data concentration in today's information age, so data must be managed in a fully centralized manner. The data that are managed in a fully centralized manner include: summary data at the national, provincial, ministerial, and departmental levels; data from industries such as education, public security, finance, and taxation; huge product data and transaction data managed by large enterprises; large-scale common data generated in social management, urban construction, etc. In the process of batch processing data, how to detect suspicious batches of large-scale and high-concurrency batch contents has become an increasingly prominent issue in data batch processing projects. For example, for banking business, in order to meet the requirements of fund supervision, it is necessary to quickly find duplicate batches from historical batches, and intercept suspicious batches to avoid financial losses to merchants. Therefore, in the scenario of distributed cluster batch processing, it is necessary to first obtain each detail of the current batch and then extract the key elements.

[0093] Step S20, generating a corresponding character string according to each of the details, and calculating an information summary feature code of the character string;

[0094] Specifically, after obtaining the details of the current batch, it is necessary to extract the key elements of each detail, and then splice the key elements of each detail into a splicing string in the same splicing order, and then put the obtained splicing string into the splicing string list.

[0095] Furthermore, the obtained concatenated string list is sorted to ensure that the sorting algorithm of each batch detail concatenated string is the same. In this embodiment, the sorting is performed according to the string sorting algorithm of low-order first. Then the sorted concatenated string list is concatenated into a string one by one, thus avoiding the problem that the same batch cannot be judged as duplicate due to the disorder of the detail order.

[0096] After generating the corresponding character string for each detail, it is necessary to further calculate the MD5 information digest feature code of the character string to obtain a distributed lock called the MD5 value.

[0097] Step S30, obtaining a distributed lock corresponding to the information summary feature code in the remote dictionary cache service;

[0098] During the batch content duplication detection process, it is necessary to perform query calls and insert calls on the Redis cache service. However, performing compound operations on Redis will cause multi-threading problems, so it is necessary to first obtain a distributed lock called the MD5 value.

[0099] After obtaining the step of obtaining the distributed lock named MD5 value, it is necessary to judge the acquisition result. If the acquisition of the distributed lock named MD5 value fails, the step of obtaining the distributed lock named MD5 value is re-executed after waiting for a preset time. In this embodiment, the preset waiting time is 2 seconds. After waiting for 2 seconds, the step of obtaining the distributed lock named MD5 value is re-executed to ensure successful acquisition.

[0100] Step S40: query and call the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information digest feature code to obtain a duplicate detection result.

[0101] After obtaining the distributed lock, the Redis query call is performed through the MD5 code to determine whether there is cached data with the MD5 value. If it already exists, the current batch is considered duplicate or suspicious. After obtaining the duplicate detection result, the distributed lock named MD5 value needs to be released so that subsequent batch requests with the same MD5 code can obtain the distributed lock, and then return the content duplicate detection result. If it is found during the query call that there is no cached data with the MD5 value, the Key / Value data with the MD5 code as the key is inserted into the Redis cache service, and the current batch is determined to be compliant, and then the distributed lock named MD5 value is released in the same way, so that subsequent batch requests with the same MD5 code can obtain the distributed lock, and then return the content duplicate detection result.

[0102] In this embodiment, by obtaining each detail of the current batch; generating a corresponding string according to each detail, and calculating the information summary feature code of the string; obtaining the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; querying and calling the remote dictionary cache service according to the distributed lock, and judging whether there is cache data with the value of the information summary feature code, so as to obtain the duplicate detection result. By generating a string according to the current batch and calculating the information summary feature code, and then querying the remote dictionary cache service to obtain the duplicate detection result, there is no need to match the content of the current batch with the content in the cache service one by one, thereby greatly improving the detection speed and thus improving the user experience.

[0103] Reference Figure 3 , Figure 3 for Figure 2 The specific flow chart of step S40 in the embodiment. Figure 2 In the embodiment shown, in this embodiment, the above step S40 includes:

[0104] Step S401, querying and calling the remote dictionary cache service according to the distributed lock to obtain query data;

[0105] After acquiring the distributed lock, a Redis query call is made through the MD5 code to obtain query data from the Redis cache service to determine whether there is cache data with the MD5 value.

[0106] Step S402, determining whether there is cache data with a value equal to the information summary feature code in the query data;

[0107] The obtained query data is judged to determine whether there is cache data with the value of MD5.

[0108] Step S403, if there is cache data with the value of the information summary feature code in the query data, then the current batch is determined to be suspicious;

[0109] If the query finds that cached data with the value of MD5 already exists, the current batch is considered duplicate or suspicious. After obtaining the duplicate detection result, the distributed lock named MD5 value needs to be released so that subsequent batch requests with the same MD5 code can obtain the distributed lock.

[0110] Step S404, if the cache data whose value is the information digest feature code does not exist in the query data, the current batch is determined to be compliant, and the cache data whose value is the information digest feature code is inserted into the remote dictionary cache service;

[0111] If it is found during the query call that no cached data with the MD5 value exists, the Key / Value data with the MD5 code as the key is inserted into the Redis cache service, and the current batch is determined to be compliant. Then, the distributed lock named as the MD5 value is released in the same way, so that subsequent batch requests with the same MD5 code can obtain the distributed lock.

[0112] Step S405, determining whether the current batch is suspicious or compliant is used as the weight determination result.

[0113] The duplicate detection result is obtained based on the suspicious or compliant results of the current batch, and the duplicate detection result is returned so that users can quickly find duplicate batches and then intercept suspicious batches.

[0114] This embodiment uses the above scheme, specifically by querying and calling the remote dictionary cache service according to the distributed lock to obtain query data; judging whether there is cache data with the value of the information digest feature code in the query data; if there is cache data with the value of the information digest feature code in the query data, then the current batch is judged to be suspicious; if there is no cache data with the value of the information digest feature code in the query data, then the current batch is judged to be compliant, and the cache data with the value of the information digest feature code is inserted into the remote dictionary cache service; the result of judging whether the current batch is suspicious or compliant is used as the duplicate judgment result. By querying in the remote dictionary cache service to obtain the duplicate judgment result, there is no need to match the content of the current batch with the content in the cache service one by one, thereby greatly improving the detection speed and thus improving the user experience.

[0115] In addition, an embodiment of the present invention further provides a batch content duplication detection system, the batch content duplication detection system comprising:

[0116] The external application is used to send a batch request to the batch processing service cluster, wherein the batch request includes details of the current batch;

[0117] The batch processing service cluster is used to obtain each detail of the current batch in the batch request sent by the external application; generate a corresponding character string according to each detail, and calculate the information summary feature code of the character string; obtain the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; query and call the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code to obtain a duplicate detection result;

[0118] The remote dictionary cache service is used to provide the batch processing service cluster with a distributed lock corresponding to the information digest feature code; provide query data according to the distributed lock corresponding to the information digest feature code, and store cache data with the value of the information digest feature code inserted by the batch processing service cluster.

[0119] Specifically, the batch content duplication detection system detects duplicate content in each batch of requests in the following process:

[0120] (1) Concatenate the key elements of each detail of the current batch in the same order, ensuring that the concatenation order of the key elements of each batch of details is the same, and then put the concatenated strings into a list;

[0121] (2) Sort the concatenated string list using a low-order-first string sorting algorithm to ensure that the sorting algorithm of each batch of detailed concatenated strings is the same, and then concatenate them one by one into a string, thereby avoiding the problem of not being able to judge duplicates due to the disorder of the detailed order of the same batch;

[0122] (3) Calculate the MD5 code of the string;

[0123] (4) Obtain a Redis distributed lock named MD5 code. As shown in steps (6) and (7), duplicate detection requires a Redis query call and an insert call. For Redis compound operations, it will cause multi-threading problems, so it is necessary to first obtain a distributed lock named MD5 value;

[0124] (5) If the acquisition of the Redis distributed lock named MD5 code fails, wait for 2 seconds and then jump to step (4);

[0125] (6) After acquiring the distributed lock, perform a Redis query call using the MD5 code to determine whether there is cache data with the MD5 value. If it already exists, the current batch is considered duplicate or suspicious, and jump to step (8)

[0126] (7) If there is no cache data with the MD5 value, insert the Key / Value data with the MD5 code as the key into the Redis cache, and the current batch is considered compliant.

[0127] (8) Release the Redis distributed lock named MD5 code so that subsequent batch requests with the same MD5 code can obtain the lock, and then perform the system logic of step (5);

[0128] (9) Return the duplicate detection result.

[0129] Reference Figure 4 , Figure 4 FIG. 1 is a schematic diagram of the working principle of the batch content duplication detection system in an embodiment of the present invention. Figure 4 As shown in the figure, when two batch requests with the same content are sent simultaneously from an external application, the batch processing service cluster first obtains the string corresponding to the first batch request, and then calculates the MD5 code of the string, and then obtains the Redis distributed lock named MD5 code from the Redis cache service. At this time, the other batch request with the same content needs to wait for the lock. When the first batch request uses the MD5 code to make a Redis query call, after determining whether there is cache data with the value of MD5, the lock is released so that the other batch request can obtain the lock. In addition, in order to ensure that the MD5 cache data is not lost, resulting in a failure in duplicate detection, Redis persistence should be done, so that the MD5 data can be restored in time after the machine is restarted.

[0130] The embodiment of the present invention calculates the information summary of the key information of the batch and then saves it to the distributed cache. The subsequent batches only need to calculate their own information summary and then compare whether there is the same information summary in the cache to quickly detect whether the current file is suspicious. Since there is no need to match the current file content with the existing file content one by one, especially in scenarios with large data volumes and high concurrency, the batch detection speed is greatly improved and the user experience of merchants is enhanced. It solves the problem of detecting suspicious batches of large batches with high concurrency in the scenario of distributed cluster batch processing, and has a greatly improved processing speed compared to checking for duplicates by matching file content.

[0131] In addition, an embodiment of the present invention further provides a batch processing content duplication detection device, the batch processing content duplication detection device comprising:

[0132] The acquisition module is used to obtain the details of the current batch;

[0133] A generating and calculating module, used to generate a corresponding character string according to each of the details, and calculate an information summary feature code of the character string;

[0134] A lock acquisition module is used to acquire a distributed lock corresponding to the information summary feature code in the remote dictionary cache service;

[0135] The duplicate detection module is used to query and call the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code to obtain a duplicate detection result.

[0136] The principle and implementation process of batch content duplication detection implemented in this embodiment can be referred to the above embodiments and will not be described in detail here.

[0137] In addition, an embodiment of the present invention also proposes a terminal device, which includes a memory, a processor, and a batch content duplication detection program stored in the memory and executable on the processor, and the batch content duplication detection program, when executed by the processor, implements the steps of the batch content duplication detection method as described above.

[0138] Since all the technical solutions of all the aforementioned embodiments are adopted when this batch content duplication judgment program is executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0139] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a batch content duplication detection program is stored. When the batch content duplication detection program is executed by a processor, the steps of the batch content duplication detection method described above are implemented.

[0140] Since all the technical solutions of all the aforementioned embodiments are adopted when this batch content duplication judgment program is executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0141] Compared with the prior art, the batch content duplication detection method, system, device, terminal device and storage medium proposed in the embodiment of the present invention obtains each detail of the current batch; generates a corresponding character string according to each detail, and calculates the information summary feature code of the character string; obtains the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; and queries and calls the remote dictionary cache service according to the distributed lock to determine whether there is cached data with the value of the information summary feature code to obtain the duplication detection result. By generating a character string according to the current batch and calculating the information summary feature code, and then querying the remote dictionary cache service to obtain the duplication detection result, there is no need to match the content of the current batch with the content in the cache service one by one, thereby greatly improving the detection speed and thus improving the user experience.

[0142] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0143] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to execute the method of each embodiment of the present application.

[0145] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A batch content duplication detection method, characterized in that: The batch content duplication detection method comprises the following steps: Get the details of each item in the current batch; Generate a corresponding character string according to each of the details, and calculate an information summary feature code of the character string; Obtain the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; The remote dictionary cache service is queried and called according to the distributed lock to determine whether there is cache data with the value of the information summary feature code to obtain a duplicate detection result; After the step of obtaining the distributed lock corresponding to the information summary feature code in the remote dictionary cache service, the following step is further included: Determine whether a distributed lock corresponding to the information summary feature code in the remote dictionary cache service is obtained; If the distributed lock corresponding to the information summary feature code in the remote dictionary cache service is not obtained, then after waiting for a preset time, return to execute the step of obtaining the distributed lock corresponding to the information summary feature code in the remote dictionary cache service and subsequent steps; The step of querying and calling the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information digest feature code to obtain a duplicate detection result includes: Performing a query call on the remote dictionary cache service according to the distributed lock to obtain query data; Determine whether there is cache data with the value of the information summary feature code in the query data; If there is cache data with the value of the information summary feature code in the query data, the current batch is determined to be suspicious; If the cache data whose value is the information digest feature code does not exist in the query data, the current batch is determined to be compliant, and the cache data whose value is the information digest feature code is inserted into the remote dictionary cache service; The result of judging the current batch to be suspicious or compliant is used as the weight determination result; The step of generating a corresponding character string according to each of the details and calculating the information summary feature code of the character string includes: Extract the key elements of each of the details; splicing the key elements in the same order to obtain a spliced ​​string; Put the concatenated string into a concatenated string list; Sorting the concatenated string list, and concatenating the sorted concatenated string list into a string; Calculate the information summary feature code of the character string.

2. The batch content duplication detection method according to claim 1, characterized in that: After the step of querying and calling the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code to obtain a duplicate detection result, the following step is also included: Releasing the distributed lock corresponding to the information summary feature code; Return the duplicate determination result.

3. The batch content duplication detection method according to claim 1, characterized in that: The step of sorting the concatenated string list and concatenating the sorted concatenated string list into a string comprises: Sorting the concatenated string list according to a low-order-first string sorting algorithm; Concatenate the sorted concatenated string list into one string one by one.

4. A batch content duplication detection system, characterized in that: The batch content duplication detection system comprises: The external application is used to send a batch request to the batch processing service cluster, wherein the batch request includes details of the current batch; The batch processing service cluster is used to obtain each detail of the current batch in the batch request sent by the external application; generate a corresponding character string according to each detail, and calculate the information summary feature code of the character string; obtain the distributed lock corresponding to the information summary feature code in the remote dictionary cache service; query and call the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code to obtain a duplicate detection result; A remote dictionary cache service is used to provide the batch processing service cluster with a distributed lock corresponding to the information digest feature code; provide query data according to the distributed lock corresponding to the information digest feature code, and store cache data whose value is the information digest feature code inserted by the batch processing service cluster; After obtaining the distributed lock corresponding to the information digest feature code in the remote dictionary cache service, the method further includes: determining whether the distributed lock corresponding to the information digest feature code in the remote dictionary cache service is obtained; if the distributed lock corresponding to the information digest feature code in the remote dictionary cache service is not obtained, after waiting for a preset time, returning to the step of obtaining the distributed lock corresponding to the information digest feature code in the remote dictionary cache service and subsequent steps; The querying and calling the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code to obtain a duplicate determination result includes: Performing a query call on the remote dictionary cache service according to the distributed lock to obtain query data; Determine whether there is cache data with the value of the information summary feature code in the query data; If there is cache data with the value of the information summary feature code in the query data, the current batch is determined to be suspicious; If the cache data whose value is the information digest feature code does not exist in the query data, the current batch is determined to be compliant, and the cache data whose value is the information digest feature code is inserted into the remote dictionary cache service; The result of judging the current batch to be suspicious or compliant is used as the weight determination result; The step of generating a corresponding character string according to each of the details and calculating the information summary feature code of the character string includes: Extract the key elements of each of the details; splicing the key elements in the same order to obtain a spliced ​​string; Put the concatenated string into a concatenated string list; Sorting the concatenated string list, and concatenating the sorted concatenated string list into a string; Calculate the information summary feature code of the character string.

5. A batch content duplication detection device, characterized in that: The batch processing content duplication detection device comprises: The acquisition module is used to obtain the details of the current batch; A generating and calculating module, used to generate a corresponding character string according to each of the details, and calculate an information summary feature code of the character string; A lock acquisition module is used to acquire a distributed lock corresponding to the information summary feature code in the remote dictionary cache service; A duplicate detection module, used for querying and calling the remote dictionary cache service according to the distributed lock, and determining whether there is cache data with the value of the information summary feature code, so as to obtain a duplicate detection result; After obtaining the distributed lock corresponding to the information digest feature code in the remote dictionary cache service, the method further includes: determining whether the distributed lock corresponding to the information digest feature code in the remote dictionary cache service is obtained; if the distributed lock corresponding to the information digest feature code in the remote dictionary cache service is not obtained, after waiting for a preset time, returning to the step of obtaining the distributed lock corresponding to the information digest feature code in the remote dictionary cache service and subsequent steps; The querying and calling the remote dictionary cache service according to the distributed lock to determine whether there is cache data with the value of the information summary feature code to obtain a duplicate determination result includes: Performing a query call on the remote dictionary cache service according to the distributed lock to obtain query data; Determine whether there is cache data with the value of the information summary feature code in the query data; If there is cache data with the value of the information summary feature code in the query data, the current batch is determined to be suspicious; If the cache data whose value is the information digest feature code does not exist in the query data, the current batch is determined to be compliant, and the cache data whose value is the information digest feature code is inserted into the remote dictionary cache service; The result of judging the current batch to be suspicious or compliant is used as the weight determination result; The step of generating a corresponding character string according to each of the details and calculating the information summary feature code of the character string includes: Extract the key elements of each of the details; splicing the key elements in the same order to obtain a spliced ​​string; Put the concatenated string into a concatenated string list; Sorting the concatenated string list, and concatenating the sorted concatenated string list into a string; Calculate the information summary feature code of the character string.

6. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a batch content duplication detection program stored in the memory and executable on the processor. When the batch content duplication detection program is executed by the processor, the steps of the batch content duplication detection method as described in any one of claims 1 to 3 are implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a batch content duplication detection program, and when the batch content duplication detection program is executed by the processor, the steps of the batch content duplication detection method according to any one of claims 1 to 3 are implemented.

Images