Expandable data desensitization method and system based on task queue

Abstract

The invention discloses an expandable data desensitization method and system based on a task queue, and the method comprises the steps: extracting data formats which can be processed in batches from original data formats of target files for different types of target files; based on the data features of the target file and the converted data format, matching corresponding data detection and data desensitization processing logic; and based on the data detection and data desensitization processing logic of the target file, executing a data detection and data desensitization task through a multi-queue task distribution and parallel processing mode of the task queue. Specific detection and processing rules are allowed to be defined for different types of files, so that the flexibility and adaptability of the desensitization process are improved. The data desensitization method and the data desensitization system have strong parallel processing capability, can deal with large-scale data desensitization tasks, remarkably improve processing efficiency, shorten overall response time, ensure efficient desensitization and maintain data integrity and safety at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of data desensitization, and in particular to an extensible data desensitization method and system based on a task queue. Background Art

[0002] Data desensitization technology ensures the security of data in various operations by protecting sensitive data, complying with legal and regulatory requirements, improving data security, and protecting commercial secrets. Current data desensitization technology mainly relies on technical means such as keyword matching algorithms, artificial intelligence technology, and data encryption methods to identify and shield sensitive parts of the data; and relies on rules and regulations such as information import and export approval systems to further ensure data security and ultimately achieve all-round data security.

[0003] With the advancement of the current digitalization process and the continuous expansion of enterprise scale, the number of digital assets that enterprises need to protect is also increasing. In the early days, data desensitization only needed to process text files such as service logs and structured database table files, but now it is also necessary to support the processing of PDF files, multimedia files, drawing design files, and various unstructured database files. This change has brought higher requirements for data desensitization technology, especially when faced with complex and diverse file formats, the limitations of traditional data desensitization methods have gradually emerged. In addition, with the rapid development of business, the amount of files involved in processing has increased exponentially, which has put higher requirements on the efficiency of data desensitization.

[0004] Currently, the main data desensitization development framework is Microsoft's Presidio. Presidio supports the processing of two data types: text and images, but there are still significant limitations in practical applications. First, Presidio lacks the ability to expand the processing of multiple file types and cannot effectively deal with file formats other than text and images, such as PDF, multimedia files, and various unstructured database files; second, Presidio's parallel capabilities and monitoring capabilities are relatively limited, and it is difficult to deploy in a distributed manner and track and manage the execution status of each desensitization task in real time in large-scale data processing scenarios. The lack of this ability to conduct partial trial operation and real-time monitoring and management has resulted in Presidio being unable to provide sufficient performance guarantees and task visibility when processing complex and diverse data types and responding to growing performance requirements.

[0005] Therefore, improving the processing capabilities of the data desensitizing framework, especially the customized processing capabilities for more file types, has become an urgent need. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a scalable data desensitization method and system based on task queues, which allows specific detection and processing rules to be defined for different types of files, thereby improving the flexibility and adaptability of the desensitization process. The data desensitization method and system of the present invention have powerful parallel processing capabilities, can cope with large-scale data desensitization tasks, significantly improve processing efficiency, shorten overall response time, and ensure that the integrity and security of the data are maintained while desensitizing efficiently.

[0007] The technical solution adopted by the present invention is as follows:

[0008] On the one hand, the present invention proposes a scalable data desensitization method based on a task queue, comprising:

[0009] For different types of target files, extract the data format that can be processed in batches from the original data format of the target file; match the corresponding data detection and data desensitization processing logic based on the data characteristics of the target file and the converted data format;

[0010] Based on the data detection and data desensitization processing logic of the target file, data detection and data desensitization tasks are executed through multi-queue task distribution and parallel processing of the task queue.

[0011] Furthermore, the data detection and data desensitization processing logic is matched based on the data characteristics of the target file and the converted data format, including: setting a detection component according to the data characteristics of the target file and the converted data format, the detection component includes a detector and a processor, the detector stores a detection process function, the processor stores a desensitization process function, and part of the output of the detector is part of the input of the processor.

[0012] Furthermore, for target files of text type, the detection component can output a tuple in a first preset format through the detector, and combine it with the original text as the input of the processor, wherein the tuple in the first preset format includes a starting point, an end point and a credibility; the processor can perform corresponding mask processing on the detection results whose credibility is greater than a set threshold, and finally output the masked text content.

[0013] Furthermore, for target files of image type, the detection component can output a tuple in a second preset format through the detector, and combine it with the original image as the input of the processor, wherein the tuple in the second preset format includes the horizontal coordinate of the starting point, the width of the detection frame, the height of the detection frame and the credibility; the processor can perform corresponding rectangular mosaic processing on the detection results whose credibility is greater than the set threshold, and finally output the processed image content.

[0014] Furthermore, the data detection and data desensitization processing logic based on the target file executes the data detection and data desensitization tasks through multi-queue task distribution and parallel processing of the task queue, including:

[0015] After the detection components are set according to the data characteristics of the target file and the converted data format, a unique ID is assigned to each detection component through the registration center unit, and the unique ID and the detection component are transmitted to the publishing unit;

[0016] The publishing unit reads the load function of the detection component through the task packaging publisher and submits it to the Celery task queue as a task; when the Celery task queue returns the task ID, the publishing unit binds the unique ID and the task ID one by one through the publishing result storage; after the publishing unit completes the binding, it transmits the binding result to the subscription center unit;

[0017] The subscription center unit detects the execution status of the tasks through a monitor according to the task ID polling until all tasks are completed; after all tasks are completed, the subscription center unit determines the success status of the tasks through an assembler, and assembles and outputs the success results in a dictionary format.

[0018] On the other hand, the present invention proposes a scalable data desensitization system based on a task queue, comprising:

[0019] The first processing module is configured to extract data formats that can be processed in batches from the original data formats of target files of different types; and match corresponding data detection and data desensitization processing logic based on the data features of the target files and the converted data formats;

[0020] The second processing module is configured to perform data detection and data desensitization tasks based on the data detection and data desensitization processing logic of the target file through multi-queue task distribution and parallel processing of the task queue.

[0021] Furthermore, in the first processing module, based on the data characteristics of the target file and the converted data format, the corresponding data detection and data desensitization processing logic is matched, including: setting a detection component according to the data characteristics of the target file and the converted data format, the detection component includes a detector and a processor, the detector stores a detection process function, the processor stores a desensitization process function, and part of the output of the detector is part of the input of the processor.

[0022] Furthermore, for target files of text type, the detection component can output a tuple in a first preset format through the detector, and combine it with the original text as the input of the processor, wherein the tuple in the first preset format includes a starting point, an end point and a credibility; the processor can perform corresponding mask processing on the detection results whose credibility is greater than a set threshold, and finally output the masked text content.

[0023] Furthermore, for target files of image type, the detection component can output a tuple in a second preset format through the detector, and combine it with the original image as the input of the processor, wherein the tuple in the second preset format includes the horizontal coordinate of the starting point, the width of the detection frame, the height of the detection frame and the credibility; the processor can perform corresponding rectangular mosaic processing on the detection results whose credibility is greater than the set threshold, and finally output the processed image content.

[0024] Furthermore, the second processing module includes a registration center unit, a publishing unit and a subscription center unit, wherein:

[0025] The registration center unit is configured to assign a unique ID to the detection component and transmit the unique ID and the detection component to the publishing unit; the detection component is set according to the data characteristics of the target file and the converted data format;

[0026] The publishing unit is configured to read the load function of the detection component through the task packaging publisher and submit it to the Celery task queue as a task; when the Celery task queue returns the task ID, the publishing unit binds the unique ID and the task ID one by one through the publishing result storage; after the publishing unit completes the binding, the binding result is transmitted to the subscription center unit;

[0027] The subscription center unit is configured to detect the execution status of the tasks through a monitor according to the task ID polling until all tasks are completed; after all tasks are completed, the subscription center unit determines the success status of the tasks through an assembler, and assembles and outputs the success results in a dictionary format.

[0028] The beneficial effects of the present invention are:

[0029] 1. Flexible support for processing of various file types. The present invention can process a variety of file formats (such as PDF, multimedia files, unstructured databases, etc.), rather than just text and images. This meets the actual needs of desensitizing various data types and can adapt to new file formats that may appear in the future.

[0030] 2. Improve parallel processing capabilities. The present invention uses task queues to process tasks in parallel and supports distributing detection components to different servers for processing. In this way, the efficiency of data processing can be significantly improved, and large-scale data tasks can be better handled.

[0031] 3. Mature monitoring capabilities. The present invention supports task queue monitoring tools, such as Celery's Flower and Celerymon tools, which can be used to monitor task execution status and worker health in real time. This can provide a comprehensive visualization of task progress, ensure that tasks can be completed smoothly, and promptly discover and handle potential problems.

[0032] 4. Optimized large file processing. By using object storage services, the present invention can efficiently process large files (such as AI model weights and large-scale JSON data), avoiding serialization performance problems and file unavailability problems caused by large files, and improving the stability and processing efficiency of the system.

[0033] 5. Flexible task allocation and management. The present invention supports the use of load balancing to distribute tasks to different servers. This flexible task allocation mechanism enables the system to adjust resource configuration according to actual needs, thereby improving the scalability and elasticity of the system.

[0034] 6. Efficient task result processing. The present invention can effectively process and format the results after the task is completed, ensuring that the output results are accurate and meet expectations. This includes checking the success status of the task and removing unnecessary output data to ensure the simplicity and effectiveness of the final result. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of an extensible data desensitization method based on a task queue in Example 1 of the present invention.

[0036] Figure 2 This is an architecture diagram of a data desensitization framework based on a Celery task queue in Example 1 of the present invention.

[0037] Figure 3 This is a flow chart of the operation of the preset text detection component of Example 1 of the present invention.

[0038] Figure 4 This is a flow chart of the operation of the preset image OCR detection component of Example 1 of the present invention.

[0039] Figure 5 This is a flow chart of the custom detection component class of Example 1 of the present invention. DETAILED DESCRIPTION

[0040] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific implementation methods of the present invention are now described. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0041] Example 1

[0042] Existing desensitization frameworks and solutions can usually only process fixed file formats (such as text, images, etc.). However, in practical applications, there are more and more file formats that need to be processed, including PDF, multimedia files, unstructured databases, etc. Existing frameworks are difficult to adapt to these flexible and diverse needs, and cannot effectively detect and desensitize these new file formats, resulting in increased data security risks.

[0043] Furthermore, with the surge in data volume and the increase in processing requirements, existing frameworks or solutions are unable to cope with the parallel processing and task monitoring capabilities. They usually lack effective parallel processing mechanisms and are difficult to support large-scale file desensitization tasks. In addition, these frameworks or solutions are also deficient in task monitoring and management, and are unable to track the execution progress of tasks and the health status of working nodes in real time, thus affecting the overall processing efficiency and system reliability.

[0044] Based on this, this embodiment provides a scalable data desensitization method based on a task queue, such as Figure 1 As shown, it includes: for different types of target files, extracting the data format that can be processed in batches from the original data format of the target file; matching the corresponding data detection and data desensitization processing logic based on the data characteristics of the target file and the converted data format; based on the data detection and data desensitization processing logic of the target file, executing data detection and data desensitization tasks through multi-queue task distribution and parallel processing of the task queue.

[0045] Preferably, matching the corresponding data detection and data desensitization processing logic based on the data characteristics of the target file and the converted data format can be achieved in the following way: setting a detection component according to the data characteristics of the target file and the converted data format, the detection component includes a detector and a processor, the detector stores a detection process function, the processor stores a desensitization process function, and part of the output of the detector is part of the input of the processor.

[0046] More preferably, for target files of text type, the detection component can output a tuple in a first preset format through the detector, and combine it with the original text as the input of the processor, the tuple in the first preset format includes a starting point, an end point and a credibility; the processor can perform corresponding mask processing on the detection results whose credibility is greater than a set threshold, and finally output the masked text content.

[0047] More preferably, for target files of image type, the detection component can output a tuple in a second preset format through the detector, and combine it with the original image as the input of the processor, the tuple in the second preset format includes the horizontal coordinate of the starting point, the detection frame width, the detection frame height and the credibility; the processor can perform corresponding rectangular mosaic processing on the detection results whose credibility is greater than the set threshold, and finally output the processed image content.

[0048] Preferably, in this embodiment, the data detection and data desensitization processing logic based on the target file is performed through multi-queue task distribution and parallel processing of the task queue, which can be implemented in the following ways:

[0049] After the detection components are set according to the data characteristics of the target file and the converted data format, a unique ID is assigned to each detection component through the registration center unit, and the unique ID and the detection component are transmitted to the publishing unit;

[0050] The publishing unit reads the load function of the detection component through the task packaging publisher and submits it to the Celery task queue as a task; when the Celery task queue returns the task ID, the publishing unit binds the unique ID and the task ID one by one through the publishing result storage; after the publishing unit completes the binding, it transmits the binding result to the subscription center unit;

[0051] The subscription center unit detects the execution status of tasks through the monitor according to the task ID polling until all tasks are completed; after all tasks are completed, the subscription center unit determines the success status of the tasks through the assembler and assembles and outputs the success results in a dictionary format.

[0052] Specifically, taking the Celery task queue as an example, this embodiment provides a data desensitization framework based on the Celery task queue, and its architecture is as follows: Figure 2 As shown in the figure. The data desensitization framework can be divided into two parts: Celery parallel execution unit and framework body. The parallel execution unit is implemented by Celery, and the Celery module can be deployed in a single machine or cluster. The framework body includes six functional modules: data preprocessing, function definition, registration center, publisher, subscription center and result assembly.

[0053] 1. Celery parallel execution unit implementation method:

[0054] The Celery parallel execution unit is mainly responsible for executing detection and desensitization tasks in parallel, while monitoring the progress of parallel tasks and the health of Celery workers. By using the prefork mode, Celery workers can use multi-core CPUs for parallel task processing. When deployed in a cluster, workers on all machines listen to highPriority and lowPriority queues at the same time, and Kombu is responsible for load balancing and distributing tasks to different workers for parallel execution. In order to monitor the health of Celery workers and the completion of tasks, users can use tools such as Flower and Celerymon. In terms of messaging middleware, Celery's broker can choose RabbitMQ, Redis or SQS, while the result storage backend supports tools such as Redis, SQLAlchemy or Django ORM. It should be noted that some monitoring tools may not work properly when using SQS as a broker.

[0055] 2. The operation process of the framework body:

[0056] The framework is responsible for the definition and management of the overall desensitization function, including:

[0057] First, when using this framework, you first need to select the specific format file to be processed and process the file into a data format that can be processed in batches, such as processing a word document into data formats such as string text, images, and tables.

[0058] Second, the detection process and desensitization process need to be defined according to the data format. The detection process function is called a detector, and the desensitization process function is called a processor, which are collectively called a detection component. When using a detection component, it is necessary to clarify the detector and processor input and output, and part of the detector output needs to be part of the processor input. The framework will verify the above process and reject the detection component whose detector output does not appear in the processor input. The framework provides common detection components, including text detection components and image detection components. The framework provides common rule-matching-based text detectors, such as dates, bank card numbers, ID card numbers, etc., and artificial intelligence-based text detectors, such as names, etc., as well as common text processors such as deletion, hash value replacement, and label value replacement. Image OCR-based detectors and rectangular mosaic processors are also provided.

[0059] Preferably, the preset text detection component runs as follows: Figure 3As shown, after the user extracts the text, the predefined text detection component is used. After being processed by the framework's registration center, publisher, Celery worker execution and subscription center, a dictionary in the format of '{detection component unique ID: [(detector output), (publisher output)]}' is output. The text detector in the predefined detection component in the framework will output a tuple in the format of '(starting point, end point, credibility)', and use the output and the original text as the input of the text processor. The processor will perform corresponding mask processing on the detection results with credibility greater than a certain threshold, and finally output the masked text content. Finally, the result assembler removes the detector output and outputs the final result dictionary.

[0060] Preferably, the preset image OCR detection component runs as follows: Figure 4 As shown, after the user extracts the image, the predefined image OCR detection component is used. After being processed by the framework's registration center, publisher, Celery worker execution and subscription center, a dictionary in the format of '{detection component unique ID: [(detector output), (publisher output)]}' is output. The OCR detector in the predefined detection component in the framework will output a tuple in the format of '(starting point horizontal coordinate, detection box width, detection box height, credibility)', and use the output and the original image as the input of the OCR processor. The processor will perform corresponding rectangular mosaic processing on the detection results with credibility greater than a certain threshold, and finally output the byte array content of the coded image. Finally, the detector output is removed by the result assembler to output the final result dictionary.

[0061] Preferably, the custom detection component class is such as Figure 5As shown in the figure, after extracting the data that can be processed in batches, the user needs to define the corresponding detectors and processors according to the data characteristics. In the framework, these detectors and processors are passed to the load function, which serves as the execution entry of the Celery worker and is decorated by the Celery decorator. It is important to note that if the detectors and processors use large files (such as AI model weight data) or reference large JSON data, some challenges may occur. Since the Celery worker and the detection component may run on different machines, the file may not be found on some nodes. In addition, Celery serializes the parameters in the task and sends them through the message broker, which may lead to performance degradation, high memory usage, and task timeout problems. Therefore, in order to effectively process large files, it is recommended to use object storage services in detectors and processors. By storing large files in object storage during preprocessing, detectors and processors can call the required files through the link of the object storage. This not only avoids the problem of file unavailability, but also reduces the performance burden during serialization, thereby improving the performance of the overall system and reducing memory usage. In addition, object storage services can also ensure the accessibility and consistency of files in a distributed processing environment, thereby better supporting the execution of large-scale data desensitization tasks.

[0062] Third, after the user defines and submits all detection components, the framework's registration center will assign a unique ID to each detection component. After the binding is completed, the unique ID and the detection component will be passed to the publisher. In the publisher, the task wrapper reads the load function of the detection component and submits it to Celery as a task. When Celery returns the task ID, the result storage in the publisher will bind the unique ID and the task ID one by one. After the publisher completes the binding, it will pass the binding result to the subscription center. The monitor in the subscription center will poll the execution status of the detection task according to the task ID until all tasks are completed. After the task is completed, the subscription center assembler will judge the success status of the task and assemble the successful result in the dictionary format of '{detection component unique ID: [(detector output), (publisher output)]}'. Finally, the result assembler removes the detector output results of all detection components in the output results of the subscription center, retains only the publisher output, and finally assembles the output in the dictionary format of '{detection component unique ID: (publisher output)}'.

[0063] Example 2

[0064] This embodiment provides a scalable data desensitization system based on a task queue, including:

[0065] The first processing module is configured to extract data formats that can be processed in batches from the original data formats of target files of different types; and match corresponding data detection and data desensitization processing logic based on the data features of the target files and the converted data formats;

[0066] The second processing module is configured to perform data detection and data desensitization tasks based on the data detection and data desensitization processing logic of the target file through multi-queue task distribution and parallel processing of the task queue.

[0067] Preferably, in the first processing module, the corresponding data detection and data desensitization processing logic is matched based on the data characteristics of the target file and the converted data format, including: setting a detection component according to the data characteristics of the target file and the converted data format, the detection component includes a detector and a processor, the detector stores a detection process function, the processor stores a desensitization process function, and part of the output of the detector is part of the input of the processor.

[0068] Preferably, for target files of text type, the detection component can output a tuple in a first preset format through the detector, and combine it with the original text as the input of the processor, the tuple in the first preset format includes a starting point, an end point and a credibility; the processor can perform corresponding mask processing on the detection results whose credibility is greater than a set threshold, and finally output the masked text content.

[0069] Preferably, for target files of image type, the detection component can output a tuple in a second preset format through the detector, and combine it with the original image as the input of the processor, the tuple in the second preset format includes the horizontal coordinate of the starting point, the detection frame width, the detection frame height and the credibility; the processor can perform corresponding rectangular mosaic processing on the detection results whose credibility is greater than the set threshold, and finally output the processed image content.

[0070] Preferably, the second processing module includes a registration center unit, a publishing unit and a subscription center unit, wherein:

[0071] The registration center unit is configured to assign a unique ID to the detection component and transmit the unique ID and the detection component to the publishing unit; the detection component is set according to the data characteristics of the target file and the converted data format;

[0072] The publishing unit is configured to read the load function of the detection component through the task packaging publisher and submit it to the Celery task queue as a task; when the Celery task queue returns the task ID, the publishing unit binds the unique ID and the task ID one by one through the publishing result storage; after the publishing unit completes the binding, it transmits the binding result to the subscription center unit;

[0073] The subscription center unit is configured to detect the execution status of tasks by polling according to the task ID through a monitor until all tasks are completed; after all tasks are completed, the subscription center unit determines the success status of the tasks through an assembler and assembles and outputs the success results in a dictionary format.

[0074] Example 3

[0075] This embodiment is based on embodiment 1:

[0076] This embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the scalable data desensitization method based on the task queue of embodiment 1 is implemented. The computer program may be in source code form, object code form, executable file, or some intermediate form.

[0077] Example 4

[0078] This embodiment is based on embodiment 1:

[0079] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the scalable data desensitization method based on a task queue of Embodiment 1. The computer program may be in source code form, object code form, executable file, or some intermediate form. The storage medium includes: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the storage medium does not include electric carrier signals and telecommunication signals.

[0080] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

Claims

1. A scalable data desensitization method based on task queue, characterized in that: include: For different types of target files, extract the data format that can be processed in batches from the original data format of the target file; Based on the data features of the target file and the converted data format, match the corresponding data detection and data desensitization processing logic; Based on the data detection and data desensitization processing logic of the target file, data detection and data desensitization tasks are executed through multi-queue task distribution and parallel processing of the task queue.

2. According to the scalable data desensitization method based on task queue in claim 1, it is characterized in that: The data characteristics of the target file and the converted data format are matched with corresponding data detection and data desensitization processing logic, including: A detection component is set according to the data features of the target file and the converted data format. The detection component includes a detector and a processor. The detector stores a detection process function, the processor stores a desensitization process function, and part of the output of the detector is part of the input of the processor.

3. According to the scalable data desensitization method based on task queue in claim 2, it is characterized in that: For a target file of a text type, the detection component can output a tuple in a first preset format through a detector, and combine the original text as an input of a processor, wherein the tuple in the first preset format includes a start point, an end point and a credibility; The processor can perform corresponding mask processing on the detection results whose credibility is greater than a set threshold, and finally output the masked text content.

4. According to the scalable data desensitization method based on task queue in claim 2, it is characterized in that: For target files of image type, the detection component can output a tuple in a second preset format through a detector, and combine it with the original image as the input of the processor. The tuple in the second preset format includes the horizontal coordinate of the starting point, the detection frame width, the detection frame height and the credibility; the processor can perform corresponding rectangular mosaic processing on the detection results whose credibility is greater than the set threshold, and finally output the processed image content.

5. The scalable data desensitization method based on task queue according to claim 1 is characterized in that: The data detection and data desensitization processing logic based on the target file executes the data detection and data desensitization tasks through multi-queue task distribution and parallel processing of the task queue, including: After the detection components are set according to the data characteristics of the target file and the converted data format, a unique ID is assigned to each detection component through the registration center unit, and the unique ID and the detection component are transmitted to the publishing unit; The publishing unit reads the load function of the detection component through the task packaging publisher and submits it to the Celery task queue as a task; when the Celery task queue returns the task ID, the publishing unit binds the unique ID and the task ID one by one through the publishing result storage; after the publishing unit completes the binding, it transmits the binding result to the subscription center unit; The subscription center unit detects the execution status of the tasks through a monitor according to the task ID polling until all tasks are completed; after all tasks are completed, the subscription center unit determines the success status of the tasks through an assembler, and assembles and outputs the success results in a dictionary format.

6. A scalable data desensitization system based on task queue, characterized in that: include: The first processing module is configured to extract data formats that can be processed in batches from original data formats of target files of different types; Based on the data features of the target file and the converted data format, match the corresponding data detection and data desensitization processing logic; The second processing module is configured to perform data detection and data desensitization tasks based on the data detection and data desensitization processing logic of the target file through multi-queue task distribution and parallel processing of the task queue.

7. The scalable data desensitization system based on task queue according to claim 6 is characterized in that: In the first processing module, based on the data features of the target file and the converted data format, the corresponding data detection and data desensitization processing logic is matched, including: A detection component is set according to the data features of the target file and the converted data format. The detection component includes a detector and a processor. The detector stores a detection process function, the processor stores a desensitization process function, and part of the output of the detector is part of the input of the processor.

8. The scalable data desensitization system based on task queue according to claim 7 is characterized in that: For a target file of a text type, the detection component can output a tuple in a first preset format through a detector, and combine the original text as an input of a processor, wherein the tuple in the first preset format includes a start point, an end point and a credibility; The processor can perform corresponding mask processing on the detection results whose credibility is greater than a set threshold, and finally output the masked text content.

9. The scalable data desensitization system based on task queue according to claim 7 is characterized in that: For target files of image type, the detection component can output a tuple in a second preset format through a detector, and combine it with the original image as the input of the processor. The tuple in the second preset format includes the horizontal coordinate of the starting point, the detection frame width, the detection frame height and the credibility; the processor can perform corresponding rectangular mosaic processing on the detection results whose credibility is greater than the set threshold, and finally output the processed image content.

10. The scalable data desensitization system based on task queue according to claim 6, characterized in that: The second processing module includes a registration center unit, a publishing unit and a subscription center unit, wherein: The registration center unit is configured to assign a unique ID to the detection component and transmit the unique ID and the detection component to the publishing unit; the detection component is set according to the data characteristics of the target file and the converted data format; The publishing unit is configured to read the load function of the detection component through the task packaging publisher and submit it to the Celery task queue as a task; when the Celery task queue returns the task ID, the publishing unit binds the unique ID and the task ID one by one through the publishing result storage; after the publishing unit completes the binding, the binding result is transmitted to the subscription center unit; The subscription center unit is configured to detect the execution status of the tasks through a monitor according to the task ID polling until all tasks are completed; after all tasks are completed, the subscription center unit determines the success status of the tasks through an assembler, and assembles and outputs the success results in a dictionary format.

Images