System data collecting and cleaning method, device and system and storage medium

Abstract

The invention discloses a system data collecting and cleaning method, device and system and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: formulating an acquisition task, and marking the acquisition task as a to-be-executed task in a data acquisition schedule; creating a thread pool to mark a to-be-executed collection task as in-progress; synchronizing the blocking queue through the thread pool and the non-fair lock to obtain original data; mapping the original data into entity objects, and obtaining newly obtained entity objects after extraction and integration; retrieving the original entity object based on the primary key to obtain an existing entity object; updating the original data through MD5 hash code comparison, and recording the updated original data into a data updating record table; and asynchronously traversing the data updating record table through the thread pool, cleaning and synchronizing, and marking the data updating record table as completed. According to the method provided by the embodiment of the invention, the problem of performance bottleneck caused by the system in the full-amount system data acquisition and cleaning process is solved, the real-time performance of data updating and the stability of the system are ensured, and resource occupation is reduced.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a system data acquisition and cleaning method, device, system and storage medium. Background Art

[0002] The data collection and cleaning process of traditional systems is highly dependent on whether the data center can provide incremental identification. In the absence of such identification, the system is forced to perform full data collection and cleaning, which is particularly heavy in the face of data centers with huge amounts of data. It not only increases the load on the system server database, but may also cause slow service response or even service crash.

[0003] In reality, most data centers are unable to provide incremental data or the incremental data they provide has quality issues, which forces the system to adopt full data collection and cleaning. However, full data collection and cleaning has the problems of resource usage surge, performance bottleneck, data synchronization delay and system stability risk, among which: Surge in resource usage: Full data processing consumes huge amounts of system resources, especially CPU, memory, and disk I / O, which can significantly reduce system response speed and affect user experience. Performance bottleneck: With the rapid growth of system concurrency and data volume, the original data processing process is difficult to meet the real-time and stability requirements of the business. The system may encounter serious performance bottlenecks, such as response delays and limited transaction processing capabilities. Data synchronization delay: Full data processing increases the delay in data synchronization, which may cause information updates on the system to lag, affecting the timeliness of information and user trust in the system. System stability risk: Long-term high resource usage and large-scale data processing will increase the risk of system downtime, especially during peak hours, which may cause system crashes, seriously affecting service continuity and availability. Summary of the invention

[0004] The purpose of this application is to provide a system data acquisition and cleaning method, device, system and storage medium, aiming to solve the performance bottleneck problem caused by the system during the full system data acquisition and cleaning process, while ensuring the real-time data update and system stability, and reducing resource usage.

[0005] In order to achieve the above objectives, the solution of this application is: In a first aspect, an embodiment of the present application provides a system data collection and cleaning method, comprising: Obtain the update frequency of the original data, formulate collection tasks according to the update frequency of the original data, create a data collection schedule, record the collection tasks in the data collection schedule, and mark them as pending; Create a thread pool and set the number of core threads. Based on the asynchronous thread, continuously scan the collection tasks to be executed and mark the collection tasks to be executed as in progress. Submit the ongoing collection tasks to the thread pool, create an unfair lock synchronization blocking queue that matches the number of core threads, manage the thread pool through the unfair lock synchronization blocking queue, and obtain the original data. Map the original data to entity objects, extract the field aliases in the entity objects, and process the data integration of many-to-one relationships to obtain the newly acquired entity objects; retrieve the original entity objects based on the primary key to obtain the existing entity objects; create a data update record table, compare the newly acquired entity objects with the existing entity objects by MD5 hash coding, and if the MD5 comparison reveals data differences, update the original data according to the primary key and record it in the data update record table; The data update record table is traversed asynchronously through the thread pool, and data cleaning is performed according to the data update record table; the cleaned data is updated to the target system based on the communication interface, and marked as completed in the data update record table to realize system data collection and cleaning.

[0006] According to the above method of the embodiment of the present application, the following additional technical features may also be provided: Furthermore, collection tasks are formulated according to the update frequency of the original data, including formulating periodic collection tasks and intermittent collection tasks according to the update frequency of the original data, and reserving a mechanism for manually starting the collection task.

[0007] Furthermore, before executing the collection task, the historical records of the previous tasks are cleaned up.

[0008] Furthermore, the method includes a fault recovery mechanism, and when an abnormality occurs in data collection, an asynchronous thread is activated to screen the collection tasks to be executed and in progress in the data collection schedule.

[0009] Furthermore, the thread pool includes a high-performance thread pool. When an abnormality occurs in data collection, the high-performance thread pool is used to recover the screened collection tasks to be executed and in progress.

[0010] Furthermore, the selected pending and ongoing collection tasks are recovered through a high-performance thread pool, including detection during the recovery process. If a recovery anomaly occurs, an alarm mechanism is triggered.

[0011] In a second aspect, an embodiment of the present application provides a system data acquisition and cleaning device, comprising: The task collection module is configured to obtain the update frequency of the original data, formulate the collection tasks according to the update frequency of the original data, create a data collection schedule, record the collection tasks in the data collection schedule, and mark them as to be executed; The data acquisition module is configured to create a thread pool and set the number of core threads, based on the asynchronous thread, continuously scan the collection tasks to be executed, mark the collection tasks to be executed as in progress; submit the collection tasks in progress to the thread pool, create an unfair lock synchronization blocking queue matching the number of core threads, manage the thread pool through the unfair lock synchronization blocking queue, and obtain the original data; A mapping comparison module is configured to map the original data to entity objects, extract field aliases in the entity objects, and process data integration of many-to-one relationships to obtain newly acquired entity objects; retrieve the original entity objects based on the primary key to obtain the existing entity objects; create a data update record table, compare the newly acquired entity objects with the existing entity objects by MD5 hash codes, and if the MD5 comparison reveals data differences, update the original data according to the primary key and record it in the data update record table; The data cleaning module is configured to asynchronously traverse the data update record table through the thread pool, and perform data cleaning according to the data update record table; update the cleaned data to the target system based on the communication interface, and mark it as completed in the data update record table to realize system data collection and cleaning.

[0012] In a third aspect, an embodiment of the present application provides a system data acquisition and cleaning system, the system including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the system data acquisition and cleaning method provided in the first aspect of the embodiment of the present application.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the system data acquisition and cleaning method as described in the first aspect of the embodiment of the present application.

[0014] Compared with the prior art, the system data collection and cleaning method provided by the embodiment of the present application has the following beneficial technical effects: The embodiment of the present application manages the execution of collection tasks by creating a thread pool and an unfair lock synchronization blocking queue, which significantly improves the concurrent processing capability of data collection, enables the system to process multiple collection tasks at the same time, and thus improves the efficiency of data collection; in the data cleaning process, MD5 hash coding is used to compare new and old entity objects, which can accurately find data differences and update the original data accordingly, ensuring the accuracy and consistency of the data and avoiding errors or problems caused by inconsistent data; collection tasks are formulated according to the update frequency of the original data, including periodic collection, intermittent collection, and a mechanism for reserving manual start of collection tasks. This flexibility enables the system to adapt to different update modes of different data sources, improving the adaptability and reliability of the system. Scalability; a fault recovery mechanism is introduced. When an exception occurs in data collection, the asynchronous thread can be automatically activated to screen and recover the affected collection tasks, especially using a high-performance thread pool for recovery and performing detection during the recovery process. Once a recovery exception occurs, an alarm mechanism is triggered. These measures together ensure the stability and reliability of the system; by creating a data collection schedule and a data update record table, the execution status of the collection task and the update of the data are recorded in detail. This traceability can clearly understand the data collection and update process, facilitate problem troubleshooting and system maintenance; the entire data collection, cleaning and update process is highly automated, reducing the need for manual intervention, reducing the risk of human error, and improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram showing a flow chart of a system data collection and cleaning method according to an embodiment of the present application is shown; Figure 2 A structural block diagram of a system data acquisition and cleaning device according to an embodiment of the present application is shown; Figure 3 A structural block diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0016] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0017] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0018] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0019] like Figure 1 As shown, the embodiment of the present application provides a system data collection and cleaning method, comprising the following steps: Step 101, obtain the update frequency of the original data, formulate a collection task according to the update frequency of the original data, create a data collection schedule, record the collection task in the data collection schedule, and mark it as to be executed.

[0020] Understanding the update frequency of raw data and formulating collection tasks based on this frequency are important steps to ensure data collection efficiency and accuracy.

[0021] First, the system needs to understand the update frequency of each data source or data set. The update frequency may vary from data source to data source, some may be updated every hour, some may be updated every day, and some may be updated every week or every month.

[0022] After understanding the update frequency of the original data, the system needs to formulate appropriate collection tasks based on this information. This includes determining the time point of collection, the collection cycle (such as hourly, daily, weekly, etc.) and the specific content of the collection. For example, if a data source is updated once an hour, the system can set a collection task that is executed once an hour to obtain the latest data.

[0023] In order to track and manage the execution of collection tasks, the system needs to create a data collection schedule. This table can contain multiple fields, such as task ID, data source name, collection time, collection status (to be executed, in progress, completed, etc.). Through this table, you can clearly understand the current status and history of each collection task.

[0024] After the collection tasks are formulated, the system needs to record these tasks in the previously created data collection schedule and mark their initial status as "to be executed". In this way, the system can arrange and manage the execution of the collection tasks according to this schedule. When the system is ready to execute a collection task, it will retrieve the task marked as "to be executed" from the schedule and update its status to "in progress".

[0025] When formulating the collection tasks, the embodiment of the present application not only considers periodic collection and intermittent collection, but also reserves a mechanism for manually starting the collection tasks, which increases the flexibility and operability of the system. Before executing the collection task, the historical records of the previous tasks are cleaned up to ensure the accuracy of the data and the efficient operation of the system.

[0026] Step 102, create a thread pool and set the number of core threads, based on the asynchronous thread continuously scanning the collection tasks to be executed, mark the collection tasks to be executed as in progress; submit the ongoing collection tasks to the thread pool, create an unfair lock synchronization blocking queue that matches the number of core threads, manage the thread pool through the unfair lock synchronization blocking queue, and obtain the original data.

[0027] The embodiment of the present application creates a thread pool through ThreadPoolExecutor, sets the number of core threads based on the specific specifications of the server, and ensures effective allocation of resources.

[0028] ThreadPoolExecutor is a class in the Java concurrent package java.util.concurrent. Thread pool is a resource reuse solution based on pooling technology, which aims to reduce the overhead of creating and destroying threads and improve system performance. ThreadPoolExecutor allows flexible control of thread pool configuration, including thread pool size, thread life cycle, task queue, etc.

[0029] ThreadPoolExecutor provides multiple constructors, including: corePoolSize: The core thread pool size, that is, the number of threads that always remain alive in the thread pool, even if these threads are idle.

[0030] maximumPoolSize: The maximum number of threads that the thread pool can accommodate executing simultaneously.

[0031] keepAliveTime and unit: When the number of threads is greater than the number of core threads, this is the maximum time that excess idle threads will wait for new tasks before terminating.

[0032] workQueue: A blocking queue used to store tasks to be executed.

[0033] threadFactory: A factory for creating new threads. You can customize how threads are created.

[0034] handler: The rejection strategy to use when a task cannot be executed by the thread pool (i.e. the thread pool has been closed or has reached maximum capacity).

[0035] The main methods include: execute(Runnable command): Submits a task to the thread pool for execution.

[0036] submit(Callable <t>task): submit a task that returns a result, and return a Future that indicates that the task is waiting to be completed <t>.

[0037] shutdown(): Starts the shutdown sequence of the thread pool. It will no longer accept new tasks, but will wait for submitted tasks to complete.

[0038] shutdownNow(): Attempts to stop all active tasks that are executing, stops processing waiting tasks, and returns a list of tasks waiting to be executed.

[0039] The advantages of thread pools are that they reduce resource consumption by reusing existing threads, reducing the overhead of thread creation and destruction; they improve response speed. When a task arrives, existing threads in the thread pool can be directly reused without waiting for the creation of new threads; they improve thread manageability. The thread pool can uniformly manage multiple threads (such as setting thread priority, thread name, etc.) to reduce competition between threads.

[0040] The embodiment of the present application creates an unfair lock synchronization blocking queue that matches the number of core threads through new Semaphore(*), providing fine resource control for concurrent tasks.

[0041] In Java, Semaphore is a class used to control the number of operations that access a particular resource or resource pool at the same time. It maintains a set of permits, each of which represents an available resource or an access right to a resource. When a thread wants to access a resource, it must first obtain a permit; if the permit is available, the thread can continue to execute; if the permit is not available, the thread will be blocked until a permit is available.

[0042] Semaphore's constructor mainly includes two versions: Semaphore(int permits): This constructor creates a semaphore with a given number of permits, all of which are available. The permits parameter specifies the number of permits the semaphore will initially have.

[0043] Semaphore(int permits, boolean fair): This constructor also creates a semaphore with a given number of permits, but with an additional fair parameter to control whether the allocation of permits is fair. If fair is true, threads will get permits in the order they request permits (FIFO), but this will reduce throughput. If fair is false, threads will get permits randomly, which is usually more efficient, but may cause starvation problems (some threads may not get permits for a long time).

[0044] In the embodiment of the present application, before the task starts, a resource token is obtained through semaphore.acquire(), and the state value of the synchronization queue is decremented. When the state value is less than zero, the new task will be suspended and added to the blocking queue. After the data comparison is completed, the token is released through semaphore.release(), the state value is restored, and the waiting thread is awakened to achieve efficient reuse of resources.

[0045] In Java's java.util.concurrent.Semaphore class, the acquire() method is used to obtain a permit to access a resource or perform a restricted operation. If there is no permit currently available, the thread calling the acquire() method will be blocked until a permit is available.

[0046] The acquire() method has no parameters and no return value (or it returns void), but it internally blocks the current thread to wait for the permit to become available. Once the permit is acquired, the thread can continue to perform its subsequent operations.

[0047] It should be noted that when a thread is blocked by the acquire() method, it will enter a waiting state until a permit is available. Once a permit is available, the thread that has been waiting the longest (by default, but it is not guaranteed to be absolutely fair because Java's Semaphore is unfair by default) will obtain the permit and continue execution.

[0048] In addition, if the thread is interrupted while waiting for permission (that is, the interrupt() method of the thread is called), the acquire() method will throw an InterruptedException and clear the interrupt status of the current thread. Therefore, in the code that calls acquire(), it is usually necessary to catch and handle InterruptedException, and reset the interrupt status when the exception is caught, or take other appropriate measures.

[0049] Step 103, map the original data to entity objects, extract the field aliases in the entity objects, and process the data integration of many-to-one relationships to obtain newly acquired entity objects; retrieve the original entity objects based on the primary key to obtain the existing entity objects; create a data update record table, compare the newly acquired entity objects with the existing entity objects by MD5 hash coding, and if the MD5 comparison reveals data differences, update the original data according to the primary key and record it in the data update record table.

[0050] The embodiment of the present application maps the acquired raw data to the corresponding entity object through mapToEntity. mapToEntity usually appears in software development, especially when using the ORM (object-relational mapping) framework. The ORM framework allows developers to operate the database in an object-oriented way, mapping the tables and data rows in the database to classes and objects in the programming language. As an operation or method name, the core concept of mapToEntity is to map or convert a certain form of data (such as DTOs - data transfer objects, JSON, XML, etc.) to the corresponding entity object.

[0051] The main uses include data conversion. When data is transferred from an external system or front-end to the back-end through an API, the data usually exists in DTOs, JSON or other formats. In order to store this data in the database, it is necessary to convert this data into an entity object corresponding to the database table structure. The mapToEntity method or operation is used for this conversion; decoupling. By encapsulating the logic of data conversion in mapToEntity, the coupling between different parts of the application (such as controllers, service layers, and data access layers) can be reduced. In this way, when the database table structure changes, only the mapping logic in mapToEntity needs to be modified without modifying the code at other levels. Data validation. When mapping data to entity objects, the mapToEntity method can also include data validation logic, which helps ensure the integrity and consistency of the data and prevents invalid or erroneous data from being saved in the database.

[0052] In the mapToEntity method, the embodiment of the present application extracts the alias of the field according to the Name attribute of the @Alias ​​annotation in the entity class, ensures that it is consistent with the field naming in the data center, and realizes accurate mapping of data.

[0053] Field aliases are extracted because they are usually more descriptive and readable than the original field names, making it easier for non-technical users or general users to understand the meaning of the data. For example, setting the alias of the POPULATION field to population makes the information on the data report or interface more intuitive and easy to understand. In complex query statements, field aliases can simplify field references, making query statements clearer and easier to maintain.

[0054] In queries involving multiple tables, there may be cases where the field names are the same but the meanings are different. By specifying different aliases for these fields, you can avoid naming conflicts and ensure the accuracy of query results. In self-join queries, the same table is referenced multiple times. At this time, specifying different aliases for the fields in the table is a necessary means to distinguish different reference instances.

[0055] When generating reports or creating database views, field aliases can be used to customize the way data is presented, making the report or view more in line with user needs and expectations. When exporting data to other systems or file formats, field aliases can be used to define the names of columns in the exported file, improving the efficiency and accuracy of data exchange.

[0056] When writing programs or scripts that interact with the database, using field aliases can make the code more concise and easier to understand. At the same time, field aliases can also be used as the names of variables or objects in the program to improve the readability and maintainability of the code. When building dynamic SQL statements, field aliases can be used to dynamically specify the column names of the query results, making the SQL statements more flexible and powerful.

[0057] In applications that need to support multiple languages, field aliases can be used to provide field descriptions in different languages. By specifying multilingual aliases for fields, you can easily achieve internationalization of your application.

[0058] For fields with many-to-one relationships, data integration is performed based on the tableName and fieldName attributes in the @Alias ​​annotation. The tableName attribute indicates the table name or interface name of the data center, and multiple data are separated by semicolons; the fieldName attribute corresponds to the specific field name, which matches the number of elements in the tableName attribute and is also separated by semicolons. When a single field needs to be mapped to multiple fields in the data center, multiple field names in the fieldName attribute are connected with commas. During the entity object conversion process, the system will automatically merge these field values.

[0059] The embodiment of the present application also efficiently retrieves the corresponding entity object instance in the memory based on the unique identifier of the currently collected data, that is, the primary key, to ensure the accuracy and speed of data comparison.

[0060] When using the primary key to retrieve an entity object, the main purpose is to find and obtain the complete entity object or part of its information corresponding to the primary key from a data source (such as a database, cache, in-memory data structure, etc.).

[0061] The primary key is a combination of one or more attributes (fields) of an entity object, which is used to uniquely identify each record in a data table. In a relational database, the primary key is usually defined as a combination of one or more fields, the values ​​of which are unique in the data table and do not allow duplication.

[0062] The newly acquired entity objects are compared with the existing objects in the memory by MD5 hash codes, and scientific algorithms are used to ensure data integrity and consistency, and timely discover and handle data changes. If the MD5 comparison reveals data differences, the obsolete records in the data exchange system database are accurately deleted based on the primary key, and then the latest data obtained from the data center is inserted into the database. At the same time, the update information, including the primary key and data type, is registered in the data update record table to provide a basis for subsequent data processing and auditing.

[0063] If the MD5 comparison reveals no data differences, no processing is required.

[0064] Specifically, the system will first formulate collection tasks based on the update frequency of the original data, and record these tasks in the data collection schedule and mark them as pending. Then, the system will continuously scan these pending collection tasks based on asynchronous threads. Once a task is found to be pending, it will be marked as in progress. Then, the thread pool is started to perform the actual collection tasks. The threads in the thread pool will work in parallel according to the set number of core threads to collect the latest raw data from data sources (such as databases, files, APIs, etc.). The collected raw data will be mapped to entity objects. This mapping process usually involves parsing the data format, verifying the data integrity and accuracy, and converting the data into an object model used within the application.

[0065] During the data mapping process, the system will extract the field aliases in the entity object, which is critical for subsequent data processing and display. For fields with many-to-one relationships, the system will also perform data integration to ensure data integrity and consistency. Ultimately, through this series of steps, the newly acquired raw data is converted into a new entity object.

[0066] Before updating data, the system needs to retrieve the existing entity object to compare with the newly acquired entity object. The retrieval process is usually based on the primary key, because the primary key is the unique identifier of the entity object in the data store. The system will use the primary key value in the newly acquired entity object to query the existing data store (such as a database) to obtain the corresponding existing entity object.

[0067] Once the newly acquired entity object and the existing entity object are obtained, the system will use MD5 hash codes to compare the data content of the two objects. If the MD5 comparison reveals that there are differences in the data, it means that the newly acquired entity object contains some updated or changed information. In this case, the system will update the original data according to the primary key and record the update record in the data update record table.

[0068] Step 104, asynchronously traverse the data update record table through the thread pool, and clean the data according to the data update record table; update the cleaned data to the target system based on the communication interface, and mark it as completed in the data update record table to realize system data collection and cleaning.

[0069] After the data comparison is completed, the thread pool is used to asynchronously traverse the data update record table, and the corresponding data cleaning model is intelligently started according to the data type in the record to ensure efficient and concurrent data processing. The data cleaning model performs data conversion and cleaning based on the system's specific business requirements and the data structure of the intermediate library, and accurately converts the original data into the business data format required by the system.

[0070] The embodiment of the present application uses a secure and reliable Webservice interface to seamlessly push the cleaned data to the system to achieve real-time data update and synchronization. The Webservice interface is a standardized interface for communication and data exchange between different systems. It is based on Web technology and standardized protocols such as HTTP, SOAP and XML, and achieves interoperability between systems on different platforms, different languages ​​and different technologies.

[0071] The system receives synchronized data, updates the specified data according to the preset synchronization strategy, and feeds back the update status to the data exchange system, forming a closed loop of data synchronization to ensure data consistency and integrity. After the execution process of a single data collection model is successfully completed, the system needs to accurately update the status of the corresponding record in the data collection schedule based on the unique identifier (ID) of the current scheduled task and the ID of the specific collection model, and clearly mark it as "completed", so as to achieve real-time tracking and management of the task execution status, ensure the integrity and controllability of the data collection process, and provide detailed data basis for subsequent monitoring and auditing.

[0072] The embodiment of the present application also includes a fault recovery mechanism (or collection task retry mechanism) after the service is restored. Once the data collection service is detected to have experienced a downtime and completed self-recovery, the system will immediately activate the asynchronous thread, go deep into the data collection schedule, and filter out all the collection model data marked as "to be executed" and "in progress". This measure is intended to quickly identify and locate those collection tasks that have not been completed due to abnormal service interruptions.

[0073] The system then starts a high-performance thread pool, schedules and executes the above-selected individual collection model tasks one by one, ensuring that even in the event of an unexpected service interruption, data collection work can be quickly and orderly restored, filling the collection gaps during the interruption period and maintaining the continuity and integrity of data collection. During the task retry process, the system can dynamically adjust the execution order of tasks in the thread pool according to the urgency of the task and the timeliness of the data, giving priority to critical or high-priority collection tasks, thereby maximizing resource utilization and shortening the overall recovery time. While the task is being retried, the system updates the task status in the data collection schedule in real time to ensure that the execution of each task is accurately recorded and monitored, facilitating subsequent auditing and troubleshooting. To enhance the robustness of the system, any exceptions that occur during the retry process will be captured and recorded by the system, and the alarm mechanism will be triggered to send an immediate notification to the operation and maintenance team to ensure that the problem is responded to and handled in a timely manner.

[0074] like Figure 2 As shown, the embodiment of the present application provides a system data acquisition and cleaning device, including a task acquisition module 201, a data acquisition module 202, a mapping comparison module 203 and a data cleaning module 204, wherein: The task collection module 201 is configured to obtain the update frequency of the original data, formulate the collection task according to the update frequency of the original data, create a data collection schedule, record the collection task in the data collection schedule, and mark it as to be executed; The data acquisition module 202 is configured to create a thread pool and set the number of core threads, based on the asynchronous thread continuously scanning the collection tasks to be executed, marking the collection tasks to be executed as in progress; submitting the in-progress collection tasks to the thread pool, creating an unfair lock synchronization blocking queue matching the number of core threads, managing the thread pool through the unfair lock synchronization blocking queue, and obtaining the original data; The mapping comparison module 203 is configured to map the original data to entity objects, extract the field aliases in the entity objects, and process the data integration of many-to-one relationships to obtain newly acquired entity objects; retrieve the original entity objects based on the primary key to obtain the existing entity objects; create a data update record table, compare the newly acquired entity objects with the existing entity objects by MD5 hash codes, and if the MD5 comparison reveals data differences, update the original data according to the primary key and record it in the data update record table; The data cleaning module 204 is configured to asynchronously traverse the data update record table through the thread pool, and perform data cleaning according to the data update record table; update the cleaned data to the target system based on the communication interface, and mark it as completed in the data update record table to realize system data collection and cleaning.

[0075] The system data acquisition cleaning device of the embodiment of the present application can be a computer device, or a component in the computer device, such as an integrated circuit or a chip. The computer device can be a terminal, or other devices other than a terminal. Exemplarily, the computer device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a vehicle-mounted computer device, a mobile Internet device (Mobile Internet Device, MID), an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), etc., which is not specifically limited in the embodiment of the present application.

[0076] The system data acquisition and cleaning device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the system data acquisition and cleaning method embodiment will not be described here.

[0077] The present application also provides a computer device, such as Figure 3 As shown, the computer device includes a processor 301 and a memory 302. The memory 302 stores programs or instructions that can be run on the processor 301. When the program or instruction is executed by the processor 301, each step of the above-mentioned system data acquisition and cleaning method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0078] It should be noted that the computer device in the embodiment of the present application includes the above-mentioned mobile computer device and non-mobile computer device.

[0079] The memory 302 can be used to store software programs and various data. The memory 302 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 302 may include a volatile memory or a non-volatile memory, or the memory 302 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 302 of the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0080] The processor 301 may include one or more processing units; optionally, the processor 301 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 301.

[0081] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned system data acquisition and cleaning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0082] An embodiment of the present application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned system data acquisition and cleaning method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0083] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0084] An embodiment of the present application also provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned system data acquisition and cleaning method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0085] It should be noted that, in the present application, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0086] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.< / t> < / t>

Claims

1. A system data collection and cleaning method, characterized in that: The method comprises: Obtaining the update frequency of the original data, formulating a collection task according to the update frequency of the original data, creating a data collection schedule, recording the collection task in the data collection schedule, and marking it as to be executed; Create a thread pool and set the number of core threads, based on the asynchronous thread uninterrupted scanning of the collection tasks to be executed, mark the collection tasks to be executed as in progress; submit the collection tasks in progress to the thread pool, create an unfair lock synchronization blocking queue matching the number of core threads, manage the thread pool through the unfair lock synchronization blocking queue, and obtain the original data; Mapping the original data to entity objects, extracting field aliases in the entity objects, and processing data integration of many-to-one relationships to obtain newly acquired entity objects; retrieving the original entity objects based on the primary key to obtain existing entity objects; creating a data update record table, performing MD5 hash code comparison between the newly acquired entity objects and the existing entity objects, and if the MD5 comparison reveals data differences, updating the original data according to the primary key and recording in the data update record table; The data update record table is asynchronously traversed through the thread pool, and data cleaning is performed according to the data update record table; the cleaned data is updated to the target system based on the communication interface, and marked as completed in the data update record table, thereby realizing system data collection and cleaning.

2. The system data collection and cleaning method according to claim 1, characterized in that: The formulation of the collection tasks according to the update frequency of the original data includes formulating periodic collection tasks and intermittent collection tasks according to the update frequency of the original data, and reserving a mechanism for manually starting the collection tasks.

3. The system data collection and cleaning method according to claim 1, characterized in that: Before executing the acquisition task, the historical records of the preceding task are cleared.

4. The system data collection and cleaning method according to claim 1, characterized in that: The method includes a fault recovery mechanism. When an abnormality occurs in data collection, an asynchronous thread is activated to screen the collection tasks to be executed and in progress in the data collection schedule.

5. The system data collection and cleaning method according to claim 4, characterized in that: The thread pool includes a high-performance thread pool. When an abnormality occurs in data collection, the high-performance thread pool is used to recover the screened collection tasks to be executed and in progress.

6. The system data collection and cleaning method according to claim 5, characterized in that: The high-performance thread pool is used to recover the screened pending and ongoing acquisition tasks, including detection during the recovery process. If a recovery abnormality occurs, an alarm mechanism is triggered.

7. A system data acquisition and cleaning device, characterized in that: The device comprises: A task collection module is configured to obtain the update frequency of the original data, formulate a collection task according to the update frequency of the original data, create a data collection schedule, record the collection task in the data collection schedule, and mark it as to be executed; The data acquisition module is configured to create a thread pool and set the number of core threads, based on the asynchronous thread continuously scanning the collection tasks to be executed, marking the collection tasks to be executed as in progress; submitting the in-progress collection tasks to the thread pool, creating an unfair lock synchronization blocking queue matching the number of core threads, managing the thread pool through the unfair lock synchronization blocking queue, and obtaining the original data; A mapping comparison module is configured to map the original data to entity objects, extract field aliases in the entity objects, and process data integration of many-to-one relationships to obtain newly acquired entity objects; retrieve the original entity objects based on the primary key to obtain existing entity objects; create a data update record table, compare the newly acquired entity objects with the existing entity objects by performing MD5 hash code comparison, and if the MD5 comparison reveals data differences, update the original data according to the primary key and record it in the data update record table; The data cleaning module is configured to asynchronously traverse the data update record table through the thread pool, perform data cleaning according to the data update record table; update the cleaned data to the target system based on the communication interface, and mark it as completed in the data update record table to realize system data collection and cleaning.

8. A system data collection and cleaning system, the system comprising a processor and a memory, wherein a computer program is stored in the memory, characterized in that: The computer program is loaded and executed by the processor to implement the system data acquisition and cleaning method according to any one of claims 1 to 6.

9. A computer-readable storage medium, wherein a computer program is stored in the storage medium, characterized in that: When the computer program is executed by a processor, it is used to implement the system data acquisition and cleaning method as described in any one of claims 1 to 6.

Images