Data processing system and analysis method based on Akka Actor model

Abstract

The invention discloses a data processing system and analysis method based on an Akka Actor model. The data processing system comprises a data production system, a data analysis and processing system, an intelligent monitoring platform and an intelligent abnormity checking platform. Wherein the data production system is connected with the data analysis and processing system; the data analysis and processing system and the intelligent abnormity checking platform are connected with the intelligent monitoring platform. According to the invention, all service modules are connected in series by utilizing the characteristics of Akka, so that efficient processing and flexible expansion of general data processing and analysis application are realized. Through the dynamic expansion capability of Akka, under the condition that a system is not restarted, a new algorithm Actor is developed and added into a cluster, and therefore rapid integration and deployment of functions are achieved. According to the method, system isolation can be effectively broken, cross-service seamless cooperation is realized, the concurrent processing capability and data processing efficiency of the system are improved, the whole process from data production, preprocessing and analysis to monitoring and task issuing is supported, and efficient monitoring and management of abnormal data are completed.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically a data processing system and analysis method based on the Akka Actor model. Background Technology

[0002] Data processing technology has undergone decades of development, with its application scope and technological level continuously improving. Modern data processing technologies utilize various sensors, data sources, and algorithms to process and analyze massive amounts of data in real time, finding wide application in multiple fields such as finance, healthcare, transportation, and environmental monitoring. These technological advancements have driven the transformation of data processing from traditional manual processing to automated processing and then to intelligent analysis, making data processing more efficient, accurate, and comprehensive.

[0003] In existing technologies, a distributed microservice architecture is typically adopted to achieve modular and unitized deployment of data processing service platforms. Through RPC calls, multi-layered service collaboration, and middleware support, business processes are driven and automated management is achieved. The unitized design of services enhances the system's scalability, flexibility, and maintainability, adapting to the needs of modern data processing systems.

[0004] The patent publication number, CN118055039A, is related to remote sensing data processing and focuses on breaking down services into processing units to maximize flexibility and decouple services. It does not mention concurrency, high availability, or cross-language communication technologies between services.

[0005] The patent publication number, CN117785416A, relates to cross-platform task scheduling and distributed service architecture. It relies on Kafka middleware for data synchronization service calls, focusing on communication issues between service modules of different architectures. There is no record of any technology related to high availability.

[0006] The patent publication number, CN117745432A, describes a data analysis application implemented using distributed microservices. It is based on traditional distributed microservices and achieves high availability through load balancing of replicated nodes. There is no record of cross-language architecture service calls. Summary of the Invention

[0007] The purpose of this invention is to provide a data processing system and analysis method based on the Akka Actor model, in order to solve the problems mentioned in the background art, which use a distributed microservice architecture in the data processing service platform, resulting in significant lag and low concurrent throughput, leading to low data processing efficiency.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A data processing system based on the Akka Actor model includes a data production system, a data analysis and processing system, an intelligent monitoring platform, and an intelligent anomaly detection platform; wherein the data production system is connected to the data analysis and processing system; and both the data analysis and processing system and the intelligent anomaly detection platform are connected to the intelligent monitoring platform.

[0010] The data production system generates data and transmits the generated data to the data analysis and processing system. The data analysis and processing system preprocesses and analyzes the data to generate structured data that can be used for monitoring. The intelligent monitoring platform triggers task processes based on the structured data obtained from the data analysis and processing system, performs anomaly judgment and decision-making, and monitors the data through a large screen. The intelligent anomaly verification platform issues verification tasks, manages the offline verification workflow, and reports the verification results.

[0011] A data analysis method based on the Akka Actor model, using the system described in claim 1 to analyze data, includes the following steps:

[0012] Step S1: Managers issue data verification tasks through the intelligent monitoring platform; Managers issue verification tasks through the intelligent monitoring platform in the data processing system, specifying the time, region and content of the verification.

[0013] Step S2: Data production is carried out through the data production system;

[0014] Step S3: Perform data preprocessing on the data obtained from the data production system using a data analysis and processing system;

[0015] Step S4: Data analysis is performed on the data through the data analysis and processing system; after the top-level message receiving Actor receives the data, it delegates it to the specified algorithm implementation Actor according to the system configuration parameters, and then the algorithm Actor delegates the sub-Actor to perform the actual analysis operation.

[0016] Step S5: The intelligent monitoring platform performs monitoring and judgment processing based on the data analysis results obtained in step S4.

[0017] Step S6: The intelligent anomaly verification platform generates specific verification tasks and assigns them to the managers responsible for the area for offline verification.

[0018] Step S7: Managers process the task through the intelligent anomaly verification platform: Managers conduct on-site verification and feed the verification results back to the intelligent monitoring platform to support the analysis and judgment of problem data;

[0019] Step S8, Monitoring screen display service on the intelligent monitoring platform: Publish the analysis results as a visualization data service and display them on the monitoring screen for managers to view in real time.

[0020] According to the above technical solution, in step S1, when the manager issues a data verification task, the task is issued by calling the data production actor in the data production system through the Actor of the start link interface in the intelligent monitoring platform.

[0021] According to the above technical solution, in step S2, the data production actor in the data production system receives the event message, delegates the sub-actor to perform the data production task, and generates data for the specified region and time.

[0022] According to the above technical solution, in step S3, the data preprocessing specifically involves: performing preliminary processing on the acquired data, having the message receiving Actor delegate the preprocessing task to the sub-Actor for non-blocking parallel execution, and forwarding the processing result to the data analysis and processing system.

[0023] According to the above technical solution, in step S4, a specific data processing algorithm is applied to calculate the specified analysis content and generate structured data.

[0024] According to the above technical solution, in step S5, based on the current verification results and historical data, an algorithm is used to determine whether there are any abnormalities in the data; the processing results are forwarded to the monitoring screen display service, and the abnormal data is forwarded to the task generation service.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] Through the technical solution of this invention, and via dynamic expansion and seamless integration mechanisms, new algorithm Actors can be added without restarting the system, thereby improving the flexibility of functional expansion. Direct message passing and chained calls between Actors solve the lag problem in traditional distributed microservice architectures, reducing system response time and complexity. Traditional systems require defining multiple topics and fixed producer-consumer logic, leading to complex and inflexible system design; the technical solution of this invention simplifies system logic and improves response speed. This invention utilizes Akka's lock-free concurrency and lightweight Actors to fully leverage physical machine performance, reduce resource consumption, and increase system throughput. Direct communication and seamless integration between services are achieved through Akka's Actor model, reducing system complexity and enhancing inter-service collaboration. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the functional structure of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example 1

[0030] like Figure 1 As shown, a data processing system based on the Akka Actor model is characterized by comprising a data production system, a data analysis and processing system, an intelligent monitoring platform, and an intelligent anomaly verification platform; wherein the data production system is connected to the data analysis and processing system; and both the data analysis and processing system and the intelligent anomaly verification platform are connected to the intelligent monitoring platform.

[0031] The data production system generates data and transmits the generated data to the data analysis and processing system. The data analysis and processing system preprocesses and analyzes the data to generate structured data that can be used for monitoring. The intelligent monitoring platform triggers task processes based on the structured data obtained from the data analysis and processing system, performs anomaly judgment and decision-making, and monitors the data through a large screen. The intelligent anomaly verification platform issues verification tasks, manages the offline verification workflow, and reports the verification results.

[0032] Specifically, the data processed by the data analysis and processing system is transmitted to the intelligent monitoring platform via Akka RPC communication. The intelligent monitoring platform, after analyzing and processing the data, displays the anomaly information on a large screen or notifies management personnel via SMS. Management personnel then create verification tasks from the anomaly data and distribute them to the intelligent anomaly verification platform via Akka RPC communication. Offline personnel then conduct on-site verification according to the workflow logic, returning the verification results to the intelligent monitoring platform to resolve the anomaly.

[0033] One implementation method for data generation; in the field of satellite remote sensing, data is satellite imagery, and data production, such as remote sensing data production (satellite remote sensing data system), involves the remote sensing data production actor in the system receiving event messages and delegating sub-actors to perform remote sensing data production, generating remote sensing data for a specified region and time. Following Akka's principles, it easily supports millions of actors and handles concurrent requests. The produced remote sensing data is then forwarded to the remote sensing data processing system.

[0034] This invention leverages the characteristics of Akka, especially its high-performance, fault-tolerant, and scalable Actor model, to connect various service modules, enabling efficient processing and flexible expansion of general data processing and analysis applications. Specifically, through Akka's dynamic expansion capabilities, new algorithmic actors can be developed and added to the cluster without restarting the system, thereby achieving rapid integration and deployment of functions. This invention effectively breaks down system isolation, enabling seamless cross-service collaboration, improving the system's concurrent processing capabilities and data processing efficiency, supporting the entire process from data production, preprocessing, analysis to monitoring and task distribution, and achieving efficient monitoring and management of abnormal data. Through this innovative method, this application significantly improves the system's flexibility and scalability while ensuring system stability and high performance.

[0035] Example 2

[0036] This embodiment is a further refinement of Embodiment 1.

[0037] A data analysis method based on the Akka Actor model, the data analysis includes the following steps:

[0038] Step S1: The manager issues a data verification task through the intelligent monitoring platform; the manager issues the verification task through the intelligent monitoring platform in the data processing system, specifying the time, region and content of the verification; the Actor of the starting link interface in the intelligent monitoring platform calls the data production Actor in the data production system.

[0039] Step S2 involves data production via a data production system. The data production actor in the system receives event messages and delegates data production tasks to sub-actors, generating data for a specified region and time. Akka can easily support millions of actors and handle concurrent requests. The produced data is then forwarded to a data analysis and processing system.

[0040] Step S3: The data obtained from the data production system is preprocessed through the data analysis and processing system; the acquired data is initially processed, such as noise reduction and correction, to ensure data quality.

[0041] Step S4: Data analysis is performed on the data through the data analysis and processing system; after the top-level message receiving Actor receives the data, it delegates it to the specified algorithm implementation Actor according to the system configuration parameters, and then the algorithm Actor delegates the sub-Actor to perform the actual analysis operation.

[0042] Step S5: The intelligent monitoring platform performs monitoring and judgment processing based on the data analysis results obtained in step S4.

[0043] Step S6: The intelligent anomaly verification platform generates specific verification tasks and assigns them to the managers responsible for the area for offline verification.

[0044] Step S7: Managers process the task through the intelligent anomaly verification platform: Managers conduct on-site verification and feed the verification results back to the intelligent monitoring platform to support the analysis and judgment of problem data;

[0045] Step S8, Monitoring screen display service on the intelligent monitoring platform: Publish the analysis results as a visualization data service and display them on the monitoring screen for managers to view in real time.

[0046] In step S1, when the manager issues a data verification task, the task is issued by calling the data production actor in the data production system through the Actor of the starting link interface in the intelligent monitoring platform.

[0047] In step S2, the data production actor in the data production system receives the event message, delegates the sub-actor to perform the data production task, and generates data for the specified region and time.

[0048] In step S3, data preprocessing specifically involves: performing preliminary processing on the acquired data; having the message receiving Actor delegate the preprocessing task to the sub-Actor for non-blocking parallel execution; and forwarding the processing result to the data analysis and processing system.

[0049] Specifically, in the field of satellite remote sensing, data processing here involves processing remote sensing data and, based on business requirements, invoking specific visual model algorithms to identify land parcels. A more detailed process involves producing remote sensing image data, issuing computational tasks (specifying the land parcels to be calculated and the computational target), and obtaining structured data that specifically identifies a particular land parcel and determines whether it conforms to the target rules. Applications include (for example, a task might be to calculate the crop planting type of farmland parcels in a specific county; the structured data of the identification result would show what crop was planted on that parcel (sorghum, corn, rice, others), and then combining this with other data to analyze whether the land should be used for corn planting). Anomalies are identified, and tasks are then issued to a verification platform for offline processing.

[0050] Specifically, the top-level Actor receives the data to be processed, and then creates delegated child Actors to perform the processing (e.g., if there are 100 new tasks to be processed, the Actor that processes these tasks can directly create 100 child Actors to process these 100 tasks concurrently).

[0051] In step S4, a specific data processing algorithm is applied to calculate the specified analysis content and generate structured data. For example, satellite imagery data is used to determine non-agricultural status. The input parameter is a raster image, specifying the area of ​​the plot. The calculation result is whether there is a problem with the area of ​​this plot, and the plot is marked as illegally occupied in this calculation task (structured data: plot ID, status, and other attributes), which requires offline verification and processing (misidentification, or temporary covering by a greenhouse, etc.).

[0052] Data processing algorithms include: temporal change detection for analyzing vegetation index changes during the crop growing season and detecting whether a plot of land has been uncultivated for a long period; classification algorithm: based on image classification, plots are divided into agricultural land and abandoned land; deep learning: a deep learning model is trained using multiple temporal images to automatically detect abandoned land.

[0053] The core implementation logic of satellite remote sensing technology can be roughly divided into: data acquisition and preprocessing, including geometric correction, radiometric correction, atmospheric correction and other steps to ensure the accuracy and consistency of the images.

[0054] Feature extraction: Target information, such as crop growth characteristics and land use change, is extracted through methods such as band combination, vegetation index, and time series analysis.

[0055] Model training and application: Based on machine learning or deep learning models, combined with ground truth data, train and validate classification models or change detection algorithms.

[0056] In step S5, based on the current verification results and historical data, an algorithm is used to determine whether there are any abnormalities in the data; the processing results are forwarded to the monitoring screen display service, and the abnormal data is forwarded to the task generation service.

[0057] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0058] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data processing system based on the Akka Actor model, characterized in that: It includes a data production system, a data analysis and processing system, an intelligent monitoring platform, and an intelligent anomaly detection platform; among them, the data production system is connected to the data analysis and processing system; the data analysis and processing system and the intelligent anomaly detection platform are both connected to the intelligent monitoring platform; The data production system generates data and transmits the generated data to the data analysis and processing system. The data analysis and processing system preprocesses and analyzes the data to generate structured data that can be used for monitoring. The intelligent monitoring platform triggers task processes based on the structured data obtained from the data analysis and processing system, performs anomaly judgment and decision-making, and monitors the data through a large screen. The intelligent anomaly verification platform issues verification tasks, manages the offline verification workflow, and reports the verification results.

2. A data analysis method based on the Akka Actor model, characterized in that: The system described in claim 1 is used to analyze data, and the data analysis includes the following steps: Step S1: Managers issue data verification tasks through the intelligent monitoring platform; Managers issue verification tasks through the intelligent monitoring platform in the data processing system, specifying the time, region and content of the verification. Step S2: Data production is carried out through the data production system; Step S3: Perform data preprocessing on the data obtained from the data production system using a data analysis and processing system; Step S4: Data analysis is performed on the data through the data analysis and processing system; after the top-level message receiving Actor receives the data, it delegates it to the specified algorithm implementation Actor according to the system configuration parameters, and then the algorithm Actor delegates the sub-Actor to perform the actual analysis operation. Step S5: The intelligent monitoring platform performs monitoring and judgment processing based on the data analysis results obtained in step S4. Step S6: The intelligent anomaly verification platform generates specific verification tasks and assigns them to the managers responsible for the area for offline verification. Step S7: Managers process the task through the intelligent anomaly verification platform: Managers conduct on-site verification and feed the verification results back to the intelligent monitoring platform to support the analysis and judgment of problem data; Step S8, Monitoring screen display service on the intelligent monitoring platform: Publish the analysis results as a visualization data service and display them on the monitoring screen for managers to view in real time.

3. The data analysis method based on the Akka Actor model according to claim 2, characterized in that: In step S1, when the manager issues a data verification task, the task is issued by calling the data production actor in the data production system through the Actor of the starting link interface in the intelligent monitoring platform.

4. The data analysis method based on the Akka Actor model according to claim 3, characterized in that: In step S2, the data production actor in the data production system receives the event message, delegates the sub-actor to perform the data production task, and generates data for the specified region and time.

5. The data processing system and analysis method based on the Akka Actor model according to claim 4, characterized in that: In step S3, data preprocessing specifically involves: performing preliminary processing on the acquired data; having the message receiving Actor delegate the preprocessing task to the sub-Actor for non-blocking parallel execution; and forwarding the processing result to the data analysis and processing system.

6. The data analysis method based on the Akka Actor model according to claim 5, characterized in that: In step S4, a specific data processing algorithm is applied to calculate the specified analysis content and generate structured data; Data processing algorithms include: time-series change detection for analyzing vegetation index changes during the crop growing season and detecting whether a plot of land has not been planted for a long time; Classification algorithm: Based on image classification, the land parcels are divided into agricultural land and abandoned land; Deep learning: Train a deep learning model using multiple temporal images to automatically detect abandoned land.

7. The data analysis method based on the Akka Actor model according to claim 6, characterized in that: In step S5, based on the current verification results and historical data, an algorithm is used to determine whether there are any abnormalities in the data; the processing results are forwarded to the monitoring screen display service, and the abnormal data is forwarded to the task generation service.

Images