Multi-source heterogeneous data acquisition method based on micro-service architecture

Abstract

The embodiment of the invention provides a multi-source heterogeneous data acquisition method based on a micro-service architecture, and belongs to the field of data acquisition. The method comprises the steps that an adaptive acquisition strategy algorithm is used in an acquisition service group module, acquisition frequency and acquisition time delay are adjusted based on states and data traffic of data sources so as to ensure real-time performance and stability of data acquisition, and an adaptive task scheduling algorithm based on deep learning is used in a monitoring and protection module so as to ensure real-time performance and stability of data acquisition. Tasks are intelligently distributed, and loads are dynamically migrated. By adopting a self-adaptive acquisition strategy algorithm, acquisition parameters can be dynamically adjusted according to real-time states of different data sources, data loss and acquisition delay are reduced, and a complex multi-source data environment can be better coped with. And moreover, by adopting a self-adaptive scheduling algorithm based on deep learning in the monitoring and protection module, tasks are intelligently allocated and loads are dynamically migrated, so that the system load fluctuation can be better adapted, and the task execution efficiency and the resource utilization rate are improved.

Description

Technical Field

[0001] The present invention relates to the field of data acquisition technology, and in particular to a method for acquiring multi-source heterogeneous data based on a microservice architecture. Background Art

[0002] With the development of the Internet of Things (IoT) and Industrial Internet technologies, the variety of devices and systems connected to port systems is increasing, including smart sensors, edge computing devices, automated guided vehicles (AGVs), and various PLC controllers. These devices use different communication protocols (such as Modbus, OPC-UA, HTTP / HTTPS, and MQTT) and multiple data formats (such as CSV, XML, and JSON), and contain large amounts of semi-structured and unstructured data (such as video streams, monitoring data, and real-time sensor status data). Traditional centralized, monolithic architectures struggle to effectively collect, process, and transmit this heterogeneous data, resulting in low data access efficiency, poor real-time performance, and insufficient scalability.

[0003] In addition, various business links (such as cargo loading and unloading, warehouse management, transportation scheduling and ship management) have extremely high requirements for the real-time, compatibility and reliability of data collection. The traditional centralized collection architecture is difficult to cope with the complexity of accessing and managing multi-source heterogeneous data. Therefore, the port system urgently needs a data collection technology based on a distributed architecture to flexibly adapt to the access and management of multiple data sources and improve the real-time and stability of data collection. As a modular and distributed system design pattern, the microservice architecture can split the system into multiple independent microservice modules. Each module is independently deployed and managed, and can be developed, operated and expanded without affecting other service functions. It is conducive to data interaction among multi-source heterogeneous systems in ports, thereby effectively realizing the exchange and sharing of different data. However, the existing data collection methods based on microservice architecture still face the following challenges when dealing with complex multi-source heterogeneous data: 1. Traditional data collection methods usually adopt fixed collection frequency and collection delay, lack adaptive collection strategies and dynamic scheduling mechanisms, and are unable to adjust the collection frequency and collection delay in time according to the real-time status of the data source and fluctuations in data traffic.

[0004] 2. In the multi-source heterogeneous data scenario of the port system, each data source has different collection loads and priorities. The existing collection solution lacks an effective scheduling strategy to reasonably allocate collection resources, resulting in the load balancing and data integrity of the collection nodes not being guaranteed. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a method for collecting multi-source heterogeneous data based on a microservice architecture, which is used to fully or at least partially solve the technical problems existing in the above-mentioned existing technologies, such as the lack of adaptive collection strategies and dynamic scheduling mechanisms, the inability to timely adjust the collection frequency and collection delay according to the real-time status of the data source and the fluctuation of data traffic, and the lack of effective scheduling strategies to reasonably allocate collection resources, resulting in the load balancing of collection nodes and the lack of guarantee of data integrity.

[0006] To achieve the above objectives, an embodiment of the present invention provides a method for collecting multi-source heterogeneous data based on a microservice architecture. The microservice architecture includes an entry layer, a gateway layer, a monitoring and protection module, a collection service group module, a service governance module, and a data transmission and management layer. The method for collecting multi-source heterogeneous data based on the microservice architecture includes: The collection service group module uses an adaptive collection strategy algorithm to adjust the collection frequency and collection delay in real time based on the status and data flow of each data source to ensure the real-time and stability of data collection. The data sources include but are not limited to port machinery and equipment, environmental monitoring sensors, port logistics and transportation vehicles, remote business management systems, and intelligent video surveillance systems. An adaptive task scheduling algorithm based on deep learning is adopted in the monitoring and protection module to intelligently allocate tasks and dynamically migrate loads.

[0007] Optionally, an adaptive collection strategy algorithm is used in the collection service group module to adjust the collection frequency and collection delay in real time based on the status and data flow of each data source to ensure the real-time and stability of data collection, including: The device status information and current data flow are acquired once every preset period. When the data flow is lower than the data flow threshold, the acquisition frequency is increased; when the data flow exceeds the data flow threshold, the acquisition frequency is reduced. Dynamically adjust the acquisition delay according to changes in the acquisition frequency. If the acquisition delay is greater than the acquisition delay threshold, increase the acquisition frequency. The acquisition frequency and acquisition delay are dynamically adjusted according to the abnormal type and degree of the device status, wherein the abnormal type of the device status includes abnormal device health status or abnormal device connection status.

[0008] Optionally, when the data flow rate is lower than the data flow rate threshold, the collection frequency is increased according to the following formula:

[0009] Where, Indicates the minimum acquisition frequency, Indicates the acquisition frequency adjustment coefficient, Indicates the current data traffic. Indicates the data traffic threshold.

[0010] Optionally, when the data flow exceeds the data flow threshold, the collection frequency is reduced according to the following formula:

[0011] Where, Indicates the maximum acquisition frequency, Indicates the acquisition frequency adjustment coefficient, Indicates the data flow threshold. Indicates the current data traffic.

[0012] Optionally, if the collection delay is greater than the collection delay threshold, the collection frequency is increased according to the following formula:

[0013] Where, Indicates the maximum acquisition frequency, is the acquisition delay adjustment coefficient, Indicates the acquisition delay, Indicates the collection delay threshold.

[0014] Optionally, dynamically adjust the acquisition frequency and latency based on the type and severity of the device status anomaly, including: When an abnormal device health status is detected, the collection frequency is reduced and the collection delay is increased. If the device status is seriously abnormal, the current collection task is suspended. When an abnormal device connection status is detected, the collection task is suspended; When the device connection status and device health status are restored, the collection frequency and collection delay are restored to the initial values ​​of the collection frequency and the initial values ​​of the collection delay.

[0015] Optionally, when an abnormal device health status is detected, the collection frequency is reduced according to the following formula:

[0016] Where, Indicates the degree to which the health status of the device deviates from the normal value. is the frequency adjustment coefficient.

[0017] Optionally, when an abnormal device health status is detected, increase the collection delay according to the following formula:

[0018] Where, is the acquisition delay adjustment coefficient, Indicates the degree to which the health status of the device deviates from the normal value. Indicates the maximum acquisition delay.

[0019] Optionally, an adaptive task scheduling algorithm based on deep learning is adopted in the monitoring and protection module to intelligently allocate tasks and dynamically migrate loads, including: A three-layer one-dimensional convolutional neural network is used to construct a temporal convolutional neural network model. Causal convolution and dilated convolution are used in each layer to ensure that the temporal convolutional neural network model captures the feature information in long time series. A temporal convolutional neural network model is used to predict the Q value of each action according to the current state sequence, the optimal action is selected, and the current state is updated based on the optimal action.

[0020] Optionally, the temporal convolutional neural network model predicts the Q value of each action using the following formula:

[0021] Where, and are the weights and biases of the output layer, Indicates that the output layer represents the final feature .

[0022] This technical solution utilizes an adaptive collection strategy algorithm to adjust collection frequency and latency in real time based on device status and data flow. Compared to traditional fixed collection frequency and latency methods, this approach can dynamically adjust collection parameters based on the real-time status of different data sources, reducing data loss and collection delays, and better navigating complex multi-source data environments. Furthermore, by employing a deep learning-based adaptive scheduling algorithm in the monitoring and protection module, tasks are intelligently assigned and loads are dynamically migrated. Compared to traditional static scheduling methods, this approach is more adaptable to system load fluctuations, improving task execution efficiency and resource utilization.

[0023] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present invention, but do not constitute a limitation of the embodiments of the present invention. In the accompanying drawings: Figure 1 This is a flowchart of an implementation method for collecting multi-source heterogeneous data based on a microservice architecture provided by an embodiment of the present invention; Figure 2 This is a microservice architecture diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following describes the specific implementation of the embodiment of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiment of the present invention and is not used to limit the embodiment of the present invention.

[0026] See Figure 1 FIG. 1 is a flowchart of an implementation method for collecting multi-source heterogeneous data based on a microservice architecture according to an embodiment of the present invention, including the following execution steps: Step 100: Adaptive collection strategy algorithm is used in the collection service group module to adjust the collection frequency and collection delay in real time based on the status and data flow of each data source to ensure the real-time and stability of data collection.

[0027] Among them, data sources include but are not limited to port machinery and equipment, environmental monitoring sensors, port logistics and transportation vehicles, remote business management systems, and intelligent video surveillance systems.

[0028] For example, port machinery and equipment (such as cranes, conveyors, and loading and unloading equipment) collects equipment operating status, energy consumption data, and fault logs. Through Modbus, OPC-UA, CAN bus, and other protocols, these systems collect information on equipment start and stop status, operating mode switching, operating instructions, energy consumption, fault alarm logs, control parameters (such as load, lifting height, and speed), and sensor data (such as temperature, pressure, and liquid level). Port environmental monitoring sensors (such as meteorological, sea condition, and noise sensors) collect environmental monitoring data such as wind speed and direction, temperature and humidity, air pressure, and wave height. These sensors communicate using protocols such as HTTP / HTTPS and MQTT to monitor the port environment in real time and upload sensor health status and abnormal alarm information. Port logistics and transport vehicles (such as AGVs, container trucks, and rail-mounted cranes) collect vehicle location information, dispatch instructions, and equipment health status, pre-processing and aggregation through edge nodes before uploading. This system supports real-time analysis and processing for multi-vehicle route planning, congestion detection, and task allocation. Remote business management systems (such as cargo management, ship scheduling, and warehouse management systems): These systems collect business data through API interfaces (RESTful API, SOAP), including cargo lists, loading and unloading progress, ship berthing and departure, warehouse status, business process records, operation logs, operator identity information, and business operation history records. These systems also perform data standardization, conversion, and integration. Intelligent video surveillance systems: These systems collect images and video streams from port operations using cameras, drones, and other devices. These systems perform pre-processing such as target detection and behavior recognition at edge computing nodes before uploading the data to the data collection module for further analysis and storage. The collected data includes real-time video streams, image snapshots, target detection (e.g., personnel, equipment), behavior analysis (e.g., illegal operations), video stream health status, and monitoring logs.

[0029] Specifically, when executing step 100, the following steps may be specifically performed: S1000: Obtain device status information and current data traffic once every preset period. When the data traffic is lower than the data traffic threshold, increase the acquisition frequency. When the data traffic exceeds the data traffic threshold, reduce the acquisition frequency.

[0030] Specifically, set the initial acquisition frequency , minimum acquisition frequency , maximum acquisition frequency . Define the current data flow and data traffic thresholds . Define the device status , including device health status (Indicates the current health status of the device, which can be provided by the device's internal sensors or management system, such as device temperature, voltage, current, etc.) and connection status (Indicates the connection status between the device and the acquisition module, which can be "Connection Normal" or "Connection Abnormal"). Define the acquisition delay and acquisition delay threshold . Initialization: Set the initial acquisition frequency , collection delay .

[0031] When the data flow rate is lower than the data flow rate threshold, the collection frequency is increased according to the following formula:

[0032] Where, Indicates the minimum acquisition frequency, Indicates the acquisition frequency adjustment coefficient, Indicates the current data traffic. Indicates the data traffic threshold.

[0033] When the data flow exceeds the data flow threshold, the collection frequency is reduced according to the following formula:

[0034] Where, Indicates the maximum acquisition frequency, Indicates the acquisition frequency adjustment coefficient, Indicates the data flow threshold. Indicates the current data traffic.

[0035] S1001: Dynamically adjust the acquisition delay according to the change of the acquisition frequency. If the acquisition delay is greater than the acquisition delay threshold, increase the acquisition frequency.

[0036] It should be understood that the acquisition delay .

[0037] Specifically, if the collection delay is greater than the collection delay threshold, the collection frequency is increased according to the following formula:

[0038] Where, Indicates the maximum acquisition frequency, is the acquisition delay adjustment coefficient, Indicates the acquisition delay, Indicates the collection delay threshold.

[0039] S1002: Dynamically adjust the acquisition frequency and acquisition delay according to the abnormal type and degree of the device status, wherein the abnormal type of the device status includes abnormal device health status or abnormal device connection status.

[0040] Specifically, when an abnormal device health status is detected, the collection frequency is reduced and the collection delay is increased. If the device status is seriously abnormal, the current collection task is suspended. When an abnormal device connection status is detected, the collection task is suspended. When the device connection status and device health status are restored, the collection frequency and collection delay are restored to the initial values ​​of the collection frequency and the initial values ​​of the collection delay.

[0041] For example, abnormal health status of a device may be characterized by excessively high temperature, excessively low current, or the device being offline.

[0042] More specifically, when an abnormal device health status is detected, the collection frequency is reduced according to the following formula:

[0043] Where, Indicates the degree to which the health status of the device deviates from the normal value. is the frequency adjustment coefficient.

[0044] When an abnormal device health status is detected, the collection delay is increased according to the following formula:

[0045] Where, is the acquisition delay adjustment coefficient, Indicates the degree to which the health status of the device deviates from the normal value. Indicates the maximum acquisition delay.

[0046] In some implementations, if the device status is seriously abnormal (e.g., the device is overheated), the current collection task is suspended and the collection service module is notified: . Data supplementary collection strategy: If the device status is within the set time If the system fails to recover within 10 seconds, the data re-collection strategy is triggered, the current data collection task is marked as "abnormal re-collection", and additional data is collected from other redundant devices or systems: ,in, It is the frequency adjustment coefficient of the supplementary mining task.

[0047] In some implementations, when an abnormal device connection status is detected, the collection task is suspended: , and notify the collection management service to handle the connection exception. The system automatically tries to Reconnect the device within the time interval and record the number of reconnections. Data recollection strategy: If the device If the data is not recovered successfully within 10 seconds, the system will execute the additional collection strategy to supplement the lost data. When the device health status and connection status return to normal, the system will automatically restore the collection frequency and collection delay to the initial values: , resume the collection task and adjust the task status to "normal": , returns the following parameters: real-time acquisition frequency and acquisition delay ; Current collection task status (Such as "normal", "pause", "resume", "exception handling", etc.).

[0048] Step 101: Adopting a deep learning-based adaptive task scheduling algorithm in the monitoring and protection module to intelligently allocate tasks and dynamically migrate loads.

[0049] Specifically, when executing step 101, the following steps may be specifically performed: S1010: A temporal convolutional neural network model is constructed using a three-layer one-dimensional convolutional neural network. Each layer uses causal convolution and dilated convolution to ensure that the temporal convolutional neural network model captures the feature information in long time series.

[0050] S1011: Use a temporal convolutional neural network model to predict the Q value of each action according to the current state sequence, select the optimal action, and update the current state based on the optimal action.

[0051] In some implementations, different data sources in multi-source heterogeneous data collection methods exhibit significant differences in data volume, transmission rate, and data priority, making it difficult for traditional scheduling algorithms to optimize task allocation strategies in real time within complex environments. Therefore, a deep learning-based adaptive task scheduling algorithm is employed in the monitoring and protection module. Multi-source heterogeneous data collection task scheduling is modeled as a Markov decision process (MDP), and the optimal scheduling strategy is automatically learned using deep learning models (e.g., temporal convolutional neural networks (TCNs)). All collection tasks are intelligently scheduled from a global perspective, dynamically assigned to different collection nodes, and migrated and load balanced based on system status, thereby improving the overall performance and resource utilization of the multi-source heterogeneous data collection method.

[0052] The specific algorithm flow is as follows: (1) Input parameters and state space: State space : System state variables in the form of time series: Load sequence of each acquisition node in the current system , indicating that each collection node has Load changes in time steps. Current collection task quantity sequence , indicating the past The number of tasks in the system changes within a time step. The current task priority sequence , which represents the change of the importance weight of each task in the system over time.

[0053]

[0054] Action Space Includes the following actions: assigning a new collection task to a target node .

[0055] Adjust the collection frequency of a collection node or acquisition delay Migrate some tasks of a node to a node with lower load.

[0056] Reward Function Define the success rate of task execution at the current moment , system load balancing and system response time .

[0057] Calculate instant rewards based on the above indicators

[0058] .

[0059] (2) Optimization goal: Maximize the system's acquisition task success rate and response efficiency. Minimize the execution delay and system load imbalance during task scheduling. Adaptive optimization of task scheduling strategies is achieved through temporal convolutional neural network modeling.

[0060] (3) Specific algorithm process: a. Environmental state initialization and data preprocessing: defining the system initial state sequence , including the load sequence of each acquisition node, the current task quantity sequence and the task priority sequence.,The system status data is normalized and time-series processed to form standardized time-series data input.

[0061] b. Temporal Convolutional Neural Network (TCN) model structure design: A three-layer one-dimensional convolutional neural network is used to build the TCN model. Each layer uses causal convolution and dilated convolution to ensure that the model can capture the characteristic information in long time series. The input of the TCN model is the state sequence of the system at the current moment. , the output is each action in the current state The Q value (i.e. .

[0062] The structure of the TCN network: Each layer contains multiple causal convolution kernels, and the convolution kernel size is In the present invention, the convolution kernel size is 3, the stride is 1, and the number of convolution kernels is 32, 64, 128, and 256. Each convolution layer uses residual connection to improve the stability of gradient transfer. Expansion coefficient As the number of layers increases, it increases exponentially (e.g. The expansion coefficient of the layer is .

[0063] c. State feature extraction and Q value calculation: for state sequence Perform convolution operation to extract feature representation at each moment:

[0064] in, and For the The weights and bias parameters of the convolutional layers, For the The input of the layer, .

[0065] The output layer represents the final feature Converted into actions Q value:

[0066] in, and are the weights and biases of the output layer.

[0067] d. Strategy update and optimization: Use experience replay technology to record the quadruple of historical state, action, reward and next state .

[0068] Use the target network to stabilize policy updates:

[0069] Update policy network parameters:

[0070] in, is the learning rate, is the discount factor, are policy network parameters.

[0071] e. Dynamic scheduling strategy generation: based on the current state , use the TCN model to predict the Q value of each action , choose the best action

[0072]

[0073] Execute the best action , and update the system status.

[0074] f. Strategy optimization and model convergence: Through multiple rounds of training and strategy updates, the model continuously learns task scheduling strategies from historical experience and ultimately obtains a stable task scheduling strategy.

[0075] By adopting a deep learning-based adaptive scheduling algorithm in the monitoring and protection module, tasks are intelligently allocated and loads are dynamically migrated. Compared with traditional static scheduling methods, it can better adapt to system load fluctuations and improve task execution efficiency and resource utilization.

[0076] In some implementations, the microservices architecture design can achieve unified collection of heterogeneous data from multiple sources by supporting parsing and processing of multiple protocols (such as Modbus, OPC-UA, HTTP / HTTPS, MQTT) and multiple data formats (such as CSV, XML, JSON). Each collection program exists in the form of an independent module and can flexibly adapt to the collection needs of different types of data sources (such as industrial equipment, sensor devices, edge computing nodes and remote data interfaces). Figure 2As shown in the figure, the system consists of an entry layer, a gateway layer, a monitoring and protection module, a collection service group module, a service governance module, and a data transmission and management layer. 1) The entry layer, as the system's external request entry point, is responsible for load balancing and forwarding all external access requests, distributing them to internal microservice modules to ensure system stability and reliability in high-concurrency scenarios. The entry layer uses a reverse proxy function to route requests from external clients to the system's gateway layer, providing basic access control and request distribution. Furthermore, the entry layer evenly distributes external requests through a load balancing mechanism to prevent overload in a single microservice module. 2) The gateway layer receives and manages requests from the entry layer, forwarding them to the corresponding collection service group based on request type (such as device data collection requests, sensor status requests, business data management requests, etc.). The gateway layer is also responsible for request authorization authentication, flow control, and routing management. The gateway layer ensures the security of data collection requests through access control mechanisms and performs flow control for different types of requests. Based on the specific content of the request, it is assigned to the corresponding collection service (such as Service A, Service B, Service C, etc.), achieving effective management and distribution of multi-source heterogeneous data. 3) The Monitoring and Protection Module monitors the execution status of all collection tasks and services, responsible for real-time fault detection and handling. When system anomalies occur or the load is excessive, it triggers appropriate protection strategies and analyzes and adjusts the system status. This module is used to implement full-link monitoring and fault recovery using tools such as Skywalking and Hystrix. The system automatically triggers circuit breaking strategies when an anomaly is detected, preventing the fault from spreading to other modules and ensuring the stability and continuity of the collection service. Furthermore, the monitoring module automatically adjusts system load distribution to avoid blocking or interrupting the execution of collection tasks. 4) The Collection Service Group Module, the core data collection module, contains multiple collection service instances responsible for collecting tasks from different data sources (such as industrial equipment, environmental monitoring equipment, and business systems). Each service group contains multiple collection services of the same type (such as Service A, Service B, and Service C), and can be dynamically scaled up or down based on actual business needs. Examples of collection services: Service A: Collects industrial equipment data (such as PLCs and sensors), using industry-standard communication protocols (such as Modbus and OPC-UA) to obtain equipment operating status, sensor data, and fault logs. Service B: Collects environmental monitoring data (such as temperature, humidity, and air pressure) to monitor the safety status of the port operating environment in real time. Service C: Collects business system data (such as warehouse management and logistics management) through API interfaces, providing real-time updates of business data such as cargo loading and unloading, ship scheduling, and inventory management. Purpose: Independently executes collection tasks for various data sources and transmits the collected data to data processing or storage modules through message middleware or direct communication, ensuring the effective collection and management of multi-source heterogeneous data.5) The service governance module manages the status of microservices through tools such as Nacos, including service registration, service discovery, configuration management, and fault recovery. When a collection service instance fails, it automatically triggers a restart or scaling operation to ensure high system availability. Function: Monitors the health of each collection service instance, automatically handles or issues alerts when anomalies are detected, and dynamically scales the number of collection service instances based on the system load, ensuring smooth system operation even under high load. 6) The data transmission and management layer is responsible for data communication and management between the collection service and the data processing service, ensuring data consistency and reliability across all stages of collection, transmission, storage, and processing. It includes: a. Message middleware: Used for data communication between the collection service and the processing service, it supports asynchronous messaging and ensures efficient data transmission. b. Distributed file system: Manages and stores unstructured data (such as logs, images, and videos), supporting distributed storage and access of large-scale data. c. Caching: Improves data access speed, reduces database access pressure, and reduces data query latency. d. Database: Manages and stores structured data (such as device status, business data, and operation logs) to provide support for data processing and business decision-making. e. Data processing: Cleans, converts, analyzes, and stores collected data to provide high-quality data support for subsequent data analysis and business decisions. Process Overview: a. Each collection service group collects data from different data sources and passes the data to the data processing component through message middleware or direct connection. b. The data processing component pre-processes the raw data (such as data cleaning, format conversion, and data analysis), and then stores the processed data in the database or cache. c. If further analysis or mining is required, the data processing module can pass the data to the advanced analysis module for data mining and model analysis to support business decision-making and task scheduling.

[0077] By integrating the functions of the aforementioned modules with the architectural design, we can effectively manage the collection and processing of heterogeneous data from multiple sources. The system boasts excellent scalability and maintainability, and provides efficient collection and processing capabilities in complex and changing data environments. Furthermore, the service governance module and monitoring and protection module ensure system stability and high availability under high load and abnormal scenarios.

[0078] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0082] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0083] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0084] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0086] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for collecting multi-source heterogeneous data based on a microservice architecture, wherein the microservice architecture includes an entry layer, a gateway layer, a monitoring and protection module, a collection service group module, a service governance module, and a data transmission and management layer, characterized in that: The method for collecting multi-source heterogeneous data based on the microservice architecture includes: The collection service group module uses an adaptive collection strategy algorithm to adjust the collection frequency and collection delay in real time based on the status and data flow of each data source to ensure the real-time and stability of data collection. The data sources include but are not limited to port machinery and equipment, environmental monitoring sensors, port logistics and transportation vehicles, remote business management systems, and intelligent video surveillance systems. An adaptive task scheduling algorithm based on deep learning is adopted in the monitoring and protection module to intelligently allocate tasks and dynamically migrate loads.

2. The method for collecting multi-source heterogeneous data based on microservice architecture according to claim 1 is characterized in that: The collection service group module uses an adaptive collection strategy algorithm to adjust the collection frequency and collection delay in real time based on the status and data flow of each data source to ensure the real-time and stability of data collection, including: The device status information and current data flow are acquired once every preset period. When the data flow is lower than the data flow threshold, the acquisition frequency is increased; when the data flow exceeds the data flow threshold, the acquisition frequency is reduced. Dynamically adjust the acquisition delay according to changes in the acquisition frequency. If the acquisition delay is greater than the acquisition delay threshold, increase the acquisition frequency. The acquisition frequency and acquisition delay are dynamically adjusted according to the abnormal type and degree of the device status, wherein the abnormal type of the device status includes abnormal device health status or abnormal device connection status.

3. The method for collecting multi-source heterogeneous data based on microservice architecture according to claim 2 is characterized in that: When the data flow rate is lower than the data flow rate threshold, the collection frequency is increased according to the following formula: Where, Indicates the minimum acquisition frequency, Indicates the acquisition frequency adjustment coefficient, Indicates the current data traffic. Indicates the data traffic threshold.

4. The method for collecting multi-source heterogeneous data based on microservice architecture according to claim 2 is characterized in that: When the data flow exceeds the data flow threshold, the collection frequency is reduced according to the following formula: Where, Indicates the maximum acquisition frequency, Indicates the acquisition frequency adjustment coefficient, Indicates the data flow threshold. Indicates the current data traffic.

5. The method for collecting multi-source heterogeneous data based on microservice architecture according to claim 2 is characterized in that: If the collection delay is greater than the collection delay threshold, the collection frequency is increased according to the following formula: Where, Indicates the maximum acquisition frequency, is the acquisition delay adjustment coefficient, Indicates the acquisition delay, Indicates the collection delay threshold.

6. The method for collecting multi-source heterogeneous data based on microservice architecture according to claim 2 is characterized in that: Dynamically adjust the acquisition frequency and latency based on the type and severity of device status anomalies, including: When an abnormal device health status is detected, the collection frequency is reduced and the collection delay is increased. If the device status is seriously abnormal, the current collection task is suspended. When an abnormal device connection status is detected, the collection task is suspended; When the device connection status and device health status are restored, the collection frequency and collection delay are restored to the initial values ​​of the collection frequency and the initial values ​​of the collection delay.

7. The method for collecting multi-source heterogeneous data based on microservice architecture according to claim 6 is characterized in that: When an abnormal device health status is detected, the collection frequency is reduced according to the following formula: Where, Indicates the degree to which the health status of the device deviates from the normal value. is the frequency adjustment coefficient.

8. The method for collecting multi-source heterogeneous data based on microservice architecture according to claim 6 is characterized in that: When an abnormal device health status is detected, the collection delay is increased according to the following formula: Where, is the acquisition delay adjustment coefficient, Indicates the degree to which the health status of the device deviates from the normal value. Indicates the maximum acquisition delay.

9. The method for collecting multi-source heterogeneous data based on microservice architecture according to claim 1, characterized in that: The monitoring and protection module uses an adaptive task scheduling algorithm based on deep learning to intelligently allocate tasks and dynamically migrate loads, including: A three-layer one-dimensional convolutional neural network is used to construct a temporal convolutional neural network model. Causal convolution and dilated convolution are used in each layer to ensure that the temporal convolutional neural network model captures the feature information in long time series. A temporal convolutional neural network model is used to predict the Q value of each action according to the current state sequence, the optimal action is selected, and the current state is updated based on the optimal action.

10. The method for collecting multi-source heterogeneous data based on microservice architecture according to claim 9, characterized in that: The temporal convolutional neural network model uses the following formula to predict the Q value of each action: Where, and are the weights and biases of the output layer, Indicates that the output layer represents the final feature .

Images