Automatically configured data acquisition and sending method, apparatus and device, and computer program product

Abstract

The invention discloses an automatic configuration data acquisition and sending method, device and equipment and a computer program product, and the method comprises the steps: initializing a configuration center client of a probe based on private configuration information of the probe, and enabling the configuration center client used by the probe to be isolated from a configuration center client used by a target application; obtaining target configuration data for data acquisition and transmission through a configuration center client of the probe and a predefined configuration center namespace; creating a message sending client according to the target configuration data; and carrying out data acquisition and sending by using a predefined data transmission strategy through a message sending client. According to the method, the independent configuration center client is initialized based on the private configuration, the isolation of the probe and the target application configuration management is realized, and the security, independence and flexibility of the configuration are improved. Data transmission is carried out through the predefined data transmission strategy, so that the fault tolerance of data transmission and the continuity of services are guaranteed.

Description

Technical Field

[0001] This application relates to the field of data acquisition and transmission technology, and in particular to an automated configuration method, apparatus and equipment, and computer program product for data acquisition and transmission. Background Technology

[0002] In the current wave of technological development, distributed microservice architecture has become the preferred architectural pattern for building many large and complex systems due to its high flexibility, scalability, and loose coupling. This architecture breaks down a large application system into multiple small, autonomous services, each of which can be independently developed, deployed, and operated, interacting through lightweight communication mechanisms (such as HTTP, RPC, etc.), thereby better adapting to rapid changes and iterations in business operations.

[0003] However, with the widespread adoption of distributed microservice architectures, system scale is constantly expanding, the number of services is increasing dramatically, and data flow within the system is becoming increasingly complex and frequent. To ensure stable system operation, promptly identify potential problems, and optimize performance, comprehensive, accurate, and real-time collection and transmission of various data within the system has become crucial. This data covers multiple aspects, including service operational status, performance metrics, and business transaction information, serving as a vital basis for system monitoring, analysis, and decision-making.

[0004] In a distributed microservice architecture, data collection and transmission typically rely on specialized probes. These probes, acting as the "tentacles" of data collection, are deployed on various service nodes to gather local data and send it to a designated data storage or processing center (such as a message queue system like Kafka) for subsequent analysis and processing. Therefore, efficiently managing these data collection and transmission probes and ensuring their stable and reliable operation has become a critical challenge in distributed microservice architectures.

[0005] Currently, several technical solutions exist for data collection and transmission in distributed microservice architectures. For example, one existing technology provides a method for end-to-end monitoring of distributed services, aiming to monitor the entire service chain and obtain system operating status and performance indicators through tracing and analyzing service call chains. Another example is a financial business monitoring method and system, primarily targeting financial business scenarios, monitoring and analyzing business transaction data to ensure the secure and stable operation of financial services.

[0006] While these existing technical solutions meet the needs of distributed service monitoring and business monitoring to some extent, they still have the following shortcomings in terms of data collection and probe management:

[0007] (1) Inconvenient configuration modification: When it is necessary to modify the configuration of the probe connecting to Kafka, it is necessary to log in to the machine where the probe is located and manually modify the local configuration file. This operation method is not only cumbersome, but also extremely inefficient for large-scale distributed systems, as it requires repeated operations on multiple nodes.

[0008] (2) Delay in taking effect: After modifying the configuration file, the probe needs to be restarted for the new configuration to take effect. Restarting the probe will cause a brief interruption in data collection and transmission, resulting in a slow response speed, which cannot meet the needs of business scenarios with high real-time requirements.

[0009] (3) Low operational security: There are certain security risks in logging into the machine to modify the configuration file and restart the probe. The system may fail due to misoperation or configuration error, affecting the continuity of business.

[0010] (4) Low management efficiency: The existing probe management method is relatively inefficient in managing probes, and cannot quickly respond to the needs of business changes and system adjustments. It is difficult to meet the requirements of efficient management of data collection and sending probes under the distributed microservice architecture.

[0011] To address the shortcomings of the existing technical solutions, this invention aims to provide an automated configuration data acquisition and transmission method and apparatus, thereby solving problems such as inconvenient configuration modification, delayed effectiveness, low operational security, and low management efficiency in the prior art. Specifically, the objectives of this invention include: Summary of the Invention

[0012] This application provides an automated configuration method, apparatus, and computer program product for data acquisition and transmission, which enables automated configuration of probes during the data acquisition and transmission process, improves operational efficiency, and ensures business continuity.

[0013] The embodiments of this application adopt the following technical solutions:

[0014] In a first aspect, embodiments of this application provide an automated configuration data acquisition and transmission method, the automated configuration data acquisition and transmission method comprising:

[0015] During the probe startup process for data acquisition and transmission, the probe's configuration center client is initialized based on the probe's private configuration information. The probe and the target application run in the same virtual machine environment, and the configuration center client used by the probe and the configuration center client used by the target application are isolated from each other.

[0016] The target configuration data for data collection and transmission is obtained through the configuration center client of the probe and the predefined configuration center namespace;

[0017] Create and maintain a message sending client based on the target configuration data;

[0018] The message sending client uses a predefined data transmission strategy to collect and send data.

[0019] Optionally, the mutual isolation includes dependency isolation, and the initialization of the probe's configuration center client based on the probe's private configuration information during the probe startup process for data acquisition and transmission includes:

[0020] When packaging the probe, the package names of the relevant dependency files of the probe's configuration center client are replaced so that the package path of the related dependency files after the package name replacement is different from the package path of the corresponding dependency files used in the target application.

[0021] Optionally, the mutual isolation includes configuration isolation, and the initialization of the probe's configuration center client based on the probe's private configuration information during the probe's data acquisition and transmission startup process includes:

[0022] Read the namespace identifier of the configuration center defined in the private configuration information of the probe;

[0023] The namespace identifier of the read configuration center is set to the system property through static synchronization code. The system property is the method of reading the namespace identifier of the configuration center with the highest priority.

[0024] The configuration center client of the probe is created based on the namespace identifier of the configuration center in the system properties;

[0025] After the configuration center client of the probe is created, restore the original namespace identifier of the configuration center in the system properties.

[0026] Optionally, the mutual isolation includes configuration isolation, and the initialization of the probe's configuration center client based on the probe's private configuration information during the probe's data acquisition and transmission startup process includes:

[0027] Based on the service provider interface mechanism, a class that implements the service provider interface defined by the configuration center is provided, and the class of the service provider interface is registered as an implementation class. The service provider interface is used to obtain the access address of the configuration server node.

[0028] When initializing the configuration center client for the probe, the registered implementation class is loaded;

[0029] The implementation class reads the access address of the configuration server node from the private configuration information of the probe.

[0030] Optionally, the predefined configuration center namespace includes a public namespace and at least one private namespace, and the step of obtaining the target configuration data for data collection and transmission through the probe's configuration center client and the predefined configuration center namespace includes:

[0031] The configuration center client of the probe loads public configuration data from the public namespace;

[0032] The configuration center client of the probe loads private configuration data from at least one of the private namespaces.

[0033] Optionally, creating and maintaining the message sending client based on the target configuration data includes:

[0034] A first message sending client is created based on the target configuration data, which serves as the currently active message sending client.

[0035] Listen for configuration change notifications in the configuration center namespace;

[0036] Upon receiving the configuration change notification, update the configuration of the first message sending client or create a second message sending client;

[0037] Perform a hot reload on the updated first message sending client or the second message sending client.

[0038] Optionally, the predefined data transmission strategy includes an asynchronous transmission strategy, and the step of collecting and sending data using the predefined data transmission strategy through the message sending client includes:

[0039] Build a thread pool in the delegate class of the message sending client;

[0040] Modify the send method in the delegate class of the message sending client to submit the message sending task to the thread pool for asynchronous sending.

[0041] Optionally, the predefined data transmission strategy includes a dual-send strategy, and the step of collecting and sending data using the predefined data transmission strategy through the message sending client includes:

[0042] Construct a multi-channel sending proxy class, which is used to proxy multiple independent message sending clients;

[0043] The multi-channel sending proxy class calls the sending methods of multiple independent message sending clients to send the same data through multiple independent message sending clients.

[0044] Secondly, embodiments of this application also provide an automated configuration data acquisition and transmission device, the automated configuration data acquisition and transmission device comprising:

[0045] An initialization unit is used to initialize the configuration center client of the probe based on the probe's private configuration information during the probe startup process of data acquisition and transmission. The probe and the target application run in the same virtual machine environment, and the configuration center client used by the probe and the configuration center client used by the target application are isolated from each other.

[0046] The acquisition unit is used to acquire target configuration data for data collection and transmission through the configuration center client of the probe and the predefined configuration center namespace;

[0047] A creation unit is used to create and maintain a message sending client based on the target configuration data;

[0048] The sending unit is used to collect and send data through the message sending client using a predefined data transmission strategy.

[0049] Thirdly, embodiments of this application also provide an apparatus, comprising:

[0050] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned automated configurations of the data acquisition and transmission method.

[0051] Fourthly, embodiments of this application also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the data acquisition and transmission method of any of the aforementioned automated configurations.

[0052] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: The automated configuration data collection and sending method of the embodiments of this application first initializes the configuration center client of the probe based on the private configuration information of the probe during the probe startup process of data collection and sending. The probe and the target application run in the same virtual machine environment, and the configuration center client used by the probe and the configuration center client used by the target application are isolated from each other. Then, the target configuration data for data collection and sending is obtained through the configuration center client of the probe and the predefined configuration center namespace. After that, a message sending client is created and maintained according to the target configuration data. Finally, the data is collected and sent through the message sending client using a predefined data transmission strategy. The automated configuration data collection and sending method of the embodiments of this application, by initializing an independent configuration center client based on private configuration when the probe starts, realizes the isolation of the configuration management of the probe and the target application, improves the security and independence of the configuration, and can dynamically adjust the collection and sending parameters without modifying the probe code, thereby improving the flexibility of the configuration. By using predefined data transmission strategies for data collection and transmission, the fault tolerance and reliability of data transmission are effectively improved, ensuring business continuity and data integrity, while reducing the risk of data loss. This provides an efficient, reliable, and flexible solution for data collection and transmission in distributed systems. Attached Figure Description

[0053] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0054] Figure 1 This is a flowchart illustrating an automated configuration data acquisition and transmission method according to an embodiment of this application;

[0055] Figure 2 This is a schematic diagram of the structure of an automated data acquisition and transmission device according to an embodiment of this application;

[0056] Figure 3 This is a schematic diagram of the structure of a device according to an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0059] This application provides an automated configuration method for data acquisition and transmission, such as... Figure 1 The diagram illustrates a flowchart of an automated configuration data acquisition and transmission method according to an embodiment of this application. The automated configuration data acquisition and transmission method includes the following steps S110 to S140:

[0060] Step S110: During the probe startup process for data acquisition and transmission, the configuration center client of the probe is initialized based on the probe's private configuration information. The probe and the target application run in the same virtual machine environment, and the configuration center client used by the probe and the configuration center client used by the target application are isolated from each other.

[0061] When a probe for data collection and transmission starts, it first retrieves relevant configuration information from its own private configuration information source. This private configuration information may be stored in a probe-specific configuration file (such as a custom JSON, XML, or properties file) or may be passed in through a specific configuration initialization interface. For example, the probe may pass private configuration information as a parameter in the startup script or retrieve it by reading a configuration file from a specific local configuration directory.

[0062] Based on the obtained private configuration information, the probe initializes its configuration center client. The private configuration information may contain key parameters such as the configuration center's address, authentication information (such as username, password, access token, etc.), and connection timeout.

[0063] The probe and the target application run in the same virtual machine (JVM) environment, but the configuration center client used by the probe and the configuration center client used by the target application need to be isolated to avoid mutual interference. For example, create a configuration center client instance for the probe that is different from the target application's instance. Each instance uses independent configuration parameters and connection pools to ensure that they do not interfere with each other when accessing the configuration center, and each obtains and manages its own configuration data.

[0064] Step S120: Obtain target configuration data for data collection and transmission through the configuration center client of the probe and the predefined configuration center namespace.

[0065] The probe connects to the configuration center service through a pre-initialized configuration center client. Simultaneously, it uses predefined configuration center namespaces to locate the required configuration data. A namespace is a mechanism in the configuration center used to classify and isolate configuration data; different namespaces can store configurations for different business operations or environments.

[0066] The configuration center client retrieves target configuration data for data collection and transmission based on a predefined namespace. This target configuration data may include key information such as the Kafka cluster address, topic name, message format, collection frequency, and transmission timeout.

[0067] Step S130: Create and maintain a message sending client based on the target configuration data.

[0068] The probe creates a message sending client based on the acquired target configuration data. For example, if the target configuration data specifies using Kafka as the message middleware, the probe will create a Kafka producer client based on the relevant Kafka configuration (such as server address, port, authentication information, etc.) and maintain it. This includes monitoring the client's running status (such as whether the connection is normal, whether any exceptions have occurred, etc.), dynamically adjusting the client's parameters, and periodically refreshing the configuration to ensure that the message sending client can work stably and efficiently.

[0069] Step S140: Data is collected and sent using the message sending client and a predefined data transmission strategy.

[0070] The probe collects data from the target application or system according to a predefined data collection strategy (such as timed collection, event-triggered collection, etc.), for example, it can collect system link data. After collecting the data, the probe uses a pre-created and maintained message sending client to send the data according to a predefined data transmission strategy, such as an asynchronous multi-send strategy. The asynchronous multi-send strategy means that the probe sends the collected data to two different message queues simultaneously (such as two different Kafka clusters) to improve the reliability and fault tolerance of data transmission.

[0071] The automated configuration data acquisition and transmission method of this application, by initializing an independent configuration center client based on private configuration when the probe starts, achieves isolation between the probe and the target application configuration management, improving configuration security and independence. It allows for dynamic adjustment of acquisition and transmission parameters without modifying the probe code, enhancing configuration flexibility. Through predefined data transmission strategies, it effectively improves the fault tolerance and reliability of data transmission, ensuring business continuity and data integrity while reducing the risk of data loss. This provides an efficient, reliable, and flexible solution for data acquisition and transmission in distributed systems.

[0072] In some embodiments of this application, the mutual isolation includes dependency isolation. The initialization of the probe's configuration center client based on the probe's private configuration information during the probe's startup process for data collection and transmission includes: when packaging the probe, replacing the package names of the relevant dependency files of the probe's configuration center client so that the package path of the related dependency files after package name replacement is different from the package path of the corresponding dependency files used in the target application.

[0073] Since the probe and the target application run in the same JVM environment, using configuration center client dependency files with the same package path may cause conflicts during class loading and other processes, affecting the normal operation of their respective functions. Therefore, dependency isolation is required, specifically through package name substitution.

[0074] Choose a suitable packaging tool, such as Maven or Gradle. Taking Maven as an example, use the shade plugin to perform the package name replacement operation. The shade plugin is a commonly used plugin in Maven for handling dependency conflicts and resource renaming. It can repackage and rename specific dependencies in the project, thereby changing the package path of the dependency files. Execute the packaging command (such as Maven's `mvn package` command). At this time, the shade plugin will start working according to the configuration information. It will scan the dependencies in the project, find the relevant dependency files of the specified configuration center client, and repackage and replace the package names of these dependency files according to the set rules. After the replacement is complete, a new package file is generated, which contains the configuration center client dependency files with the replaced package names. The package path of this file is different from the corresponding dependency file package path in the target application, thus achieving dependency isolation.

[0075] During the probe's data collection and transmission process, the probe reads its own private configuration information, which includes connection information for the configuration center. Based on this private configuration information, the probe initializes the configuration center client using a repackaged configuration center client dependency file with a different package name. Because the package names are different, the probe's configuration center client and the configuration center client in the target application do not interfere with each other during class loading and runtime, achieving mutual isolation.

[0076] This embodiment of the application achieves isolation between the probe and the target application regarding their dependencies on the configuration center client by replacing the package names of the dependency files related to the probe's configuration center client. This dependency isolation method effectively avoids class conflicts that may arise when both applications use configuration center client dependencies with the same package path within the same JVM environment, ensuring that the probe and the target application can run independently and stably. Simultaneously, this isolation mechanism allows the probe and the target application to use different versions or configurations of the configuration center client, improving the system's flexibility and scalability.

[0077] In some embodiments of this application, the mutual isolation includes configuration isolation. During the probe startup process for data acquisition and transmission, initializing the probe's configuration center client based on the probe's private configuration information includes: reading the namespace identifier of the configuration center defined in the probe's private configuration information; setting the read namespace identifier of the configuration center to a system property through static synchronization code, wherein the system property is a method for reading the namespace identifier of the configuration center with the highest priority; creating the probe's configuration center client based on the namespace identifier of the configuration center in the system property; and restoring the original namespace identifier of the configuration center in the system property after the configuration center client is created.

[0078] Since the configuration discovery logic reads the namespace identifier (appId) of the configuration center sequentially from a fixed location when the configuration center client is created, and the reading order is systemProperty, system environment variables, springboot configuration file and app.properties configuration file, in order to prevent the probe from reading the configuration center configuration used by the target application and to achieve configuration isolation, it is necessary to ensure that the probe uses its own independent namespace identifier to create the configuration center client.

[0079] During probe startup, the predefined namespace identifier (appId) of the configuration center is first read from the probe's private configuration information. This private configuration information can be stored in a probe-specific configuration file, and the probe obtains the namespace identifier through the corresponding configuration reading tool.

[0080] Since the highest priority way for the configuration center client to read the namespace identifier during creation is from `systemProperty`, static synchronization code is used to set the read probe's namespace identifier into the system property `systemProperty`. The use of static synchronization code ensures that setting system properties is thread-safe in a multi-threaded environment, avoiding data races and inconsistencies, and ensuring that only one thread can perform system property setting operations at any given time.

[0081] Based on the probe's namespace identifier set in system properties, a configuration center client for the probe is created. During initialization, the configuration center client follows the preset configuration discovery logic, first reading the namespace identifier from systemProperty. Since systemProperty has already been set to the probe's namespace identifier at this point, it ensures the creation of a configuration center client that uses the probe's independent configuration.

[0082] After the probe's configuration center client is created, in order not to affect the normal operation of other parts of the system (such as functions that depend on the namespace identifier in the original systemProperty), the system properties can be restored to the original namespace identifier of the configuration center.

[0083] Furthermore, probes can use custom class loaders to further isolate themselves from the target application's appId through static synchronized code during class loading. Custom class loaders can control the class loading process, ensuring that when the probe loads classes related to the configuration center, it is not affected by classes with the same name in the target application, thus guaranteeing the integrity of configuration isolation.

[0084] This application's embodiment first reads the configuration center namespace identifier from its own private configuration during the probe startup process, and then uses static synchronization code to set it to the `systemProperty` with the highest read priority, thereby creating an independent configuration center client. After creation, the system properties are restored. Combined with a custom class loader and static synchronization code during class loading, this achieves effective isolation between the probe and the target application in terms of configuration center configuration. This configuration isolation method avoids configuration conflicts that may result from shared configurations, ensuring that the probe can perform data collection and transmission operations according to its own independent configuration, thus improving system stability and reliability. Furthermore, this solution does not require large-scale modifications to the underlying logic of the configuration center; it achieves configuration isolation at a low cost by performing reasonable configuration operations and class loading control during the probe startup process. It has good scalability and compatibility, and can adapt to system environments of varying complexity.

[0085] In some embodiments of this application, the mutual isolation includes configuration isolation. During the probe startup process for data acquisition and transmission, initializing the probe's configuration center client based on the probe's private configuration information includes: providing a class that implements the service provider interface defined by the configuration center based on the service provider interface mechanism, and registering the service provider interface class as an implementation class. The service provider interface is used to obtain the access address of the configuration server node. When initializing the probe's configuration center client, the registered implementation class is loaded; and the access address of the configuration server node is read from the probe's private configuration information through the implementation class.

[0086] The meta URL of the configuration center is the initial access address for obtaining the actual location information of the configuration server node when the client starts up; it acts as a bootstrapping entry point. The configuration center provides discovery logic for eight meta URLs, but does not support direct setting via API. Simultaneously, to meet the configuration isolation requirements between the probe and the target application, it is necessary to ensure that the probe can use its own independent configuration to obtain the access address of the configuration server node, avoiding reading the configuration used by the target application. The configuration center supports the SPI (Service Provider Interface) mechanism for customizing URL discovery logic, providing a feasible way to achieve configuration isolation for probes.

[0087] Based on the service provider interface (Meta Server Provider) defined by the configuration center, develop a dedicated implementation class for obtaining the access address of the configuration server node. This implementation class needs to implement the corresponding methods according to the interface specification. Its core function is to read the access address (i.e., metaURL) of the configuration server node from the probe's private configuration information. Using the SPI mechanism, register the class implementing the configuration center service provider interface as an implementation class. This way, during the configuration center client initialization, this implementation class can be discovered and loaded via the SPI mechanism.

[0088] After loading the implementation class, the configuration center client calls the method in the implementation class to read the access address of the configuration server node from the probe's private configuration information, thus obtaining the probe's independent meta URL. This meta URL will be used to guide the configuration center client to discover the real configuration service cluster, thereby ensuring that the probe can perform subsequent data collection and sending operations based on its own independent configuration, achieving configuration isolation from the target application.

[0089] This application embodiment utilizes the SPI mechanism supported by the configuration center to implement a service provider interface class specifically for obtaining the access address of the configuration server node from the probe's private configuration information, and registers this class as an implementation class. When the probe starts and initializes the configuration center client, this implementation class is loaded to obtain an independent meta URL, thereby achieving effective isolation between the probe and the target application in terms of configuration center configuration. This scheme avoids configuration conflicts that may occur when the probe reads the target application's configuration, ensuring that the probe can accurately obtain configuration server node information according to its own independent configuration and stably perform data collection and transmission operations.

[0090] In some embodiments of this application, the predefined configuration center namespace includes a public namespace and at least one private namespace. Obtaining target configuration data for data collection and transmission through the probe's configuration center client and the predefined configuration center namespace includes: loading public configuration data from the public namespace through the probe's configuration center client; and loading private configuration data from at least one of the private namespaces through the probe's configuration center client.

[0091] Two types of namespaces are predefined in the configuration center: one public namespace and at least one private namespace. The public namespace is used to store common configuration data that all probe instances need to share, such as some common parameter settings and global rules. The private namespace is used to store private configuration data unique to specific systems or probe instances. The different namespaces are independent of each other to ensure that the configurations of different systems or probe types do not interfere with each other.

[0092] In the configuration file of each probe, set a special configuration item `using_namespace.xxx`, where `xxx` is the system ID of this system, such as `app.business_id` in the probe configuration file. The value of this configuration item specifies which private namespace the probe instance should load private configuration data from. For example, if the system ID is 111, and you want this system to load the configuration of the namespace `ns111`, then set `using_namespace.111=ns111` in the probe configuration file. If no special settings are made, by default, all namespaces are `application`, meaning all probes load the configuration of the `application` namespace.

[0093] During the probe's data acquisition and transmission startup process, the probe's configuration center client first connects to the configuration center and loads common configuration data from the common namespace. This common configuration data provides unified parameter and rule support for the probe's basic operation.

[0094] The configuration center client determines the private namespace from which to load private configuration data based on the `using_namespace.xxx` configuration item in the probe configuration file. Then, the configuration center client loads the private configuration data from that specified private namespace. For example, if `using_namespace.111=ns111`, the configuration center client loads system-specific private configuration data from the `ns111` namespace.

[0095] This application embodiment achieves the goal of enabling probes on different instances to load different configurations by dividing the configuration center into public and private namespaces and flexibly specifying the private namespace using special configuration items in the probe configuration file. This scheme allows for centralized management and sharing of public configuration data, improving configuration reusability and consistency; at the same time, the private namespace provides independent configuration space for different systems or different types of probes, ensuring the personalization and independence of configurations and avoiding configuration conflict issues.

[0096] In some embodiments of this application, creating and maintaining a message sending client based on the target configuration data includes: creating a first message sending client based on the target configuration data as the currently active message sending client; listening for configuration change notifications in the configuration center namespace; updating the configuration of the first message sending client or creating a second message sending client when the configuration change notification is detected; and performing hot reload on the updated first message sending client or the second message sending client.

[0097] After obtaining the target configuration data, a first message sending client is created based on this configuration data and set as the currently active message sending client. For example, if the target configuration data specifies the use of Kafka as the message sending middleware and includes configuration parameters such as the Kafka server address, port, and authentication information, then a Kafka message sending client instance is created based on these parameters. This instance is the first message sending client and will be responsible for the actual data sending tasks thereafter.

[0098] After the probe starts, it uses the configuration center client's functionality to monitor configuration changes in the configuration center namespace. The configuration center client establishes a persistent connection with the configuration center to monitor configuration data changes in real time. Once the configuration data in the configuration center is updated, the configuration center will proactively send a configuration change notification to the client. For example, when operations personnel modify Kafka-related configuration parameters in the configuration center, such as adding a Kafka server node or changing the message sending partitioning strategy, the configuration center will immediately send a change notification to all probe clients that have subscribed to the relevant namespace.

[0099] Upon receiving a configuration change notification, determine if updating the existing first message sending client's configuration can accommodate the new requirements. If the changed configuration parameters support dynamic updates, such as the Kafka client's send buffer size or message compression method, then directly update the corresponding configuration on the first message sending client. After the update, the first message sending client continues to be used as the currently active client. If the configuration change involves situations where updating the original Kafka client's configuration is not possible, such as when the connection between the original first message sending client and the Kafka server is interrupted and cannot be restored, then a second message sending client needs to be created.

[0100] To achieve smooth client switching, a Kafka client delegate class is created using the delegate design pattern. This delegate class proxies the actual Kafka client, providing a unified message sending interface to the outside world.

[0101] When hot reloading of the updated first or second message sending client is required, the synchronous method `switchTo` of the delegate class is called. In the `switchTo` method, the `client` reference maintained internally by the delegate class is first replaced with the new client instance (i.e., the updated first or second message sending client), ensuring that subsequent message sending operations use the new client. Then, the old client is shut down with a set shutdown timeout, such as 5 seconds, to ensure that the old client can release resources properly and avoid resource leaks.

[0102] To improve the fault tolerance of configuration changes, in addition to the configuration center proactively pushing configuration change notifications, the configuration center client will also proactively pull the latest configuration every 5 minutes. If the push fails due to network issues or other reasons during the previous process, the periodic proactive retrieval ensures that the probe client can obtain the latest configuration data in a timely manner, guaranteeing the consistency and accuracy of the configuration.

[0103] This application embodiment creates an initial message sending client and listens for configuration change notifications in real time. This allows for flexible updates to existing client configurations or the creation of new clients when configurations change. Furthermore, it utilizes a delegated design pattern and a hot reload mechanism to achieve seamless updates to the message sending client. This approach avoids restarting the probe program due to configuration changes, significantly reducing system downtime and improving system availability and stability. Simultaneously, the configuration retrieval mechanism, combining proactive push from the configuration center with periodic client retrieval, enhances fault tolerance for configuration changes, ensuring that the probe client can obtain the latest configuration data promptly and accurately, thereby guaranteeing the normal operation of data collection and sending functions.

[0104] In some embodiments of this application, the predefined data transmission strategy includes an asynchronous sending strategy. The step of collecting and sending data using the predefined data transmission strategy through the message sending client includes: constructing a thread pool in the delegate class of the message sending client; modifying the sending method in the delegate class of the message sending client to submit the message sending task to the thread pool for asynchronous sending.

[0105] Construct a thread pool in the delegate class of the message sending client. The thread pool configuration should be set according to actual needs. For example, setting the number of threads to 1 means that only one thread is responsible for handling message sending tasks at a time, avoiding resource contention and complex synchronization issues that may arise from multiple threads operating simultaneously. Simultaneously, set the queue size, for example, to 50. By reasonably configuring the number of threads and the queue size, certain message sending concurrency requirements can be met while ensuring reasonable utilization of system resources.

[0106] The `send` method in the message sending client delegate class was modified. Originally, the Kafka client's `send` method sent messages asynchronously, but the operation of fetching cluster metadata was synchronous. When an exception occurred in the Kafka cluster, the synchronous metadata operation would be blocked until a one-minute timeout, affecting subsequent message sending by other clients. To solve this problem, the modified `send` method no longer directly calls the original Kafka client's `send` method. Instead, it submits the message sending task to a previously built thread pool. Threads in the thread pool are responsible for executing the actual message sending operations, including handling the fetching of Kafka cluster metadata. This makes the message sending task asynchronous. Even if a Kafka cluster exception causes metadata fetching to be blocked, it will not block other operations of the current delegate class, especially not affecting the normal operation of other message sending clients. For example, when multiple message sending clients with different configurations (such as clients corresponding to different systems) send messages through this delegate class, if one client's message sending task is blocked due to a Kafka cluster exception, the message sending tasks of other clients can still be processed normally in the thread pool.

[0107] This application embodiment achieves further asynchronous message sending by constructing a thread pool in the message sending client delegate class and modifying the sending method. This solution effectively solves the stuttering problem caused by Kafka clients synchronously pulling cluster metadata when the cluster is abnormal, and avoids the situation where the message sending of a single client is blocked, affecting the message sending of other clients, thus greatly improving the stability and reliability of message sending. At the same time, the use of the thread pool reasonably controls the concurrent processing capability, avoiding the resource consumption and synchronization complexity caused by too many threads, and improving resource utilization while ensuring system performance.

[0108] In some embodiments of this application, the predefined data transmission strategy includes a dual-send strategy. The step of collecting and sending data using the predefined data transmission strategy through the message sending client includes: constructing a multi-channel sending proxy class, which is used to proxy multiple independent message sending clients; and calling the sending methods of multiple independent message sending clients through the multi-channel sending proxy class to send the same data through multiple independent message sending clients.

[0109] Create a multi-channel message sending proxy class. This class acts as a proxy for multiple independent message sending clients. When designing this proxy class, define interface methods related to message sending. These interface methods function similarly to the sending methods of a single message sending client, but in practice, multiple clients will be involved.

[0110] During the initialization of the multi-channel send proxy class, multiple independent message sending clients need to be created and configured separately. When sending data through the multi-channel send proxy class, the send method of the proxy class calls the send methods of multiple independent message sending clients, sending the same data to these clients simultaneously. For example, when a message to be sent is received, the proxy class will pass the data to the send interfaces of multiple Kafka clients, which will then send the data to their respective Kafka clusters. In this way, even if one Kafka cluster fails or experiences a network outage, the other clusters can still receive and store data normally, ensuring reliable data transmission.

[0111] This application implements a dual-send strategy for data transmission by constructing a multi-channel sending proxy class and proxying multiple independent message sending clients. This solution significantly improves the reliability and fault tolerance of data transmission. When one message sending channel (such as a Kafka cluster) fails, experiences network problems, or suffers from performance bottlenecks, the other channel can still function normally, ensuring that data is not lost and reaches the target system in a timely and accurate manner. Simultaneously, this dual-send strategy can also be used for data backup and redundant storage, meeting the needs of business scenarios with extremely high requirements for data security and reliability.

[0112] This application also provides an automated data acquisition and transmission device 200, such as... Figure 2 As shown, a schematic diagram of an automated configuration data acquisition and transmission device 200 according to an embodiment of this application is provided. The automated configuration data acquisition and transmission device 200 includes:

[0113] The initialization unit 210 is used to initialize the configuration center client of the probe based on the private configuration information of the probe during the probe startup process of data acquisition and transmission. The probe and the target application run in the same virtual machine environment, and the configuration center client used by the probe and the configuration center client used by the target application are isolated from each other.

[0114] The acquisition unit 220 is used to acquire target configuration data for data collection and transmission through the configuration center client of the probe and the predefined configuration center namespace;

[0115] Creation unit 230 is used to create and maintain message sending clients according to the target configuration data;

[0116] The sending unit 240 is used to collect and send data through the message sending client using a predefined data transmission strategy.

[0117] In some embodiments of this application, the mutual isolation includes dependency isolation, and the initialization unit 210 is specifically used to: when packaging the probe, replace the package name of the relevant dependency files of the configuration center client of the probe, so that the package path of the relevant dependency files after package name replacement is different from the package path of the corresponding dependency files used in the target application.

[0118] In some embodiments of this application, the mutual isolation includes configuration isolation. The initialization unit 210 is specifically used to: read the namespace identifier of the configuration center defined in the private configuration information of the probe; set the read namespace identifier of the configuration center to the system properties through static synchronization code, wherein the system properties are the method of reading the namespace identifier of the configuration center with the highest priority; create the configuration center client of the probe based on the namespace identifier of the configuration center in the system properties; and restore the original namespace identifier of the configuration center in the system properties after the configuration center client of the probe is created.

[0119] In some embodiments of this application, the mutual isolation includes configuration isolation. The initialization unit 210 is specifically used to: provide a class that implements the service provider interface defined by the configuration center based on the service provider interface mechanism, and register the class of the service provider interface as an implementation class. The service provider interface is used to obtain the access address of the configuration server node. When initializing the configuration center client of the probe, the registered implementation class is loaded. The access address of the configuration server node is read from the private configuration information of the probe through the implementation class.

[0120] In some embodiments of this application, the predefined configuration center namespace includes a public namespace and at least one private namespace. The acquisition unit 220 is specifically used to: load public configuration data from the public namespace through the configuration center client of the probe; and load private configuration data from at least one of the private namespaces through the configuration center client of the probe.

[0121] In some embodiments of this application, the creation unit 230 is specifically used to: create a first message sending client based on the target configuration data, as the currently active message sending client; listen for configuration change notifications in the configuration center namespace; update the configuration of the first message sending client or create a second message sending client when the configuration change notification is heard; and perform hot reload on the updated first message sending client or the second message sending client.

[0122] In some embodiments of this application, the predefined data transmission strategy includes an asynchronous sending strategy, and the sending unit 240 is specifically used to: construct a thread pool in the delegate class of the message sending client; modify the sending method in the delegate class of the message sending client to submit the message sending task to the thread pool for asynchronous sending.

[0123] In some embodiments of this application, the predefined data transmission strategy includes a dual-send strategy, and the sending unit 240 is specifically used to: construct a multi-channel sending proxy class, which is used to proxy multiple independent message sending clients; and through the multi-channel sending proxy class, call the sending methods of multiple independent message sending clients respectively, so as to send the same data through multiple independent message sending clients.

[0124] It is understood that the above-mentioned automated configuration data acquisition and transmission device can realize each step of the automated configuration data acquisition and transmission method provided in the foregoing embodiments. The relevant explanations of the automated configuration data acquisition and transmission method are applicable to the automated configuration data acquisition and transmission device, and will not be repeated here.

[0125] Figure 3 This is a schematic diagram of the structure of a device according to an embodiment of this application. For example... Figure 3 As shown, the device includes one or more processors (or processing units), and may also include one or more memories coupled to the processors, and may also include a communication module coupled to the processors.

[0126] A communication module can be used to communicate with other devices or apparatuses, such as sending or receiving data and / or signals. A communication module may have at least one communication module for communication. A communication module may include any interface necessary for communicating with other devices. Exemplarily, a communication module may be a transceiver, circuit, bus, module, or other type of communication module.

[0127] The processor may include, but is not limited to, one or more of the following: a general-purpose computer, a special-purpose computer, a microcontroller, a digital signal processor (DSP), or a controller-based multi-core controller architecture. The device may have multiple processors, such as application-specific integrated circuit (ASIC) chips, which are time-dependent on a clock synchronized with the main processor.

[0128] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM), or other volatile memories that do not persist during the duration of a power outage.

[0129] A computer program consists of computer-executable instructions that are executed by an associated processor. Programs can be stored in ROM. A processor can perform any appropriate action and processing by loading the program into RAM.

[0130] Possible implementations of this application can be achieved through a program, enabling the communication device to execute any of the processes discussed in the foregoing embodiments. Possible implementations of this application can also be achieved through hardware or a combination of software and hardware.

[0131] In some implementations, the program may be tangibly contained in a computer-readable storage medium, which may include in a device (such as in memory) or other storage device accessible by the device. The program may be loaded from the computer-readable storage medium into RAM for execution. The computer-readable storage medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc.

[0132] This application also provides a computer-readable storage medium storing computer instructions or program code thereon, which, when executed by a processor, causes the processor to perform the methods and functions involved in any of the above embodiments. A computer-readable medium can be any tangible medium that contains or stores a program for or relating to an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. More detailed examples of computer-readable storage media include electrical connections with one or more wires, magnetic media (e.g., disks, floppy disks, hard disks, magnetic tapes, magnetic storage devices), optical media (e.g., optical storage devices, DVDs), semiconductor media (e.g., solid-state drives), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or any suitable combination thereof.

[0133] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. Embodiments of this application also provide at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. This computer program product includes one or more computer-executable instructions, such as instructions included in a program module, which execute in a device on a target real or virtual processor to perform the processes, methods, and functions involved in any of the above embodiments. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0134] This application also proposes a computer program product, including a computer program or instructions that, when run on a computer, cause the computer to perform the processes, methods, and functions described in the above embodiments. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided as needed. The machine-executable instructions for the program modules can be executed locally or in a distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0135] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0136] It should be noted that although embodiments of this application have been described above with reference to the accompanying drawings, these embodiments are not independent of each other, and they can be combined to obtain other embodiments. The methods, situations, categories, and classifications of embodiments in this application are only for the convenience of description and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined with each other if logically consistent. The various embodiments of this application can be arbitrarily combined to achieve different technical effects. The embodiments of this application will not list various combinations.

[0137] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0138] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for automated configuration of data acquisition and transmission, characterized in that, The automated configuration data acquisition and transmission method includes: During the probe startup process for data acquisition and transmission, the probe's configuration center client is initialized based on the probe's private configuration information. The probe and the target application run in the same virtual machine environment, and the configuration center client used by the probe and the configuration center client used by the target application are isolated from each other. The target configuration data for data collection and transmission is obtained through the configuration center client of the probe and the predefined configuration center namespace; Create and maintain a message sending client based on the target configuration data; The message sending client uses a predefined data transmission strategy to collect and send data.

2. The automated configuration data acquisition and transmission method according to claim 1, characterized in that, The mutual isolation includes dependency isolation. The initialization of the probe's configuration center client based on the probe's private configuration information during the probe's startup process for data acquisition and transmission includes: When packaging the probe, the package names of the relevant dependency files of the probe's configuration center client are replaced so that the package path of the related dependency files after the package name replacement is different from the package path of the corresponding dependency files used in the target application.

3. The automated configuration data acquisition and transmission method according to claim 1, characterized in that, The mutual isolation includes configuration isolation. The initialization of the probe's configuration center client based on the probe's private configuration information during the probe's data acquisition and transmission startup process includes: Read the namespace identifier of the configuration center defined in the private configuration information of the probe; The namespace identifier of the read configuration center is set to the system property through static synchronization code. The system property is the method of reading the namespace identifier of the configuration center with the highest priority. The configuration center client of the probe is created based on the namespace identifier of the configuration center in the system properties; After the configuration center client of the probe is created, restore the original namespace identifier of the configuration center in the system properties.

4. The automated configuration data acquisition and transmission method according to claim 1, characterized in that, The mutual isolation includes configuration isolation. The initialization of the probe's configuration center client based on the probe's private configuration information during the probe's data acquisition and transmission startup process includes: Based on the service provider interface mechanism, a class that implements the service provider interface defined by the configuration center is provided, and the class of the service provider interface is registered as an implementation class. The service provider interface is used to obtain the access address of the configuration server node. When initializing the configuration center client for the probe, the registered implementation class is loaded; The implementation class reads the access address of the configuration server node from the private configuration information of the probe.

5. The automated configuration data acquisition and transmission method according to claim 1, characterized in that, The predefined configuration center namespace includes a public namespace and at least one private namespace. Obtaining the target configuration data for data collection and transmission through the probe's configuration center client and the predefined configuration center namespace includes: The configuration center client of the probe loads public configuration data from the public namespace; The configuration center client of the probe loads private configuration data from at least one of the private namespaces.

6. The automated configuration data acquisition and transmission method according to claim 1, characterized in that, The step of creating and maintaining a message sending client based on the target configuration data includes: A first message sending client is created based on the target configuration data, which serves as the currently active message sending client. Listen for configuration change notifications in the configuration center namespace; Upon receiving the configuration change notification, update the configuration of the first message sending client or create a second message sending client; Perform a hot reload on the updated first message sending client or the second message sending client.

7. The automated configuration data acquisition and transmission method according to claim 1, characterized in that, The predefined data transmission strategy includes an asynchronous transmission strategy, and the process of collecting and sending data using the predefined data transmission strategy through the message sending client includes: Build a thread pool in the delegate class of the message sending client; Modify the send method in the delegate class of the message sending client to submit the message sending task to the thread pool for asynchronous sending.

8. The automated configuration data acquisition and transmission method according to claim 1, characterized in that, The predefined data transmission strategy includes a dual-send strategy, and the data collection and transmission using the predefined data transmission strategy through the message sending client includes: Construct a multi-channel sending proxy class, which is used to proxy multiple independent message sending clients; The multi-channel sending proxy class calls the sending methods of multiple independent message sending clients to send the same data through multiple independent message sending clients.

9. An automated data acquisition and transmission device, characterized in that, The automated configuration data acquisition and transmission device includes: An initialization unit is used to initialize the configuration center client of the probe based on the probe's private configuration information during the probe startup process of data acquisition and transmission. The probe and the target application run in the same virtual machine environment, and the configuration center client used by the probe and the configuration center client used by the target application are isolated from each other. The acquisition unit is used to acquire target configuration data for data collection and transmission through the configuration center client of the probe and the predefined configuration center namespace; A creation unit is used to create and maintain a message sending client based on the target configuration data; The sending unit is used to collect and send data through the message sending client using a predefined data transmission strategy.

10. An apparatus comprising: processor; And a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the data acquisition and transmission method of any of claims 1 to 7 with automated configuration.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the data acquisition and transmission method of any one of claims 1 to 7 for automated configuration.

Images