Data acquisition device and method based on RabbitMQ

Abstract

The invention discloses a data acquisition device and method based on RabbitMQ, and relates to the technical field of data acquisition. Comprising the following steps of: 1, dynamically configuring a RabbitMQ message queue, 2, performing consumption feedback by using the RabbitMQ message queue, 21, performing dissimilation processing on messages, namely performing dissimilation analysis on the messages received in real time, and extracting key information, and 22, adaptively adjusting a threshold, namely dynamically adjusting a monitoring threshold and parameters according to the key information, and step 3, self-adaptive processing of dead messages is carried out by using the RabbitMQ message queue: abnormal messages in dissimilatory information are identified through a preset rule and a machine learning model, feature extraction is carried out on the abnormal messages, dimension reduction processing is carried out on the extracted features, and a retransmission strategy is self-adaptively adjusted according to the processed features so as to carry out message retransmission.

Description

Technical Field

[0001] The present invention discloses a data acquisition device and method based on RabbitMQ, which relates to the technical field of data acquisition. Background Art

[0002] There are still some defects in the existing data acquisition methods or devices through RabbitMQ. For example:

[0003] Insufficient dynamic configuration: When the connection status between the device and the server changes frequently, it is difficult for the existing methods to automatically adjust parameters to maintain a stable connection.

[0004] Low monitoring efficiency: The real-time monitoring of a large number of message flows takes a long time and is costly, and it is difficult to meet the high-efficiency data acquisition requirements in industrial and commercial scenarios.

[0005] Insufficient dead letter processing: For abnormal or alienated information (such as dead letters, invalid receipts, etc.), the existing devices rely on predefined standards for filtering, resulting in a high false alarm rate and inability to adaptively adjust.

[0006] When data is collected and aggregated based on the existing data acquisition device through RabbitMQ, due to the inability to flexibly configure message consumption, the maintenance cost of the device will increase; if effective consumption monitoring is not carried out, the integrity of the data cannot be guaranteed, and at the same time, a large number of messages will accumulate in the dead letter queue. Summary of the Invention

[0007] Aiming at the problems of the existing technology, the present invention provides a data acquisition device and method based on RabbitMQ, which introduces a dynamic configuration mechanism, a consumption feedback monitoring mechanism, and a dead letter adaptive processing mechanism, and realizes the efficient acquisition and processing of complex alienated data. It can automatically adjust parameters according to real-time data characteristics, monitor data streams in real time and optimize the acquisition efficiency. At the same time, when abnormal or alienated information appears, it can quickly identify and resend relevant messages to ensure the accuracy and integrity of data acquisition.

[0008] The specific solution proposed by the present invention is as follows:

[0009] The present invention provides a data acquisition method based on RabbitMQ, including:

[0010] Step 1: Perform dynamic configuration of the RabbitMQ message queue:

[0011] Step 11: Detect the real-time connection status between the data acquisition device and the RabbitMQ message queue server, dynamically adjust the transmission parameters, and dynamically adjust the data acquisition parameters through preset rules or machine learning models.

[0012] Step 12: Dynamic configuration message listening process: Create a ConnectionFactory for creating connections; create a SimpleMessageListenerContainer class for managing message listening, add the specified queue name to the object container of the SimpleMessageListenerContainer class, and start the container to start listening to the queue.

[0013] Step 2: Use the RabbitMQ message queue for consumption feedback:

[0014] Step 21: Perform message alienation processing: Analyze the alienation of the real-time received messages and extract key information.

[0015] Step 22: Adaptively adjust the threshold: Dynamically adjust the monitoring threshold and parameters according to the key information.

[0016] Step 3: Use the RabbitMQ message queue for self-adaptive processing of dead letters:

[0017] Identify abnormal messages in the alienated information through preset rules and machine learning models, extract features of the abnormal messages, and perform dimensionality reduction processing on the extracted features. According to the processed features, adaptively adjust the retransmission strategy for message retransmission.

[0018] Furthermore, in step 11 of the data collection method based on RabbitMQ, the data collection parameters are dynamically adjusted using threshold rules or rules based on time windows.

[0019] Or the data collection parameters are dynamically adjusted using a supervised learning model, and the supervised learning model includes a decision tree / random forest model and a support vector machine.

[0020] Furthermore, before starting the container to start listening to the queue in step 12 of the data collection method based on RabbitMQ, check whether the container is empty. If it is empty, perform initialization:

[0021] Create a new SimpleMessageListenerContainer object using the connectionFactory.

[0022] Set a message listener. When the message listener receives a message, convert the message body into a string.

[0023] Set the container not to start automatically.

[0024] Add the specified queue name to the container and start the container to begin listening to the queue.

[0025] Further, in step 3 of the data collection method based on RabbitMQ, the threshold rule or regular expression matching rule is used to identify abnormal messages in the alienated information.

[0026] Or an unsupervised learning model is used to identify abnormal messages in the alienated information. The unsupervised learning model includes a model constructed using the K-means clustering algorithm, a model using the Isolation Forest algorithm, or a model constructed using one-class support vector machines.

[0027] The present invention also provides a data collection device based on RabbitMQ, including a configuration module, a message feedback module, and a dead letter processing module.

[0028] The configuration module performs dynamic configuration of the RabbitMQ message queue:

[0029] Step 11: Detect the real-time connection status between the data collection device and the RabbitMQ message queue server, dynamically adjust the transmission parameters, and dynamically adjust the data collection parameters through a preset rule or a machine learning model.

[0030] Step 12: Dynamically configure the message listening process: Create a ConnectionFactory for creating a connection; create a SimpleMessageListenerContainer class for managing message listening, add the specified queue name to the object container of the SimpleMessageListenerContainer class, and start the container to begin listening to the queue.

[0031] The message feedback module uses the RabbitMQ message queue for consumption feedback:

[0032] Step 21: Perform message alienation processing: Analyze the alienation of the real-time received messages and extract key information.

[0033] Step 22: Adaptively adjust the threshold: Dynamically adjust the monitoring threshold and parameters according to the key information.

[0034] The dead letter processing module uses the RabbitMQ message queue for adaptive processing of dead letters:

[0035] Identify abnormal messages in the alienated information through a preset rule and a machine learning model, extract features of the abnormal messages, perform dimensionality reduction processing on the extracted features, and adaptively adjust the retransmission strategy for message retransmission according to the processed features.

[0036] Furthermore, when the configuration module of the data acquisition device based on RabbitMQ executes step 11, it dynamically adjusts the data acquisition parameters by using threshold rules or time window-based rules,

[0037] or dynamically adjusts the data acquisition parameters by using a supervised learning model, and the supervised learning model includes decision tree / random forest model and support vector machine.

[0038] Furthermore, when the configuration module of the data acquisition device based on RabbitMQ executes step 12, before starting the container to start listening to the queue, it checks whether the container is empty. If it is empty, it initializes:

[0039] Creates a new SimpleMessageListenerContainer object using the connectionFactory,

[0040] Sets a message listener. When the message listener receives a message, it converts the message body into a string,

[0041] Sets the container not to start automatically;

[0042] Adds the specified queue name to the container and starts the container to start listening to the queue.

[0043] Furthermore, when the dead letter processing module of the data acquisition device based on RabbitMQ executes step 3, it identifies abnormal messages in the dissimilated information by using threshold rules or regular expression matching rules,

[0044] or identifies abnormal messages in the dissimilated information by using an unsupervised learning model, and the unsupervised learning model includes a model constructed using the K-means clustering algorithm, a model constructed using the Isolation Forest algorithm, or a model constructed using one-class support vector machine.

[0045] The advantages of the present invention are:

[0046] Through the dynamic configuration mechanism, the consumption feedback monitoring mechanism, and the dead letter-based adaptive processing mechanism, the device can automatically adjust parameters to ensure the stability and efficiency of data acquisition. When abnormal or dissimilated information appears, it can quickly identify and process it, thus ensuring the accuracy and integrity of data acquisition. Through flexible message consumption configuration, the operation and maintenance cost of the system can be greatly reduced, and at the same time, the system can adapt to more business application scenarios; effective consumption monitoring can ensure the integrity of data; the application of the dead letter queue can reduce the data volume while retaining key information and improve the success rate of data acquisition. Brief Description of the Drawings

[0047] Figure 1 It is a schematic diagram of the application framework of the method of the present invention.

[0048] Figure 2 It is a schematic diagram of the code example of the configuration message listening process. Detailed Embodiments

[0049] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the specific embodiments cited are not intended to limit the present invention.

[0050] Embodiment 1

[0051] The present invention provides a data collection method based on RabbitMQ, including:

[0052] Step 1: Perform dynamic configuration of the RabbitMQ message queue:

[0053] Step 11: Detect the real-time connection status between the data collection device and the RabbitMQ message queue server, dynamically adjust the transmission parameters, and dynamically adjust the data collection parameters through preset rules or machine learning models.

[0054] Among them, in step 11, the data collection parameters are dynamically adjusted by using threshold rules or time window-based rules. For example, set the thresholds of some key indicators, and when the collected data exceeds or is lower than these thresholds, the collection parameters are automatically adjusted. For another example, based on the time window-based rule, when the data changes greatly within a period of time, the collection frequency is increased; when the data changes little, the collection frequency is decreased.

[0055] Or use a supervised learning model to dynamically adjust the data collection parameters. The supervised learning model includes decision tree / random forest model and support vector machine. For example, use historical data to train a decision tree or random forest model, predict the change trend of future data, and adjust the collection parameters according to the prediction results, etc.

[0056] Step 12: Dynamically configure the message listening process: Create a ConnectionFactory for creating a connection; create a SimpleMessageListenerContainer class for managing message listening, add the specified queue name to the object container of the SimpleMessageListenerContainer class, and start the container to start listening to the queue. For the code example of dynamically configuring the message listening process, reference can be made to Figure 2 .

[0057] Before starting the container to start listening to the queue in step 12, check if the container is empty. If it is empty, initialize it:

[0058] Create a new SimpleMessageListenerContainer object using the connectionFactory,

[0059] Set a message listener. When the message listener receives a message, convert the message body to a string.

[0060] Set the container not to start automatically;

[0061] Add the specified queue name to the container and start the container to start listening to the queue.

[0062] Step 2: Use the RabbitMQ message queue for consumption feedback:

[0063] Step 21: Perform message alienation processing: Analyze the real-time received messages for alienation, extract key information.

[0064] Step 22: Adaptively adjust the threshold: Dynamically adjust the monitoring threshold and parameters according to the key information.

[0065] Step 3: Use the RabbitMQ message queue for adaptive processing of dead letters:

[0066] Identify abnormal messages in the alienated information through preset rules and machine learning models, extract features of the abnormal messages, and perform dimensionality reduction processing on the extracted features. According to the processed features, adaptively adjust the retransmission strategy for message retransmission.

[0067] Among them, use threshold rules or regular expression matching rules to identify abnormal messages in the alienated information.

[0068] Or use an unsupervised learning model to identify abnormal messages in the alienated information. The unsupervised learning model includes a model constructed using the K-means clustering algorithm, a model using the Isolation Forest algorithm, or a model constructed using one-class support vector machines. For example, use the K-means algorithm to classify messages into normal and abnormal categories, thereby identifying the messages corresponding to abnormal behaviors. Use the Isolation Forest algorithm to identify abnormal messages or attack behaviors in the messages.

[0069] Embodiment 2

[0070] The present invention also provides a data acquisition device based on RabbitMQ, including a configuration module, a message feedback module, and a dead letter processing module.

[0071] The configuration module performs dynamic configuration of the RabbitMQ message queue:

[0072] Step 11: Detect the real-time connection status between the data acquisition device and the RabbitMQ message queue server, dynamically adjust the transmission parameters, and dynamically adjust the data acquisition parameters through preset rules or machine learning models.

[0073] Step 12: Dynamically configure the message listening process: Create a ConnectionFactory to create a connection; create a SimpleMessageListenerContainer class to manage message listening, add the specified queue name to the object container of the SimpleMessageListenerContainer class, and start the container to start listening to the queue.

[0074] The message feedback module uses the RabbitMQ message queue for consumption feedback:

[0075] Step 21: Perform message alienation processing: Analyze the alienation of the real-time received messages and extract key information.

[0076] Step 22: Adaptively adjust the threshold: Dynamically adjust the monitoring threshold and parameters according to the key information.

[0077] The dead letter processing module uses the RabbitMQ message queue for adaptive processing of dead letters:

[0078] Identify abnormal messages in the alienated information through preset rules and machine learning models, extract features of the abnormal messages, perform dimensionality reduction processing on the extracted features, and adaptively adjust the retransmission strategy for message retransmission according to the processed features.

[0079] Regarding the information interaction, execution process, etc. among the above-mentioned modules in the device, since they are based on the same concept as the method embodiment of the present invention, the specific content can be referred to the description in the method embodiment of the present invention, and will not be elaborated here.

[0080] Similarly, through the dynamic configuration mechanism, consumption feedback monitoring mechanism and dead letter-based adaptive processing mechanism of the device of the present invention, the device can automatically adjust parameters to ensure the stability and efficiency of data acquisition. When abnormal or alienated information appears, it can quickly identify and process it, thereby ensuring the accuracy and integrity of data acquisition. Through flexible message consumption configuration, the operation and maintenance cost of the system can be greatly reduced, and at the same time, the system can be adapted to more business application scenarios; effective consumption monitoring can ensure the integrity of data; the application of the dead letter queue can reduce the data volume while retaining key information, improving the success rate of data acquisition.

[0081] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above-mentioned embodiments can be physical structures or logical structures, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities respectively, or some components in multiple independent devices may jointly implement them.

[0082] The above-mentioned embodiments are only preferred embodiments given to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A data acquisition method based on RabbitMQ, characterized in that Including: Step 1: Perform dynamic configuration of the RabbitMQ message queue: Step 11: Detect the real-time connection status between the data acquisition device and the RabbitMQ message queue server, dynamically adjust the transmission parameters, and dynamically adjust the data acquisition parameters through preset rules or machine learning models. Step 12: Dynamically configure the message listening process: Create a ConnectionFactory for creating connections; create a SimpleMessageListenerContainer class for managing message listening, add the specified queue name to the object container of the SimpleMessageListenerContainer class, and start the container to start listening to the queue. Step 2: Use the RabbitMQ message queue for consumption feedback: Step 21: Perform message alienation processing: Analyze the alienation of the real-time received messages and extract key information. Step 22: Adaptively adjust the threshold: Dynamically adjust the monitoring threshold and parameters according to the key information. Step 3: Use the RabbitMQ message queue for adaptive processing of dead letters: Identify abnormal messages in the alienated information through preset rules and machine learning models, extract features of the abnormal messages, perform dimensionality reduction processing on the extracted features, and adaptively adjust the retransmission strategy for message retransmission according to the processed features.

2. The data acquisition method based on RabbitMQ according to claim 1, wherein in step 11, the data acquisition parameters are dynamically adjusted by using threshold rules or rules based on time windows. Or a supervised learning model is used to dynamically adjust the data acquisition parameters, and the supervised learning model includes a decision tree / random forest model and a support vector machine.

3. The data acquisition method based on RabbitMQ according to claim 1, characterized in that Before starting the container to start listening to the queue in step 12, check whether the container is empty, and if it is empty, perform initialization: Create a new SimpleMessageListenerContainer object using the connectionFactory. Set a message listener that converts the message body to a string when receiving a message. Set the container not to start automatically. Add the specified queue name to the container and start the container to start listening to the queue.

4. The data acquisition method based on RabbitMQ according to claim 1, wherein in step 3, abnormal messages in the alienated information are identified by using threshold rules or regular expression matching rules. Or an unsupervised learning model is used to identify abnormal messages in the alienated information, and the unsupervised learning model includes a model constructed using the K-means clustering algorithm, a model using the Isolation Forest algorithm, or a model constructed using a one-class support vector machine.

5. A data acquisition device based on RabbitMQ, characterized in that Including a configuration module, a message feedback module, and a dead letter processing module. The configuration module performs dynamic configuration of the RabbitMQ message queue: Step 11: Detect the real-time connection status between the data acquisition device and the RabbitMQ message queue server, dynamically adjust the transmission parameters, and dynamically adjust the data acquisition parameters through preset rules or machine learning models. Step 12: Dynamically configure the message listening process: Create a ConnectionFactory for creating connections; create a SimpleMessageListenerContainer class for managing message listening, add the specified queue name to the object container of the SimpleMessageListenerContainer class, and start the container to start listening to the queue. The message feedback module uses the RabbitMQ message queue for consumption feedback. Step 21: Perform message alienation processing: Analyze the alienation of the real-time received messages and extract key information. Step 22: Adaptively adjust the threshold: Dynamically adjust the monitoring threshold and parameters according to the key information. The dead letter processing module uses the RabbitMQ message queue for adaptive processing of dead letters. Identify abnormal messages in the alienated information through preset rules and machine learning models, extract features from the abnormal messages, perform dimensionality reduction processing on the extracted features, and adaptively adjust the retransmission strategy for message retransmission according to the processed features.

6. The data acquisition device based on RabbitMQ according to claim 5, characterized in that When the configuration module executes Step 11, it dynamically adjusts the data acquisition parameters using threshold rules or time window-based rules. Or use a supervised learning model to dynamically adjust the data acquisition parameters. The supervised learning model includes decision tree / random forest models and support vector machines.

7. The data acquisition device based on RabbitMQ according to claim 5, characterized in that When the configuration module executes Step 12, before starting the container to start listening to the queue, check if the container is empty. If it is empty, initialize it: Create a new SimpleMessageListenerContainer object using the connectionFactory. Set a message listener that converts the message body to a string when the message is received. Set the container not to start automatically. Add the specified queue name to the container and start the container to start listening to the queue.

8. The data acquisition device based on RabbitMQ according to claim 5 is characterized in that the dead When the letter processing module executes Step 3, it identifies abnormal messages in the alienated information using threshold rules or regular expression matching rules. Or use an unsupervised learning model to identify abnormal messages in the alienated information. The unsupervised learning model includes models constructed using the K-means clustering algorithm, the Isolation Forest algorithm, or one-class support vector machines.

Images