Data real-time reading delivery method and system

Abstract

The application discloses a data real-time reading and transmission method and system, belongs to the technical field of data storage, and aims to solve the technical problem of how to read data in real time and transmit and store data. The method comprises the following steps: S100, subscribing to a message topic based on a Redis message queue, and agreeing on the message topic; S200, setting a data storage mode; S300, setting data storage information based on the set storage mode; S400, entering data in a message topic corresponding to a data direction; S500, judging whether the data needs to be stored, if yes, storing the data according to the configured storage mode, and executing step S600, if not, executing step S600; S600, judging whether the data needs to be transmitted, if yes, transmitting the data according to the configured storage mode, and executing step S700, if not, executing step S700; and S700, deleting the data that has been processed in Redis.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, specifically to a method and system for real-time data reading and transmission. Background Technology

[0002] With the rapid development of science and technology, artificial intelligence (AI) technology has advanced by leaps and bounds and quickly entered people's lives. The booming development of AI technology has propelled people's lives to a higher level, ushering in the intelligent era of human civilization. AI exists in all aspects of life; for example, it can improve people's work and study efficiency, reduce their workload, and enrich their lives.

[0003] The development of various technologies is essentially inseparable from the support of various data. Data refers to symbols that record and identify objective events; it is a combination of physical symbols or physical symbols that record the nature, state, and interrelationships of objective things. It is a recognizable, abstract symbol. In artificial intelligence systems, data is the general term for all media of symbols that can be input into a computer and processed by computer programs. It is a general term for numbers, letters, symbols, and analog quantities that are used for input into electronic computers for processing and have certain meaning. In artificial intelligence systems, data allows us to understand the current state of something and perform specific actions based on that state. If the data acquisition is delayed, timely processing is impossible; therefore, the real-time nature of data is crucial.

[0004] How to read, forward, and store data in real time is a technical problem that needs to be solved. Summary of the Invention

[0005] The technical objective of this invention is to address the above-mentioned shortcomings by providing a method and system for real-time data reading and transmission, thereby solving the technical problem of how to read, forward, and store data in real time.

[0006] In a first aspect, the present invention provides a method for real-time data reading and transmission, comprising the following steps:

[0007] S100: A message topic subscription method based on Redis message queues, with agreed-upon message topics;

[0008] S200, Set the data storage method;

[0009] S300: Based on the set storage method, set the data storage information, including the storage address;

[0010] Enter data in the message topic corresponding to the S400 production data direction;

[0011] S500. For the entered data, determine whether the data needs to be stored. If yes, store the data according to the configured storage method and execute step S600. If no, execute step S600.

[0012] S600. Determine whether the data needs to be transmitted. If yes, transmit the data according to the configured storage method and execute step S700. If no, execute step S700.

[0013] S700: Delete data that has already been processed in Redis.

[0014] As a preferred approach, the message topic subscription method based on Redis message queues specifies message topics in key value format, and production data is entered into the corresponding message topics according to the predefined message topics.

[0015] Preferred storage methods include Elasticsearch, databases, and Redis; setting the data storage method also includes setting up HTTP connections to transmit the data again.

[0016] Preferably, if the storage method is a database, the corresponding storage information includes the database type, URL, port, username, and password.

[0017] As a preferred method, when entering data into the message topic corresponding to the production data direction, simulate data reading by manually entering data through commands.

[0018] In a second aspect, the present invention provides a real-time data reading and transmission system for real-time reading and transmission of data using a real-time data reading and transmission method as described in any of the first aspects, the system comprising:

[0019] The topic agreement module is used to agree on message topics based on the message topic subscription method of Redis message queue.

[0020] The configuration module is used to set the data storage method and, based on the set storage method, set the data storage information, including the storage address;

[0021] The data entry module is used to support data entry into message topics corresponding to the production data direction;

[0022] The transmission module is configured to perform the following:

[0023] L100. For the entered data, determine whether the data needs to be stored. If yes, store the data according to the configured storage method and execute step L200. If no, execute step L200.

[0024] L200: Determine whether the data needs to be transmitted. If yes, transmit the data according to the configured storage method and execute step S700. If no, execute step L300.

[0025] L300: Delete data that has already been processed in Redis.

[0026] Preferably, the topic agreement module is used to define the message topic subscription method and the key value format of the message topic based on the Redis message queue;

[0027] The data entry module is used to support the entry of production data into the corresponding message topic according to the predetermined message topic.

[0028] Preferably, storage methods include Elasticsearch, database, and Redis; the configuration module is also used to set the HTTP connection for retransmission of data.

[0029] Preferably, if the storage method is a database, the corresponding storage information includes the database type, URL, port, username, and password.

[0030] Preferably, when entering data into the message topic corresponding to the production data direction, the entry module is used to support manual data entry via command to simulate reading data.

[0031] The real-time data reading and transmission method and system of the present invention have the following advantages:

[0032] 1. It can realize real-time data reading and transmission, ensuring the timeliness of system data;

[0033] 2. Data can be cleaned up promptly after reading to ensure the environment does not lag due to increased data volume;

[0034] 3. Compatibility issues in a fully domestically produced environment have been fully considered. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] The invention will be further described below with reference to the accompanying drawings.

[0037] Figure 1 This is a flowchart of a real-time data reading and transmission process in Example 1. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0039] This invention provides a method and system for real-time data reading and transmission, which solves the technical problem of how to read, forward and store data in real time.

[0040] Example 1:

[0041] The present invention discloses a method for real-time data reading and transmission, comprising the following steps:

[0042] S100: A message topic subscription method based on Redis message queues, with agreed-upon message topics;

[0043] S200, Set the data storage method;

[0044] S300: Based on the set storage method, set the data storage information, including the storage address;

[0045] Enter data in the message topic corresponding to the S400 production data direction;

[0046] S500. For the entered data, determine whether the data needs to be stored. If yes, store the data according to the configured storage method and execute step S600. If no, execute step S600.

[0047] S600. Determine whether the data needs to be transmitted. If yes, transmit the data according to the configured storage method and execute step S700. If no, execute step S700.

[0048] S700: Delete data that has already been processed in Redis.

[0049] In step S100 of this embodiment, the message topic is agreed upon. Since the Redis message queue is used, the topic needs to be agreed upon, that is, a topic similar to a key value is defined. The data production party needs to enter data into the topic according to the agreement.

[0050] Step S200 configures the storage method. In this embodiment, Elasticsearch, database, and Redis are supported. Alternatively, an HTTP connection can be set up to transmit the data again.

[0051] Step S300: Fill in different storage information according to the selected storage method. For example, if the storage method is a database, you need to fill in the database type (this embodiment supports ShenTong, Dameng, Kingbase, Hangao, Nanda General, and MySQL), URL, port, username, and password.

[0052] Once step S400 is configured, begin reading data. You can also manually enter data via commands to simulate data reading.

[0053] Here's a specific example from this embodiment: Set the topic name to ai-video-analysis-stream, select a database as the storage method, choose ShenTong database, set the URL to jdbc:oscar: / / 10.10.10.10:2003 / 2003, port number to 2003, and the username and password to oscar / 123456. Save the settings. Then, set up a new HTTP link with the URL http: / / 10.10.10.10:8080 / dealData / getRedisData. After setup, you can manually simulate adding data to Redis using the following command: `xadd ai-video-analysis-stream *msg"{\"timestamp\":\"1669965873042\",\"channel\":\"channel-1\"....}"`. After execution, a data entry will be inserted into Redis. The system will immediately read this data and, based on the previous settings, insert the read data into the ShenTong database. It will then send the read data to the other party via an HTTP request. After reading and processing are complete, the data entry will be removed from Redis.

[0054] The method in this embodiment is compatible with domestic system environments, and can delete data in a timely manner after reading or transmitting, ensuring the stable operation of the system; it can read data in real time, and read data promptly when it is stored; it can transmit data in real time, ensuring the timeliness of the data; and it can set the data storage location or transmit data links as needed, making it relatively flexible.

[0055] Example 2:

[0056] The present invention discloses a real-time data reading and transmission system, including a topic agreement module, a configuration module, an input module, and a transmission module. The system can execute the method disclosed in Embodiment 1 to perform real-time data reading and transmission.

[0057] The topic convention module is used to define message topics based on the Redis message queue's message topic subscription method.

[0058] In this embodiment, the topic agreement module is used to define the message topic subscription method and the key value format of message topics based on the Redis message queue.

[0059] The configuration module is used to set the data storage method and, based on the set storage method, to set the data storage information, including the storage address.

[0060] In this example, storage methods include Elasticsearch, database, and Redis. The configuration module sets the data storage method and also configures the HTTP connection for data transmission. If the storage method is a database, the corresponding storage information includes the database type, URL, port, username, and password.

[0061] The data entry module supports data entry into message topics corresponding to the production data direction. When entering data into message topics corresponding to the production data direction, the data entry module also supports simulating data reading by manually entering data via commands.

[0062] The delivery module is used to perform the following:

[0063] L100. For the entered data, determine whether the data needs to be stored. If yes, store the data according to the configured storage method and execute step L200. If no, execute step L200.

[0064] L200: Determine whether the data needs to be transmitted. If yes, transmit the data according to the configured storage method and execute step S700. If no, execute step L300.

[0065] L300: Delete data that has already been processed in Redis.

[0066] The system in this embodiment can read data in real time based on domestic CPUs and operating systems, so that the generated data can be displayed in real time; after the data is acquired, it can be transmitted or saved according to the system configuration.

[0067] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the means in the different embodiments described above, and these embodiments are also within the protection scope of the present invention.

Claims

1. A method for real-time data reading and transmission, characterized in that, Includes the following steps: S100: A message topic subscription method based on Redis message queues, with agreed-upon message topics; S200, setting the data storage method, also includes setting up HTTP connections to transmit data again, and storage methods include Elasticsearch, database, and Redis; S300: Based on the set storage method, set the data storage information, including the storage address; Enter data in the message topic corresponding to the S400 production data direction; S500. For the entered data, determine whether the data needs to be stored. If yes, store the data according to the configured storage method and execute step S600. If no, execute step S600. S600. Determine whether the data needs to be transmitted. If yes, transmit the data according to the configured storage method and execute step S700. If no, execute step S700. S700: Delete data that has already been processed in Redis.

2. The real-time data reading and transmission method according to claim 1, characterized in that, Based on the Redis message queue, the message topic subscription method specifies the message topic in key value format, and production data is entered into the corresponding message topic according to the predefined message topic.

3. The real-time data reading and transmission method according to claim 1, characterized in that, If the storage method is a database, the corresponding storage information includes the database type, URL, port, username, and password.

4. The real-time data reading and transmission method according to claim 1, characterized in that, When entering data into the message topic corresponding to the production data direction, simulate data reading by manually entering data through commands.

5. A real-time data reading and transmission system, characterized in that, The system is used for real-time reading and transmission of data using a real-time data reading and transmission method as described in any one of claims 1-4, the system comprising: The topic agreement module is used to agree on message topics based on the message topic subscription method of Redis message queue. The configuration module is used to set the data storage method and, based on the set storage method, set the data storage information. It is used to set the HTTP link to transmit the data again. The storage methods include Elasticsearch, database, and Redis, and the storage information includes the storage address. The data entry module is used to support data entry into message topics corresponding to the production data direction; The transmission module is configured to perform the following: L100. For the entered data, determine whether the data needs to be stored. If yes, store the data according to the configured storage method and execute step L200. If no, execute step L200. L200: Determine whether the data needs to be transmitted. If yes, transmit the data according to the configured storage method and execute step S700. If no, execute step L300. L300: Delete data that has already been processed in Redis.

6. The real-time data reading and transmission system according to claim 5, characterized in that, The topic convention module is used to define the message topic subscription method and the key value format of message topics based on the Redis message queue. The data entry module is used to support the entry of production data into the corresponding message topic according to the predetermined message topic.

7. A real-time data reading and transmission system according to claim 5, characterized in that, If the storage method is a database, the corresponding storage information includes the database type, URL, port, username, and password.

8. A real-time data reading and transmission system according to claim 5, characterized in that, When entering data into the message topic corresponding to the production data direction, the data entry module is used to support manual data entry via command to simulate reading data.

Images