Real-time data exchange method, device and equipment and readable storage medium
By using a real-time data exchange method with a public data terminal and message queue, the problem of high complexity in data exchange between software systems is solved, achieving efficient and secure data transmission and storage, and improving data flow and availability.
Patent Information
- Application Number
- CN202410968024.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-19
AI Technical Summary
Existing data exchange between software systems is complex, costly, and fails to meet the needs of modern enterprises for efficient, real-time data exchange.
Data is integrated and distributed through a public data terminal. Real-time data exchange between data producers, public data terminals, and data consumers is achieved using message queues. Data is converted and stored using object storage format. Data integrity is verified using hash values. Data transmission is optimized through aggregation, direct download, and slice download methods.
It enables real-time data exchange and rapid discovery, improves data flow and value, simplifies data acquisition processes, ensures secure and efficient data storage and transmission, and enhances data availability and interoperability.
Smart Images

Figure CN118981384A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a real-time data exchange method, device, equipment and readable storage medium. Background Art
[0002] With the continuous deepening of informatization and digitalization, construction companies have gradually built multiple software systems to support different business areas, such as business preparation systems, project management systems, human resources systems, labor systems, centralized procurement systems, cost estimation and cost accounting systems, and BI analysis systems. These systems mainly solved the tooling and informatization problems of a single business field in the early stage, but as the demand for overall solutions grows, the various software systems need to be integrated and collaborated to form a unified business data exchange and processing platform.
[0003] However, existing technologies have obvious deficiencies in data exchange between software systems. Traditionally, data exchange is a point-to-point connection carried out on demand, lacking a unified top-level plan. As the number of systems increases, a complex mesh data exchange structure is formed, which not only increases the complexity of the overall system, but also significantly increases the cost of docking and adaptation when each software system docks with other systems. This model is difficult to meet the needs of modern enterprises for efficient and real-time data exchange.
[0004] In view of this, how to overcome the problems existing in the prior art and provide an efficient and real-time business data exchange solution is an important issue that needs to be urgently solved by those skilled in the art. Summary of the invention
[0005] The object of the present invention is to provide a real-time data exchange method, device, equipment and readable storage medium, which can complete the data exchange between the data generating end and the data consuming end in real time.
[0006] According to one aspect of the present invention, a real-time data exchange method is provided, the method comprising:
[0007] Obtaining a data generation event subscribed to from a message queue; wherein the data generation event is a notification event sent by a data generation end to the message queue after generating data to be exchanged;
[0008] Parsing the data generation event to obtain an object storage link, and downloading the data to be exchanged through the object storage link;
[0009] Convert the data to be exchanged according to the object storage format, store the converted data to be exchanged as object storage data, and send a data conversion completion event to the message queue for subscription by the data consumer;
[0010] A data acquisition request is received from the data consumer, and a data download link is sent to the data consumer according to the data acquisition request, so that the data consumer downloads data according to the data download link.
[0011] Optionally, the parsing from the data generation event to obtain an object storage link, and downloading the to-be-exchanged data through the object storage link includes:
[0012] Obtaining the data packet size of the to-be-exchanged data from the object storage link;
[0013] When the size of the data packet is less than or equal to the first data threshold, the data packets corresponding to the data to be exchanged are aggregated and downloaded;
[0014] When the size of the data packet is greater than the first data threshold and less than the second data threshold, directly downloading the data packet from the object storage link to obtain the data to be exchanged; wherein the first data threshold is less than the second data threshold;
[0015] When the data packet size is greater than or equal to the second data threshold, the data packet is segmented according to the data packet size to obtain multiple fragments for downloading separately, and after all fragments are downloaded, all fragments are reassembled to obtain the data to be exchanged.
[0016] Optionally, when the size of the data packet is less than or equal to a first data threshold, aggregating and downloading the data packets corresponding to the data to be exchanged includes:
[0017] When the size of the data packet is less than or equal to a first data threshold, caching the data packet in a temporary buffer area;
[0018] Whenever a new data packet is added to the temporary buffer, the sum of the sizes of all data packets currently in the temporary buffer is checked;
[0019] When the total size of the data packets in the temporary buffer area is greater than the first data threshold, all the data packets in the temporary buffer area are downloaded in batches to obtain the data to be exchanged, and the temporary buffer area is cleared.
[0020] Optionally, after obtaining an object storage link by parsing the data generation event and downloading the data to be exchanged through the object storage link, the method further includes:
[0021] Parsing the data generation event to obtain a first hash value of the data to be exchanged; wherein the first hash value is obtained by performing a hash calculation on the data to be exchanged generated by the data generation end;
[0022] A second Hash value of the data to be exchanged downloaded from the object storage link is calculated, and the integrity of the data to be exchanged during the downloading process is verified by comparing whether the first Hash value and the second Hash value are the same.
[0023] Optionally, converting the data to be exchanged according to an object storage format and storing the converted data to be exchanged as object storage data includes:
[0024] Parsing the data generation event to obtain a unique identifier of the data to be exchanged;
[0025] Before performing data conversion on the data to be exchanged, using the unique identifier as a query condition to find out whether the unique identifier has been stored;
[0026] If yes, determining whether to ignore storage or update storage of the converted data to be exchanged according to the service attributes of the data to be exchanged;
[0027] If not, the converted data to be exchanged and the unique identifier are stored.
[0028] Optionally, converting the data to be exchanged according to an object storage format and storing the converted data to be exchanged as object storage data includes:
[0029] Cleaning the data to be exchanged and formatting the data according to the data format of the data to be exchanged to obtain the data to be stored;
[0030] Obtain a JSON subclass defined based on a domain model layer and a logical model layer; wherein the domain model layer is a model determined based on the business layer, the logical model layer is a model abstracted from the domain model layer for defining entity objects and relationships, and the JSON subclass is used to extract entity objects from the data to be exchanged;
[0031] The data to be stored is converted into a format of a JSON subclass, the data is serialized according to the structure of the JSON subclass, and the converted data to be stored is stored according to a preset storage specification of the object storage to obtain the object storage data.
[0032] In order to achieve the above object, the present invention further provides a real-time data exchange device, the device comprising:
[0033] An acquisition module, used to acquire a data generation event subscribed from a message queue; wherein the data generation event is a notification event sent to the message queue by the data generation end after generating the data to be exchanged;
[0034] A parsing module, used for parsing the data generation event to obtain an object storage link, and downloading the data to be exchanged through the object storage link;
[0035] A conversion module, used for converting the data to be exchanged according to the object storage format, storing the converted data to be exchanged as object storage data, and sending a data conversion completion event to the message queue for subscription by the data consumer;
[0036] The consumption module is used to receive a data acquisition request from the data consumption end, and send a data download link to the data consumption end according to the data acquisition request, so that the data consumption end can download data according to the data download link.
[0037] Optionally, the parsing module is further used to:
[0038] Obtaining the data packet size of the to-be-exchanged data from the object storage link;
[0039] When the size of the data packet is less than or equal to the first data threshold, the data packets corresponding to the data to be exchanged are aggregated and downloaded;
[0040] When the size of the data packet is greater than the first data threshold and less than the second data threshold, directly downloading the data packet from the object storage link to obtain the data to be exchanged; wherein the first data threshold is less than the second data threshold;
[0041] When the data packet size is greater than or equal to the second data threshold, the data packet is segmented according to the data packet size to obtain multiple fragments for downloading separately, and after all fragments are downloaded, all fragments are reassembled to obtain the data to be exchanged.
[0042] In order to achieve the above-mentioned purpose, the present invention also provides a computer device, which specifically includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the real-time data exchange method introduced above when executing the computer program.
[0043] In order to achieve the above object, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the real-time data exchange method introduced above are implemented.
[0044] The real-time data exchange method, device, equipment and readable storage medium provided by the present invention integrate and distribute data through a public data terminal, and realize real-time data exchange among the data generation terminal, the public data terminal and the data consumption terminal through a message queue, thereby ensuring that the data can be quickly discovered and utilized once generated, thereby enhancing the liquidity and value of the data. The data is downloaded through a download link and stored in an object mode, thereby not only providing a safe and efficient download and storage solution for massive data, but also simplifying the data acquisition process and improving the liquidity of the data. In addition, the data storage format is standardized through the data format conversion function, so that the public data can be presented to various consumer terminals in a standardized form during data storage and data consumption, thereby improving the availability and interoperability of the data. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0046] Figure 1 A schematic diagram of an optional flow chart of the real-time data exchange method provided in Example 1;
[0047] Figure 2 A schematic diagram of an optional composition structure of a real-time data exchange device provided in Embodiment 2;
[0048] Figure 3 A schematic diagram of an optional hardware architecture of a computer device provided in Example 3. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0050] Embodiment 1
[0051] The embodiment of the present invention provides a real-time data exchange method, such as Figure 1 As shown, the method is applied to a public data terminal and specifically includes the following steps:
[0052] Step S101: obtaining a data generation event subscribed from a message queue; wherein the data generation event is a notification event sent by the data generating end to the message queue after generating data to be exchanged.
[0053] Specifically, the public data end is an intermediate data platform established based on the common data environment (COMMON DATA ENVIRONMENT) to connect the data generation end and the data consumption end. It is used to obtain the data from the data generation end and convert it into a standard format for consumption by the data consumption end. It converts the original point-to-point data interaction between the data generation end and the data consumption end into a centralized star structure, reducing the complexity of data interaction.
[0054] In this embodiment, a message queue is introduced to establish a real-time data notification function for the data generation end, the public data end, and the data consumption end. In addition, the message queue can also help to unlock a new data generation end or data consumption end at any time. Only by subscribing to the corresponding notification event can you quickly access the data exchange system.
[0055] Step S102: parsing the data generation event to obtain an object storage link, and downloading the data to be exchanged through the object storage link.
[0056] Among them, the data generation event includes an object storage link, and the object storage link is used to download the data packet of the data to be exchanged, so that the public data end can download the data to be exchanged through the object storage link, and transmit the consumed data based on the object storage external link, so that the business data interface and the ordinary interface are separated to ensure data transmission efficiency, while the bandwidth of the ordinary interface is not occupied.
[0057] In this embodiment, corresponding download transmission logic is designed for the data packet size of the data to be exchanged. When the data packet size is less than or equal to the first data threshold, an aggregate download method is adopted. When the data packet size is greater than the first data threshold and less than the second data threshold, a direct download method is adopted. When the data packet size is greater than or equal to the second data threshold, a slice download method is adopted. Data packets of different sizes can be downloaded quickly and orderly, saving transmission resources and improving download efficiency.
[0058] Step S103: converting the data to be exchanged according to the object storage format, storing the converted data to be exchanged as object storage data, and sending a data conversion completion event to the message queue for subscription by the data consumer.
[0059] The object storage format is a data storage architecture that manages data as objects. Each object contains the data itself, metadata, and a unique identifier. The data to be exchanged is converted into the object storage format so that it can be stored and retrieved in the object storage system of the public data end.
[0060] Specifically, the data conversion completion event includes detailed information of the object data that has been converted, such as storage location, unique identifier, etc., so that the data consumer can consume the data according to the information in the above event.
[0061] Step S104: receiving a data acquisition request from the data consumer, and sending a data download link to the data consumer according to the data acquisition request, so that the data consumer downloads data according to the data download link.
[0062] Among them, the message queue is used to achieve efficient data flow between the data generation end, the public data end and the data consumption end. Once the data generation end generates the data to be exchanged, it immediately sends a notification event through the message queue to ensure the timely delivery of data. At the same time, after the data conversion is completed, the data consumption end is also notified through the message queue, realizing the rapid exchange of data.
[0063] Wherein, the step S102 includes the following steps:
[0064] Step A1: Obtain the data packet size of the data to be exchanged from the object storage link.
[0065] Step A2: when the size of the data packet is less than or equal to the first data threshold, the data packets corresponding to the data to be exchanged are aggregated and downloaded.
[0066] Among them, smaller data packets are aggregated in the cache area, and batch aggregated and downloaded after the download conditions are met, which can save bandwidth and reduce the frequency of calling the interface.
[0067] Step A3: When the size of the data packet is greater than the first data threshold and less than the second data threshold, directly download the data packet from the object storage link to obtain the data to be exchanged; wherein the first data threshold is less than the second data threshold.
[0068] Step A4: When the data packet size is greater than or equal to the second data threshold, the data packet is segmented according to the data packet size to obtain multiple fragments for downloading separately, and after all fragments are downloaded, all fragments are reassembled to obtain the data to be exchanged.
[0069] Among them, after slicing and downloading the super large data packet, in order to ensure the integrity and accuracy of the data to be exchanged before and after the slice transmission, it is also necessary to use the hash value method to perform integrity verification on the data before and after the slice to ensure fast data transmission while ensuring data security. If data corruption is found, the data to be exchanged in the data packet needs to be downloaded again.
[0070] Specifically, the step A2 further includes the following steps:
[0071] Step B1: When the size of the data packet is less than or equal to a first data threshold, the data packet is cached in a temporary cache area.
[0072] Step B2: Whenever a new data packet is added to the temporary buffer, the total size of all data packets currently in the temporary buffer is checked.
[0073] Step B3: When the total size of the data packets in the temporary buffer area is greater than the first data threshold, all the data packets in the temporary buffer area are downloaded in batches to obtain the data to be exchanged, and the temporary buffer area is cleared.
[0074] Furthermore, after obtaining an object storage link by parsing the data generation event and downloading the data to be exchanged through the object storage link, the method further includes:
[0075] Step C1: parsing the data generation event to obtain a first hash value of the data to be exchanged; wherein the first hash value is obtained by performing a hash calculation on the data to be exchanged generated by the data generation end;
[0076] Step C2: Calculate a second hash value of the data to be exchanged downloaded from the object storage link, and verify the integrity of the data to be exchanged during the download process by comparing whether the first hash value and the second hash value are the same.
[0077] Since the public data terminal will convert the data format downloaded from the data generation end and store it, when the data is stored in the public data terminal, it is not exactly the same as the data to be exchanged generated by the original data generation end, so it is impossible to compare the stored data with the original data to be exchanged. Therefore, by comparing the first hash value of the original data to be exchanged and the second hash value of the downloaded data to be exchanged to see if they are the same, it can be checked whether the data is missing or damaged during the transmission and download process, resulting in data incompleteness, so as to solve data transmission problems in a timely manner and ensure data integrity.
[0078] Wherein, the step S103 further includes the following steps:
[0079] Step D1: parse the data generation event to obtain a unique identifier of the data to be exchanged.
[0080] Step D2: Before performing data conversion on the data to be exchanged, use the unique identifier as a query condition to find out whether the unique identifier has been stored.
[0081] Step D3: If yes, determining whether to perform ignore storage or update storage on the converted data to be exchanged according to the service attributes of the data to be exchanged.
[0082] Step D4: If not, store the converted data to be exchanged and the unique identifier.
[0083] In this embodiment, since the public data terminal converts the data format downloaded from the data generation terminal and stores it, after the data is stored in the public data terminal, it is not completely the same as the data to be exchanged generated by the original data generation terminal, so it is impossible to compare the stored data with the original data to be exchanged, that is, it is impossible to avoid the situation where duplicate data storage causes data waste. Therefore, in this embodiment, a unique identifier is used to detect whether the public data terminal has stored the same data. If the same data exists, the processing method can be determined according to the business type, such as ignoring the storage or updating the storage. In order to save the storage resources of the public data terminal.
[0084] Specifically, the step S103 further includes the following steps:
[0085] Step E1: clean the data to be exchanged, and format the data according to the data format of the data to be exchanged to obtain data to be stored.
[0086] The data cleaning includes data processing such as removing duplicate data and invalid characters. The formatting includes adjusting the format of the target type of data according to a preset format, such as processing the data representing the time type "April 12, 2023" as "2023.04.12" according to the preset format, so as to ensure a high degree of uniformity of the data format, improve the data quality, and provide a reliable data foundation for subsequent consumer ends. It is convenient for data to be exchanged and shared between different systems or applications.
[0087] Step E2: Obtain a JSON subclass defined based on a domain model layer and a logical model layer; wherein the domain model layer is a model determined based on the business level, the logical model layer is a model abstracted from the domain model layer for defining entity objects and relationships, and the JSON subclass is used to extract entity objects from the data to be exchanged.
[0088] Wherein, the domain model layer is a model built based on the business level, reflecting the entities, attributes and relationships in the actual business scenario. The logical model layer is further abstracted on the basis of the domain model layer, defining more general entity objects and the relationships between them, so as to facilitate cross-domain data processing and exchange. The JSON subclass is a JSON structure template generated according to the specific entity object defined in the logical model layer. These JSON subclasses are JSON structure templates generated according to the specific entity object defined in the logical model layer. These subclasses are used to extract information of specific entity objects from the data to be exchanged.
[0089] In this embodiment, through the construction of domain model and logical model, the abstraction and standardization of business data are realized, which is convenient for data management and reuse. As a medium for data exchange, JSON subclass is lightweight, easy to read and write, and convenient for data serialization and deserialization. It improves the flexibility and scalability of data and facilitates data storage and data consumption.
[0090] Step E3: convert the data to be stored into the format of a JSON subclass, serialize the data according to the structure of the JSON subclass, and store the converted data to be stored according to a preset object storage specification to obtain the object storage data.
[0091] The serialization process is a process of converting the data to be stored into the format of a JSON subclass and then converting it into a format that can be stored or transmitted according to the data structure object state of the JSON subclass. The serialization process ensures the consistency and integrity of the data during storage and transmission.
[0092] In this embodiment, data is integrated and distributed through a public data terminal, and real-time data exchange is achieved among the data generation terminal, the public data terminal, and the data consumption terminal through a message queue, ensuring that the data can be quickly discovered and utilized once generated, thereby enhancing the liquidity and value of the data. Data is downloaded through a download link and stored in object mode, which not only provides a safe and efficient download and storage solution for massive data, but also simplifies the data acquisition process and improves the liquidity of data. In addition, the data storage format is standardized through the data format conversion function, so that public data can be presented to various consumer terminals in a standardized form during data storage and consumption, thereby improving the availability and interoperability of data.
[0093] Embodiment 2
[0094] The embodiment of the present invention provides a real-time data exchange device, such as Figure 2 As shown, the device specifically includes the following components:
[0095] The acquisition module 201 is used to acquire a data generation event subscribed from a message queue; wherein the data generation event is a notification event sent by the data generation end to the message queue after generating the data to be exchanged;
[0096] A parsing module 202, configured to parse the data generation event to obtain an object storage link, and download the data to be exchanged through the object storage link;
[0097] The conversion module 203 is used to convert the data to be exchanged according to the object storage format, store the converted data to be exchanged as object storage data, and send a data conversion completion event to the message queue for subscription by the data consumer;
[0098] The consumption module 204 is used to receive a data acquisition request from the data consumer, and send a data download link to the data consumer according to the data acquisition request, so that the data consumer can download data according to the data download link.
[0099] The analysis module 202 is used to:
[0100] Obtaining the data packet size of the to-be-exchanged data from the object storage link;
[0101] When the size of the data packet is less than or equal to the first data threshold, the data packets corresponding to the data to be exchanged are aggregated and downloaded;
[0102] When the size of the data packet is greater than the first data threshold and less than the second data threshold, directly downloading the data packet from the object storage link to obtain the data to be exchanged; wherein the first data threshold is less than the second data threshold;
[0103] When the data packet size is greater than or equal to the second data threshold, the data packet is segmented according to the data packet size to obtain multiple fragments for downloading separately, and after all fragments are downloaded, all fragments are reassembled to obtain the data to be exchanged.
[0104] Specifically, the parsing module 202 is further used for:
[0105] When the size of the data packet is less than or equal to a first data threshold, caching the data packet in a temporary buffer area;
[0106] Whenever a new data packet is added to the temporary buffer, the sum of the sizes of all data packets currently in the temporary buffer is checked;
[0107] When the total size of the data packets in the temporary buffer area is greater than the first data threshold, all the data packets in the temporary buffer area are downloaded in batches to obtain the data to be exchanged, and the temporary buffer area is cleared.
[0108] Furthermore, the parsing module 202 is also used for:
[0109] Parsing the data generation event to obtain a first hash value of the data to be exchanged; wherein the first hash value is obtained by performing a hash calculation on the data to be exchanged generated by the data generation end;
[0110] A second Hash value of the data to be exchanged downloaded from the object storage link is calculated, and the integrity of the data to be exchanged during the downloading process is verified by comparing whether the first Hash value and the second Hash value are the same.
[0111] The conversion module 203 is used for:
[0112] Parsing the data generation event to obtain a unique identifier of the data to be exchanged;
[0113] Before performing data conversion on the data to be exchanged, using the unique identifier as a query condition to find out whether the unique identifier has been stored;
[0114] If yes, determining whether to ignore storage or update storage of the converted data to be exchanged according to the service attributes of the data to be exchanged;
[0115] If not, the converted data to be exchanged and the unique identifier are stored.
[0116] Specifically, the conversion module 203 is further used for:
[0117] Cleaning the data to be exchanged and formatting the data according to the data format of the data to be exchanged to obtain the data to be stored;
[0118] Obtain a JSON subclass defined based on a domain model layer and a logical model layer; wherein the domain model layer is a model determined based on the business layer, the logical model layer is a model abstracted from the domain model layer for defining entity objects and relationships, and the JSON subclass is used to extract entity objects from the data to be exchanged;
[0119] The data to be stored is converted into a format of a JSON subclass, the data is serialized according to the structure of the JSON subclass, and the converted data to be stored is stored according to a preset storage specification of the object storage to obtain the object storage data.
[0120] Embodiment 3
[0121] This embodiment also provides a computer device, such as a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs. Figure 3 As shown, the computer device 30 of this embodiment includes at least but not limited to: a memory 301 and a processor 302 that can communicate with each other via a system bus. It should be noted that Figure 3 Only computer device 30 is shown with components 301 - 302 , but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.
[0122] In this embodiment, the memory 301 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 301 may be an internal storage unit of the computer device 30, such as a hard disk or a memory of the computer device 30. In other embodiments, the memory 301 may also be an external storage device of the computer device 30, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. equipped on the computer device 30. Of course, the memory 301 may also include both the internal storage unit and the external storage device of the computer device 30. In this embodiment, the memory 301 is generally used to store the operating system and various application software installed on the computer device 30. In addition, the memory 301 may also be used to temporarily store various types of data that have been output or are to be output.
[0123] In some embodiments, the processor 302 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other real-time data exchange chips. The processor 302 is generally used to control the overall operation of the computer device 30 .
[0124] Specifically, in this embodiment, the processor 302 is used to execute the program of the real-time data exchange method stored in the memory 301, and the program of the real-time data exchange method implements the following steps when being executed:
[0125] Obtaining a data generation event subscribed to from a message queue; wherein the data generation event is a notification event sent by a data generation end to the message queue after generating data to be exchanged;
[0126] Parsing the data generation event to obtain an object storage link, and downloading the data to be exchanged through the object storage link;
[0127] Convert the data to be exchanged according to the object storage format, store the converted data to be exchanged as object storage data, and send a data conversion completion event to the message queue for subscription by the data consumer;
[0128] A data acquisition request is received from the data consumer, and a data download link is sent to the data consumer according to the data acquisition request, so that the data consumer downloads data according to the data download link.
[0129] The specific implementation process of the above method steps can be found in Example 1, and this embodiment will not be repeated here.
[0130] Embodiment 4
[0131] This embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored. When the computer program is executed by a processor, the following method steps are implemented:
[0132] Obtaining a data generation event subscribed to from a message queue; wherein the data generation event is a notification event sent by a data generation end to the message queue after generating data to be exchanged;
[0133] Parsing the data generation event to obtain an object storage link, and downloading the data to be exchanged through the object storage link;
[0134] Convert the data to be exchanged according to the object storage format, store the converted data to be exchanged as object storage data, and send a data conversion completion event to the message queue for subscription by the data consumer;
[0135] A data acquisition request is received from the data consumer, and a data download link is sent to the data consumer according to the data acquisition request, so that the data consumer downloads data according to the data download link.
[0136] The specific implementation process of the above method steps can be found in Example 1, and this embodiment will not be repeated here.
[0137] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0138] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.
[0140] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A real-time data exchange method, characterized in that: Applied to a public data terminal, the method comprises: Obtaining a data generation event subscribed to from a message queue; wherein the data generation event is a notification event sent by a data generation end to the message queue after generating data to be exchanged; Parsing the data generation event to obtain an object storage link, and downloading the data to be exchanged through the object storage link; Convert the data to be exchanged according to the object storage format, store the converted data to be exchanged as object storage data, and send a data conversion completion event to the message queue for subscription by the data consumer; A data acquisition request is received from the data consumer, and a data download link is sent to the data consumer according to the data acquisition request, so that the data consumer downloads data according to the data download link.
2. The real-time data exchange method according to claim 1, characterized in that: The step of parsing the data generation event to obtain an object storage link, and downloading the data to be exchanged through the object storage link, includes: Obtaining the data packet size of the to-be-exchanged data from the object storage link; When the size of the data packet is less than or equal to the first data threshold, the data packets corresponding to the data to be exchanged are aggregated and downloaded; When the size of the data packet is greater than the first data threshold and less than the second data threshold, directly downloading the data packet from the object storage link to obtain the data to be exchanged; wherein the first data threshold is less than the second data threshold; When the data packet size is greater than or equal to the second data threshold, the data packet is segmented according to the data packet size to obtain multiple fragments for downloading separately, and after all fragments are downloaded, all fragments are reassembled to obtain the data to be exchanged.
3. The real-time data exchange method according to claim 2, characterized in that: When the size of the data packet is less than or equal to the first data threshold, the data packets corresponding to the data to be exchanged are aggregated and downloaded, including: When the size of the data packet is less than or equal to a first data threshold, caching the data packet in a temporary buffer area; Whenever a new data packet is added to the temporary buffer, the sum of the sizes of all data packets currently in the temporary buffer is checked; When the total size of the data packets in the temporary buffer area is greater than the first data threshold, all the data packets in the temporary buffer area are downloaded in batches to obtain the data to be exchanged, and the temporary buffer area is cleared.
4. The real-time data exchange method according to claim 1, characterized in that: After obtaining an object storage link by parsing the data generation event and downloading the data to be exchanged through the object storage link, the method further includes: Parsing the data generation event to obtain a first hash value of the data to be exchanged; wherein the first hash value is obtained by performing a hash calculation on the data to be exchanged generated by the data generation end; A second Hash value of the data to be exchanged downloaded from the object storage link is calculated, and the integrity of the data to be exchanged during the downloading process is verified by comparing whether the first Hash value and the second Hash value are the same.
5. The real-time data exchange method according to claim 1, characterized in that: The step of converting the data to be exchanged according to the object storage format and storing the converted data to be exchanged as object storage data includes: Parsing the data generation event to obtain a unique identifier of the data to be exchanged; Before performing data conversion on the data to be exchanged, using the unique identifier as a query condition to find out whether the unique identifier has been stored; If yes, determining whether to ignore storage or update storage of the converted data to be exchanged according to the service attributes of the data to be exchanged; If not, the converted data to be exchanged and the unique identifier are stored.
6. The real-time data exchange method according to claim 1, characterized in that: The step of converting the data to be exchanged according to the object storage format and storing the converted data to be exchanged as object storage data includes: Cleaning the data to be exchanged and formatting the data according to the data format of the data to be exchanged to obtain the data to be stored; Obtain a JSON subclass defined based on a domain model layer and a logical model layer; wherein the domain model layer is a model determined based on the business layer, the logical model layer is a model abstracted from the domain model layer for defining entity objects and relationships, and the JSON subclass is used to extract entity objects from the data to be exchanged; The data to be stored is converted into a format of a JSON subclass, the data is serialized according to the structure of the JSON subclass, and the converted data to be stored is stored according to a preset storage specification of the object storage to obtain the object storage data.
7. A real-time data exchange device, characterized in that: The device comprises: An acquisition module, used to acquire a data generation event subscribed from a message queue; wherein the data generation event is a notification event sent to the message queue by the data generation end after generating the data to be exchanged; A parsing module, used for parsing the data generation event to obtain an object storage link, and downloading the data to be exchanged through the object storage link; A conversion module, used for converting the data to be exchanged according to the object storage format, storing the converted data to be exchanged as object storage data, and sending a data conversion completion event to the message queue for subscription by the data consumer; The consumption module is used to receive a data acquisition request from the data consumption end, and send a data download link to the data consumption end according to the data acquisition request, so that the data consumption end can download data according to the data download link.
8. The real-time data exchange device according to claim 7, characterized in that: The parsing module is also used for: Obtaining the data packet size of the to-be-exchanged data from the object storage link; When the size of the data packet is less than or equal to the first data threshold, the data packets corresponding to the data to be exchanged are aggregated and downloaded; When the size of the data packet is greater than the first data threshold and less than the second data threshold, directly downloading the data packet from the object storage link to obtain the data to be exchanged; wherein the first data threshold is less than the second data threshold; When the data packet size is greater than or equal to the second data threshold, the data packet is segmented according to the data packet size to obtain multiple fragments for downloading separately, and after all fragments are downloaded, all fragments are reassembled to obtain the data to be exchanged.
9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
