A data query method and device based on a real-time atlas platform

Abstract

The application provides a data query method and device based on a real-time atlas platform. The method comprises the following steps: constructing a real-time atlas platform; receiving a data query request; performing data query in the real-time atlas platform according to the data query request, obtaining a data query result; performing correlation analysis on the data query result through the real-time atlas platform, obtaining correlation atlas data; and outputting the correlation atlas data. It can be seen that the method can construct a real-time atlas platform to realize the effect of stream batch integration, thereby reducing the cost of maintenance operation and learning; and the atlas platform can also generate higher-quality atlas data, thereby improving the development efficiency of the model and realizing the rapid iteration of the model.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data query method and apparatus based on a real-time graph platform. Background Technology

[0002] The current industry-standard data warehouse architecture is the Lambda architecture, which adds a layer for processing real-time data on top of a traditional offline data warehouse. Data from the offline warehouse and the real-time data pipeline is then merged at the Serving layer to enable querying of both offline and real-time data. However, as data volume and application complexity increase, the Lambda architecture inevitably encounters several problems, such as high development and maintenance costs, high learning curves, wasted space and poor data consistency due to storing data multiple times in different views, and difficulty in use due to the use of different languages. Summary of the Invention

[0003] The purpose of this application is to provide a data query method and apparatus based on a real-time graph platform, which can build a real-time graph platform to achieve the effect of integrated batch processing, thereby reducing the cost of maintenance, operation and learning; it can also generate higher quality graph data through the graph platform, thereby improving the development efficiency of the model and realizing rapid iteration of the model.

[0004] The first aspect of this application provides a data query method based on a real-time graph platform, including:

[0005] Build a real-time graph platform;

[0006] Receive data query requests;

[0007] According to the data query request, a data query is performed on the real-time graph platform to obtain the data query results;

[0008] The real-time graph platform is used to perform correlation analysis on the data query results to obtain correlated graph data;

[0009] Output the associated map data.

[0010] In the above implementation process, this method can first construct a real-time graph platform; receive data query requests; then perform data queries on the real-time graph platform according to the data query requests to obtain data query results; then, perform correlation analysis on the data query results through the real-time graph platform to obtain correlated graph data; finally, output the correlated graph data. It is evident that this method can construct a real-time graph platform to achieve integrated stream and batch processing, thereby reducing maintenance, operation, and learning costs; it can also generate higher-quality graph data through this graph platform, thereby improving model development efficiency and enabling rapid model iteration.

[0011] Furthermore, the real-time map construction platform includes:

[0012] Configure data sources, real-time data warehouses, graph databases, message subscription platforms, and stream processing frameworks; wherein, the data sources include financial business data, insurance business data, user behavior data, data from third-party platforms, and operational result data;

[0013] A real-time graph platform is constructed based on the data source, the real-time data warehouse, the graph database, the message subscription platform, and the stream processing framework.

[0014] Furthermore, after constructing the real-time atlas platform, the method further includes:

[0015] Receive raw data from the data source; wherein the raw data includes real-time data and / or offline data;

[0016] The raw data is stored in the real-time data warehouse and the graph database of the real-time graph platform, respectively.

[0017] Further, storing the raw data in the real-time data warehouse and the graph database of the real-time graph platform, respectively, includes:

[0018] When the raw data includes real-time data, the real-time data is sent to the message subscription platform;

[0019] The real-time data of the message subscription platform is processed by the stream processing framework to obtain the first processed data;

[0020] The first processed data is stored in the real-time data warehouse and the graph database, respectively.

[0021] Further, storing the raw data in the real-time data warehouse and the graph database of the real-time graph platform, respectively, includes:

[0022] When the raw data includes offline data, the offline data is sent to the real-time data warehouse for storage using data warehouse technology;

[0023] The real-time data warehouse is connected through the connector of the streaming framework to process the offline data in real time through the streaming framework to obtain streaming data;

[0024] The stream processing data is stored in the graph database, and data labels are set for the stream processing data using the machine learning library of the stream processing framework to obtain the second processed data;

[0025] The second processed data is stored in the real-time data warehouse.

[0026] Furthermore, the type characteristics of the real-time graph platform include location identification type, device association type, graph relationship type, and statistical community feature type.

[0027] A second aspect of this application provides a data query device based on a real-time graph platform, the data query device based on the real-time graph platform comprising:

[0028] Building blocks are used to construct real-time graph platforms;

[0029] The receiving unit is used to receive data query requests;

[0030] The query unit is used to perform data query on the real-time graph platform according to the data query request and obtain the data query result;

[0031] The correlation analysis unit is used to perform correlation analysis on the data query results through the real-time graph platform to obtain correlation graph data;

[0032] The output unit is used to output the associated map data.

[0033] In the above implementation process, the device can construct a real-time graph platform through a construction unit; receive data query requests through a receiving unit; perform data queries on the real-time graph platform according to the data query requests through a query unit to obtain data query results; perform correlation analysis on the data query results through the real-time graph platform through a correlation analysis unit to obtain correlated graph data; and finally, output the correlated graph data through an output unit. It is evident that this device can construct a real-time graph platform to achieve integrated batch processing, thereby reducing maintenance, operation, and learning costs; it can also generate higher-quality graph data through this graph platform, thereby improving model development efficiency and enabling rapid model iteration.

[0034] Furthermore, the building unit includes:

[0035] The configuration subunit is used to configure the data source, real-time data warehouse, graph database, message subscription platform, and stream processing framework; wherein, the data source includes financial business data, insurance business data, user behavior data, data from third-party platforms, and operational result data;

[0036] A sub-unit is constructed to build a real-time graph platform based on the data source, the real-time data warehouse, the graph database, the message subscription platform, and the stream processing framework.

[0037] Furthermore, the data query device based on the real-time graph platform also includes:

[0038] The receiving unit is further configured to receive raw data from the data source after the real-time map platform is constructed; wherein the raw data includes real-time data and / or offline data;

[0039] A storage unit is used to store the raw data into the real-time data warehouse and the graph database of the real-time graph platform, respectively.

[0040] Furthermore, the storage unit includes:

[0041] A sending subunit is used to send the real-time data to the message subscription platform when the original data includes real-time data;

[0042] The processing subunit is used to process the real-time data of the message subscription platform through the stream processing framework to obtain the first processed data;

[0043] The storage subunit is used to store the first processed data into the real-time data warehouse and the graph database, respectively.

[0044] Furthermore, the storage unit includes:

[0045] The sending subunit is used to send the offline data to the real-time data warehouse for storage using data warehouse technology when the original data includes offline data;

[0046] The processing subunit is used to connect to the real-time data warehouse through the connector of the streaming processing framework, so as to process the offline data in real time through the streaming processing framework to obtain streaming processed data;

[0047] The storage subunit is used to store the stream processing data in the graph database and set data tags for the stream processing data through the machine learning library of the stream processing framework to obtain the second processed data;

[0048] The storage subunit is also used to store the second processed data to the real-time data warehouse.

[0049] Furthermore, the type characteristics of the real-time graph platform include location identification type, device association type, graph relationship type, and statistical community feature type.

[0050] A third aspect of this application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the data query method based on a real-time graph platform as described in any one of the first aspects of this application.

[0051] The fourth aspect of this application provides a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the data query method based on a real-time graph platform as described in any one of the first aspects of this application. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating a data query method based on a real-time graph platform provided in this application embodiment;

[0054] Figure 2 A flowchart illustrating another data query method based on a real-time graph platform provided in this application embodiment;

[0055] Figure 3 A schematic diagram of a data query device based on a real-time graph platform provided in this application embodiment;

[0056] Figure 4 This is a schematic diagram of another data query device based on a real-time graph platform provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0058] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0059] Example 1

[0060] Please refer to Figure 1 , Figure 1 This embodiment provides a flowchart illustrating a data query method based on a real-time geographic graph platform. The data query method based on the real-time geographic graph platform includes:

[0061] S101. Construct a real-time graph platform.

[0062] In this embodiment, the type characteristics of the real-time graph platform include location identification type, device association type, graph relationship type, and statistical community feature type.

[0063] S102, Receive data query request.

[0064] S103. Perform a data query on the real-time graph platform according to the data query request and obtain the data query results.

[0065] S104. Perform correlation analysis on the data query results through the real-time graph platform to obtain the correlation graph data.

[0066] S105, Output the correlation map data.

[0067] In this embodiment, the subject executing the method can be a computing device such as a computer or server, and no limitation is made in this embodiment.

[0068] In this embodiment, the subject executing the method can also be a smart device such as a smartphone or tablet, and no limitation is made in this embodiment.

[0069] As can be seen, implementing the data query method based on a real-time graph platform described in this embodiment can build a real-time graph platform, achieve batch processing integration, reduce learning and operation costs, improve model development efficiency, and quickly achieve model iteration.

[0070] Example 2

[0071] Please refer to Figure 2 , Figure 2 This embodiment provides a flowchart illustrating a data query method based on a real-time geographic graph platform. The data query method based on the real-time geographic graph platform includes:

[0072] S201, configure data sources, real-time data warehouse, graph database, message subscription platform and stream processing framework.

[0073] In this embodiment, the data sources include financial business data, insurance business data, user behavior data, data from third-party platforms, and operational result data.

[0074] S202. Based on the data source, real-time data warehouse, graph database, message subscription platform, and stream processing framework, construct a real-time graph platform.

[0075] In this embodiment, the type characteristics of the real-time graph platform include location identification type, device association type, graph relationship type, and statistical community feature type.

[0076] S203. Receive raw data from the data source; wherein the raw data includes real-time data and / or offline data.

[0077] S204. Store the raw data in the real-time data warehouse and graph database of the real-time graph platform, respectively.

[0078] As an optional implementation, the raw data is stored in the real-time data warehouse and graph database of the real-time graph platform, respectively, including:

[0079] When the raw data includes real-time data, the real-time data is sent to the message subscription platform.

[0080] The real-time data from the message subscription platform is processed using a stream processing framework to obtain the first processed data;

[0081] The first processed data is stored in a real-time data warehouse and a graph database, respectively.

[0082] As an optional implementation, the raw data is stored in the real-time data warehouse and graph database of the real-time graph platform, respectively, including:

[0083] When the raw data includes offline data, data warehouse technology is used to send the offline data to a real-time data warehouse for storage.

[0084] Connect to the real-time data warehouse through the connector of the stream processing framework to process offline data in real time and obtain stream-processed data.

[0085] The stream processing data is stored in a graph database, and data labels are set for the stream processing data using the machine learning library of the stream processing framework to obtain the second-processed data;

[0086] The second-processed data is stored in a real-time data warehouse.

[0087] S205, Receive data query request.

[0088] S206. Perform a data query on the real-time graph platform according to the data query request and obtain the data query results.

[0089] S207. Perform correlation analysis on the data query results through a real-time graph platform to obtain correlated graph data.

[0090] S208, Output the correlation map data.

[0091] In this embodiment, a method for implementing Hologres stream and batch processing in the financial field is proposed.

[0092] (1) Regarding Hologres' storage-compute separation architecture: users can elastically scale up or down according to business needs. In distributed storage, the following three architectures are commonly used:

[0093] Shared Disk / Storage: This refers to a storage cluster with many disks mounted on it, which can be directly accessed by each compute node.

[0094] Shared Nothing: This architecture means that each compute node mounts its own storage. Nodes can communicate with each other, but the disks between nodes are not shared, which can lead to resource waste.

[0095] Storage Disaggregation: This is equivalent to treating the storage cluster as a large disk that can be accessed by each compute node. Each compute node also has a certain amount of cache space to access cached data, and there is no need to worry about the management of the storage cluster. This storage-compute separation architecture facilitates flexible expansion and can effectively save resources.

[0096] (2) Regarding Hologres' positioning as a unified storage for stream and batch processing: After data collection, data can go through different processing links, but the results after processing can be directly written to Hologres. This solves the data consistency problem and eliminates the need to distinguish between offline and real-time tables, reducing complexity and significantly lowering the learning cost for users.

[0097] (3) Hologres supports two file formats: row storage and column storage. Row storage is suitable for point query scenarios based on PK, while column storage is suitable for complex OLAP query scenarios.

[0098] (4) The Hologres execution engine is a general-purpose distributed query engine that focuses on optimizing real-time queries with high concurrency and low latency.

[0099] (5) Based on trillions of data, it provides multidimensional analysis and services. 99.99% of queries can return results within 80ms, truly achieving real-time and offline integration of data, supporting online application services, and bringing a more intelligent consumption experience to merchants and consumers.

[0100] In this embodiment, the NebulaGraph graph computation engine has the following features:

[0101] (1) NebulaGraph uses a shard-nothing distributed engine for storage, which can support the computation of trillions of graphs. It is mainly divided into three parts for combined services:

[0102] The graph service is primarily responsible for real-time graph computation.

[0103] The meta service is mainly responsible for data management, schema operations, and user permissions, etc.

[0104] The storage service is primarily responsible for data storage.

[0105] (2) Nubula adopts a compute-storage separation architecture, where both the compute layer and the storage layer can be cloned independently. It also supports compute transfer, reducing data migration. Both the meta layer and the storage layer achieve eventual data consistency through the Raft protocol.

[0106] In this embodiment, the construction process of the map platform is as follows:

[0107] Data sources include financial business data, insurance business data, user behavior data, third-party platform data, and operational results data. Real-time data is sent to Pulsar, then processed by Flink before being sent partly to Hologress and partly to NebulaGraph. Offline data is processed via ETL and sent to Hologress. Data in Hologress is processed in real-time via Flink-connector, with one part being processed using FLINK-ML online learning to handle real-time labels, and the other part being sent to NebulaGraph for further processing.

[0108] In this embodiment, Pulsar is a distributed message publish / subscribe delivery platform with a highly flexible messaging model and an intuitive client API.

[0109] In this embodiment, Apache Flink is an open-source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data stream engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute both batch processing and stream processing programs.

[0110] In this embodiment, hologress is a real-time data warehouse.

[0111] In this embodiment, NebulaGraph is an open-source, distributed, and easily scalable native graph database capable of handling massive datasets containing hundreds of billions of vertices and trillions of edges, and providing millisecond-level queries. A graph database is specifically designed to store and retrieve information from vast graph networks. It can efficiently store graph data as vertices and edges, and can also attach properties to vertices and edges.

[0112] In this embodiment, ETL stands for Extract-Transform-Load, a data warehouse technology used to describe the process of extracting, transforming, and loading data from the source to the destination.

[0113] In this embodiment, flink-connector is the connector of Flink.

[0114] In this embodiment, Flink-ML machine learning is FlinkML, which is Flink's machine learning (ML) library.

[0115] In this embodiment, the features of NebulaGraph4, the entire graph platform, can be summarized into four categories:

[0116] (1) The first type is location recognition, which is mainly based on the user's location information and the GeoHash algorithm to realize the data calculation of location cluster features. For example, this method discovers some suspicious users through location cluster features, and then through anti-fraud investigation, it checks the facial recognition photos of these users and finds that their backgrounds are very similar and that they are all applying for business with the same company. This method can combine location features with the AI ​​capabilities of image recognition to more accurately locate similar fraudulent behaviors;

[0117] (2) The second category is device association, which is mainly achieved through relationship graphs. By obtaining information on users associated with the same device, some simple fraudulent activities can be located relatively quickly;

[0118] (3) The third type is graph relationships, such as user login, registration, self-use, credit granting, etc. This method will capture information such as device fingerprints, mobile phone numbers, and contacts of users in these scenarios in real time to construct the neighbor relationships of the relationship graph. Then, by using such neighbor relationships and the degree of the nodes associated with the user, it is determined whether they are associated with some blacklisted or graylisted users to identify risks;

[0119] (4) The fourth category is the statistical community features based on community discovery algorithms. By judging the size of the community and the behavior of users in the community, statistical rule features are extracted.

[0120] In this embodiment, the query language provided by NebulaGraph supports a rich set of graph functions, such as adjacent edges, maximum path, and shortest path. Data in NebulaGraph can be used to construct valuable relationship graphs using these graph functions.

[0121] In this embodiment, the subject executing the method can be a computing device such as a computer or server, and no limitation is made in this embodiment.

[0122] In this embodiment, the subject executing the method can also be a smart device such as a smartphone or tablet, and no limitation is made in this embodiment.

[0123] As can be seen, implementing the data query method based on a real-time graph platform described in this embodiment can build a real-time graph platform, achieve batch processing integration, reduce learning and operation costs, improve model development efficiency, and quickly achieve model iteration.

[0124] Example 3

[0125] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a data query device based on a real-time graph platform provided in this embodiment. Figure 3 As shown, the data query device based on the real-time graph platform includes:

[0126] Building unit 310 is used to build a real-time graph platform;

[0127] The receiving unit 320 is used to receive data query requests;

[0128] The query unit 330 is used to perform data queries on the real-time graph platform according to the data query request and obtain the data query results.

[0129] The correlation analysis unit 340 is used to perform correlation analysis on the data query results through the real-time graph platform to obtain correlation graph data;

[0130] Output unit 350 is used to output correlation map data.

[0131] In this embodiment, the explanation of the data query device based on the real-time graph platform can be referred to the description in Embodiment 1 or Embodiment 2, and will not be repeated here.

[0132] As can be seen, implementing the data query device based on the real-time graph platform described in this embodiment can build a real-time graph platform, realize the integration of batch and stream processing, reduce learning and operation costs, and improve model development efficiency, enabling rapid model iteration.

[0133] Example 4

[0134] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of a data query device based on a real-time graph platform provided in this embodiment. Figure 4 As shown, the data query device based on the real-time graph platform includes:

[0135] Building unit 310 is used to build a real-time graph platform;

[0136] The receiving unit 320 is used to receive data query requests;

[0137] The query unit 330 is used to perform data queries on the real-time graph platform according to the data query request and obtain the data query results.

[0138] The correlation analysis unit 340 is used to perform correlation analysis on the data query results through the real-time graph platform to obtain correlation graph data;

[0139] Output unit 350 is used to output correlation map data.

[0140] As an optional implementation, the building unit 310 includes:

[0141] Configuration subunit 311 is used to configure data sources, real-time data warehouses, graph databases, message subscription platforms, and stream processing frameworks; among which, data sources include financial business data, insurance business data, user behavior data, data from third-party platforms, and operational result data;

[0142] Subunit 312 is constructed to build a real-time graph platform based on data sources, real-time data warehouses, graph databases, message subscription platforms, and stream processing frameworks.

[0143] As an optional implementation, the data query device based on the real-time map platform further includes:

[0144] The receiving unit 320 is used to receive raw data from the data source after the real-time graph platform is built; wherein the raw data includes real-time data and / or offline data;

[0145] Storage unit 360 is used to store the raw data into the real-time data warehouse and graph database of the real-time graph platform, respectively.

[0146] As an optional implementation, the storage unit 360 includes:

[0147] Sending subunit 361 is used to send real-time data to the message subscription platform when the raw data includes real-time data;

[0148] Processing subunit 362 is used to process real-time data from the message subscription platform through a stream processing framework to obtain the first processed data;

[0149] Storage subunit 363 is used to store the first processed data into the real-time data warehouse and the graph database respectively.

[0150] As an optional implementation, the storage unit 360 includes:

[0151] The sending subunit 361 is used to send offline data to a real-time data warehouse for storage using data warehouse technology when the original data includes offline data.

[0152] Processing subunit 362 is used to connect to a real-time data warehouse through a connector of the stream processing framework to process offline data in real time through the stream processing framework to obtain stream-processed data;

[0153] Storage subunit 363 is used to store stream processing data in a graph database and set data labels for the stream processing data through the machine learning library of the stream processing framework to obtain the second processing data;

[0154] Storage subunit 363 is also used to store the second processed data to a real-time data warehouse.

[0155] In this embodiment, the type characteristics of the real-time graph platform include location identification type, device association type, graph relationship type, and statistical community feature type.

[0156] In this embodiment, the explanation of the data query device based on the real-time graph platform can be referred to the description in Embodiment 1 or Embodiment 2, and will not be repeated here.

[0157] As can be seen, implementing the data query device based on the real-time graph platform described in this embodiment can build a real-time graph platform, realize the integration of batch and stream processing, reduce learning and operation costs, and improve model development efficiency, enabling rapid model iteration.

[0158] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to execute the data query method based on a real-time graph platform as described in Embodiment 1 or Embodiment 2 of this application.

[0159] This application provides a computer-readable storage medium storing computer program instructions. When these computer program instructions are read and executed by a processor, they perform the data query method based on a real-time graph platform as described in Embodiment 1 or Embodiment 2 of this application.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0161] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0162] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0165] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0166] In addition, the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A data query method based on a real-time graph platform, characterized in that, include: Build a real-time graph platform; Receive data query requests; According to the data query request, a data query is performed on the real-time graph platform to obtain the data query results; The real-time graph platform is used to perform correlation analysis on the data query results to obtain correlated graph data; Output the associated map data; The real-time graph platform is built upon data sources, a real-time data warehouse, a graph database, a message subscription platform, and a stream processing framework. After the real-time graph platform is constructed, the raw data received from the data source is stored in the real-time data warehouse and the graph database of the real-time graph platform, respectively. The step of storing the raw data in the real-time data warehouse and the graph database of the real-time graph platform includes: When the raw data includes real-time data, the real-time data is sent to the message subscription platform; The real-time data of the message subscription platform is processed by the stream processing framework to obtain the first processed data; The first processed data is stored in the real-time data warehouse and the graph database, respectively. The step of storing the raw data in the real-time data warehouse and the graph database of the real-time graph platform, respectively, further includes: When the raw data includes offline data, the offline data is sent to the real-time data warehouse for storage using data warehouse technology; The real-time data warehouse is connected through the connector of the streaming framework to process the offline data in real time through the streaming framework to obtain streaming data; The stream processing data is stored in the graph database, and data labels are set for the stream processing data using the machine learning library of the stream processing framework to obtain the second processed data; The second processed data is stored in the real-time data warehouse.

2. The data query method based on a real-time graph platform according to claim 1, characterized in that, The real-time map construction platform includes: Configure data sources, real-time data warehouses, graph databases, message subscription platforms, and stream processing frameworks; wherein, the data sources include financial business data, insurance business data, user behavior data, data from third-party platforms, and operational result data; A real-time graph platform is constructed based on the data source, the real-time data warehouse, the graph database, the message subscription platform, and the stream processing framework.

3. The data query method based on a real-time graph platform according to claim 1, characterized in that, The type characteristics of the real-time graph platform include location identification type, device association type, graph relationship type, and statistical community feature type.

4. A data query device based on a real-time graph platform, characterized in that, The data query device based on the real-time graph platform includes: Building blocks are used to construct real-time graph platforms; The receiving unit is used to receive data query requests; The query unit is used to perform data query on the real-time graph platform according to the data query request and obtain the data query result; The correlation analysis unit is used to perform correlation analysis on the data query results through the real-time graph platform to obtain correlation graph data; Output unit, used to output the associated map data; The real-time graph platform is built upon data sources, a real-time data warehouse, a graph database, a message subscription platform, and a stream processing framework. The data query device based on the real-time graph platform further includes: A storage unit is used to store the raw data received from the data source into the real-time data warehouse and the graph database respectively after the construction unit builds the real-time graph platform; The storage unit includes a sending subunit, a processing subunit, and a storage subunit: When the raw data includes real-time data, the sending subunit is used to send the real-time data to the message subscription platform; The processing subunit is used to process the real-time data of the message subscription platform through the stream processing framework to obtain the first processed data; The storage subunit is used to store the first processed data into the real-time data warehouse and the graph database, respectively; When the original data includes offline data, the sending subunit is used to send the offline data to the real-time data warehouse for storage using data warehouse technology; The processing subunit is used to connect to the real-time data warehouse through the connector of the stream processing framework, so as to process the offline data in real time through the stream processing framework to obtain stream-processed data; The storage subunit is used to store the stream processing data in the graph database and set data tags for the stream processing data through the machine learning library of the stream processing framework to obtain the second processed data; The storage subunit is also used to store the second processed data to the real-time data warehouse.

5. The data query device based on a real-time graph platform according to claim 4, characterized in that, The building unit includes: The configuration subunit is used to configure the data source, real-time data warehouse, graph database, message subscription platform, and stream processing framework; wherein, the data source includes financial business data, insurance business data, user behavior data, data from third-party platforms, and operational result data; A sub-unit is constructed to build a real-time graph platform based on the data source, the real-time data warehouse, the graph database, the message subscription platform, and the stream processing framework.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the data query method based on the real-time graph platform as described in any one of claims 1 to 3.

7. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions, which are read and executed by a processor to perform the data query method based on a real-time graph platform as described in any one of claims 1 to 3.

Images