Data processing method and device, equipment and medium

Abstract

The invention belongs to the field of big data, and relates to a data processing method which comprises the following steps: acquiring initial data; sending the initial data to a pre-constructed real-time queue system, wherein the real-time queue system is used for caching and scheduling the received data; according to a preset priority strategy and a preset scheduling strategy, performing real-time processing on the initial data in the real-time queue system to obtain candidate data; analyzing the candidate data by adopting a preset graph data model to obtain graph structure data corresponding to the initial data; and importing the graph structure data into a graph database. The invention further provides a data processing device, equipment and a medium. In addition, the invention further relates to a block chain technology, and the initial data, the candidate data and the graph structure data can be stored in a block chain. According to the method, the real-time performance, accuracy and efficiency of importing the data into the graph database can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method, device, equipment and medium. Background Art

[0002] Currently, the field of financial risk control widely uses technologies based on knowledge graphs to improve the accuracy and efficiency of risk assessment. Traditionally, using Apache Spark micro-batch processing mode to import data into graph databases has become the mainstream practice, but this method faces significant challenges.

[0003] First, the inherent latency of micro-batch processing limits its application in scenarios with high real-time requirements, such as market fluctuation monitoring and instant fraud warning. The timeliness of data processing is greatly reduced due to the batch interval. Secondly, resource utilization is inefficient. The resources reserved for micro-batch processing are wasted under low load, and resource allocation is rigid, making it difficult to dynamically adapt to load changes, affecting the overall performance of the system. Therefore, there is an urgent need for a method to improve the real-time performance of data import into graph databases. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data processing method, device, equipment and medium to solve the low real-time problem of existing data import into a graph database.

[0005] In order to solve the above technical problems, the present application provides a data processing method, which adopts the following technical solutions:

[0006] Obtain initial data; send the initial data to a pre-built real-time queue system, which is used to cache and schedule the received data; process the initial data in the real-time queue system in real time according to a preset priority strategy and a preset scheduling strategy to obtain candidate data; use a preset graph data model to parse the candidate data to obtain graph structure data corresponding to the initial data; import the graph structure data into the graph database.

[0007] Furthermore, the step of sending the initial data to the pre-built real-time queue system specifically includes:

[0008] According to the preset format conversion protocol, the data format of the initial data is converted into a standardized data exchange format to obtain the optimized target initial data; according to the preset routing rules, the target real-time queue is determined from the pre-built real-time queue system; the target initial data is pushed to the target real-time queue.

[0009] Further, according to the preset priority strategy and the preset scheduling strategy, the initial data in the real-time queue system is processed in real time to obtain the step of candidate data, which specifically includes:

[0010] According to the preset priority strategy, the priority of the target initial data in the target real-time queue is determined; the current data processing status of the target real-time queue is obtained; based on the preset scheduling strategy, priority and data processing status, the target initial data in the target real-time queue is processed in real time to obtain the processing result; the processing result is output as candidate data, and the status of the target real-time queue in the real-time queue system is updated.

[0011] Furthermore, the step of using a preset graph data model to parse the candidate data to obtain graph structure data corresponding to the initial data specifically includes:

[0012] Extract the node information, edge information and attribute key information of the candidate data; use the preset graph data model to reorganize the node information, edge information and attribute key information to generate graph structure data corresponding to the initial data.

[0013] Furthermore, the step of importing the graph structure data into the graph database specifically includes:

[0014] Parse the graph structure data to obtain node type information and edge type information of the graph structure data; obtain operation type information of the graph database; generate a first operation statement corresponding to the graph structure data based on the node type information and the operation type information, and generate a second operation statement corresponding to the graph structure data based on the edge type information and the operation type information; determine the first operation statement and the second operation statement as graph database operation statements corresponding to the graph structure data; execute the graph database operation statements through the interface of the graph database to import the graph structure data into the graph database in real time.

[0015] Furthermore, after the step of importing the graph structure data into the graph database, the method further includes:

[0016] If the graph structure data is successfully imported into the graph database, the graph structure data is removed from the real-time queue system; if the graph structure data is not successfully imported into the graph database, the graph structure data is put back into the real-time queue system and a prompt message is sent to the user terminal.

[0017] Furthermore, after the step of importing the graph structure data into the graph database, the method further includes:

[0018] Calculate the similarity between the graph structure data in the graph database and the existing data in the knowledge graph; based on the similarity, establish the association relationship between the graph structure data in the graph database and the existing data in the knowledge graph; based on the association relationship, integrate the entity information, attribute information and relationship information of the graph structure data in the graph database into the corresponding positions of the knowledge graph, and update the knowledge graph.

[0019] In order to solve the above technical problems, the embodiment of the present application further provides a data processing device, which adopts the following technical solution:

[0020] An acquisition module, used to acquire initial data;

[0021] A sending module is used to send the initial data to a pre-built real-time queue system, and the real-time queue system is used to cache and schedule the received data;

[0022] A processing module, used to process the initial data in the real-time queue system in real time according to a preset priority strategy and a preset scheduling strategy to obtain candidate data;

[0023] A parsing module is used to parse the candidate data using a preset graph data model to obtain graph structure data corresponding to the initial data;

[0024] Import module, used to import graph structure data into the graph database.

[0025] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above data processing method are implemented.

[0026] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable at least one processor to perform the steps of the above data processing method.

[0027] Compared with the prior art, the embodiment of the present application has the following beneficial effects: the real-time queue system can cache and schedule the received initial data in real time, and perform real-time processing according to the preset priority and scheduling strategy. This method significantly reduces the delay of data processing, making it more applicable to financial risk control application scenarios with extremely high real-time requirements. Through real-time analysis, it is possible to respond to market changes and potential risks more quickly, and improve the timeliness and accuracy of risk prevention and control. The real-time queue system can dynamically adjust resource allocation according to the current data processing load, avoiding the problem of resource waste or uneven load that may occur in micro-batch processing. This flexible resource management method enables the system to maintain efficient operation under different workloads and improves the overall efficiency of resource utilization. The data processing process in the real-time queue system is more continuous and controllable, reducing the data consistency problem caused by the discontinuity of data processing between batches. At the same time, through the preset priority and scheduling strategy, concurrent operations and data dependencies can be better managed to reduce the risk of data inconsistency or loss. This is crucial to ensure the accuracy and reliability of financial risk control decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0030] Figure 2 It is a flowchart of a data processing method provided by this application;

[0031] Figure 3 It is a structural schematic diagram of a data processing device provided by the present application;

[0032] Figure 4 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0037] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0038] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0039] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0040] It should be noted that the data processing method provided in the embodiment of the present application is generally executed by a server, and accordingly, the data processing device is generally set in the server.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0042] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data processing method according to the present application. The data processing method comprises the following steps:

[0043] Step S201, obtaining initial data.

[0044] In this embodiment, the electronic device (eg, Figure 1The server shown in the figure) can obtain the initial data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0045] Among them, initial data refers to structured, semi-structured or unstructured data from various channels required for building and enriching knowledge graphs. Initial data can be obtained from various business systems within financial institutions, such as credit systems, transaction systems, customer information systems, etc. Initial data can also be relevant data captured from the Internet, such as social media information, public news, etc.

[0046] Step S202: sending the initial data to a pre-built real-time queue system, where the real-time queue system is used to cache and schedule the received data.

[0047] The real-time queue system refers to a technical architecture for processing data streams, which can receive, cache and schedule data at a near real-time speed. The real-time queue system can be used to process high-concurrency, large-scale data streams to ensure that data can be effectively managed and processed in the system. The real-time queue system can temporarily store initial data and process data according to certain priority strategies and scheduling strategies.

[0048] Step S203 , according to a preset priority strategy and a preset scheduling strategy, the initial data in the real-time queue system is processed in real time to obtain candidate data.

[0049] Priority strategy refers to sorting the initial data according to certain rules or standards during the initial data processing to determine the order of processing. For example, it can be formulated based on the urgency, importance, source or other business logic of the data to ensure that key data can be processed first. For example, for data in the field of financial risk control, data involving high-risk transactions will be given a higher priority.

[0050] The scheduling policy refers to the rules and algorithms used to manage the flow and processing of data in the real-time queue system. The real-time queue system includes a target real-time queue, which includes a processing unit (such as a processor, thread or task). The scheduling policy determines when, how and by which processing unit of the target real-time queue the initial data is processed to ensure that the initial data can be processed efficiently and orderly.

[0051] Real-time processing refers to the immediate or near-instant processing and analysis of initial data to meet the time-sensitive needs of the business. Real-time processing means that after the initial data is sent to the real-time queue system, it can be quickly processed and converted to generate candidate data for subsequent steps. This processing method helps to quickly respond to business changes and improve the timeliness and accuracy of data processing.

[0052] Among them, candidate data refers to the data set after the initial data has been processed in real time. The candidate data has been preliminarily screened and sorted according to the preset priority strategy and scheduling strategy, but has not yet undergone the final analysis or decision process. Candidate data is the basis for subsequent data analysis and graph structure data generation.

[0053] Step S204: parse the candidate data using a preset graph data model to obtain graph structure data corresponding to the initial data.

[0054] Among them, the graph data model is a mathematical model used to represent and store graph structured data. The graph data model defines the types and attributes of nodes and the representation of the relationships between nodes. The graph data model is used to guide how to parse candidate data into graph structured data for storage and query in the graph database.

[0055] Among them, graph structured data refers to a data set organized according to a graph data model with clear node and edge relationships. This data form can intuitively represent the complex relationships between entities and is a core component of applications such as knowledge graphs. Graph structured data is obtained by parsing candidate data, which reflects the association and attribute information between entities in the initial data.

[0056] Step S205, importing the graph structure data into the graph database.

[0057] Among them, graph database is a database system specially used for storing and querying graph structure data. Different from traditional relational database, graph database uses graph structure as data model, providing a more intuitive and efficient way to process complex relational data. Graph database is used to store parsed graph structure data for subsequent knowledge graph construction, query and analysis operations.

[0058] In one embodiment, after the graph structure data is imported into the graph database, if the graph structure data is successfully imported into the graph database, the graph structure data can be removed from the real-time queue system. If the graph structure data is not successfully imported into the graph database, the graph structure data needs to be put back into the real-time queue system and a prompt message is sent to the user terminal.

[0059] In one embodiment, after the graph structure data is imported into the graph database, the existing knowledge graph can be updated based on the graph structure data by calculating the similarity between the graph structure data in the graph database and the existing data in the knowledge graph.

[0060] The real-time queue system of the embodiment of the present application can cache and schedule the received initial data in real time, and perform real-time processing according to the preset priority and scheduling strategy. This method significantly reduces the delay of data processing, making it more applicable to financial risk control application scenarios with extremely high real-time requirements. Through real-time analysis, it is possible to respond to market changes and potential risks more quickly, and improve the timeliness and accuracy of risk prevention and control. The real-time queue system can dynamically adjust resource allocation according to the current data processing load, avoiding the waste of resources or uneven load problems that may occur in micro-batch processing. This flexible resource management method enables the system to maintain efficient operation under different workloads, improving the overall efficiency of resource utilization. The data processing process in the real-time queue system is more continuous and controllable, reducing the data consistency problems caused by the discontinuity of data processing between batches. At the same time, through the preset priority and scheduling strategy, concurrent operations and data dependencies can be better managed to reduce the risk of data inconsistency or loss. This is crucial to ensure the accuracy and reliability of financial risk control decisions.

[0061] In some optional implementations of this embodiment, step 202, sending the initial data to a pre-built real-time queue system, specifically includes the following steps:

[0062] According to the preset format conversion protocol, the data format of the initial data is converted into a standardized data exchange format to obtain the optimized target initial data; according to the preset routing rules, the target real-time queue is determined from the pre-built real-time queue system; the target initial data is pushed to the target real-time queue.

[0063] In one example, when the initial data can be converted into a standardized data exchange format, a predefined JSON Schema (a standard and specification for describing and verifying JSON data structures) can be used as a data format conversion protocol. For example, unstructured text data is extracted and mapped according to the fields defined in the template, and structured fields such as title, author, release time, and text are extracted and converted into a lightweight data exchange format (JavaScript Object Notation, JSON). At the same time, the Base64 encoding algorithm (a coding method based on 64 printable characters to represent binary data) is used to encode multimedia data such as pictures and videos and convert them into a storable string format. The converted target initial data is obtained. According to the preset routing rules, the consistent hashing algorithm is used to distribute the optimized target initial data to different real-time queues of the real-time queue system.

[0064] The embodiment of the present application can convert the initial data into a standardized data exchange format through a preset format conversion protocol, which can ensure the consistency and comparability of data between different systems, platforms or applications. This standardization helps to reduce the complexity of data parsing and conversion, and improve the efficiency and accuracy of data processing. The standardized data format enhances the interoperability between systems, allowing data to flow and exchange seamlessly in different environments and systems. This helps to build a more flexible and scalable data processing architecture. Through preset routing rules, the target real-time queue can be intelligently selected for data push. This flexibility helps to cope with complex network environments and changing business needs, ensuring that data can reach the target location accurately and in a timely manner.

[0065] In some optional implementations of this embodiment, in step S203, the initial data in the real-time queue system is processed in real time according to a preset priority strategy and a preset scheduling strategy to obtain candidate data, which specifically includes the following steps:

[0066] According to the preset priority strategy, the priority of the target initial data in the target real-time queue is determined; the current data processing status of the target real-time queue is obtained; based on the preset scheduling strategy, priority and data processing status, the target initial data in the target real-time queue is processed in real time to obtain the processing result; the processing result is output as candidate data, and the status of the target real-time queue in the real-time queue system is updated.

[0067] In one example, the target initial data in the target real-time queue can be prioritized according to a preset priority strategy. The priority strategy can comprehensively consider factors such as the timeliness of the data and the importance of the business, and set multiple priorities. For example, a priority of 1-5 is set, with the highest priority at level 1 and the lowest priority at level 5. For example, news data generated within 1 hour is divided into priority level 1, those within 1-12 hours are divided into level 2, those within 12-24 hours are divided into level 3, those within 1-7 days are divided into level 4, and those above 7 days are divided into level 5. The current processing information of the target real-time queue is obtained, including the amount of data waiting to be processed in the target real-time queue, the average processing time, the CPU and memory usage, etc. Based on the priority and data processing status, the target initial data is processed using a scheduling algorithm, such as a weighted polling scheduling algorithm. The algorithm dynamically adjusts the weight according to the priority level and processing time, and data with high priority and short processing time obtains a higher weight, thereby giving priority to processing. After the processing is completed, the result is output as candidate data, and the real-time queue status is updated according to the processing time, data volume, etc., for subsequent task scheduling. The entire process can achieve efficient and low-latency data processing, ensuring the real-time nature of key data to the greatest extent.

[0068] The embodiment of the present application can automatically identify and sort the target initial data in the target real-time queue through a preset priority strategy. High-priority data can be processed first. Based on the preset scheduling strategy and data processing status, the data in the queue can be processed in real time, reducing waiting time and delay, and improving the efficiency of data processing. Through the combination of priority and scheduling strategy, computing resources can be allocated more reasonably to ensure that high-priority tasks receive sufficient resource support. After processing the data, the status of the real-time queue can be updated in time to ensure the accuracy and consistency of the data.

[0069] In some optional implementations of this embodiment, step S204, using a preset graph data model to parse the candidate data to obtain graph structure data corresponding to the initial data, specifically includes the following steps:

[0070] Extract the node information, edge information and attribute key information of the candidate data; use the preset graph data model to reorganize the node information, edge information and attribute key information to generate graph structure data corresponding to the initial data.

[0071] In one example, node information, edge information, and attribute key information of candidate data can be extracted. For example, named entity recognition technology can be used to identify named entities from candidate data to obtain node information. Relational words or phrases in the candidate data can be analyzed, which usually imply the relationship between entities. Relation extraction technology (such as rule-based methods, machine learning-based methods, etc.) can be used to identify these relationships, which can be used to map to edges of a graph structure to obtain edge information. Identify the attribute information of these named entities and edges to obtain attribute key information. The extracted information is normalized and converted into a unified data format. Then, the extracted node information, edge information, and attribute key information are classified and mapped according to the node and edge types defined in the graph data model to obtain graph structure data corresponding to the initial data.

[0072] The embodiments of the present application can display complex data relationships in a graphical manner through graph structure data, so that the relationship between the data is clear at a glance. In graph structure data, the association relationship between data is clearly expressed, which makes relationship-based queries very efficient. Using a preset graph data model for data reorganization can ensure that data from different sources are structurally consistent. This helps integrate and share data, making data exchange between different systems easier and more efficient.

[0073] In some optional implementations of this embodiment, step S205, importing the graph structure data into the graph database, specifically includes the following steps:

[0074] Parse the graph structure data to obtain node type information and edge type information of the graph structure data; obtain operation type information of the graph database; generate a first operation statement corresponding to the graph structure data based on the node type information and the operation type information, and generate a second operation statement corresponding to the graph structure data based on the edge type information and the operation type information; determine the first operation statement and the second operation statement as graph database operation statements corresponding to the graph structure data; execute the graph database operation statements through the interface of the graph database to import the graph structure data into the graph database in real time.

[0075] In one example, the graph structure data is parsed to identify and extract all nodes and edges. Each node and edge is classified to determine its type information, and the node type information and edge type information of the graph structure data are obtained. The operation type that the graph database needs to perform, such as insert, update, delete, etc., is determined, and the operation type information corresponding to the operation type is obtained. Based on the node type information and the operation type information, an operation statement for the node is generated to obtain the first operation statement corresponding to the graph structure data. Based on the edge type information and the operation type information, an operation statement for the edge is generated to obtain the second operation statement corresponding to the graph structure data. The first operation statement and the second operation statement can be assembled to form a complete graph database operation statement. Using the interface provided by the graph database, the generated graph database operation statement (i.e., a graph database query language) is submitted to the database for real-time writing by executing an HTTP request. After the graph data is stored in the database, it can be queried and analyzed through the graph database operation statement.

[0076] The application embodiment can significantly improve the efficiency of data import by directly parsing the graph structure data and generating operation statements for the graph database. During the data import process, the operation statements are directly generated according to the node and edge type information of the graph structure data, which can ensure that the data imported into the graph database is consistent with the original graph structure data.

[0077] In some optional implementations of this embodiment, after the graph structure data is imported into the graph database in step S205, the following steps may also be included:

[0078] If the graph structure data is successfully imported into the graph database, the graph structure data is removed from the real-time queue system; if the graph structure data is not successfully imported into the graph database, the graph structure data is put back into the real-time queue system and a prompt message is sent to the user terminal.

[0079] In one example, the storage status can be monitored in real time during the process of graph structure data being stored in the warehouse. The execution result of the write operation, such as the number of matched nodes, the number of created relationships, etc., can be returned through the return clause of the graph database operation statement (the return clause is used to specify which data should be returned after the query is executed). If the returned result is consistent with the graph structure data, it means that the storage is successful, and the graph structure data can be removed from the real-time queue system. If there is an execution failure, such as timeout, syntax error, etc., the exception can be captured, and the graph structure data can be put back into the target real-time queue and wait for the next retry. At the same time, notifications such as emails and text messages are pushed to the user terminal to prompt the reasons for the storage failure and handling suggestions.

[0080] The embodiment of the present application can avoid repeated processing of the graph structure data by timely removing the graph structure data that has been successfully imported into the graph database from the real-time queue, thereby improving the overall data processing efficiency. When the graph structure data is not successfully imported into the graph database, it is put back into the real-time queue and a prompt message is sent to the user terminal. By sending prompt messages to the user terminal, the user can monitor the status and results of data processing in real time, discover and solve problems in a timely manner, thereby enhancing the reliability and maintainability of the system.

[0081] In some optional implementations of this embodiment, after step S205, importing the graph structure data into the graph database, the following steps may also be included:

[0082] Calculate the similarity between the graph structure data in the graph database and the existing data in the knowledge graph; based on the similarity, establish the association relationship between the graph structure data in the graph database and the existing data in the knowledge graph; based on the association relationship, integrate the entity information, attribute information and relationship information of the graph structure data in the graph database into the corresponding positions of the knowledge graph, and update the knowledge graph.

[0083] In one example, during the process of graph structure data being stored in the database, the similarity between the graph structure data and the existing data in the knowledge graph can be calculated. The cosine similarity algorithm is used to represent the attribute information of the entity as a vector, and the cosine value of the angle between the vectors is calculated. The larger the value, the higher the similarity. For example, the similarity between the newly added entity "singer A" and the existing entity "singer B" in the graph is 8, which exceeds the preset threshold of 7, and an association relationship is established between the two entities. Based on the association relationship, the attribute information of "singer A" in the target data, such as date of birth, representative works, etc., is integrated into the corresponding position of "singer B" in the knowledge graph. For the relationship information, the random walk algorithm is used, starting from the target entity, randomly selecting an outgoing edge, stopping when reaching an existing entity, and generating a relationship path. Repeating the random walk multiple times, multiple relationship paths between entities are obtained, and the relationship type on the path is extracted as the potential association relationship between the target entity and the existing entity. For example, "singer A" and "singer B" both have a "cooperation" relationship with "director A", thereby mining the relationship between "singer A" and "singer B". Finally, the fused entity, attribute and relationship information is updated to the knowledge graph to expand the knowledge graph.

[0084] The embodiments of the present application can ensure that the entity information, attribute information and relationship information in the graph structure data are accurately placed in the corresponding positions of the knowledge graph based on the similarity fusion strategy, thereby improving the consistency and accuracy of the knowledge graph. Through similarity calculation, the graph structure data may provide more detailed and comprehensive information for the entities or relationships already in the knowledge graph. Through the fusion process, this knowledge can be used to improve and optimize the existing entries in the knowledge graph. The graph structure data may contain new entities, new attributes or new relationships that are not yet covered in the knowledge graph. Through similarity calculation and fusion process, this new knowledge can be effectively integrated into the knowledge graph, thereby enhancing its richness and completeness.

[0085] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned initial data, candidate data, and graph structure data, the above-mentioned initial data, candidate data, and graph structure data can also be stored in a node of a blockchain.

[0086] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0087] The embodiments of the present application can acquire and process initial data, candidate data, and graph structure data based on the relevant algorithms of natural language processing (NLP) technology under artificial intelligence. Among them, artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0088] Natural language processing technology is a key branch in the field of artificial intelligence. It focuses on giving computers the ability to understand and process human language, so that they can parse, understand and generate natural language text like humans. By simulating some core mechanisms of human language processing, NLP technology can deeply process, analyze and interpret text data, extract key information from it, understand semantic content, and perform complex tasks or generate natural language responses based on it. Natural language processing technology can be used to parse candidate data to obtain graph structure data that will be imported into the graph database.

[0089] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0090] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0091] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0092] like Figure 3 As shown, the data processing device 400 of this embodiment includes: an acquisition module 401, a sending module 402, a processing module 403, a parsing module 404 and an importing module 405. Among them:

[0093] An acquisition module 401 is used to acquire initial data;

[0094] A sending module 402 is used to send the initial data to a pre-built real-time queue system, and the real-time queue system is used to cache and schedule the received data;

[0095] The processing module 403 is used to process the initial data in the real-time queue system in real time according to the preset priority strategy and the preset scheduling strategy to obtain candidate data;

[0096] The parsing module 404 is used to parse the candidate data using a preset graph data model to obtain graph structure data corresponding to the initial data;

[0097] The import module 405 is used to import the graph structure data into the graph database.

[0098] The real-time queue system of the embodiment of the present application can cache and schedule the received initial data in real time, and perform real-time processing according to the preset priority and scheduling strategy. This method significantly reduces the delay of data processing, making it more applicable to financial risk control application scenarios with extremely high real-time requirements. Through real-time analysis, it is possible to respond to market changes and potential risks more quickly, and improve the timeliness and accuracy of risk prevention and control. The real-time queue system can dynamically adjust resource allocation according to the current data processing load, avoiding the waste of resources or uneven load problems that may occur in micro-batch processing. This flexible resource management method enables the system to maintain efficient operation under different workloads, improving the overall efficiency of resource utilization. The data processing process in the real-time queue system is more continuous and controllable, reducing the data consistency problems caused by the discontinuity of data processing between batches. At the same time, through the preset priority and scheduling strategy, concurrent operations and data dependencies can be better managed to reduce the risk of data inconsistency or loss. This is crucial to ensure the accuracy and reliability of financial risk control decisions.

[0099] In one embodiment, the sending module 402 includes:

[0100] The format conversion submodule is used to convert the data format of the initial data into a standardized data exchange format according to a preset format conversion protocol to obtain optimized target initial data;

[0101] The queue determination submodule is used to determine the target real-time queue from the pre-built real-time queue system according to the preset routing rules;

[0102] The push submodule is used to push the target initial data to the target real-time queue.

[0103] The embodiment of the present application can convert the initial data into a standardized data exchange format through a preset format conversion protocol, which can ensure the consistency and comparability of data between different systems, platforms or applications. This standardization helps to reduce the complexity of data parsing and conversion, and improve the efficiency and accuracy of data processing. The standardized data format enhances the interoperability between systems, allowing data to flow and exchange seamlessly in different environments and systems. This helps to build a more flexible and scalable data processing architecture. Through preset routing rules, the target real-time queue can be intelligently selected for data push. This flexibility helps to cope with complex network environments and changing business needs, ensuring that data can reach the target location accurately and in a timely manner.

[0104] In one embodiment, the processing module 403 includes:

[0105] The priority determination submodule is used to determine the priority of the target initial data in the target real-time queue according to a preset priority strategy;

[0106] The first acquisition submodule is used to obtain the current data processing status of the target real-time queue;

[0107] The processing submodule is used to process the target initial data in the target real-time queue in real time based on the preset scheduling strategy, priority and data processing status to obtain the processing result;

[0108] The output submodule is used to output the processing results as candidate data and update the state of the target real-time queue in the real-time queue system.

[0109] The embodiment of the present application can automatically identify and sort the target initial data in the target real-time queue through a preset priority strategy. High-priority data can be processed first. Based on the preset scheduling strategy and data processing status, the data in the queue can be processed in real time, reducing waiting time and delay, and improving the efficiency of data processing. Through the combination of priority and scheduling strategy, computing resources can be allocated more reasonably to ensure that high-priority tasks receive sufficient resource support. After processing the data, the status of the real-time queue can be updated in time to ensure the accuracy and consistency of the data.

[0110] In one embodiment, the parsing module 404 includes:

[0111] The extraction submodule is used to extract the node information, edge information and attribute key information of the candidate data;

[0112] The reorganization submodule is used to reorganize the node information, edge information and attribute key information using a preset graph data model to generate graph structure data corresponding to the initial data.

[0113] The embodiment of the present application uses graph structure data to display complex data relationships in a graphical manner, so that the relationship between the data is clear at a glance. In graph structure data, the association relationship between data is clearly expressed, which makes relationship-based queries very efficient. Using a preset graph data model for data reorganization can ensure that data from different sources are structurally consistent. This helps integrate and share data, making data exchange between different systems easier and more efficient.

[0114] In one embodiment, the import module 405 includes:

[0115] The parsing submodule is used to parse the graph structure data to obtain the node type information and edge type information of the graph structure data;

[0116] The second acquisition submodule is used to obtain operation type information of the graph database;

[0117] A generation submodule, used to generate a first operation statement corresponding to the graph structure data based on the node type information and the operation type information, and to generate a second operation statement corresponding to the graph structure data based on the edge type information and the operation type information;

[0118] A determination submodule, used for determining the first operation statement and the second operation statement as graph database operation statements corresponding to the graph structure data;

[0119] The execution submodule is used to execute graph database operation statements through the graph database interface to import graph structure data into the graph database in real time.

[0120] The embodiment of the present application can significantly improve the efficiency of data import by directly parsing the graph structure data and generating operation statements for the graph database. During the data import process, the operation statements are directly generated according to the node and edge type information of the graph structure data, which can ensure that the data imported into the graph database is consistent with the original graph structure data.

[0121] In one embodiment, the data processing device 400 further includes:

[0122] A removal module is used to remove the graph structure data from the real-time queue system if the graph structure data is successfully imported into the graph database;

[0123] The information sending module is used to put the graph structure data back into the real-time queue system and send a prompt message to the user terminal if the graph structure data is not successfully imported into the graph database.

[0124] The embodiment of the present application can avoid repeated processing of the graph structure data that has been successfully imported into the graph database from the real-time queue in a timely manner, thereby improving the overall data processing efficiency. When the graph structure data is not successfully imported into the graph database, it is put back into the real-time queue and a prompt message is sent to the user terminal. By sending prompt messages to the user terminal, the user can monitor the status and results of data processing in real time, discover and solve problems in a timely manner, thereby enhancing the reliability and maintainability of the system.

[0125] In one embodiment, the data processing device 400 further includes:

[0126] The calculation module is used to calculate the similarity between the graph structure data in the graph database and the existing data in the knowledge graph;

[0127] Establish a module for establishing the association relationship between the graph structure data in the graph database and the existing data in the knowledge graph based on similarity;

[0128] The fusion module is used to fuse the entity information, attribute information and relationship information of the graph structure data in the graph database into the corresponding positions of the knowledge graph based on the association relationship, and update the knowledge graph.

[0129] The embodiments of the present application can ensure that the entity information, attribute information and relationship information in the graph structure data are accurately placed in the corresponding positions of the knowledge graph based on the similarity fusion strategy, thereby improving the consistency and accuracy of the knowledge graph. Through similarity calculation, the graph structure data may provide more detailed and comprehensive information for the entities or relationships already in the knowledge graph. Through the fusion process, this knowledge can be used to improve and optimize the existing entries in the knowledge graph. The graph structure data may contain new entities, new attributes or new relationships that are not yet covered in the knowledge graph. Through similarity calculation and fusion process, this new knowledge can be effectively integrated into the knowledge graph, thereby enhancing its richness and completeness.

[0130] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0131] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0132] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0133] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of data processing methods, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0134] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running a data processing method.

[0135] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0136] The real-time queue system of the embodiment of the present application can cache and schedule the received initial data in real time, and perform real-time processing according to the preset priority and scheduling strategy. This method significantly reduces the delay of data processing, making it more applicable to financial risk control application scenarios with extremely high real-time requirements. Through real-time analysis, it is possible to respond to market changes and potential risks more quickly, and improve the timeliness and accuracy of risk prevention and control. The real-time queue system can dynamically adjust resource allocation according to the current data processing load, avoiding the waste of resources or uneven load problems that may occur in micro-batch processing. This flexible resource management method enables the system to maintain efficient operation under different workloads, improving the overall efficiency of resource utilization. The data processing process in the real-time queue system is more continuous and controllable, reducing the data consistency problems caused by the discontinuity of data processing between batches. At the same time, through the preset priority and scheduling strategy, concurrent operations and data dependencies can be better managed to reduce the risk of data inconsistency or loss. This is crucial to ensure the accuracy and reliability of financial risk control decisions.

[0137] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data processing method as described above.

[0138] The real-time queue system of the embodiment of the present application can cache and schedule the received initial data in real time, and perform real-time processing according to the preset priority and scheduling strategy. This method significantly reduces the delay of data processing, making it more applicable to financial risk control application scenarios with extremely high real-time requirements. Through real-time analysis, it is possible to respond to market changes and potential risks more quickly, and improve the timeliness and accuracy of risk prevention and control. The real-time queue system can dynamically adjust resource allocation according to the current data processing load, avoiding the waste of resources or uneven load problems that may occur in micro-batch processing. This flexible resource management method enables the system to maintain efficient operation under different workloads, improving the overall efficiency of resource utilization. The data processing process in the real-time queue system is more continuous and controllable, reducing the data consistency problems caused by the discontinuity of data processing between batches. At the same time, through the preset priority and scheduling strategy, concurrent operations and data dependencies can be better managed to reduce the risk of data inconsistency or loss. This is crucial to ensure the accuracy and reliability of financial risk control decisions.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0140] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0141] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data processing method, characterized in that: The steps include: Get initial data; Sending the initial data to a pre-built real-time queue system, the real-time queue system is used to cache and schedule the received data; According to a preset priority strategy and a preset scheduling strategy, the initial data in the real-time queue system is processed in real time to obtain candidate data; Using a preset graph data model, the candidate data is parsed to obtain graph structure data corresponding to the initial data; The graph structure data is imported into a graph database.

2. The method according to claim 1, characterized in that The step of sending the initial data to a pre-built real-time queue system specifically includes: According to a preset format conversion protocol, the data format of the initial data is converted into a standardized data exchange format to obtain optimized target initial data; According to the preset routing rules, determine the target real-time queue from the pre-built real-time queue system; The target initial data is pushed to the target real-time queue.

3. The method according to claim 2, characterized in that The step of performing real-time processing on the initial data in the real-time queue system according to the preset priority strategy and the preset scheduling strategy to obtain candidate data specifically includes: Determining the priority of the target initial data in the target real-time queue according to a preset priority strategy; Obtaining the current data processing status of the target real-time queue; Based on a preset scheduling strategy, the priority and the data processing state, the target initial data in the target real-time queue is processed in real time to obtain a processing result; The processing result is output as candidate data, and the state of the target real-time queue in the real-time queue system is updated.

4. The method according to claim 1, characterized in that: The step of using a preset graph data model to parse the candidate data to obtain graph structure data corresponding to the initial data specifically includes: Extracting node information, edge information, and attribute key information of the candidate data; The node information, the edge information and the attribute key information are reorganized by using a preset graph data model to generate graph structure data corresponding to the initial data.

5. The method according to claim 1, characterized in that The step of importing the graph structure data into a graph database specifically includes: Parsing the graph structure data to obtain node type information and edge type information of the graph structure data; Get the operation type information of the graph database; Based on the node type information and the operation type information, a first operation statement corresponding to the graph structure data is generated; based on the edge type information and the operation type information, a second operation statement corresponding to the graph structure data is generated; Determine the first operation statement and the second operation statement as graph database operation statements corresponding to the graph structure data; The graph database operation statement is executed through the interface of the graph database to import the graph structure data into the graph database in real time.

6. The method according to claim 1, characterized in that After the step of importing the graph structure data into the graph database, the method further includes: If the graph structure data is successfully imported into the graph database, the graph structure data is removed from the real-time queue system; If the graph structure data is not successfully imported into the graph database, the graph structure data is put back into the real-time queue system, and a prompt message is sent to the user terminal.

7. The method according to claim 1, characterized in that After the step of importing the graph structure data into the graph database, the method further includes: Calculate the similarity between the graph structure data in the graph database and the existing data in the knowledge graph; Based on the similarity, establishing an association relationship between the graph structure data in the graph database and the existing data in the knowledge graph; Based on the association relationship, the entity information, attribute information and relationship information of the graph structure data in the graph database are integrated into the corresponding positions of the knowledge graph, and the knowledge graph is updated.

8. A data processing device, characterized in that: include: An acquisition module, used to acquire initial data; A sending module, used for sending the initial data to a pre-built real-time queue system, where the real-time queue system is used for caching and scheduling the received data; A processing module, used for processing the initial data in the real-time queue system in real time according to a preset priority strategy and a preset scheduling strategy to obtain candidate data; A parsing module, used to parse the candidate data using a preset graph data model to obtain graph structure data corresponding to the initial data; The import module is used to import the graph structure data into the graph database.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images