Data processing method and device
By intercepting data query requests and generating blood-related data, it solves the problem that data centers find it difficult to quickly determine the data source and path after the statistical caliber changes, and achieves efficient data inspection.
Patent Information
- Application Number
- CN202510592410.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2025-08-15
AI Technical Summary
After the statistical caliber of the data center is changed, it is difficult for the data source and data request path to quickly determine the impact, which makes data inspection time-consuming and labor-intensive.
By intercepting data query requests, the request path information and data source information are extracted, the log data is generated and reported to the data warehouse, data processing is performed, and blood-related data is generated to record the association relationship between the request path and the data source.
Quickly determine the data source and data request path affected by the change in statistical caliber, reduce data inspection time, and improve data inspection efficiency.
Smart Images
Figure CN120492708A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of data processing technology, and in particular, to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] By integrating multi-dimensional business indicator data within the video platform, the data center can provide users with data support for video content analysis, video content quality assessment, and video cover title optimization.
[0003] In related technologies, data centers often struggle to manage these complex data sources, as they rely on multiple data sources to provide services. For example, if the statistical caliber of a particular dimension of data recorded in a data center changes, identifying the affected data sources requires a step-by-step, depth-first tracing of the data sources to identify all affected scenarios, which is time-consuming and laborious.
[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention
[0005] The embodiments of the present application provide a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.
[0006] One aspect of an embodiment of the present application provides a data processing method, the method comprising: Intercepting a data query request sent to a data center service and extracting request path information from the data query request; Obtaining data source information of response data returned by the data center service based on the data query request; Splicing the request path information and the data source information into log data, and reporting the log data to the data warehouse of the data center service; Data processing is performed on the multiple log data stored in the data warehouse to generate lineage data, which is used to record the data sources associated with different request paths, and to record the request paths associated with different data sources.
[0007] Optionally, intercepting the data query request sent to the data center service and extracting request path information from the data query request includes: When the data query request is an HTTP request, intercepting the data query request through an HTTP interceptor, and extracting request path information from the data query request through the HTTP interceptor; When the data query request is a GRPC request, the data query request is intercepted by the GRPC interceptor, and the request path information is extracted from the data query request by the GRPC interceptor.
[0008] Optionally, processing the plurality of log data stored in the data warehouse to generate lineage data includes: Performing data cleaning on the plurality of log data stored in the data warehouse to obtain the plurality of cleaned log data; Aggregation processing is performed on the cleaned multiple log data to obtain the blood relationship data.
[0009] Optionally, the method further includes: A bloodline data interface is established for querying the bloodline data.
[0010] Optionally, the method further includes: Get the questions to be queried; Performing semantic understanding on the query question to obtain a semantic understanding result; Performing keyword extraction on the speech understanding result to extract at least one target keyword; When the at least one target keyword includes a bloodline keyword corresponding to the bloodline data query question, the bloodline data interface and the target tool function are called based on the at least one target keyword to obtain relevant data corresponding to the at least one target keyword, the bloodline keyword is used to call the bloodline data interface, the target tool function is a tool function associated with the target keyword other than the bloodline keyword, and the target tool function is used to obtain relevant data corresponding to the target keyword other than the bloodline keyword; The data query result corresponding to the question to be queried is outputted through the large language model based on the preset prompt word project and the acquired relevant data corresponding to the at least one target keyword.
[0011] Optionally, performing semantic understanding on the query question to obtain a semantic understanding result includes: The large language model is used to perform semantic understanding on the query question to obtain a semantic understanding result.
[0012] Another aspect of an embodiment of the present application provides a data processing device, the device comprising: An interception module, configured to intercept a data query request sent to a data center service and extract request path information from the data query request; An acquisition module, configured to acquire data source information of response data returned by the data center service based on the data query request; A reporting module, configured to combine the request path information and the data source information into log data, and report the log data to the data warehouse of the data center service; A generation module is used to process the multiple log data stored in the data warehouse to generate lineage data, which is used to record the data sources associated with different request paths and the request paths associated with different data sources.
[0013] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.
[0014] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.
[0015] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.
[0016] The embodiment of the present application adopts the above technical solution, which may include the following advantages: First, by intercepting the data query request sent to the data center service, and extracting the request path information from the data query request, and obtaining the data source information of the response data returned by the data center service based on the data query request. Afterwards, by splicing the request path information and the data source information into log data, and reporting the log data to the data warehouse of the data center service, so that the multiple log data stored in the data warehouse can be processed and lineage data can be generated. The above data processing method can obtain data sources associated with different request paths, and obtain request paths associated with different data sources. In this way, when the statistical caliber of data in a certain dimension changes, the data source and data request path affected by it can be quickly determined based on the generated lineage data, thereby reducing the time required for data screening and improving data screening efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.
[0018] Figure 1 The following schematically shows an operating environment diagram of the data processing method according to the first embodiment of the present application; Figure 2 The following schematically shows a flow chart of a data processing method according to the first embodiment of the present application; Figure 3 A schematic diagram shows a detailed flow chart of the steps of processing multiple log data stored in the data warehouse to generate bloodline data; Figure 4 Schematically shows a newly added flow chart of the data processing method according to the first embodiment of the present application; Figure 5 A block diagram schematically shows a data processing device according to the second embodiment of the present application; and Figure 6 The following schematically shows a hardware architecture diagram of a computer device according to the third embodiment of the present application. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0020] It should be noted that the descriptions of "first", "second", etc. in the embodiments of the present application are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0021] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed. They are only used to facilitate the description of this application and to distinguish each step. Therefore, they cannot be understood as limitations on this application.
[0022] First, an explanation of the terms used in this application is provided: An interceptor is a mechanism used in software systems to intercept, process, or modify requests or responses. It is typically used to insert additional logic before or after an operation, such as logging, permission verification, or data conversion.
[0023] Data Collection: refers to the process of systematically collecting, recording and storing information from various sources.
[0024] Prompt Engineering: This refers to designing and optimizing input prompts to guide AI models to generate more accurate, relevant, or expected outputs.
[0025] Large Language Model (LLM): refers to a deep learning model trained using large amounts of text data that can generate natural language text or understand the meaning of language text.
[0026] AI Agent (Artificial Intelligence Agent): refers to a program or system that can simulate human thinking and behavior to automatically perform tasks and solve complex problems.
[0027] To facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the following describes the relevant technologies: Because data centers rely on a wide range of data sources to provide services, managing these complex data sources often leads to chaos. For example, if the statistical caliber of data recorded in a data center changes, users need to conduct a depth-first, step-by-step tracing of the data sources to identify the affected data sources, which is time-consuming and labor-intensive.
[0028] To this end, an embodiment of the present application provides a data processing technical solution. In this technical solution, first, by intercepting the data query request sent to the data center service, and extracting the request path information from the data query request, and obtaining the data source information of the response data returned by the data center service based on the data query request. Afterwards, by splicing the request path information and the data source information into log data, and reporting the log data to the data warehouse of the data center service, so that the multiple log data stored in the data warehouse can be processed and lineage data can be generated. Through the above data processing method, data sources associated with different request paths and request paths associated with different data sources can be obtained. In this way, when the statistical caliber of data in a certain dimension changes, the data source and data request path affected by it can be quickly determined based on the generated lineage data, thereby reducing the time required for data screening and improving data screening efficiency. See below for details.
[0029] Finally, for ease of understanding, an exemplary operating environment is provided below.
[0030] like Figure 1 As shown, the environment diagram includes a data center service 2, a network 4, and a client 6, wherein: The data center service 2 can be comprised of a single or multiple computing devices. These multiple computing devices can include virtualized computing instances. Virtualized computing instances can include virtual machines, such as emulations of computer systems, operating systems, servers, and the like. A computing device can load a virtual machine based on a virtual image and / or other data defining the specific software (e.g., operating system, specialized application, server) used for the emulation. As demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.
[0031] The data center service 2 can be configured to communicate with clients 6 and the like via a network 4. The network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 can include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, and the like, or wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.
[0032] Data center service 2 can provide storage, reading, writing, querying, deleting and other services.
[0033] Client 6 can be an electronic device running an operating system such as Windows, Android™, or iOS, such as a smartphone, tablet, laptop, virtual reality device, gaming device, set-top box, in-vehicle terminal, or smart TV. Based on these operating systems, various application programs, such as data processing programs, can be run.
[0034] The client 6 may provide / configure a user access page for manipulating the service platform 2 or uploading objects, etc.
[0035] It should be noted that the above devices are exemplary, and the number and type of devices can be adjusted in different scenarios or according to different needs.
[0036] The technical solutions of the present application are described below through a number of embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described herein.
[0037] Example 1 Figure 2 The flowchart of the data processing method according to the first embodiment of the present application is schematically shown.
[0038] like Figure 2 As shown, the data processing method may include steps S200 to S206, wherein: Step S200: intercepting a data query request sent to a data center service, and extracting request path information from the data query request.
[0039] Step S202: Acquire data source information of response data returned by the data center service based on the data query request.
[0040] Step S204: Combine the request path information and the data source information into log data, and report the log data to the data warehouse of the data center service.
[0041] Step S206 , processing the multiple log data stored in the data warehouse to generate lineage data, where the lineage data is used to record data sources associated with different request paths, and request paths associated with different data sources.
[0042] The data processing method provided in this embodiment first intercepts the data query request sent to the data center service, extracts the request path information from the data query request, and obtains the data source information of the response data returned by the data center service based on the data query request. Afterwards, the request path information and the data source information are spliced into log data, and the log data is reported to the data warehouse of the data center service, so that the multiple log data stored in the data warehouse can be processed to generate lineage data. Through the above data processing method, data sources associated with different request paths and request paths associated with different data sources can be obtained. In this way, when the statistical caliber of data in a certain dimension changes, the data source and data request path affected by it can be quickly determined based on the generated lineage data, thereby reducing the time required for data screening and improving data screening efficiency.
[0043] The following combination Figure 2 , each step in steps S200~S206 and other optional steps are described in detail.
[0044] Step S200 , intercepting the data query request sent to the data center service, and extracting the request path information from the data query request.
[0045] The data center service is a service that provides data query services to users. The data center service can provide users with data query services in multiple dimensions, such as providing users with a service to query the number of fans, the number of likes on a certain video, and the number of views of a certain video.
[0046] In actual applications, whenever a user needs to query a data center service for certain data, they can trigger a data query request to obtain the data. At this point, the data query request can be intercepted by an interceptor or proxy. After intercepting the data query request, parameters such as the user ID and request path can be extracted from the request to obtain the user ID, request path, and other parameters.
[0047] The request path is the URL path specified by the client when sending a request, which tells the server which resource or function to access. The request path information typically includes the protocol, domain name, port, and interface path.
[0048] In actual applications, request path information can be extracted in a variety of ways. An exemplary method is provided below.
[0049] In an optional embodiment, intercepting a data query request sent to a data center service and extracting request path information from the data query request includes: when the data query request is an HTTP request, intercepting the data query request through an HTTP interceptor, and extracting request path information from the data query request through the HTTP interceptor; when the data query request is a GRPC request, intercepting the data query request through a GRPC interceptor, and extracting request path information from the data query request through the GRPC interceptor.
[0050] In this embodiment, different interceptors are used to intercept data query requests based on different request types, allowing for flexible extraction of information such as the request path and user ID. Furthermore, using interceptors to intercept data query requests and extract the request path information reduces forwarding and processing at the middle layer, thereby reducing latency, compared to using a proxy approach.
[0051] In one embodiment, the data query request may be intercepted directly by creating a Proxy object through a proxy, and the request path information may be extracted therefrom.
[0052] Step S202 , obtain data source information of the response data returned by the data center service based on the data query request.
[0053] The response data is the data obtained by the data center service based on the data query request. The data source information is used to represent the area where the response data is stored. The data source information can be the name of the data table corresponding to the response data.
[0054] In actual applications, when the data center service performs a data query based on the data query request, it reads the corresponding data table to obtain response data. At this point, the data center service can return the table name of the read data table as the data source information to the client, allowing the client to obtain the data source information. In another embodiment, the data center service can also store the data source data in a designated location, and the client can subsequently obtain the data source information from the designated location.
[0055] Step S204 , splicing the request path information and the data source information into log data, and reporting the log data to the data warehouse of the data center service.
[0056] In this embodiment, by splicing the request path information and the data source information into a log data, the request path information and the data source information can be reported to the data warehouse for storage at one time.
[0057] Among them, a data warehouse is an application or system running in a data center service for data integration, storage and analysis.
[0058] Step S206 , data processing is performed on the multiple log data stored in the data warehouse to generate lineage data, and the lineage data is used to record the data sources associated with different request paths, and to record the request paths associated with different data sources.
[0059] In this embodiment, the data warehouse stores log data obtained by processing data query requests based on multiple users.
[0060] The data processing includes data cleaning, data verification, data aggregation, etc.
[0061] As an example, lineage data 1 shows that the data sources associated with request path A are data source a and data source b. Lineage data 2 shows that the data sources associated with request path B are data source a and data source c. Lineage data 3 shows that the request paths associated with data source a are request path A, request path D, and request path F. Lineage data 4 shows that the request paths associated with data source b are request path C and request path E.
[0062] In this embodiment, by processing the multiple log data stored in the data warehouse, blood relationship data that is convenient for analyzing and troubleshooting the data can be generated.
[0063] In an alternative embodiment, see Figure 3 The processing of the plurality of log data stored in the data warehouse to generate lineage data may include: Step S300 , performing data cleaning on the plurality of log data stored in the data warehouse to obtain a plurality of cleaned log data.
[0064] Step S302 , performing aggregation processing on the multiple cleaned log data to obtain the bloodline data.
[0065] In some embodiments, the purpose of data cleaning is to remove noise data, correct erroneous data, and ensure data accuracy and consistency. In particular, data that does not conform to the format or is meaningless or repeated data can be regarded as noise data.
[0066] The purpose of data aggregation is to summarize the cleaned data according to business needs and generate more meaningful lineage data.
[0067] In another embodiment, when aggregating log data, the log data may also be aggregated according to time dimension, user dimension, business dimension, etc., so as to obtain aggregated data of multiple dimensions, so that subsequent users can query the data of these dimensions.
[0068] In this embodiment, the log data is cleaned and aggregated to generate lineage data, which is convenient for subsequent query and analysis.
[0069] In an optional embodiment, the method further includes: A bloodline data interface is established for querying the bloodline data.
[0070] The bloodline data interface is used to query bloodline data.
[0071] In this embodiment, a bloodline data interface for querying the bloodline data is established, so that the bloodline data can be queried through the bloodline data interface later.
[0072] In an alternative embodiment, see Figure 4 , the method further comprises: Step S400: Obtain the question to be queried.
[0073] Step S402: performing semantic understanding on the query question to obtain a semantic understanding result; Step S404: extracting keywords from the speech understanding result to obtain at least one target keyword; Step S406, when the at least one target keyword includes a bloodline keyword corresponding to the bloodline data query question, the bloodline data interface and the target tool function are called based on the at least one target keyword to obtain relevant data corresponding to the at least one target keyword, the bloodline keyword is used to call the bloodline data interface, the target tool function is a tool function associated with the target keyword other than the bloodline keyword, and the target tool function is used to obtain relevant data corresponding to the target keyword other than the bloodline keyword.
[0074] Step S408: Outputting a data query result corresponding to the query question through a large language model based on a preset prompt word project and the acquired relevant data corresponding to the at least one target keyword.
[0075] In actual applications, when a user needs to query a question, the user can input the question to be queried through the interactive module provided by the client. After receiving the question to be queried input by the user, the interactive module will perform semantic understanding of the question to facilitate the subsequent extraction of target keywords based on the semantic understanding results.
[0076] After obtaining at least one target keyword, we can check whether these target keywords contain related keywords. If so, it indicates that the query question entered by the user involves querying related data, and the related data interface will be called. If the target keywords do not contain related keywords, it indicates that the query question entered by the user only involves other non-related data, and the target tool function will be called to obtain relevant data.
[0077] The lineage data query problem is a problem related to lineage data, such as an interface problem in the request path or a problem related to the data source.
[0078] As an example, the information input by the user can be all data sources used by interface A and the number of views of manuscript A. Then "all data sources used by interface A" is a lineage data query problem, and the number of views of manuscript A is a lineage data query problem.
[0079] The target keyword is used as a parameter for calling a lineage data interface or a target tool function. By matching the target keyword with multiple target tool functions or calling a lineage data interface, the lineage data interface or target tool function that needs to be called can be determined to obtain relevant data based on the corresponding target keyword.
[0080] It should be noted that the target tool function may include functions for calling the manuscript information query interface, user information query interface, interface description interface, data source query interface, etc.
[0081] After obtaining the relevant data, the large language model can be used to analyze and process the relevant data based on the preset prompt word project, so as to output data query results that meet the user's expectations for the user to view.
[0082] It should be noted that in order to efficiently implement the above steps S400-S408, the above steps S400-S408 can be scheduled and implemented by an artificial intelligence agent.
[0083] In this embodiment, by leveraging the powerful analysis and summarization capabilities of the large language model, combined with lineage data, manuscript data, data source data, etc., the data query results corresponding to the question to be queried can be quickly queried, so that users can accurately derive the dependency relationship between the interface and the data source based on the data query results, which is convenient for subsequent troubleshooting of business problems.
[0084] In an optional embodiment, performing semantic understanding on the query question to obtain a semantic understanding result includes: The large language model is used to perform semantic understanding on the query question to obtain a semantic understanding result.
[0085] In this embodiment, the semantic understanding of the query question is performed through the large language model to obtain a semantic understanding result, so that the powerful semantic understanding ability of the large language model can be used to realize the conversion of the query question, which is convenient for subsequent keyword extraction.
[0086] Example 2 Figure 5 The block diagram of the data processing device 500 according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 5 As shown, the apparatus 500 may include: an interception module 510, an acquisition module 520, a reporting module 530, and a generation module 540, wherein: An interception module 510 is configured to intercept a data query request sent to a data center service and extract request path information from the data query request; An acquisition module 520 is configured to acquire data source information of response data returned by the data center service based on the data query request; A reporting module 530 is configured to combine the request path information and the data source information into log data, and report the log data to the data warehouse of the data center service; The generation module 540 is used to process the multiple log data stored in the data warehouse to generate lineage data, and the lineage data is used to record the data sources associated with different request paths, and to record the request paths associated with different data sources.
[0087] As an optional embodiment, intercepting the data query request sent to the data center service and extracting the request path information from the data query request includes: When the data query request is an HTTP request, intercepting the data query request through an HTTP interceptor, and extracting request path information from the data query request through the HTTP interceptor; When the data query request is a GRPC request, the data query request is intercepted by the GRPC interceptor, and the request path information is extracted from the data query request by the GRPC interceptor.
[0088] As an optional embodiment, the processing of the plurality of log data stored in the data warehouse to generate lineage data includes: Performing data cleaning on the plurality of log data stored in the data warehouse to obtain the plurality of cleaned log data; Aggregation processing is performed on the cleaned multiple log data to obtain the blood relationship data.
[0089] As an optional embodiment, the apparatus 500 is further configured to: A bloodline data interface is established for querying the bloodline data.
[0090] As an optional embodiment, the apparatus 500 is further configured to: Get the questions to be queried; Performing semantic understanding on the query question to obtain a semantic understanding result; Performing keyword extraction on the speech understanding result to extract at least one target keyword; When the at least one target keyword includes a bloodline keyword corresponding to the bloodline data query question, the bloodline data interface and the target tool function are called based on the at least one target keyword to obtain relevant data corresponding to the at least one target keyword, the bloodline keyword is used to call the bloodline data interface, the target tool function is a tool function associated with the target keyword other than the bloodline keyword, and the target tool function is used to obtain relevant data corresponding to the target keyword other than the bloodline keyword; The data query result corresponding to the question to be queried is outputted through the large language model based on the preset prompt word project and the acquired relevant data corresponding to the at least one target keyword.
[0091] As an optional embodiment, performing semantic understanding on the query question to obtain a semantic understanding result includes: The large language model is used to perform semantic understanding on the query question to obtain a semantic understanding result.
[0092] Example 3 Figure 6 The following schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a data processing method according to the third embodiment of the present application. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers), etc. Figure 6As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other via a system bus. Memory 10010 includes at least one type of computer-readable storage medium, including flash memory, a hard disk, a multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, and the like. In some embodiments, memory 10010 may be an internal storage module of computer device 10000, such as a hard disk or memory of computer device 10000. In other embodiments, memory 10010 may also be an external storage device of computer device 10000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like equipped on computer device 10000. Of course, memory 10010 may also include both internal storage modules and external storage devices of computer device 10000. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed on the computer device 10000, such as program codes of data processing methods, etc. In addition, the memory 10010 can also be used to temporarily store various data that has been output or is to be output.
[0093] In some embodiments, processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data exchange or communication with computer device 10000. In this embodiment, processor 10020 is used to execute program code stored in memory 10010 or process data.
[0094] Network interface 10030 may include a wireless network interface or a wired network interface. Network interface 10030 is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and a communication link between computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.
[0095] It should be pointed out that Figure 6 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementing all of the shown components is not a requirement, and more or fewer components may alternatively be implemented.
[0096] In this embodiment, the data processing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.
[0097] Example 4 An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the data processing method in the embodiment when executed by a processor.
[0098] In this embodiment, computer-readable storage media include flash memory, hard disks, multimedia cards, card-type memories (e.g., SD or DX memories), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disks, optical disks, and the like. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the computer device's hard disk or memory. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like. Of course, the computer-readable storage medium may also include both the internal storage unit and external storage devices of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the data processing method described in the embodiments. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that has been output or is about to be output.
[0099] Example 5 An embodiment of the present application further provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.
[0100] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented using general-purpose computer devices. They can be concentrated on a single computer device or distributed on a network composed of multiple computer devices. Alternatively, they can be implemented using program codes executable by the computer device, so that they can be stored in a storage device and executed by the computer device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.
[0101] It should be noted that the above are only preferred embodiments of the present application and do not limit the scope of patent protection of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present application.
Claims
1. A data processing method, characterized in that: The method comprises: Intercepting a data query request sent to a data center service and extracting request path information from the data query request; Obtaining data source information of response data returned by the data center service based on the data query request; Splicing the request path information and the data source information into log data, and reporting the log data to the data warehouse of the data center service; Data processing is performed on the multiple log data stored in the data warehouse to generate lineage data, which is used to record the data sources associated with different request paths, and to record the request paths associated with different data sources.
2. The method according to claim 1, characterized in that The intercepting the data query request sent to the data center service and extracting the request path information from the data query request includes: When the data query request is an HTTP request, intercepting the data query request through an HTTP interceptor, and extracting request path information from the data query request through the HTTP interceptor; When the data query request is a GRPC request, the data query request is intercepted by the GRPC interceptor, and the request path information is extracted from the data query request by the GRPC interceptor.
3. The method according to claim 1, characterized in that The processing of the plurality of log data stored in the data warehouse to generate lineage data includes: Performing data cleaning on the plurality of log data stored in the data warehouse to obtain the plurality of cleaned log data; Aggregation processing is performed on the cleaned multiple log data to obtain the blood relationship data.
4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: A bloodline data interface is established for querying the bloodline data.
5. The method according to claim 4, characterized in that The method further comprises: Get the questions to be queried; Performing semantic understanding on the query question to obtain a semantic understanding result; Performing keyword extraction on the speech understanding result to extract at least one target keyword; When the at least one target keyword includes a bloodline keyword corresponding to the bloodline data query question, the bloodline data interface and the target tool function are called based on the at least one target keyword to obtain relevant data corresponding to the at least one target keyword, the bloodline keyword is used to call the bloodline data interface, the target tool function is a tool function associated with the target keyword other than the bloodline keyword, and the target tool function is used to obtain relevant data corresponding to the target keyword other than the bloodline keyword; The data query result corresponding to the question to be queried is outputted through the large language model based on the preset prompt word project and the acquired relevant data corresponding to the at least one target keyword.
6. The method according to claim 5, characterized in that The semantic understanding of the query question to obtain a semantic understanding result includes: The large language model is used to perform semantic understanding on the query question to obtain a semantic understanding result.
7. A data processing device, characterized in that: The device comprises: An interception module, configured to intercept a data query request sent to a data center service and extract request path information from the data query request; An acquisition module, configured to acquire data source information of response data returned by the data center service based on the data query request; A reporting module, configured to combine the request path information and the data source information into log data, and report the log data to the data warehouse of the data center service; A generation module is used to process the multiple log data stored in the data warehouse to generate lineage data, which is used to record the data sources associated with different request paths and the request paths associated with different data sources.
8. A computer device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 6 are implemented.