Method and device for determining data association information

Abstract

The invention discloses a method and device for determining data association information, and relates to the technical field of big data. A specific embodiment of the method comprises the steps of obtaining to-be-analyzed data from an associated information analysis request; acquiring historical log data and script task data of the to-be-analyzed data; and on the basis of the historical log data and the script task data, determining association information of the to-be-analyzed data by utilizing a preset association rule. According to the embodiment, the duration of the data association information determination process can be shortened, the data analysis efficiency is improved, and the data analysis cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method and device for determining data association information. Background Art

[0002] The existing technology mainly determines data correlation information through business research or other manual research methods. For example, in the waybill analysis scenario, different business personnel such as R&D personnel or decision makers are required to communicate a lot to determine the correlation information between waybill details and waybill plans.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0004] Manually determining data association information is time-consuming, inefficient, and requires a lot of labor costs. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and apparatus for determining data association information, which can shorten the duration of the data association information determination process, improve the efficiency of data analysis and reduce the cost of data analysis.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for determining data association information is provided, comprising:

[0007] Obtaining data to be analyzed from a related information analysis request;

[0008] Obtaining historical log data and script task data of the data to be analyzed;

[0009] Based on the historical log data and the script task data, the association information of the data to be analyzed is determined using preset association rules.

[0010] Optionally, after determining the association information of the data to be analyzed by using a preset association rule based on the historical log data and the script task data, the method further includes:

[0011] The historical log data, script task data and related information of the data to be analyzed are input into a data result model, so as to use the data result model to process the related information of the data to be analyzed; wherein the data processing includes at least one of the following: data cleaning, data conversion and data storage.

[0012] Optionally, the method further comprises:

[0013] In response to a related information query request input by a user, obtaining the to-be-queried data corresponding to the related information query request;

[0014] The data to be queried is input into the data result model, and the data result model outputs the associated information result of the data to be queried.

[0015] Optionally, in response to a related information query request input by a user, before obtaining the to-be-queried data corresponding to the related information query request, the process includes:

[0016] It is determined that the user has access rights.

[0017] Optionally, after determining the association information of the data to be associated based on the historical log data and the script task data using a preset association rule, the method further includes:

[0018] A guide interface is configured based on the association information of the data to be analyzed, and the guide interface is used to query the data association information.

[0019] Optionally, based on the historical log data and the script task data, the associated information of the data to be analyzed is determined using a preset association rule, including:

[0020] Keywords are extracted from the historical log data and script task data, and the associated information of the data to be analyzed is determined based on the matching degree of the keywords; or, the historical log data and script task data are input into a pre-trained associated information model to obtain the associated information of the data to be analyzed.

[0021] Optionally, before obtaining the historical log data of the data to be analyzed, the process includes:

[0022] Determine that the execution result of the historical log data is successful.

[0023] According to a second aspect of an embodiment of the present invention, there is provided a device for determining data association information, including:

[0024] A request module, used to obtain the data to be analyzed from the associated information analysis request;

[0025] An acquisition module, used to acquire the historical log data and script task data of the data to be analyzed;

[0026] The determination module is used to determine the association information of the data to be analyzed based on the historical log data and the script task data using a preset association rule.

[0027] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0028] one or more processors;

[0029] a memory for storing one or more programs,

[0030] When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method described in any one of the above embodiments.

[0031] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0032] An embodiment of the above invention has the following advantages or beneficial effects: by obtaining the data to be analyzed from the associated information analysis request; obtaining the historical log data and script task data of the data to be analyzed; based on the historical log data and script task data, determining the associated information of the data to be analyzed using preset association rules; this embodiment can shorten the duration of the data association information determination process, improve the efficiency of data analysis and reduce the cost of data analysis.

[0033] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0035] Figure 1 is a schematic diagram of the main process of the method for determining data association information according to an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of the main process of a method for determining data association information according to a reference embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of the main process of data association relationship analysis according to a reference embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of the main process of data association information query according to a reference embodiment of the present invention;

[0039] Figure 5 It is a schematic diagram of the main process of log and script data parsing according to a reference embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of main modules of an apparatus for determining data association information according to an embodiment of the present invention;

[0041] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0042] Figure 8It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0044] It should be noted that the acquisition, storage and application of personal information involved in the embodiments of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0045] Logistics enterprise users are paying more and more attention to information construction and information planning, and are systematically building more and more internal business links, so that traditional production materials can be recombined to maximize rational utilization and improve enterprise productivity. Digital transformation generates a large number of data report requirements. The strategic deployment of operations and management is decided through data reports, and reasonable planning is carried out through data analysis, which leads to a large number of data requirements. For example, in the waybill analysis scenario, the waybill details and waybill plans are analyzed and data is processed. The existing method is to obtain the relationship between data tables through business research or system metadata research and analysis, and then perform data processing. This solution requires a lot of communication between many departments or personnel, and the development efficiency is low. The long time to determine the table relationship leads to slow model iteration. It takes about 2 days to investigate and determine the table relationship to process the data model. Therefore, it takes a long time and inefficient to manually determine the data association information, and it requires a lot of labor costs.

[0046] In view of this, according to one aspect of an embodiment of the present invention, a method for determining data association information is provided.

[0047] Figure 1 FIG. 1 is a schematic diagram of the main process of the method for determining data association information according to an embodiment of the present invention. Figure 1 As shown, the method for determining data association information according to an embodiment of the present invention includes:

[0048] Step S101: obtaining data to be analyzed from a correlation information analysis request.

[0049] The association information analysis request refers to a request issued by a user or system to determine the association information between data, including expected output, conditions and restrictions, data range, etc. The data range is used to specify the range of the data set or data table to be analyzed, the conditions and restrictions include specific filtering conditions, data processing restrictions or support for analysis, confidence thresholds, etc. The expected output is used to indicate the desired output form, such as a list of association rules, a correlation score of data items, a chart display, etc.

[0050] The data to be analyzed refers to the data set specified in the association information analysis request, that is, the object of the actual association information analysis operation. The data to be analyzed may include various types of data, such as structured data, unstructured data, time series data, or geographic data, etc.

[0051] Specifically, first, parse the received related information analysis request, and configure the corresponding data source information according to the parsing result; then, write SQL query (Structured Query Language) or data extraction script to extract the data to be analyzed from the database or data source according to the conditions of the related information analysis request. In addition, the extracted data to be analyzed can also be pre-processed, such as data cleaning, data conversion or data filtering, etc., to improve the quality and consistency of the data.

[0052] Step S102, obtaining historical log data and script task data of the data to be analyzed.

[0053] The historical log data refers to a log file that records the operation of a system or application over a period of time. The historical log file can record various types of events and operations, as well as the associated data between operations and system events, such as user access logs, system operation logs, or database operations, etc. The user access log records various operations of the user on the system, such as login, query, update data, etc. The system operation log records the system's own operations and events, such as startup, stop, error, and warning, etc. The database operation log records database operations, such as query execution, data insertion, update, and delete operations, etc.

[0054] The script task data refers to the records of various scripts and tasks executed in the system, including the time of script execution, execution results, parameters used, related information, etc. For example, the name of the script or task, the time when the script starts and ends, the parameters used when the script is running, the dependencies between different scripts, the records of multiple executions of the same script, the association between script execution and system events, etc.

[0055] Specifically, the historical log data of the data to be analyzed can be obtained by querying the log table of the database, or by searching and analyzing the application log file, or by using the log management tool or command provided by the operating system, etc. Script task data can be obtained through the interface or API (Application Programming Interface) of the scheduling system; in addition, during the script execution process, the execution information can be recorded in the log file, and the script task data can be obtained by analyzing the log file; a custom script recording mechanism can also be configured, such as adding a log output function to the script, and obtaining the script task data by analyzing the script record, etc.

[0056] Step S103, based on the historical log data and the script task data, using preset association rules to determine association information of the data to be analyzed.

[0057] The preset association rules are rules or patterns predefined according to business needs or analysis purposes, which are used to determine the association information between the data to be analyzed. Association rules can be set based on empirical knowledge, statistical analysis, or machine learning. For example, the support and confidence thresholds of frequent item sets can be defined to discover frequent data item combinations, or the minimum support and minimum confidence of association rules can be defined to determine the association relationship between data items. After the setting is completed, analysts or systems can use the association rules to automatically discover the association information between data items.

[0058] The association information of the data to be analyzed refers to the association relationship and pattern between the data to be analyzed. The association information may include a combination of data items with high operation frequency, the association between specific operations and system events, the dependency relationship between script tasks, etc.

[0059] Specifically, the above historical log data and script task data are analyzed or matched using preset association rules to obtain analysis or matching results; then, the analysis results can be reprocessed through statistical analysis, visualization tools, etc., and finally the association information of the data to be analyzed is obtained. Data association analysis and business decisions can also be made based on the association information, such as optimizing system processes, improving user experience, and improving task execution efficiency.

[0060] The embodiment of the present invention obtains the data to be analyzed from the association information analysis request; obtains the historical log data and script task data of the data to be analyzed; and determines the association information of the data to be analyzed based on the historical log data and the script task data using preset association rules; this embodiment can shorten the duration of the data association information determination process, improve the efficiency of data analysis and reduce the cost of data analysis.

[0061] Optionally, after determining the association information of the data to be analyzed based on the historical log data and the script task data by using a preset association rule, step S103 further includes:

[0062] The historical log data, script task data and related information of the data to be analyzed are input into a data result model, so as to use the data result model to process the related information of the data to be analyzed; wherein the data processing includes at least one of the following: data cleaning, data conversion and data storage.

[0063] The data result model is a tool or system used to process and analyze data, which can be a data processing flow, a data warehouse, a data mining model, etc. Data processing includes operations such as data cleaning, data conversion, and data storage. The data cleaning includes removing duplicate data, processing missing values, and filtering abnormal data; the data conversion includes converting data into a format suitable for analysis, calculating new indicators or features, and merging related data sets; the data storage refers to storing the processed data in a target database, data warehouse, or file system.

[0064] The historical log data, script task data and related information of the data to be analyzed are input into the data result model, and the data result model processes the input data, analyzes the results after data processing, evaluates the quality and availability of the data, checks whether there are problems or errors in the processing process, and continuously improves the data processing process by monitoring data quality, feedback mechanism, model update, etc., thereby improving data processing efficiency and business value; the processed data can be used for association relationship query, business analysis development, etc.

[0065] Optionally, Figure 2 FIG. 1 is a schematic diagram of the main process of a method for determining data association information according to a reference embodiment of the present invention. Figure 2 As shown, the method also includes:

[0066] Step S201: in response to a related information query request input by a user, obtaining the to-be-queried data corresponding to the related information query request.

[0067] Step S202: input the data to be queried into the data result model, and the data result model outputs the associated information result of the data to be queried.

[0068] Specifically, a user can input an associated information query request through an interface or a system interface. The associated information query request includes information such as query data objects, query conditions, associated rules, required output results, etc. The associated information query request can involve different business scenarios and needs, such as finding data with specific association relationships, performing correlation analysis, and finding data with high correlation.

[0069] According to the user's associated information query request, the data to be queried is obtained from the database or other data sources, and the data to be queried includes historical log data, script task data or other business data; the obtained data to be queried is input into the data result model, and the data result model performs data processing, association rule mining, association analysis and other operations. The data result model can be a pre-established model, data processing flow or data analysis tool, etc.; after being processed by the data result model, the associated information results of the data to be queried are output, and the associated information results include association rules, association indicators, data relationship diagrams and other forms, and clear data display and explanation can also be performed to improve the user experience.

[0070] Optionally, step S201 includes, before obtaining the to-be-queried data corresponding to the associated information query request in response to the associated information query request input by the user: determining whether the user has access rights.

[0071] Specifically, before responding to the associated information query request input by the user, the user's identity and authority are verified and authorized so that the user can only access the data and functions for which he has the authority.

[0072] The system can authenticate users through username and password verification, single sign-on, multi-factor authentication, etc. to determine whether the user is a legitimate system user; once the user identity verification is passed, the system can authorize the user according to the user's role and permissions. Permission authorization includes data access permission authorization and function operation permission authorization, that is, determining which data the user can access and which functional operations the user can perform (such as query, modification, deletion, etc.); after permission authorization is completed, the system can implement access control mechanisms through access control lists, role permission management, policy-based access control, data masking and other technologies, so that users can only access data and functions for which they have permissions.

[0073] Before a user initiates a query request for related information, the system verifies the user's access rights, such as whether the user has the authority to query related data and whether the query conditions and rules are met; if the authority verification fails, the system will give a corresponding prompt or reject the user's request to improve data security and privacy protection. During the entire authority authorization and access control process, the system can also audit and record the user's access behavior, authority change records, etc., and can also monitor the system's access status, promptly detect abnormal access behavior and take corresponding measures to improve the security and stability of the system.

[0074] The embodiment of the present invention performs permission verification and authorization before responding to a user-related information query request, thereby improving the legitimacy and security of user access to data, effectively preventing unauthorized access and data leakage risks, and improving the security and reliability of the system.

[0075] Optionally, after determining the association information of the data to be associated based on the historical log data and the script task data using a preset association rule, step S103 further includes:

[0076] A guide interface is configured based on the association information of the data to be analyzed, and the guide interface is used to query the data association information.

[0077] Specifically, the guidance interface configured based on the association information of the data to be associated is used to display the association information of the data to be analyzed and perform query operations. The guidance interface includes elements such as data table association diagram, association rule list, query input box, etc. to intuitively display data association information and query functions.

[0078] The associated information of the data to be analyzed is displayed in the guidance interface, and the associated information includes the associated relationship, associated fields, associated rules, etc. between data tables. Specifically, it can be displayed in a graphical manner, such as a relationship diagram, a tree diagram, etc. The user can understand the association between the data tables by viewing the associated information.

[0079] The query function of data association information is implemented in the guidance interface. Users can perform query operations by entering keywords, selecting association rules, etc. The query function retrieves relevant information from the associated information based on the information or conditions entered by the user, and displays the query results on the interface. The query results include data tables, associated fields, association rules and other information that meet the query conditions. The query results can be displayed in tables, charts or other forms. Users can view the query results and export, save or further analyze them according to specific needs.

[0080] In addition, you can also add user interaction and feedback mechanisms to the guidance interface, such as error prompts, query suggestions, query history, and other functions to enhance the user experience.

[0081] The guiding interface configured based on association rules in the embodiment of the present invention is conducive to users to query and analyze data association information. The interface design can improve the user's work efficiency and data comprehension ability, and improve the query efficiency of data association information.

[0082] Optionally, step S103 determines the association information of the data to be analyzed based on the historical log data and the script task data using a preset association rule, including:

[0083] Keywords are extracted from the historical log data and script task data, and the associated information of the data to be analyzed is determined based on the matching degree of the keywords; or, the historical log data and script task data are input into a pre-trained associated information model to obtain the associated information of the data to be analyzed.

[0084] Specifically, keywords are extracted from historical log data and script task data. The keywords may be important fields, associated conditions, operation instructions, etc. in the data. Keyword extraction may be performed through text processing technology, data mining algorithms, etc., for example, based on keyword extraction algorithms, word frequency statistics, NLP (Natural Language Processing) technology, etc. After the keywords are extracted, the extracted keywords are matched or the keywords are matched with preset association rules or associated dictionaries to determine the relevance and relevance between the keywords, and the degree of matching between the keywords is calculated using text similarity algorithms, word vector models, etc.

[0085] Historical log data and script task data can also be input into a pre-trained association information model, which is a model built based on machine learning, deep learning and other technologies for automatically learning the association relationship of data. The association information model outputs the association information of the data to be analyzed.

[0086] The embodiments of the present invention can obtain the associated information of the data to be analyzed based on keyword matching or an associated information model, thereby improving the efficiency of determining the data associated information.

[0087] Optionally, before step S102 of acquiring the historical log data of the data to be analyzed, it includes: determining that the execution result of the historical log data is successful.

[0088] Specifically, when recording historical log data, an identifier can be added to the execution result of each executed task or operation, such as adding an identifier to use success / failure or other related status to indicate the execution result; the execution result of each task or operation is judged based on the execution result information recorded in the historical log data, and the successful execution result includes the completion time, execution status, related information, etc. For situations where the execution result is a failure or an exception, corresponding exception handling and analysis are performed, such as capturing and recording error logs, investigating and repairing the causes of the exception, and other operations.

[0089] Confirm that the execution result of the historical log data is successful, and manual review and confirmation can also be performed, such as manual review of the execution result, manual comparison with the actual situation, manual confirmation of the accuracy of the data, etc.; the confirmed execution result is recorded as part of the historical log data for subsequent data acquisition or analysis, etc.

[0090] Figure 3 Schematic diagram of the main process of data association relationship analysis according to a reference embodiment of the present invention; Figure 3 As shown in the figure, the SQL historical log collection and SQL script tasks are obtained, and the table association relationship is obtained by association rule analysis, which is then input into the data model processing link, and finally the data association relationship is analyzed and displayed.

[0091] Specifically, SQL history logs include SQL statements, execution time, execution results, error messages, etc. SQL history logs can be collected in real time or in batches at regular intervals. You can use log collection tools, the database's own log function, or custom scripts to collect SQL execution logs during database operation, such as database server log files, log records within the database management system, and so on.

[0092] SQL script tasks include SQL script content, execution time, execution status (success or failure), related configuration parameters, etc. SQL script tasks can be obtained through API interfaces, task logs, functions provided by the scheduling system, or custom scripts. For example, the above methods are used to collect information on all executed SQL script tasks from the scheduled task scheduling system or data warehouse system.

[0093] Use association rules to analyze the data of historical logs and script tasks, and determine the association between data tables. You can extract key fields and table names from historical logs and script tasks, for example, extract table names and field names in SELECT (filtering), JOIN (joining) and other operations; determine the association between tables according to preset association rules, such as table name matching, foreign key relationships, etc. Association rules can be explicit database constraints (such as foreign keys) or implicit relationships (such as tables that are often queried together); calculate the frequency and strength of associations between tables through statistical analysis, and determine their degree of association, for example, in how many queries are Table A and Table B used simultaneously.

[0094] The acquired association relationships are processed using data models. Data processing includes data cleaning, data conversion or data storage. Data cleaning includes clearing invalid data or repairing errors in data. Data conversion can convert raw data into a standard format, such as data normalization and dimensional model construction. Data storage can store processed data in a database or data warehouse.

[0095] After data processing, the association relationship between data tables is displayed in a visual way. Data visualization tools or customized front-end display interfaces can be used for display. The display form can be relationship diagrams, association matrices, charts, etc., and user interaction functions such as zooming in and out, clicking nodes to view detailed information, dynamic query, etc. are configured to enhance the user experience. The final effect is shown in Table 1. Through this embodiment, the association logic can be quickly obtained, and the association relationship can be obtained from the survey logic to the system analysis, so that the workload that originally required 2 or 3 days can be completed in 1 hour, and the efficiency is increased by 1600%, which greatly improves the development efficiency.

[0096]

[0097] The embodiments of the present invention can collect and analyze SQL historical logs and SQL script task data, use preset association rules to determine the association relationship between data tables, and provide users with clear association diagrams and data query tools through data model processing and visual display, thereby improving the efficiency of data management, enhancing the flexibility and ease of use of data, and improving the user experience.

[0098] Figure 4 is a schematic diagram of the main process of data association information query according to a reference embodiment of the present invention; Figure 4 As shown, in the BDP (Business Data Platform, benchmark development program) query interface, the association guidance function is enabled to query and display the association information of the data table.

[0099] The user first enters the BDP query interface. In the BDP query interface, the system first determines whether the user has enabled the association guidance function. If the association guidance is not enabled, the system directly displays the table structure, and the user can view the table structure information, and the fields can be copied; if the association guidance is enabled, the system will add an underline under the field when displaying the table structure field, indicating that these fields can be clicked and the fields cannot be copied; when the user clicks an underlined field, the system reads the association information of the table related to the field, such as reading `table_test` The information in the table includes the association relationship data between the tables obtained through the association rule analysis; before displaying the association information, the system needs to judge the table permissions to determine whether the user has the permission to access the table involved. If the user does not have the permission, the system prompts the user that "there is no permission for the associated tables" and terminates the operation. If the user has the permission, the system continues to perform subsequent operations; when the user has the permission to view the associated table, the system will jump to a new page, in which the association information related to the field selected by the user that the user has the permission to view will be displayed; the user can view the association information in the new page, and return to the previous query interface after completing the view to continue other operations or queries.

[0100] The embodiment of the present invention determines whether the user has enabled association guidance, reads and displays relevant association information, so that the user can effectively query and analyze the relationship between data tables within the scope of authority, thereby improving the efficiency of user data query and enhancing the security and standardization of data access.

[0101] Figure 5 Schematic diagram of the main process of log and script data parsing according to a reference embodiment of the present invention; Figure 5 As shown, the association information between tables is parsed and processed based on the online scripts and data query logs.

[0102] Get the online scripts, which are SQL scripts that have been deployed to the production environment. After getting the scripts, the system checks the execution status of the online scripts on the previous day (T-1). If the execution on the previous day was unsuccessful, the system chooses not to parse these scripts. If the execution on the previous day was successful, the system parses the association information contained in the successfully executed online scripts. The association information refers to the association relationship between different tables in the script, such as foreign key constraints, JOIN operations, etc.

[0103] At the same time, the system can also obtain data query logs, which are used to record various data query operations performed in the system, including the executed SQL statements, execution time, execution results, etc.; for each record in the data query log, the system checks its execution status. If the execution is unsuccessful, the system chooses not to parse the data query log. If the execution is successful, the system parses the related information in the data query log.

[0104] The system merges the successfully parsed online scripts and the associated information in the data query log. The merged associated information includes the associations between all data tables in the system. The system creates a table named `table_test`, which is used to store the parsed table association information. The associated information obtained by the merged parsing is written to the `table_test` table. The written `table_test` table includes all successfully parsed associated information for query and analysis.

[0105] The embodiment of the present invention checks whether the script and log are executed successfully, so that the system only parses reliable data sources and stores the parsed association information in the `table_test` table, thereby improving the accuracy and reliability of data association analysis.

[0106] According to a second aspect of the embodiment of the present invention, Figure 6 As shown, a device 600 for determining data association information is provided, including:

[0107] The request module 601 is used to obtain the data to be analyzed from the associated information analysis request;

[0108] An acquisition module 602 is used to acquire the historical log data and script task data of the data to be analyzed;

[0109] The determination module 603 is used to determine the association information of the data to be analyzed based on the historical log data and the script task data using a preset association rule.

[0110] Optionally, the apparatus 600 for determining data association information further includes a processing module, and the processing module is used to:

[0111] The historical log data, script task data and related information of the data to be analyzed are input into a data result model, so as to use the data result model to process the related information of the data to be analyzed; wherein the data processing includes at least one of the following: data cleaning, data conversion and data storage.

[0112] Optionally, the apparatus 600 for determining data association information further includes a query module, and the query module is used to:

[0113] In response to a related information query request input by a user, obtaining the to-be-queried data corresponding to the related information query request;

[0114] The data to be queried is input into the data result model, and the data result model outputs the associated information result of the data to be queried.

[0115] Optionally, the apparatus 600 for determining data association information further includes a permission module, and the permission module is used to:

[0116] It is determined that the user has access rights.

[0117] Optionally, the apparatus 600 for determining data association information further includes a guiding module, and the guiding module is used to:

[0118] A guide interface is configured based on the association information of the data to be analyzed, and the guide interface is used to query the data association information.

[0119] Optionally, the determining module is further used for:

[0120] Keywords are extracted from the historical log data and script task data, and the associated information of the data to be analyzed is determined based on the matching degree of the keywords; or, the historical log data and script task data are input into a pre-trained associated information model to obtain the associated information of the data to be analyzed.

[0121] Optionally, the apparatus 600 for determining data association information further includes an execution module, and the execution module is used to:

[0122] Determine that the execution result of the historical log data is successful.

[0123] It should be noted that the specific implementation content of the device for determining data association information in the present invention has been described in detail in the method for determining data association information described above, so the repeated content will not be described again here.

[0124] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0125] one or more processors;

[0126] a storage device for storing one or more programs,

[0127] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiment of the present invention.

[0128] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0129] Figure 7 An exemplary system architecture 700 is shown to which the method for determining data association information or the apparatus for determining data association information according to the embodiments of the present invention can be applied.

[0130] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0131] The user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 701, 702, 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0132] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0133] The server 705 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 701, 702, and 703. The backend management server may analyze and process the received product information query request and other data, and feed back the processing results (such as target association information - only an example) to the terminal device.

[0134] It should be noted that the method for determining data association information provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the device for determining data association information is generally disposed in the server 705 .

[0135] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0136] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0137] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can execute various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0138] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 runs a communication process via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0139] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for running the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0140] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually run basically in parallel, and they can sometimes run in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that runs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a request module, a determination module and an acquisition module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the request module may also be described as "a module for obtaining data to be analyzed from an associated information analysis request".

[0143] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining data to be analyzed from a correlation information analysis request; obtaining historical log data and script task data of the data to be analyzed; and determining the correlation information of the data to be analyzed based on the historical log data and script task data using preset correlation rules.

[0144] According to the technical solution of the embodiment of the present invention, it has the following advantages or beneficial effects: by obtaining the data to be analyzed from the associated information analysis request; obtaining the historical log data and script task data of the data to be analyzed; based on the historical log data and script task data, using preset association rules to determine the associated information of the data to be analyzed; this embodiment can shorten the duration of the data association information determination process, improve the efficiency of data analysis and reduce the cost of data analysis.

[0145] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0146] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method for determining data association information, characterized in that: include: Obtaining data to be analyzed from a related information analysis request; Obtaining historical log data and script task data of the data to be analyzed; Based on the historical log data and the script task data, the association information of the data to be analyzed is determined using preset association rules.

2. The method according to claim 1, characterized in that After determining the association information of the data to be analyzed based on the historical log data and the script task data using the preset association rules, the method further includes: The historical log data, script task data and related information of the data to be analyzed are input into a data result model, so as to use the data result model to process the related information of the data to be analyzed; wherein the data processing includes at least one of the following: data cleaning, data conversion and data storage.

3. The method according to claim 2, characterized in that The method further comprises: In response to a related information query request input by a user, obtaining the to-be-queried data corresponding to the related information query request; The data to be queried is input into the data result model, and the data result model outputs the associated information result of the data to be queried.

4. The method according to claim 3, characterized in that In response to a related information query request input by a user, before obtaining the to-be-queried data corresponding to the related information query request, the method includes: It is determined that the user has access rights.

5. The method according to claim 1, characterized in that: After determining the association information of the data to be associated based on the historical log data and the script task data using a preset association rule, the method further includes: A guide interface is configured based on the association information of the data to be analyzed, and the guide interface is used to query the data association information.

6. The method according to claim 1, characterized in that Based on the historical log data and the script task data, the associated information of the data to be analyzed is determined using preset association rules, including: Keywords are extracted from the historical log data and script task data, and the associated information of the data to be analyzed is determined based on the matching degree of the keywords; or, the historical log data and script task data are input into a pre-trained associated information model to obtain the associated information of the data to be analyzed.

7. The method according to claim 1, characterized in that Before obtaining the historical log data of the data to be analyzed, the following steps are included: Determine that the execution result of the historical log data is successful.

8. A device for determining data association information, characterized in that: include: A request module, used to obtain the data to be analyzed from the associated information analysis request; An acquisition module, used to acquire the historical log data and script task data of the data to be analyzed; The determination module is used to determine the association information of the data to be analyzed based on the historical log data and the script task data using a preset association rule.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images