Data processing method and system, electronic equipment and storage medium

Abstract

The invention provides a data processing method and system, electronic equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that original demand data is obtained, demand clarification is conducted on the original demand data, and target demand data is obtained; responding to the target demand data as an index query problem, and performing data source detection based on the target demand data to obtain a data source detection result; determining a data screening condition according to the data source detection result, a code value information table and a dynamic cue word; and generating a target structured query statement according to the data screening condition. By applying the scheme, the data query efficiency and accuracy can be greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and in particular, to a data processing method, system, electronic device, and storage medium. Background Art

[0002] In the context of digital operation, the indicator platform and the big data platform, as key data asset platforms, are important bridges connecting business data and technology. In the traditional mode, business personnel usually directly query existing indicators through the GUI (Graphical User Interface) mode of the platform or submit data extraction requirements to the technology department. However, with the development of business, the data environment has become increasingly complex and faces many challenges: First, there are a large number of indicators with similar semantics, making it difficult for business personnel to accurately select the required indicators, resulting in low query efficiency and easy errors; Second, the data caliber and data logic complexity are extremely high, and there are differences in the understanding and use of data in different business scenarios, increasing the difficulty of data understanding and application; Third, there are many business-related tables, and the demand for flexible data query is increasing day by day. The traditional query method is difficult to meet the diverse data acquisition needs, restricting the timeliness and accuracy of business decisions. Summary of the Invention

[0003] The present disclosure provides a data processing method, system, electronic device, and storage medium to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, and the method includes: Obtain original demand data, clarify the demand for the original demand data, and obtain target demand data; In response to the target demand data being an indicator query problem, perform data exploration based on the target demand data to obtain a data exploration result; Determine data screening conditions according to the data exploration result, the code value information table, and the dynamic prompt words; Generate a target structured query statement according to the data screening conditions.

[0005] In an implementable manner, the performing data exploration based on the target demand data to obtain a data exploration result includes: Convert the target demand data to obtain a query vector; Obtain multi-source heterogeneous data, convert the multi-source heterogeneous data to obtain a data vector; Perform multi-way retrieval on the data vector based on the query vector to obtain a multi-way retrieval result; Fuse and reorder the multi-way retrieval results to obtain a preliminary retrieval result; Construct a frequently asked questions retrieval module based on the preliminary retrieval results, and obtain intermediate retrieval results through the frequently asked questions retrieval module; Re-rank the intermediate retrieval results to obtain data provenance results.

[0006] In an implementable manner, the determining of data screening conditions according to the data provenance results, the code value information table, and the dynamic prompt words includes: Obtain auxiliary decision-making information; Based on the data provenance results, the target demand data, and the auxiliary decision-making information, construct dynamic prompt words through a structured prompt strategy set; Parse the data provenance results according to the dynamic prompt words, and determine data screening conditions by referring to the code value information table.

[0007] In an implementable manner, the generating of a target structured query statement according to the data screening conditions includes: Obtain an initial structured query statement according to the data screening conditions; Correct errors in the initial structured query to obtain a corrected initial structured query statement; Perform an optimization operation on the corrected initial structured query statement to obtain a target structured query statement.

[0008] In an implementable manner, the obtaining of the original demand data and the clarification of the original demand data to obtain the target demand data includes: Obtain the original demand data; Perform semantic analysis, ambiguity elimination, and key information extraction on the original demand data through demand clarification prompt words to obtain the target demand data.

[0009] In an implementable manner, the method further includes: Execute the target structured query statement to obtain a query result; Visually display the query result.

[0010] In an implementable manner, the method further includes: In response to the target demand data being a non-index query problem, determine whether the target demand data is a calculation problem; If so, obtain a query result through a calculation function; If not, obtain a query result through knowledge Q&A retrieval.

[0011] According to a second aspect of the present disclosure, there is provided a data processing system, the system includes: A demand clarification module, configured to obtain the original demand data, clarify the original demand data, and obtain the target demand data; A source exploration module, configured to, in response to the target requirement data being an index query problem, perform data source exploration based on the target requirement data to obtain a data source exploration result; A statement generation module, configured to determine data filtering conditions according to the data source exploration result, a code value information table, and dynamic prompt words; The statement generation module is further configured to generate a target structured query statement according to the data filtering conditions.

[0012] According to a third aspect of the present disclosure, there is provided an electronic device, including: At least one processor; And a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the present disclosure.

[0013] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to cause a computer to execute the method described in the present disclosure.

[0014] The data processing method, system, electronic device, and storage medium of the present disclosure accurately locate the target requirements through requirement clarification, then collect relevant data through data source exploration, determine data filtering conditions in combination with the code value information table and dynamic prompt words, and finally generate a target structured query statement. It realizes the full-process automatic processing from fuzzy requirements to the generation of accurate query statements, effectively improves the accuracy, comprehensiveness, and efficiency of data processing, reduces the manual participation cost, enhances the system's understanding and response ability to business requirements, and can better meet the requirements of complex and changeable business scenarios.

[0015] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understandable through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By referring to the accompanying drawings and reading the following detailed description, the above and other purposes, features, and advantages of the exemplary embodiments of the present disclosure will become easily understandable. In the drawings, several embodiments of the present disclosure are shown in an exemplary rather than restrictive manner, wherein: In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0017] Figure 1 Shows a schematic implementation flow diagram of the data processing method according to the embodiments of the present disclosure; Figure 2 Shows the implementation process schematic diagram of the data provenance method according to an embodiment of the present disclosure; Figure 3 Shows the implementation scenario schematic of an embodiment of the present disclosure Figure 1 ; Figure 4 Shows the implementation scenario schematic of an embodiment of the present disclosure Figure 2 ; Figure 5 Shows the composition structure schematic diagram of the data processing system according to an embodiment of the present disclosure; Figure 6 Shows the composition structure schematic diagram of an electronic device according to an embodiment of the present disclosure. Detailed implementation manners

[0018] To make the objectives, features, and advantages of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present disclosure.

[0019] According to the first aspect of the embodiments of the present disclosure, a data processing method is provided. As Figure 1 shown, the method specifically includes the following steps: Step 101, obtain the original requirement data, clarify the requirements for the original requirement data, and obtain the target requirement data.

[0020] Receive the original requirement data from the user through a user interaction interface such as a text input box or a voice interaction interface. The original requirement data may have problems such as semantic ambiguity, incomplete expression, containing industry jargon, or being ambiguous. Use requirement clarification prompt words to conduct multi-round dialogue interaction and semantic analysis on the original requirement data to clarify the user's ambiguous expressions, supplement missing information, and resolve ambiguous content. Obtain clear, complete, and data processing specification-compliant target requirement data.

[0021] Step 102, in response to the target requirement data being an index query problem, perform data provenance based on the target requirement data to obtain a data provenance result.

[0022] Among them, the indicator query problem refers to the query requirements centered around specific data indicators. Such problems usually involve the calculation, analysis, and display of quantitative data. For example, "What is the balance of credit loans of a certain branch every month as of a certain day?" and "Compare the gross profit margin fluctuation trends of different product lines." etc. Its essence is to obtain measurable numerical results through structured data processing. When determining that the target demand data is an indicator query problem, first, based on the target demand, obtain data information closely related to the target demand from multiple dimensions and channels. Specifically, according to the target demand data, clarify multiple data retrieval dimensions and channels such as data standards, screening rules, data granularity, and associated data tables. On this basis, the knowledge base information can be combined to supplement and improve the missing or ambiguous retrieval dimensions and channels. In addition, by actively guiding user interaction, key information can be further confirmed, and accurate demand details confirmed manually can be supplemented to ensure the accuracy of demand understanding.

[0023] In the data source exploration process, analyze the data structure association relationships between multiple tables, and construct weight factors to evaluate the importance of the field information recalled from each channel. Through data fusion, screening, and reordering, etc., the originally disordered data information is integrated into a data source exploration result that fits the target demand, thereby providing reliable and effective data support for subsequent data processing.

[0024] Step 103, determine the data screening conditions according to the data source exploration result, the code value information table, and the dynamic prompt words.

[0025] The code value information table includes various types of code value information, such as product type numbers, structure name numbers, etc. The dynamic prompt words are guiding instruction information dynamically generated according to the target demand data and the data source exploration result, combined with business domain knowledge. Specifically, analyze the data source exploration result according to the dynamic prompt words to obtain the data characteristics that match the target demand, and at the same time, calibrate and convert the data characteristics according to the standard specifications in the code value information table. Through in-depth analysis of the data and rule matching, gradually clarify the data screening logic that meets the target demand, and finally obtain the data screening conditions.

[0026] Step 104, generate a target structured query statement according to the data screening conditions.

[0027] Specifically, a natural language to SQL (Structured Query Language) tool or a rule engine can be used to generate an initial structured query statement containing elements such as field selection and filtering conditions according to the data screening conditions. Then, syntax checking, logical optimization, and index adaptation can be performed on the initial statement, such as correcting spelling mistakes, supplementing missing JOIN conditions, adding execution plan optimization hints, etc. Finally, a target structured query statement with correct syntax, precise logic, and high execution efficiency is obtained.

[0028] The method of this embodiment accurately locates the target requirements through requirement clarification, then collects relevant data through data exploration, determines the data screening conditions by combining the code value information table and dynamic prompt words, and finally generates a target structured query statement. It realizes the full-process automated processing from fuzzy requirements to the generation of accurate data query statements, effectively improves the accuracy, comprehensiveness, and efficiency of data processing, reduces the manual participation cost, enhances the system's understanding and response ability to business requirements, and can better meet the requirements of complex and changeable business scenarios.

[0029] In another feasible embodiment of the present disclosure, the data exploration based on the target requirement data to obtain a data exploration result can be specifically realized by the following technical means: transforming the target requirement data to obtain a query vector; acquiring multi-source heterogeneous data and transforming the multi-source heterogeneous data to obtain data vectors; performing multi-way retrieval on the data vectors based on the query vector to obtain multi-way retrieval results; fusing and reordering the multi-way retrieval results to obtain a preliminary retrieval result; constructing a common problem retrieval module based on the preliminary retrieval result, and obtaining an intermediate retrieval result through the common problem retrieval module; reordering the intermediate retrieval result to obtain a data exploration result.

[0030] Multi-source heterogeneous data refers to a collection of data from different data sources with different structures and formats, which can specifically include table structure / field information, business caliber descriptions, historical archived data, etc. Among them, the table structure / field information is used to clarify the framework for data storage and field definitions, the business caliber description is used to define the calculation rules and meanings of business indicators, and the historical archived data is used to provide past data processing experiences and cases. The target demand data and multi-source heterogeneous data are respectively cleaned and vectorized to obtain query vectors and data vectors that can be recognized by machines. Multi-way retrieval is to use multiple retrieval strategies to parallelly search for relevant data in the data vectors based on the query vectors, such as BM25 retrieval based on keyword matching, knowledge graph retrieval based on semantic understanding, and similarity retrieval based on historical cases. Then, the multi-way retrieval results are fused and re-ranked to eliminate semantic ambiguity. The fusion process integrates the results obtained by different strategies, and the re-ranking adjusts the result priorities according to criteria such as relevance and authority. By constructing a frequently asked questions retrieval module and reusing historical query experiences, the retrieval accuracy is further improved. This module can quickly match the solutions and data sources of similar historical questions, and finally obtain the data source exploration results after re-ranking again.

[0031] The following uses a specific example to illustrate this embodiment, as Figure 2 shown. This example includes: First, the multi-source heterogeneous data and target demand data are processed through Embedding (vectorization) technology to obtain data vectors and query vectors, which are then stored in the database. Next, BM25 and Knowledge (knowledge graph retrieval) are used to parallelly perform multi-way retrieval on the data vectors stored in the database based on the query vectors to obtain multi-way retrieval results. Subsequently, the multi-way retrieval results are fused and re-ranked to eliminate semantic ambiguity and obtain preliminary retrieval results. After that, a FAQ Retriever (frequently asked questions retrieval module) is constructed based on the preliminary retrieval results to obtain intermediate retrieval results. Then, the constructed Reranker (re-ranking module) is used to re-rank the intermediate retrieval results. Finally, the data source exploration results are obtained.

[0032] In this embodiment, by obtaining multi-source heterogeneous data and parsing it into data vectors, it can comprehensively cover information with different structures and sources. The multi-way retrieval performs parallel search and combines multiple strategies to improve the retrieval breadth and depth. The fusion and re-ranking of the multi-way retrieval results eliminate semantic ambiguity and make the preliminary retrieval results more accurate. Based on this, a frequently asked questions retrieval module is constructed to reuse historical experiences, further improving the retrieval accuracy. Finally, the data source exploration results obtained after re-ranking again have the characteristics of accuracy, comprehensiveness, and efficiency, and can better meet the complex requirements of the business for data exploration.

[0033] In another embodiment of the present disclosure, determining the data screening conditions according to the data provenance result, the code value information table, and the dynamic prompt words includes: obtaining auxiliary decision-making information; constructing dynamic prompt words based on the data provenance result, the target demand data, and the auxiliary decision-making information through a structured prompt strategy set; parsing the data provenance result according to the dynamic prompt words, and determining the data screening conditions by referring to the code value information table.

[0034] The auxiliary decision-making information is a set of data and knowledge that can improve the understanding of requirements, including historical query cases, relevant knowledge information, user requirement questions, context information (including short-term memory and long-term memory), proprietary knowledge bases, etc. Among them, historical query cases provide past query ideas and experience for reference; relevant knowledge information supplements professional knowledge in the field to help accurately understand requirements; user requirement questions clarify the direction of business demands; context information helps understand the background and context of requirements; proprietary knowledge bases help understand professional knowledge within the industry.

[0035] The structured prompt strategy set is a systematic prompt word logic that covers six elements: relevance, problem, plan, result, trade-off, and meaning. It processes the input information through specific logic to guide the generation of accurate and effective prompt words. Specifically, with the help of the large language model (LLM), the data provenance result, the target demand data, and the auxiliary decision-making information, etc., are subjected to feature extraction, semantic analysis, and logical integration according to the rules of the structured prompt strategy set, so that information from different sources complements and correlates with each other, thereby constructing dynamic prompt words that conform to the actual business requirements and can accurately convey the intention of data processing. Finally, the data provenance result is parsed according to the dynamic prompt words, and combined with the code value information table, the data screening conditions are determined. Figure 6 The method of this embodiment constructs dynamic prompt words by leveraging the structured prompt strategy set and the large language model, and accurately extracts the key elements of information. This method can deeply understand industry jargon and requirement context, and is more flexible and accurate than traditional methods, greatly improving the accuracy and effectiveness of determining data screening conditions.

[0036] In another embodiment of the present disclosure, generating the target structured query statement according to the data screening conditions can be specifically implemented through the following steps: obtaining the initial structured query statement according to the data screening conditions; correcting the errors in the initial structured query to obtain the corrected initial structured query statement; performing an optimization operation on the corrected initial structured query statement to obtain the target structured query statement.

[0037]

[0038] ​First, based on the determined data screening conditions, construct an initial structured query statement. Subsequently, the system automatically checks for various basic problems such as execution errors and syntax errors in this statement and repairs them, thereby obtaining a corrected initial structured query statement with correct syntax and execution logic.

[0039] Next, perform an optimization operation on the corrected initial structured query statement. Specifically, analyze and adjust its execution plan to improve query efficiency. On the one hand, it can be optimized through a policy tree and the MCTS algorithm; on the other hand, it is also possible to use prompt words to guide the large language model to further optimize the statement. After this series of optimizations, the target structured query statement is obtained.

[0040] In an implementable manner, industry knowledge information can also be combined to logically interpret the generated target structured query statement for easy understanding of the statement intention. At the same time, verify the data fetching results from aspects such as data type, constraint conditions, and data consistency to ensure the accuracy and reliability of the finally obtained data.

[0041] The method of this embodiment ensures the correctness and efficiency of the query statement through error checking and repair and multi-dimensional optimization.

[0042] In another embodiment of the present disclosure, the obtaining of the original requirement data and the clarification of the original requirement data to obtain the target requirement data can be specifically implemented through the following steps: obtaining the original requirement data; performing semantic parsing, ambiguity elimination, and key information extraction on the original requirement data through requirement clarification prompt words to obtain the target requirement data.

[0043] First, pre-construct requirement clarification prompt words by analyzing common business requirement types, collecting professional vocabulary in the business field, and sorting out business process specifications. During the processing, first perform semantic parsing on the original requirement data based on the requirement clarification prompt words, using natural language processing technology to deeply analyze the meaning of each word and sentence in the original requirement data, and accurately understand the business logic and user intention behind it. Then perform ambiguity elimination. Since the original requirement data may have unclear expressions or multiple meanings of a word, through careful comparison with the business knowledge base, industry specifications, etc., clarify the accurate direction of the requirement and effectively avoid misunderstandings. Finally, perform key information extraction. From the requirement data after semantic parsing and ambiguity elimination, screen out key elements valuable for subsequent data processing, such as business objects involved, time ranges, data indicators, etc. The target requirement data is composed of these key information.

[0044] The method of this embodiment can accurately analyze the semantics of the original requirements, deeply understand the business logic and user intentions by constructing targeted requirement clarification prompts. By comparing with the business knowledge base and industry specifications to eliminate ambiguities, it ensures the accurate understanding of the requirements. The obtained target requirement data provides clear and accurate guidance for subsequent data processing, greatly improving the efficiency and accuracy of requirement processing.

[0045] In an implementable embodiment of the present disclosure, the method further includes: executing the target structured query statement to obtain a query result; and visually displaying the query result.

[0046] First, execute the target structured query statement to extract a data set that meets the requirements, thereby obtaining a query result. Then, visually display the query result, that is, present the obtained data in the form of intuitive and easy-to-understand charts such as bar charts, line charts, pie charts, or in the form of a graphical interface.

[0047] In an implementable embodiment of the present disclosure, the method further includes: in response to the target requirement data being a non-metric query problem, determining whether the target requirement data is a calculation problem; if so, obtaining a query result through a calculation function; if not, obtaining a query result through knowledge Q&A retrieval.

[0048] A non-metric query problem is a query requirement that does not focus on specific data metric calculations, but rather on non-quantitative information such as business knowledge, concept explanations, and process descriptions. When determining whether it is a non-metric problem, first determine whether the non-metric query problem belongs to a calculation problem. If it is a calculation problem, such as involving addition, subtraction, multiplication, division, statistical operations, etc. of numerical values, call a specific calculation function to perform calculation processing on the relevant data to obtain a query result.

[0049] If it is determined that the non-metric query problem does not belong to a calculation problem, such as asking knowledge-based questions such as the definition of a certain business concept or the steps of a business process, a knowledge Q&A retrieval mechanism will be activated. Search and match in a pre-constructed knowledge Q&A library, and screen out the content with the highest relevance to the question from a large number of knowledge entries, and use this as the query result to feedback to the user. Through such a process, the system can adopt appropriate methods to obtain accurate query results for different types of non-metric query problems, meeting diverse user needs.

[0050] The following uses two specific implementation scenarios to illustrate the above embodiments. Scenario 1 is an indicator platform, referring to Figure 3 , the method includes: The user first raises a question, which enters the nl2sql-api-service backend service. Intent understanding is performed by calling the "Requirement Clarification Prompt" of the LLM to obtain the target requirement data and determine the nature of the question. If it is determined to be an irrelevant question, the system guides the user to ask a new question; if it is determined to be an indicator query question, it enters the indicator query processing flow.

[0051] In the indicator query processing flow, data source exploration is first carried out. Specifically, fixed indicator source exploration is carried out first, which is implemented by calling the embedding model and the fixed indicator library. Specifically, the fixed indicator retrieval module in the nl2sql-datasources algorithm service is called through an interface. If the fixed indicator is an indicator that has been queried historically, the SQL generator is directly called to generate an SQL statement. If it is not an indicator that has been queried historically, data standard source exploration, measurement field source exploration, and statistical granularity source exploration are carried out in sequence, all of which are implemented by calling the corresponding data standard retrieval module, measurement field retrieval module, and statistical granularity module through an interface. These modules implement functions based on the Embedding model. Then, screening condition generation is carried out. Specifically, the SQL generation Prompt (screening rule generation) of the LLM is called, that is, dynamic prompt words, code value information tables, and data source exploration results are used to generate data screening conditions. Subsequently, the SQL generator is called, and with the help of the LLM's SQL post-processing and sql-engine (sql engine), an SQL statement is generated, and the reasoning process description is implemented. Among them, both screening rule generation and SQL generation are implemented by calling filter_rules.so. Finally, the generated SQL statement calls the indicator platform interface to fetch data to complete data presentation.

[0052] If it is determined to be a non-indicator query question, it will be further determined whether it is a calculation question. If it is a calculation question, the calculation Function is called, and then the answer is generated by the LLM, and finally the user is answered by Chat; if it is not a calculation question, knowledge question and answer retrieval is called, and then the answer is generated by the LLM, and the user is also answered by Chat.

[0053] In the whole process, the nl2sql-datasources algorithm service provides important support, which includes functional modules such as data preprocessing and Embedding vectorization, and related functions are implemented through Data_all.so. The indicator platform performs data synchronization through data-sync-service, triggers data processing, and reads data. The indicator platform knowledge base stores information such as fixed indicators, multi-dimensional models / templates, granularity, fields, code values, knowledge question and answer pairs, etc., which are stored in file storage, MySQL, and vector libraries respectively. Users can also upload knowledge question and answer pairs to enrich the knowledge base. In addition, the nbcb-sql-engine indicator SQL generation service helps generate the final available SQL statement in the SQL generation link through interface calls.

[0054] Scenario 2 is a big data platform. Refer to Figure 4 , the method includes: After the user asks a question, it first enters the intention understanding stage. If it is judged to be an irrelevant question, the system will guide the user to ask again; if it is judged to be an indicator type question, the subsequent process will continue.

[0055] For indicator type questions, first clarify the requirements, call the requirement clarification prompt of the LLM, and combine the known and clear dimensions and knowledge base supplementation to make the requirement dimensions more clear. If the requirements are incomplete, the user needs to supplement the question, and then the LLM generates a complete work order description, and then the user confirms the work order. The confirmed work order enters the Embedding model table exploration link to obtain the top 12 tables. Then the LLM explores the key fields according to the matching situation between the table fields and keywords, and then the user performs the table selection operation.

[0056] After the table selection is completed, it enters the SQL generation stage. If the SQL generation is successful, the SQL statement will be executed to query data from the data warehouse and present the data. If the SQL generation fails, it needs to return for reprocessing.

[0057] In the whole process, tool sets such as Prompt generation, LLM information extraction, data retrieval, configuration information query, and SQL generation are used. In the data retrieval link, data is obtained through the Retrieve interface from sources such as industry knowledge bases, data graphs, indicators and data standards, table metadata, custom configuration data information, and prompt engineering. These data are stored in different media such as vector libraries, graph databases, MySQL, and file storage. The system initialization is responsible for by data managers, and the status Memory is used to record relevant status information to assist the smooth progress of the whole process.

[0058] This solution shortens the data query work that originally took 10 minutes to within 1 minute through technical means such as requirement clarification, rapid data source exploration, and automatic generation of query statements, changes the traditional working mode, and greatly improves the data usage efficiency. At the same time, during the data source exploration process, the data standards and indicator systems are inspected and improved to help optimize the data asset construction system. At the same time, this solution has extremely strong scalability and can be seamlessly applied to multiple departments. By combining and docking with BI (Business Intelligence) tools and other methods, it helps data analysts conveniently generate reports and charts, lowers the threshold of data visualization, and further releases the data value.

[0059] According to the second aspect of the embodiments of the present disclosure, a data processing system is provided, as Figure 5 shown, the system includes: A requirements clarification module 501, configured to obtain original requirement data, clarify the original requirement data to obtain target requirement data; A data source exploration module 502, configured to, in response to the target requirement data being an index query problem, perform data source exploration based on the target requirement data to obtain a data source exploration result; A statement generation module 503, configured to determine data filtering conditions according to the data source exploration result, a code value information table, and dynamic prompt words; The statement generation module 503 is further configured to generate a target structured query statement according to the data filtering conditions.

[0060] In an implementable embodiment of the present disclosure, the data source exploration module 502 is further configured to transform the target requirement data to obtain a query vector; obtain multi-source heterogeneous data, transform the multi-source heterogeneous data to obtain a data vector; perform multi-way retrieval on the data vector based on the query vector to obtain a multi-way retrieval result; fuse and reorder the multi-way retrieval result to obtain a preliminary retrieval result; construct a frequently asked questions retrieval module based on the preliminary retrieval result, and obtain an intermediate retrieval result through the frequently asked questions retrieval module; reorder the intermediate retrieval result to obtain a data source exploration result.

[0061] In an implementable embodiment of the present disclosure, the statement generation module 503 is further configured to obtain auxiliary decision-making information; construct dynamic prompt words through a structured prompt strategy set based on the data source exploration result, the target requirement data, and the auxiliary decision-making information; parse the data source exploration result according to the dynamic prompt words, and determine data filtering conditions by referring to the code value information table.

[0062] In an implementable embodiment of the present disclosure, the statement generation module 503 is further configured to obtain an initial structured query statement according to the data filtering conditions; correct errors in the initial structured query to obtain a corrected initial structured query statement; perform an optimization operation on the corrected initial structured query statement to obtain a target structured query statement.

[0063] In an implementable embodiment of the present disclosure, the requirements clarification module 501 is further configured to obtain original requirement data; perform semantic parsing, ambiguity elimination, and key information extraction on the original requirement data through requirements clarification prompt words to obtain target requirement data.

[0064] In an implementable embodiment of the present disclosure, the system further includes an execution module (not shown in the figure), and the execution module is configured to execute the target structured query statement to obtain a query result; perform visual display on the query result.

[0065] In an implementable embodiment of the present disclosure, the execution module is further configured to, in response to the target requirement data being a non-index query problem, determine whether the target requirement data is a calculation problem; if so, obtain a query result through a calculation function; if not, obtain a query result through knowledge question and answer retrieval.

[0066] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device and a readable storage medium.

[0067] Figure 6 A schematic block diagram of an example electronic device that can be used to implement the embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0068] As Figure 6 shown, the device 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0069] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disc, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0070] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 executes the various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above can be executed. Alternatively, in other embodiments, the computing unit 801 can be configured to execute the data processing method by any other suitable means (e.g., by means of firmware).

[0071] Various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0072] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, executed partially on the machine as an independent software package and partially on a remote machine, or executed entirely on a remote machine or server.

[0073] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0074] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0075] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0076] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, can also be a server of a distributed system, or a server incorporating a blockchain.

[0077] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0078] In addition, the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this disclosure, "a plurality of" means two or more, unless otherwise specifically defined.

[0079] As described above, the above are only specific embodiments of this disclosure, but the protection scope of this disclosure is not limited thereto. Any person skilled in the art within the technical scope disclosed by this disclosure can easily think of changes or substitutions, which should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the original requirement data, clarify the requirements of the original requirement data, and obtain the target requirement data; In response to the target requirement data being an index query problem, perform data source exploration based on the target requirement data to obtain a data source exploration result; Determine data screening conditions according to the data source exploration result, the code value information table, and the dynamic prompt words; Generate a target structured query statement according to the data screening conditions.

2. The method according to claim 1, wherein The performing data source exploration based on the target requirement data to obtain a data source exploration result includes: Transform the target requirement data to obtain a query vector; Obtain multi-source heterogeneous data, transform the multi-source heterogeneous data to obtain a data vector; Perform multi-way retrieval on the data vector based on the query vector to obtain a multi-way retrieval result; Fuse and re-rank the multi-way retrieval results to obtain a preliminary retrieval result; Construct a common problem retrieval module based on the preliminary retrieval result, and obtain an intermediate retrieval result through the common problem retrieval module; Re-rank the intermediate retrieval result to obtain a data source exploration result.

3. The method according to claim 1, wherein The determining data screening conditions according to the data source exploration result, the code value information table, and the dynamic prompt words includes: Obtain auxiliary decision-making information; Based on the data source exploration result, the target requirement data, and the auxiliary decision-making information, construct dynamic prompt words through a structured prompt strategy set; Parse the data source exploration result according to the dynamic prompt words, and determine data screening conditions by referring to the code value information table.

4. The method according to claim 1, characterized in that, The generating a target structured query statement according to the data screening conditions includes: Obtain an initial structured query statement according to the data screening conditions; Correct the errors of the initial structured query to obtain a corrected initial structured query statement; Perform an optimization operation on the corrected initial structured query statement to obtain a target structured query statement.

5. The method according to claim 1, wherein The obtaining the original requirement data, clarifying the requirements of the original requirement data, and obtaining the target requirement data includes: Obtain the original requirement data; Perform semantic analysis, ambiguity elimination, and key information extraction on the original requirement data through requirement clarification prompt words to obtain the target requirement data.

6. The method according to claim 1, wherein The method further includes: Execute the target structured query statement to obtain a query result; Visually display the query result.

7. The method according to claim 1, wherein The method further includes: In response to the target requirement data being a non-index query problem, determine whether the target requirement data is a calculation problem; If so, obtain a query result through a calculation function; If not, obtain a query result through knowledge question and answer retrieval.

8. A data processing system, characterized in that, The system includes: A requirement clarification module for obtaining the original requirement data, clarifying the requirements of the original requirement data, and obtaining the target requirement data; A source exploration module for performing data source exploration based on the target requirement data in response to the target requirement data being an index query problem to obtain a data source exploration result; A statement generation module for determining data screening conditions according to the data source exploration result, the code value information table, and the dynamic prompt words; The statement generation module is further configured to generate a target structured query statement according to the data screening conditions.

9. An electronic device, characterized in that, including: At least one processor; and a memory communicatively coupled to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when executed by the at least one processor, cause the at least one processor to be capable of performing the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are for causing a computer to perform the method according to any one of claims 1-7.

Images