Data processing method, system, electronic device and storage medium

Abstract

The present disclosure provides a data processing method, system, electronic device, and storage medium, relating to the field of data processing technology. The method comprises: obtaining original demand data, clarifying the original demand data, and obtaining target demand data; in response to the target demand data being an indicator query question, performing data source exploration based on the target demand data to obtain data source exploration results; determining data screening conditions based on the data source exploration results, a code value information table, and dynamic prompt words; and generating a target structured query statement based on the data screening conditions. Application of this solution can significantly improve data query efficiency and accuracy.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, system, electronic device, and storage medium. Background Art

[0002] In the context of digital operations, indicator platforms and big data platforms, as key data asset platforms, serve as crucial bridges connecting business data and technology. Traditionally, business personnel typically query existing indicators directly through the platform's GUI (Graphical User Interface) or submit data retrieval requests to the technology department. However, as business grows, the data environment becomes increasingly complex, presenting numerous challenges: First, the large number of semantically similar indicators makes it difficult for business personnel to accurately select the required indicators, resulting in inefficient and error-prone queries; second, the data caliber and logic complexity are extremely complex, and the understanding and use of data vary across business scenarios, increasing the difficulty of data understanding and application; third, with the numerous business-related tables and the growing demand for flexible data queries, traditional query methods are unable to meet diverse data acquisition needs, hindering the timeliness and accuracy of business decisions. Summary of the Invention

[0003] The present disclosure provides a data processing method, system, electronic device and storage medium to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, the method comprising:

[0005] Obtain original demand data, clarify the original demand data, and obtain target demand data;

[0006] In response to the target demand data being an indicator query question, performing data source exploration based on the target demand data to obtain a data source exploration result;

[0007] Determine data screening conditions based on the data source exploration results, the code value information table and the dynamic prompt words;

[0008] Generate a target structured query statement based on the data screening conditions.

[0009] In one possible implementation, performing data source discovery based on the target demand data to obtain a data source discovery result includes:

[0010] Converting the target demand data to obtain a query vector;

[0011] Acquire multi-source heterogeneous data, and transform the multi-source heterogeneous data to obtain data vectors;

[0012] Performing a multi-path search on the data vector based on the query vector to obtain a multi-path search result;

[0013] fusing and reordering the multiple search results to obtain preliminary search results;

[0014] Building a FAQ retrieval module based on the preliminary retrieval results, and obtaining intermediate retrieval results through the FAQ retrieval module;

[0015] Reorder the intermediate search results to obtain data source exploration results.

[0016] In one possible implementation, determining data screening conditions based on the data source discovery results, the code value information table, and the dynamic prompt word includes:

[0017] Obtain information to support decision-making;

[0018] Based on the data source exploration results, the target demand data and the auxiliary decision information, a dynamic prompt word is constructed through a structured prompt strategy set;

[0019] The data source search result is analyzed according to the dynamic prompt word, and the data screening condition is determined by comparing it with the code value information table.

[0020] In one possible implementation, generating a target structured query statement according to the data screening condition includes:

[0021] Obtaining an initial structured query statement according to the data screening condition;

[0022] Correcting errors in the initial structured query to obtain a corrected initial structured query statement;

[0023] An optimization operation is performed on the corrected initial structured query statement to obtain a target structured query statement.

[0024] In one embodiment, the obtaining of original demand data, clarifying the original demand data, and obtaining target demand data includes:

[0025] Obtain original demand data;

[0026] The original demand data is semantically parsed, ambiguity eliminated, and key information extracted through demand clarification prompt words to obtain target demand data.

[0027] In one embodiment, the method further comprises:

[0028] Executing the target structured query statement to obtain a query result;

[0029] The query results are visually displayed.

[0030] In one embodiment, the method further comprises:

[0031] In response to the target demand data being a non-index query problem, determining whether the target demand data is a calculation problem;

[0032] If so, the query result is obtained by calculating the function;

[0033] If not, obtain the query results through knowledge question and answer search.

[0034] According to a second aspect of the present disclosure, there is provided a data processing system, the system comprising:

[0035] The demand clarification module is used to obtain original demand data, clarify the original demand data, and obtain target demand data;

[0036] A source discovery module, configured to, in response to the target demand data being an indicator query question, perform data source discovery based on the target demand data to obtain a data source discovery result;

[0037] A statement generation module is used to determine data screening conditions based on the data source exploration results, the code value information table and the dynamic prompt words;

[0038] The statement generation module is further configured to generate a target structured query statement based on the data screening condition.

[0039] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0040] at least one processor;

[0041] and a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0043] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0044] The disclosed data processing method, system, electronic device, and storage medium precisely locate target requirements through demand clarification, then collect relevant data through data source exploration, determine data screening conditions by combining a code value information table with dynamic prompts, and ultimately generate a target structured query statement. This fully automated process, from fuzzy requirements to precise query statement generation, effectively improves the accuracy, comprehensiveness, and efficiency of data processing, reduces manual effort, and enhances the system's ability to understand and respond to business needs, enabling it to better meet the demands of complex and ever-changing business scenarios.

[0045] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0047] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

[0048] Figure 1 A schematic diagram of the implementation flow of the data processing method according to an embodiment of the present disclosure is shown;

[0049] Figure 2 A schematic diagram of the implementation process of the data source detection method according to an embodiment of the present disclosure is shown;

[0050] Figure 3 Shows a schematic diagram of the implementation scenario of the embodiment of the present disclosure Figure 1 ;

[0051] Figure 4 Shows a schematic diagram of the implementation scenario of the embodiment of the present disclosure Figure 2 ;

[0052] Figure 5 A schematic diagram showing the structure of a data processing system according to an embodiment of the present disclosure is shown;

[0053] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0054] To make the purposes, features, and advantages of the present disclosure more apparent and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative work shall fall within the scope of protection of the present disclosure.

[0055] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, such as Figure 1 As shown, the method specifically includes the following steps:

[0056] Step 101: obtain original demand data, clarify the original demand data, and obtain target demand data.

[0057] Receive raw demand data from users through user interfaces such as text input boxes and voice interfaces. This raw demand data may be semantically ambiguous, incomplete, contain industry jargon, or be ambiguous. Using demand clarification prompts, multiple rounds of dialogue interaction and semantic analysis are conducted on the raw demand data to clarify ambiguous statements, supplement missing information, and resolve ambiguous content. This results in clear, complete, and compliant target demand data.

[0058] Step 102 : In response to the target demand data being an indicator query question, data source discovery is performed based on the target demand data to obtain a data source discovery result.

[0059] Among them, indicator query questions refer to query requirements centered around specific data indicators. These questions usually involve the calculation, analysis, and presentation of quantitative data, such as "What is the monthly credit loan balance of a certain branch as of a certain date?" and "Comparing the fluctuation trends of gross profit margins of different product lines." Their essence is to obtain measurable numerical results through structured data processing. When it is determined that the target demand data is an indicator query question, first, based on the target demand, data information closely related to the target demand is obtained from multiple dimensions and channels. Specifically, based on the target demand data, multiple data retrieval dimensions and channels such as data standards, screening rules, data granularity, and related data tables are clarified. On this basis, the knowledge base information can be combined to supplement and improve missing or ambiguous retrieval dimensions and channels. In addition, key information can be further confirmed by actively guiding user interaction, supplementing the precise demand details confirmed by manual verification, and ensuring the accuracy of demand understanding.

[0060] During data provenance analysis, we analyze the data structure relationships between multiple tables and construct weighting factors to assess the importance of field information retrieved from each channel. By integrating, filtering, and reordering data, we can integrate previously disorganized data into data provenance results that meet target requirements, providing reliable and effective data support for subsequent data processing.

[0061] Step 103: Determine data screening conditions based on the data source discovery results, the code value information table, and the dynamic prompt words.

[0062] The code value information table includes various code value information, such as product type numbers and structure name numbers. Dynamic prompts are dynamically generated guiding instructions based on target demand data and data sourcing results, combined with business domain knowledge. Specifically, the data sourcing results are parsed based on the dynamic prompts to identify data features that match the target demand. These features are then calibrated and converted against the standards and specifications in the code value information table. Through in-depth data analysis and rule matching, the data screening logic that meets the target demand is gradually clarified, ultimately resulting in the data screening criteria.

[0063] Step 104: Generate a target structured query statement based on the data screening condition.

[0064] Specifically, you can use a natural language-to-SQL (Structured Query Language) tool or a rules engine to generate an initial structured query statement based on data screening criteria, including field selection, filter conditions, and other elements. You can then perform syntax verification, logic optimization, and index adaptation on this initial statement. For example, this can correct spelling errors, add missing JOIN conditions, and add execution plan optimization tips. Ultimately, you'll obtain the target structured query statement with correct syntax, precise logic, and efficient execution.

[0065] The method of this embodiment precisely identifies the target requirement through demand clarification, then collects relevant data through data source exploration, combines the code value information table with dynamic prompts to determine data screening conditions, and ultimately generates the target structured query statement. This automates the entire process from fuzzy requirements to precise data query statement generation, effectively improving the accuracy, comprehensiveness, and efficiency of data processing, reducing manual participation costs, and enhancing the system's understanding and response capabilities to business needs, enabling it to better meet the needs of complex and ever-changing business scenarios.

[0066] In another possible embodiment of the present disclosure, the data source discovery based on the target demand data to obtain the data source discovery result can be specifically achieved through the following technical means: transforming the target demand data to obtain a query vector; acquiring multi-source heterogeneous data, transforming the multi-source heterogeneous data, and obtaining a data vector; performing multi-way retrieval on the data vector based on the query vector to obtain a multi-way retrieval result; fusing and reordering the multi-way retrieval results to obtain a preliminary retrieval result; constructing a common problem retrieval module based on the preliminary retrieval result, and obtaining an intermediate retrieval result through the common problem retrieval module; and reordering the intermediate retrieval results to obtain a data source discovery result.

[0067] Multi-source heterogeneous data is a collection of data from different data sources with varying structures and formats. Specifically, it can include table structure / field information, business scope descriptions, and historical archived data. Table structure / field information clarifies the data storage framework and field definitions; business scope descriptions define the calculation rules and meaning of business indicators; and historical archived data provides historical data processing experience and case studies. The target demand data and multi-source heterogeneous data are cleansed and vectorized to generate machine-readable query vectors and data vectors. Multi-way search employs multiple search strategies based on the query vector to concurrently search for relevant data within the data vector, such as keyword-matching BM25 search, semantic-based knowledge graph search, and historical case-based similarity search. The multi-way search results are then fused and re-ranked to eliminate semantic ambiguity. The fusion process integrates the results obtained from different strategies, while the re-ranking prioritizes the results based on relevance, authority, and other criteria. By building a common problem search module and reusing historical query experience, search accuracy is further improved. This module quickly matches solutions and data sources for similar historical problems, and ultimately, through further re-ranking, the data provenance is determined.

[0068] The following describes the implementation method through a specific example. Figure 2 As shown in the figure, this example involves: first, using embedding technology to process multi-source heterogeneous data and target demand data, obtaining data vectors and query vectors, which are then stored in a database. Next, BM25 and Knowledge (knowledge graph retrieval) are used to perform multi-way searches on the stored data vectors in parallel based on the query vectors, obtaining multi-way search results. Subsequently, these multi-way search results are fused and re-ranked to eliminate semantic ambiguity, resulting in preliminary search results. Finally, a FAQ Retriever (FAQ retrieval module) is constructed based on the preliminary search results to obtain intermediate search results. The constructed Reranker (reranking module) is then used to re-rank the intermediate search results. Finally, the data provenance results are obtained.

[0069] This embodiment acquires heterogeneous data from multiple sources and parses it into data vectors, comprehensively covering information of varying structures and origins. Multi-channel retrieval and parallel search, combined with various strategies, enhance the breadth and depth of retrieval. The results of these multiple retrievals are fused and re-ranked to eliminate semantic ambiguity and make the initial search results more accurate. Based on this, a frequently asked question retrieval module is constructed to reuse historical experience, further improving retrieval precision. Finally, the resulting data source exploration results, obtained through further re-ranking, are accurate, comprehensive, and efficient, better meeting the complex business needs for data exploration.

[0070] In another embodiment of the present disclosure, determining data screening conditions based on the data sourcing results, the code value information table and the dynamic prompt words includes: obtaining auxiliary decision information; constructing dynamic prompt words based on the data sourcing results, the target demand data and the auxiliary decision information through a structured prompt strategy set; parsing the data sourcing results according to the dynamic prompt words, and determining data screening conditions by comparing with the code value information table.

[0071] Decision-support information is a collection of data and knowledge that can improve understanding of needs. It includes historical query cases, relevant knowledge information, user demand questions, contextual information (including short-term and long-term memory), and proprietary knowledge bases. Historical query cases provide insights and experience from past queries; relevant knowledge information supplements professional domain knowledge to help accurately understand needs; user demand questions clarify the direction of business demands; contextual information helps understand the background of needs and the connections between them; and proprietary knowledge bases help understand industry expertise.

[0072] The structured prompt strategy set is a systematic prompt word logic covering relevance, problem, plan, result, trade-off and intention. Figure 6 It processes input information through specific logic to generate accurate and effective prompts. Specifically, using the LLM large language model, it extracts features, performs semantic analysis, and logically integrates data provenance results, target demand data, and supporting decision-making information according to the rules of a structured prompt strategy set. This allows information from different sources to complement and connect with each other, thereby constructing dynamic prompts that meet actual business needs and accurately convey data processing intent. Finally, the dynamic prompts are used to analyze the data provenance results and, combined with the code value information table, determine the data screening criteria.

[0073] The method in this embodiment leverages a structured prompt strategy set and a large language model to construct dynamic prompt words and accurately extract key information elements. This approach deeply understands industry jargon and demand context, is more flexible and accurate than traditional methods, and greatly improves the accuracy and effectiveness of determining data screening conditions.

[0074] In another embodiment of the present disclosure, generating a target structured query statement based on the data screening condition can be specifically achieved through the following steps: obtaining an initial structured query statement based on the data screening condition; performing error correction on the initial structured query to obtain a corrected initial structured query statement; and performing optimization operations on the corrected initial structured query statement to obtain a target structured query statement.

[0075] First, an initial structured query statement is constructed based on the determined data screening conditions. The system then automatically checks for fundamental issues such as execution errors and syntax errors, and repairs them, ultimately producing a revised initial structured query statement with correct syntax and execution logic.

[0076] Next, the revised initial structured query statement is optimized. Specifically, its execution plan is analyzed and adjusted to improve query efficiency. This optimization can be achieved using a strategy tree and the MCTS algorithm. Furthermore, hint words can be used to guide the large language model for further optimization. After this series of optimizations, the target structured query statement is obtained.

[0077] In one implementation, the generated target structured query statement can be logically interpreted in conjunction with industry knowledge to facilitate understanding of the statement's intent. At the same time, the data retrieval results are verified based on data types, constraints, and data consistency to ensure the accuracy and reliability of the final data obtained.

[0078] The method of this embodiment ensures the correctness and efficiency of query statements through error detection and repair and multi-dimensional optimization.

[0079] In another embodiment of the present disclosure, the obtaining of original demand data, the clarifying of the original demand data, and the obtaining of target demand data can be specifically achieved through the following steps: obtaining original demand data; performing semantic parsing, ambiguity elimination, and key information extraction on the original demand data through demand clarification prompt words to obtain target demand data.

[0080] First, by analyzing common business requirement types, collecting professional vocabulary from the business domain, and sorting out business process specifications, we pre-construct demand clarification prompts. During the processing, we first perform semantic parsing of the original demand data based on the demand clarification prompts. Using natural language processing technology, we deeply analyze the meaning of each word and sentence in the original demand data to accurately understand the underlying business logic and user intent. Next, we perform ambiguity elimination. Since the original demand data may be unclear or have multiple meanings, we conduct a detailed comparison with business knowledge bases and industry specifications to clarify the exact direction of the demand and effectively avoid misunderstandings. Finally, we perform key information extraction. From the demand data that has undergone semantic parsing and ambiguity elimination, we screen out key elements that are valuable for subsequent data processing, such as the business objects involved, time ranges, and data indicators. This key information constitutes the target demand data.

[0081] By constructing targeted demand clarification prompts, the method of this embodiment accurately analyzes the semantics of the original demand and provides a deep understanding of the business logic and user intent. By comparing with business knowledge bases and industry standards to eliminate ambiguity, the demand is accurately understood. The resulting target demand data provides clear and precise guidance for subsequent data processing, significantly improving the efficiency and accuracy of demand processing.

[0082] In one possible embodiment of the present disclosure, the method further includes: executing the target structured query statement to obtain a query result; and visually displaying the query result.

[0083] First, the target structured query statement is executed to extract the data set that meets the requirements, thereby obtaining the query results. Then, the query results are visualized, that is, the obtained data is presented in the form of intuitive and easy-to-understand charts such as bar charts, line charts, pie charts, or graphical interfaces.

[0084] In one possible embodiment of the present disclosure, the method further includes: in response to the target demand data being a non-indicator query problem, determining whether the target demand data is a calculation problem; if so, obtaining the query result through a calculation function; if not, obtaining the query result through knowledge question and answer retrieval.

[0085] Non-metric queries are queries that don't focus on specific data metric calculations, but rather on non-quantitative information such as business knowledge, concept explanations, and process descriptions. When determining whether a non-metric query is a computational problem, we first determine whether it's a computational problem. If it's a computational problem, such as one involving numerical addition, subtraction, multiplication, and division, or statistical operations, we call specific computational functions to process the relevant data and generate the query results.

[0086] If the non-metric query is determined not to be a computational problem, such as a knowledge-based question about the definition of a business concept or the steps of a business process, the knowledge question-and-answer retrieval mechanism is activated. A search and matching process is performed within a pre-built knowledge question-and-answer database, filtering out the most relevant content from a large number of knowledge items and providing this as the query result to the user. This process enables the system to obtain accurate query results for different types of non-metric queries using appropriate methods, meeting diverse user needs.

[0087] The following two specific implementation scenarios are used to illustrate the above embodiments. Scenario 1 is an indicator platform. Figure 3 , the method comprising:

[0088] The user first asks a question, which is then sent to the nl2sql-api-service backend. The system then uses the LLM's "Requirement Clarification Prompt" to understand the user's intent, obtain the target requirement data, and determine the nature of the question. If the question is deemed irrelevant, the system prompts the user to ask again. If it is determined to be a performance indicator query, the system then proceeds to the performance indicator query process.

[0089] The indicator query process begins with data provenance. Specifically, it investigates the provenance of fixed indicators, implementing this by invoking the embedding model and fixed indicator library. Specifically, the fixed indicator retrieval module in the nl2sql-datasources algorithm service is invoked through an interface. If the fixed indicator has been previously queried, the SQL generator is directly invoked to generate SQL statements. If the indicator has not been previously queried, data standard provenance, metric field provenance, and statistical granularity provenance are sequentially investigated. These are all implemented through interfaces invoking the corresponding data standard retrieval module, metric field retrieval module, and statistical granularity modules. These modules implement their functions based on the embedding model. Next, filtering conditions are generated. Specifically, the SQL prompt (filtering rule generation) in the LLM is invoked to generate dynamic prompt words, code value information tables, and data provenance results to generate data filtering conditions. The SQL generator is then invoked to generate SQL statements using the LLM's SQL post-processing and SQL-engine, and to describe the inference process. Both filtering rule generation and SQL generation are implemented by invoking filter_rules.so. Finally, the generated SQL statements invoke the indicator platform interface to retrieve data, completing data presentation.

[0090] If the question is determined to be a non-metric query, it will be further determined whether it is a calculation question. If so, the calculation function is called, and the LLM generates an answer, which is ultimately answered by Chat. If it is not a calculation question, the knowledge question search is called, and the LLM generates an answer, which is also answered by Chat.

[0091] Throughout the entire process, the nl2sql-datasources algorithm service provides crucial support. It includes functional modules such as data preprocessing and embedding vectorization, implemented through Data_all.so. The indicator platform synchronizes data through the data-sync-service, triggering data processing and reading data. The indicator platform's knowledge base stores information such as fixed indicators, multidimensional models / templates, granularity, fields, code values, and knowledge question and answer pairs, stored in file storage, MySQL, and a vector library. Users can also upload knowledge question and answer pairs to enrich the knowledge base. Furthermore, the nbcb-sql-engine indicator SQL generation service, invoked through an interface, helps generate final, usable SQL statements during the SQL generation phase.

[0092] Scenario 2 is a big data platform, refer to Figure 4 , the method comprising:

[0093] After a user asks a question, the system first enters the intent understanding phase. If the question is deemed irrelevant, the system will guide the user to ask again. If it is determined to be an indicator-based question, the subsequent process will continue.

[0094] For metrics-related questions, requirements are clarified first. LLM's requirements clarification prompt is invoked, and the requirements are clarified by combining known dimensions and the knowledge base. If the requirements are incomplete, the user is asked to provide additional information. LLM then generates a complete work order description, which the user then confirms. After confirmation, the work order enters the table discovery phase of the Embedding model, obtaining the top 12 tables. LLM then searches for key fields based on matches between table fields and keywords, and the user then selects a table.

[0095] After selecting the table, the SQL generation phase begins. If SQL generation succeeds, the SQL statement is executed, querying data from the data warehouse and presenting the data. If SQL generation fails, the process returns to the original state and reprocesses the data.

[0096] The entire process utilizes tools such as prompt generation, LLM information extraction, data retrieval, configuration information query, and SQL generation. Data retrieval uses the Retrieve interface to acquire data from industry knowledge bases, data graphs, metrics and data standards, table metadata, custom configuration data, and prompt word projects. This data is stored in various media, including vector libraries, graph databases, MySQL, and file storage. System initialization is the responsibility of data management personnel, while status memory is used to record relevant status information to facilitate the smooth progress of the entire process.

[0097] This solution uses technical means such as demand clarification, rapid data sourcing, and automatic query generation to reduce data lookup tasks from 10 minutes to just one minute, transforming traditional work models and significantly improving data efficiency. Furthermore, data standards and indicator systems are tested and refined during the data sourcing process, helping to optimize the data asset development system. Furthermore, this solution is highly scalable and can be seamlessly applied across multiple departments. By integrating with BI (Business Intelligence) tools, it helps data analysts easily generate reports and charts, lowering the barrier to data visualization and further unlocking the value of data.

[0098] According to a second aspect of the embodiment of the present disclosure, a data processing system is provided. Figure 5 As shown, the system includes:

[0099] The demand clarification module 501 is used to obtain original demand data, clarify the original demand data, and obtain target demand data;

[0100] A source discovery module 502 is configured to, in response to the target demand data being an indicator query question, perform data source discovery based on the target demand data to obtain a data source discovery result;

[0101] A statement generation module 503 is used to determine data screening conditions based on the data source exploration results, the code value information table and the dynamic prompt words;

[0102] The statement generation module 503 is further configured to generate a target structured query statement according to the data screening condition.

[0103] In one possible embodiment of the present disclosure, the source discovery module 502 is further used to transform the target demand data to obtain a query vector; obtain multi-source heterogeneous data, transform the multi-source heterogeneous data to obtain a data vector; perform multi-way retrieval on the data vector based on the query vector to obtain a multi-way retrieval result; fuse and reorder the multi-way retrieval results to obtain a preliminary retrieval result; construct a common problem retrieval module based on the preliminary retrieval result, and obtain an intermediate retrieval result through the common problem retrieval module; reorder the intermediate retrieval results to obtain a data source discovery result.

[0104] In one possible embodiment of the present disclosure, the statement generation module 503 is also used to obtain auxiliary decision information; based on the data sourcing results, the target demand data and the auxiliary decision information, dynamic prompt words are constructed through a structured prompt strategy set; the data sourcing results are parsed according to the dynamic prompt words, and the data screening conditions are determined by comparing with the code value information table.

[0105] In one possible embodiment of the present disclosure, the statement generation module 503 is further used to obtain an initial structured query statement based on the data screening condition; perform error correction on the initial structured query to obtain a corrected initial structured query statement; and perform optimization operations on the corrected initial structured query statement to obtain a target structured query statement.

[0106] In one embodiment of the present disclosure, the demand clarification module 501 is further configured to obtain original demand data; perform semantic analysis, eliminate ambiguity, and extract key information from the original demand data using demand clarification prompt words to obtain target demand data.

[0107] In one embodiment of the present disclosure, the system further includes an execution module (not shown in the figure), which is configured to execute the target structured query statement to obtain a query result; and visually display the query result.

[0108] In one possible embodiment of the present disclosure, the execution module is also used to determine whether the target demand data is a calculation problem in response to the target demand data being a non-indicator query problem; if so, obtain the query result through the calculation function; if not, obtain the query result through knowledge question and answer retrieval.

[0109] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0110] Figure 6 A schematic block diagram of an example electronic device that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0111] like Figure 6 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. Computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0112] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0113] The computing unit 801 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data processing method by any other suitable means (e.g., via firmware).

[0114] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0119] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this disclosure can be achieved, and this document is not limited here.

[0121] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0122] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain original demand data, clarify the original demand data, and obtain target demand data; In response to the target demand data being an indicator query question, performing data source exploration based on the target demand data to obtain a data source exploration result; Determine data screening conditions based on the data source exploration results, the code value information table and the dynamic prompt words; Generate a target structured query statement according to the data screening conditions; The data source discovery based on the target demand data to obtain the data source discovery result includes: transforming the target demand data to obtain a query vector; acquiring multi-source heterogeneous data and transforming the multi-source heterogeneous data to obtain a data vector; performing a multi-path search on the data vector based on the query vector to obtain a multi-path search result; fusing and reordering the multi-path search results to obtain a preliminary search result; constructing a common problem search module based on the preliminary search result, and obtaining an intermediate search result through the common problem search module; and reordering the intermediate search results to obtain the data source discovery result. The data screening conditions are determined based on the data sourcing results, the code value information table and the dynamic prompt words, including: obtaining auxiliary decision information; constructing dynamic prompt words based on the data sourcing results, the target demand data and the auxiliary decision information through a structured prompt strategy set; parsing the data sourcing results according to the dynamic prompt words, and determining the data screening conditions by comparing with the code value information table.

2. The method according to claim 1, characterized in that Generating a target structured query statement according to the data screening condition includes: Obtaining an initial structured query statement according to the data screening condition; Correcting errors in the initial structured query to obtain a corrected initial structured query statement; An optimization operation is performed on the corrected initial structured query statement to obtain a target structured query statement.

3. The method according to claim 1, characterized in that The obtaining of original demand data, clarifying the original demand data, and obtaining target demand data includes: Obtain original demand data; The original demand data is semantically parsed, ambiguity eliminated, and key information extracted through demand clarification prompt words to obtain target demand data.

4. The method according to claim 1, wherein The method further comprises: Executing the target structured query statement to obtain a query result; The query results are visually displayed.

5. The method according to claim 1, wherein The method further comprises: In response to the target demand data being a non-index query problem, determining whether the target demand data is a calculation problem; If so, the query result is obtained by calculating the function; If not, obtain the query results through knowledge question and answer search.

6. A data processing system, characterized in that: The system comprises: The demand clarification module is used to obtain original demand data, clarify the original demand data, and obtain target demand data; A source discovery module, configured to, in response to the target demand data being an indicator query question, perform data source discovery based on the target demand data to obtain a data source discovery result; A statement generation module is used to determine data screening conditions based on the data source exploration results, the code value information table and the dynamic prompt words; The statement generation module is further used to generate a target structured query statement according to the data screening condition; The source discovery module is further configured to transform the target demand data to obtain a query vector; acquire multi-source heterogeneous data and transform the multi-source heterogeneous data to obtain a data vector; perform a multi-path search on the data vector based on the query vector to obtain a multi-path search result; fuse and reorder the multi-path search results to obtain a preliminary search result; construct a common problem search module based on the preliminary search result, obtain an intermediate search result through the common problem search module; and reorder the intermediate search results to obtain a data source discovery result. The statement generation module is also used to obtain auxiliary decision information; based on the data sourcing results, the target demand data and the auxiliary decision information, a dynamic prompt word is constructed through a structured prompt strategy set; the data sourcing results are analyzed according to the dynamic prompt word, and the data screening conditions are determined by comparing with the code value information table.

7. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images