Multi-mode heterogeneous data query method and device

Abstract

The invention relates to the technical field of data analysis, in particular to a multi-mode heterogeneous data query method which comprises the following steps: receiving a query statement sent by a user side, performing lexical analysis on the query statement through an analyzer, and performing semantic check on the analyzed query statement to obtain an initial query statement; sending the initial query statement to a query optimizer to generate a corresponding query syntax tree; after the query syntax tree is sent to a Data-speed engine for prompt word engineering processing, indexing is carried out in combination with an index vector library to obtain an index result, and an execution plan is intelligently predicted based on the index result; inputting the intelligent prediction execution plan into a large language model system, outputting an execution code corresponding to a current execution task and operators of an available instruction set corresponding to different execution tasks in real time, and introducing an entity identifier into an attention mechanism of the large language model system for supervision and fine adjustment; and calling an operator corresponding to the execution code through a Data-speed JIT generator, and generating an execution result based on the operator.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to a multi-mode heterogeneous data query method and device. Background Art

[0002] With the rapid development of information technology, the amount of global data has shown an explosive growth trend, and the data types are becoming increasingly diverse, covering structured data (such as tables in relational databases), semi-structured data (such as JSON, XML, etc.) and unstructured data (such as text, images, audio, etc.). This change poses a severe challenge to traditional single-model databases, because they often find it difficult to effectively process and analyze such complex and diverse data sets. Therefore, multi-mode heterogeneous data SQL query analysis technology has emerged, aiming to provide a comprehensive solution to meet the extensive needs of modern applications for data processing and analysis.

[0003] Although there are some multi-mode heterogeneous data query and analysis platforms on the market that can improve data processing and analysis efficiency to a certain extent, they still face many challenges. Specifically, different computing tasks on multi-mode data need to be performed by operators that best suit their data characteristics. However, existing technologies often find it difficult to achieve efficient switching and collaborative work between multiple operators, resulting in performance bottlenecks.

[0004] In addition, although GPU operators excel in accelerating large-scale parallel computing tasks (such as graphics processing and scientific computing), CPU operators are suitable for more general computing scenarios, and SIMD (single instruction multiple data) operators can achieve parallel data processing on modern processors, the existing technology still has obvious deficiencies in fusing these operators to jointly optimize query analysis performance. This deficiency directly limits the overall efficiency of the query analysis system, especially when processing large-scale data sets, where performance bottlenecks are particularly prominent.

[0005] In view of this, how to provide a multi-mode heterogeneous data query and analysis method that essentially integrates heterogeneous computing, forms real-time specific processing of heterogeneous data, and significantly improves the processing performance of multi-mode heterogeneous data has become a technical problem that urgently needs to be solved. Summary of the invention

[0006] The embodiments of the present application provide a multi-mode heterogeneous data query and analysis method, a multi-mode heterogeneous data query and analysis device, a computing device and a computer storage medium, which are used to solve the problems of performance bottlenecks and functional deficiencies in the prior art when processing multi-mode heterogeneous data.

[0007] In a first aspect of an embodiment of the present application, a multi-mode heterogeneous data query method is provided, comprising:

[0008] Receiving a query statement sent by a user terminal, performing lexical analysis and grammatical analysis on the query statement through a parser, and performing semantic checking on the parsed query statement to obtain an initial query statement;

[0009] Sending the initial query statement to a query optimizer to generate a corresponding query syntax tree;

[0010] After sending the query syntax tree to the Data-speed engine for prompt word engineering processing, it is indexed in combination with an index vector library carrying multi-mode heterogeneous data to obtain an index result, and based on the index result, an execution plan is intelligently predicted;

[0011] The intelligent prediction execution plan is input into the large language model system, and the execution code corresponding to the current execution task and the operators of the available instruction sets corresponding to different execution tasks are output in real time, wherein entity identification is introduced into the attention mechanism of the large language model system for supervised fine-tuning;

[0012] The operator corresponding to the execution code is called through the Data-speed JIT generator, and an execution result is generated based on the operator.

[0013] In a second aspect of an embodiment of the present application, a multi-mode heterogeneous data query device is provided, including:

[0014] The parsing module is configured to receive a query statement sent by a user terminal, perform lexical parsing and grammatical parsing on the query statement through a parser, and perform semantic checking on the parsed query statement to obtain an initial query statement;

[0015] A generation module is configured to send the initial query statement to a query optimizer to generate a corresponding query syntax tree;

[0016] An indexing module is configured to send the query syntax tree to the Data-speed engine for prompt word engineering processing, and then perform indexing in combination with an index vector library carrying multi-mode heterogeneous data to obtain an index result, and intelligently predict an execution plan based on the index result;

[0017] An output module is configured to input the intelligent prediction execution plan into a large language model system, and output in real time the execution code corresponding to the current execution task and operators of available instruction sets corresponding to different execution tasks, wherein entity identification is introduced into the attention mechanism of the large language model system for supervised fine-tuning;

[0018] The calling module is configured to call the operator corresponding to the execution code through the Data-speed JIT generator, and generate an execution result based on the operator.

[0019] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0020] Memory and processor;

[0021] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned multi-mode heterogeneous data query method are implemented.

[0022] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned multi-mode heterogeneous data query method are implemented.

[0023] The present application provides a multi-mode heterogeneous data query method, comprising: first, receiving a query statement sent by a user end, performing lexical parsing and grammatical parsing on the query statement through a parser, and performing semantic checking on the parsed query statement to obtain an initial query statement; then, sending the initial query statement to a query optimizer to generate a corresponding query syntax tree; second, sending the query syntax tree to a Data-speed engine for prompt word engineering processing, and then indexing it in combination with an index vector library carrying multi-mode heterogeneous data to obtain an index result, and based on the index result, intelligently predicting an execution plan; third, inputting the intelligently predicted execution plan into a large language model system, and outputting in real time the execution code corresponding to the current execution task and the operators of the available instruction sets corresponding to different execution tasks, wherein entity identification is introduced into the attention mechanism of the large language model system for supervised fine-tuning; finally, calling the operator corresponding to the execution code through a Data-speed JIT generator, and generating an execution result based on the operator.

[0024] The multi-mode heterogeneous data query method provided in the embodiment of the present application is applied to achieve specific processing of heterogeneous data by integrating heterogeneous computing, rather than general processing of diverse data, which can effectively cope with the high-performance computing requirements of non-independent and identically distributed data. On the one hand, it solves the accuracy and efficiency bottlenecks of the traditional Attention mechanism: by introducing entity identification and generating efficient adaptive operators, the present invention significantly improves the accuracy and efficiency of multi-mode heterogeneous data processing; on the other hand, it generates a large language model that adapts to the multi-mode heterogeneous data processing engine: the model can optimize the characteristics of multi-mode heterogeneous data and generate efficient query plans and execution codes; on the other hand, it realizes an intelligent adaptive execution engine: by real-time generation and compilation of LLVM JIT instructions, the present invention realizes intelligent adaptive processing of multi-mode heterogeneous data, greatly improving computing performance; on the other hand, it provides comprehensive intelligent operation and maintenance management: through real-time analysis and intelligent management of cluster status, the present invention ensures the stability and efficiency of the system.

[0025] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, which can be implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0027] Figure 1 A flowchart of a multi-mode heterogeneous data query method provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of an enhanced distributed multi-mode heterogeneous data SQL query analysis engine based on a Data-speed large language model provided in an embodiment of the present application;

[0029] Figure 3 It is a structural schematic diagram of a multi-mode heterogeneous data query device provided by an embodiment of this specification;

[0030] Figure 4 A structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] In order to efficiently integrate diverse operators into a multi-mode heterogeneous data query and analysis system, a series of complex technical difficulties need to be overcome. These difficulties mainly stem from the following technical limitations:

[0033] Deepening challenges of hardware heterogeneity: The diversity of hardware platforms, especially the significant differences between CPU and GPU in computing architecture, memory model and instruction set, constitute the primary obstacle to operator fusion. Developers need to have a deep understanding of the underlying details of each hardware in order to design operators that can fully utilize the hardware potential while ensuring cross-platform compatibility. However, this in-depth understanding is often accompanied by high learning costs and development time, and with the rapid iteration of hardware technology, the original optimization strategies may quickly become outdated.

[0034] Data transmission bottleneck is aggravated: In multi-mode heterogeneous data processing, the frequent migration of data between CPU and GPU becomes a bottleneck for performance improvement. This is not only due to the time consumption of data transmission itself, but also due to the additional overhead caused by data format conversion and synchronization operations. Especially when processing large-scale data sets, these overheads may occupy a large part of the overall computing time, thus severely limiting the overall efficiency of the system.

[0035] Increased programming complexity: In order to utilize the powerful computing power of GPUs, developers need to master specific parallel programming models (such as CUDA and OpenCL), which are often very different from traditional CPU programming paradigms, increasing the difficulty of development and maintenance. In addition, the incompatibility of programming interfaces between different hardware platforms further exacerbates the complexity of cross-platform development and deployment.

[0036] Algorithm adaptation problem: The diversity and complexity of multimodal data require algorithms to be highly flexible and adaptable. However, existing operator designs are often optimized for specific data types or modalities and are difficult to directly apply to other types of data. This results in the need to frequently switch algorithms or operators when processing mixed data modes, increasing the complexity and uncertainty of the processing flow.

[0037] Resource management complexity: In a multi-mode heterogeneous environment, the reasonable allocation and dynamic adjustment of computing resources become the key to ensure the efficient operation of the system. However, due to hardware heterogeneity and task diversity, it becomes extremely difficult to accurately predict and allocate resources. In addition, resource competition and conflicts may also lead to performance degradation, especially in high-load or real-time processing scenarios.

[0038] In summary, the existing technologies have significant defects in terms of operator fusion, data transmission optimization, programming complexity simplification, algorithm adaptability improvement and effective resource management, which seriously restrict the performance and scalability of multi-mode heterogeneous data query and analysis systems. This application aims to overcome the above technical challenges through innovative technical solutions and achieve more efficient, flexible and scalable multi-mode heterogeneous data processing capabilities.

[0039] This application proposes an innovative solution for an enhanced distributed multi-mode heterogeneous data SQL query analysis engine based on a large language model in response to the performance bottlenecks and functional deficiencies of the prior art when processing multi-mode heterogeneous data. This solution aims to achieve specific processing of heterogeneous data by integrating heterogeneous computing, rather than general processing of diverse data, so as to effectively meet the high-performance computing requirements of non-independent and identically distributed data. In addition, the present invention also makes up for the functional deficiencies of conventional query analysis engines in hybrid retrieval, RAG retrieval, intelligent load, and intelligent selection of execution plans based on large models.

[0040] See also Figure 1 , Figure 1 A flowchart of a multi-mode heterogeneous data query method provided in an embodiment of the present application. Figure 1 As shown, the specific steps include the following steps.

[0041] Step S102: receiving a query statement sent by the user terminal, performing lexical analysis and grammatical analysis on the query statement through a parser, and performing semantic checking on the parsed query statement to obtain an initial query statement.

[0042] Step S104: Send the initial query statement to the query optimizer to generate a corresponding query syntax tree.

[0043] Step S106: After sending the query syntax tree to the Data-speed engine for prompt word engineering processing, it is indexed in conjunction with an index vector library carrying multi-mode heterogeneous data to obtain an index result, and based on the index result, an execution plan is intelligently predicted.

[0044] Step S108: Input the intelligent prediction execution plan into the large language model system, and output the execution code corresponding to the current execution task and the operators of the available instruction sets corresponding to different execution tasks in real time, wherein entity identification is introduced into the attention mechanism of the large language model system for supervised fine-tuning.

[0045] Step S110: calling the operator corresponding to the execution code through the Data-speed JIT generator, and generating an execution result based on the operator.

[0046] The embodiments of the present application, on the one hand, utilize the powerful capabilities of the large language model to generate JIT (Just-In-Time) code for current tasks and data in real time, thereby optimizing computing performance; on the other hand, utilize the large language model to generate operators that can fully utilize the GPU and SIMD instruction set, thereby improving parallel computing efficiency; and on the other hand, by directly inputting entity identifiers into the attention mechanism, the model can better perceive and utilize entity information, thereby enhancing the ability to represent entity semantics.

[0047] In the embodiment of the present application, after sending the query syntax tree to the Data-speed engine for prompt word engineering processing, the query syntax tree is indexed in combination with an index vector library carrying multi-mode heterogeneous data to obtain an index result, including:

[0048] After sending the query syntax tree to the Data-speed engine for prompt word engineering processing, RAG indexing is performed in combination with an index vector library carrying multi-mode heterogeneous data to obtain an initial index result;

[0049] The initial query statement and the first index result are combined to perform mixed indexing to obtain an index result, wherein the mixed index includes vector retrieval for approximate query and scalar retrieval for complex condition filtering.

[0050] It should be noted that RAG retrieval is combined with Retrieval-augmented Generation technology, which uses vector databases to store corporate private data and generates accurate answers through large models to improve response speed and stability.

[0051] Hybrid search combines vector search and scalar search, using vectors to solve approximate query problems and scalars to solve complex condition filtering problems. This requires the database to support multiple filtering mechanisms, such as pre-search filtering, filtering during search, and post-search filtering.

[0052] The conventional large-model prompt word method has limited improvement in the ability to understand the "sql execution plan". This application constructs a large number of case answers and adds them to the RAG system. With the help of prompt engineering, the system's ability to decompose and generate sql execution plans is further enhanced.

[0053] In an embodiment of the present application, the entity identification is introduced into the attention mechanism of the large language model system for supervised fine-tuning, including:

[0054] Based on the attention mechanism of the entity identification vector, the large language model system is fine-tuned through supervision, wherein the attention mechanism based on the entity identification vector includes:

[0055]

[0056]

[0057] Q'=W Q (Q+E);

[0058] L'=W K (K+E);

[0059] V'=W V (V+E);

[0060] Among them, E is the entity identification vector related to the input sequence; Q is Query, K is key, and V is Value, which are the three core components of the attention mechanism and are vectors derived from the input data.

[0061] Used to calculate attention weights and output results.

[0062] Conventional fine-tuning methods have limited ability to improve the understanding of SQL. This application introduces entity identification in the attention mechanism, trains the Data-speed basic model, and enhances the large model's ability to pay attention to SQL.

[0063] In the embodiment of the present application, the calling of the operator corresponding to the execution code by the Data-speed JIT generator and the generation of the execution result based on the operator include:

[0064] Determine the data characteristics of multi-mode heterogeneous data, the current computing power node and the instruction set supported by the current computing power node, generate adaptive LLVM instructions in real time through the Data-speed JIT generator, and compile the LLVM instructions into machine code;

[0065] Based on the machine code, an operator corresponding to the execution code is called, and based on the operator, an execution result is generated.

[0066] It should be noted that JIT compilation can optimize the execution plan of database queries at runtime, reduce function call overhead and loop overhead through inline optimization, loop unrolling, constant folding and other means, and improve execution efficiency.

[0067] In an embodiment of the present application, the method for generating the index vector library includes:

[0068] Collect multi-mode heterogeneous data in real time, and perform vectorization processing on the multi-mode heterogeneous data based on a vectorization engine to obtain a vectorized file;

[0069] Based on the vectorization file, an index vector library is generated.

[0070] Specifically, the vectorization engine is used to vectorize the multi-mode heterogeneous data to obtain a vectorized file, including:

[0071] Slicing the multi-mode heterogeneous data to obtain multi-mode heterogeneous slice data;

[0072] Extracting heterogeneous data from the multi-mode heterogeneous slice data through an embedded model;

[0073] An index is constructed for the heterogeneous data through an index optimizer to obtain a vectorized file.

[0074] It should be noted that through the big model technology, semantic analysis and vectorization processing can be performed on the data in the database to establish a more accurate index, thereby improving retrieval efficiency. For example, by converting data blocks into semantic vectors through the embedding model and creating indexes, the quality and speed of retrieval can be improved.

[0075] In another embodiment, the method further comprises:

[0076] Understanding at least one inputted execution prediction execution plan based on the large language model system, and determining an index form of at least one of the prediction execution plans;

[0077] Generate different versions of query questions based on different index forms;

[0078] Vector retrieval is performed based on different versions of query questions to determine the execution code corresponding to the current execution task and the operators of the available instruction sets corresponding to different execution tasks.

[0079] It should be noted that the large model is used to understand and rewrite the user's query into a form that is more suitable for database retrieval. This can be achieved by generating multiple versions of the query question, improving the quality of the question, then performing vector retrieval, and finally comprehensively ranking the results.

[0080] See also Figure 2 , Figure 2 A schematic diagram of an enhanced distributed multi-mode heterogeneous data SQL query analysis engine based on a Data-speed large language model provided in an embodiment of the present application. Figure 2 As shown, the enhanced distributed multi-mode heterogeneous data SQL query and analysis engine based on the big model includes five units: multi-mode heterogeneous data source, Data-speed vectorization engine, Data-speed engine, query engine, and Data-speed intelligent adaptive execution engine.

[0081] Specifically, the multi-mode heterogeneous data source is the source of multi-mode heterogeneous data, including from MySQL HBase CSV JSON files, etc.

[0082] Data-speed vectorization engine is used to vectorize multi-mode heterogeneous data. The input is multi-mode heterogeneous data, and the output is vectorized files stored in the data-speed precise index vector database. This module includes data slicing, embeddingmodel extraction of embedding, and data-speed index optimizer for building indexes for vectorized heterogeneous data, which are finally stored in the data-speed precise index vector database.

[0083] The Data-speed engine is used to receive the user query of the query engine and the query that is combined with the data-speed precise index vector database and send it to the data-speed engine. When the data-speed engine performs a query, it will start the data-speed intelligent adaptive execution engine to improve the query speed, accuracy and other capabilities of the data-speed engine. This module includes a pre-training system and a supervised fine-tuning system for training the data-speed large model, which has intelligent load, query rewriting and intelligent prediction functions.

[0084] The query engine is used to receive user query statements, parse and semantically check the user's query statements. The data-speed optimizer relies on the data-speed engine, the data-speed precise index vector database, and the data-speed intelligent adaptive execution engine to ultimately provide accurate results for heterogeneous queries.

[0085] Data-speed intelligent adaptive execution engine is used to support operators when the query engine calls the data-speed engine, providing faster, smarter, and more universal operators.

[0086] In addition, it also includes an intelligent operation and maintenance module. The large model can be used for intelligent operation and maintenance of the database. For example, by analyzing historical data and real-time monitoring data, it can predict database performance bottlenecks and potential problems, and perform optimization and adjustments in advance.

[0087] The multi-mode heterogeneous data query method provided in the embodiment of the present application is applied to achieve specific processing of heterogeneous data by integrating heterogeneous computing, rather than general processing of diverse data, which can effectively cope with the high-performance computing requirements of non-independent and identically distributed data. On the one hand, it solves the accuracy and efficiency bottlenecks of the traditional Attention mechanism: by introducing entity identification and generating efficient adaptive operators, the present invention significantly improves the accuracy and efficiency of multi-mode heterogeneous data processing; on the other hand, it generates a large language model that adapts to the multi-mode heterogeneous data processing engine: the model can optimize the characteristics of multi-mode heterogeneous data and generate efficient query plans and execution codes; on the other hand, it realizes an intelligent adaptive execution engine: by real-time generation and compilation of LLVM JIT instructions, the present invention realizes intelligent adaptive processing of multi-mode heterogeneous data, greatly improving computing performance; on the other hand, it provides comprehensive intelligent operation and maintenance management: through real-time analysis and intelligent management of cluster status, the present invention ensures the stability and efficiency of the system.

[0088] Corresponding to the above method embodiment, this specification also provides a multi-mode heterogeneous data query device embodiment, Figure 3 Schematic diagram of a multi-mode heterogeneous data query device provided by an embodiment of this specification. Figure 3 As shown, the device comprises:

[0089] The parsing module 302 is configured to receive a query statement sent by a user terminal, perform lexical parsing and grammatical parsing on the query statement through a parser, and perform semantic checking on the parsed query statement to obtain an initial query statement;

[0090] A generating module 304 is configured to send the initial query statement to a query optimizer to generate a corresponding query syntax tree;

[0091] The indexing module 306 is configured to send the query syntax tree to the Data-speed engine for prompt word engineering processing, and then perform indexing in combination with an index vector library carrying multi-mode heterogeneous data to obtain an index result, and intelligently predict an execution plan based on the index result;

[0092] The output module 308 is configured to input the intelligent prediction execution plan into the large language model system, and output the execution code corresponding to the current execution task and the operators of the available instruction sets corresponding to different execution tasks in real time, wherein the entity identification is introduced into the attention mechanism of the large language model system for supervised fine-tuning;

[0093] The calling module 310 is configured to call the operator corresponding to the execution code through the Data-speed JIT generator, and generate an execution result based on the operator.

[0094] In an optional embodiment, the index module 306 is further configured to:

[0095] After sending the query syntax tree to the Data-speed engine for prompt word engineering processing, RAG indexing is performed in combination with an index vector library carrying multi-mode heterogeneous data to obtain an initial index result;

[0096] The initial query statement and the first index result are combined to perform mixed indexing to obtain an index result, wherein the mixed index includes vector retrieval for approximate query and scalar retrieval for complex condition filtering.

[0097] In an optional embodiment, the output module 308 is further configured to:

[0098] Based on the attention mechanism of the entity identification vector, the large language model system is fine-tuned through supervision, wherein the attention mechanism based on the entity identification vector includes:

[0099]

[0100]

[0101] Q'=W Q (Q+E);

[0102] L'=W K (K+E);

[0103] V'=W V (V+E);

[0104] Among them, E is the entity identification vector related to the input sequence; Q is Query, K is key, and V is Value. They are the three core components in the attention mechanism. They are vectors derived from the input data and are used to calculate the attention weight and output results.

[0105] In an optional embodiment, the calling module 310 is further configured to:

[0106] Determine the data characteristics of multi-mode heterogeneous data, the current computing power node and the instruction set supported by the current computing power node, generate adaptive LLVM instructions in real time through the Data-speed JIT generator, and compile the LLVM instructions into machine code;

[0107] Based on the machine code, an operator corresponding to the execution code is called, and based on the operator, an execution result is generated.

[0108] In an optional embodiment, the index module 306 is further configured to:

[0109] Collect multi-mode heterogeneous data in real time, and perform vectorization processing on the multi-mode heterogeneous data based on a vectorization engine to obtain a vectorized file;

[0110] Based on the vectorization file, an index vector library is generated.

[0111] In an optional embodiment, the index module 306 is further configured to:

[0112] Slicing the multi-mode heterogeneous data to obtain multi-mode heterogeneous slice data;

[0113] Extracting heterogeneous data from the multi-mode heterogeneous slice data through an embedded model;

[0114] An index is constructed for the heterogeneous data through an index optimizer to obtain a vectorized file.

[0115] In an optional embodiment, the device further includes:

[0116] a form determination module, configured to understand at least one inputted execution prediction execution plan based on the large language model system, and determine an index form of at least one of the prediction execution plans;

[0117] A question generation module is configured to generate different versions of query questions based on different index forms;

[0118] The task generation module is configured to perform vector retrieval based on different versions of query questions, determine the execution code corresponding to the current execution task and operators of available instruction sets corresponding to different execution tasks.

[0119] The multi-mode heterogeneous data query device provided in the embodiment of the present application is applied to realize specific processing of heterogeneous data by integrating heterogeneous computing, rather than general processing of diverse data, which can effectively cope with the high-performance computing requirements of non-independent and identically distributed data. On the one hand, it solves the accuracy and efficiency bottlenecks of the traditional Attention mechanism: by introducing entity identification and generating efficient adaptive operators, the present invention significantly improves the accuracy and efficiency of multi-mode heterogeneous data processing; on the other hand, it generates a large language model that adapts to the multi-mode heterogeneous data processing engine: the model can optimize the characteristics of multi-mode heterogeneous data and generate efficient query plans and execution codes; on the other hand, it realizes an intelligent adaptive execution engine: by real-time generation and compilation of LLVM JIT instructions, the present invention realizes intelligent adaptive processing of multi-mode heterogeneous data and greatly improves computing performance; on the other hand, it provides comprehensive intelligent operation and maintenance management: through real-time analysis and intelligent management of cluster status, the present invention ensures the stability and efficiency of the system.

[0120] Figure 4 This is a block diagram of a computing device provided in an embodiment of the present application. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0121] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0122] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0123] The computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 400 may also be a mobile or stationary server.

[0124] The processor 420 is used to execute the following computer executable instructions, which, when executed by the processor, can implement the above-mentioned multi-mode heterogeneous data query function.

[0125] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the multi-mode heterogeneous data query embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the multi-mode heterogeneous data query embodiment.

[0126] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which can implement the above-mentioned multi-mode heterogeneous data query function when executed by a processor.

[0127] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the multi-mode heterogeneous data query embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the multi-mode heterogeneous data query embodiment.

[0128] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is enabled to perform the above-mentioned multi-mode heterogeneous data query function.

[0129] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer program embodiment, since it is basically similar to the multi-mode heterogeneous data query embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the multi-mode heterogeneous data query embodiment.

[0130] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0131] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0132] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0133] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0134] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A multi-mode heterogeneous data query method, characterized in that: include: Receiving a query statement sent by a user terminal, performing lexical analysis and grammatical analysis on the query statement through a parser, and performing semantic checking on the parsed query statement to obtain an initial query statement; Sending the initial query statement to a query optimizer to generate a corresponding query syntax tree; After sending the query syntax tree to the Data-speed engine for prompt word engineering processing, it is indexed in combination with an index vector library carrying multi-mode heterogeneous data to obtain an index result, and based on the index result, an execution plan is intelligently predicted; The intelligent prediction execution plan is input into the large language model system, and the execution code corresponding to the current execution task and the operators of the available instruction sets corresponding to different execution tasks are output in real time, wherein entity identification is introduced into the attention mechanism of the large language model system for supervised fine-tuning; The operator corresponding to the execution code is called through the Data-speed JIT generator, and an execution result is generated based on the operator.

2. The method according to claim 1, characterized in that After the query syntax tree is sent to the Data-speed engine for prompt word engineering processing, it is indexed in conjunction with an index vector library carrying multi-mode heterogeneous data to obtain an index result, including: After sending the query syntax tree to the Data-speed engine for prompt word engineering processing, RAG indexing is performed in combination with an index vector library carrying multi-mode heterogeneous data to obtain an initial index result; The initial query statement and the first index result are combined to perform mixed indexing to obtain an index result, wherein the mixed index includes vector retrieval for approximate query and scalar retrieval for complex condition filtering.

3. The method according to claim 1, characterized in that The entity identification is introduced into the attention mechanism of the large language model system for supervised fine-tuning, including: Based on the attention mechanism of the entity identification vector, the large language model system is fine-tuned through supervision, wherein the attention mechanism based on the entity identification vector includes: Q'=W Q (Q+E); W'=W K (W+E); V'=W V (V+E); Among them, E is the entity identification vector related to the input sequence; Q is Query, K is key, and V is Value. They are the three core components in the attention mechanism. They are vectors derived from the input data and are used to calculate the attention weight and output results.

4. The method according to claim 1, characterized in that: The calling of the operator corresponding to the execution code by the Data-speed JIT generator and generating the execution result based on the operator include: Determine the data characteristics of multi-mode heterogeneous data, the current computing power node and the instruction set supported by the current computing power node, generate adaptive LLVM instructions in real time through the Data-speed JIT generator, and compile the LLVM instructions into machine code; Based on the machine code, an operator corresponding to the execution code is called, and based on the operator, an execution result is generated.

5. The method according to claim 1, characterized in that The method for generating the index vector library comprises: Collect multi-mode heterogeneous data in real time, and perform vectorization processing on the multi-mode heterogeneous data based on a vectorization engine to obtain a vectorized file; Based on the vectorization file, an index vector library is generated.

6. The method according to claim 5, characterized in that The vectorization engine is used to vectorize the multi-mode heterogeneous data to obtain a vectorized file, including: Slicing the multi-mode heterogeneous data to obtain multi-mode heterogeneous slice data; Extracting heterogeneous data from the multi-mode heterogeneous slice data through an embedded model; An index is constructed for the heterogeneous data through an index optimizer to obtain a vectorized file.

7. The method according to claim 1, characterized in that The method further comprises: Understanding at least one inputted execution prediction execution plan based on the large language model system, and determining an index form of at least one of the prediction execution plans; Generate different versions of query questions based on different index forms; Vector retrieval is performed based on different versions of query questions to determine the execution code corresponding to the current execution task and the operators of the available instruction sets corresponding to different execution tasks.

8. A multi-mode heterogeneous data query device, characterized in that: include: The parsing module is configured to receive a query statement sent by a user terminal, perform lexical parsing and grammatical parsing on the query statement through a parser, and perform semantic checking on the parsed query statement to obtain an initial query statement; A generation module is configured to send the initial query statement to a query optimizer to generate a corresponding query syntax tree; An indexing module is configured to send the query syntax tree to the Data-speed engine for prompt word engineering processing, and then perform indexing in combination with an index vector library carrying multi-mode heterogeneous data to obtain an index result, and intelligently predict an execution plan based on the index result; An output module is configured to input the intelligent prediction execution plan into a large language model system, and output in real time the execution code corresponding to the current execution task and operators of available instruction sets corresponding to different execution tasks, wherein entity identification is introduced into the attention mechanism of the large language model system for supervised fine-tuning; The calling module is configured to call the operator corresponding to the execution code through the Data-speed JIT generator, and generate an execution result based on the operator.

9. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images