61results about How to "Improve query performance" patented technology

The invention provides a natural language SQL (Structured Query Language) conversion method based on a data platform and a large language model, which relates to the technical field of data processing and comprises the following steps of: performing vectorization storage on metadata, blood relationship, indexes and historical query information, performing intention judgment and vector matching on a user query request, and determining a query condition based on a matching result. An association path is constructed according to data consanguinity, and an optimal SQL statement is generated by utilizing a historical query optimization table connection sequence and combining index matching. According to the method, intelligent conversion from the natural language to the efficient SQL can be realized, the query efficiency is improved, and the use threshold of a user is reduced.

The invention relates to the technical field of medical care, in particular to an intelligent query method for a nursing knowledge base, which comprises the following steps of: performing multi-form standardized processing on a nursing problem input by a user, deeply analyzing a query request by utilizing a semantic understanding model, and obtaining an intelligent query result based on an inference algorithm; the method comprises the following steps: automatically selecting related literatures, clinical data and nursing experience data from a nursing knowledge base, generating a preliminary nursing scheme candidate set in combination with a nursing professional knowledge graph, and performing personalized screening and optimization on the scheme by using an intelligent recommendation model in combination with the current illness state, treatment progress and historical nursing data of a patient; the intelligent recommendation model is self-optimized through user feedback, and the nursing scheme recommendation precision is continuously improved. The nursing knowledge base automatically fuses newest medical research results and clinical nursing practice data. The problems that in the prior art, nursing scheme recommendation accuracy is low, updating is not timely, and the individuation degree is insufficient are effectively solved.

The invention relates to an insurance industry-oriented customer full-life-cycle event data modeling method and system. The method comprises the steps of collecting customer full-life-cycle event data through a customer contact channel and a business processing link of an insurance industry; preprocessing the collected data, and converting the data into structured data; key elements are extracted from the preprocessed structured detail data; performing hierarchical and classified storage on the structured data based on a hierarchical and classified event data model, and respectively establishing a trunk index model and an expansion detail model in combination with the key elements; generating a unique event ID (Identity) for each event from the key elements; and performing data query optimization and providing data service based on the trunk index model, the expansion detail model and the event ID. Compared with the prior art, the method has the advantages that massive customer event data can be efficiently integrated and analyzed, the data processing and query capability is improved, more efficient and flexible data services are provided, and decision support is provided for the insurance industry.

The invention discloses a database query method, device and system and a medium, and the method comprises the steps: receiving a database query statement, analyzing the query statement to generate an execution plan of a tree structure, segmenting the execution plan, and obtaining a plurality of execution plan subtrees; matching the plurality of execution plan sub-trees with the intermediate result set in the cache, and determining whether intermediate result set sub-trees equal to the execution plan sub-trees exist in the intermediate result set or not; if an intermediate result set sub-tree equal to the execution plan sub-tree exists in the cache, replacing the successfully matched execution plan sub-tree with the equal intermediate result set sub-tree, and generating a replacement execution plan; and performing scheduling execution according to the replacement execution plan to obtain a corresponding database query result. The subtrees in the execution plan are matched and replaced through the intermediate result set in the cache, so that the intermediate result set in the cache can be directly used in subsequent execution operators, resource consumption and execution time are reduced, and database query performance is improved.

The invention discloses an analysis query method and device for a database, and the method comprises the steps: obtaining data corresponding to a query instruction from a pre-constructed preset storage structure through a preset index algorithm when the query instruction is received in a data query process of a database system, the preset storage structure is obtained by optimizing a storage structure design layer of a multi-version concurrency control mechanism, the preset storage structure is used for avoiding volume expansion caused by multi-version control and improving the performance of data analysis and query, and data is compressed through a data compression algorithm. And storing the compressed data in a preset storage structure in a column storage mode.

The invention relates to the technical field of data processing, in particular to a data statistical method based on a large data volume, which comprises the following steps of: S1, defining a many-to-many first subject table and a second subject table of different data magnitudes in a relational database; s2, acquiring a mapping relation between the first main body table and a second main body table, traversing the second main body table, acquiring IDs of all records corresponding to each record in the second main body table in the first main body table, and splicing an ID list of nodes on a path by traversing a path from a node where the ID is located to a root node, after splicing is completed, an array is formed and stored in a field added to a corresponding record in the first main body table, and storage of the data mapping relation is completed; and S3, completing storage of a data mapping relationship between the first main body table and the second main body table for subsequent adjustment, statistics or query calling. According to the method, the data storage cost can be greatly reduced, the data maintenance cost can be reduced, and real-time statistics and query of a large amount of data can be realized.

The invention discloses a business data management method and system, electronic equipment and a storage medium. The method comprises the following steps: constructing a preset business index library; wherein the preset business index library comprises a first storage unit and a second storage unit; obtaining to-be-stored business data; wherein the business data to be stored comprises index metadata and index numerical data, and the index metadata corresponds to the index numerical data; storing the index metadata to the first storage unit, storing the index numerical data to the second storage unit, and constructing to obtain a target business index library; and when it is determined that the query index request is received, querying from the target business index library according to the query index request to obtain target business data. According to the embodiment of the invention, the management of the business data can be realized, and the query efficiency and query performance of the business data are effectively improved. The method can be widely applied to the technical field of big data.

The present invention provides a method and system for automatically generating database query rewriting rules, including: step 1: modeling the query plan and query rewriting rules of the database into an enumerable form; step 2: enumerating the constraints between each pair of query plan templates to generate a first query rewriting rule. Compared with the prior art, the present invention can generate new rewriting rules that do not exist in the existing database system, can match and rewrite more complex queries generated by frameworks such as object-relational mapping in Internet applications, and improve the database query rewriting capability and the performance of database queries under Internet applications.

This invention relates to an index construction method, device, program product, and storage medium, and pertains to the technical field of vector databases. The method includes: acquiring query vectors; calculating the coordinate region identifiers of each query vector in the vector space; statistically analyzing the query frequency of each coordinate region identifier within a preset time window to obtain regional query frequency distribution data; calculating the frequency growth rate of the regional query frequency distribution data of each coordinate region identifier at the current moment compared to the regional query frequency distribution data at the previous moment; calculating a query popularity fluctuation coefficient based on each frequency growth rate; monitoring the load parameters of the vector database system in real time and calculating a comprehensive system load index based on the load parameters; and constructing an index for each query vector based on the query popularity fluctuation coefficient and the comprehensive system load index. Implementing the technical solution provided in this application can achieve a balance between indexing effectiveness and system stability.

This application provides one or more embodiments of a relational database management method and apparatus. The method includes: obtaining a creation instruction for creating a target data table in the relational database; wherein the creation instruction includes column information corresponding to basic data columns in the target data table; responding to the creation instruction, creating the target data table in the relational database, including the basic data columns and hidden data columns, based on the column information; wherein the hidden data columns are extended data columns independent of the basic data columns; after the target data table is created, establishing a unique tree-structured index for the target data table based on the data in the hidden data columns; wherein the data in the hidden data columns corresponding to data rows in the target data table includes monotonically increasing values.

The invention relates to the field of knowledge graphs, and discloses a bitmap-based graph data storage and query method, system, device and medium, the method adopts a unified bitmap model to store graph data, converts graph structure data into a bitmap suitable for parallel computing, maps a graph query request into a bitmap operation expression tree, and stores the graph data in the bitmap operation expression tree; therefore, graph query is converted into bitmap operation, and the query performance of high-concurrency and complex graph query is remarkably improved; moreover, a reasonable computing resource scheduling decision is dynamically set according to an input graph query request, and for high-concurrency and complex graph query, thousands of cores of a large-scale parallel computing engine are used for parallel processing, so that the random access bottleneck and the CPU computing power bottleneck of traditional graph traversal are thoroughly avoided, the query speed of order of magnitude can be increased, and the query efficiency is improved. A plurality of complex atlas analysis queries can be processed at the same time, and the real-time requirement of complex atlas query can be met.

The invention discloses a historical data archiving implementation method and device and a medium. The method comprises the following steps: creating a system architecture and a table; under the condition that table creation is completed, archiving conditions are configured; under the condition that the archiving condition configuration is completed, the archiving task is executed regularly; and aiming at the execution process of the archiving task, the query function is optimized. The method has the beneficial effects that the comprehensive functions of reducing the data volume of the table, improving the query performance, supporting flexible condition configuration, reducing the storage space occupied by the table, avoiding associated data migration, querying archived historical data, querying archived and unarchived total data and periodically deleting useless data through configuration are realized.

The invention relates to the technical field of data storage, in particular to a data processing method and device, equipment and a storage medium, and is used for improving the deletion efficiency of different keys in an LSM number. The method comprises the following steps: for any hierarchy of a log structure merging LSM tree, obtaining each to-be-processed file, namely a read-only memory table or a source SST associated with the hierarchy, sorting and traversing a first type of keys in the files, and determining whether the keys are invalid or not according to a preset judgment condition; wherein the judgment condition comprises at least one of the following conditions: the key itself or the value corresponding to the key itself meets a first value taking condition, and the key itself or the value corresponding to the key itself and the preorder key itself or the value corresponding to the key itself meets a second value taking condition; the first type of keys are different keys in the to-be-processed files; and for each to-be-processed file, generating a target SST through the key value pair data of the effective key.

The application discloses a system and method for accelerating reading compressed files based on A Virtual File System, and belongs to the technical field of time series databases; the technical problem to be solved by the application is how to realize column format compressed storage and reading of time series data types, reduce IO, memory and CPU consumption caused by decompression, and improve the ability of the system to process time series data; the technical scheme adopted is as follows: in the time series engine of the open database ZNBase, an execution engine generates an execution plan according to SQL semantic analysis, calls a storage layer interface of ZNBase, and queries accurate time series history partition compressed data; the storage layer completes fast execution of the query in the storage layer based on various compressed query interfaces provided by AVFS, and accelerates the query of the database; the specific steps are as follows: pre-analysis; compressed file pre-reading; reading compressed data.

This application discloses a data aggregation method, data aggregation device, electronic device, and storage medium, relating to the field of computer technology. It can improve the query performance of data after it is stored in the database, and reduce the difficulty and cost of querying. The specific solution includes: acquiring network data stream traffic data; generating at least one first data log and at least one second data log based on the traffic data; aggregating all first data logs according to a first aggregation rule based on the link identifier corresponding to the first data log to obtain at least one first aggregated log; and aggregating all second data logs according to a second aggregation rule based on the link identifier corresponding to the second data log to obtain at least one second aggregated log.

The present invention relates to a query system based on data analysis, and in particular to the technical field of database management. The query system comprises a user information management module for implementing dynamic authority management for users, a data source management module for managing user requests, caches, data sources and data security, and a database management module for managing a number of databases. The present invention analyzes the query request of the user to determine whether the user request exists in the cache, so as to determine the selection of the query mode for the query request, thereby making flexible selections in the query process, and setting different supervision strengths in the corresponding query mode to supervise the query process, thereby improving the accuracy of control over the data query process.

The invention discloses a database distributed version synchronization and intelligent compression method and system, and relates to the technical field of distributed databases. The method comprises the following steps: a database fragment management mechanism in a distributed environment divides a database into a plurality of fragments, and each fragment independently maintains a versioned storage manager; a cross-node version coordination mechanism ensures the consistency of global version numbers; according to a multi-copy synchronization mechanism, the consistency of multiple copies is ensured by adopting a two-stage submission protocol; an intelligent copy selection algorithm; a distributed garbage recycling mechanism; the version compression mechanism is used for analyzing historical version data and storing the historical version data by adopting increment; intelligently combining strategies, and determining a compression strategy according to the data access frequency. According to the method, the problem of consistency of data version synchronization in a distributed environment is solved, and meanwhile, storage space occupation is reduced through intelligent compression.

The invention discloses a network security dynamic early warning method and system based on a knowledge graph, and relates to the technical field of network security, and the method comprises the following steps: S1, multi-dimensional network security knowledge acquisition and integration, S2, knowledge graph construction, S3, network security dynamic early warning and query, and S4, system optimization and updating. Multi-dimensional knowledge integration: constructing a comprehensive and accurate knowledge graph by integrating multi-dimensional knowledge in the field of network security, providing a solid foundation for early warning and query, and constructing a high-efficiency graph data improving the construction efficiency and query performance of the knowledge graph by utilizing an advanced graph database technology and an index optimization strategy; and intelligent early warning and query: combining real-time monitoring and intelligent analysis technologies to realize network security dynamic early warning and accurate query functions and improve the applicability and user experience of the system, and continuous optimization and iteration: continuously optimizing and updating the system according to the user feedback and the development change of the network security field to ensure the timeliness and accuracy of the system.

The invention relates to the technical field of computers, and discloses a non-deterministic relation entry search method, device and equipment and a knowledge graph construction method, device and equipment. Performing vector conversion on the target relation data to obtain a target relation vector corresponding to the target relation data; according to the target relation vector, obtaining a candidate relation vector having similarity with the target relation vector and a vector index to which the candidate relation vector belongs from a pre-constructed knowledge graph library; and according to the similarity between each candidate relation vector and the target relation vector, the vector index to which the candidate relation vector belongs and the target object data, obtaining an output result corresponding to the search request from a knowledge graph library. The method does not need to consider whether the relationship between different entries in the knowledge graph database is uniquely specified or not. And operations such as semantic alignment and normalization processing do not need to be executed during searching. The searching efficiency is higher, the cost is lower, and the method can be suitable for multiple types of knowledge maps and searching tasks.

The invention relates to a data integration method and device based on geographic space information. The method comprises the following steps: selecting a to-be-analyzed file, and configuring a geographic space data integration source end and a target end; obtaining metadata of the geographic space data file; performing conversion according to the obtained metadata to obtain a target end field and assembling a table building statement; creating a spatial index of the target end according to the obtained metadata; and segmenting according to the number of geographic space data files, segmenting a corresponding number of subtasks by a manage management node, and concurrently executing file analysis, SQL assembly and data writing operation. According to the method, the geographic space data file is analyzed through the program to obtain the metadata information, the analysis process of files in different formats is unified, the technical background requirement and the learning cost of operators are reduced, the data integration efficiency is improved, and the complexity and the time cost of geographic space data integration are reduced. According to the method, the memory overflow risk is reduced by adopting a block reading mode, and the stability of metadata analysis is improved.

The invention discloses an optimized query method based on multi-table statistical information cardinality estimation. The method comprises the following steps: constructing a star structure in an offline stage, and converting historical multi-table connection query into virtual key connection between nodes in the star structure; calculating and counting a structure cardinal number and a degree sequence in the star-shaped structure through a merged degree sequence, and compressing and outputting a finally obtained degree sequence through a streaming degree sequence; at the present stage, according to the connection condition of the multi-table connection query input in real time, the multi-table connection query is decomposed into a plurality of star structure instances, and after the degree sequence obtained in each corresponding offline stage is queried, the final cardinality upper bound estimation value of the whole query is obtained through middle structure merging iteration. According to the method, on the premise that the upper bound is estimated, the precision and efficiency of multi-table connection cardinal number estimation are remarkably improved, and the method is suitable for a query optimization scene of a relational database.

This invention discloses a method and system for automated generation and optimization of database table structures. The method specifically includes: receiving the user's backend project source code; extracting a semantic information network graph related to data persistence through static code scanning; inputting the semantic information network graph into an intelligent parsing engine to analyze code logic relationships and obtain index recommendation strategies and potential table structure design flaws; based on the index recommendation strategies and potential table structure design flaws, identifying sensitive fields and recommending encryption or de-identification schemes to obtain a first table structure; performing automated stress testing on the first table structure and adjusting the index strategy and partitioning scheme based on the test results; and optimizing the first table structure according to the adjusted index strategy and partitioning scheme to obtain a second table structure. This invention achieves intelligent generation and optimization of database table structures, improving development efficiency, enhancing maintainability, improving user experience, and ensuring data security.

The invention provides a medical big data ES wide table generation method and device based on efficient dynamic data configuration, and belongs to the technical field of medical data processing, and the method comprises the steps: carrying out dynamic data configuration based on a built ES distributed cluster; according to dynamic data configuration, performing personnel main table query in the dimension of people and source table query in the dimension of doctor seeing from a specified table of a data source through multiple threads, and collecting data of required fields; converting the collected data into a unified ES field structure type according to a data type so as to obtain standardized data; and carrying out data assembly according to the dimensions of people and doctor seeing based on the standardized data, generating ES personnel wide table data and ES doctor seeing wide table data, and updating the two pieces of ES wide table data to an ES distributed cluster in real time. According to the invention, large-scale medical data can be efficiently integrated and managed, powerful data retrieval and analysis capabilities are provided, and the capabilities of cross-mechanism data sharing, rapid query and accurate analysis are improved.

This specification provides a unified indexing method, apparatus, electronic device, and medium for multiple metric vectors. The method includes: in the index creation phase, constructing a shared graph topology structure decoupled from similarity metrics; in the query execution phase, enabling the query optimizer to generate an execution plan containing a single index scan node and generating a corresponding metric type identifier; performing graph traversal based on the shared graph topology structure, and if no match is found, calling the underlying calculation kernel function corresponding to the target similarity metric type to perform lazy distance calculation and writing it into a shared memory cache pool; dynamically adjusting the pruning threshold and search width parameters of the graph traversal, continuing to traverse the shared graph topology structure until the termination condition is met, and outputting the final candidate node set; and returning the final candidate node set and its corresponding distance values ​​to the query executor layer of the database system through the index scan callback interface in the custom index access method plugin, and outputting the final query result set.

The invention provides a database operation statement optimization method, a medium and a product. The method comprises the following steps: acquiring a database operation statement; judging whether the database operation statement contains a target keyword or not, wherein the target keyword is used for processing a result set obtained by the parallel query branch connected with the target keyword; if yes, the database operation statement is optimized according to the corresponding relation between the parallel query branches and the type of the target keyword, and therefore the query branches are reduced. Through the method, the number of parallel query branches can be reduced, so that unnecessary database operation and table scanning are reduced, the query efficiency is improved, the query logic is simplified, the database query performance is improved, and resource consumption is reduced.

The present invention discloses an HBase optimization scheme based on row-column hybrid storage, comprising the following steps: S1, constructing a new data table and index table, wherein the table structure of the data table and index is as follows: storing multiple values ​​of the same field continuously, and recording the maximum value, minimum value, and starting offset of the same field of the relevant columns in the data table and index table; S2, constructing a new row-column hybrid storage query scheme based on the new table structure; S3, converting Get to Scan on the data table; S4, rewriting Scan on the index table. This HBase optimization scheme based on row-column hybrid storage performs data modeling based on a row-column hybrid storage mode, can make full use of data similarity, significantly reduce data storage, and is verified by actual measurement to occupy 50% less storage than the original scheme; based on the new data aggregation storage mode, it can reduce the number of random data reads and improve overall query performance.

The invention discloses a database construction method based on hybrid heap allocation and versioned storage, and relates to the technical field of databases. The method comprises the following steps: establishing a database file structure system comprising a data file, a metadata file, a function file, a mode file and an extension file; the method comprises the following steps: constructing a versioned storage manager adopting a hybrid heap allocation algorithm, dividing a heap memory into memory pages and memory blocks with fixed sizes, managing the free memory pages by using a free linked list, recording the allocation state of the memory blocks in a bitmap, and adding version header information for the memory blocks; establishing a version index mechanism based on a persistent B + tree; and a historical data tracking function is realized. According to the method, the advantages of the idle linked list and the bitmap are combined, the memory allocation efficiency is guaranteed, the damage resistance of the system is improved, and meanwhile, complete historical tracking of the data is achieved through a versioning storage mechanism.

The invention provides a cloud data warehouse, a data reading method, a server and a computer storage medium, a computing node and an executor node are deployed on a physical node through affinity configuration, after the computing node receives a query request, position information of a target execution node is retrieved in a manager node, and a retrieval result is obtained; if the position information of the target execution node is the position information of an executor node corresponding to the calculation node, reading target data through a local file system; if the position information of the target execution node is not the position information of the executor node corresponding to the computing node, reading target data in the target execution node, and writing a copy of the target data in the executor node corresponding to the computing node; and if the position information of the target execution node is not retrieved, the computing node reads the target data in the object storage. Therefore, the problem that the cloud data warehouse cannot be cached and shared is solved, and the query performance is effectively improved.

The application provides a kind of time series data write-time incremental adaptive downsampling method, belong to data processing technical field, specifically include: in original time series data write-in triggers bucket division determination, maintain current open bucket and extract P5 and P95 quantile value of data in bucket, calculate fluctuation amplitude based on quantile range, determine bucket boundary by two-dimensional condition of time upper limit and fluctuation threshold, generate summary result containing first value, P5 value, P95 value, last value and its corresponding timestamp when bucket is closed and is stored, while dynamically updating fluctuation threshold using exponentially weighted moving average method according to fluctuation amplitude of closed bucket.The scheme of the application moves downsampling from query side to write-in side, directly reads summary result when querying, avoids repeated scanning of original data, and threshold can automatically evolve according to data characteristics, effectively resists transient glitch interference, and deployment requirement is low.

The invention belongs to the technical field of data supervision, and discloses a data quality supervision method, which specifically comprises the following steps of: 1, creating a quality rule for checking a data table and a field according to a service quality rule through a unified scheduling platform; according to the method, the quality rules for verifying the data tables and the fields are created through the unified scheduling platform, flexible creation and automatic verification of the service quality rules are achieved, and therefore the data quality is improved, the automation level of data processing is enhanced, secondly, the task execution condition is monitored in real time through SLA monitoring, it is ensured that the data processing task is ready on time, and the data processing efficiency is improved. According to the method and the system, the business continuity is enhanced, then the table information, the field information and the index caliber are comprehensively managed through the metadata management platform, the data structure and meaning can be more accurately understood, so that the analysis efficiency and accuracy are improved, and finally, the data development efficiency is improved by establishing a perfect data blood relationship analysis function.