27results about How to "Improve query performance" patented technology

This invention relates to an index construction method, device, program product, and storage medium, and pertains to the technical field of vector databases. The method includes: acquiring query vectors; calculating the coordinate region identifiers of each query vector in the vector space; statistically analyzing the query frequency of each coordinate region identifier within a preset time window to obtain regional query frequency distribution data; calculating the frequency growth rate of the regional query frequency distribution data of each coordinate region identifier at the current moment compared to the regional query frequency distribution data at the previous moment; calculating a query popularity fluctuation coefficient based on each frequency growth rate; monitoring the load parameters of the vector database system in real time and calculating a comprehensive system load index based on the load parameters; and constructing an index for each query vector based on the query popularity fluctuation coefficient and the comprehensive system load index. Implementing the technical solution provided in this application can achieve a balance between indexing effectiveness and system stability.

This application provides one or more embodiments of a relational database management method and apparatus. The method includes: obtaining a creation instruction for creating a target data table in the relational database; wherein the creation instruction includes column information corresponding to basic data columns in the target data table; responding to the creation instruction, creating the target data table in the relational database, including the basic data columns and hidden data columns, based on the column information; wherein the hidden data columns are extended data columns independent of the basic data columns; after the target data table is created, establishing a unique tree-structured index for the target data table based on the data in the hidden data columns; wherein the data in the hidden data columns corresponding to data rows in the target data table includes monotonically increasing values.

The invention relates to the field of knowledge graphs, and discloses a bitmap-based graph data storage and query method, system, device and medium, the method adopts a unified bitmap model to store graph data, converts graph structure data into a bitmap suitable for parallel computing, maps a graph query request into a bitmap operation expression tree, and stores the graph data in the bitmap operation expression tree; therefore, graph query is converted into bitmap operation, and the query performance of high-concurrency and complex graph query is remarkably improved; moreover, a reasonable computing resource scheduling decision is dynamically set according to an input graph query request, and for high-concurrency and complex graph query, thousands of cores of a large-scale parallel computing engine are used for parallel processing, so that the random access bottleneck and the CPU computing power bottleneck of traditional graph traversal are thoroughly avoided, the query speed of order of magnitude can be increased, and the query efficiency is improved. A plurality of complex atlas analysis queries can be processed at the same time, and the real-time requirement of complex atlas query can be met.

The application discloses a system and method for accelerating reading compressed files based on A Virtual File System, and belongs to the technical field of time series databases; the technical problem to be solved by the application is how to realize column format compressed storage and reading of time series data types, reduce IO, memory and CPU consumption caused by decompression, and improve the ability of the system to process time series data; the technical scheme adopted is as follows: in the time series engine of the open database ZNBase, an execution engine generates an execution plan according to SQL semantic analysis, calls a storage layer interface of ZNBase, and queries accurate time series history partition compressed data; the storage layer completes fast execution of the query in the storage layer based on various compressed query interfaces provided by AVFS, and accelerates the query of the database; the specific steps are as follows: pre-analysis; compressed file pre-reading; reading compressed data.

This application discloses a data aggregation method, data aggregation device, electronic device, and storage medium, relating to the field of computer technology. It can improve the query performance of data after it is stored in the database, and reduce the difficulty and cost of querying. The specific solution includes: acquiring network data stream traffic data; generating at least one first data log and at least one second data log based on the traffic data; aggregating all first data logs according to a first aggregation rule based on the link identifier corresponding to the first data log to obtain at least one first aggregated log; and aggregating all second data logs according to a second aggregation rule based on the link identifier corresponding to the second data log to obtain at least one second aggregated log.

The invention discloses a database distributed version synchronization and intelligent compression method and system, and relates to the technical field of distributed databases. The method comprises the following steps: a database fragment management mechanism in a distributed environment divides a database into a plurality of fragments, and each fragment independently maintains a versioned storage manager; a cross-node version coordination mechanism ensures the consistency of global version numbers; according to a multi-copy synchronization mechanism, the consistency of multiple copies is ensured by adopting a two-stage submission protocol; an intelligent copy selection algorithm; a distributed garbage recycling mechanism; the version compression mechanism is used for analyzing historical version data and storing the historical version data by adopting increment; intelligently combining strategies, and determining a compression strategy according to the data access frequency. According to the method, the problem of consistency of data version synchronization in a distributed environment is solved, and meanwhile, storage space occupation is reduced through intelligent compression.

The invention discloses an optimized query method based on multi-table statistical information cardinality estimation. The method comprises the following steps: constructing a star structure in an offline stage, and converting historical multi-table connection query into virtual key connection between nodes in the star structure; calculating and counting a structure cardinal number and a degree sequence in the star-shaped structure through a merged degree sequence, and compressing and outputting a finally obtained degree sequence through a streaming degree sequence; at the present stage, according to the connection condition of the multi-table connection query input in real time, the multi-table connection query is decomposed into a plurality of star structure instances, and after the degree sequence obtained in each corresponding offline stage is queried, the final cardinality upper bound estimation value of the whole query is obtained through middle structure merging iteration. According to the method, on the premise that the upper bound is estimated, the precision and efficiency of multi-table connection cardinal number estimation are remarkably improved, and the method is suitable for a query optimization scene of a relational database.

This invention discloses a method and system for automated generation and optimization of database table structures. The method specifically includes: receiving the user's backend project source code; extracting a semantic information network graph related to data persistence through static code scanning; inputting the semantic information network graph into an intelligent parsing engine to analyze code logic relationships and obtain index recommendation strategies and potential table structure design flaws; based on the index recommendation strategies and potential table structure design flaws, identifying sensitive fields and recommending encryption or de-identification schemes to obtain a first table structure; performing automated stress testing on the first table structure and adjusting the index strategy and partitioning scheme based on the test results; and optimizing the first table structure according to the adjusted index strategy and partitioning scheme to obtain a second table structure. This invention achieves intelligent generation and optimization of database table structures, improving development efficiency, enhancing maintainability, improving user experience, and ensuring data security.

This specification provides a unified indexing method, apparatus, electronic device, and medium for multiple metric vectors. The method includes: in the index creation phase, constructing a shared graph topology structure decoupled from similarity metrics; in the query execution phase, enabling the query optimizer to generate an execution plan containing a single index scan node and generating a corresponding metric type identifier; performing graph traversal based on the shared graph topology structure, and if no match is found, calling the underlying calculation kernel function corresponding to the target similarity metric type to perform lazy distance calculation and writing it into a shared memory cache pool; dynamically adjusting the pruning threshold and search width parameters of the graph traversal, continuing to traverse the shared graph topology structure until the termination condition is met, and outputting the final candidate node set; and returning the final candidate node set and its corresponding distance values ​​to the query executor layer of the database system through the index scan callback interface in the custom index access method plugin, and outputting the final query result set.

The invention discloses a database construction method based on hybrid heap allocation and versioned storage, and relates to the technical field of databases. The method comprises the following steps: establishing a database file structure system comprising a data file, a metadata file, a function file, a mode file and an extension file; the method comprises the following steps: constructing a versioned storage manager adopting a hybrid heap allocation algorithm, dividing a heap memory into memory pages and memory blocks with fixed sizes, managing the free memory pages by using a free linked list, recording the allocation state of the memory blocks in a bitmap, and adding version header information for the memory blocks; establishing a version index mechanism based on a persistent B + tree; and a historical data tracking function is realized. According to the method, the advantages of the idle linked list and the bitmap are combined, the memory allocation efficiency is guaranteed, the damage resistance of the system is improved, and meanwhile, complete historical tracking of the data is achieved through a versioning storage mechanism.

The application provides a kind of time series data write-time incremental adaptive downsampling method, belong to data processing technical field, specifically include: in original time series data write-in triggers bucket division determination, maintain current open bucket and extract P5 and P95 quantile value of data in bucket, calculate fluctuation amplitude based on quantile range, determine bucket boundary by two-dimensional condition of time upper limit and fluctuation threshold, generate summary result containing first value, P5 value, P95 value, last value and its corresponding timestamp when bucket is closed and is stored, while dynamically updating fluctuation threshold using exponentially weighted moving average method according to fluctuation amplitude of closed bucket.The scheme of the application moves downsampling from query side to write-in side, directly reads summary result when querying, avoids repeated scanning of original data, and threshold can automatically evolve according to data characteristics, effectively resists transient glitch interference, and deployment requirement is low.

A target detection model training method, a target detection method, a device and equipment, relate to the technical field of computer vision algorithms. The above method comprises: obtaining a multi-scale feature map of a training image according to a target detection model to be trained, and encoding the multi-scale feature map to obtain encoded features; generating noise samples, decoding the noise samples and the encoded features based on a query vector, and outputting a plurality of candidate prediction boxes according to the obtained decoded features; the query vector is obtained according to anchor box parameters and multi-scale feature vectors of the multi-scale feature map, and the anchor box parameters include at least one learnable point coordinate generated based on the multi-scale feature vectors; all candidate prediction boxes are matched with the true value of the training image, and the loss is calculated, and then the target detection model is trained based on the loss. The present application realizes the combination of features and the positions reflecting the features, and increases key point sampling, thereby realizing better query effect.

This application relates to a method, system, and medium for querying order communication data based on order status changes. The method includes: monitoring the status of a target order, where the target order status includes an order creation status used to trigger data preloading; when an order creation status is detected, acquiring historical communication data associated with the target order and storing the historical communication data in a preloaded cache area; in response to receiving a compliant query request for the target order, merging the historical communication data with real-time acquired incremental data to obtain the target order communication data; and performing a compliant query on the target order communication data to obtain the query results. This addresses the problem of how to improve the performance of order communication data querying in related technologies.

This application belongs to the field of computer technology, specifically relating to a data query method, a data query device, a computer-readable medium, an electronic device, and a computer program product. The method includes: parsing a data query request to obtain query conditions, the query conditions including one or more of a subject, an object, and a predicate, the predicate representing the logical relationship between the subject and the object; determining a target data partition in a graph database based on the query conditions, the graph database including a first-level data partition obtained by dividing regions according to the density relationship between nodes, the first-level data partition including a second-level data partition obtained by dividing regions according to the logical relationship between nodes; and retrieving data query results matching the query conditions from the target data partition. This application can improve data query efficiency and reduce resource consumption in the data query process.

The invention discloses a data fragmentation method, device and equipment, a medium and a program product, and relates to the technical field of database management. The method comprises the steps of obtaining a current database table structure and a current service query log of each data fragment of the distributed database; according to the current database table structure and the current business query log of each data fragment, constructing a current data distribution relation graph corresponding to each data fragment; identifying the current data distribution relation graph by adopting a data distribution adjustment model, and outputting a target data fragmentation mode of the distributed database; and according to the target data fragmentation mode of the distributed database, performing data distribution adjustment on each data fragment of the distributed database. According to the technical scheme provided by the embodiment of the invention, the query efficiency and load balance of the distributed database are improved.

The application relates to the field of databases, and particularly provides a small-range out-of-order time sequence data fast query method and device, which limits the out-of-order degree, makes the time sequence data be in order from the overall trend, and uses a dichotomy search and local traversal mode to query the record at a certain time point or filter all records in a certain time period. Compared with the prior art, the application can limit the out-of-order degree of the time sequence data, use the dichotomy search and local traversal mode, make it unnecessary to traverse all records when the record at a certain time point or in a certain time period is queried, improve the overall query performance of the database, reduce the IO of the disk, and improve the use experience of the user.

This invention discloses a lightweight database statistics recommendation method, relating to the field of database technology. Addressing the shortcomings of existing database statistics collection strategies, the proposed solution includes: automatically creating the `sql_statistics` and `statistics_used` tables; loading the data from the `statistics_used` table into memory to form a statistics usage list `MapList M`; recording historical query statements in real-time to the `sql_statistics` table; collecting basic statistics, periodically determining whether the cold start phase has ended, and if so, periodically querying the system load; if the load does not exceed a preset threshold, executing corresponding processing logic to generate a temporary `MapList H`; filtering columns from the temporary `MapList H` to form a set `N`; creating statistics for all columns in set `N`, removing statistics from `MapList M` as needed, and then migrating the column names and corresponding frequencies from `N` from the temporary `MapList H` to `MapList M`, completing the update of `MapList M`, and synchronizing it to the `statistics_used` table; when the database triggers a schema change that involves statistics column names, modifying the corresponding column names in the `statistics_used` table and synchronously updating `MapList M`.

The invention provides a file merging method and device, equipment, a storage medium and a program product, and particularly relates to the technical field of data processing. The method comprises the steps that small files, stored in a distributed storage system, of a target data table are recognized, and format information of all the small files is recorded; according to the recorded format information, combining the small files through a distributed computing framework to generate a plurality of aggregation files; and writing the aggregation file into the temporary storage directory, and migrating the aggregation file to a formal storage directory of the target data table after the aggregation file passes the integrity verification. The method is used for achieving the effects that efficient small file merging is achieved, and meanwhile continuous access of streaming data is supported.

The application discloses a kind of database query method and device fusing three layer metadata atlas and historical feedback, belong to database query technical field.The method includes: constructing the metadata atlas of database;Obtain natural language query sentence, identify the business entity of natural language query sentence;Business entity is mapped to target database and table;From the table level relationship subgraph associated with target database and table in metadata atlas;According to table level relationship subgraph, determine the field connection path between field;Field connection path is converted into SQL logic component;Search similar historical query record with natural language query sentence, obtain historical SQL statement and its execution result abstract;Based on field connection path, SQL logic component, historical SQL statement and execution result abstract, generate target SQL statement;Execute target SQL statement.The application improves the accuracy of natural language to SQL query conversion by layered reasoning mechanism.

The invention discloses a fast query method and device for metadata and a storage medium, the method is applied to a driving node of a distributed computing engine, the distributed computing engine further comprises an execution node, and the driving node is in communication connection with the execution node. Determining a list file corresponding to the metadata query request; in response to the fact that the total file size of the list file is larger than a distributed analysis threshold value, task grouping processing is conducted on the list file, a plurality of query tasks are obtained, and each query task corresponds to part of the list file; each query task is sent to an execution node for task execution, target metadata returned by the execution node is obtained, and the target metadata is obtained by the execution node through analyzing a part of inquired list files according to the query tasks. According to the scheme, the query performance of the metadata can be improved.

The invention discloses a multi-protocol intelligent power distribution station equipment access management system, particularly relates to the technical field of intelligent power distribution, and comprises a protocol adaptation module, a data management module, an intelligent analysis module and a business service module. The protocol adaptation module automatically identifies and analyzes a plurality of communication protocols through a preset protocol library, and outputs equipment data streams in a unified format; the data management module receives and persistently stores the data, and establishes a queried data environment; the intelligent analysis module obtains data from a data environment for analysis based on equipment health degree evaluation, energy efficiency analysis and a load prediction model, and generates a specific operation instruction; and the business service module responds to the instruction and executes automatic services such as operation and maintenance work order distribution, reactive compensation control or power purchase plan generation and the like. According to the invention, plug and play, unified data management and intelligent analysis and decision of heterogeneous equipment are realized, and the operation efficiency, the equipment reliability and the energy efficiency management level of a power distribution station are improved.

The invention belongs to the field of high-performance storage systems and computer system structures, and particularly discloses an LSM tree reading optimization design method based on a partition ordered hash table, which is characterized in that a hash table is maintained in a memory, the table is used for caching a key read from a disk recently, and a file number and a file size corresponding to the key are recorded; when a user initiates a query request, if a target key is not hit in both the memtable and the immutable memtable, a traditional LSM tree mechanism can directly enter a disk for layer-by-layer query, but in the method, the table can be queried in a memory preferentially. According to the method, the key and the file number and the file size of the corresponding stable are maintained in real time by utilizing the hash table of the partition; whenever a new stable is generated, the system checks whether a key in the new stable is overlapped with an existing key in the hash table, and if the key in the new stable is overlapped with the existing key in the hash table, the file number and the file size of the corresponding key in the hash table are updated. In this way, the target stable can be directly positioned during query, layer-by-layer traversal is not needed, and therefore the reading speed is effectively increased.

The present application belongs to the technical field of program analysis, and relates to a high-efficiency analysis method and system for a result of a pollution analysis. The method comprises the following steps: receiving and verifying an input propagation process file and a node relationship file; constructing a mapping relationship from a source node to a target node for the node relationship file and extracting instruction numbers, generating a source node-target node mapping file and an index file according to the mapping relationship from the source node to the target node, and generating an instruction number file according to the instruction numbers; performing segmented processing on the propagation process file to obtain a segmented file and a segmented information file; and locating and querying pollution propagation information of any instruction number or index position by using the files. The present application realizes high-efficiency processing and fast querying of a large-scale pollution analysis result file, solves the problems of excessively large file size, low querying efficiency and excessive memory occupation in the prior art, and significantly improves the analysis and querying capacity for pollution analysis result data.

The invention provides a database operation statement optimization method, a medium, a product and equipment. The method comprises the following steps: acquiring a database operation statement; identifying a target parent query and a target child query which have an internal connection relationship in the database operation statement, and determining a connection key corresponding to the internal connection; judging whether the target parent query and the target child query meet a preset predicate push-up condition or not; and under the condition that the target parent query and the target child query meet a preset predicate push-up condition, pushing up a screening condition corresponding to the connection key in the target child query to the target parent query. Through the method, the data in the data table in the parent query connection operation process can be filtered in advance, so that the scanning time is shortened, and the query performance of the database is improved.

The invention provides a software package dependency relationship modeling method based on a graph structure and a storage system thereof. The software package dependency relationship modeling method comprises the following steps: analyzing metadata of a software package; constructing a multivariate node graph model, wherein the model comprises a basic package node and a version node which are used for distinguishing a package name and a specific version instance, a capability node used for uniformly expressing capability, a relation node used for processing or dependence, and an assertion node and a clause node which are used for forming a Boolean dependence expression tree; and selecting a corresponding mapping strategy according to the analyzed dependency relationship type, creating corresponding nodes and edges in a graph database, and storing the constructed graph model in the graph database. According to the method and the device, unified modeling of dependency relationships of different packet formats is realized through unified graph model design; by introducing an expression tree, complex Boolean logic can be accurately expressed; and through hierarchical node design, the multi-version storage efficiency is optimized, and efficient analysis on a large-scale software dependence network can be supported.

The application provides a database query optimization method and system based on large model enhancement, and belongs to the technical field of database management systems. The application deeply embeds a large model into key processes of database query optimization, including SQL rewriting, cardinality estimation, cost estimation and execution plan enumeration, so as to improve the understanding and optimization ability of an optimizer for complex SQL. Through deep analysis of SQL semantics, large-scale statistical pattern recognition and pruning of a complex plan search space, the application significantly improves the quality of a query plan, reduces cardinality estimation errors, reduces a plan search space and improves the overall execution performance of a database. The application can be applied to any relational database system and has universality and high practical value.

The invention discloses a data analysis and report generation system based on asynchronous processing. The data analysis and report generation system comprises the steps that a user directly inputs query information and a. Xlsx file for query on a front-end display module; the EDS interface module extracts effective information in the. Xlsx file to generate a query telemetering Map set, generates a telemetering theoretical value range Map and sends the Map to the data comparison module; the database connection module judges that the connection request is valid according to the query telemetering Map set and then sends the batch query request to the asynchronous processing module; the asynchronous processing module sends the asynchronous query request set to the database storage and query module for query, combines the result sets sent by the data comparison module and then sends the result sets to the document generation module; the database storage and query module stores the telemetry data, generates a query result set according to the asynchronous query request set and sends the query result set to the data comparison module; the data comparison module obtains a result set and sends the result set to the asynchronous processing module; the document generation module generates a telemetry query result table and a Word document.