476results about How to "Improve query performance" patented technology

A system and methodology for parallel query optimization using semantic-based partitioning is described. In one embodiment, for example, in a database system comprising a database storing data in database tables, a method is described for improving query performance by dynamically partitioning the data, the method comprises steps of: receiving a query requesting data from the database; generating a plurality of subplans for executing the query, each subplan including one or more operators for performing relational operations; adding operators for partitioning data and performing a given relational operation in parallel to at least some of the plurality of subplans; and building a plan for execution of the query based, at least in part, upon selecting subplans having favorable execution costs

Cached information that includes multiple network addresses is delivered in a manner that changes the order of the network addresses each time the cached information is delivered to a client, thereby achieving load balancing in a manner that reduces network traffic and improves system performance. In the preferred embodiments, the user defines a minimum time to live (TTL) and one or more order change criteria. When a query is made to a DNS, the DNS returns an answer that includes a time to live (TTL). This answer may be stored in a cache, and may include multiple network addresses. If the TTL for the answer received from a DNS is less that the minimum TTL, the TTL for the answer is set to the minimum TTL before storing the answer in the cache. When a query may be satisfied by a cached answer that includes multiple network addresses, a cache mechanism delivers the multiple cached network addresses in an order determined by the one or more order change criteria if the TTL for the cached results is a positive number. In this manner load balancing can be achieved while still benefitting from the performance enhancements of caching DNS query results.

The invention discloses a method and a system for improving the large data volume query performance and belongs to the technical field of large data volume query. The method comprises A, loading data in a disk database into distributed caches in a cache ID-entity data key value pair mode, and storing the cache ID and key information of the entity data in a cache ID table of a memory database simultaneously; B, querying the cache ID table according to a query request when the query request sent by a client is obtained to selecting an ID set meeting the query request; C, obtaining the entity data from corresponding distributed caches according to the cache ID set and returning the entity data to the client. By means o the system and the method, loads of the disk database can be effectively reduced, and the big data query performance is improved.

The invention pertains to a system and method for dispatching and executing the compute-intensive parts of the workflow for database queries on an attached high-performance, parallel computing platform. The performance overhead for moving the required data and results between the database platform and the high-performance computing platform where the workload is executed is amortized in several ways, for example,

• • by exploiting the fine-grained parallelism and superior hardware performance on the parallel computing platform for speeding up compute-intensive calculations,
  • by using in-memory data structures on the parallel computing platform to cache data sets between a sequence of time-lagged queries on the same data, so that these queries can be processed without further data transfer overheads,
  • by replicating data within the parallel computing platform so that multiple independent queries on the same target data set can be simultaneously processed using independent parallel partitions of the high-performance computing platform.

A specific embodiment of this invention was used for deploying a bio-informatics application involving gene and protein sequence matching using the Smith-Waterman algorithm on a database system connected via an Ethernet local area network to a parallel supercomputer.

A system and method for analyzing predicate usage and proposing solutions to improve query performance. The system comprises a system for collecting performance statistics for database queries; an aggregation system that organizes the performance statistics based on predicate combinations used in the database queries;. and an evaluation system that compares the performances of queries having similar predicate combinations. Further features include a problem identification system that identifies a predicate, which when added to a query, results in a low performing query, and a solution proposal system that automatically proposes a solution to address the low performing query.

The invention relates to a method and a device for realizing a column storage based relational database. The method comprises: establishing data files, and serially numbering the data blocks which form the data files; defining a table segment; inserting records into the table segment; generating the only record identification number in the table segment for the record inserted in the table segment, and separating the records by columns; for each column in the records, executing the following operation: storing column values and the record identification numbers as value data in the data blocks and sorting according to the size of the column values; storing the record identification numbers and the serial numbers of the data blocks for storing the value data which are taken as connecting data in new data blocks, and sorting according to the size of the record identification numbers; and establishing indexes for the data blocks for storing the value data and the data blocks for storing the connecting data, and generating index data blocks. The embodiment of the invention improves the query performance of the database.

An information retrieval system implements a search language, through which a querying entity (e.g., a user, a program or process, or the like) formulates a search query. Preferably, a search query is composed of an ordered set of clause definitions, and each clause can have set membership operations applied to it. Each clause includes a clause pipeline, and a time constraint. A clause pipeline includes an ordered set of clause specifications separated by a pipeline operator. A clause specification can be either an expansion operation or a filtering operation. Preferably, a first clause specification in a pipeline operates on an initial universe of all objects, and each subsequent clause specification operates on a set of objects produced from the previous clause specification. The search language is exposed to users (typically, IT administrators), and one or more builder programs within the system (each referred to as a “model builder”) are used internally to present data models to the search language. A model builder extracts data from a given type of data source (including, without limitation, a relational database system, an application programming interface (API), or the like), and enables that data to be presented to one or more constructs of the search language according to a single unified data model.

The invention discloses a system and method for task processing. The system comprises a request receiving module, a task splitting module, a task processing module, a result integration module and a result returning module. According to the characteristic that historical data of a real-time database are managed through a non-shared mechanism in the prior art, a shared management mechanism is adopted in the system. A query task requested by a user is split into a plurality of sub-tasks by various functional modules of the system on the basis of a preset splitting strategy. Parallel processing is conducted on the sub-tasks on the basis of a data source shared mechanism. When the method is specifically implemented, a plurality of database servers can be arranged to serve as equal nodes to be connected to the real-time database, so that a data access right of the real-time database is shared. Parallel access is conducted on the real-time database, so that the sub-tasks are processed in parallel. Compared with an existing task processing method based on the non-shared management mechanism, the inquiry performance of the real-time database is greatly improved through the system and method for task processing.

The invention discloses an extendible repeated data detection method, belongs to the technical field of computer storage, and solves the problem that in the existing repeated data detecting method, the storage capacity cannot be efficiently extended, so as to meet the requirements of the current situation that the storage demand increases and repeatedly deleted systems need upgrading and updating. The extendible repeated data detection method comprises the following steps: partitioning processing, fingerprint extraction, retrieving of Bloom filters, retrieving of fingerprint subset table, judgment of unfulfilled Bloom filters, new fingerprint marking, judgment of Bloom filter quantity, and extending of Bloom filter array. In the invention, the Bloom filter array is used to retrieve the fingerprint data, so as to quickly locate the retrieval range, improve the retrieval efficiency and realize detection on the repeated data; the extendible repeated data detection method is high in expansibility and querying performance, can support element location and control the misjudgment rate, and further can effectively reduce the memory overhead. The Bloom filter array is composed of a series of isomorphic Bloom filters, so that once the misjudgment rate epsilon' and the pre-established retrieving fingerprint total quantity nmax are provided, the quantity of the required Bloom filters and the number of the hush functions can be worked out.

The invention discloses a method and device for inquiring data. The method comprises the steps that an inquiry statement is decomposed into one or more inquiry clauses; traversal is carried out on the inquiry clauses, the inquiry clauses are matched with metadata, and the inquiry clauses are replaced with execution results stored in the metadata; otherwise, the inquiry clauses are executed, and the metadata are updated; the final execution result is used for main inquiry to obtain an inquiry result. The executed inquiry clauses and the execution results of the executed inquiry clauses are used as the metadata to be managed, the identical inquiry clauses in the current session or different sessions are directly replaced with the execution results stored in the metadata, the inquiry clauses are prevented from being executed repeatedly, and therefore the inquiry performance of the whole system is improved to a greater extent. Meanwhile, compared with optimization of the existing inquiry clause level, the inquiry optimization granularity is thinner, the method and device are more suitable for inquiring large data in data warehouse application, and therefore the inquiry performance of the whole system is further improved.

The invention discloses a mass data querying method, belonging to the information technology field. The method comprises: a) dividing the mass data entity table according to the given time spans into sub-areas; b) establishing a two-dimensional statistical table for attribute names in the entity table, wherein one dimension represents all the given time spans in the time interval, the other dimension represents the attribute data in the attribute column of the attribute names, and the contents in the statistical table represent the name of entity table of certain attribute data existing in certain time span; c) during querying, acquiring a collection of the entity table sub-areas in the time interval according to the statistical table for the attribute name and the time interval given in the querying condition, if the querying condition comprises the statistically treated attribute name; and d) narrowing the mass data scope according to the collection and then carrying out query. The method of the invention can be used for mass data query in large-scale systems such as charging query systems of telecommunications industry, large website transaction platform and the like.

The embodiment of the invention discloses a mass data based query method and device. The mass data based query method includes: receiving an SQL which is input by a user via a system query page; performing logic analysis and physical analysis on the SQL; converting a query field required by user query into a code defined in metadata, and acquiring a query result according to a table, a storage type, and a storage physical surface in the SQL; and gathering the query results, converting the query result into a preset format, and displaying the query results in the query page. A semantic analysis technique based mass data query system can solve the problem that SQL query cannot supported while big data storage techniques such as haoop and solr are used; and a data pre-fetching technique and a Hash query technique are adopted in a system design, the query performance of the mass data query system can be optimized, and the query real-time performance of mass data can be improved.

The invention provides a mobile object continuous k-nearest neighbor (CKNN) query method based on a road based road networks tree (RRN-Tree) in a road network and relates to a data query method. The mobile object CKNN query method aims to solve the problem that in the prior art, an index structure is utilized to index the road network, the road network is modeled to a directed/ undirected graph, and a nearest neighbor query request is processed based on a memory data structure; however, when the road network has a large data size and multiple rod segments, the query efficiency is decreased rapidly; furthermore, the modeling mode based on the graph cannot reflect the steering relation of a mobile object at the crossing, and the nearest neighbor query of the complex road network with crossing steering and U-shaped turning constraints cannot achieved. According to the mobile object CKNN query method, an RRN-Tree index structure is provided to index the road network and interesting point objects, an adjacency linked list is established for crossed points on paths in the index structure, the connection relations between road segments at the crossing are stored, and therefore, the CKNN query of the road network under complex constrain conditions is completed. The mobile object CKNN query method is used for inquiring CKNN query of the road network.

Method and system for providing results for task list queries to Workflow Management Systems (WfMSs) and Human Task Management Systems (HTMSs). In one aspect, a method for providing the result of a task list query to a human task database includes caching data from the database in a plurality of materialized views. A new request for a list of tasks from the database is received, and a new query is created when the new request applies to the data in a particular materialized view. When the particular materialized view includes data that requires updating from more recent data in the database, the materialized view is so updated. The new query is executed against the data in the materialized view instead of executing the new query against the data in the database.

The invention discloses an image retrieval system integrating classification with hash partitioning, which comprises a downloading module, a classification module training module, an image classification module, a characteristic extraction module, a recording table building module, a partitioning module, a request processing module, a retrieval module, a similarity acquiring module and a result returning module. The downloading module is used for downloading images so as to build an image library, the classification model training module classifies the images in the image library according tothe shapes and selects representative sample images from the image library to form a sample library firstly, then extracts characteristic descriptors of the classification bottom layer of all images in the sample library and performs trainings on the characteristic descriptors of the classification bottom layer by a support vector machine so as to obtain discriminant of each classification. Classification models are formed according to the discriminants of all the classifications. Precision ratio of the image retrieval system is increased, the problem of low recall ratio during classificationmistakes is overcome, and the retrieval speed of the image retrieval system is increased integrally.

The invention discloses a parallel query method for a distributed database. In the method, due to the introduction of the design of proxy and qcs modules, a query system is able to support a high-concurrency distributed database and meets requirements of a large-sized application system. Meanwhile, based on a concept of 'dividing and ruling' query tasks, the qcs module initiates query concurrently, granularity data which can be queried quickly are fed back first, and thus, the overall query performance of the system is improved obviously and the needs for quick query of a majority of application systems are met.

The invention discloses a method and system for extracting a knowledge graph from software project data and answering the knowledge graph. The method comprises the following steps that for each type of software project data in a software project database, entities and an incidence relation between entities are extracted from the type of the software project data to be stored in a corresponding graph database; based on a traceability correlation technology of software data, correlation processing is carried out on the data in each graph database, and the correlation between entities of different types of software project data is obtained; and a corresponding edge is added into each graph database according to the incidence relation between entities of different types of software project data, and the entities with different sources are connected to generate the knowledge graph of the software project data; for the input natural language query statement, a matched communication sub-graphis inquired from the knowledge graph to serve as an answer. By virtue of the method and system, the problems of data correlation deletion, serious information isolation and difficulty in connection query and analysis of the software project are solved.

The invention relates to a processing method and a device of tree type data. The method includes obtaining a father node and child nodes of the tree type data; displaying the structure of the tree type data in a set mode according to the subjection relations between the father node and the child nodes and between the child nodes; and starting from one side of the boundary of the set at the outermost layer to number boundary points of each set sequentially, wherein the numbers at two sides of the boundary of each set are used for limiting the range of the set. According to the method and the device, the problem of insufficient structural management of the tree type data in the prior art is solved, the management of the tree type data is more efficient, the query performance is higher, simultaneously, limit of the tree type data hierarchy is eliminated, and the method and the device are particularly suitable for the management of mass tree type data such as a family tree, a gene spectrum and a bill of material (BOM).

The invention discloses a distributed device log collection method. According to the method, an integrated data intermediate layer is built through a distributed log processing framework in a mediator mode, integrated data intermediate management service is formed, the data intermediate management service collects device logs, the logs are stored on each distributed storage point in a distributed way, in addition, data collection is carried out, and if the distributed storage points need to be increased, the goal is achieved by adopting a distributed storage point dynamic expansion mechanism. The distributed device log collection method has the advantages that the integrated data intermediate layer is built by adopting the mediator mode, the logs are subjected to collection and formatting processing in a unified way, and the distributed storage points are uniformly managed and dispatched in a centralized way; the sub table structure is adopted, the multithreading processing advantages are better realized, a particular sub table indexing mechanism is built, and forms a super-volume data grading indexing system together with MariaDb database indexes of each data storage point, meanwhile, the performance advantage of a distributed server is utilized, and the storage and query performance of log data is greatly improved.