57results about How to "Realize read-write separation" patented technology

The invention relates to a system and a method for data management. The system comprises a main server and a standby server, the main server comprises a main database which can be used for internal users to perform data read-write operations, and the standby server comprises a standby database which only allows external users to perform data read-only operation. At least one switchable environment which can be equivalent to the main database exists in the standby database for switching. By the system and the method for data management, system load can be shared, high usability can be realized, and system economy is improved.

The invention provides a dynamic switching method and device based on multiple databases. The dynamic switching method based on the multiple databases comprises the steps that a relevant configuration file is read when a computer application system is started, wherein the configuration file comprises more than one key-value pair, each key is a database identification, and each value is a method name for accessing of a corresponding database; the database identifications in the configuration file are set in a thread pool of the computer application system; the method is intercepted when the method of the computer application system is called, a database identification corresponding to the method is obtained in the thread pool, and the intercepted method is released; the method is executed by means of the obtained database identification. The dynamic switching method and device based on the multiple databases have the advantages that read-write separation based on configuration can be achieved, and method-level database switching is achieved.

The invention discloses a method for realizing separation of reading and writing of a database and a device, and belongs to the technical field of data processing. The database comprises a main base and a secondary base; the secondary base is the backup of the main base; the method includes steps of receiving a structured query language SQL access request sent by an application server; according to the SQL access request, acquiring an operation type corresponding to the SQL access request and request logo of the SQL access request; if the operation type is a writing operation type, writing a to-be-written data corresponding to the SQL access request to the main base; if the operation type is a reading operation type, confirming if the separation of reading and writing is required according to the request logo and the reading-writing separation table; if the reading-writing separation is required, reading data corresponding to the SQL access request from the secondary base, wherein the reading-writing separation table comprises a request logo of SQL access request in need of separation of reading and writing. The device comprises a receiving module, a first acquiring module, a writing module and a reading module. The method and the device can reduce the development difficulty of the application procedure.

The invention relates to an access method and system for lots of small files. The method comprises the following steps of compressing business files corresponding to obtained business data to obtain a file package; extracting a characteristic value of the business data according to a preset extraction rule; when the characteristic value is not stored in a file index database, encoding the characteristic value to obtain data coding according to a preset encoding rule, and storing a corresponding relation between the characteristic value and the data coding into the file index database; generating a corresponding directory structure according to the data coding and decompressing the file package to obtain the business files; storing the business files into a storage path of the directory structure; and reading the business files. The method and the system have the beneficial effects that lots of small files are effectively stored and read, the system access is simple, the efficiency of writing and reading mass data is high, and multidirectional elastic extension can be realized.

The embodiment of the invention discloses a read-write separation method, a device, a computer device and a storage medium. The method comprises the following steps of: receiving a data request of anapplication program through a middleware; Selecting a database for executing the data request according to a preset sub-database rule; If the data request is a write data request, writing the data into a table of a main database according to a preset table division rule; And if the data request is a read data request, reading data from a table of a database according to a preset table splitting rule to return the read data to the application program. By implementing the method of the embodiment of the invention, the read-write separation can be realized, and the high concurrent read-write efficiency of the database can be improved.

The invention provides a detection system and a detection method of advertisement clicking anomaly based on Spark Streaming, and relates to the field of computer technique application. Logs are collected when a user clicks the advertisements on a webpage, data collected in real time are cleaned, data field format is standardized, and the standardized data is transferred to the Kafka data information system by Flume, data are classified through a KNN neighborhood algorithm of Spark Streaming, and the three classes of abnormal data, suspicious data, and normal data can be obtained. The abnormal data and the normal data are stored in a database, the suspicious data are sent to the Kafka data information system, and naive Bayes classifiers are trained through the abnormal data, the classification information of the suspicious data can be obtained using the classifier, and data are saved in the database. Advertiser expenses are justly collected by the amount of normal data, in the meantime, the popularities of each advertisement are obtained by analyses, the directions for industrial developments are provided for the advertisers, and the information such as user distributions in the country is provided.

The invention discloses a method for improving data read-write performance based on a mysql database cluster. The method comprises the following steps: establishing a mysql database cluster system; realizing read-write separation and library division functions of the mysql database cluster system by utilizing mycat intermediate; using MYISAM as a storage engine of a table; when a new table is created, implementing a mysql partitioning strategy; and performing data caching by using a redis database. A set of large mysql database cluster system is designed by utilizing mycat + mysql, landslide and debris flow real-time monitoring data can be quickly written into a database, and Beidou high-precision positioning data is stored by using an MYISAM data table; the data storage reliability can beimproved by utilizing a multi-master and multi-slave structure, and the query efficiency is further improved by using mysql partitions; a keyword index can be used for increasing the retrieval speed,and redis cache is used for improving the reading performance; the functions of read-write separation and library division can be achieved, and the data read-write speed is effectively increased. Onthe whole, the read-write performance of the system is not reduced along with the increasing of the data volume.

The invention discloses a multi-library separation method based on business data intelligent analysis. The method comprises the following steps of obtaining an access request of the front-end businessdata, performing semantic analysis on the access request, analyzing the access object condition, generating an access strategy, executing the corresponding access strategy according to the type of the statement, recording the statement access object information, and generating an access frequency table, regularly and intelligently analyzing an object in the access frequency table to generate a migration strategy, analyzing the service data to determine an object needing to be migrated in the access frequency table, migrating the determined object needing to be migrated into a corresponding database, and after the data migration is completed, updating a data access strategy, and performing data cleaning on the database before the migration. The method classifies and intelligently analyzesthe data accessed by the business, can automatically analyze and detect the access frequency and the access content of the business processing database object, completes the automatic migration and cleaning of the object, and achieves the physical separation and read-write separation of the data of multiple databases.

The invention provides a method for realizing database dynamic switching, which comprises the following steps of analyzing a data source configuration file to form a corresponding database connectionpool; defining a set of annotations of data source switching; analyzing the annotation, and placing the database name in a local thread copy; returning the database object corresponding to the database name in the thread copy to the Spring framework to realize dynamic switching of the database. The invention further provides a computer readable storage medium, high cohesion and low coupling can beachieved without other service control codes, and the performance is not affected. On the basis, database read-write separation can be realized through configuration.

An unreliable environment-oriented audit log read-write managing method and system. The method comprises a writing-in step, in which audit logs produced during an audit procedure by an audit device are periodically backed up to a log file related to time and in a hard disk, a read step, in which relative files are extracted according to user selected time, and importing the extracted log file to a database for check use. The audit logs are written into the log file, so even when power is off during the writing, only the last write-in log is affected; therefore system damage probability during sudden power-off can be reduced; when a user checks a log, the log file can be extracted according to user selected time and imported to the database; during the log file explicit importing process, the device power supply is not turned off due to the awareness of the import process, so possibility of errors during the importing of the log to the database can be greatly reduced; and further, during the check, a database of column storage type is employed, so log writing-in speed can be improved.

The invention aims to provide an audit log data processing method and system, and the method comprises the steps: building a temporary table, and enabling the writing operation of the temporary tableto be executed by a MyISAM engine; after auditing log data is collected, only storing the auditing log data in a temporary table, and only storing the data of the first duration in the temporary table; setting a data cleaning strategy of a second duration by the temporary table; adding a trigger to the temporary table, and executing synchronization operation of the temporary table and the originaltable regularly; and cancelling statistics of all operations of the temporary table. According to the technical scheme of combining the Esper engine and the Memcached, read-write separation of the original table is indirectly achieved, the data query efficiency is improved, and meanwhile data statistics is improved.

The invention discloses a method and a device for reading and writing separation of a database. One embodiment of the method comprises the steps that a business request of a user is acquired through a web access layer; a business method of the business request is named according to a business type of the business request; the business method is intercepted through a pre-declared aspect oriented programming interception class; and according to an interception result of the aspect oriented programming interception class, the business request is linked to a master base of the database for corresponding writing operation, or the business request is linked to a slave base of the database for corresponding reading operation. Through the embodiment, reading and writing separation of the database is realized.

The invention discloses a database capacity expansion method and system and relates to the field of databases. The method includes: acquiring a capacity expansion identity; monitoring the database according to the capacity expansion identity, and performing dynamic smooth capacity expansion on the database. The database is monitored through the capacity expansion identity, dynamic and smooth capacity expansion is performed on the database according to the capacity expansion identity, and capacity expansion is performed on a storage table itself in the database, so that the problems of hotspotimbalance and difficulty in migration of capacity expansion data in the process of capacity expansion of the database can be solved.

The invention provides a data processing method and device, electronic equipment and a storage medium. According to the data processing method provided by the invention, firstly, a query request is obtained, and the query request comprises a query demand of a user for service data; and then service data generated in the operation of the e-commerce system from a preset database are obtained througha real-time data transmission module, storing the obtained service data to a query database different from the preset database, and determining a data processing result according to a query demand and the query database. Therefore, read-write separation of the service data is realized, the corresponding data meeting the query requirement can be determined quickly in real time according to the service data, the data processing pressure of the preset database is greatly reduced, the self operation and maintenance of the e-commerce system are facilitated, and the use experience of the e-commercesystem is further improved.

The invention provides a big data management disaster tolerance method for realizing reading and writing separation based on a partition table technology. The big data management disaster tolerance method comprises the following steps: S1, establishing a corresponding partition table according to a range partition, a Hash partition and a composite partition; S2, creating an index for the partitiontable established in the step S1; S3, partitioning a database into a primary database and a slave database; S4, processing a transactional operation by the primary database, and processing a SELECT query operation by the salve database; and S5, synchronizing the slave database and the primary database. Through reading and writing operation, a plurality of libraries coexist, so that the disaster occurrence is avoided effectively, and the data stability and security are improved. On the aspect of data performance, I/O operation is balanced; the physical disk space is allocated reasonably; and the big data query performance is improved greatly. On the other hand, data maintenance is facilitated, and the data availability is enhanced.

The invention discloses a distributed unstructured database association query method based on big data, and relates to the technical field of data processing. The method comprises the following stepsof: S1, generating data tables, and designing data table instances associated with multiple tables by adopting mapping in combination with service requirements so as to obtain a plurality of data tables; s2, monitoring a data table; s3, data synchronization; s4, ETL processing and data aggregation: cleaning and preprocessing the data subjected to data synchronization in the step S3 through a kafkacluster, and storing the aggregated data into an unstructured database in a landing manner; s5, selection of a fragmentation algorithm: when the data reaches the Tb-level data volume, designing the fragmentation algorithm in combination with a query scene to realize distributed mass storage of the data; the method has the advantages that read-write separation of the structured database and the unstructured database can be achieved, and the mass data storage problem is solved.

The invention relates to a load balancing system for database servers.The load balancing system comprises a server cluster composed of the database servers; one database server in the server cluster is selected as a data reading server, and the other database servers in the server cluster serve as data writing servers; databases needing to be processed are partitioned, obtained corresponding database partitions are stored in the corresponding data writing servers, and data to be written is written into the database partitions of the corresponding data writing servers through the data reading server; after the data to be written is written into the database partitions of the corresponding data writing servers, all the database servers in the server cluster are synchronized in real time, so that all the database servers in the server cluster store the databases with the same data content.The load balancing system for the database servers is compact in structure, capable of effectively reducing the load of the database servers, increasing the handling capacity of the database servers and improving the operating efficiency, safe and reliable.

The invention provides a data acquisition system and method. According to the data acquisition system and method, data acquisition and configuration are achieved in a BS mode, a user can access data under the condition that the user communicates with a server network, the problem of independent faults is avoided, read-write separation is achieved, and the pressure of a main node is reduced. Meanwhile, the system also supports the acquisition of universal protocols and various statistical reports, and can meet the demands of more customers.

The invention discloses a data storage and query method based on a distributed database, and the method comprises the following steps: S1, setting a connected data source through a client, and determining the data type of to-be-processed data; S2, determining that the data is core business data, and storing the core business data in an ORACLE database; s3, confirming that the data is log data, andasynchronously storing the log data into a MySQL database at the rear end of the database middleware through a message queue; s4, storing the database middleware the data to fragmentation nodes of aMySQL database main library; and S5, enabling the MySQL database main library to regularly extract the data of each fragment node and synchronizes the data to a historical database. The message queueasynchronously lands the log service data into the log data center library, so that the data volume of the core transaction library is reduced, the performance of the core service library is guaranteed, and the overall operation efficiency of core services is improved. Meanwhile, the query and storage problems of log historical data are efficiently solved.

The invention relates to the field of data processing, and discloses a data query method based on a database cluster, which comprises the following steps of: associating at least one proxy server for each database in the database cluster, and defining a master proxy server and a slave proxy server in the proxy servers; receiving a data query task transmitted by the client, and fragmenting the data query task to obtain a plurality of fragmented query tasks; transmitting the plurality of fragmented query tasks to the master proxy server, and receiving a distribution result of the slave proxy server of each fragmented query task returned by the master proxy server; and according to the distribution result, sending a data query instruction of the fragment query task to the slave proxy server, receiving a data query result obtained by performing data query on a database of the database cluster by the slave proxy server according to the data query instruction, and returning the data query result to the client. In addition, the invention also relates to a block chain technology, and the data query result can be stored in a block chain. According to the invention, the data query efficiency can be improved.

The invention discloses a cache read-write method and a terminal. A read-write cache is arranged between a read-only cache and a common cache; in a reading scene, the read-write cache is inquired whenthe read-only cache does not inquire the corresponding data, the common cache is inquired when the read-write cache does not inquire the corresponding data, and the read-only cache and the read-writecache are respectively updated according to the data inquired in the common cache; and in a writing scene, the read-write cache is emptied after the common cache is updated successfully. The read-write cache is used as middleware; after the common cache is updated successfully, the read-write cache is firstly emptied, the read-write pressure under high concurrency can be effectively relieved whenthe read-write operation occurs at the same time, the method is different from the prior art that read-write waiting is caused due to the fact that data read-write is locked, read-write separation ofcache data can be achieved, and cache data read-write under a high concurrency scene is supported.

The invention discloses a data storage method and computer equipment, and the data storage method comprises the steps: deploying a data pool of a storage system, marking a to-be-stored area of a first recording disk as a write data pool, and marking a data area of a second recording disk as a read data pool; writing real-time data into the first recording disk; in response to completion of writing of the real-time data, modifying data pool deployment of the storage system, integrally marking the first recording disk as a read data pool, marking a to-be-stored area of the second recording disk as a write data pool, and canceling a read data pool label of a data area of the second recording disk; reading real-time data of the first recording disk, and writing the real-time data into the second recording disk so as to synchronize the data of the first recording disk and the data of the second recording disk; in this way, read-write separation of the mechanical disk can be achieved.