Database inquiry machine

Abstract

The invention discloses a database query machine which comprises a data layer unit, a service layer unit, a communication layer unit and a presentation layer unit, the data layer unit is connected with the service layer unit, the service layer unit is connected with the communication layer unit, and the communication layer unit is connected with the presentation layer unit. The database query system is standard in formulation, unified in management and focused on a unified platform, efficiency is improved, management is convenient, risks are controlled, log records are queried, tracing is facilitated, operations such as probing and alarming, slave library matching and ddl, update and deelete forbidding are achieved, the system is safer, the complete database query system is constructed, research and development work efficiency can be improved, management is convenient, and the research and development efficiency is improved. Meanwhile, the safety of the database is guaranteed, and data query is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of databases, and in particular to a database query machine. Background Art

[0002] Database is a warehouse that organizes, stores and manages data according to data structure. It was created more than 60 years ago. With the development of information technology and market, especially after the 1990s, data management is no longer just about storing and managing data, but has turned into various data management methods required by users. There are many types of databases, from the simplest tables that store various data to large database systems that can store massive amounts of data, which have been widely used in various aspects.

[0003] In the information society, fully and effectively managing and utilizing various information resources is a prerequisite for scientific research and decision-making management. Database technology is the core part of various information systems such as management information systems, office automation systems, and decision support systems, and is an important technical means for scientific research and decision-making management.

[0004] With the development and expansion of the Internet, network traffic has also increased, and enterprises have higher security awareness requirements for R&D personnel. In this case, data security is particularly important. Once a data leak occurs, it will cause serious losses to the company. How to query and export data safely is still a relatively troublesome matter.

[0005] Currently, there is no unified standard or tool in the industry that can be safely used as a database query machine. The existing database query machines are not standardized and there is no statistical query tool. Each R&D department uses its own bastion machine, which is difficult to manage. The risk is high. There is no log record for query. Some R&D departments even query the main database, resulting in slow query and no traceability. The permissions are too large, and the configured main database has update and delete operations, which is extremely dangerous and easy to damage data. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a database query machine in order to overcome the defects of the database query machine in the prior art, such as non-standardization, high risk and excessive authority.

[0007] The present invention solves the above technical problems through the following technical solutions:

[0008] The present invention provides a database query machine, comprising a data layer unit, a service layer unit, a communication layer unit and a presentation layer unit, wherein the data layer unit is connected to the service layer unit, the service layer unit is connected to the communication layer unit, and the communication layer unit is connected to the presentation layer unit.

[0009] In this technical solution, a complete database query system is constructed, which can improve the efficiency of R&D work and facilitate management, while ensuring the security of the database and facilitating data query.

[0010] Preferably, the data layer unit includes a momgodb module, a mysql module, a sqlserver module, a redis module and an elasticsearch module.

[0011] In this technical solution, the types of data in the data layer unit are explained.

[0012] Preferably, the service layer unit includes a monitoring and protection module, a business cluster module and a governance configuration module.

[0013] Preferably, the communication layer unit includes a netty module, a socket module and an http\https module.

[0014] Preferably, the presentation layer unit includes a web module.

[0015] In this technical solution, web includes html, html5 and vue.

[0016] Preferably, the monitoring and protection module includes a skywalking link tracking module, a gateway cluster monitoring module and a database detection and monitoring module.

[0017] In this technical solution, the monitoring and protection module is used to monitor the performance and security of the database of the database query machine.

[0018] Preferably, the business cluster module includes a query machine module, an execution log module, an online data review and export module and a collection module.

[0019] In this technical solution, the collection module can be used to collect commonly used commands.

[0020] Preferably, the governance configuration module includes an input module, a modification module and an export module.

[0021] In this technical solution, the entry module is used for database entry, the modification module is used for database connection modification, and the export module is used for data export, but data export requires leadership approval.

[0022] Preferably, the web module includes html, html5 and vue.

[0023] Preferably, the query machine module (111) includes a mysql query machine, a redis query machine, a mongo query machine and an es query machine.

[0024] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present invention.

[0025] The positive and progressive effects of the present invention are:

[0026] The database query system of the present invention formulates specifications, unified management, focuses on a unified platform, improves efficiency, facilitates management, controls risks, has query log records, is easy to trace, and has detection and alarm, slave library, disables DDL, update, delete and other operations, is safer, and builds a complete database query system, which can improve R&D work efficiency and facilitate management, while ensuring the security of the database and facilitating data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a schematic diagram of the overall structure of a database query machine according to an embodiment of the present invention.

[0028] Description of Reference Numerals

[0029] 1. Data layer unit;

[0030] 2. Service layer unit;

[0031] 3. Communication layer unit;

[0032] 4. Presentation layer unit;

[0033] 5. momgodb module;

[0034] 6. mysql module;

[0035] 7. sqlserver module;

[0036] 8. redis module;

[0037] 9. elasticsearch module;

[0038] 10. Monitoring and protection module; 101. Skywalking link tracking module; 102. Gateway cluster monitoring module; 103. Database detection and monitoring module;

[0039] 11. Business cluster module; 111. Query machine module; 112. Execution log module; 113. Online data review and export module; 114. Collection module;

[0040] 12. Governance configuration module; 121. Input module; 122. Modify module; 123. Export module;

[0041] 14. netty module;

[0042] 15. Socket module;

[0043] 16. http\https module;

[0044] 17. Web module. DETAILED DESCRIPTION

[0045] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0046] Figure 1 The structure diagram of the embodiment of the database query machine of the present invention is shown. The database query machine includes a data layer unit 1, a service layer unit 2, a communication layer unit 3 and a presentation layer unit 4, wherein the data layer unit 1 is connected to the service layer unit 2, the service layer unit 2 is connected to the communication layer unit 3, and the communication layer unit 3 is connected to the presentation layer unit 4.

[0047] In this technical solution, a complete database query system is constructed, which can improve the efficiency of R&D work and facilitate management, while ensuring the security of the database and facilitating data query.

[0048] The data layer unit 1 includes a momgodb module 5, a mysql module 6, a sqlserver module 7, a redis module 8 and an elasticsearch module 9.

[0049] In this technical solution, the type of data in the data layer unit 1 is explained.

[0050] Mongodb in momgodb module 5 is a database based on distributed file storage, written in C++ language. It aims to provide scalable and high-performance data storage solutions for WEB applications. Its biggest feature is that it supports a very powerful query language, and its syntax is somewhat similar to an object-oriented query language. It can almost realize most of the functions of single-table queries in relational databases, and also supports indexing of data. Mongodb is characterized by high performance, easy deployment, easy use, and very convenient data storage.

[0051] MySQL module 6 MySQL is a relational database management system that stores data in different tables instead of putting all data in a large warehouse, which increases speed and flexibility. The SQL language used by MySQL is the most commonly used standardized language for accessing databases.

[0052] SQL Server Module 7 SQL Server is a relational database management system that is easy to use, scalable, and highly integrated with related software. It can be used across a variety of platforms, from laptops running Microsoft Windows 98 to large multi-processor servers running Microsoft Windows 2 Management Configuration Module 12. It is a comprehensive database platform that provides enterprise-level data management using integrated business intelligence (BI) tools. The SQL Server database engine provides more secure and reliable storage for relational and structured data, allowing users to build and manage high-availability and high-performance data applications for business.

[0053] Redis in Redis module 8 is an open source log-type, Key-Value database written in ANSI C language, supporting the network, memory-based and persistent, and provides APIs in multiple languages. Redis is a key-value storage system. Similar to Memcached, it supports relatively more value types for storage, including string, list, set and zset. These data types support push / pop, add / remove, intersection, union and difference, and richer operations, and these operations are atomic. On this basis, Redis supports various different sorting methods. Like Memcached, in order to ensure efficiency, data is cached in memory. The difference is that Redis will periodically write updated data to disk or write modification operations to additional record files, and on this basis, master-slave synchronization is realized. Redis is a high-performance key-value database. The emergence of Redis has largely compensated for the shortcomings of key-value storage such as Memcached, and can play a good supplementary role to relational databases in some occasions. It provides Python, Ruby, Erlang, and PHP clients, which are very easy to use.

[0054] Elasticsearch in elasticsearch module 9 is a search server based on Lucene. It provides a distributed multi-user full-text search engine based on a RESTful web interface. Elasticsearch is developed in Java and released as open source under the terms of the Apache license. It is currently a popular enterprise-level search engine. Designed for cloud computing, it can achieve real-time search, is stable, reliable, fast, and easy to install and use.

[0055] The service layer unit 2 includes a monitoring and protection module 10 , a business cluster module 11 and a governance configuration module 12 .

[0056] The communication layer unit 3 includes a netty module 14 , a socket module 15 and an http\https module 16 .

[0057] Netty in the netty module 14 is a java open source framework, which is now an independent project on Github. Netty provides asynchronous, event-driven network application frameworks and tools for quickly developing high-performance, high-reliability network servers and client programs. Netty is a client and server-side programming framework based on NIO. Using Netty can ensure that you can quickly and easily develop a network application, such as a client and server application that implements a certain protocol. Netty is equivalent to simplifying and streamlining the programming and development process of network applications, such as: socket service development based on TCP and UDP. Netty is a project that absorbs the implementation experience of multiple protocols (including various binary text protocols such as FTP, SMTP, HTTP, etc.) and has been quite carefully designed. In the end, Netty successfully found a way to ensure the performance, stability and scalability of its applications while ensuring ease of development.

[0058] The socket in the socket module 15 is a two-way communication connection between two programs on the network to exchange data. One end of this connection is called a socket. The essence of socket is a programming interface (API). TCP / IP also provides an interface for programmers to use for network development. This is the Socket programming interface; HTTP is a car that provides a specific form of encapsulating or displaying data; Socket is an engine that provides the ability of network communication.

[0059] In the http\https module 16, http stands for Hypertext Transfer Protocol, which is a transmission protocol for transmitting hypertext from a WWW server to a local browser. It can make the browser more efficient and reduce network transmission. It not only ensures that the computer correctly and quickly transmits the hypertext document, but also determines which part of the transmitted document and which part of the content is displayed first, such as text before graphics.

[0060] https is an HTTP channel with security as its goal. In simple terms, it is a secure version of HTTP, that is, an SSL layer is added to HTTP. The security foundation of HTTPS is SSL, so the details of encryption require SSL. It is a URI scheme (abstract identifier system) with a syntax similar to the http: system. It is used for secure HTTP data transmission.

[0061] Communication layer unit 3 is used to check the client connection IP of the database.

[0062] The presentation layer unit 4 includes a web module 17 .

[0063] In this technical solution, web includes html, html5 and vue.

[0064] Html is a hypertext markup language, an application under the standard generalized markup language. "Hypertext" means that the page can contain pictures, links, and even non-text elements such as music and programs.

[0065] HTML5 is a language description method for building web content. HTML5 is the next generation standard of the Internet. It is a language method for building and presenting Internet content. It is considered one of the core technologies of the Internet. HTML5 is the specification of HTML, the core language of the Web. When users browse web pages by any means, the content they see is originally in HTML format. In the browser, it is converted into recognizable information through some technical processing.

[0066] Vue is an open source front-end JavaScript framework with powerful data-driven capabilities and a flexible template system that enables developers to quickly build complex single-page applications (SPAs). Vue also allows developers to extend application functionality by defining custom methods.

[0067] The monitoring and protection module 10 includes a skywalking link tracking module 101 , a gateway cluster monitoring module 102 and a database detection and monitoring module 103 .

[0068] In this technical solution, the monitoring and protection module 10 is used to monitor the performance and security of the database of the database query machine.

[0069] The skywalking in the skywalking link tracking module 101 is a link tracking analysis system, which requires a data source, which can be reported by various services in the distributed system as nodes, or by various functions inside the service. It has very complete support for JAVA, with point tracking, no intrusion, and automatically captures common indicators such as http, rpc, mysql, redis, etc. For other languages, the compatibility may not be so good, and manual tracking is required, which is somewhat intrusive.

[0070] The gateway in the gateway cluster monitoring module 102 is a gateway that can be used for wide area network interconnection or local area network interconnection. It is a computer system or device that performs the important task of conversion. It is used between two systems with different communication protocols, data formats or languages, or even completely different architectures. The gateway is a translator. Unlike a bridge that simply transmits information, the gateway repackages the received information to meet the needs of the destination system.

[0071] The business cluster module 11 includes a query machine module 111 , an execution log module 112 , an online data review and export block 113 and a collection module 114 .

[0072] In this technical solution, the collection module 114 can be used to collect frequently used commands.

[0073] The governance configuration module 12 includes an input module 121 , a modification module 122 and an export module 123 .

[0074] In this technical solution, the input module 121 is used for database input, the modification module 122 is used for database connection modification, and the export module 123 is used for data export, but data export requires approval by the leader.

[0075] The web module 17 includes html, html5 and vue.

[0076] The query machine module 111 includes a mysql query machine, a redis query machine, a mongo query machine and an es query machine.

[0077] In the present invention, no matter whether the system adopts a monolithic application architecture or a microservice application architecture, a distributed cluster deployment method needs to be adopted when deploying the application.

[0078] Hot deployment mechanism

[0079] The delivered applications can be hot-deployed, and most common maintenance operations such as software package version upgrades and simple middleware maintenance do not require downtime. A detailed hot deployment plan and error rollback plan are provided, which are formed into operation documents or scripts and delivered to the operation and maintenance team along with the installation and deployment manual for preservation and updating.

[0080] High availability and redundancy requirements

[0081] For commonly used middleware such as cache, message queue, relational database, non-relational database, log middleware, and load balancing, no matter which department or team maintains them, they need to achieve high availability in distributed cluster deployment. The way each middleware implements distributed deployment is different and needs to be treated differently. The high availability standard is 99.9%, that is, the planned downtime and service outage time shall not exceed 8.8 hours per year, and unplanned downtime and service outage are all characterized as accidents. The application itself and the middleware used need to have proper data backup, formulate abnormal or error recovery plans (which can be combined with relevant plans of the basic department and operation and maintenance team), ensure real-time hot standby for key data, and back up non-critical data on a daily or weekly basis.

[0082] Security

[0083] Comprehensive permission control

[0084] Functional permissions: Each function of the application needs to be properly assigned to the appropriate role or user;

[0085] Interface permissions: The visible or operable permissions of various interface elements of the application need to be appropriately assigned to appropriate roles or users.

[0086] Data permissions: The corresponding operations of adding, deleting, modifying and checking various data (classes, models) stored in the application backend need to be properly assigned to appropriate roles or users.

[0087] Operation trace tracking

[0088] The application needs to implement a security audit function that can record various operations performed by users at various levels on the system. The security audit log can be easily queried and can save records for at least 6 months. For core critical systems, the security audit log must be kept for no less than 36 months.

[0089] Safety standards and specifications requirements

[0090] Except for very few special cases, all applications need to adopt international and industry standard security protocols.

[0091] According to Yiche's technology development standards and specifications, all internal applications need to be connected to the group's unified authentication service center and implement single sign-on (SSO). Any application must comply with national and industry data security standards, regulations and specifications. It is strictly prohibited to save key information in plain text, such as user passwords, bank card numbers, ID numbers, facial data, etc.

[0092] Any application needs to properly keep the passwords of operation and maintenance personnel and management personnel at all levels of the system, including middleware management passwords, and weak passwords are strictly prohibited.

[0093] Any application needs to continue to pay attention to the security vulnerability notifications issued by Yiche’s security department or the vulnerability notifications issued by national and industry counterparts, and promptly repair the security vulnerabilities in the application.

[0094] Scalability

[0095] Apply horizontal expansion mechanism

[0096] After the application is deployed in a distributed cluster, the application itself or the middleware used by the application can perform horizontal performance expansion. The application deliverables must include a horizontal expansion operation manual and be delivered to the operation and maintenance team along with the installation and deployment files.

[0097] Application vertical expansion mechanism

[0098] Applications can achieve vertical expansion by applying for higher-configuration server resources, including vertical expansion of CPU, memory, disk, and network resources.

[0099] The technical stack requirements for this system are:

[0100] According to the current technology stack of Bitauto, newly designed and developed applications need to meet the requirements of the unified technology stack. That is, Spring Boot (which also has unified requirements for versions) is used as the main development framework and tool to implement containerized development and deployment, and the DevOps tool chain (ARK platform and workHub platform) provided by Bitauto is used to implement CI, and teams with conditions can implement CD. Since the technology stack is frequently updated, the application development team needs to obtain the latest detailed list of technology stacks from the technology platform. Each potential supplier needs to certify and learn the list. Bidding means understanding Bitauto's requirements for a unified technology stack and promising to carry out technology development in accordance with the requirements.

[0101] The technology stack released by Bitauto specifies the various technologies and tools used in the back-end, front-end development and testing processes, as well as the types and versions of various middleware services that DBaaS can provide. The compiler of the technical specification can download the latest version from the relevant wiki space and directory and provide it to the development team.

[0102] For the reconstruction of existing applications, the requirements for unified technology stacks can be appropriately relaxed. Applications with conditions can migrate development frameworks or DevOps to upgrade the application technology architecture. Except for individual cases, newly designed and developed applications should use containerized development and deployment. The available container development and deployment platforms are TSF and TKE. For newly designed and developed applications that introduce non-standard technology stacks, a technical architecture review meeting needs to be initiated, and the technical team of the Digitalization Department will review and decide whether to merge and adopt the newly introduced technology components into the technology stack.

[0103] The front-end design requirements are:

[0104] Depending on the requirements of each project for UI style, the application needs to use the UI development tools or resource packages provided by Yiche and perform front-end development in accordance with the specified UI / UE standards and specifications.

[0105] Other requirements are:

[0106] As a backend management system, all systems with interfaces need to be connected to Yiche's unified login platform (SSO) in accordance with Yiche's requirements, thereby simplifying the "identity authentication" function.

[0107] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. Database query machine, characterized by: The invention comprises a data layer unit (1), a service layer unit (2), a communication layer unit (3) and a presentation layer unit (4), wherein the data layer unit (1) is connected to the service layer unit (2), the service layer unit (2) is connected to the communication layer unit (3), and the communication layer unit (3) is connected to the presentation layer unit (4).

2. The database query machine according to claim 1, characterized in that: The data layer unit (1) comprises a momgodb module (5), a mysql module (6), a sqlserver module (7), a redis module (8) and an elasticsearch module (9).

3. The database query machine according to claim 1, characterized in that: The service layer unit (2) comprises a monitoring and protection module (10), a business cluster module (11) and a governance configuration module (12).

4. The database query machine according to claim 1, characterized in that: The communication layer unit (3) includes a netty module (14), a socket module (15) and an http\https module (16).

5. The database query machine according to claim 1, characterized in that: The presentation layer unit (4) includes a web module (17).

6. The database query machine according to claim 3, characterized in that: The monitoring and protection module (10) comprises a skywalking link tracking module (101), a gateway cluster monitoring module (102) and a database activity detection monitoring module (103).

7. The database query machine according to claim 3, characterized in that: The business cluster module (11) comprises a query machine module (111), an execution log module (112), an online data review and export block (113) and a collection module (114).

8. The database query machine according to claim 3, characterized in that: The governance configuration module (12) includes an input module (121), a modification module (122) and an export module (123).

9. The database query machine according to claim 1, characterized in that: The web module (17) includes html, html5 and vue.

10. The database query machine according to claim 7, characterized in that: The query machine module (111) includes a mysql query machine, a redis query machine, a mongo query machine and an es query machine.

Images