OceanBase database initialization method and device, equipment and medium

Abstract

The invention relates to the technical field of databases, in particular to an OceanBase database initialization method and device, equipment and a medium. Obtaining an input command, and putting parameters in the command into a command set; executing database loading and file scanning according to parameters in the command set; after file scanning is completed, when the command set does not contain SQL statements, the command set is transmitted to the native Django framework; when the command set contains the SQL statement, verifying the SQL statement according to the incompatible rule set; after the verification is passed, the command set is transmitted to the native Django framework; and when the native Django framework generates the migrating file, checking, and / or correcting the feature field information in the migrating file according to the parameters in the command set. Through a Django command parameter mode, the OceanBase and non-OceanBase conditions are supported at the same time, the characteristics of a Django framework automatic management database are maintained, and the development efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of database technology, and in particular to an OceanBase database initialization method, device, equipment and medium. Background Art

[0002] In the field of information technology, it involves multiple levels such as software and hardware, services, and communications, such as chips, operating systems, servers, databases, and network security. In database migration and transformation, Alibaba's self-developed distributed relational database OceanBase has become the preferred replacement target for databases such as MySQL and Oracle with its high availability, high performance, strong consistency, and high compatibility. Since the ORM of third-party open source frameworks such as Spring and Django is basically used for database mapping during software application development, it is necessary to clarify the support of the ORM of the third-party framework for data during database migration and transformation.

[0003] Django framework is an open source application framework developed in Python and following the MVC (Model-View-Controller) principle. It provides URL distributor, template system, form processing, scalability, rich ORM interface, and built-in database drivers such as MySQL and Oracle. It is particularly suitable for more complex Python projects. It can quickly help users complete the development of business functions. It is currently the most mainstream Web application development framework in Python. However, software applications using Django framework have the following problems during the migration and transformation of the OceanBase database:

[0004] The MySQL mode of the OceanBase database is compatible with most of the functions and syntax of MySQL 5.7 / 8.0. However, due to different product architectures, some MySQL functions, syntax, and optimizers are not supported. Manual identification and inspection are required during migration, which greatly increases the migration cost of the database.

[0005] Although OceanBase is compatible with almost all Oracle R&D functions, it is different from Oracle in terms of underlying architecture and product form. Some data types are still not supported and need to be identified by developers themselves. This is especially difficult for old historical projects.

[0006] In certain cases, the migration files generated by the migration mechanism of the Django framework may create fields and establish foreign key associations at the same time. The database SQL statements mapped to these fields can run correctly in native MySQL / Oracle databases, but problems may occur when executed in OceanBase. OceanBase currently has no solution to this problem, and the native Django framework currently has no compatible processing method.

[0007] When migrating a large project to OceanBase, all existing SQL code files and data fields need to be checked one by one. If the project has multiple environments at the same time, the OceanBase database resource consumption is too large.

[0008] In summary, there is an urgent need for an OceanBase database initialization technology solution based on the Django framework to solve compatibility issues, reduce manual troubleshooting, and improve efficiency. Summary of the invention

[0009] In response to the above problems, the present disclosure provides an OceanBase database initialization method, apparatus, device and medium.

[0010] In a first aspect, a method for initializing an OceanBase database is provided, the method comprising:

[0011] Obtaining input commands and placing parameters in the commands into a command set; performing database loading and file scanning according to the parameters in the command set; and transmitting the command set to the native Django framework when the file scanning is completed and the command set does not contain SQL statements;

[0012] When the command set contains SQL statements, the SQL statements are verified according to the incompatible rule set; after the verification passes, the command set is transferred to the native Django framework;

[0013] After the native Django framework generates a migration file, the characteristic field information in the migration file is checked and / or corrected according to the parameters in the command set.

[0014] Further, executing database loading and file scanning according to the parameters in the command set includes:

[0015] When the command set contains the OceanBase parameter, database loading and file scanning are performed in sequence.

[0016] Furthermore, the performing of database loading and file scanning according to the parameters in the command set further includes:

[0017] When the command set does not contain the OceanBase parameter, the command set is transferred to the native Django framework.

[0018] Furthermore, the database loading includes:

[0019] Load all databases configured under the native Django framework.

[0020] Furthermore, the file scanning includes:

[0021] Scan all py and SQL files in the current project and search for SQL statements.

[0022] Furthermore, the checking of the SQL statement according to the incompatible rule set includes:

[0023] Verify SQL statements according to the preset incompatible rule set, and call the MySQL verifier or Oracle verifier to check and verify SQL statements according to the database loading type.

[0024] Further, the checking and / or modifying the characteristic field information in the migration file according to the parameters in the command set includes:

[0025] When the command set contains the OceanBase parameter, the migration file generated by the native Django framework is checked according to the parameters in the command set; when the feature field check in the migration file is inconsistent with the parameters in the command set, the feature field statement of the migration file is corrected to make the feature field in the migration file consistent with the parameters in the command set.

[0026] In a second aspect, an OceanBase database initialization device includes: a Command component, a Checker component, and an Autodetector component;

[0027] The Command component is used to obtain input commands and put the parameters in the commands into a command set; perform database loading and file scanning according to the parameters in the command set; after the file scanning is completed, if the command set does not contain SQL statements, transmit the command set to the native Django framework;

[0028] Checker component, used to check SQL statements according to incompatible rule sets when the command set contains SQL statements; after the check passes, the command set is transferred to the native Django framework;

[0029] The Autodetector component is used to check and / or correct the feature field information in the migration file according to the parameters in the command set after the native Django framework generates the migration file.

[0030] In a third aspect, an electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0031] a memory storing a computer program;

[0032] The processor is used to implement the above-mentioned OceanBase database initialization method when executing the computer program stored in the memory.

[0033] In a fourth aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned OceanBase database initialization method is implemented.

[0034] The present disclosure includes at least the following beneficial effects:

[0035] This paper uses Django command parameters to support both OceanBase and non-OceanBase situations, maintaining the Django framework's feature of automatically managing databases and improving development efficiency. By using commands, OceanBase incompatibility information in the project is checked, automatically helping to identify problems during development and repairing problematic migration statements.

[0036] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures indicated in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a schematic diagram of the initialization method flow in an embodiment of the present disclosure;

[0039] Figure 2 This is a schematic diagram of the structure of the initialization device according to an embodiment of the present disclosure;

[0040] Figure 3 A schematic diagram of the composition of the initialization device according to an embodiment of the present disclosure;

[0041] Figure 4 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0043] like Figure 1 As shown, a method for initializing an OceanBase database includes:

[0044] Obtaining input commands and placing parameters in the commands into a command set; performing database loading and file scanning according to the parameters in the command set; and transmitting the command set to the native Django framework when the file scanning is completed and the command set does not contain SQL statements;

[0045] When the command set contains SQL statements, the SQL statements are verified according to the incompatible rule set; after the verification passes, the command set is transferred to the native Django framework;

[0046] After the native Django framework generates a migration file, the characteristic field information in the migration file is checked and / or corrected according to the parameters in the command set.

[0047] The specific implementation is as follows:

[0048] This database initialization device (referred to as django_ob in the code reference, hereinafter referred to as django_ob) mainly involves hot patch loading, parameter acquisition, database loading, file scanning, incompatible rule set acquisition, MySQL / Oracle verification check, migration file inspection and correction, etc. in the specific execution process. The specific process is as follows Figure 1 shown.

[0049] In the overall process, the Command component is responsible for accepting Django parameters, putting the parameter information into the command set, and performing database loading and file scanning according to the parameter conditions; the Checker component is responsible for obtaining incompatible rule sets, and calling the MySQL validator or Oracle validator to check and verify SQL statements based on the database loading results; the Autodetector component mainly checks and changes the feature field information in the migration file according to the command set parameters.

[0050] When using the django_ob installation, you need to replace the Command component in the native Django command first to ensure that the project can obtain and execute the installation-related methods when it is initialized. The principle is to use the dynamic language characteristics of Python to replace the native Django method with the modified method through the monkey patch mode, so as to meet the support of implementing new functions without changing the original Django framework, while ensuring the unbinding of the installation from the Django framework and the independence of the installation code. The replacement method is to replace the Command class in the original Django with the OceanBaseCommand class in the Command component of this installation in the manage.py file in the project root directory. The code is as follows:

[0051]

[0052] Command component implementation steps:

[0053] The Command component structure is as follows:

[0054]

[0055]

[0056]

[0057] 1. Command set extension: The command set is compatible with the original Django commands and adds support for the "oceanbase" parameter. When the "oceanbase" parameter is recognized in the user command, the data loading and file scanning methods in the django_ob device will be started. If the "oceanbase" parameter is not recognized, the original Django method is directly called for loading. The Command component of this device inherits the Command component of the original Django to achieve adaptation in the subclass, which does not make any invasive modifications to the original Django and also achieves the device's own functions.

[0058] 2. Database loading: Database loading automatically identifies the database configuration in the Django framework and the database type. Since Django supports multiple data sources, it is necessary to put the identified database types into the current loader and pass the loader information as a parameter when scanning text.

[0059] 3. Text scanning: The text scanner will automatically scan all py files and SQL files under the current project. In Django's own ORM interface, OceanBase supports it, so you only need to judge the native SQL statement information. By calling Python's built-in AST abstract number, you can get all the applications in the py file, and then determine whether the database ORM mapping is used to identify whether the current file performs database operations. When judging the database mapping connection, in addition to Django, other database connection tools such as PyMySQL, cx_Oracle, etc. also need to be judged to ensure that the judgment is fully covered. When the SQL statement is identified, start the verifier component for verification. The main code for text scanning is as follows:

[0060]

[0061]

[0062] Checker component implementation steps:

[0063] The Checker component consists of the following:

[0064]

[0065] Incompatible rule set: The incompatible rule set includes two parts: initialization data set and online data set acquisition. The initialization data set is the incompatible rule information that has been clearly defined in the current device. When the online data set acquisition fails, the initialization data set is directly used for verification. The online data set obtains the latest incompatible rule set information by crawling the OceanBase official document. The online data set has a default update flag, which is used to check whether it is consistent with the OceanBase official document. If it is inconsistent, an online update operation is performed.

[0066] Verifier: The verifier mainly verifies the database syntax, database functions, SQL statements, and data fields that OceanBase has clearly stated are incompatible, determines whether there is incompatible information in the current project, and prompts the developer. The MySQL verifier and Oracle verifier are based on different verification implementations. Since OceanBase has only explained the compatibility of MySQL and Oracle data, this device currently only supports the MySQL verifier and Oracle verifier.

[0067] After the file is scanned and identified, the database type and SQL statement information are passed to the verifier. The verifier selects a specific MySQL or Oracle verifier through the strategy mode to instantiate, and checks the syntax, functions, and fields of the current SQL statement, and writes incompatible verification information into the result. After all project documents are scanned and checked, if there are incompatible results, the results will be printed to the current command console, including file path, incompatible rules and other information, and the current operation will be terminated. For example, the main implementation code of the MySQL verifier is as follows:

[0068]

[0069]

[0070] Autodetector component implementation steps: The Autodetector component structure is as follows:

[0071]

[0072]

[0073]

[0074] The Autodetector component only contains the Migration modifier, whose main function is to check whether the migration file generated by the native Django framework contains feature field information, which may be incompatible when operated in the OceanBase database through the Django framework, and to modify the feature field information. The main steps include:

[0075] Scan all migration files in the current project. Django will generate a migration file each time the makemigrations operation is executed, so you need to filter the files based on their creation time and only check the migration files generated by this command operation.

[0076] Check whether there is feature field information in the migration file. The feature field information is added through "migrations.AddField" and a foreign key connection relationship is established for the field through "field=models.ForeignKey". The complete feature field information is as follows:

[0077]

[0078]

[0079] When Django generates SQL statements for database execution, it puts field creation and foreign key relationship establishment into one SQL statement. When this statement is executed in OceanBase, a "Column not found" error is prompted. In this case, the mapped SQL is:

[0080]

[0081] Because OceanBase is a distributed database, when the same SQL statement contains both field creation and field foreign key relationship establishment, some nodes in OceanBase will execute foreign key relationship establishment operations without executing field creation operations, resulting in field errors. There is currently no solution to this problem in the OceanBase official documentation.

[0082] Split the feature field "migrations.AddField" operation, first replace "models.ForeignKey" with "models.CharField" in the current line, and only create a normal field; add the "migrations.AlterField" field mapping information at the end of the file, and add the previously canceled foreign key relationship to the field operation, and write the modified information into the original migration file. The modified information is as follows:

[0083]

[0084] When Django maps field information to SQL statements, it generates two SQL statements. The first one is to simply create the field; the second one modifies the field and establishes a foreign key relationship. In this case, OceanBase will not report an error when executing SQL. The specific statement information is as follows:

[0085] ALTER TABLE

[0086] ˋtable_user_auth_settingˋ

[0087] ADD

[0088] COLUMNˋrole_idˋchar(32)NOT NULL;

[0089] ALTER TABLE

[0090] ˋtable_user_auth_settingˋ

[0091] ADD

[0092] CONSTRAINTˋtable_user_auth_setting_role_id_a4569432_fk_table_roleˋFOREIGN KEY

[0093] (ˋrole_idˋ)REFERENCESˋtable_roleˋ(ˋuidˋ)

[0094] The manual method in the prior art is time-consuming and inefficient. Every time the database is initialized or changed, manual inspection and processing are required, which increases the risk and loses the automatic management feature of the database by the Django framework.

[0095] This feature automatically helps developers check and handle SQL compatibility issues in projects when migrating OceanBase databases, allowing developers to quickly focus on the functions themselves. Its main features are as follows:

[0096] Filling the gap: Currently, there is no dedicated compatible device for the OceanBase database in the industry's database support extension tools based on the Django framework.

[0097] Automation: Provides automated verification and automated solutions. Automatically helps developers check the compatibility issues of SQL syntax, functions, and fields in the project when migrating the OceanBase database, without the need for developers to check manually. At the same time, it automatically corrects the characteristic fields in the migration file automatically generated by the Django framework. The SQL statements generated by the characteristic fields will fail to execute in the OceanBase database, resulting in the inability to use Django commands to change the database normally.

[0098] Simple and easy to use: This device integrates database compatibility verification into the Django framework command, perfectly adapting to the Django framework. Developers can perform compatibility verification only through parameters without any other learning costs. Based on the original Django framework command, this device only needs to add the oceanbase parameter, that is, "python manage.py makemigrations--oceanbase".

[0099] Low invasiveness: This device only needs to be referenced in manage.py, and no changes are required in other parts of the project.

[0100] Good scalability: Since OceanBase has only officially explained the compatibility with MySQL and Oracle data, this device currently only supports MySQL validators and Oracle validators. If there are other database compatibility solutions such as PostgreSQL in the future, you only need to add the PostgreSQL validator to implement it.

[0101] The method for scanning files disclosed in the present invention is to identify whether a py file introduces a database mapping class. This method obtains abstract data through AST provided by Python, but can also directly verify whether there are database statement keywords such as "select, update" in the current file through keyword matching.

[0102] The present invention solves the problem of projects developed based on the Django framework. When migrating the database OceanBase from the original MySQL and Oracle databases, it is necessary to manually check for incompatibility issues such as database syntax, fields, functions, and blocking errors of feature fields in the migration file. After a long period of repeated demonstration, this device can obviously identify compatibility issues between OceanBase and MySQL and Oracle databases based on the Django framework, and help developers automatically identify and modify OceanBase incompatibility issues. The device supports both OceanBase and non-OceanBase situations through Django command parameters, maintains the characteristics of the Django framework's automatic management of the database, improves development efficiency, and has very good results in data information system trials.

[0103] like Figure 2 As shown, an OceanBase database initialization device includes: a Command component, a Checker component and an Autodetector component;

[0104] The Command component is used to obtain input commands and put the parameters in the commands into a command set; perform database loading and file scanning according to the parameters in the command set; after the file scanning is completed, if the command set does not contain SQL statements, transmit the command set to the native Django framework;

[0105] Checker component, used to check SQL statements according to incompatible rule sets when the command set contains SQL statements; after the check passes, the command set is transferred to the native Django framework;

[0106] The Autodetector component is used to check and / or correct the feature field information in the migration file according to the parameters in the command set after the native Django framework generates the migration file.

[0107] The specific implementation is as follows:

[0108] This disclosure mainly includes three parts: Checker, Command, and Autodetector. The solution device and the native Django framework are structured as follows Figure 2 shown.

[0109] Command component: Identify the Django version, obtain input commands, and achieve extended compatibility with OceanBase commands; the database loader loads the current database information; identifies whether to introduce Django ORM mapping and other third-party database driver mappings through text feature scanning, etc.

[0110] Checker component: implements syntax, function, and field verification between OceanBase and MySQL and Oracle databases, mainly including MySQL validator, Oracle validator, and incompatible rule sets.

[0111] Autodetector component: accepts command set information, scans the generated migration files according to specific parameters, corrects problematic feature field statements in migration files, and is compatible with the OceanBase database mode.

[0112] This paper solves the OceanBase database initialization check and compatibility issues of the Django framework. The specific principle is: the developer executes the Django database initialization command, and the Command component checks whether the command carries the oceanbase parameter. According to the oceanbase parameter, the Checker component is started to check the database syntax, function, and method, and the migration file is checked and corrected through the Autodetector component. The specific composition of each component is as follows: Figure 3 shown.

[0113] Command component:

[0114] Verification: Check whether the Django framework exists in the current operating environment and verify the Django framework version to ensure that the component can run correctly.

[0115] Command set: Extends native Django commands and adds support for the "oceanbase" command parameter, allowing users to enter OceanBase check commands. Database loading and text scanning will rely on this command parameter, and subsequent operations can obtain command parameters.

[0116] Database loading: Load all databases configured under the current Django framework and pass the database information into the Check component.

[0117] Text scanning: Scan all py and SQL files in the current project, use Python's built-in AST abstraction to identify whether the current py file contains database ORM mapping, find SQL statements, and pass the information to the Check component.

[0118] Checker component:

[0119] Incompatible rule set: includes incompatible data information written directly during initialization, and updated information obtained by automatically obtaining OceanBase official page documents through crawlers. In the event of network failure, the initialization information shall prevail.

[0120] Validator: includes MySQL and Oracle validators, which mainly perform database syntax validation, database function validation, SQL statement validation, and data field validation.

[0121] Autodetector components:

[0122] Migration modifier: Checks whether there are any incompatibility issues in the migration files generated by the native Django. This method is the least intrusive and modifies the native Django, and if the original Django version is upgraded, it will not affect the use of the current components.

[0123] like Figure 4 As shown, the present disclosure provides an electronic device, including a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 communicate with each other through the communication bus 404;

[0124] Memory 403, storing computer programs;

[0125] The processor 401 is configured to implement the above method when executing the computer program stored in the memory 403 .

[0126] The present disclosure provides a computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0127] The computer-readable storage medium may be included in the device / apparatus described in the above embodiment; or it may exist independently without being assembled into the device / apparatus. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0128] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0129] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. An OceanBase database initialization method, characterized in that: The method comprises: Obtaining input commands and placing parameters in the commands into a command set; performing database loading and file scanning according to the parameters in the command set; and transmitting the command set to the native Django framework when the file scanning is completed and the command set does not contain SQL statements; When the command set contains SQL statements, the SQL statements are verified according to the incompatible rule set; after the verification passes, the command set is transferred to the native Django framework; After the native Django framework generates a migration file, the characteristic field information in the migration file is checked and / or corrected according to the parameters in the command set.

2. The OceanBase database initialization method according to claim 1, characterized in that: The performing of database loading and file scanning according to the parameters in the command set includes: When the command set contains the OceanBase parameter, database loading and file scanning are performed in sequence.

3. The OceanBase database initialization method according to claim 1, characterized in that: The performing of database loading and file scanning according to the parameters in the command set also includes: When the command set does not contain the OceanBase parameter, the command set is transferred to the native Django framework.

4. An OceanBase database initialization method according to any one of claims 1 to 3, characterized in that: The database loading includes: Load all databases configured under the native Django framework.

5. The OceanBase database initialization method according to claim 1, characterized in that: The document scanning includes: Scan all py and SQL files in the current project and search for SQL statements.

6. The OceanBase database initialization method according to claim 1, characterized in that: The checking of the SQL statement according to the incompatible rule set includes: Verify SQL statements according to the preset incompatible rule set, and call the MySQL verifier or Oracle verifier to check and verify SQL statements according to the database loading type.

7. The OceanBase database initialization method according to claim 1, characterized in that: The checking and / or modifying the characteristic field information in the migration file according to the parameters in the command set includes: When the command set contains the OceanBase parameter, the migration file generated by the native Django framework is checked according to the parameters in the command set; when the feature field check in the migration file is inconsistent with the parameters in the command set, the feature field statement of the migration file is corrected to make the feature field in the migration file consistent with the parameters in the command set.

8. An OceanBase database initialization device, characterized in that: include: Command component, Checker component and Autodetector component; The Command component is used to obtain input commands and put the parameters in the commands into a command set; perform database loading and file scanning according to the parameters in the command set; after the file scanning is completed, if the command set does not contain SQL statements, transmit the command set to the native Django framework; Checker component, used to check SQL statements according to incompatible rule sets when the command set contains SQL statements; after the check passes, the command set is transferred to the native Django framework; The Autodetector component is used to check and / or correct the feature field information in the migration file according to the parameters in the command set after the native Django framework generates the migration file.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a memory storing a computer program; A processor, used to implement an OceanBase database initialization method according to any one of claims 1 to 7 when executing a computer program stored in a memory.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, an OceanBase database initialization method according to any one of claims 1 to 7 is implemented.

Images