Data access method, system and equipment based on Django framework and medium

Abstract

The invention is applicable to the technical field of data processing, and provides a data access method based on a Django framework, which comprises the following steps of: analyzing a subclass statement to obtain a table name, a table structure, a primary key and a partition key of a partition table to be established; determining the partition type of the partition table to be established by judging the data type of the partition key; expanding the Django framework on the basis of migrating of the Django framework and the partition type, and generating a partition table building statement on the basis of the expanded Django framework; executing the partition table building statement through a database transaction, and building the partition table; and generating a data management statement based on the ORM of the Django framework, wherein the data management statement is used for adding, updating, filtering or deleting the data records in the partition table. According to the method, a set of complete ORM does not need to be developed for partition table establishment, the development and maintenance cost is reduced, meanwhile, database operation is simplified, the development efficiency is improved, and in addition, the method has the single-table 10 billion-level large-scale data access capacity.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular relates to a data access method, system, device and medium based on a Django framework. Background Art

[0002] In the digital age, the widespread popularity of the Internet, the Internet of Things, and social media has driven the explosive growth of global data. This trend not only marks a surge in the amount of information, but also means that data has become an indispensable key resource in corporate innovation and decision-making. The rise of data collection platforms provides companies with an effective tool to collect and integrate massive amounts of data and transform them into valuable assets. These platforms have become the key basic systems for companies to achieve data-driven development, supporting companies to stay ahead in the fierce market competition.

[0003] In the prior art, the acquisition of data in the data collection platform is usually implemented through a data warehouse or a database. Specifically, as the core of enterprise-level data storage and management, the data warehouse optimizes the data storage structure by adopting technologies such as partitioning, column storage, and data compression. Combined with the high capacity and reliability of the distributed storage system, the data warehouse can effectively support the storage requirements of large-scale data access based on Django. In addition, the access performance of large-scale data access based on Django in the data warehouse can be significantly improved by using technologies such as indexes, query optimizers, specialized computing engines, and hierarchical storage management. By integrating the data access layer and intelligent SQL statement generation, the data warehouse can provide efficient data access services to meet the high requirements of enterprises for data access speed and accuracy; for databases, taking NoSQL databases as an example, NoSQL databases have become an ideal choice for storing unstructured or semi-structured large-scale data access based on Django with their flexible data models and excellent horizontal expansion capabilities. NoSQL databases provide efficient data access services through technologies such as client integration, API integration, and shard-aware queries. These technologies enable NoSQL databases to flexibly handle various types of data and to cope with the growing amount of data through horizontal expansion. These characteristics of NoSQL databases enable them to occupy an important position in the field of big data and meet the diverse needs of data storage and access in the big data era.

[0004] In the above-mentioned data access method using data warehouse, the data warehouse construction and maintenance process usually involves complex technology, which requires professionals to have deep knowledge and skills to design, implement and optimize. In addition, some data warehouse systems have shown low adaptability and flexibility in the face of rapid changes in business needs and data types. The data integration process, especially the integration of data from different sources and formats into the data warehouse, often requires tedious data cleaning, conversion and mapping steps. In order to ensure the security, maintainability and performance of the code, additional development work is usually required for the data warehouse, such as developing a complete set of ORM or other measures. Therefore, the data warehouse has problems of complexity and flexibility. For the data access method using NoSQL database, NoSQL database lacks support for traditional ACID transactions, which in turn limits the processing of complex transactions. In addition, NoSQL databases are also limited in query capabilities. When executing complex queries and multi-table associations, they lack some mature tools and integrations, such as ORM support and strong automatic fault recovery capabilities. As a result, NoSQL databases have problems with complex transaction support and query capability limitations. Summary of the invention

[0005] The embodiments of the present application provide a data access method, system, device and medium based on the Django framework, which can solve one of the above-mentioned prior art problems.

[0006] In a first aspect, an embodiment of the present application provides a data access method based on a Django framework, comprising:

[0007] By parsing the subclass declaration, obtain the table name, table structure, primary key and partition key of the partition table to be created;

[0008] Determine the partition type of the partition table to be established by judging the data type of the partition key;

[0009] Generate partition table creation statements based on the Django framework's migrate and the partition type;

[0010] Execute the partition table creation statement through a database transaction to create the partition table;

[0011] The ORM based on the Django framework generates data management statements, wherein the data management statements are used to add, update, filter or delete data records in the partition table.

[0012] Further, determining the partition type of the partition table to be established by judging the data type of the partition key includes:

[0013] If there is only one partition key, and the designated column of the partition key is id, the partition type of the partition table is determined to be KEY type;

[0014] If there is one or more partition keys, and the data type of the designated column of the partition key is INT type or CHART type, then the partition type of the partition table is determined to be HASH type;

[0015] If there is only one partition key, and the data type of the designated column of the partition key is DATETIME type, then the partition type of the partition table is determined to be TO_DATE type;

[0016] If the partition type of the partition table cannot be determined by the partition key, a partition key error declaration message is issued.

[0017] Furthermore, the partition table to be established is also provided with an associated attribute, and the associated attribute is used to determine whether the partition table allows foreign key constraints;

[0018] Before the migration of the Django framework and the partition type generate partition table creation statements, the following are included:

[0019] Based on the partition type, determining an attribute value of the associated attribute;

[0020] If the partition type is a KEY type, determining that the attribute value of the associated attribute is TRUE;

[0021] If the partition type is a HASH type or a TO_DATE type, determining that the attribute value of the associated attribute is FALSE;

[0022] If there is a partition table to be associated and the attribute value of the associated attribute of the partition table is FALSE, an association failure message is issued and generation of a partition table creation statement is prevented.

[0023] Furthermore, the migrate of the Django framework and the partition type generate a partition table creation statement, including:

[0024] Generate the first table creation statement based on the migrate of Django framework;

[0025] Generate a second table creation statement according to the partition type, where the second table creation statement is used to modify the primary key of the first table creation statement to a joint primary key, where the joint primary key includes a primary key and a partition key;

[0026] Generate a third table creation statement based on the partition type and the joint primary key;

[0027] A partition table creation statement is generated based on the first table creation statement, the second table creation statement, and the third table creation statement.

[0028] Further, after executing the partition table creation statement through the database transaction to create the partition table, the following steps are included:

[0029] When adding a field to a non-partitioned table, generating a table modification statement and executing the table modification statement;

[0030] Before executing the table modification statement, determining whether the table modification statement satisfies an online execution operation, wherein the online execution operation supports the database to remain online during the execution process of the table modification statement;

[0031] If the table modification statement satisfies the online execution operation, a configuration statement is added on the basis of the table modification statement, wherein the configuration statement is used to enable the table modification statement to execute the operation online;

[0032] If the table modification statement does not satisfy the online execution operation, an error alarm mechanism is triggered.

[0033] Furthermore, the ORM based on the Django framework generates data management statements, and the data management statements are used to add, update, filter or delete data records in the partition table, including:

[0034] Based on the data operation requirements, corresponding data operation statements are generated, wherein the data operation statements include data adding statements, data updating statements, data filtering statements and data deleting statements;

[0035] A corresponding data operation method is created for each data operation, and a data management statement is generated based on the data operation statement and the data operation method to operate the data in the partition table.

[0036] Furthermore, before the ORM based on the Django framework generates a data management statement, the data management statement is used to add, update, filter or delete data records in the partition table, including:

[0037] Obtain key information of the partition table, the key information including class name and partition key;

[0038] Detecting an operation parameter of the data operation statement, and determining whether the operation parameter includes a partition key;

[0039] If the operation parameters do not contain the partition key, a ValueError exception is raised and the generation of the data management statement is prevented.

[0040] In a second aspect, an embodiment of the present application provides a data access system based on a Django framework, comprising:

[0041] Parsing module: used to obtain the table name, table structure, primary key and partition key of the partition table to be created by parsing the subclass declaration;

[0042] A partition type determination module: used to determine the partition type of the partition table to be established by judging the data type of the partition key;

[0043] Table creation statement generation module: used to generate partition table creation statements based on the Django framework's migrate and the partition type;

[0044] A partition table creation module: used to execute the partition table creation statement through a database transaction to create the partition table;

[0045] Partition table data operation module: used to generate data management statements based on the ORM of the Django framework, and the data management statements are used to add, update, filter or delete data records in the partition table.

[0046] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data access method based on the Django framework when executing the computer program.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned data access method based on the Django framework is implemented.

[0048] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0049] The present application provides a data access method based on the Django framework, which adds partition table management to the Django development framework to achieve the expansion of the Django framework. The partition table creation statement is automatically generated and executed through the expansion of the Django framework, thereby realizing the establishment of the partition table. There is no need to develop a complete ORM to establish the partition table, which reduces the development and maintenance costs. In addition, a strict detection mechanism for foreign key constraints on partition tables is introduced to achieve the ability to access large-scale data of tens of billions of levels for a single table. In addition, the data operation of the partition table is abstracted at a high level through ORM, allowing developers to use Python code instead of direct SQL statements to operate the database. Furthermore, by declaring ORM partition keys and constraining the data management statements of the partition table, developers do not need to perceive the underlying database partitioning implementation principle and can correctly use the database partition table, which not only simplifies database operations but also improves development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 It is a flow chart of a data access method based on the Django framework provided by one embodiment of the present invention;

[0052] Figure 2 It is a structural diagram of a data access system based on the Django framework provided by one embodiment of the present invention;

[0053] Figure 3 It is a structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0055] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0056] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0057] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0058] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0059] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0060] See also Figure 1 As shown, the present invention is a data access method based on the Django framework, comprising the following steps:

[0061] S100, obtaining the table name, table structure, primary key and partition key of the partition table to be established by parsing the subclass declaration;

[0062] Django is an advanced Python web framework that encourages rapid development and clean, practical design. Django follows the MVC (Model-View-Controller) design pattern and divides the application into three core components. The Django development process usually includes defining data models, creating views to handle business logic, and writing templates to display data. In this application, based on the Django framework, a partition table is established for the database. Through Django's object-relational mapping (ORM) function, the dynamic management of the partition table is flexibly implemented. There is no need for business developers to directly write partition definitions, which can meet large-scale data access. The above method not only improves data processing efficiency, but also reduces the complexity of business code, which can better meet the needs of enterprises.

[0063] Specifically, based on the ORM of the Django framework, a parent class named PartitionedModel is defined. The PartitionedModel is an ORM model parent class that supports the partition table function. The parent class contains attributes and methods related to establishing partition tables, such as defining partition keys, specifying partition strategies, and processing special logic related to partition tables, etc., and defines specific objects through subclasses inheriting the PartitionedModel parent class. These objects are ultimately mapped to the partition table of the database, thereby achieving effective management and operation of the partition table. This method not only improves the flexibility and scalability of database design, but also automates and simplifies the data integration process, reduces the tedious steps of data cleaning, conversion, and mapping, thereby simplifying the use and maintenance of partition tables. In addition, ORM can also intelligently process data from different sources and formats to improve the efficiency of data integration.

[0064] In this embodiment, the subclass declaration for declaring the PartitionedModel subclass is first parsed to determine the table name, table structure, primary key, partition key and other information of the partition table to be established, so as to provide information support for the subsequent establishment of the partition table.

[0065] S200, determining the partition type of the partition table to be established by judging the data type of the partition key;

[0066] Partition tables established with different data types will affect the storage efficiency of data and the performance of data operations. Therefore, in this embodiment, the partition key of the partition table is determined by parsing the subclass declaration, and then the partition type of the partition table is determined by judging the data type of the specified partition key. Based on the partition type, a partition table matching the data characteristics is established, thereby intelligently processing data of different data types, improving the efficiency of data integration, and making large-scale data processing and analysis faster and more flexible.

[0067] In some embodiments, the step S200 includes:

[0068] If there is only one partition key, and the designated column of the partition key is id, the partition type of the partition table is determined to be KEY type;

[0069] If there is one or more partition keys, and the data type of the designated column of the partition key is INT type or CHART type, then the partition type of the partition table is determined to be HASH type;

[0070] If there is only one partition key, and the data type of the designated column of the partition key is DATETIME type, then the partition type of the partition table is determined to be TO_DATE type;

[0071] If the partition type of the partition table cannot be determined by the partition key, a partition key error declaration message is issued.

[0072] In this embodiment, the partition type that best suits the current data characteristics is configured according to the data type of the partition key, such as the HASH type for random hash partitioning, the KEY type for partition key range partitioning, and the TO_DATE type for time partitioning.

[0073] In this embodiment, when the partition type is KEY type, the partition key and the primary key must be the same, that is, the specified columns of the partition key and the primary key are both id fields, and their data type is INT type.

[0074] In this embodiment, when establishing a partition table, one or more fields of the database table may be selected as partition keys. Therefore, a HASH type partitioning method is used to calculate the hash value of the partition key to determine which partitions the data should be stored in, making the data distribution more even, thereby improving the efficiency of query and management to a certain extent.

[0075] In this embodiment, when establishing a partition table, there is a situation where the date time (DATETIME) field is used as the partition key. Therefore, the TO_DATR partitioning method is adopted. By setting a preset time division strategy, such as dividing a zone by natural day, dividing a zone by natural month, and dividing a zone by natural year, an appropriate time division strategy is selected according to actual needs to establish a partition table.

[0076] In this embodiment, if the partition key does not belong to the data type corresponding to the KEY type, the HASH type, and the TO_DATE type, it means that the declaration of the partition key in the subclass declaration of the PartitionedModel subclass is wrong, and a partition key error declaration message is issued to prompt the user to check the corresponding partition key declaration.

[0077] Specifically, in one embodiment of the present application, a subclass declaration of a PartitionedModel subclass is as follows:

[0078] class News(PartitionedModel):

[0079] id=models.AutoField(primary_key=True)

[0080] date = models.DateField()

[0081] title = models.TextField()

[0082] content = models.TextField()

[0083] partition_by = to_date(

[0084] columns = ['date'],

[0085] algorithm = 'by_day' )

[0087] objects=PartitionedManager()

[0088] In the above subclass, a partition model with the class name News is defined, which inherits from the PartitionedModel abstract base class, that is, PartitionedModel is the parent class of the partition model. Through the above subclass declaration, it can be known that the primary key of the partition table to be established is id. In addition, the class name defined by the partition model is used as the table name of the partition table to be established. At the same time, the partition_by attribute is also declared in the partition model. Through the partition_by attribute, it can be known that the partition key of the partition table to be established is date, the partition type is TO_DATE, and the time partitioning strategy in the partition type is by_day, that is, one zone is divided according to natural days. Before establishing a partition table for the News to be partitioned, based on the time partitioning strategy by_day, multiple partition data are generated in advance according to the release time, thereby improving the partitioning efficiency of the partition table.

[0089] S300, based on the migrate of the Django framework and the partition type, extending the Django framework, and generating a partition table creation statement based on the extended Django framework;

[0090] In this embodiment, when establishing a partition table, it is necessary to use the migrate of the Django framework to migrate data between the partition table and the original data table. Specifically, by defining a Django management command named partition_migrate, the Django management command is a command logic definition for generating and executing partition table creation statements. It inherits from the BaseCommand class. It can be understood that based on the subclass declared based on the PartitionedModel parent class, partition_migrate generates partition table creation statements and executes corresponding partition table creation statements.

[0091] Specifically, the subclass declaration of partition_migrate inherited from BaseCommand is as follows:

[0092] class partition_migrate(BaseCommand):

[0093] help = 'Generate partition table'

[0094] def handle(self,*args,**options):

[0095] #Traverse all partition models inherited from PartitionedModel

[0096] for model in apps.get_models():

[0097] if issubclass(model,PartitionedModel):

[0098] #Get the partition configuration information of the partition model

[0099] partition_by=getattr(model,'partition_by',None)

[0100] if partition_by:

[0101] #Generate partition table creation statement

[0102] partition_ddl=generate_partition_ddl(model._meta.db_table)

[0103] #Execute the partition table creation statement on the database

[0104] with connection.cursor()as cursor:

[0105]

[0106] In the above statement, the handle method is declared. In this method, the regular database migration operation is first performed to ensure that the table structure of the partition table is created. Then, all partition models inherited from PartitionedModel are traversed to obtain the partition configuration information of the partition model. The partition configuration information includes the primary key, partition key, and partition type. For each partition model, the generate_partition_ddl method is called to generate the partition table creation statement partition_ddl. Then, based on the mysql_client library, the partition_ddl statement is executed to complete the partition table creation process. By adding partition_migrate for partition table management to the Django framework, the Django framework is extended. When executing the database migration command, the Django framework automatically generates a partition table creation statement for the News table in the MySQL database and executes the partition table creation statement. Developers do not need to manually write any SQL code related to the establishment of partition tables, which improves development efficiency and security.

[0107] In some embodiments, before the above step S300, the following steps are included:

[0108] Based on the partition type, determining an attribute value of the associated attribute;

[0109] If the partition type is a KEY type, determining that the attribute value of the associated attribute is TRUE;

[0110] If the partition type is a HASH type or a TO_DATE type, determining that the attribute value of the associated attribute is FALSE;

[0111] If there is a partition table to be associated and the attribute value of the associated attribute of the partition table is FALSE, an association failure message is issued and generation of a partition table creation statement is prevented.

[0112] In this embodiment, the foreign key constraint is used to ensure the consistency of data in one table with data in another table. When the partition type is KEY type, because the partition key is the primary key id field and is unique, the partition table created by the PartitionedModel subclass can support foreign key constraints. However, when the partition type is HASH type or TO_DATE type, the partition key is not necessarily unique, so the partition table created by HASH type or TO_DATE type does not support foreign key constraints.

[0113] In this embodiment, in the subclass declaration of the PartitionedModel subclass, the associated attribute AllowForeignKey is set. When the partition table is established, the attribute value of the associated attribute AllowForeignKey is checked to determine whether the associated partition table supports foreign key constraints. When AllowForeignKey=YES, it means that the database table does not support foreign key constraints. On this basis, the partition table creation statement cannot be generated, that is, the partition table cannot be established.

[0114] In this embodiment, partition_migrate for partition table management and AllowForeignKey association attribute for constraining correct use capability are added to the Django development framework, thereby achieving large-scale data access capability of tens of billions of data in a single table.

[0115] In some embodiments, generating a partition table creation statement includes:

[0116] Generate the first table creation statement based on the migrate of Django framework;

[0117] Generate a second table creation statement according to the partition type, where the second table creation statement is used to modify the primary key of the first table creation statement to a joint primary key, where the joint primary key includes a primary key and a partition key;

[0118] Generate a third table creation statement based on the partition type and the joint primary key;

[0119] A partition table creation statement is generated based on the first table creation statement, the second table creation statement, and the third table creation statement.

[0120] In this embodiment, a conventional database migration operation based on migrate is adopted, and the basic table structure of the partition table is generated through the first table creation statement DDL1. According to the partition type of the partition table, the second table creation statement DDL2 is generated, and the primary key in the basic table structure is modified to a joint primary key. Specifically, taking the partition type of the partition table as key as an example, DDL2 "alter table {table name} drop primary key, add primary key (id, {partition key});" is generated, and then the third table creation statement DDL3 "ALTER TABLE {table name} PARTITION BY KEY (id, {partition key});" is generated through the partition type and the joint primary key. Similarly, if the partition type is HASH type, the third table creation statement DDL3 is "ALTER TABLE {table name} PARTITION BY HASH ({id, partition key});", if the partition type is TO_DATE type, the third table creation statement DDL3 is "ALTER TABLE {table name} PARTITION BY TO_DATE({id, partition key});", and then, the first table creation statement, the second table creation statement and the third table creation statement are concatenated to form the partition table creation statement DDL.

[0121] In this embodiment, taking the above-mentioned embodiment of generating a partition table named News as an example, a partition table creation statement is generated. Specifically, based on the handle method of partition_migrate, a partition table creation statement is generated. Specifically, first, the conventional database migration operation of migrate is performed to generate the first table creation statement DDL1 for establishing the basic table structure of the partition table, and then the generate_partition_ddl method is called. In the generate_partition_ddl method, the partition table declaration and the database table in the PartitionedModel parent class are traversed to obtain an unpartitioned database table set, and a partition operation is performed on each database table in the unpartitioned database table set to generate two database tables. In this process, based on the partition type and the primary key in the basic table structure, a second table creation statement DDL2 is generated. The second table creation statement DDL2 is as follows "alter table news drop primary key, add primary key key(id,date)", through the second table creation statement DDL2, a joint primary key key(id,date) is generated, and through the joint primary key, a third table creation statement DDL3 is generated. In one embodiment, the third table creation statement DDL3 is as follows "PARTITION BY RANGE(TO_DAYS(date))(PARTITION p20180101 VALUES LESS THAN(TO_DAYS('2018-01-01'), PARTITION p20180102 VALUES LESS THAN(TO_DAYS('2018-01-02')......PARTITION pmax VALUES LESS THANMAXVALUE)", finally, the above DDL1, DDL2, and DDL3 are spliced ​​to form a complete partition table creation statement DDL, and the partition table creation statement DDL is as follows "DDL(DDL1, DDL2, DDL3)".

[0122] S400, executing the partition table creation statement through a database transaction to create the partition table;

[0123] In this embodiment, a database transaction is used to execute the partition table creation statement. Since database transactions have the characteristics of atomicity, consistency, isolation, and persistence, during the table creation process, this means that the entire table creation operation is either completed successfully or rolled back to the state before the operation when any error occurs, thereby ensuring the consistency of the database state. In this way, we safely and reliably create the required partition table, laying a solid foundation for subsequent data storage and query operations.

[0124] In some embodiments, after the above step S400, the following steps are included:

[0125] When adding a field to a non-partitioned table, generating a table modification statement and executing the table modification statement;

[0126] Before executing the table modification statement, determining whether the table modification statement satisfies an online execution operation, wherein the online execution operation supports the database to remain online during the execution process of the table modification statement;

[0127] If the table modification statement satisfies the online execution operation, a configuration statement is added on the basis of the table modification statement, wherein the configuration statement is used to enable the table modification statement to execute the operation online;

[0128] If the table modification statement does not satisfy the online execution operation, an error alarm mechanism is triggered.

[0129] In this embodiment, if a new field in a non-partition table is added to the generated partition table, when executing a table modification statement, it is necessary to evaluate whether the execution of the table modification statement satisfies the online execution operation to reduce the impact on the database service.

[0130] In this embodiment, for table modification statements that do not meet the requirements for online execution, an error alarm mechanism is triggered when the table modification statements are executed. The developer is notified that the database migration being attempted cannot be completed online and may affect the normal operation of the database, thereby prompting the developer to make correct processing operations.

[0131] In this embodiment, for table modification statements that meet the requirements of online execution, a configuration statement "ALGORITHM=INPLACE,LOCK=SHARED" is generated at the same time as the table modification statement is generated. ALGORITHM=INPLACE means that the modification should be performed in situ as much as possible to reduce the copying and movement of data. LOCK=SHARED means that during the modification process, the partition table should be locked in shared mode, allowing read operations but not write operations. In this way, the ORM model can modify the partition table without blocking, improve the availability of the partition table, and reduce the impact on database services.

[0132] S500. Generate data management statements based on the ORM of the Django framework, where the data management statements are used to add, update, filter or delete data records in the partition table.

[0133] In some embodiments, the step S500 includes:

[0134] Based on the data operation requirements, corresponding data operation statements are generated, wherein the data operation statements include data adding statements, data updating statements, data filtering statements and data deleting statements;

[0135] A corresponding data operation method is created for each data operation, and a data management statement is generated based on the data operation statement and the data operation method to operate the data in the partition table.

[0136] In this embodiment, the models.Manager class provided by the ORM based on the Django framework manages the ORM object behavior and realizes the generation of data management statements. Specifically, by using Python's dynamic class definition capability, with models.Manager as the parent class, PartitionedManager is dynamically defined to inherit models.Manager, and the create, update, delete, filter and other data operation methods are overloaded in the subclass PartitionedManager. It can be understood that the subclass declared based on the PartitionedModel parent class generates data management statements by calling the data operation methods encapsulated in PartitionedManager, thereby adding, updating, filtering or deleting data records in the partition table.

[0137] Specifically, the above subclass of PartitionedManager inherited from models.Manager is declared as follows:

[0138] class PartitionedManager(models.Manager):

[0139] def filter(self,from_result,**kwargs):

[0140] partition_by=getattr(self.model,'partition_by')

[0141] kwargs[partition_by.partition_key]=get_and_check_partition_value(kwargs)

[0142] return super().filter(**kwargs)

[0143] def create(self,**kwargs):

[0144] partition_by=getattr(self.model,'partition_by')

[0145] kwargs[partition_by.partition_key]=get_and_check_partition_value(kwargs)

[0146] return super().create(**kwargs)

[0147] def update(self,**kwargs):

[0148] partition_by=getattr(self.model,'partition_by')

[0149] kwargs[partition_by.partition_key]=get_and_check_partition_value(kwargs)

[0150] return super().update(**kwargs)

[0151] def delete(self,**kwargs):

[0152] partition_by=getattr(self.model,'partition_by')

[0153] kwargs[partition_by.partition_key]=get_and_check_partition_value(kwargs)

[0154] return super().delete(**kwargs)

[0155] objects=PartitionedManager()

[0156] In the above statement, four data operation methods, namely create, update, delete, and filter, are encapsulated. By setting the objects attribute, the News partition table defined in the above embodiments is pointed to the PartitionedManager class. Thus, queries, creations, deletions, and modifications of the partition table can be processed through this PartitionedManager class. Specifically, corresponding data operation statements are generated based on data operation requirements, enabling them to call the corresponding data operation methods encapsulated in the PartitionedManager. For example, for an article filtered by title through filter, its data operation statement is "News.objects.filter(title='title', date='20240101')". Through the filter method encapsulated in the PartitionedManager, super().filter(**kwargs) is returned, and then the corresponding data management statement, that is, the DML statement, is generated, and this DML statement is automatically executed, finally obtaining a list of News instances that meet the title condition.

[0157] In this embodiment, the ORM performs a high-level abstraction of the data operations on the partition table, allowing developers to operate the database using Python code instead of direct SQL statements. This not only simplifies database operations but also improves development efficiency. In addition, the ORM usually provides built-in mechanisms to prevent security issues such as SQL injection, thereby reducing data risks.

[0158] In some of the embodiments, before the above step S500, it includes:

[0159] Obtain the key information of the partition table, where the key information includes the class name and the partition key;

[0160] Detect the operation parameters of the data operation statement and determine whether the operation parameters contain the partition key;

[0161] If the operation parameters do not contain the partition key, a ValueError exception is raised, and at the same time, the generation of the data management statement is blocked.

[0162] Under a large amount of data, if no specific partition key is specified for querying data, it will cause the system to scan all the data in the partitioned table during data operations, which is likely to result in slow queries and reduce the data query efficiency. Therefore, in the operation parameters of the data operation statement generated in this embodiment, a partition key needs to be set so that when performing data operations on the partitioned table, corresponding data can be returned based on the partition key, thereby improving the data query efficiency. It can be understood that each data operation method encapsulated in PartitionedManager includes a method for detecting whether the operation parameters of the data operation statement contain a partition key.

[0163] In this embodiment, taking the above example of filtering articles that meet the title conditions through filter, if the data operation statement generated by the developer based on data operation requirements is "News.objects.filter(title='标题')", when calling the filter method in PartitionedManager, the partition key defined in the News class obtained through get_and_check_partition_value is "date", and the data operation statement is checked for parameters. If the specific partition key "date" cannot be obtained, a ValueError exception is raised and a log is recorded to prevent the data operation statement from continuing to execute to generate the corresponding data management statement. When it is detected that the partition key "date" exists in the data operation statement, then super().filter(**kwargs) is returned, and a list of News instances is automatically obtained while generating the corresponding DML statement. Similarly, in the create, delete, and update methods, PartitionedManager obtains the partition key, verifies the value of the partition key by calling the get_and_check_partition_value function, and adds it to the creation parameters, and finally calls the corresponding method of PartitionedManager to execute the operation.

[0164] In this embodiment, by declaring the ORM partition key to constrain the DML operations of the partitioned table, the use of the partitioned table is simplified, ensuring that the partition key is always correctly set during data operations. In addition, through PartitionedManager and defining the partition configuration in the ORM model, the complex details related to the partitioned table can be hidden, enabling developers to correctly use the database partitioned table without having to understand the underlying database partition implementation principle, providing a more user-friendly experience for developers, and thus improving the application efficiency of accessing large-scale data.

[0165] Please refer to Figure 2 As shown, the present invention also provides a data access system based on the Django framework, and the system includes:

[0166] Parsing module 201: used to obtain the table name, table structure, primary key and partition key of the partition table to be established by parsing the subclass declaration;

[0167] Partition type determination module 202: used to determine the partition type of the partition table to be established by judging the data type of the partition key;

[0168] Table creation statement generation module 203: used to generate partition table creation statements based on the migrate of the Django framework and the partition type;

[0169] Partition table creation module 204: used to execute the partition table creation statement through a database transaction to create the partition table;

[0170] Partition table data operation module 205: used to generate data management statements based on the ORM of the Django framework, wherein the data management statements are used to add, update, filter or delete data records in the partition table.

[0171] It is understandable that if Figure 1 The contents of the data access method embodiment based on the Django framework shown in the figure are all applicable to the data access system embodiment based on the Django framework. The functions specifically implemented by the data access system embodiment based on the Django framework are similar to those in the figure. Figure 1 The data access method embodiment based on the Django framework shown in FIG. 1 is the same as that shown in FIG. 1 , and the beneficial effects achieved are the same as those of FIG. Figure 1 The beneficial effects achieved by the data access method embodiment based on the Django framework shown are also the same.

[0172] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0173] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0174] See also Figure 3 As shown, an embodiment of the present invention further provides a computer device 3, comprising: a memory 302 and a processor 301 and a computer program 303 stored in the memory 302. When the computer program 303 is executed on the processor 301, a data access method based on the Django framework as described in any one of the above methods is implemented.

[0175] The computer device 3 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 It is only an example of computer device 3 and does not constitute a limitation on computer device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0176] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0177] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may also be an external storage device of the computer device 3, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 3. Further, the memory 302 may also include both an internal storage unit of the computer device 3 and an external storage device. The memory 302 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 302 may also be used to temporarily store data that has been output or is to be output.

[0178] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data access method based on the Django framework as described in any one of the above methods is implemented.

[0179] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0180] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data access method based on Django framework, characterized in that: include: By parsing the subclass declaration, obtain the table name, table structure, primary key and partition key of the partition table to be created; Determine the partition type of the partition table to be established by judging the data type of the partition key; Based on the migrate of the Django framework and the partition type, the Django framework is extended, and based on the extended Django framework, a partition table creation statement is generated; Execute the partition table creation statement through a database transaction to create the partition table; The ORM based on the Django framework generates data management statements, wherein the data management statements are used to add, update, filter or delete data records in the partition table.

2. The method according to claim 1, characterized in that The determining the partition type of the partition table to be established by judging the data type of the partition key includes: If there is only one partition key, and the designated column of the partition key is id, the partition type of the partition table is determined to be KEY type; If there is one or more partition keys, and the data type of the designated column of the partition key is INT type or CHART type, then the partition type of the partition table is determined to be HASH type; If there is only one partition key, and the data type of the designated column of the partition key is DATETIME type, then the partition type of the partition table is determined to be TO_DATE type; If the partition type of the partition table cannot be determined by the partition key, a partition key error declaration message is issued.

3. The method according to claim 2, characterized in that The partition table to be established is also provided with an associated attribute, and the associated attribute is used to determine whether the partition table allows foreign key constraints; Before the migration of the Django framework and the partition type, the Django framework is extended, and before a partition table creation statement is generated based on the extended Django framework, the following is included: Based on the partition type, determining an attribute value of the associated attribute; If the partition type is a KEY type, determining that the attribute value of the associated attribute is TRUE; If the partition type is a HASH type or a TO_DATE type, determining that the attribute value of the associated attribute is FALSE; If there is a partition table to be associated and the attribute value of the associated attribute of the partition table is FALSE, an association failure message is issued and generation of a partition table creation statement is prevented.

4. The method according to claim 1, characterized in that The generating of partition table creation statements includes: Generate the first table creation statement based on the Django framework's migrate; Generate a second table creation statement according to the partition type, where the second table creation statement is used to modify the primary key of the first table creation statement to a joint primary key, where the joint primary key includes a primary key and a partition key; Generate a third table creation statement based on the partition type and the joint primary key; A partition table creation statement is generated based on the first table creation statement, the second table creation statement, and the third table creation statement.

5. The method according to claim 1, characterized in that After the partition table creation statement is executed through the database transaction to create the partition table, the method includes: When adding a field to a non-partitioned table, generating a table modification statement and executing the table modification statement; Before executing the table modification statement, determining whether the table modification statement satisfies an online execution operation, wherein the online execution operation supports the database to remain online during the execution process of the table modification statement; If the table modification statement satisfies the online execution operation, a configuration statement is added on the basis of the table modification statement, wherein the configuration statement is used to enable the table modification statement to execute the operation online; If the table modification statement does not satisfy the online execution operation, an error alarm mechanism is triggered.

6. The method according to claim 1, characterized in that The ORM based on the Django framework generates data management statements, which are used to add, update, filter or delete data records in the partition table, including: Based on the data operation requirements, corresponding data operation statements are generated, wherein the data operation statements include data adding statements, data updating statements, data filtering statements and data deleting statements; A corresponding data operation method is created for each data operation, and a data management statement is generated based on the data operation statement and the data operation method to operate the data in the partition table.

7. The method according to claim 6, characterized in that Before the ORM based on the Django framework generates a data management statement, the data management statement is used to add, update, filter or delete data records in the partition table, including: Obtain key information of the partition table, the key information including class name and partition key; Detecting an operation parameter of the data operation statement, and determining whether the operation parameter includes a partition key; If the operation parameters do not contain the partition key, a ValueError exception is raised and the generation of the data management statement is prevented.

8. A data access system based on Django framework, characterized in that: include: Parsing module: used to obtain the table name, table structure, primary key and partition key of the partition table to be created by parsing the subclass declaration; A partition type determination module: used to determine the partition type of the partition table to be established by judging the data type of the partition key; Table creation statement generation module: used to generate partition table creation statements based on the Django framework's migrate and the partition type; A partition table creation module: used to execute the partition table creation statement through a database transaction to create the partition table; Partition table data operation module: used to generate data management statements based on the ORM of the Django framework, and the data management statements are used to add, update, filter or delete data records in the partition table.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images