Database-based data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a database-based data processing method and device, electronic equipment and a storage medium, and the method comprises the steps that in response to a business operation, to-be-processed data and an object configuration file are determined, and first field information contained in the object configuration file is consistent with second field information of a database table corresponding to the business operation; generating a target object according to the to-be-processed data and the object configuration file; and operating the database table according to the target object to obtain a target database table. The target object corresponding to the to-be-processed data can be directly obtained based on the object configuration file without reflection processing of database fields, operation on the database table is rapidly achieved, and the performance of database data processing is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of database technology, and specifically to a data processing method, device, electronic device and storage medium based on a database, wherein the storage medium includes a computer-readable storage medium. Background Art

[0002] Database technology has become the core technology and important foundation of information infrastructure. As the most effective means of data management, database technology has greatly promoted the development of computer applications. Database technology is mainly used to solve non-numerical computing problems in data processing. The main content of data processing is data storage, query, modification, sorting and statistics.

[0003] Among them, the database data is processed by the database operation engine. When processing data, the traditional database operation engine needs to generate a DO file and reflect the database fields based on the DO file, and then perform insert assignment operations to obtain a DO object for encapsulating data read from or written to the database, and transfer data between the business logic layer and the data access layer based on the DO object.

[0004] However, data transfer based on DO objects requires reflection processing of database fields, which leads to performance degradation. Especially in the field of semiconductor equipment, the performance requirements of data processing are particularly important. Summary of the invention

[0005] The embodiments of the present application provide a database-based data processing method, device, electronic device and storage medium, which can improve the performance of database data processing.

[0006] In a first aspect, an embodiment of the present application provides a data processing method based on a database, the method comprising:

[0007] In response to the business operation, determine the data to be processed and the object configuration file, wherein the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation;

[0008] Generate a target object according to the data to be processed and the object configuration file;

[0009] The database table is operated according to the target object to obtain a target database table.

[0010] In a second aspect, an embodiment of the present application further provides a data processing device based on a database, the device comprising:

[0011] A response module, configured to determine, in response to a business operation, data to be processed and an object configuration file, wherein first field information contained in the object configuration file is consistent with second field information of a database table targeted by the business operation;

[0012] A generating module, used for generating a target object according to the data to be processed and the object configuration file;

[0013] The operation module is used to operate the database table according to the target object to obtain a target database table.

[0014] Optionally, in some embodiments of the present application, the object configuration file is configured with field operation permissions for each of the first field information;

[0015] The generating a target object according to the data to be processed and the object configuration file comprises:

[0016] Determine the field operation authority corresponding to each of the data to be processed from the object configuration file;

[0017] Verify each of the data to be processed according to the field operation permissions to obtain a field permission verification result;

[0018] If the field permission check result includes a check passed, the data to be processed is inserted into the object configuration file to obtain a target object.

[0019] Optionally, in some embodiments of the present application, the object configuration file is configured with table operation permissions for the database table;

[0020] The operating the database table according to the target object comprises:

[0021] Determine a table operation type based on the business operation;

[0022] Extracting the table operation permission from the object configuration file;

[0023] Verify the table operation type according to the table operation authority to obtain a table operation authority verification result;

[0024] According to the table operation permission verification result, control is performed to operate the database table based on the target object.

[0025] Optionally, in some embodiments of the present application, controlling the operation of the database table based on the target object according to the table operation permission verification result includes:

[0026] If the table operation permission verification result includes verification passed, a structured query statement is generated according to the target object, and the structured query statement is executed to operate the database table.

[0027] Optionally, in some embodiments of the present application, the object configuration file is configured with version attribute information for the data version;

[0028] The operating the database table according to the target object comprises:

[0029] Acquire the current version information of the database table from the version attribute information;

[0030] The database table is operated according to the target object, and the current version information is updated to obtain the target version information.

[0031] Optionally, in some embodiments of the present application, a database environment identifier for a database type is configured in the object configuration file, the target object includes at least one, and each target object corresponds to a different database environment identifier;

[0032] The operating the database table according to the target object comprises:

[0033] Packing each of the target objects into a target database operation engine;

[0034] Performing transaction grouping processing on each of the target objects based on the database environment identifier by the target database operation engine to obtain at least one transaction to be processed, wherein the database environment identifier of at least one target object corresponding to each of the transactions to be processed is the same;

[0035] For each transaction to be processed, generating a structured query statement based on at least one target object combination of the transaction to be processed;

[0036] Each database connection is obtained according to each database environment identifier, and the structured query statement of each transaction to be processed is processed respectively according to each database connection.

[0037] Optionally, in some embodiments of the present application, after operating the database table according to the target object, the method further includes:

[0038] Record operation logs through the target database operation engine;

[0039] In response to a log operation on the operation log, performing target processing on the operation log;

[0040] The log operation includes at least one of log query or log statistics.

[0041] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the above-mentioned database-based data processing method when executed by the processor.

[0042] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned database-based data processing method are implemented.

[0043] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0044] In response to a business operation, an embodiment of the present application determines data to be processed and an object configuration file, wherein the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, generates a target object according to the data to be processed and the object configuration file, operates the database table according to the target object, and obtains a target database table.

[0045] Among them, since the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, there is no need to perform reflection processing of the database field. The target object corresponding to the data to be processed can be obtained directly based on the object configuration file, and the operation of the database table can be quickly implemented, thereby improving the performance of database data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a schematic diagram of a scenario in which a terminal device according to an embodiment of the present application executes the database-based data processing method;

[0048] Figure 2 It is a flowchart of a data processing method based on a database provided in an embodiment of the present application;

[0049] Figure 3is another flowchart of the data processing method based on the database provided in an embodiment of the present application;

[0050] Figure 4 is a schematic diagram of the structure of a data processing device based on a database provided in an embodiment of the present application;

[0051] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0053] The embodiments of the present application provide a data processing method, device, electronic device and computer-readable storage medium based on a database. Specifically, the embodiments of the present application provide a data processing device suitable for an electronic device, which is used to process database data to improve the performance of database data processing. Specifically, the electronic device includes a terminal device or a server, and the terminal device includes but is not limited to mobile phones, laptops, desktop computers and other devices. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms and other basic cloud computing services, etc. The server can be directly or indirectly connected via wired or wireless communication.

[0054] See also Figure 1 , Figure 1 : is a schematic diagram of a scenario in which a terminal device according to an embodiment of the present application executes the data processing method based on a database, wherein the specific execution process of the terminal device executing the data processing method based on a database is as follows:

[0055] The user performs business operations through the front-end page on the terminal device 10, and the terminal device 10 determines the data to be processed and the object configuration file in response to the business operation, wherein the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, generates a target object according to the data to be processed and the object configuration file, operates the database table according to the target object, and obtains the target database table.

[0056] In summary, since the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, there is no need to perform reflection processing of the database field. The target object corresponding to the data to be processed can be obtained directly based on the object configuration file, and the operation of the database table can be quickly implemented, thereby improving the performance of database data processing.

[0057] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0058] See also Figure 2 , Figure 2 A flowchart of a data processing method based on a database provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the flowchart. Specifically, the process of the data processing method based on a database specifically includes:

[0059] 101. In response to a business operation, determine data to be processed and an object configuration file, wherein first field information included in the object configuration file is consistent with second field information of a database table targeted by the business operation.

[0060] Among them, business operations are usually generated by users based on business needs through web front-end page operations, wherein the business operations include operations such as adding, deleting, modifying, querying, sorting, and paging of data in database tables. For example, users insert data, delete data, edit and modify data in the database through the front-end page.

[0061] The data to be processed is the data targeted by the business operation. For example, for a business operation such as adding data to insert data into a database table, the data to be inserted into the database table is the data to be processed. For a business operation such as deleting data, the data to be deleted from the database table is the data to be processed.

[0062] The object configuration file is a configuration file for a target object, and the target object refers to an object used to encapsulate data read from or written to a database. For example, the target object includes a DTO object. In an embodiment of the present application, the object configuration file is configured with field information consistent with the database table field, and the field information is configured by code editing.

[0063] Since a database table usually includes multiple objects, multiple object configuration files can be configured. For example, a corresponding object configuration file is configured for each database table, and the object configuration file corresponding to the current business operation is obtained by file identification. Each object configuration file can be distinguished by an id or name (Name), that is, the object configuration file includes at least basic information and field information. The basic information includes:

[0064] DTO Id: unique identifier, corresponding to the database table name.

[0065] Name: The display name of the object configuration file for easy identification.

[0066] Env: Database environment identifier, bound to a specific database instance.

[0067] The field information mainly refers to the specific fields of the database table. For example, the field information includes identification, name, age, gender, etc. (It can be understood that in a specific database, the field name is usually expressed in English, and Chinese is used as an example here).

[0068] For example, take an object configuration file (such as a DTO configuration file) defined in JSON format as follows:

[0069] DTO Id: "DO_SYS_CLUSTER_E";

[0070] Name: "System Cluster";

[0071] Env:"Production_DB".

[0072] It should be noted that in the traditional solution, the data read from or written to the database is encapsulated based on data objects (DO, Data Object). However, since the field names in the DO objects are inconsistent with the names of the database table fields, it is necessary to reflect the database fields when transferring data based on the DO objects, that is, reflect the fields of the business data generated by the front end into database fields, establish a correspondence between the front-end fields and the database fields, and then facilitate the operation of the data in the database table through the front-end data.

[0073] However, since field reflection processing is required when transferring data based on DO objects, the processing efficiency is low and the processing performance of database data is reduced. Based on this, in an embodiment of the present application, an object configuration file consistent with the field in the database table is configured, and reflection processing is no longer required when generating data objects, thereby improving data processing efficiency and improving the performance of database data processing.

[0074] 102. Generate a target object according to the data to be processed and the object configuration file.

[0075] That is, according to the requirements of the object configuration file, a target object corresponding to the data to be processed is generated.

[0076] Among them, since the first field information in the object configuration file is consistent with the second field information in the database table, the fields in the generated target object are consistent with the fields in the database table. Therefore, there is no need to perform reflection processing on the target object, and the target object can be directly used for data transfer.

[0077] 103. Operate the database table according to the target object to obtain a target database table.

[0078] The target database table is obtained after the target object operates on the database table, that is, the target database table is obtained after the data is updated.

[0079] Among them, since the target object contains the data to be processed corresponding to the business operation, the database table can be directly operated based on the target object, that is, the database table can be updated using the data to be processed, for example, the data to be processed can be added to the database table, or the data in the database table can be modified using the data to be processed.

[0080] In summary, in the embodiment of the present application, since the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, there is no need to perform reflection processing of the database field. The target object corresponding to the data to be processed can be obtained directly based on the object configuration file, and the operation of the database table can be quickly implemented, thereby improving the performance of database data processing.

[0081] It is understandable that in order to ensure the validity and security of data and improve the accuracy of database table operations, the data to be processed can be verified. Usually, the verification of database table fields mainly depends on the design of data table fields, for example, designing them as non-empty or read-only attributes. However, this design can only be performed when the database operation engine receives the data, that is, when it is actually about to be updated to the database table. There is a certain lag, and if the final data does not meet the attribute requirements, it will lead to invalid transmission of the data object.

[0082] Therefore, in an embodiment of the present application, the attribute requirements (field operation permissions) of the database table fields are set in the object configuration file, so that the target object is generated by the data to be processed through the attribute requirements, and then the generation of the target object can be canceled when the attribute requirements are not met, and then the transfer of the target object is not required. That is, optionally, in some embodiments of the present application, the object configuration file is configured with field operation permissions for each of the first field information, and the step of "generating the target object according to the data to be processed and the object configuration file" includes:

[0083] Determine the field operation authority corresponding to each of the data to be processed from the object configuration file;

[0084] Verify each of the data to be processed according to the field operation permissions to obtain a field permission verification result;

[0085] If the field permission check result includes a check passed, the data to be processed is inserted into the object configuration file to obtain a target object.

[0086] Among them, table field operation permissions refer to the operation behavior permissions for the field, which is also the definition of the table field. For example, the field operation permissions include the field data type, whether it is allowed to be empty, whether it is a primary key or whether it is a read-only field, etc. It is expressed as:

[0087] Name: Field name.

[0088] Type: Field data type.

[0089] Empty: Whether to allow empty.

[0090] PK: Whether it is the primary key field.

[0091] ReadOnly: Whether it is a read-only field.

[0092] For example, take an object configuration file (such as a DTO configuration file) defined in JSON format as follows:

[0093] DTO Id: "DO_SYS_CLUSTER_E";

[0094] Name: "System Cluster";

[0095] Env:"Production_DB";

[0096] Fields:

[0097] -Name:"SID",Type:"int",Empty:false,PK:true,

[0098] ReadOnly:false;

[0099] -Name:"AUTO_TYPE",Type:"string",Empty:true,

[0100] ReadOnly: true;

[0101] -Name: "SN", Type: "int", Empty: true,

[0102] OpeLevel:3.

[0103] That is, for a field named "SID", its field data type is "int", cannot be empty, is the primary key, and is not read-only; for a field named "AUTO_TYPE", its field data type is "string", can be empty, and is read-only.

[0104] Among them, verifying the data to be processed according to the field operation permission refers to: determining whether the field corresponding to the data to be processed can be set to empty, whether the data type of the data to be processed meets the field requirements, etc.

[0105] Among them, by configuring the operation permissions for each database table field in the object configuration file, field permission verification can be performed when generating the target object, and data validity can be judged in advance. And by configuring field operation permissions based on the object configuration file, the convenience of data operation is also improved. Users or developers do not need to operate the database table structure to change field properties, which improves efficiency.

[0106] Among them, in the embodiment of the present application, since the object configuration file contains field information consistent with the database table, the data to be processed can be directly inserted into the object configuration file to obtain the target object, that is, the database table fields and the data to be processed corresponding to the field mapping are stored in the target object.

[0107] Similarly, the object configuration file may also store operation permissions for the database table, i.e., table operation permissions, so as to be verified when generating the target object. That is, optionally, in some embodiments of the present application, the object configuration file is configured with table operation permissions for the database table, and the step of "operating the database table according to the target object" includes:

[0108] Determine a table operation type based on the business operation;

[0109] Extracting the table operation permission from the object configuration file;

[0110] Verify the table operation type according to the table operation authority to obtain a table operation authority verification result;

[0111] According to the table operation permission verification result, control is performed to operate the database table based on the target object.

[0112] The table operation permission refers to the operation permission on the database table, which includes but is not limited to one of read-only, read-write, or read-write-delete, etc., which respectively means that data can only be queried from the database table, can be queried and written, or can be queried, written, and deleted.

[0113] Among them, the table operation type refers to the operation type of the business operation on the database table, including addition, deletion, modification and query, etc., which is determined by the user through the web page based on specific business needs. For example, when the user clicks the Add control in the page, the Add table operation type is generated, and when the user clicks the Delete control in the page, the Delete table operation type is generated.

[0114] Among them, by configuring table operation permissions in the object configuration file and verifying the table operation type of the business operation based on the table operation permissions, operation verification processing can be performed when the data object to be transferred is generated to avoid invalid data transfer caused by invalid operations.

[0115] Among them, in the embodiments of the present application, a structured query statement (SQL statement) can be generated through the target object, and a database table can be operated based on the structured query statement, that is, optionally, in some embodiments of the present application, the step of "controlling the operation of the database table based on the target object according to the table operation permission verification result" includes:

[0116] If the table operation permission verification result includes verification passed, a structured query statement is generated according to the target object, and the structured query statement is executed to operate the database table.

[0117] For example, an SQL statement is generated based on the field information corresponding to the database table and the data to be processed corresponding to the field information. It should be noted that when generating an SQL statement, it is also necessary to combine the table operation type, for example, to generate SQL for adding, inserting, deleting, or updating SQL for the database table.

[0118] Among them, in the embodiment of the present application, the processing of the data in the database table includes the processing of adding, deleting, modifying and checking a single piece of data, and also includes batch processing, and correspondingly, also includes dynamic query processing, that is, the processing of the data in the database table includes:

[0119] 1. Single record operations include: insert, update, delete, query, etc.

[0120] For example, for the insert operation of a single data record:

[0121] The definition of the insert operation is: inserting the field value corresponding to the target object (the data to be processed corresponding to the field) into the database table.

[0122] The method is defined as: func Insert(dto*DTO)error, where (dto*DTO) represents the target object, that is, the DTO object.

[0123] The implementation logic is: generate an INSERT SQL statement based on the field definition of DTO; check whether the required field (Empty: false) has a value, that is, if the field attribute is not allowed to be empty, there is a field non-empty check; execute the SQL statement to insert the record.

[0124] For another example, for the update operation of a single data:

[0125] The update operation is defined as: locating the record based on the primary key field (PK) and updating the specified field.

[0126] Method definition: func Update(dto*DTO)error, where (dto*DTO) represents the target object, that is, the DTO object.

[0127] Implementation logic: Check whether the primary key field (PK) has a value; check the read-only field (ReadOnly: true) to prevent modification of restricted fields; dynamically generate UPDATE SQL statements; execute SQL statements to update records.

[0128] For another example, to delete a single piece of data:

[0129] The definition of the delete operation is: locate and delete records based on the primary key field.

[0130] Method definition: func Delete(dto*DTO)error, where (dto*DTO) represents the target object, that is, the DTO object.

[0131] Implementation logic: Check whether the primary key field has a value; dynamically generate a DELETE SQL statement based on DTO; execute the SQL statement to delete the record.

[0132] For example, query operation for a single data:

[0133] The query operation is defined as: retrieving a single record by primary key field or other conditions.

[0134] Method definition: func Query(dto*DTO,fields...string)(map[string]interface{},error), where (dto*DTO) represents the target object, that is, the DTO object.

[0135] Implementation logic: dynamically build SELECT SQL statements; optionally return specified fields; execute SQL and return query results.

[0136] 2. Batch operation:

[0137] Batch insert, update, delete (Batch Operations), multi-table, multi-database operations, etc.

[0138] For example, for bulk inserts in a bulk operation:

[0139] The definition of batch insert is: batch insert multiple records into the same database table.

[0140] Method definition: func BatchInsert(dtos[]*DTO)error, where (dtos[]*DTO) represents multiple target objects, that is, multiple DTO objects.

[0141] Implementation logic: Verify the field value of each DTO object; dynamically generate batch INSERT statements (such as using INSERT INTO table VALUES(...)); execute SQL statements to insert multiple records.

[0142] For another example, for batch updates in batch operations:

[0143] The batch update is defined as: batch update of multiple records, supporting different primary key positioning.

[0144] Method definition: func BatchUpdate(dtos[]*DTO)error, where (dtos[]*DTO) represents multiple target objects, that is, multiple DTO objects.

[0145] Implementation logic: Verify the primary key and field permissions of each DTO; dynamically generate multiple UPDATE SQL statements; execute SQL in batches.

[0146] For another example, for batch deletion in a batch operation:

[0147] The definition of batch deletion is: batch delete multiple records.

[0148] Method definition: func BatchDelete(dtos[]*DTO)error, where (dtos[]*DTO) represents multiple target objects, that is, multiple DTO objects.

[0149] Implementation logic: Verify the primary key of each DTO; dynamically generate multiple DELETE SQL statements; execute SQL in batches.

[0150] 3. Dynamic query:

[0151] Conditional query (QueryWithCondition), paging and sorting support (PaginatedQuery), etc.

[0152] For example, for conditional queries in dynamic queries:

[0153] The definition of this conditional query is: query multiple records based on dynamic conditions.

[0154] Method definition: func QueryWithCondition(dto*DTO,conditions[]*Condition,limitint)([]map[string]interface{},error), where (dtos[]*DTO) represents multiple target objects, that is, multiple DTO objects.

[0155] Implementation logic: dynamically build WHERE conditions; support multi-condition combination query; specify the number of query results (limit); execute SQL statements and return results.

[0156] For another example, for paging queries in dynamic queries:

[0157] The definition of the paging query is: execute the paging query and return the results.

[0158] Method definition: func PaginatedQuery(dto*DTO,page,pageSize int,orderBystring)([]map[string]interface{},error), where (dtos[]*DTO) represents multiple target objects, that is, multiple DTO objects.

[0159] Implementation logic: Calculate the offset based on page and pageSize; dynamically generate SQL statements containing LIMIT and OFFSET; support specifying sort fields (orderBy); execute SQL and return paginated results.

[0160] Among them, in the embodiment of the present application, multi-terminal operations are controlled by setting version attribute information in the object configuration file to prevent data conflicts, that is, optionally, in some embodiments of the present application, the object configuration file is configured with version attribute information for the data version, and the step of "operating the database table according to the target object" includes:

[0161] Acquire the current version information of the database table from the version attribute information;

[0162] The database table is operated according to the target object, and the current version information is updated to obtain the target version information.

[0163] Among them, the version attribute information records the version of the data in the database table, and the current version information is the current version of the data in the database table. By obtaining the version attribute information and updating the version when operating the database table, multi-terminal operation control is achieved to prevent data conflicts.

[0164] For example, taking the multi-terminal modification or multi-terminal deletion operation of a single piece of data as an example, after the front end queries the data, it also obtains the current version information corresponding to the data. After the target object is generated through the object configuration file, the target object and the current version information are handed over to the database operation engine for processing. That is, the database operation engine checks the version. If the data has a version and the version is consistent with the version of the current data in the database table, the current version information is used as the SQL condition (such as the where condition). If it is an update operation, the current version information is also updated to the target version information, where the target version information is the next version of the current version information; if it is a deletion operation, the current version information is not processed, that is, the data has been deleted, and the field of the version attribute information contained in the data no longer exists.

[0165] Similarly, for multi-terminal batch processing operations on multiple data, you can refer to it and understand that the version of the data is recorded through version attribute information. That is, in order to prevent multiple users from operating on the same data, which leads to ambiguity in the resulting data, the embodiment of the present application adds version control to ensure that users can perceive that the data has been modified and cannot be directly updated or deleted.

[0166] Optionally, in an embodiment of the present application, by setting a database environment identifier in an object configuration file, multi-transaction processing across databases can be implemented, that is, optionally, in some embodiments of the present application, the object configuration file is configured with a database environment identifier for a database type, the target object includes at least one, each target object corresponds to a different database environment identifier, and the step of "operating the database table according to the target object" includes:

[0167] Packing each of the target objects into a target database operation engine;

[0168] Performing transaction grouping processing on each of the target objects based on the database environment identifier by the target database operation engine to obtain at least one transaction to be processed, wherein the database environment identifier of at least one target object corresponding to each of the transactions to be processed is the same;

[0169] For each transaction to be processed, generating a structured query statement based on at least one target object combination of the transaction to be processed;

[0170] Each database connection is obtained according to each database environment identifier, and the structured query statement of each transaction to be processed is processed respectively according to each database connection.

[0171] Among them, transactions include operation items, including single-database transactions and cross-database transaction groups. Specifically:

[0172] 1. Single Database Transaction: If the target object of the operation belongs to one database, you can directly start the transaction and execute SQL.

[0173] That is, a single-database transaction is defined as executing a transaction in a single database.

[0174] Method definition: func ExecuteInTransaction(dtos[]*DTO,actions[]Action)error;

[0175] Implementation logic: Start a database transaction; execute multiple SQL operations (Insert / Update / Delete); commit the transaction if the operation is successful; roll back if it fails.

[0176] 2. Cross-Database Transactions: multiple target objects of the operation belong to different databases, and they are grouped according to the configured Env and transaction processing and storage are enabled separately.

[0177] The cross-database transaction group is defined as: executing transactions across multiple databases.

[0178] Method definition: func ExecuteAcrossDatabases(groups map[string][]*DTO)error;

[0179] Implementation logic: Group operations according to DTO's Env; start an independent transaction for each group; execute SQL statements in group order to ensure transaction independence.

[0180] For example, scene processing example:

[0181] Taking the processing of semiconductor equipment information and operation record information as an example, it is assumed that there are three target objects, namely: equipment information storage and equipment status storage, corresponding to the Oracle database; operation information storage, corresponding to the PostgreSQL database. At this time, the new equipment is running, recording the equipment operation information and equipment information. The specific storage process is as follows:

[0182] The processing of the first operation information includes obtaining Env; assembling SQL with the same Env and executing them together, that is, assembling device information and device status SQL together, and distinguishing execution with the database ";"; Env obtains the corresponding database connection; opens a transaction; executes SQL; closes the transaction. Then, the next operation information is processed according to the above steps.

[0183] Accordingly, when performing cross-database multi-transaction processing based on the database environment identifier, the corresponding database connection object can be matched based on the database environment identifier to perform group storage management through the database connection object. The management of database connections also includes dynamically managing database connections to optimize resource utilization in high-concurrency scenarios. Specifically, it includes:

[0184] (1) Connection pool management:

[0185] a. Each database instance is equipped with an independent connection pool.

[0186] b. Dynamically adjust the number of connections to support peak loads.

[0187] (2)Multiple database support:

[0188] Use the Env property of DTO to manage connections to multiple database instances.

[0189] (3) Cross-database operations:

[0190] Automatically coordinate transaction execution across multiple databases.

[0191] Take the complex supply chain in the semiconductor industry as an example. The supply chain includes: raw material procurement, supplier management, product transportation, etc.:

[0192] There may be a scenario where suppliers use their own databases to record material delivery times, and the company's internal system needs to integrate this data and link it with the production system.

[0193] The specific requirements are as follows: one database manages internal semiconductor equipment information (such as Oracle). Another database stores equipment operation records (such as PostgreSQL). When processing business, it is necessary to store equipment information and equipment operation records at the same time.

[0194] Based on the above scenarios and requirements, traditional database operation engines need to execute different database connections to perform data storage operations separately.

[0195] In the embodiment of the present application, the database operation engine (getdbo) can be directly handed over for processing, and the database operation engine can perform group management according to the configured database environment identifier Env in a batch processing manner. The specific process is as follows:

[0196] (1) Before the system is started, add the DB configuration item in the system configuration file (e.g., cfg configuration file). The configuration item has the following main contents:

[0197] a. DB connection information, including connection address, number of connections, etc.

[0198] b. The unique Env (environment) identifier of the DB library.

[0199] (2) As mentioned above, when configuring the target object, there is a database environment identifier Env attribute, which has a corresponding relationship with the DB configuration.

[0200] (3) When performing multiple types of DTO data operations on the DB, the corresponding DB connection object is found according to the configured Env for group storage management.

[0201] Optionally, in an embodiment of the present application, for operations on the database, operation logs are recorded to facilitate log query, statistics and other operations. That is, optionally, in some embodiments of the present application, after the step of "operating the database table according to the target object", the method further includes:

[0202] Record operation logs through the target database operation engine;

[0203] In response to a log operation on the operation log, performing target processing on the operation log;

[0204] The log operation includes at least one of log query or log statistics.

[0205] The operation log is a log generated when operating the database, including the operation type, execution time, and user information, which is used to record what operations are performed on the database. When there are many logs, the logs can be managed by log query or log statistics.

[0206] Correspondingly, in the embodiment of the present application, abnormal operations on the database can also be recorded, that is, error information of the operation can be recorded to locate the abnormal cause of the operation, etc.

[0207] In summary, in the embodiment of the present application, since the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, there is no need to perform reflection processing of the database field. The target object corresponding to the data to be processed can be obtained directly based on the object configuration file, and the operation of the database table can be quickly implemented, thereby improving the performance of database data processing.

[0208] Among them, the security verification of data operations is achieved through the configuration of field operation permissions and table field operation permissions in the object configuration file, thereby improving the security of database table data.

[0209] Among them, by configuring the version attribute information in the object configuration file, data control is achieved during multi-terminal operations to prevent data conflicts.

[0210] Among them, through the configuration of the database environment identifier in the object configuration file, group processing of multiple transactions across databases based on the database environment identifier is achieved, thereby supporting cross-database transactions.

[0211] In order to highlight the superiority of the embodiments of the present application, a comparative analysis is conducted with a traditional database operation engine (such as GORM) to test the performance and effects in different scenarios.

[0212] Scenario 1: Single batch insertion.

[0213] Operation Engine Number of connections Number of rows written Write data once time consuming GORM 10 2000 2000 152.897ms getdbo 10 2000 2000 112.662ms

[0214] Result analysis:

[0215] In batch insertion, since the fields configured in the object configuration file correspond one-to-one to the fields in the database table, the reflection consumption is reduced, and the SQL can be directly spliced ​​out and submitted to the database at one time, which reduces the time consumption of SQL splicing and parameter processing at runtime and improves performance by about 26%.

[0216] Scenario 2: High concurrent writing.

[0217] Operation Engine Number of connections Concurrency Write data once time consuming GORM 10 20 1 549.219ms getdbo 10 20 1 249.338ms

[0218] Result analysis:

[0219] In high-concurrency scenarios, the embodiments of the present application utilize the configuration features of the object configuration file (such as batch operations) to increase the data operation speed, significantly reduce the delay caused by resource competition, and improve performance by more than 50%.

[0220] Scenario 3: Cross-database batch operations.

[0221] Operation Engine Number of databases Bulk Operation Entries Cross-database transaction support GORM 3 1000 Not supported getdbo 3 1000 support

[0222] Result analysis:

[0223] Traditional engines have limited support for cross-database transactions, but the embodiments of the present application achieve consistency and efficiency of cross-database operations through the group execution strategy of the object configuration file.

[0224] In order to more clearly understand the data processing method based on the database in the embodiment of the present application, the data processing method in the embodiment of the present application is described in relatively complete steps. Figure 3 , Figure 3 is another flow chart of a data processing method based on a database provided in an embodiment of the present application, wherein the data processing method based on a database includes:

[0225] 111. In response to the business operation, obtaining the object configuration file through the database operation engine;

[0226] 112. Assign the to-be-processed data corresponding to the business operation to the object configuration file;

[0227] 113. During the assignment process, verification is performed based on the field operation permissions and table operation permissions in the object configuration file;

[0228] 114. After the verification is passed, the target object is assigned;

[0229] 115. Performing an update operation on the database table based on the target object and the current version attribute information of the database table through the database operation engine;

[0230] 116. In the update operation, transaction management is performed based on the database environment identifier in the target object, and the transaction is started;

[0231] 117. Execute the structured query statement corresponding to the current transaction;

[0232] 118. Close the transaction and execute the next transaction;

[0233] 119. When each transaction is executed, record the operation log and exception log.

[0234] In summary, the embodiment of the present application realizes all-round security verification at the operation level and field level through deep integration of object configuration files, and effectively avoids multi-terminal conflicts through version control. Especially in the semiconductor field, where equipment operation requirements are strict, the advantages of the solution of the embodiment of the present application are becoming more and more obvious.

[0235] In order to better implement the database-based data processing method of the present application, the present application also provides a database-based data processing device based on the above-mentioned database-based data processing method. The meanings of the terms are the same as those in the above-mentioned database-based data processing method, and the specific implementation details can refer to the description in the method embodiment.

[0236] See also Figure 4 , Figure 4 : is a schematic diagram of the structure of a data processing device based on a database provided in an embodiment of the present application. The data processing device based on a database is applied to a terminal device. The data processing device based on a database may be specifically as follows:

[0237] A response module 201 is used to determine the data to be processed and the object configuration file in response to the business operation, wherein the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation;

[0238] A generating module 202, configured to generate a target object according to the data to be processed and the object configuration file;

[0239] The operation module 203 is used to operate the database table according to the target object.

[0240] Optionally, in some embodiments of the present application, the object configuration file is configured with field operation permissions for each of the first field information;

[0241] The generating a target object according to the data to be processed and the object configuration file comprises:

[0242] Determine the field operation authority corresponding to each of the data to be processed from the object configuration file;

[0243] Verify each of the data to be processed according to the field operation permissions to obtain a field permission verification result;

[0244] If the field permission check result includes a check passed, the data to be processed is inserted into the object configuration file to obtain a target object.

[0245] Optionally, in some embodiments of the present application, the object configuration file is configured with table operation permissions for the database table;

[0246] The operating the database table according to the target object comprises:

[0247] Determine a table operation type based on the business operation;

[0248] Extracting the table operation permission from the object configuration file;

[0249] Verify the table operation type according to the table operation authority to obtain a table operation authority verification result;

[0250] According to the table operation permission verification result, control is performed to operate the database table based on the target object.

[0251] Optionally, in some embodiments of the present application, controlling the operation of the database table based on the target object according to the table operation permission verification result includes:

[0252] If the table operation permission verification result includes verification passed, a structured query statement is generated according to the target object, and the structured query statement is executed to operate the database table.

[0253] Optionally, in some embodiments of the present application, the object configuration file is configured with version attribute information for the data version;

[0254] The operating the database table according to the target object comprises:

[0255] Acquire the current version information of the database table from the version attribute information;

[0256] The database table is operated according to the target object, and the current version information is updated to obtain the target version information.

[0257] Optionally, in some embodiments of the present application, a database environment identifier for a database type is configured in the object configuration file, the target object includes at least one, and each target object corresponds to a different database environment identifier;

[0258] The operating the database table according to the target object comprises:

[0259] Packing each of the target objects into a target database operation engine;

[0260] Performing transaction grouping processing on each of the target objects based on the database environment identifier by the target database operation engine to obtain at least one transaction to be processed, wherein the database environment identifier of at least one target object corresponding to each of the transactions to be processed is the same;

[0261] For each transaction to be processed, generating a structured query statement based on at least one target object combination of the transaction to be processed;

[0262] Each database connection is obtained according to each database environment identifier, and the structured query statement of each transaction to be processed is processed respectively according to each database connection.

[0263] Optionally, in some embodiments of the present application, after operating the database table according to the target object, the method further includes:

[0264] Record operation logs through the target database operation engine;

[0265] In response to a log operation on the operation log, performing target processing on the operation log;

[0266] The log operation includes at least one of log query or log statistics.

[0267] In the embodiment of the present application, the response module 201 determines the data to be processed and the object configuration file in response to the business operation, wherein the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, the generation module 202 generates a target object according to the data to be processed and the object configuration file, and the operation module 203 operates the database table according to the target object.

[0268] Among them, in the embodiment of the present application, since the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, there is no need to perform reflection processing of the database field. The target object corresponding to the data to be processed can be obtained directly based on the object configuration file, and the operation of the database table can be quickly implemented, thereby improving the performance of database data processing.

[0269] In addition, the present application also provides an electronic device, such as Figure 5 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0270] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0271] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0272] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing based on the database by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0273] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0274] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0275] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby implementing the steps in any one of the database-based data processing methods provided in the embodiments of the present application.

[0276] In response to a business operation, an embodiment of the present application determines data to be processed and an object configuration file, wherein the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, generates a target object based on the data to be processed and the object configuration file, and operates the database table based on the target object.

[0277] Among them, since the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation, there is no need to perform reflection processing of the database field. The target object corresponding to the data to be processed can be obtained directly based on the object configuration file, and the operation of the database table can be quickly implemented, thereby improving the performance of database data processing.

[0278] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0279] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0280] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any database-based data processing method provided in the present application.

[0281] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0282] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0283] Since the instructions stored in the computer-readable storage medium can execute the steps in any database-based data processing method provided in the present application, the beneficial effects that can be achieved by any database-based data processing method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0284] The above is a detailed introduction to a database-based data processing method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method based on a database, characterized in that: The method comprises: In response to the business operation, determine the data to be processed and the object configuration file, wherein the first field information contained in the object configuration file is consistent with the second field information of the database table targeted by the business operation; Generate a target object according to the data to be processed and the object configuration file; The database table is operated according to the target object to obtain a target database table.

2. The method for data processing based on a database according to claim 1, characterized in that: The object configuration file is configured with field operation permissions for each of the first field information; The generating a target object according to the data to be processed and the object configuration file comprises: Determine the field operation authority corresponding to each of the data to be processed from the object configuration file; Verify each of the data to be processed according to the field operation permissions to obtain a field permission verification result; If the field permission check result includes a check passed, the data to be processed is inserted into the object configuration file to obtain a target object.

3. The method for data processing based on a database according to claim 1, characterized in that: The object configuration file is configured with table operation permissions for the database table; The operating the database table according to the target object comprises: Determine a table operation type based on the business operation; Extracting the table operation permission from the object configuration file; Verify the table operation type according to the table operation authority to obtain a table operation authority verification result; According to the table operation permission verification result, control is performed to operate the database table based on the target object.

4. The method for data processing based on a database according to claim 3, characterized in that: The controlling the operation of the database table based on the target object according to the table operation permission verification result includes: If the table operation permission verification result includes verification passed, a structured query statement is generated according to the target object, and the structured query statement is executed to operate the database table.

5. The method for data processing based on a database according to claim 1, characterized in that: The object configuration file is configured with version attribute information for the data version; The operating the database table according to the target object comprises: Acquire the current version information of the database table from the version attribute information; The database table is operated according to the target object, and the current version information is updated to obtain the target version information.

6. The method for data processing based on a database according to claim 1, characterized in that: The object configuration file is configured with a database environment identifier for the database type, the target object includes at least one, and each target object corresponds to a different database environment identifier; The operating the database table according to the target object comprises: Packing each of the target objects into a target database operation engine; Performing transaction grouping processing on each of the target objects based on the database environment identifier by the target database operation engine to obtain at least one transaction to be processed, wherein the database environment identifier of at least one target object corresponding to each of the transactions to be processed is the same; For each transaction to be processed, generating a structured query statement based on at least one target object combination of the transaction to be processed; Each database connection is obtained according to each database environment identifier, and the structured query statement of each transaction to be processed is processed respectively according to each database connection.

7. The method for data processing based on a database according to claim 1, characterized in that: After operating the database table according to the target object, the method further includes: Record operation logs through the target database operation engine; In response to a log operation on the operation log, performing target processing on the operation log; The log operation includes at least one of log query or log statistics.

8. A data processing device based on a database, characterized in that: The device comprises: A response module, configured to determine, in response to a business operation, data to be processed and an object configuration file, wherein first field information contained in the object configuration file is consistent with second field information of a database table targeted by the business operation; A generating module, used for generating a target object according to the data to be processed and the object configuration file; The operation module is used to operate the database table according to the target object to obtain a target database table.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the database-based data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the database-based data processing method according to any one of claims 1 to 7 are implemented.

Images