A method for processing data in batches based on memory

Abstract

A method for batch processing data in memory includes the following steps: S1, the front end assembles the data to be saved, removes data that already exists but is no longer needed, and sends a save request to memory; S2, a pre-set data set; S3, the back end receives the request parameters and queries all existing related data in the database according to the business logic; S4, the deletion flags of the query results data are set to deleted in batches, and the data is added to the update class set; S5, the data is parsed and modified in memory; S6, after the data is processed in memory, the data is persisted according to the update class data set and the addition class data set, and the database batch update operation and batch add operation are executed respectively.

Description

Technical Field

[0001] This invention relates to the field of software development, and more particularly to a method for batch processing data based on memory. Background Technology

[0002] Typically, Java backend interfaces have a single responsibility in software development: either handling adding or modifying data, or deleting data. To achieve the same function, the frontend needs to determine and call the corresponding interfaces separately, resulting in numerous calls and complex logic. Now, the backend provides a method for batch processing requests that include adding, modifying, and deleting data based on memory, aggregating multiple operations into one interface to simplify frontend development.

[0003] This engineering problem can be abstracted as follows: For data generated by a certain function, it may need to be persisted to the database, updated, or deleted. Calling multiple interfaces to process this data is inefficient, involves a large number of calls, and has complex logic. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for batch data processing based on memory.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for batch processing data based on memory, comprising the following steps:

[0006] S1, the front end assembles the data that needs to be saved. Data that already exists in the data but is no longer needed needs to be removed, and a request to save the data is sent to memory.

[0007] The memory may be a hard disk, a database, etc.

[0008] S2, Preset data set;

[0009] Predefine the data set for the business data that needs to be saved;

[0010] The data set includes two types: an update-type set and an addition-type set.

[0011] S3: The backend receives the request parameters and queries all existing related data in the database according to the business requirements.

[0012] S4, batch set the deletion flag of the query result data to deleted, and add the data to the update class set;

[0013] S5 parses and modifies the data in memory;

[0014] Parse the data and loop through it, checking if the primary key id of the data is null;

[0015] S6: After the data is processed in memory, the data is persisted according to the update-type data set and the addition-type data set, and the database batch update operation and batch addition operation are executed respectively.

[0016] Preferably, step S4 further includes the following sub-steps,

[0017] S4-1, query the MySQL database for data that already exists and has not been deleted;

[0018] S4-2, mark the retrieved data as deleted in memory first;

[0019] S4-3: Place the data marked as deleted into the updated collection, and copy the data of this type and store it in the Map collection.

[0020] Preferably, step S5 further includes the following sub-steps,

[0021] S5-1, If ​​the primary key id of the data is null, then execute the logic to add new data;

[0022] Repeat step S5 for a subset of the data;

[0023] S5-2, If the primary key id of the data is not null, then execute the data update logic;

[0024] S5-3 If the data to be saved in the request does not include the deleted data, then the deleted data will not be reset to not deleted.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: the data processing method according to the present invention greatly reduces the workload of front-end and back-end development, simplifies business processing logic, improves development efficiency and business processing efficiency, and greatly reduces the number of database accesses and improves system performance by processing data deletion and batch saving of the database based on memory processing. Attached Figure Description

[0026] Figure 1 This is a flowchart of a method for batch processing data based on memory according to the present invention. Detailed Implementation

[0027] To provide a further understanding of the purpose, structure, features, and functions of the present invention, detailed descriptions are provided below with reference to specific embodiments.

[0028] Please see Figure 1 An embodiment of the present invention provides a method for batch processing data in memory, comprising the following steps:

[0029] S1, the front end assembles the data that needs to be saved. Data that already exists in the data but is no longer needed needs to be removed, and a request to save the data is sent to memory.

[0030] The memory may be a hard disk, a database, etc.

[0031] S2, Preset data set;

[0032] Predefine the data set for the business data that needs to be saved;

[0033] The data set includes two types: an update-type set and an addition-type set.

[0034] S3: The backend receives the request parameters and queries all existing related data in the database according to the business requirements.

[0035] S4, batch set the deletion flag of the query result data to deleted, and add the data to the update class set;

[0036] S4-1, query the MySQL database for data that already exists and has not been deleted;

[0037] S4-2, mark the retrieved data as deleted in memory first;

[0038] S4-3, put the data marked as deleted into the updated class collection, and at the same time copy the data of this class and store it in the Map collection;

[0039] A Map collection is a key-value collection where the key is the primary key ID of the data and the value is the data itself, facilitating subsequent data operations.

[0040] Marking data in memory does not require database operations because the data is not yet in its final state. This reduces the number of database accesses, which is much more efficient than accessing and modifying the database data state. It also reduces database resource consumption and improves database performance.

[0041] S5 parses and modifies the data in memory;

[0042] Parse the data and loop through it, checking if the primary key id of the data is null;

[0043] S5-1, If ​​the primary key id of the data is null, it means that this data is newly added. Execute the data addition logic, set the status to not deleted, and store it in the addition class collection;

[0044] Repeat step S5 for a subset of the data;

[0045] S5-2, if the primary key id of the data is not null, it means that the data already exists in the database. Execute the data update logic, retrieve the data from the Map collection of deleted data according to the primary key id, replace the data content with the data that needs to be saved in the request, and reset the status from deleted to not deleted.

[0046] S5-3: If the data to be saved in the request does not include the deleted data, then the deleted data will not be reset to not deleted. Instead, it will remain in the update class collection and eventually be updated to the database together, becoming the actual deleted data.

[0047] S6: After the data is processed in memory, the data is persisted according to the update-type data set and the addition-type data set, and the database batch update operation and batch addition operation are executed respectively.

[0048] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for batch processing data based on memory, characterized in that: Includes the following steps, S1, the front end assembles the data that needs to be saved, removes data that already exists in the data but is no longer needed, and sends a request to save the data to memory; The memory can be a hard disk or a database; S2, Preset data set; Predefine the data set for the business data that needs to be saved; The data set includes two types: an update-type set and an addition-type set. S3: The backend receives the request parameters and queries all existing related data in the database according to the business requirements. S4, batch set the deletion flag of the query result data to deleted, and add the data to the update class set; S4-1, query the MySQL database for data that already exists and has not been deleted; S4-2, mark the retrieved data as deleted in memory first; S4-3, put the data marked as deleted into the updated class collection, and at the same time copy the data of this class and store it in the Map collection; S5 parses and modifies the data in memory; Parse the data and loop through it, checking if the primary key id of the data is null; S5-1, If ​​the primary key id of the data is null, then execute the logic to add new data; Repeat step S5 for a subset of the data; S5-2, If the primary key id of the data is not null, then execute the data update logic; S5-3, if the data to be saved in the request does not include the deleted data, then the deleted data will not be reset to not deleted; S6: After the data is processed in memory, the data is persisted according to the update-type data set and the addition-type data set, and the database batch update operation and batch addition operation are executed respectively.

Images