Data processing method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a medium, and belongs to the field of data processing. The method is applied to the electronic equipment, the electronic equipment performs data processing by applying a resilient distributed data set RDD, and the method comprises the following steps: acquiring to-be-processed data; target operation of the to-be-processed data is determined, the target operation comprises M sub-operations, and the M sub-operations have an operation sequence; and in the process of performing data processing on the to-be-processed data by adopting the RDD according to the sequence of the M sub-operations, interrupting the generation of an RDD link in the memory, and storing result data obtained by completing the execution of the Nth sub-operation into the temporary table. Therefore, the problem that when the electronic equipment applies an engine which uses RDD for data processing, such as Apache Spark, an RDD link based on the dependency relationship among multiple steps of data processing is generated, and pressure is caused to the memory of the electronic equipment is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method, device, electronic device and medium. Background Art

[0002] In the business data processing process of many industries, there is a situation where the object of a certain step of data processing is the result of the previous step of data processing, that is, there is a dependency relationship (i.e., sequence) between multiple data processing steps. For example, in the process of calculating reinsurance and cession business data in the financial insurance field, there are multiple reinsurance and cession types such as percentage reinsurance, excess reinsurance, and limit percentage reinsurance. These reinsurance types will have different priority relationships under the same customer or the same policy, that is, the calculation of excess reinsurance may rely on the result of percentage reinsurance calculation to calculate the result value.

[0003] In the related art, due to business expansion, the business data that needs to be processed increases, and many industries gradually migrate the data processing originally performed in the database to large-scale data processing engines such as Apache Spark for calculation and processing. For example, in the large-scale data processing engine of Apache Spark, there is a data processing engine that uses the Resilient Distributed Dataset (RDD) structure for data processing. Due to the lazy evaluation and immutability of RDD, the dependency between each round of data processing will form a long chain (i.e., RDD link) in RDD. The data processing engine will retain these long chains for data processing, which puts pressure on the memory of electronic devices.

[0004] Therefore, how to avoid the generation of RDD links based on dependencies between multi-step data processing when applying data processing engines such as Apache Spark that use RDD for data processing is an urgent problem to be solved. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a data processing method, device, electronic device and medium, aiming to avoid the generation of RDD links based on dependencies between multi-step data processing when an electronic device application such as Apache Spark, an engine that uses RDD for data processing, thereby avoiding the occupation of a large amount of memory resources.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a data processing method, which is applied to an electronic device, and the electronic device uses a resilient distributed dataset (RDD) to perform data processing, and the method includes:

[0007] Get the data to be processed;

[0008] Determine a target operation of the data to be processed, wherein the target operation includes M sub-operations, and the M sub-operations have a sequential operation order;

[0009] In the process of using the RDD to process the data to be processed according to the sequence of the M sub-operations, the generation of the RDD link in the memory is interrupted, and the result data obtained after the Nth sub-operation is executed is stored in a temporary table;

[0010] The RDD link is used to represent the order of operations among M sub-operations, N and M are both positive integers, and N is less than M.

[0011] In some implementations, storing the result data obtained after executing the Nth sub-operation in a temporary table includes:

[0012] Marking result data obtained after the Nth sub-operation is executed according to the order of the Nth sub-operation in the M sub-operations;

[0013] The marked result data is stored in the temporary table.

[0014] In some embodiments, the label is an identification number generated using N;

[0015] The storing the marked result data into the temporary table includes:

[0016] When the data marked by the identification number does not exist in the temporary table, the marked result data is stored in the temporary table.

[0017] In some implementations, there are multiple temporary tables, and storing the result data obtained after the Nth sub-operation is completed in the temporary table includes:

[0018] The result data obtained after the Nth sub-operation is executed is stored in a temporary table corresponding to the Nth sub-operation in the plurality of temporary tables.

[0019] In some implementations, after storing the result data obtained after executing the Nth sub-operation in a temporary table corresponding to the Nth sub-operation in the plurality of temporary tables, the method further includes:

[0020] Generate a time mark using the time point when the result data obtained after the Nth sub-operation is completed is stored in the temporary table;

[0021] Marking the temporary table corresponding to the Nth sub-operation among the plurality of temporary tables by using the time identifier;

[0022] When the time difference between the time point represented by the time identifier and the current time point is greater than a preset time threshold, the temporary table corresponding to the Nth sub-operation in the plurality of temporary tables is deleted.

[0023] In some implementations, after storing the result data obtained after executing the Nth sub-operation in a temporary table corresponding to the Nth sub-operation in the plurality of temporary tables, the method further includes:

[0024] Using the result data in the temporary table, perform the operation on the N+1th sub-operation;

[0025] When the execution information indicates that the N+1th sub-operation is completed, the temporary table corresponding to the Nth sub-operation among the plurality of temporary tables is deleted.

[0026] In some implementations, before storing the result data obtained after executing the Nth sub-operation in the temporary table, the method further includes:

[0027] Acquire a distributed lock, where the distributed lock is used to lock access to the temporary table;

[0028] The step of storing the result data obtained after executing the Nth sub-operation into the temporary table includes:

[0029] When the distributed lock is acquired successfully, the result data obtained after the Nth sub-operation is executed is stored in a temporary table.

[0030] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a data processing device, which is applied to an electronic device, and the electronic device uses a resilient distributed dataset (RDD) to perform data processing, and the device includes:

[0031] A data acquisition module, used to acquire data to be processed;

[0032] An operation execution module, used for determining a target operation of the data to be processed, wherein the target operation includes M sub-operations, and the M sub-operations have a sequential operation order;

[0033] A data storage module, used for interrupting the generation of the RDD link in the memory during the process of processing the data to be processed using the RDD according to the sequence of the M sub-operations, and storing the result data obtained after the Nth sub-operation is executed in a temporary table;

[0034] The RDD link is used to represent the order of operations among M sub-operations, N and M are both positive integers, and N is less than M.

[0035] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the data processing method described in the first aspect when executing the computer program.

[0036] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the data processing method described in the first aspect.

[0037] The data processing method, device, electronic device and medium proposed in the present application, when the electronic device applies RDD to perform data processing, by storing the result data in a temporary table after each step of multi-step calculation, the calculation result of each step can be saved, reducing the dependence on the previous operation, and avoiding the problem that when the electronic device uses a data processing engine such as Apache Spark that uses RDD for data processing to perform data processing, an RDD link based on the dependency relationship between multi-step data processing is generated, thereby causing pressure on the memory of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0039] Figure 2 It is a flowchart of the data storage process provided by the embodiment of the present application;

[0040] Figure 3 It is a flowchart of a temporary table deletion process provided by an embodiment of the present application;

[0041] Figure 4 It is a flowchart of another temporary table deletion process provided by an embodiment of the present application;

[0042] Figure 5 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0043] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0047] First, some nouns involved in this application are analyzed:

[0048] Resilient Distributed Dataset (RDD): RDD is the core structure used to store and process data in Apache Spark (hereinafter referred to as Spark). It can be regarded as a large table that can be distributed and stored on multiple machines. Spark will process data in parallel through RDD. RDD has the following characteristics:

[0049] Immutability: Once created, it cannot be modified. Each time an RDD is operated (such as filtering, conversion), a new RDD is generated.

[0050] Lazy evaluation: RDD does not perform any operations immediately, but waits until the final result data is needed to actually calculate.

[0051] Dependency: Each RDD has its "parent RDD", which is where it comes from. These parent-child relationships are connected together like a chain, forming an execution "dependency chain", that is, an RDD link.

[0052] When performing multi-step calculations in a database such as Oracle, each step can directly modify or use the result of the previous step. The results are stored in memory or disk, which makes it easy to perform subsequent operations without generating much memory pressure.

[0053] In Spark, RDD operations are executed lazily, which means that Spark will always "remember" each step of the calculation, but will not actually execute it until the final calculation result is needed. This means that if there are many steps, each step will depend on the previous step, and this dependency will form an RDD link in the RDD, and the data must be traced back to the original RDD along this chain each time.

[0054] For example, in the field of finance and insurance, suppose you need to calculate the reinsurance of a large data set. This process requires multiple calculations to handle different types of reinsurance, such as:

[0055] The first step is to calculate the percentage of reinsurance.

[0056] The second step is to calculate the excess reinsurance based on the results of the proportional reinsurance.

[0057] The third step is to calculate the limit percentage reinsurance based on the results of excess reinsurance.

[0058] In Oracle, electronic devices can perform calculations step by step, and store the results directly in the database after each calculation. The next time they calculate, they can directly read the previous results and continue the calculation without retaining the previous operations.

[0059] This approach does not bring too much memory pressure, because each calculation is independent and the results are "saved".

[0060] In Spark, each step of the calculation is done using RDD operations:

[0061] The percentage of the guarantee generates an RDD, generating RDD1;

[0062] Excess reinsurance generates RDD2 based on RDD1;

[0063] The limit percentage reinsurance generates RDD3 based on RDD2;

[0064] Spark's RDD operations are lazily evaluated, so when you finally need to calculate the result, Spark will not directly calculate RDD3, but will trace back to RDD1 along the chain and recalculate all the operations.

[0065] If this RDD link is very long, for example, it needs to go through multiple steps of calculation, and the result of each operation depends on the previous operation, Spark will retain all these dependencies, forming a complex long chain. At this time, each backtracking must go back to the earliest RDD. In the case of a large amount of data, the memory usage will increase rapidly, which may lead to insufficient memory or even memory overflow, and the program cannot be completed.

[0066] Based on this, the embodiments of the present application provide a data processing method, device, electronic device and medium, which aim to avoid the generation of RDD links based on dependencies between multi-step data processing when an electronic device application such as Apache Spark, a data processing engine that uses RDD for data processing, performs data processing.

[0067] The data processing method, device, electronic device and medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data processing method in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The data processing method provided in the embodiment of the present application relates to the field of data processing. The data processing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data processing method, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0073] See also Figure 1 , Figure 1 It is a flow chart of the data processing method provided in an embodiment of the present application. In a first aspect, an embodiment of the present application provides a data processing method, including steps S110 to S130. The method is applied to an electronic device, and the electronic device uses RDD to perform data processing.

[0074] Step S110: Obtain data to be processed.

[0075] The electronic device first obtains the raw data to be processed from the specified data source as the data to be processed. The data to be processed can be financial data, insurance data, user behavior data, etc. imported from the database. The obtained data to be processed can include multi-dimensional information, such as policy number, insurance type, amount, date and other basic information, which will be used in the subsequent calculation process.

[0076] For example, in the process of reinsurance and cession calculation, the data to be processed may be insurance information of multiple policies. For example:

[0077] Policy number: A1, Insurance type: BB, Premium: 1 million yuan, Assessment date: 20230630;

[0078] Policy number: B1, insurance type: DD, premium: 2 million yuan, assessment date: 20230630;

[0079] These data to be processed are the basic data for the subsequent reinsurance calculation.

[0080] Step S120: Determine a target operation of the data to be processed, where the target operation includes M sub-operations, and there is a sequential operation order among the M sub-operations.

[0081] The electronic device determines the target operation to be performed according to the data to be processed. The target operation is a combination of multiple sub-operations (M sub-operations), each sub-operation represents a specific operation step, and there is a strict execution order between these sub-operations.

[0082] For example, in the calculation of reinsurance, different types of reinsurance calculations (such as percentage reinsurance and excess reinsurance) may have a sequence, and one reinsurance calculation needs to be performed based on the result of the previous reinsurance calculation.

[0083] In the reinsurance calculation, the target operation can include the following sequence of sub-operations:

[0084] The first sub-operation: calculation of percentage reinsurance;

[0085] The second sub-operation: excess reinsurance calculation;

[0086] The third sub-operation: calculation of limit percentage reinsurance.

[0087] These sub-operations have a sequence of priority. The calculation of percentage reinsurance must be completed before the calculation of excess reinsurance, and the calculation of limit percentage reinsurance depends on the results of the first two steps.

[0088] Step S130: in the process of using RDD to process the data to be processed according to the sequence of M sub-operations, interrupting the generation of the RDD link in the memory, and storing the result data obtained after the Nth sub-operation is executed in a temporary table.

[0089] Among them, the RDD link is used to represent the order of operations between M sub-operations, N and M are both positive integers, and N is less than M.

[0090] In step S130, the electronic device executes the sub-operations in the order defined in step S120. Each sub-operation uses RDD for data processing. However, in order to avoid too long RDD links occupying a large amount of memory, causing memory overflow or performance degradation, after completing each sub-operation (the Nth sub-operation), the electronic device will interrupt the generation of the RDD link and write the result data of the sub-operation into a temporary table. The temporary table is used to temporarily store intermediate results to ensure that the next sub-operation can be processed based on the result, thereby avoiding the generation of RDD links based on dependencies between multi-step data processing.

[0091] For example, in the reinsurance calculation, assuming that the first sub-operation is the calculation of the percentage of reinsurance, after completion, the result data is stored in the temporary table, as shown below:

[0092] Policy number: A1, insurance type: BB, reinsurance result: RMB 800,000, assessment date: 20230630;

[0093] Policy number: B1, insurance type: DD, reinsurance result: RMB 1.6 million, assessment date: 20230630.

[0094] Then, the excess reinsurance calculation will read the results in temporary table A and perform the next calculation based on the results of the percentage reinsurance.

[0095] Through the above steps S110 to S130, when the electronic device applies RDD for data processing, the result data is stored in a temporary table after each step of the multi-step calculation, so that the calculation result of each step can be saved, reducing the dependence on the previous operation, and avoiding the problem of generating RDD links based on the dependency relationship between multi-step data processing when the electronic device uses a data processing engine such as Apache Spark that uses RDD for data processing to process data, thereby causing pressure on the memory of the electronic device.

[0096] See also Figure 2 , Figure 2 : is a flow chart of a data storage process provided in an embodiment of the present application. In some implementations, the result data obtained after the Nth sub-operation is executed is stored in a temporary table, including:

[0097] Step S210: marking the result data obtained after the Nth sub-operation is executed according to the order of the Nth sub-operation in the M sub-operations;

[0098] Step S220: Store the marked result data in a temporary table.

[0099] In this embodiment, when the Nth sub-operation is completed, the electronic device marks the result data of the sub-operation and stores the marked data in a temporary table. Specifically, according to the order of the Nth sub-operation in the M sub-operations, the electronic device will mark the result data with a specific mark. These marks can uniquely identify that the result data is generated by the Nth sub-operation, and the mark can also contain other information about data processing (such as calculation rounds, data versions, timestamps, etc.).

[0100] In this way, by marking the result data, the electronic device can distinguish the results generated by different sub-operations, and can reference or use these marks in subsequent operations for traceability or optimization. Finally, the marked result data will be stored in a temporary table to provide a reliable data source for the next round of data processing. This process ensures the order and accuracy of data processing and provides a flexible data management mechanism for electronic devices.

[0101] For example, in the reinsurance calculation process, it is assumed that the first sub-operation is the excess reinsurance calculation. When the excess reinsurance calculation is completed, the electronic device marks the result generated by the operation. For example:

[0102] Policy number: A1, insurance type: BB, excess reinsurance result: 700,000 yuan, assessment date: 20230630, marked as "Excess_1";

[0103] Policy number: B1, insurance type: DD, excess reinsurance result: RMB 1.5 million, assessment date: 20230630, marked as "Excess_1".

[0104] These marks indicate that the data is the result of excess reinsurance calculation in step 1, and subsequent processing will be traced or verified based on these marks. Next, these marked data will be stored in a temporary table to ensure that the excess reinsurance result data can be used correctly in the next step of percentage reinsurance calculation.

[0105] In some embodiments, the label is an identification number generated using N;

[0106] Store the marked result data in a temporary table, including:

[0107] If there is no data marked with the identification number in the temporary table, the marked result data is stored in the temporary table.

[0108] In this embodiment, the marking process is to generate an identification number by using the sequence number of the Nth sub-operation. This identification number represents the order of the Nth sub-operation and serves as a unique mark to distinguish the calculation results of different rounds. Before storing the marked result data in the temporary table, the electronic device will first check whether there is already data marked with the same identification number in the temporary table. If the marked data corresponding to the identification number does not exist in the temporary table, the electronic device will store the marked result data in the temporary table. In this way, it can be ensured that the results of each round of calculation will not be written repeatedly to the temporary table, and only new data will be stored, thereby avoiding redundancy and repeated storage of data.

[0109] For example, in the process of reinsurance calculation, it is assumed that the Nth round of operation is excess reinsurance calculation, and the identification number of this round is "excess_N". After the calculation is completed, the result data generated is:

[0110] Policy number: A1, insurance type: BB, excess reinsurance result: RMB 700,000, assessment date: 20230630, identification number: excess_N;

[0111] Policy number: B1, insurance type: DD, excess reinsurance result: RMB 1.5 million, valuation date: 20230630, identification number: Excess_N.

[0112] Before storing the marked data, the electronic device first checks whether there is data with the identification number "Overage_N" in the temporary table. If there is no data with the identification number in the temporary table, the electronic device stores the data in the temporary table.

[0113] If data with the same identification number already exists in the table, it will not be written again. This mechanism ensures that the reinsurance results of each round can be stored correctly and will not cause data duplication or conflict.

[0114] In some implementations, there are multiple temporary tables, and the result data obtained after the Nth sub-operation is executed is stored in the temporary table, including:

[0115] The result data obtained after the Nth sub-operation is executed is stored in a temporary table corresponding to the Nth sub-operation in multiple temporary tables.

[0116] In this embodiment, there are multiple temporary tables, each of which corresponds to a specific sub-operation. The execution result of each sub-operation will be stored in the temporary table corresponding to the sub-operation. This design allows the result of each sub-operation to be stored independently, which is convenient for subsequent calculations and data management. When executing the Nth sub-operation, the electronic device will store its calculation result in the corresponding temporary table according to the specific identification of the Nth sub-operation. In this way, the result of each operation can be reasonably isolated in its own temporary table to avoid confusion between different operation results.

[0117] For example, in the process of reinsurance calculation, it is assumed that there are three types of reinsurance operations: percentage reinsurance, excess reinsurance and limit percentage reinsurance. After each round of reinsurance operation is completed, the result data is stored in a temporary table corresponding to the reinsurance operation. For example, it is assumed that the operation in step N is excess reinsurance, and the operation in step N-1 is percentage reinsurance.

[0118] After the calculation of step N (excess reinsurance) is completed, the generated data is as follows:

[0119] Policy number: A1, insurance type: BB, excess reinsurance result: RMB 700,000, assessment date: 20230630;

[0120] Policy number: B1, insurance type: DD, excess reinsurance result: RMB 1.5 million, valuation date: 20230630.

[0121] The result data of these excess reinsurance will be stored in a temporary table B specifically used for excess reinsurance.

[0122] The result data calculated in step N-1 (percentage reinsurance) has been stored in another temporary table, temporary table A.

[0123] This design ensures that the results of each round of reinsurance can be stored independently in their respective temporary tables, avoiding mixed storage of results of different reinsurance types and facilitating subsequent query and processing.

[0124] See also Figure 3 , Figure 3 is a flowchart of a temporary table deletion process provided by an embodiment of the present application. In some implementations, after the result data obtained after the Nth sub-operation is executed is stored in a temporary table corresponding to the Nth sub-operation in multiple temporary tables, the method further includes:

[0125] Step S310: Generate a time mark using the time point when the result data obtained after the Nth sub-operation is completed is stored in the temporary table;

[0126] Step S320: Mark the temporary table corresponding to the Nth sub-operation in the multiple temporary tables by using the time mark;

[0127] Step S330: when the time difference between the time point represented by the time identifier and the current time point is greater than a preset time threshold, deleting the temporary table corresponding to the Nth sub-operation from the multiple temporary tables.

[0128] In this embodiment, when the result data is stored in the corresponding temporary table after each sub-operation is completed, additional operations are performed to optimize storage management. Specifically, when the Nth sub-operation is executed and the result data is stored in the corresponding temporary table, the electronic device will record the time point when the data is stored and generate a time identifier. This time identifier will be used to mark the temporary table. After that, the electronic device will monitor the time difference between the storage time point represented by the time identifier and the current time point. If the time difference exceeds the preset time threshold, the electronic device will automatically delete the temporary table corresponding to the Nth sub-operation to free up storage space. In this way, not only can the data in the temporary table be guaranteed to be available when needed, but also expired temporary tables can be cleaned up regularly to avoid occupying unnecessary storage resources.

[0129] Exemplarily, in the reinsurance calculation process, it is assumed that after the excess reinsurance calculation of step N is performed, the generated excess reinsurance result data is stored in temporary table B.

[0130] When the operation of storing these data is completed, the electronic device will generate a time mark (for example, the time point of storage completion is "2024-10-09 10:00:00"), and use this time mark to mark temporary table B. If the preset time threshold set by the electronic device is 24 hours, then when the current time is "2024-10-10 10:01:00", the electronic device will check the time mark of temporary table B and find that the time difference with the current time point is greater than 24 hours, exceeding the preset time threshold, so the electronic device will automatically delete temporary table B. This mechanism effectively avoids the long-term occupation of the temporary table, while ensuring that the data can be used for subsequent calculation steps within the validity period of the temporary table.

[0131] See also Figure 4 , Figure 4 is a flowchart of another temporary table deletion process provided by an embodiment of the present application. In some implementations, after the result data obtained after the Nth sub-operation is executed is stored in a temporary table corresponding to the Nth sub-operation in multiple temporary tables, the method further includes:

[0132] Step S410: using the result data in the temporary table, perform an operation on the N+1th sub-operation;

[0133] Step S420: When the N+1th sub-operation is completed, the temporary table corresponding to the Nth sub-operation among the multiple temporary tables is deleted.

[0134] In some embodiments, the electronic device first uses the result data of the Nth sub-operation in the temporary table to perform the execution process of the N+1th sub-operation. This step means that the calculation results of the previous round will be used as input data and applied to the next round of sub-operations. After the N+1th sub-operation is completed, the execution information of the operation is obtained. The execution information may include a flag, time point or related operation log of successful execution. When it is determined that the N+1th sub-operation is completed, the multiple temporary tables corresponding to the Nth sub-operation are deleted. In this way, storage space can be effectively released, unnecessary data occupation can be avoided, and efficient use of electronic device resources can be ensured.

[0135] Exemplarily, during the reinsurance calculation process, assuming that after the excess reinsurance calculation of step N is completed, the result data is stored in multiple temporary tables (such as temporary table A and temporary table B): in step S410, the data in temporary table A will be used as input to start the N+1th round of percentage reinsurance calculation. At this time, the reinsurance calculation program will use the result data of the excess reinsurance to calculate the next round of percentage reinsurance amount. In step S420, the electronic device confirms that the N+1th round of percentage reinsurance calculation has been successfully completed, and obtains the execution information of this round of calculation, such as a successful completion flag or log record. Finally, in step S430, since the N+1th sub-operation has been successfully executed, the electronic device will delete the temporary tables A and B related to the Nth round of calculation to ensure that they no longer occupy storage resources and prepare to release space for subsequent calculations.

[0136] This method can avoid the accumulation of a large amount of useless data by gradually clearing temporary data tables that are no longer needed, thereby effectively optimizing the storage resource management of electronic devices.

[0137] In some implementations, before storing the result data obtained after the Nth sub-operation is executed in the temporary table, the method further includes:

[0138] Obtain a distributed lock, which is used to lock access to temporary tables;

[0139] The result data obtained after the Nth sub-operation is executed is stored in a temporary table, including:

[0140] When the distributed lock is acquired successfully, the result data obtained after the Nth sub-operation is executed is stored in the temporary table.

[0141] In this embodiment, before the result data of the Nth sub-operation is stored in the temporary table, the process of acquiring a distributed lock is first performed. The role of the distributed lock is to ensure that the access to the temporary table is unique and safe to avoid data conflicts or inconsistencies caused by concurrent operations. Specifically, before storing the result data of the Nth sub-operation, the electronic device will try to acquire a distributed lock. The role of this lock is to ensure that only one task can access the temporary table when multiple concurrent tasks simultaneously access and modify the same temporary table, thereby ensuring the integrity and consistency of the data. When the distributed lock is successfully acquired, the electronic device stores the result data obtained after the Nth sub-operation is executed in the temporary table. In this way, other tasks cannot read and write to the temporary table before releasing the lock, ensuring the exclusivity of the operation.

[0142] Exemplarily, during the reinsurance calculation process, it is assumed that after the excess reinsurance calculation of the Nth step is completed, the electronic device is ready to store the result data in a temporary table. At this time, because multiple computing tasks may write to the temporary table at the same time, the electronic device needs to use a distributed lock to ensure the security and consistency of the data. The electronic device first obtains a distributed lock to lock access to temporary table A. For example, if task 1 obtains a distributed lock, other computing tasks (such as task 2) cannot access temporary table A temporarily. Once task 1 successfully obtains the distributed lock, it will write the result data of the Nth round of excess reinsurance to temporary table A. At this time, only task 1 can operate on table A. After the data is written successfully, task 1 will release the distributed lock, allowing other tasks (such as task 2) to continue to access and modify temporary table A. In this way, the distributed lock ensures the security and exclusivity of each write operation, avoiding data inconsistency or conflict problems caused by multiple tasks operating the table at the same time.

[0143] See also Figure 5 , Figure 5 is a structural diagram of a data processing device provided in an embodiment of the present application. The embodiment of the present application further provides a data processing device 500. The device 500 is applied to an electronic device. The electronic device uses a resilient distributed data set RDD to process data and can implement the above data processing method. The device 500 includes:

[0144] A data acquisition module 510 is used to acquire data to be processed;

[0145] An operation execution module 520 is used to determine a target operation of the data to be processed, where the target operation includes M sub-operations, and there is a sequential operation order between the M sub-operations;

[0146] The data storage module 530 is used to interrupt the generation of the RDD link in the memory and store the result data obtained after the Nth sub-operation is executed in a temporary table during the process of processing the data to be processed according to the sequence of the M sub-operations using the RDD;

[0147] Among them, the RDD link is used to represent the order of operations between M sub-operations, N and M are both positive integers, and N is less than M.

[0148] The specific implementation of the data processing device 500 is substantially the same as the specific implementation of the above-mentioned data processing method, and will not be described in detail herein.

[0149] In some embodiments, the data storage module 530 includes:

[0150] a marking unit, used for marking result data obtained after the Nth sub-operation is executed according to the order of the Nth sub-operation in the M sub-operations;

[0151] The storage unit stores the marked result data in a temporary table.

[0152] In some embodiments, the label is an identification number generated using N;

[0153] The storage unit includes a storage subunit, which is used for storing the marked result data in the temporary table when there is no data marked with the identification number in the temporary table.

[0154] In some implementations, there are multiple temporary tables, and the data storage module 530 includes:

[0155] The second storage unit is used to store the result data obtained after the Nth sub-operation is executed in a temporary table corresponding to the Nth sub-operation in the multiple temporary tables.

[0156] In some embodiments, the data storage module 530 further includes:

[0157] An identification unit, used to generate a time identification by using the time point when the result data obtained after the Nth sub-operation is completed is stored in the temporary table;

[0158] A marking unit, used for marking a temporary table corresponding to the Nth sub-operation among the multiple temporary tables by using a time mark;

[0159] The deleting unit is used to delete the temporary table corresponding to the Nth sub-operation from the multiple temporary tables when the time difference between the time point represented by the time identifier and the current time point is greater than a preset time threshold.

[0160] In some embodiments, the data storage module 530 further includes:

[0161] An execution unit, used to perform the execution process of the N+1th sub-operation using the result data in the temporary table;

[0162] An acquisition unit, used for acquiring execution information of the N+1th sub-operation;

[0163] The deleting unit is used to delete the temporary table corresponding to the Nth sub-operation from the multiple temporary tables when the execution information indicates that the N+1th sub-operation is completed.

[0164] In some embodiments, the apparatus 500 further comprises:

[0165] A distributed lock acquisition unit is used to acquire a distributed lock, which is used to lock access to a temporary table;

[0166] The data storage module 530 includes:

[0167] When the distributed lock is acquired successfully, the result data obtained after the Nth sub-operation is executed is stored in the temporary table.

[0168] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0169] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0170] The processor 601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0171] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes the data processing method of the embodiment of this application;

[0172] Input / output interface 603, used to implement information input and output;

[0173] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0174] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0175] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0176] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned data processing method when executed by a processor.

[0177] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The data processing method, device, electronic device and medium provided in the embodiments of the present application, when the electronic device applies RDD for data processing, by storing the result data in a temporary table after each step of multi-step calculation, the calculation result of each step can be saved, reducing the dependence on the previous operation, and avoiding the problem of generating RDD links based on the dependency relationship between multi-step data processing when the electronic device uses a data processing engine such as Apache Spark that uses RDD for data processing to process data, thereby causing pressure on the memory of the electronic device.

[0179] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0180] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0181] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0183] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0184] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0185] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0189] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to an electronic device, and the electronic device uses a resilient distributed data set (RDD) to process data. The method includes: Get the data to be processed; Determine a target operation of the data to be processed, wherein the target operation includes M sub-operations, and the M sub-operations have a sequential operation order; In the process of using the RDD to process the data to be processed according to the sequence of the M sub-operations, the generation of the RDD link in the memory is interrupted, and the result data obtained after the Nth sub-operation is executed is stored in a temporary table; The RDD link is used to represent the order of operations among M sub-operations, N and M are both positive integers, and N is less than M.

2. The method according to claim 1, characterized in that The step of storing the result data obtained after executing the Nth sub-operation into the temporary table includes: Marking result data obtained after the Nth sub-operation is executed according to the order of the Nth sub-operation in the M sub-operations; The marked result data is stored in the temporary table.

3. The method according to claim 2, characterized in that The mark is an identification number generated using N; The storing the marked result data into the temporary table includes: When the data marked by the identification number does not exist in the temporary table, the marked result data is stored in the temporary table.

4. The method according to claim 1, characterized in that: The number of the temporary tables is multiple, and the step of storing the result data obtained after the Nth sub-operation is completed in the temporary table includes: The result data obtained after the Nth sub-operation is executed is stored in a temporary table corresponding to the Nth sub-operation in the plurality of temporary tables.

5. The method according to claim 4, characterized in that After storing the result data obtained after executing the Nth sub-operation in the temporary tables corresponding to the Nth sub-operation in the plurality of temporary tables, the method further includes: Generate a time mark using the time point when the result data obtained after the Nth sub-operation is completed is stored in the temporary table; Marking the temporary table corresponding to the Nth sub-operation among the plurality of temporary tables by using the time identifier; When the time difference between the time point represented by the time identifier and the current time point is greater than a preset time threshold, the temporary table corresponding to the Nth sub-operation in the plurality of temporary tables is deleted.

6. The method according to claim 4, characterized in that After storing the result data obtained after executing the Nth sub-operation in the temporary tables corresponding to the Nth sub-operation in the plurality of temporary tables, the method further includes: Using the result data in the temporary table, perform the operation on the N+1th sub-operation; When the N+1th sub-operation is completed, the temporary table corresponding to the Nth sub-operation among the plurality of temporary tables is deleted.

7. The method according to claim 1, characterized in that Before storing the result data obtained after executing the Nth sub-operation in the temporary table, the method further includes: Acquire a distributed lock, where the distributed lock is used to lock access to the temporary table; The step of storing the result data obtained after executing the Nth sub-operation into the temporary table includes: When the distributed lock is acquired successfully, the result data obtained after the Nth sub-operation is executed is stored in a temporary table.

8. A data processing device, characterized in that: The device is applied to an electronic device, and the electronic device uses a resilient distributed data set (RDD) to process data. The device includes: A data acquisition module, used to acquire data to be processed; An operation execution module, used for determining a target operation of the data to be processed, wherein the target operation includes M sub-operations, and the M sub-operations have a sequential operation order; A data storage module, used for interrupting the generation of the RDD link in the memory during the process of processing the data to be processed using the RDD according to the sequence of the M sub-operations, and storing the result data obtained after the Nth sub-operation is executed in a temporary table; The RDD link is used to represent the order of operations among M sub-operations, N and M are both positive integers, and N is less than M.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images