Data access processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data access processing method and device, electronic equipment and storage medium, wherein the data access processing method comprises: obtaining to-be-accessed data and an access standard corresponding to the to-be-accessed data; obtaining an access standard configuration table, and determining a data processing flow corresponding to the access standard of the to-be-accessed data according to the access standard configuration table, wherein in the access standard configuration table, different access standards correspond to different data processing flows; and processing the to-be-accessed data according to the data processing flow to obtain processed accessed data. The present disclosure realizes the use of the same processing flow for the access of different data, thereby ensuring the standardization and flow of data access processing, and improving the system resource utilization and the maintainability in the later period.

Description

Technical Field

[0001] This disclosure relates to the field of data access technology, and in particular to a data access processing method, apparatus, electronic device and storage medium. Background Technology

[0002] In existing technologies, there are scenarios where data is synchronized in the form of large files. Taking the insurance industry as an example, for data access scenarios such as insurance purchase, cancellation, or claims, a separate processing program is often developed for products from different channels or business types, or the file parsing part is processed as a separate public service. This results in multiple processing programs for different data accesses, wasting system resources and causing poor maintainability in the later stages. Summary of the Invention

[0003] This disclosure provides a data access processing method, apparatus, electronic device, and storage medium to solve the defects in the prior art where multiple processing procedures exist for different data accesses, resulting in wasted system resources and poor maintainability in the later stages. It realizes the use of the same processing flow for access to different data, thereby ensuring the standardization and process-orientation of data access processing, improving system resource utilization and later maintainability.

[0004] This disclosure provides a data access processing method, the method comprising: acquiring data to be accessed and an access standard corresponding to the data to be accessed; acquiring an access standard configuration table, and determining a data processing flow corresponding to the access standard of the data to be accessed based on the access standard configuration table, wherein different access standards correspond to different data processing flows in the access standard configuration table; and processing the data to be accessed according to the data processing flow to obtain processed access data.

[0005] According to a data access processing method provided in this disclosure, the data processing flow includes a data parsing flow and a data mapping flow. The step of processing the data to be accessed according to the data processing flow to obtain processed access data specifically includes: parsing the data to be accessed according to the data parsing flow to obtain parsed data; and mapping the parsed data to corresponding message fields according to the data mapping flow to obtain processed access data, wherein the message fields are pre-stored in the access standard configuration table.

[0006] According to a data access processing method provided in this disclosure, in the access standard configuration table, different access standards also correspond to different message field mapping relationships. The message field mapping relationship is the mapping relationship between the parsed data and message fields after parsing the data to be accessed. The step of mapping the parsed data to the corresponding message fields according to the data mapping process to obtain processed access data specifically includes: determining the message field mapping relationship corresponding to the access standard of the parsed data based on the access standard configuration table; and mapping the parsed data to the corresponding message fields according to the data mapping process based on the message field mapping relationship to obtain processed access data.

[0007] According to a data access processing method provided in this disclosure, the data to be accessed includes multiple sub-data items to be accessed. The step of parsing the data to be accessed according to the data parsing process to obtain parsed data specifically includes: when it is determined that the data volume of the sub-data to be accessed is greater than or equal to a data volume threshold, dividing the data to be accessed into multiple execution units; parsing the sub-data to be accessed in each execution unit in parallel according to the data parsing process to obtain parsed execution units; and integrating the parsed execution units to obtain the parsed data.

[0008] According to a data access processing method provided in this disclosure, the sub-data to be accessed corresponds to a data row number; the step of parsing the sub-data to be accessed in the execution unit in parallel according to the data parsing process specifically includes: storing the data row number of the parsed sub-data to be accessed; responding to a parsing interruption instruction, interrupting the parsing of the execution unit; determining the next sub-data to be accessed after the sub-data to be accessed corresponding to the data row number based on the data row number; and continuing to parse the execution unit starting from the next sub-data to be accessed.

[0009] According to a data access processing method provided in this disclosure, after parsing the sub-data to be accessed in the execution unit in parallel according to the data parsing process, the method further includes: saving the parsing results of the sub-data to be accessed to a data detail table, wherein the parsing results include parsing success results and parsing failure results; if it is determined that the number of sub-data to be accessed actually parsed in the execution unit is different from the number of sub-data to be accessed contained in the execution unit, it is determined that there is sub-data to be accessed that failed to be parsed in the execution unit; traversing the data detail table, determining the sub-data to be accessed that failed to be parsed, and re-parsing the sub-data to be accessed that failed to be parsed according to the data parsing process.

[0010] According to a data access processing method provided in this disclosure, the step of obtaining the data to be accessed and the access standard corresponding to the data to be accessed specifically includes: in response to receiving a data access request, parsing the metadata of the data access request to obtain the storage path of the data to be accessed in the data access request and the access standard corresponding to the data to be accessed; and obtaining the data to be accessed based on the storage path.

[0011] This disclosure also provides a data access processing apparatus, the apparatus comprising: an acquisition module for acquiring data to be accessed and an access standard corresponding to the data to be accessed; a determination module for acquiring an access standard configuration table and determining a data processing flow corresponding to the access standard of the data to be accessed based on the access standard configuration table, wherein different access standards correspond to different data processing flows in the access standard configuration table; and a processing module for processing the data to be accessed according to the data processing flow to obtain processed access data.

[0012] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data access processing method as described above.

[0013] This disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data access processing method as described above.

[0014] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the data access processing method as described above.

[0015] The data access processing method, apparatus, electronic device, and storage medium disclosed herein determine the correspondence between access standards and data processing procedures by pre-setting an access standard configuration table. Upon acquiring data to be accessed, the data processing procedure for the data to be accessed can be directly determined based on the access standard of the data to be accessed, and the data to be accessed can be processed according to the data processing procedure to obtain the processed access data. This disclosure enables the use of the same processing procedure for accessing different types of data, thereby ensuring the standardization and streamlining of data access processing, improving system resource utilization and future maintainability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is one of the flowcharts illustrating the data access processing method provided in this disclosure;

[0018] Figure 2 This is a schematic diagram of the interaction execution process between the external interaction module and the channel access system provided in this public document;

[0019] Figure 3 This is a flowchart provided in this publication illustrating the process of parsing the data to be accessed according to the data parsing procedure to obtain the parsed data;

[0020] Figure 4 This is the second flowchart of the data access processing method provided in this disclosure;

[0021] Figure 5 This is the third flowchart of the data access processing method provided in this disclosure;

[0022] Figure 6 This is a schematic diagram of the structure of the data access processing device provided in this disclosure;

[0023] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this disclosure. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0025] As is known from relevant technologies, there are scenarios where data is synchronized in the form of large files. For ease of explanation, this disclosure uses the insurance industry as an example, but it is understood that this disclosure is not limited to insurance scenarios and can also be applied to other application scenarios that require data access. For data access scenarios such as insurance purchase, cancellation, or claims, a separate processing program is often developed for products from different channels or with different business types, or the file parsing part is processed as a separate public service.

[0026] Understandably, for the aforementioned insurance application scenarios, there is no fixed and highly compatible development standard for data synchronization. This results in the data from each channel being processed by the developers according to their own development habits, and similar businesses often reinvent the wheel. This leads to a waste of system resources and poor maintainability in the later stages.

[0027] For the multi-channel, multi-product, and multi-business type document-based order acceptance scenarios in the insurance business, the data access processing method disclosed herein can provide a unified, flexible, easily scalable, and easily maintainable processing solution, thereby improving business access efficiency and reducing development costs.

[0028] This disclosure will describe the process of the data access processing method in conjunction with the following embodiments.

[0029] Figure 1 This is one of the flowcharts of the data access processing method provided in this disclosure.

[0030] In an exemplary embodiment of this disclosure, such as Figure 1 As shown, the data access processing method may include steps 110 to 130, which will be described in detail below.

[0031] In step 110, the data to be accessed and the access standard corresponding to the data to be accessed are obtained.

[0032] In one embodiment, the data to be accessed can originate from different channels, have different business types, or belong to different product types. Different access standards correspond to different sources, business types, or product types. Taking the insurance business scenario as an example, data from different channels could refer to data from different business partners (such as Douyin, Kuaishou, etc.). Data with different business types could refer to data related to policy purchase, policy cancellation, or claims. Data belonging to different product types could refer to the product type of the data to be accessed, such as accident insurance or life insurance.

[0033] In another embodiment, the data to be accessed and the access standard corresponding to the data to be accessed can be obtained in the following way: in response to receiving a data access request, the metadata of the data access request is parsed to obtain the storage path of the data to be accessed in the data access request and the access standard corresponding to the data to be accessed; based on the storage path, the data to be accessed is obtained.

[0034] In one example, the access party (or customer) can initiate a data access request. This request may include information such as the name of the data to be accessed, the source channel, business type, or product type, and the storage address of the data. During application, the metadata of the data access request can be parsed to obtain the storage path of the data to be accessed and the access standard corresponding to the data. Based on the storage path, the data to be accessed can then be retrieved.

[0035] In another example, based on the received data access request, an access task can be generated by parsing the metadata of the data access request and saved to a task table. Each time a task is accessed, a corresponding task record is generated in the task table to indicate successful acceptance. The task table can also record the storage path and task flow information for the access task (which may correspond to the data to be accessed). The task flow information is determined according to the access standard configuration table. During application, the data processing flow (corresponding to the task flow information) corresponding to the access standard of the data to be accessed can be determined according to the access standard configuration table. It is understandable that storing the storage path of the data to be accessed and the task flow information in the task table facilitates subsequent processing directly based on the task table.

[0036] In another example, after receiving a data access request, it can also determine whether the request belongs to a data access request that should be processed by this system, that is, to perform authentication processing on the data access request.

[0037] It should be noted that the module capable of performing the aforementioned steps can be referred to as the external interaction module. After obtaining the data processing flow of the data to be accessed, the external interaction module can assemble the storage path of the data to be accessed and the data processing flow into a general message to be pushed downwards. After assembly, it can be pushed to the file processing module for subsequent processing. At this point, the order acceptance work of the external interaction module is completed. The other task is to wait for the notification of task completion. Once the notification is received, the external interaction module can notify the channel party of the final processing result to complete the entire closed loop of a task.

[0038] To further illustrate the data access and processing method provided in this disclosure, the following explanation will focus on the interaction process between the channel access system and the external interaction module.

[0039] Figure 2 This is a schematic diagram of the interaction execution process between the external interaction module and the channel access system provided in this public document.

[0040] In one embodiment, combined with Figure 2It is understood that the channel access system will forward the file and initiate a data access request for the file. Upon receiving the data access request, the external interaction module will first authenticate the request (verifying the legitimacy of the access party) and perform idempotency verification. Secondly, it will determine whether the file to be forwarded (corresponding to the current business) has already been processed. If it has been processed, to avoid reprocessing, the message about the forwarded file can be directly pushed to the file processing module, which will then execute the corresponding steps. For example, it can directly return the historical processing results to the requesting party, or wait for feedback on the results already being processed. If the current business has not been processed, the task related to the data access request (corresponding to the storage path of the data to be accessed) can be tagged with business type and channel. Based on a pre-set access standard configuration table, the data processing flow corresponding to the task (corresponding to the data to be accessed) can be determined. The storage path and the data processing flow of the data to be accessed are then assembled into a general message to be pushed downwards. After assembly, it can be pushed to the file processing module for subsequent processing (corresponding to task distribution). It should be noted that after receiving a data access request on the external interaction module side, the receiving result of the data access request can also be fed back to the channel access system side.

[0041] During application, it's also possible to determine if the task was successfully sent. If successful, the file processing module can directly execute the corresponding steps. If unsuccessful, the storage path and the data processing flow for the data to be accessed can be reassembled into a general message to be pushed down, and then pushed back to the file processing module for further processing. If the number of times the task is sent down exceeds the predetermined number of retries, it indicates an anomaly in the external interaction module, which can generate an anomaly summary alarm.

[0042] In step 120, an access standard configuration table is obtained, and the data processing flow corresponding to the access standard of the data to be accessed is determined according to the access standard configuration table. Different access standards correspond to different data processing flows in the access standard configuration table.

[0043] In one embodiment, different access standards correspond to different data processing flows in the access standard configuration table. During application, once the access standard for the data to be accessed is determined, the corresponding data processing flow can be directly determined based on the access standard configuration table. In one example, the steps of the data processing flow can be executed based on the file processing module to achieve the parsing and processing of the data to be accessed.

[0044] In one embodiment, the access standard configuration table can be understood as a table jointly composed of a channel business type configuration table and a process configuration table. The channel business type configuration table configures a set of processes and message parsing schemes for the access channel, insurance type, and business type of the data to be accessed. Subsequent task execution will use this configuration to automate the process. As described above, data from different channels, with different business types, or belonging to different product types corresponds to different access standards. Therefore, the processing flow for different access standards can be determined in the channel business type configuration table. The process configuration table configures the processes that may be involved in the task. Each process corresponds to a set of processing flows in the system, ensuring flexible combination and expansion of the system. It can be understood that based on the channel business type configuration table, the processing flow for the access standard (which can be understood as the process name) can be determined. Furthermore, based on the process configuration table, the specific data processing flow (which can be understood as the process details) for the processing flow can be determined.

[0045] In step 130, the data to be accessed is processed according to the data processing flow to obtain the processed access data.

[0046] During application, downstream access systems corresponding to data from different channels have access requirements for that data. The data needs to be processed to meet these requirements. These requirements correspond to the data processing flow. In one embodiment, a data processing flow corresponding to the access standard of the data to be accessed can be determined based on an access standard configuration table. The data is then processed according to this flow to obtain the processed access data. It is understood that the processed access data satisfies the access requirements of the corresponding downstream access system.

[0047] The data access processing method disclosed herein establishes a pre-defined access standard configuration table to determine the correspondence between access standards and data processing procedures. Upon acquiring data to be accessed, the data processing procedure can be directly determined based on the access standards of the data to be accessed, and the data to be accessed can be processed according to the data processing procedure to obtain the processed access data. This disclosure enables the use of the same processing procedure for accessing different types of data, thereby ensuring the standardization and streamlining of data access processing, improving system resource utilization and future maintainability.

[0048] To further illustrate the data access processing method provided in this disclosure, the following embodiments will be used for explanation.

[0049] In an exemplary embodiment of this disclosure, the data processing flow may include a data parsing flow and a data mapping flow. The data processing flow for processing the data to be accessed can be implemented in the following ways: parsing the data to be accessed according to the data parsing flow to obtain parsed data; and mapping the parsed data to corresponding message fields according to the data mapping flow to obtain processed access data.

[0050] During application, after parsing the data to be accessed according to the data parsing process, the parsed data can be obtained. This parsing process includes multiple processing points, such as retrieving the source file (corresponding to the data to be accessed), file splitting, dividing into minimum processing units, processing planning and storage, and parsing file content. The processing status of each node is recorded by the system to prevent repetitive processing after interruption and restart.

[0051] Since the parsed data lacks business meaning, it needs to be assigned that meaning. In one example, the parsed data can be mapped to corresponding message fields according to a data mapping process to give it business meaning, thus obtaining the processed access data. During application, because the processed access data meets the access requirements of the downstream access system, it can be directly connected to the downstream access system.

[0052] Understandably, message fields can be pre-stored in the access standard configuration table. In the access standard configuration table, different access standards correspond to different message field mapping relationships. The message field mapping relationship is the mapping relationship between the parsed data and the message fields after the data to be accessed is parsed.

[0053] In one embodiment, mapping the parsed data to the corresponding message fields according to the data mapping process to obtain the processed access data can be achieved in the following way: based on the access standard configuration table, determine the message field mapping relationship corresponding to the access standard of the parsed data; based on the message field mapping relationship, map the parsed data to the corresponding message fields according to the data mapping process to obtain the processed access data.

[0054] Once the data to be accessed is parsed into parsed data, the access standard for the parsed data is the same as that for the data to be accessed. During application, since the access standard corresponds to the access requirements of the downstream access system, the mapping relationship between the parsed data and the corresponding message fields can be determined based on the access standard configuration table. Furthermore, based on the message field mapping relationship, the parsed data is mapped to the corresponding message fields according to the data mapping process, so that the parsed data is given business meaning, that is, the processed access data meets the access requirements of the downstream access system.

[0055] In the application process, the data to be accessed is often a large amount of data, or there may be a large amount of data to be accessed and processed at the same time. In order to improve the access processing efficiency, the data to be accessed can be segmented.

[0056] The process of segmenting the data to be accessed to improve access processing efficiency will be described below with reference to the following embodiments.

[0057] Figure 3 This is a flowchart provided in this publication illustrating the process of parsing the data to be accessed according to the data parsing procedure to obtain the parsed data.

[0058] In an exemplary embodiment of this disclosure, the data to be accessed may include multiple sub-data items to be accessed. For example... Figure 3 As shown, the data to be accessed is parsed according to the data parsing process. The parsed data can be obtained through steps 310 to 330, which will be described in detail below.

[0059] In step 310, if it is determined that the amount of data to be accessed is greater than or equal to the data amount threshold, the data to be accessed is divided into multiple execution units.

[0060] In one embodiment, the data to be accessed may include multiple sub-data items. It is understood that the size of the data to be accessed can be measured based on the data volume of the sub-data items. Specifically, when the data to be accessed is large (corresponding to a sub-data volume greater than or equal to a data volume threshold), it will take longer to parse the data. To effectively improve the parsing efficiency of the data to be accessed, it can be segmented into multiple execution units during the application process. It is understood that the data volume of the sub-data items contained in each execution unit is significantly smaller than the data volume of the sub-data items contained within the main data. In another example, if the data to be accessed is small (corresponding to a sub-data volume less than a data volume threshold), it may not need to be segmented, and the corresponding parsing processing can be performed directly.

[0061] In one example, when the amount of data to be accessed is detected to be greater than or equal to a data volume threshold, the data to be accessed can be divided into multiple execution units. The data volume threshold can be adjusted according to actual circumstances; in this embodiment, no specific limit is imposed on the data volume threshold.

[0062] In step 320, the sub-data to be accessed in the execution unit is parsed in parallel according to the data parsing process to obtain the parsed execution unit.

[0063] In step 330, the parsed execution units are integrated to obtain the parsed data.

[0064] In one embodiment, the sub-data to be accessed in the execution unit can be parsed in parallel by threads according to the data parsing process to obtain each parsed execution unit. Furthermore, the parsed execution units can be integrated to obtain the parsed data. In this embodiment, by dividing the data to be accessed into multiple execution units and parsing these units in parallel by threads, the parsing efficiency of the data to be accessed can be improved.

[0065] To further illustrate the data access processing method provided in this disclosure, the following embodiments will be used as examples.

[0066] Continue with Figure 3 The above embodiment is used as an example for illustration. The sub-data to be accessed corresponds to a data row number. The parallel parsing of the sub-data to be accessed in the execution unit according to the data parsing process can also be implemented in the following way: store the data row number of the parsed sub-data to be accessed; in response to the parsing interruption instruction, interrupt the parsing of the execution unit; based on the data row number, determine the next sub-data to be accessed after the sub-data to be accessed corresponding to the data row number; and continue to parse the execution unit starting from the next sub-data to be accessed.

[0067] In one embodiment, to prevent duplicate processing of the sub-data to be accessed in the execution unit due to parsing interruption or restart, it is necessary to ensure the number of parsed contents and the normal storage of each parsed data item during execution. In one example, the row numbers of the parsed sub-data to be accessed can be stored in a disaster recovery device, such as a Redis cluster. During the application process, the completion of all execution units marks the completion of a file (corresponding to the data to be accessed) processing task.

[0068] During application, the parsing results of each sub-data to be accessed in the execution unit can also be stored in the data detail table. This facilitates subsequent business processing item by item. During the parsing of the data to be accessed, the details of the data to be accessed (corresponding to each sub-data to be accessed) will be obtained, and all the parsed detail data (corresponding to the parsing results of each sub-data to be accessed) can be stored in the data detail table.

[0069] To further illustrate the data access and processing methods provided in this disclosure, the following will be combined with... Figure 4 Please provide an explanation.

[0070] Figure 4 This is the second flowchart of the data access processing method provided in this disclosure.

[0071] In an exemplary embodiment of this disclosure, as Figure 4As shown, the data access processing method may include steps 410 to 460, wherein steps 410 to 420 are the same as or similar to steps 310 to 320, and step 460 is the same as or similar to step 330. For specific implementation methods and beneficial effects, please refer to the previous description. In this embodiment, they will not be repeated. Steps 420 to 450 will be described below.

[0072] In step 430, the parsing results of the sub-data to be accessed are saved to the data details table, where the parsing results include successful parsing results and unsuccessful parsing results.

[0073] In one embodiment, during the parsing of the data to be accessed, the details of the data to be accessed (corresponding to each sub-data to be accessed) are obtained. All the parsed detailed data (corresponding to the parsing results of each sub-data to be accessed, including parsing success results and parsing failure results) can be stored in the data detail table to facilitate subsequent business processing.

[0074] In step 440, if the number of actual parsed sub-data to be accessed in the execution unit is different from the number of sub-data to be accessed contained in the execution unit, it is determined that there is sub-data to be accessed in the execution unit that has failed to be parsed.

[0075] In step 450, the data detail table is traversed to identify the sub-data to be accessed that failed to be parsed, and the sub-data to be accessed that failed to be parsed is parsed again according to the data parsing process to obtain the parsed execution unit.

[0076] During application, if the actual number of sub-data to be accessed parsed in the execution unit is different from the number of sub-data to be accessed contained in the execution unit, it can be determined that there is sub-data to be accessed that failed to be parsed in the execution unit. Further, the data detail table can be traversed, and based on the parsing results stored in the data detail table, the sub-data to be accessed that failed to be parsed can be identified. Then, the sub-data to be accessed that failed to be parsed can be parsed again according to the data parsing process, thereby obtaining the parsed execution unit.

[0077] It should be noted that the module capable of statistically analyzing the parsing results of the data to be accessed can be called the task processing and analysis module. In another example, the task processing and analysis module can perform phased statistical analysis on the execution status of the entire data parsing process. If any process is found to have failed, the data compensation module will be notified to perform compensation processing. For example, if unparsed data is found in the execution unit, the unparsed data will be parsed again according to the data parsing process to obtain the parsed execution unit. If all processes are processed correctly, the file processing module will be notified to aggregate the final processing results.

[0078] The data compensation module is used to compensate for data that requires compensation. During application, real-time or offline compensation is also performed based on this module. In practice, the data compensation module can have a corresponding temporary table for storing abnormal data. All abnormal data occurring during the entire process is stored in this temporary table; its purpose is to facilitate the location and handling of abnormalities. The data in the table is stored according to the specific process, down to the node level. This makes it very easy to locate abnormalities in each process, facilitating the triggering of different responsible modules for compensation. In one example, the temporary table for storing abnormal data could be composed of the unparsed sub-data to be accessed from the data details table.

[0079] It should be noted that the module capable of performing the data parsing steps can also be called a file processing module. To further explain the data access processing method provided in this disclosure, the following description will focus on the file processing module to illustrate the data access processing process.

[0080] Figure 5 This is the third flowchart of the data access processing method provided in this disclosure.

[0081] In one embodiment, combined with Figure 5 As can be seen, the data access processing method may include steps 501 to 515, and each step will be described below.

[0082] In step 501, the file processing task is monitored.

[0083] In step 502, the channel service configuration is obtained based on the identifier.

[0084] In one example, file processing tasks can be monitored, where each task can be understood as a general message issued by the external interaction module. This general message includes the storage path of the data to be accessed and the data processing flow for that data. Furthermore, based on the access standard of the data to be accessed, the channel service configuration corresponding to the access standard (corresponding to the data processing flow described in this document) can be determined through an access standard configuration table.

[0085] In step 503, it is determined whether file parsing is required.

[0086] In step 504, the corresponding steps are executed by the data push module.

[0087] In step 505, Redis obtains the current execution flow.

[0088] During application, it can be determined whether the data to be accessed needs file parsing. If file parsing is already complete, it is not necessary to repeat the parsing process; the parsed data can be directly pushed to the data push module to execute the corresponding steps. If the file is not parsed, it needs to be parsed (corresponding to the parsing of the data to be accessed). In this case, Redis can obtain the current execution flow of the data to be accessed and start execution from the current flow to avoid re-executing completed steps.

[0089] It should be noted that the file processing module can asynchronously push the parsed data of the data to be accessed to the data push module. Since this data is often just plain text, if insurance business processing is to be performed on this text data, the data push module needs to process the parsed data as input parameters, packaging it according to the requirements of the downstream access system, i.e., assigning business meaning to the parsed data. During application, relevant configurations (i.e., message field mapping relationships) can be obtained from the access standard configuration table based on the channel code, insurance type, and business type. Based on the configuration, each piece of data in the plain text is automatically mapped to the corresponding message field to obtain the processed access data. Furthermore, the validity of the message fields can be checked. If correct, the assembled data (corresponding to the processed access data) can be directly pushed to the downstream access system. In another example, the processing status returned by the downstream access system can also be updated in the data details table for later process statistics and result feedback.

[0090] In step 506, it is determined whether the initial state has not been processed.

[0091] In step 507, the status of each node in the execution process is scanned to restore the process.

[0092] In step 508, the file is retrieved.

[0093] In one embodiment, it can be determined whether the data to be accessed is in an initial, unprocessed state. If not, the status of each node in the execution process can be scanned to restore the process. If so, the file (corresponding to the data to be accessed) can be retrieved from the corresponding storage address based on the storage path of the data to be accessed.

[0094] In step 509, the file is physically sliced ​​according to the cutting configuration to generate an execution unit.

[0095] In step 510, the execution units are parsed in parallel.

[0096] In step 511, each thread processes the execution unit.

[0097] In application processes, the data to be accessed is often large in volume, or a large amount of data may need to be accessed and processed simultaneously. To improve access processing efficiency, the data to be accessed can be segmented. In one example, based on the segmentation configuration, for example, the data to be accessed can be segmented into execution units of 1000 data entries each. It can be understood that an execution unit includes 1000 sub-data entries to be accessed. Further, the sub-data entries within the execution unit are parsed in parallel according to the data parsing process, resulting in parsed execution units. These parsed execution units are then integrated to obtain the parsed data.

[0098] In step 512, the file processing range of the current execution unit is obtained.

[0099] In step 513, each line of data in the file is traversed, the parsing result is stored, and the current line number is recorded.

[0100] In another example, the data row number of the parsed sub-data to be accessed can be stored; in response to a parsing interruption of the execution unit, based on the data row number, the next sub-data to be accessed after the sub-data to be accessed corresponding to the data row number is determined, and the parsing of the execution unit continues from the next sub-data to be accessed. Through this embodiment, the repeated parsing of already parsed sub-data to be accessed can be avoided.

[0101] In step 514, the parsing results are pushed to the data push module.

[0102] In step 515, check whether the push fails. If the number of failures exceeds the threshold, an abnormal alarm is issued.

[0103] During application, the parsed data of the data to be accessed can be asynchronously pushed to the data push module, and the data push module will process the input parameters of the parsed data to give it meaning, thereby meeting the packaging requirements of the downstream access system.

[0104] In another example, it's also possible to verify whether the parsing results were successfully pushed to the data push module. If unsuccessful, it can be pushed again. If multiple pushes fail consecutively, an error alert can be triggered. Specifically, if the number of consecutive failed pushes exceeds a certain threshold, it can be determined that the parsing results have been repeatedly pushed unsuccessfully.

[0105] The data access and processing method disclosed herein standardizes and streamlines offline insurance order-taking scenarios. It binds the data processing flow to the access channels, insurance types, and business types (corresponding to the access standards for the data to be accessed). The system's processing, from order acceptance, file retrieval, segmentation, and parsing to personalized message parsing, achieves full automation and streamlined workflow. Furthermore, when connecting to new channels or new products (corresponding to new access standards), simply configuring the new access standards in the access standard configuration table requires no additional development. Simultaneously, strategy configurations can be made for processing details, allowing the system to more easily handle personalized channels.

[0106] As described above, the data access processing method provided in this disclosure determines the correspondence between access standards and data processing procedures by pre-setting an access standard configuration table. After acquiring the data to be accessed, the data processing procedure for the data to be accessed can be directly determined based on the access standard of the data to be accessed, and the data to be accessed is processed according to the data processing procedure to obtain the processed access data. This disclosure achieves the use of the same processing procedure for accessing different types of data, thereby ensuring the standardization and streamlining of data access processing, improving system resource utilization and future maintainability.

[0107] Based on the same concept, this disclosure also provides a data access processing apparatus.

[0108] The data access processing apparatus provided in this disclosure is described below. The data access processing apparatus described below can be referred to in correspondence with the data access processing method described above.

[0109] Figure 6 This is a schematic diagram of the data access processing device provided in this disclosure.

[0110] In an exemplary embodiment of this disclosure, as Figure 6 As shown, the data access processing device may include an acquisition module 610, a determination module 620, and a processing module 630. Each module will be described in detail below.

[0111] The acquisition module 610 can be configured to acquire the data to be accessed and the access standard corresponding to the data to be accessed.

[0112] The determination module 620 can be configured to obtain an access standard configuration table and, based on the access standard configuration table, determine the data processing flow corresponding to the access standard of the data to be accessed. Different access standards in the access standard configuration table correspond to different data processing flows. The processing module 630 can be configured to process the data to be accessed according to the data processing flow to obtain processed access data.

[0113] In an exemplary embodiment of this disclosure, the data processing flow may include a data parsing flow and a data mapping flow. The processing module 630 may process the data to be accessed according to the data processing flow in the following manner to obtain the processed access data: parse the data to be accessed according to the data parsing flow to obtain parsed data; map the parsed data to the corresponding message field according to the data mapping flow to obtain the processed access data, wherein the message field is pre-stored in the access standard configuration table.

[0114] In an exemplary embodiment of this disclosure, different access standards in the access standard configuration table also correspond to different message field mapping relationships. The message field mapping relationship is the mapping relationship between the parsed data and the message fields after the data to be accessed is parsed. The processing module 630 can map the parsed data to the corresponding message fields according to the data mapping process in the following manner to obtain the processed access data: Based on the access standard configuration table, determine the message field mapping relationship corresponding to the access standard of the parsed data; based on the message field mapping relationship, map the parsed data to the corresponding message fields according to the data mapping process to obtain the processed access data.

[0115] In an exemplary embodiment of this disclosure, the data to be accessed may include multiple sub-data items to be accessed. The processing module 630 may parse the data to be accessed according to the data parsing process in the following manner to obtain parsed data: if it is determined that the data volume of the sub-data to be accessed is greater than or equal to the data volume threshold, the data to be accessed is divided into multiple execution units; the sub-data to be accessed in the execution units are parsed in parallel according to the data parsing process to obtain parsed execution units; and the parsed execution units are integrated and processed to obtain parsed data.

[0116] In an exemplary embodiment of this disclosure, the sub-data to be accessed corresponds to a data row number; the processing module 630 can also parse the sub-data to be accessed in the execution unit in parallel according to the data parsing process in the following manner: storing the data row number of the parsed sub-data to be accessed; in response to the parsing interruption instruction, interrupting the parsing of the execution unit, determining the next sub-data to be accessed after the sub-data to be accessed corresponding to the data row number based on the data row number, and continuing to parse the execution unit starting from the next sub-data to be accessed.

[0117] In an exemplary embodiment of this disclosure, the processing module 630 may further be configured to save the parsing results of the sub-data to be accessed to a data detail table, wherein the parsing results may include parsing success results and parsing failure results; if the number of sub-data to be accessed actually parsed in the execution unit is different from the number of sub-data to be accessed contained in the execution unit, it is determined that there is sub-data to be accessed that has failed to be parsed in the execution unit; the data detail table is traversed to determine the sub-data to be accessed that has failed to be parsed, and the sub-data to be accessed that has failed to be parsed is parsed again according to the data parsing process.

[0118] In an exemplary embodiment of this disclosure, the acquisition module 610 may acquire the data to be accessed and the access standard corresponding to the data to be accessed in the following manner: in response to receiving a data access request, the metadata of the data access request is parsed to obtain the storage path of the data to be accessed in the data access request and the access standard corresponding to the data to be accessed; based on the storage path, the data to be accessed is acquired.

[0119] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a data access processing method. This method includes: acquiring the data to be accessed and the access standard corresponding to the data; acquiring an access standard configuration table and determining the data processing flow corresponding to the access standard of the data to be accessed based on the access standard configuration table, wherein different access standards correspond to different data processing flows in the access standard configuration table; and processing the data to be accessed according to the data processing flow to obtain processed access data.

[0120] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] On the other hand, this disclosure also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data access processing method provided by the above methods. The method includes: acquiring data to be accessed and an access standard corresponding to the data to be accessed; acquiring an access standard configuration table and determining a data processing flow corresponding to the access standard of the data to be accessed based on the access standard configuration table, wherein different access standards in the access standard configuration table correspond to different data processing flows; and processing the data to be accessed according to the data processing flow to obtain processed access data.

[0122] In another aspect, this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the data access processing method provided by the above methods. The method includes: acquiring data to be accessed and an access standard corresponding to the data to be accessed; acquiring an access standard configuration table and determining a data processing flow corresponding to the access standard of the data to be accessed based on the access standard configuration table, wherein different access standards in the access standard configuration table correspond to different data processing flows; and processing the data to be accessed according to the data processing flow to obtain processed access data.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A data access processing method, characterized in that, The method includes: Obtain the data to be accessed and the access standard corresponding to the data to be accessed; Obtain the access standard configuration table, and determine the data processing flow corresponding to the access standard of the data to be accessed based on the access standard configuration table. In the access standard configuration table, different access standards correspond to different data processing flows. The data to be accessed is processed according to the data processing flow to obtain processed access data. The data processing flow includes a data parsing flow and a data mapping flow. Specifically, processing the data to be accessed according to the data processing flow to obtain processed access data includes: The data to be accessed is parsed according to the data parsing process described above to obtain the parsed data; The parsed data is mapped to the corresponding message fields according to the data mapping process to obtain processed access data, wherein the message fields are pre-stored in the access standard configuration table.

2. The data access processing method according to claim 1, characterized in that, In the access standard configuration table, different access standards also correspond to different message field mapping relationships, wherein the message field mapping relationship is the mapping relationship between the parsed data after the data to be accessed is parsed and the message fields; The step of mapping the parsed data to the corresponding message fields according to the data mapping process to obtain processed access data specifically includes: Based on the access standard configuration table, determine the message field mapping relationship corresponding to the access standard of the parsed data; Based on the message field mapping relationship, the parsed data is mapped to the corresponding message fields according to the data mapping process to obtain the processed access data.

3. The data access processing method according to claim 1, characterized in that, The data to be accessed includes multiple sub-data items to be accessed. The process of parsing the data to be accessed according to the data parsing procedure to obtain the parsed data specifically includes: If the amount of data to be accessed is greater than or equal to the data amount threshold, the data to be accessed is divided into multiple execution units. The data parsing process described above is used to parse the sub-data to be accessed in the execution unit in parallel to obtain the parsed execution unit. The parsed execution unit is integrated and processed to obtain the parsed data.

4. The data access processing method according to claim 3, characterized in that, The sub-data to be accessed corresponds to a data row number; The step of parsing the sub-data to be accessed in the execution unit in parallel according to the data parsing process specifically includes: Store the parsed data row number of the sub-data to be accessed; In response to the parsing interrupt instruction, the execution unit is interrupted. Based on the data row number, the next sub-data to be accessed after the sub-data to be accessed corresponding to the data row number is determined, and the parsing of the execution unit continues from the next sub-data to be accessed.

5. The data access processing method according to claim 3, characterized in that, After parsing the sub-data to be accessed in the execution unit in parallel according to the data parsing process, the method further includes: The parsing results of the sub-data to be accessed are saved to a data detail table, wherein the parsing results include parsing success results and parsing failure results; If the number of sub-data to be accessed that is actually parsed in the execution unit is different from the number of sub-data to be accessed contained in the execution unit, it is determined that there is sub-data to be accessed that cannot be parsed successfully in the execution unit. Traverse the data details table to identify the unsuccessfully parsed sub-data to be accessed, and re-parse the unsuccessfully parsed sub-data according to the data parsing process.

6. The data access processing method according to claim 1, characterized in that, The acquisition of the data to be accessed and the access standard corresponding to the data to be accessed specifically includes: In response to receiving a data access request, the metadata of the data access request is parsed to obtain the storage path of the data to be accessed and the access standard corresponding to the data to be accessed. Based on the storage path, obtain the data to be accessed.

7. A data access processing device, characterized in that, The apparatus is used to implement the data access processing method according to any one of claims 1 to 6, and the apparatus comprises: The acquisition module is used to acquire the data to be accessed and the access standard corresponding to the data to be accessed; The determination module is used to obtain the access standard configuration table and determine the data processing flow corresponding to the access standard of the data to be accessed based on the access standard configuration table. In the access standard configuration table, different access standards correspond to different data processing flows. The processing module is used to process the data to be accessed according to the data processing flow to obtain the processed accessed data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data access processing method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data access processing method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data access processing method as described in any one of claims 1 to 6.

Images