Data processing method, data processing system, storage medium and program product
The data processing method using non-relational databases and data identifier queues addresses inefficiencies in data retrieval and recalculations, optimizing resource usage and scalability in business logic processing.
Patent Information
- Application Number
- CN202411280671.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-31
AI Technical Summary
Existing data processing systems face inefficiencies in repeatedly obtaining data from upstream sources, leading to resource occupation and recalculations when small data changes occur, especially in business logic processing involving multiple tables and memory databases.
A data processing method utilizing a non-relational database with data identifiers stored in queues, allowing selective retrieval of required data types based on association relationships, reducing redundant data access and optimizing resource usage.
Enhances efficiency by minimizing redundant data retrieval and recalculations, optimizing resource usage, and enabling scalable data processing in distributed systems.
Smart Images
Figure CN119226350A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, a data processing system, a readable storage medium, and a computer program product. Background Art
[0002] When the data processing system performs business logic processing, it needs to obtain data from multiple different data sources for calculation. Before calculation, the data needs to be pulled from different upstream data sources to the MySQL database of the data processing system. During calculation, the pulled data is loaded into the memory database, and fixed business rules are used to match the corresponding data, and then the data is processed.
[0003] However, each business module has the problem of repeatedly obtaining data from upstream data; and the data processing of each business module involves the association processing of multiple tables, which is inefficient. Using MySQL database to store the acquired data and subsequent calculation data cannot achieve flexible expansion, and a large amount of data is stored in the memory database waiting for calculation, which occupies a lot of resources; when a small amount of data changes, all data needs to be recalculated, which is inefficient. Summary of the invention
[0004] In order to solve one of the above technical problems, the present disclosure provides a data processing method, a data processing system, a readable storage medium and a computer program product.
[0005] According to one aspect of the present disclosure, there is provided a data processing method, comprising: In response to a first business processing request, obtaining a first data association relationship corresponding to the first business processing request, where the first data association relationship represents a data type required by the first business processing request; Acquire a data identifier of data of a data type required by the first business processing request from a public database based on the first data association relationship, and store the data identifier in a queue of data to be calculated; In response to the data identifiers of the data types required by the first business processing request being stored in the to-be-calculated data queue, the data identifiers corresponding to the data types required by the first business processing request are read from the to-be-calculated data queue; Based on the read data identifier, data corresponding to the data identifier is obtained from the public database to obtain the data to be calculated, and the data to be calculated is processed based on the business processing rule of the first business processing request to obtain a data processing result.
[0006] According to the data processing method of at least one embodiment of the present disclosure, storing the data identifiers in the data queue to be calculated includes: storing the data identifiers of data of different data types in the data queue to be calculated corresponding to each data type respectively.
[0007] According to the data processing method of at least one embodiment of the present disclosure, the reading of the data identifier corresponding to the data type required by the first business processing request from the data queue to be calculated includes: reading the data identifiers separately from the data queues to be calculated corresponding to the respective data types.
[0008] According to at least one embodiment of the present disclosure, the data processing method further includes: In response to the second business processing request, obtaining a second data association relationship corresponding to the second business processing request, where the second data association relationship represents a data type required by the second business processing request; Reading a data identifier corresponding to the data type required by the second business processing request from the to-be-calculated data queue; In response to the fact that not all data identifiers of data of the data type required by the second business processing request are obtained from the queue of data to be calculated, determining the data type missing from the queue of data to be calculated; Acquire the data identifier of the missing data type from the public database, and store the data identifier in the to-be-calculated data queue; In response to the data identifiers of the data types required by the second business processing request being stored in the to-be-calculated data queue, the data identifiers corresponding to the data types required by the second business processing request are read from the to-be-calculated data queue; Based on the read data identifier, data corresponding to the data identifier is obtained from the public database to obtain the data to be calculated, and the data to be calculated is processed based on the business processing rule of the second business processing request to obtain a data processing result.
[0009] According to the data processing method of at least one embodiment of the present disclosure, in response to the data identifiers of the data types required by the second business processing request, all the data identifiers of the data are obtained from the data queue to be calculated, then based on the obtained data identifiers, the data corresponding to the data identifiers are obtained from the public database to obtain the data to be calculated, and the data to be calculated are processed based on the business processing rules of the second business processing request to obtain a data processing result.
[0010] According to the data processing method of at least one embodiment of the present disclosure, a plurality of data sets are configured on the public database, each data set stores data of the same data type, and different data sets store data of different data types.
[0011] According to the data processing method of at least one embodiment of the present disclosure, the data in the public database is updated based on the following processing process: Get new data from upstream data sources; Extracting the data type of the new data to obtain the data type of the new data; Extracting index information of the new data based on the data type of the new data; Based on the data type and index information of the new data, a data identifier is configured for the new data, and the new data configured with the data identifier is updated into a data set corresponding to the data type in the public database.
[0012] According to the data processing method of at least one embodiment of the present disclosure, the data identifier is a globally unique identifier.
[0013] According to the data processing method of at least one embodiment of the present disclosure, the public database is a non-relational database, and the non-relational database stores the new data in the form of column families.
[0014] According to the data processing method of at least one embodiment of the present disclosure, updating the data in the public database includes: one or more of adding data, deleting data, and changing data.
[0015] According to another aspect of the present disclosure, there is provided a data processing system, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the data processing method of any one embodiment of the present disclosure.
[0016] According to another aspect of the present disclosure, a readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the data processing method described in any embodiment of the present disclosure.
[0017] According to another aspect of the present disclosure, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, it is at least used to implement the data processing method described in any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.
[0019] Figure 1It is a flowchart of a data processing method according to an embodiment of the present disclosure.
[0020] Figure 2 It is a flowchart of a data processing method according to another embodiment of the present invention.
[0021] Figure 3 It is a flowchart of a data processing method according to another embodiment of the present invention.
[0022] Figure 4 It is a flowchart of a method for updating data in a public database according to an embodiment of the present invention.
[0023] Figure 5 A schematic diagram showing a data processing method executed by a data processing system and a method for updating data in a public database according to an embodiment of the present disclosure is shown.
[0024] Figure 6 It is a schematic block diagram of the structure of a data processing system according to one embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.
[0026] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0027] Business processing requests of data processing systems (such as human resource management systems) may require obtaining data from multiple different data sources for calculation. Before calculation, data needs to be obtained from different upstream data sources and stored in the database of the data processing system. During calculation, the data is processed using pre-set business processing rules.
[0028] Taking employee salary calculation as an example of a business processing request, employee salary calculation requires the participation of multiple business modules, such as basic salary calculation module, attendance statistics module, performance calculation module, etc.
[0029] The implementation method of the related technology is that each business module obtains the required data from the upstream data source, such as the basic salary calculation module obtains the required employee basic information and other data, the attendance statistics module obtains the required employee basic information, employee attendance and other data, and the performance calculation module obtains the required employee basic information, employee performance and other data, and uses the MySQL database to store the obtained data. However, each business module has the problem of repeatedly obtaining data from the upstream data source. For example, the basic salary calculation module, the attendance statistics module and the performance calculation module will repeatedly obtain the basic information of the employees, and use the MySQL database to store the obtained data. A large amount of data is stored in the database waiting for calculation, which occupies a large amount of resources; when a small amount of data changes, such as when the basic information of the employee changes (such as employee resignation, employee recruitment, employee rank change, etc.), the basic salary calculation module, the attendance statistics module, the performance calculation module, etc. need to re-obtain data and recalculate, which is inefficient.
[0030] In view of this, the present disclosure provides a new data processing method and a data processing system, which can solve the above technical problems.
[0031] Figure 1 It is a flowchart of a data processing method according to an embodiment of the present disclosure.
[0032] refer to Figure 1 In some embodiments of the present disclosure, the data processing method M100 includes: S102: In response to the first business processing request, obtain a first data association relationship corresponding to the first business processing request, where the first data association relationship represents a data type required by the first business processing request; S104, obtaining a data identifier of data of a data type required by the first business processing request from a public database based on the first data association relationship, and storing the data identifier in a queue of data to be calculated; S106, in response to the data identifiers of the data type required by the first business processing request being stored in the to-be-calculated data queue, read the data identifier corresponding to the data type required by the first business processing request from the to-be-calculated data queue; S108. Based on the read data identifier, obtain data corresponding to the data identifier from a public database to obtain the data to be calculated, and process the data to be calculated based on the business processing rule of the first business processing request to obtain a data processing result.
[0033] The data identifier described in the present disclosure is a globally unique identifier. The public database described in the present disclosure is preferably a non-relational database, wherein the non-relational database can store data in the form of column families.
[0034] Among them, non-relational databases are Nosql (Not Only SQL). The data storage method of non-relational databases is to combine large blocks of data together and store them in data sets, and can be stored in the form of document results, graph structures or key-value. The non-relational databases used in this disclosure include but are not limited to Redis, MongoDB, etc.
[0035] The data processing method disclosed in the present invention can be applied to a distributed business processing system. For example, a human resource management system can be configured as an architecture of a distributed business processing system.
[0036] The first business processing request described above may be, for example, an employee attendance statistics request or an employee basic salary calculation request. The present disclosure may pre-configure respective data association relationships (i.e., data dependency relationships) for employee attendance statistics requests, employee basic salary calculation requests, etc. to characterize the data type required for each business processing request. For example, the data type required for the employee attendance statistics request is employee basic information data and employee attendance data.
[0037] After obtaining the data type required for the first business processing request, the data identifier of the data type required for the first business processing request is obtained from the public database based on the data association relationship, and the data identifier is stored in the data queue to be calculated, wherein the data queue to be calculated is configured on a database outside the above-mentioned public database, such as a redis database.
[0038] When the data identifiers of the data types required by the first business processing request (for example, an employee attendance statistics request) have all been stored in the queue of data to be calculated, the data identifier corresponding to the data type required by the first business processing request is read from the above-mentioned queue of data to be calculated, and based on the read data identifier, the data corresponding to the data identifier is obtained from the above-mentioned public database to obtain the data to be calculated. The data to be calculated can be acquired into the memory corresponding to the attendance statistics module of the distributed business processing system, and the data to be calculated can be processed based on the business processing rules of the first business processing request (for example, employee attendance statistics rules) to obtain a data processing result.
[0039] The data processing method of the above-mentioned embodiment first reads the data identifier of the data of the required data type from the public database to the data queue to be calculated based on the data type required by the first business processing request. When the data identifiers of the data of the required data type are all stored in the data queue to be calculated, the corresponding data is read from the public database to the memory for calculation based on the data identifier in the data queue to be calculated, thereby avoiding directly reading a large amount of data from the public database and causing a large amount of memory occupation.
[0040] In some embodiments of the present disclosure, after the data identifiers in the data queue to be calculated are read to read the corresponding data from the public database, the data identifiers in the data queue to be calculated are assigned a read identifier or a timestamp, and after a preset time length (for example, 1 hour, 5 hours, 24 hours, etc.) has passed, these data identifiers are removed from the data queue to be calculated.
[0041] Figure 2 It is a flowchart of a data processing method according to another embodiment of the present invention.
[0042] See also Figure 2 In some embodiments of the present disclosure, the data processing method M200 includes: S202: In response to the second business processing request, obtain a second data association relationship corresponding to the second business processing request, where the second data association relationship represents a data type required by the second business processing request; S204, reading a data identifier corresponding to the data type required by the second business processing request from the to-be-calculated data queue; S206, in response to the fact that the data identifiers of the data types required by the second service processing request are not all obtained from the queue of data to be calculated, determining the data types missing from the queue of data to be calculated; S208, obtaining data identifiers of data of the missing data type from the public database, and storing the data identifiers in a queue of data to be calculated; S210, in response to the data identifiers of the data type required by the second business processing request being stored in the to-be-calculated data queue, read the data identifier corresponding to the data type required by the second business processing request from the to-be-calculated data queue; S212. Based on the read data identifier, obtain data corresponding to the data identifier from a public database to obtain the data to be calculated, and process the data to be calculated based on the business processing rule of the second business processing request to obtain a data processing result.
[0043] Based on the data processing method of this embodiment, the data identifier corresponding to the data type required by the second business processing request (for example, an employee performance calculation request) is first read from the data queue to be calculated, and the data identifier is first read from the data queue to be calculated to avoid repeatedly reading the data identifier in the public database.
[0044] Figure 3 It is a flowchart of a data processing method according to another embodiment of the present invention.
[0045] refer to Figure 3 In some embodiments of the present disclosure, the data processing method M200 includes: S202: In response to the second business processing request, obtain a second data association relationship corresponding to the second business processing request, where the second data association relationship represents a data type required by the second business processing request; S204, reading a data identifier corresponding to the data type required by the second business processing request from the to-be-calculated data queue; S206, in response to the second business processing request, all data identifiers of the data types required by the request are obtained from the queue of data to be calculated, and based on the obtained data identifiers, data corresponding to the data identifiers are obtained from a public database to obtain the data to be calculated; S208. Process the data to be calculated based on the business processing rule of the second business processing request to obtain a data processing result.
[0046] In some embodiments of the present disclosure, Figure 2 , Figure 3 The data processing process of the data processing method shown can be Figure 1 The data processing method shown is executed after it is completed. For example, the first business processing request is a request for calculating the basic salary of an employee, and the second business processing request is a request for employee attendance statistics. For example, the request for calculating the basic salary of an employee will store the data identifier of the employee's basic information data in the queue of data to be calculated. After these data identifiers are used by the request for calculating the basic salary of an employee, they can be assigned a read identifier or a timestamp. Within the preset time length described above, these data identifiers are still in the queue of data to be calculated. Therefore, during the execution of the second business processing request, the data identifier of the data of the required data type can be read from the queue of data to be calculated first, avoiding repeated reading of related data identifiers from the public database.
[0047] In some embodiments of the present disclosure, after the second business processing request has completed the use of the data identifiers in the data queue to be processed, these data identifiers will be reassigned with a read identifier or timestamp so that subsequent business processing requests can still read the data identifiers of the required type of data from the data queue to be processed.
[0048] It should be noted that in other embodiments of the present disclosure, Figure 2 , Figure 3 The data processing process of the data processing method shown may also not depend on Figure 1 Execution of the data processing method is shown.
[0049] In some embodiments of the present disclosure, data identifiers of data of different data types may be stored in the to-be-calculated data queues corresponding to the respective data types. In other words, multiple to-be-calculated data queues are configured, each to-be-calculated data queue stores data identifiers of data of a data type, and accordingly, in the process of executing the above-described reading of the data identifier corresponding to the data type required for the first business processing request or the second business processing request from the to-be-calculated data queue, the data identifiers are read separately from the to-be-calculated data queues corresponding to the respective data types.
[0050] In some embodiments of the present disclosure, multiple data sets are configured on the public database described in the present disclosure, each data set stores data of the same data type, and different data sets store data of different data types.
[0051] The present disclosure also provides a method for updating data in a public database. Figure 4 It is a flowchart of a method for updating data in a public database according to an embodiment of the present invention.
[0052] refer to Figure 4 In some embodiments of the present disclosure, the data updating method M300 of the public database of the present disclosure includes: S302, obtaining new data from an upstream data source; S304, extracting the data type of the new data to obtain the data type of the new data; S306, extracting index information of the new data based on the data type of the new data (index information such as employee ID: E0001); S308: Configure the new data based on the data type and index information of the new data, and update the new data configured with the data identifier into the data set corresponding to the data type in the public database.
[0053] Among them, the data type can be determined based on data characteristics or data field characteristics. For example, employee basic information can be used as a data type (which can be identified by employinfo), employee attendance data can be used as another data type (which can be identified by AttendanceInfo), employee performance data can be used as another data type, and so on.
[0054] For example, the data obtained from the upstream data source may be employee basic information data, employee attendance data, and employee performance data. The index information of the employee basic information may be the employee ID of the employee, the index information of the employee attendance data may also be the employee ID, and the index information of the employee performance data may be the contract number of the performance contract and the employee ID of the employee who completed the performance contract.
[0055] Based on the above process, index information of various types of data can be obtained.
[0056] The data identification described above may be configured as, for example, employeeinfo-E0001 based on the data type and index information.
[0057] Figure 5 A schematic diagram showing a data processing method executed by a data processing system and a method for updating data in a public database according to an embodiment of the present disclosure is shown.
[0058] refer to Figure 5 In step 101, at least one upstream data source (e.g. Figure 5 New data is obtained from upstream data source 1, upstream data source 2, and upstream data source 3).
[0059] Among them, the method of obtaining data from the upstream data source can be as follows Figure 5 The file reading, interface calling, message consumption and other methods shown in the text are not particularly limited in this disclosure and all fall within the protection scope of this disclosure.
[0060] Exemplarily, upstream data source 1 may be employee basic information data of all employees, upstream data source 2 may be employee attendance data of all employees, and upstream data source 3 may be completed employee performance data (eg, performance contract data) of all employees.
[0061] When collecting data from each upstream data source, the data can be collected in the form of data packets.
[0062] In step 102, data type extraction and index information extraction are performed on the collected data based on pre-configured extraction rules, and a data identifier (unique identifier) is constructed for the collected data.
[0063] Next, the new data configured with the data identifier is updated to the data set corresponding to the data type in the public database.
[0064] Among them, employee basic information data may include employee work number, employee name, company length of service, employee rank, etc.; employee attendance data may include employee work number, employee name, monthly attendance days, daily attendance hours, etc.; employee performance data may include employee work number, employee name, number of performance contracts, performance contract number, performance contract amount, etc.
[0065] When service 1 (e.g. basic salary calculation module), service 2 (e.g. attendance statistics module), and service 3 (e.g. performance calculation module) in a distributed business processing system (e.g. human resource management system) issues a business processing request, taking service 1 issuing a business processing request as an example, the rule engine (which can be implemented through a rule algorithm module) will obtain the data association relationship corresponding to the business processing request, obtain the data identifier of the data type required for the business processing request from the public database based on the data association relationship, and store the data identifier in the data queue to be calculated.
[0066] When the data identifiers of the data types required by the business processing request have been stored in the data queue to be calculated, service 1 reads the data identifier corresponding to the required data type from the data queue to be calculated, and obtains the data corresponding to the data identifier from the public database based on the read data identifier to obtain the data to be calculated, and then processes the data to be calculated based on the business processing rules related to the basic salary calculation to obtain the basic salary calculation results of each employee.
[0067] At this time, the data identifier of the employee's basic information data has been stored in the data queue to be calculated. When service 2 issues a business processing request to perform employee attendance statistics, the rule engine will obtain the data association relationship corresponding to the business processing request. Based on the data association relationship, it can first read the data identifier of the data of the required data type from the data queue to be calculated. The data identifier of the employee's basic information data will be read, and the data identifier of the employee's attendance data cannot be read. The data identifier of the employee's attendance data can continue to be obtained from the public database and stored in the data queue to be calculated.
[0068] When the rule engine determines that the data identifiers of the data types required for the above-mentioned business processing request issued by service 2 have been stored in the data queue to be calculated, service 2 reads the data identifier corresponding to the required data type from the data queue to be calculated, and obtains the data corresponding to the data identifier from the public database based on the read data identifier, obtains the data to be calculated (put into the memory), and processes the data to be calculated based on the business processing rules related to employee attendance statistics to obtain the attendance statistics results of each employee.
[0069] The non-relational database used in the present disclosure, such as MongoDB database, is used in the data processing system to obtain data from upstream data (such as Figure 5 When upstream data sources 1, 2, and 3 in the data source database obtain new data, columns can be dynamically added based on the column family database characteristics of the MongoDB database, and the newly obtained data can be stored based on the dynamically added columns, thereby meeting the real-time nature of data acquisition by the data processing system and improving the scalability of the data processing system in acquiring data.
[0070] The updating of data in the public database described above in the present disclosure includes one or more of adding data, deleting data, and changing data.
[0071] Based on the above description, the present disclosure further provides a data processing system, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the data processing method of any embodiment of the present disclosure.
[0072] Figure 6 It is a schematic block diagram of the structure of a data processing system according to one embodiment of the present disclosure.
[0073] refer to Figure 6 In some embodiments of the present disclosure, the data processing system 1000 includes: A data collection module 1002, configured to collect new data from at least one upstream data source; Public database 1004; The data identification construction module 1006 extracts the data type of the new data to obtain the data type of the new data; based on the data type of the new data; based on the data type and index information of the new data, configures a data identification for the new data and updates the new data configured with the data identification to the data set corresponding to the data type in the public database 1004; The data association relationship library module 1008 is used to store the data association relationship corresponding to the business processing request; The rule engine module 1010 obtains the data association relationship corresponding to the business processing request from the data association relationship library module 1008, obtains the data identifier of the data type required by the business processing request from the public database 1004, and stores the data identifier in the data queue to be calculated; The to-be-calculated data queue module 1012 may be configured on a Redis database; The distributed business processing system 1014 is used to issue at least one business processing request.
[0074] The above modules of the data processing system 1000 can be implemented based on software modules. Inspired by the technical solution of the present disclosure, those skilled in the art can add, reduce or change the module configuration of the data processing system, which all fall within the protection scope of the present disclosure.
[0075] refer to Figure 6 ,The data processing system can be implemented based on hardware architecture such as processor and memory, Figure 6 The hardware architecture shown in FIG. 1 is exemplary and should not be construed as limiting the implementation of data processing system 1000 .
[0076] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.
[0077] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.
[0078] The computer program or instructions may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.
[0079] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0080] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0083] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, technical staff in this field may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, without contradiction.
[0084] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0085] It should be understood by those skilled in the art that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.
Claims
1. A data processing method, characterized in that: include: In response to a first business processing request, obtaining a first data association relationship corresponding to the first business processing request, where the first data association relationship represents a data type required by the first business processing request; Acquire a data identifier of data of a data type required by the first business processing request from a public database based on the first data association relationship, and store the data identifier in a queue of data to be calculated; In response to the data identifiers of the data types required by the first business processing request being stored in the to-be-calculated data queue, the data identifiers corresponding to the data types required by the first business processing request are read from the to-be-calculated data queue; as well as Based on the read data identifier, data corresponding to the data identifier is obtained from the public database to obtain the data to be calculated, and the data to be calculated is processed based on the business processing rule of the first business processing request to obtain a data processing result.
2. The data processing method according to claim 1, characterized in that: The step of storing the data identifier in the to-be-calculated data queue includes: The data identifiers of data of different data types are respectively stored in the to-be-calculated data queues corresponding to each data type.
3. The data processing method according to claim 1, characterized in that: The step of reading a data identifier corresponding to a data type required by the first business processing request from the to-be-calculated data queue includes: The data identifiers are read respectively from the to-be-calculated data queues corresponding to the data types.
4. The data processing method according to any one of claims 1 to 3, characterized in that: The data processing method further includes: In response to the second business processing request, obtaining a second data association relationship corresponding to the second business processing request, where the second data association relationship represents a data type required by the second business processing request; Reading a data identifier corresponding to the data type required by the second business processing request from the to-be-calculated data queue; In response to the fact that not all data identifiers of data of the data type required by the second business processing request are obtained from the queue of data to be calculated, determining the data type missing from the queue of data to be calculated; Acquire the data identifier of the missing data type from the public database, and store the data identifier in the to-be-calculated data queue; In response to the data identifiers of the data types required by the second business processing request being stored in the to-be-calculated data queue, the data identifiers corresponding to the data types required by the second business processing request are read from the to-be-calculated data queue; and Based on the read data identifier, data corresponding to the data identifier is obtained from the public database to obtain the data to be calculated, and the data to be calculated is processed based on the business processing rule of the second business processing request to obtain a data processing result.
5. The data processing method according to claim 4, characterized in that: In response to the second business processing request, all data identifiers of data of the required data type are obtained from the queue of data to be calculated, and then the data corresponding to the data identifiers are obtained from the public database based on the obtained data identifiers to obtain the data to be calculated, and the data to be calculated are processed based on the business processing rules of the second business processing request to obtain the data processing result.
6. The data processing method according to claim 1, characterized in that: The public database is configured with multiple data sets, each data set stores data of the same data type, and different data sets store data of different data types.
7. The data processing method according to any one of claims 1 to 6, characterized in that: The data in the public database is updated based on the following processing: Get new data from upstream data sources; Extracting the data type of the new data to obtain the data type of the new data; Extracting index information of the new data based on the data type of the new data; as well as Based on the data type and index information of the new data, configure a data identifier for the new data and update the new data configured with the data identifier into a data set corresponding to the data type in the public database; Optionally, the data identifier is a globally unique identifier; Optionally, the public database is a non-relational database, and the non-relational database stores the new data in the form of column families; Optionally, updating the data in the public database includes: one or more of adding data, deleting data, and changing data.
8. A data processing system, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the data processing method according to any one of claims 1 to 7.
9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the data processing method according to any one of claims 1 to 7.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program is used to implement at least the data processing method according to any one of claims 1 to 7.