Data processing method, device, computer equipment and storage medium

Abstract

The present disclosure proposes a data processing method, apparatus, computer equipment, and storage medium. The specific scheme is as follows: determining reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage status of the data to be processed; determining the data association relationship corresponding to the data to be processed based on the reference storage information; and performing a target processing operation on the target data associated with the data to be processed based on the data association relationship. Through the present disclosure, it is possible to efficiently determine the target data that has a data association relationship with the data to be processed based on the storage status of the data to be processed, and perform corresponding processing operations on the target data, which can effectively improve the efficiency of data processing and effectively simplify the data processing logic, thereby effectively improving the data processing effect.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] In the field of big data technology, when it comes to copying large quantities of business data, the usual method is to write code to copy the data of the entire business module into the program code, which includes a series of processing codes such as data relationships, data extraction, data recovery, and customized operations.

[0003] In this way, when data relationships change, a large number of dependencies and data copy codes can be adjusted. It is impossible to distinguish between data extraction and data recovery, and cross-platform data copying cannot be performed, resulting in more complex data processing logic and affecting data processing effects. Summary of the Invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the purpose of the present disclosure is to propose a data processing method, device, computer equipment and storage medium, which can efficiently determine target data that has a data association relationship with the data to be processed based on the storage situation of the data to be processed, and perform corresponding processing operations on the target data, which can effectively improve the efficiency of data processing and effectively simplify the data processing logic, thereby effectively improving the data processing effect.

[0006] To achieve the above-mentioned purpose, the data processing method proposed in the embodiment of the first aspect of the present disclosure includes: determining reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage status of the data to be processed; based on the reference storage information, determining the data association relationship corresponding to the data to be processed; based on the data association relationship, performing target processing operations on the target data associated with the data to be processed.

[0007] The data processing method proposed in the embodiment of the first aspect of the present disclosure determines reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage situation of the data to be processed, and determines the data association relationship corresponding to the data to be processed based on the reference storage information, and then performs target processing operations on the target data associated with the data to be processed based on the data association relationship. It can efficiently determine the target data that has a data association relationship with the data to be processed based on the storage situation of the data to be processed, and perform corresponding processing operations on the target data, which can effectively improve the efficiency of data processing and effectively simplify the data processing logic, thereby effectively improving the data processing effect.

[0008] To achieve the above-mentioned purpose, the data processing device proposed in the second aspect embodiment of the present disclosure includes: a first determination module, used to determine reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage status of the data to be processed; a second determination module, used to determine the data association relationship corresponding to the data to be processed based on the reference storage information; and an execution module, used to perform a target processing operation on the target data associated with the data to be processed based on the data association relationship.

[0009] The data processing device proposed in the second aspect embodiment of the present disclosure determines reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage situation of the data to be processed, and determines the data association relationship corresponding to the data to be processed based on the reference storage information, and then performs target processing operations on the target data associated with the data to be processed based on the data association relationship. It can efficiently determine the target data having a data association relationship with the data to be processed based on the storage situation of the data to be processed, and perform corresponding processing operations on the target data, which can effectively improve the efficiency of data processing, and effectively simplify the data processing logic, thereby effectively improving the data processing effect.

[0010] The third embodiment of the present disclosure proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the data processing method proposed in the first embodiment of the present disclosure.

[0011] The fourth embodiment of the present disclosure proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the data processing method proposed in the first embodiment of the present disclosure.

[0012] The fifth embodiment of the present disclosure proposes a computer program product. When the instruction processor in the computer program product executes, it performs the data processing method proposed in the first embodiment of the present disclosure.

[0013] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and / or additional aspects and advantages of the present disclosure will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a flowchart of a data processing method proposed in one embodiment of the present disclosure;

[0016] Figure 2 This is a schematic diagram of a simple storage of multiple data sources proposed in one embodiment of the present disclosure;

[0017] Figure 3 This is a schematic diagram of storing a data source through sharding proposed in one embodiment of the present disclosure;

[0018] Figure 4 is a flowchart of a data processing method proposed in another embodiment of the present disclosure;

[0019] Figure 5 is a flowchart of a data processing method proposed in another embodiment of the present disclosure;

[0020] Figure 6 is a structural diagram of a data processing device proposed in one embodiment of the present disclosure;

[0021] Figure 7 is a structural diagram of a data processing device proposed in another embodiment of the present disclosure;

[0022] Figure 8 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0023] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present disclosure and are not to be construed as limiting the present disclosure. On the contrary, the embodiments of the present disclosure include all variations, modifications, and equivalents that fall within the spirit and scope of the appended claims.

[0024] Figure 1 It is a flowchart of a data processing method proposed in one embodiment of the present disclosure.

[0025] It can be explained that the executor of the data processing method of this embodiment is a data processing device, which can be implemented by software and / or hardware. The device can be configured in a computer device, which can include but is not limited to a terminal, a server, etc.

[0026] like Figure 1 As shown, the data processing method includes:

[0027] S101: Determine reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage status of the data to be processed.

[0028] Among them, the data currently to be processed can be called the data to be processed. Correspondingly, the information used to describe the storage status of the data to be processed can be called reference storage information. The reference storage information can be specific, for example, the storage format of the data to be processed, the storage address of the data to be processed, the storage type of the data to be processed, etc., and there is no restriction on this.

[0029] The data processing method described in the embodiments of the present disclosure can be applied to scenarios where massive amounts of data are processed. Of course, the data processing method described in the embodiments of the present disclosure can also be applied to any other possible data processing scenarios without limitation.

[0030] In massive data processing scenarios, considering the data source and database working principles, different types of business data are usually split into different data sources (for example: relational databases can be used to store data involving complex data relationships, document databases can be used to store document-type data, and distributed file data sources can be used to store data that is read more and written less). When users analyze massive data, they can use the Online Analytical Processing (OLAP) method without any restrictions.

[0031] In some embodiments, massive amounts of data are usually stored in multiple data sources in the form of different instances of the same data source. Due to the working principle of relational / document data, single / document data (same type of data) can also be split into multiple data shards due to its large volume and stored in different data sources or the same data source. Figure 2 and Figure 3 As shown, Figure 2 This is a schematic diagram of a simple multi-data source storage method proposed in an embodiment of the present disclosure. Figure 3 This is a schematic diagram of storage of a data source through sharding proposed in one embodiment of the present disclosure.

[0032] When in the above-mentioned massive data processing scenario, the reference storage information can be used to describe the storage situation of the data to be processed, that is, the data source where the data to be processed is stored, or the stored data table, or the stored data table shards, without any restrictions.

[0033] When determining the reference storage information corresponding to the data to be processed, the embodiments of the present disclosure may determine the data type of the data to be processed, and determine the working principle or data size of the data to be processed, and then determine the storage location information of the data to be processed based on the data type of the data to be processed and the working principle or data size of the data to be processed, and use the location information as the reference storage information corresponding to the data to be processed, without any limitation.

[0034] S102: Determine a data association relationship corresponding to the data to be processed based on the reference storage information.

[0035] Among them, different data may have an association relationship, which can be called a data association relationship. The data association relationship can be specifically, for example, an association relationship of the semantic dimension of the data to be processed, an association relationship of the business dimension of the data to be processed, an association relationship of the application dimension of the data to be processed, etc., and there is no limitation on this.

[0036] In some embodiments, the data association relationship corresponding to the data to be processed is determined based on the reference storage information. This can be done by determining the data to be processed corresponding to the reference storage information from multiple data based on the reference storage information, and then parsing the data to be processed to obtain a business association relationship corresponding to the data to be processed, and using the business association relationship as the data association relationship. Alternatively, any other possible method, such as a model prediction method or an information matching method, can be used to determine the data association relationship corresponding to the data to be processed based on the reference information, and there is no limitation on this.

[0037] S103: Execute a target processing operation on target data associated with the data to be processed according to the data association relationship.

[0038] After determining the data association relationship corresponding to the data to be processed, the corresponding processing operation (for example, corresponding to the business scenario, or corresponding to the data processing scenario, without limitation) can be performed on the target data associated with the data to be processed. The corresponding processing operation can be called the target processing operation.

[0039] The target processing operation may specifically include data extraction, data recovery, etc., for example, and is not limited thereto.

[0040] In some embodiments, according to the data association relationship, the target processing operation is performed on the target data associated with the processed data. The corresponding target processing operation instruction can be generated according to the data association relationship, and then the data processing device can respond to the target operation instruction to perform the target processing operation on the target data associated with the processed data. There is no limitation on this.

[0041] In other embodiments, target processing operations are performed on target data associated with the data to be processed based on the data association relationship. Alternatively, target data associated with the data to be processed can be determined from multiple data based on the data association relationship, and then target processing operations can be performed on the target data. There is no limitation on this.

[0042] In this embodiment, by determining reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage situation of the data to be processed, and based on the reference storage information, determining the data association relationship corresponding to the data to be processed, and then performing target processing operations on the target data associated with the data to be processed based on the data association relationship, it is possible to efficiently determine the target data having a data association relationship with the data to be processed based on the storage situation of the data to be processed, and perform corresponding processing operations on the target data, which can effectively improve the efficiency of data processing, and effectively simplify the data processing logic, thereby effectively improving the data processing effect.

[0043] Figure 4 It is a flowchart of a data processing method proposed in another embodiment of the present disclosure.

[0044] like Figure 4 As shown, the data processing method includes:

[0045] S401: In response to a data processing request, a data identifier of the data to be processed is obtained by parsing the data processing request.

[0046] Among them, the request generated by the computer device that triggers data processing can be called a data processing request. The data processing request can be, for example, a data processing device that provides a data processing request interface in advance and receives a request for processing data based on the data processing request interface.

[0047] The above-mentioned data identifier can be used to identify the data to be processed, that is, the data to be processed corresponding to the data identifier can be determined according to the data identifier.

[0048] In the embodiment of the present disclosure, in the process of configuring the association relationship of sharded multiple data sources, the configuration of a business-table business table can be supported. Since the data volume of the business-table business table is large and there is a demand for online transaction processing (OLTP), the data of the business table can be split and configured in the datasource2 library and the datasource3 library respectively, and the primary key identification (ID) of the business table can exist in the datasource1 library.

[0049] Among them, the data identifier of the data to be processed can be the primary key ID of the business table existing in the above-mentioned datasource1 library. Accordingly, the data identifier of the data to be processed is obtained by parsing the data processing request. The data processing request can be parsed to obtain the business table corresponding to the data to be processed, and then the primary key ID of the business table is further determined, and the primary key ID is used as the data identifier of the data to be processed. There is no restriction on this.

[0050] S402: Determine reference storage information according to the data identifier.

[0051] After the data identifier of the data to be processed is obtained by parsing the data processing request, the reference storage information corresponding to the data to be processed can be determined based on the data identifier. Since the data processing request is parsed to obtain the data identifier of the data to be processed, and then the reference storage information is determined in combination with the data processing identifier, the efficiency of obtaining the reference storage information can be effectively improved, the timeliness of subsequent data processing can be guaranteed, and the accuracy and reliability of the reference storage information can be effectively improved, so that the data processing method can be effectively applicable to application scenarios with high real-time requirements for data processing.

[0052] In the embodiment of the present disclosure, after determining the primary key ID corresponding to the data to be processed, the primary key ID and the business-table can be associated to determine the location and range information of the business-table corresponding to the primary key ID, and the information can be used as reference storage information, without any restriction.

[0053] S403: According to the reference storage information, configure the business association relationship involved in the data to be processed.

[0054] Optionally, in some embodiments, configuring the business association relationships involved in the data to be processed based on the reference storage information can be, when the reference storage information is of a data source association type or a data table shard type, configuring a first relationship field for the data to be processed, and configuring the business association relationships of the database dimension, the business association relationships of the data table dimension, and the business association relationships of the field dimension for the first relationship field. Since the corresponding business association relationships are configured for the data to be processed when the reference storage information is of a data source association type or a data table shard type, the reference storage information is finely divided, and based on the data source association type or data table shard type obtained by the fine division, more detailed business association relationships are assisted in generating, so that the business association relationships can be effectively adapted to the storage conditions of the data to be processed, and the business association relationships can be used to characterize the association relationships between different data, so that when the target data is subsequently determined based on the business association relationships, the impact of the storage conditions of the target data can be effectively avoided, and the target data can be quickly and accurately located in the subsequent process. The overall data processing effect can be improved based on the business association relationships adapted to the storage conditions of the data to be processed.

[0055] Among them, the data source association type and data table sharding type refer to the description of the association relationship for different types of data sources. For example, in the process of configuring the business association relationship of simple multiple data sources, for simple data association between data sources, you can create a data source and formulate the data source link and type, configure the tables, libraries, and fields in the data source, and then configure the corresponding business association relationship for the fields (the association relationship can reflect complex information such as libraries, tables, and fields).

[0056] When the reference storage information is of a data data source association type or a data table shard type, the field configured for the data to be processed may be referred to as a first relationship field.

[0057] When the reference storage information is of a data data source association type or a data table shard type, the field configured for the target data may be referred to as a second relationship field.

[0058] The database to which the data to be processed belongs can be referred to as the first database, and the database to which the target data belongs can be referred to as the second database.

[0059] The business association relationship in the database dimension describes the business association relationship between the first database to which the data to be processed belongs and the second database to which the target data belongs.

[0060] The data table to which the data to be processed belongs can be referred to as a first data table, and the data table to which the target data belongs can be referred to as a second data table.

[0061] The business association relationship in the data table dimension describes the business association relationship between the first data table to which the to-be-processed data belongs and the second data table to which the target data belongs.

[0062] The business association relationship of the field dimension describes the business association relationship between the first relationship field corresponding to the data to be processed and the second relationship field corresponding to the target data.

[0063] The business association relationships in the above-mentioned different dimensions can be specifically configured according to the business scenario requirements of data processing, or can be adaptively adjusted, and there is no restriction on this.

[0064] The business association relationship in the database dimension may be, for example, an association relationship of a data extraction business between a first database and a second database, an association relationship of a data recovery business, etc., and is not limited thereto.

[0065] The business association relationship of the field dimension may be, for example, an association relationship of a data extraction business between a first data table and a second data table, an association relationship of a data recovery business, etc., and there is no limitation to this.

[0066] The business association relationship of the data table dimension may be, for example, an association relationship of a data extraction business between a first relationship field and a second relationship field, an association relationship of a data recovery business, etc., and there is no limitation to this.

[0067] In an embodiment of the present disclosure, the business association relationships involved in the data to be processed are configured based on the reference storage information. When the reference storage information is a data association type or a data table sharding type, the business association relationships of the database dimension, the business association relationships of the data table dimension, and the business association relationships of the field dimension are configured for the first relationship field corresponding to the data to be processed, and the aforementioned multiple association relationships are collectively used as the business association relationships involved in the data to be processed.

[0068] S404: Read the data to be processed according to the reference storage information.

[0069] When in the above-mentioned massive data processing scenario, the data to be processed is read according to the reference storage information. This can be done by determining the data corresponding to the reference storage information from the massive data and using it as the data to be processed. There is no limitation on this.

[0070] S405: Acquire the business association relationship involved in the data to be processed, and use the business association relationship as the data association relationship, wherein the business association relationship is used to describe the association relationship between the data to be processed and the target data based on the business dimension.

[0071] After reading the data to be processed according to the reference storage information, the business association relationship involved in the data to be processed can be obtained, and the business association relationship can be used as the data association relationship. Since the business association relationship of the data to be processed is used as the data association relationship, the target data determined according to the reference storage information of the data to be processed can have a higher business relevance to the data to be processed, ensuring that the determined target data meets the data processing requirements of the business scenario, improving the positioning accuracy of the target data, and can also effectively guarantee the execution of the data processing method during the subsequent execution of the data processing method.

[0072] S406: Execute a target processing operation on the target data associated with the data to be processed according to the data association relationship.

[0073] The description of S406 can be found in the above embodiment and will not be repeated here.

[0074] In this embodiment, by responding to a data processing request, parsing the data identifier of the data to be processed from the data processing request, and determining the reference storage information based on the data identifier, the efficiency of obtaining the reference storage information can be effectively improved, the timeliness of subsequent data processing can be guaranteed, and the accuracy and reliability of the reference storage information can be effectively improved, so that the data processing method can be effectively applicable to application scenarios with high real-time requirements for data processing. Then, according to the reference storage information, the business association relationship involved in the data to be processed is configured, and after reading the data to be processed according to the reference storage information, the business association relationship involved in the data to be processed can be obtained, and the business association relationship can be used as the data association relationship. Since the business association relationship of the data to be processed is used as the data association relationship, the target data determined according to the reference storage information of the data to be processed can have a higher business relevance to the data to be processed, ensuring that the determined target data meets the data processing requirements of the business scenario, improving the positioning accuracy of the target data, and effectively guaranteeing the execution of the data processing method during the subsequent execution of the data processing method.

[0075] Figure 5 It is a flowchart of a data processing method proposed in another embodiment of the present disclosure.

[0076] like Figure 5 As shown, the data processing method includes:

[0077] S501: Determine reference storage information corresponding to the data to be processed, where the reference storage information is used to describe the storage status of the data to be processed.

[0078] The description of S501 can be found in the above embodiment and will not be repeated here.

[0079] S502: According to the reference storage information, configure the business association relationship involved in the data to be processed.

[0080] That is to say, after determining the reference storage information corresponding to the data to be processed, the business association relationship involved in the data to be processed can be configured according to the reference storage information. Since the business association relationship involved in the data to be processed is configured with reference to the reference storage information, the storage situation of the data to be processed can, to a certain extent, match the business association relationship between the data to be processed and the target data, so that the configuration of the business association relationship can adapt to personalized storage needs, thereby improving the flexibility of the data processing method.

[0081] The reference storage information may be any one of the following or a combination of the following: data source association type, database sharding type, data table sharding type, and hot and cold separation type.

[0082] Optionally, in some embodiments, the business association relationship involved in the data to be processed is configured according to the reference storage information. When the reference storage information is a database sharding type, the database sharding method is determined, and the database instance address corresponding to the database sharding method is determined. The database sharding method and the database instance address are configured as the business association relationship involved in the data to be processed, thereby achieving a detailed division of the reference storage information, and based on the database sharding type obtained by the detailed division, assisting in generating more detailed business association relationships, so that the business association relationship can be effectively adapted to the storage situation of the data to be processed, and the business association relationship can be used to characterize the association relationship between different data, so that when the target data is subsequently determined based on the business association relationship, it can effectively avoid being affected by the storage situation of the target data, assisting in the subsequent rapid and accurate positioning of the target data, and assisting in improving the overall data processing effect based on the business association relationship adapted to the storage situation of the data to be processed.

[0083] The database may be sharded in a manner such as by year, month, and day, or by the hash digit of a number, and there is no limitation on this.

[0084] That is to say, in the embodiment of the present disclosure, the type of reference storage information can be determined. When the reference storage information is of a database sharding type, the database sharding method is determined, and the database instance address corresponding to the database sharding method is determined. The database sharding method and the database instance address are then configured as the business association relationship involved in the data to be processed.

[0085] Optionally, in other embodiments, the business association relationship involved in the data to be processed is configured according to the reference storage information. It can also be that when the reference storage information is a hot-cold separation type, multiple sharding intervals are determined, and the data to be processed is divided with reference to the multiple sharding intervals respectively to obtain multiple sub-data to be processed corresponding to the multiple sharding intervals, and the multiple sub-data to be processed together constitute the data to be processed, and then corresponding multiple business association relationships are configured for the multiple sub-data to be processed respectively, thereby achieving a detailed division of the reference storage information, and based on the hot-cold separation type obtained by the detailed division, assisting in generating more detailed business association relationships, so that the business association relationships can be effectively adapted to the storage situation of the data to be processed, and the business association relationships can be used to characterize the association relationships between different data, so that when the target data is subsequently determined based on the business association relationships, it can effectively avoid being affected by the storage situation of the target data, assisting in quickly and accurately locating the target data, and assisting in improving the overall data processing effect based on the business association relationships adapted to the storage situation of the data to be processed.

[0086] Among them, the hot and cold separation type is a type that divides data according to the hot and cold separation method, that is, the data is divided into multiple data tables according to year, month, day, active data and inactive data. Multiple data tables can respectively form multiple databases (multiple databases can also be called multiple shard intervals).

[0087] Among them, multiple data can be stored in multiple shard intervals, and the data can be called sub-data to be processed. Multiple sub-data to be processed together constitute the data to be processed.

[0088] Among them, in multiple shard intervals, the target data associated with the data to be processed can be called target sub-data.

[0089] Among them, the database to which the sub-data to be processed belongs can be called the third database, the data table to which the sub-data to be processed belongs can be called the third data table, the database to which the target sub-data belongs can be called the fourth database, and the data table to which the target data belongs can be called the fourth data table.

[0090] Among them, the business association relationship describes the business association relationship between the third database and / or third data table to which the corresponding sub-data to be processed belongs, and the fourth database and / or fourth data table to which the target sub-data belongs. The target sub-data is obtained by dividing the target data with reference to the sharding interval.

[0091] That is to say, in the embodiment of the present disclosure, after the data is divided into multiple data tables in a hot and cold separation manner, multiple sharding intervals consisting of the multiple data tables can be determined, and multiple sub-data to be processed corresponding to the sharding intervals can be determined. Then, corresponding multiple business association relationships can be configured for the multiple sub-data to be processed respectively, and the business association relationship can be used as the business association relationship involved in the data to be processed, without any restriction.

[0092] S503: Determine the data association relationship corresponding to the data to be processed based on the reference storage information.

[0093] The description of S503 can be found in the above embodiment and will not be repeated here.

[0094] S504: Parse the data association relationship to obtain target storage information, wherein the target storage information is used to describe the storage status of the target data.

[0095] The information used to describe the storage status of the target data may be referred to as target storage information. The target storage information may specifically include, for example, the storage location of the target data and the storage type of the target data, etc., which is not limited.

[0096] After determining the data association relationship corresponding to the data to be processed based on the reference storage information, the data association relationship can be parsed to obtain the target storage information.

[0097] S505: Execute the target processing operation on the data to be processed, and simultaneously execute the target processing operation on the target data according to the target storage information.

[0098] After parsing the data association relationship to obtain the target storage information, the target processing operation can be performed on the data to be processed, and the target processing operation can be performed on the target data according to the target storage information, thereby effectively improving the data processing efficiency. The target storage information is determined based on the data association relationship, and the target storage information can be used to quickly locate the corresponding target data, so that the data processing method can be effectively adapted to the personalized data processing needs of the actual business scenario, thereby improving the robustness and applicability of the data processing method.

[0099] In the embodiment of the present disclosure, the target storage information can be used to locate the target data, that is, the data source where the target data is stored, or the stored data table, or the stored data table shard can be determined based on the target storage information, without any limitation.

[0100] That is to say, in the embodiments of the present disclosure, while performing target processing operations on the data to be processed, the data source where the target data is stored, or the stored data table, or the stored data table shard can be determined based on the target storage information, and then the target data can be read from the data source, data table or data table shard, and the target processing operation can be performed on the target data.

[0101] In this embodiment, by determining reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage situation of the data to be processed, and according to the reference storage information, configuring the business association relationship involved in the data to be processed, and then determining the data association relationship corresponding to the data to be processed according to the reference storage information, and parsing the data association relationship to obtain the target storage information, the target processing operation can be performed on the data to be processed, and the target processing operation can be performed on the target data according to the target storage information, thereby effectively improving the data processing efficiency, and the target storage information is determined based on the data association relationship, and the target storage information can be used to quickly locate the corresponding target data, so that the data processing method can be effectively adapted to the personalized data processing needs of the actual business scenario, thereby improving the robustness and applicability of the data processing method.

[0102] Figure 6 It is a structural diagram of a data processing device proposed in one embodiment of the present disclosure.

[0103] like Figure 6 As shown, the data processing device 60 includes:

[0104] A first determining module 601 is configured to determine reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage status of the data to be processed;

[0105] A second determining module 602 is configured to determine a data association relationship corresponding to the data to be processed based on the reference stored information;

[0106] The execution module 603 is configured to execute a target processing operation on target data associated with the data to be processed according to the data association relationship.

[0107] In some embodiments of the present disclosure, Figure 7 As shown, Figure 7 FIG6 is a structural diagram of a data processing device proposed in another embodiment of the present disclosure, wherein the execution module 603 is specifically configured to:

[0108] Parsing the data association relationship to obtain target storage information, wherein the target storage information is used to describe the storage status of the target data;

[0109] Target processing operations are performed on the data to be processed, and target processing operations are performed on the target data according to target storage information.

[0110] In some embodiments of the present disclosure, the first determining module 601 is specifically configured to:

[0111] In response to the data processing request, parsing the data processing request to obtain a data identifier of the data to be processed;

[0112] Determine the reference storage information based on the data identifier.

[0113] In some embodiments of the present disclosure, the second determining module 602 includes:

[0114] The reading submodule 6021 is used to read the data to be processed according to the reference storage information;

[0115] The acquisition submodule 6022 is used to acquire the business association relationship involved in the data to be processed and use the business association relationship as the data association relationship, wherein the business association relationship is used to describe the association relationship between the data to be processed and the target data based on the business dimension.

[0116] In some embodiments of the present disclosure, the reference stored information is any one or a combination of the following:

[0117] Data source association type, database sharding type, data table sharding type, and hot and cold separation type.

[0118] In some embodiments of the present disclosure, the second determining module 602 further includes:

[0119] The configuration submodule 6023 is used to configure the business association relationship involved in the data to be processed according to the reference storage information before reading the data to be processed according to the reference storage information.

[0120] In some embodiments of the present disclosure, the configuration submodule 6023 is specifically configured to:

[0121] If the reference storage information is of a data source association type or a data table shard type, configuring a first relationship field for the data to be processed;

[0122] Configuring a business association relationship of a database dimension, a business association relationship of a data table dimension, and a business association relationship of a field dimension for the first relationship field;

[0123] The business association relationship in the database dimension describes the business association relationship between the first database to which the data to be processed belongs and the second database to which the target data belongs;

[0124] The business association relationship of the data table dimension describes the business association relationship between the first data table to which the data to be processed belongs and the second data table to which the target data belongs;

[0125] The business association relationship of the field dimension describes the business association relationship between the first relationship field corresponding to the data to be processed and the second relationship field corresponding to the target data.

[0126] In some embodiments of the present disclosure, the configuration submodule 6023 is specifically configured to:

[0127] If the reference storage information is a database sharding type, determine the database sharding method;

[0128] Determine the database instance address corresponding to the database sharding method;

[0129] Configure the database sharding method and database instance address based on the business relationship involved in the data to be processed.

[0130] In some embodiments of the present disclosure, the configuration submodule 6023 is specifically configured to:

[0131] If the reference storage information is of hot and cold separation type, multiple shard intervals are determined;

[0132] Dividing the data to be processed by referring to the multiple sharding intervals respectively to obtain multiple sub-data to be processed corresponding to the multiple sharding intervals respectively, and the multiple sub-data to be processed together constitute the data to be processed;

[0133] Configuring multiple corresponding business association relationships for multiple sub-data to be processed;

[0134] Among them, the business association relationship describes the business association relationship between the third database and / or third data table to which the corresponding sub-data to be processed belongs, and the fourth database and / or fourth data table to which the target sub-data belongs. The target sub-data is obtained by dividing the target data with reference to the sharding interval.

[0135] With the above Figures 1 to 5 Corresponding to the data processing method provided in the embodiment, the present disclosure also provides a data processing device. Since the data processing device provided in the embodiment of the present disclosure is consistent with the above Figures 1 to 5 The data processing method provided in the embodiment corresponds to the data processing method, so the implementation of the data processing method is also applicable to the data processing device proposed in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0136] In this embodiment, by determining reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage situation of the data to be processed, and based on the reference storage information, determining the data association relationship corresponding to the data to be processed, and then performing target processing operations on the target data associated with the data to be processed based on the data association relationship, it is possible to efficiently determine the target data having a data association relationship with the data to be processed based on the storage situation of the data to be processed, and perform corresponding processing operations on the target data, which can effectively improve the efficiency of data processing, and effectively simplify the data processing logic, thereby effectively improving the data processing effect.

[0137] In order to implement the above embodiments, the present disclosure also proposes a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the data processing method proposed in the above embodiments of the present disclosure is implemented.

[0138] In order to implement the above embodiments, the present disclosure further proposes a non-transitory computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data processing method proposed in the above embodiments of the present disclosure is implemented.

[0139] In order to implement the above embodiments, the present disclosure further proposes a computer program product. When an instruction processor in the computer program product executes, the data processing method proposed in the above embodiments of the present disclosure is executed.

[0140] Figure 8 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Figure 8 The computer device 12 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0141] like Figure 8As shown, computer device 12 is implemented as a general-purpose computing device. Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0142] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnection (PCI) bus.

[0143] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0144] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 8 Not shown, often called a "hard drive").

[0145] although Figure 8Although not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a Compact Disc Read Only Memory (hereinafter referred to as: CD-ROM), a Digital Video Disc Read Only Memory (hereinafter referred to as: DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present disclosure.

[0146] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0147] The computer device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). This communication can occur via an input / output (I / O) interface 22. Furthermore, the computer device 12 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0148] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the data processing method mentioned in the above embodiment.

[0149] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0150] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0151] It should be noted that, in the description of this disclosure, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of this disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0152] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0153] It should be understood that various parts of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0154] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0155] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0156] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0157] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present disclosure. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0158] Although the embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are illustrative and are not to be construed as limitations on the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: Determining reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe the storage status of the data to be processed; Determining, based on the reference storage information, a data association relationship corresponding to the data to be processed; performing a target processing operation on target data associated with the data to be processed according to the data association relationship; Before reading the data to be processed according to the reference storage information, the method further includes: If the reference storage information is of a database sharding type, determining a database sharding method; Determine the database instance address corresponding to the database sharding method; Configuring the database sharding method and the database instance address as the business association relationship involved in the data to be processed; Wherein, determining the data association relationship corresponding to the data to be processed based on the reference storage information includes: Reading the data to be processed according to the reference storage information; The business association relationship involved in the data to be processed is obtained, and the business association relationship is used as the data association relationship, wherein the business association relationship is used to describe the association relationship between the data to be processed and the target data based on the business dimension.

2. The method according to claim 1, wherein The performing a target processing operation on the target data associated with the data to be processed according to the data association relationship includes: parsing the data association relationship to obtain target storage information, wherein the target storage information is used to describe the storage status of the target data; The target processing operation is performed on the data to be processed, and the target processing operation is performed on the target data according to the target storage information.

3. The method according to claim 1, wherein The determining of the reference storage information corresponding to the data to be processed includes: In response to a data processing request, parsing the data processing request to obtain a data identifier of the data to be processed; The reference storage information is determined according to the data identifier.

4. The method according to claim 1, wherein Before reading the data to be processed according to the reference storage information, the method further includes: If the reference storage information is of a data source association type or a data table sharding type, configuring a first relationship field for the data to be processed; Configuring the first relationship field with a business association relationship in the database dimension, a business association relationship in the data table dimension, and a business association relationship in the field dimension; The business association relationship in the database dimension describes the business association relationship between the first database to which the data to be processed belongs and the second database to which the target data belongs; The business association relationship of the data table dimension describes the business association relationship between the first data table to which the to-be-processed data belongs and the second data table to which the target data belongs; The business association relationship of the field dimension describes the business association relationship between the first relationship field corresponding to the data to be processed and the second relationship field corresponding to the target data.

5. The method according to claim 1, wherein Before reading the data to be processed according to the reference storage information, the method further includes: If the reference storage information is of a cold and hot separation type, determining a plurality of sharding intervals; Dividing the data to be processed by referring to the multiple sharding intervals respectively to obtain multiple sub-data to be processed corresponding to the multiple sharding intervals respectively, wherein the multiple sub-data to be processed together constitute the data to be processed; configuring corresponding multiple business association relationships for the multiple sub-data to be processed respectively; Among them, the business association relationship describes the business association relationship between the third database and / or third data table to which the corresponding sub-data to be processed belongs, and the fourth database and / or fourth data table to which the target sub-data belongs. The target sub-data is obtained by dividing the target data with reference to the sharding interval.

6. A data processing device, characterized in that: The device comprises: A first determining module is configured to determine reference storage information corresponding to the data to be processed, wherein the reference storage information is used to describe a storage condition of the data to be processed; A second determining module is configured to determine a data association relationship corresponding to the data to be processed based on the reference storage information; an execution module, configured to execute a target processing operation on target data associated with the data to be processed according to the data association relationship; The device further includes a configuration submodule: Before reading the data to be processed according to the reference storage information, the configuration submodule is used to determine the database sharding mode if the reference storage information is a database sharding type; determine the database instance address corresponding to the database sharding mode; and configure the database sharding mode and the database instance address as a business association relationship involved in the data to be processed; The second determining module is specifically configured to: Reading the data to be processed according to the reference storage information; The business association relationship involved in the data to be processed is obtained, and the business association relationship is used as the data association relationship, wherein the business association relationship is used to describe the association relationship between the data to be processed and the target data based on the business dimension.

7. The device according to claim 6, characterized in that The execution module is specifically used to: parsing the data association relationship to obtain target storage information, wherein the target storage information is used to describe the storage status of the target data; The target processing operation is performed on the data to be processed, and the target processing operation is performed on the target data according to the target storage information.

8. The device according to claim 6, wherein The first determining module is specifically configured to: In response to a data processing request, parsing the data processing request to obtain a data identifier of the data to be processed; The reference storage information is determined according to the data identifier.

9. The device according to claim 6, wherein The configuration submodule is specifically used to: If the reference storage information is of a data source association type or a data table sharding type, configuring a first relationship field for the data to be processed; Configuring the first relationship field with a business association relationship in the database dimension, a business association relationship in the data table dimension, and a business association relationship in the field dimension; The business association relationship in the database dimension describes the business association relationship between the first database to which the data to be processed belongs and the second database to which the target data belongs; The business association relationship of the data table dimension describes the business association relationship between the first data table to which the to-be-processed data belongs and the second data table to which the target data belongs; The business association relationship of the field dimension describes the business association relationship between the first relationship field corresponding to the data to be processed and the second relationship field corresponding to the target data.

10. The device according to claim 6, wherein The configuration submodule is specifically used to: If the reference storage information is of a cold and hot separation type, determining a plurality of sharding intervals; Dividing the data to be processed by referring to the multiple sharding intervals respectively to obtain multiple sub-data to be processed corresponding to the multiple sharding intervals respectively, wherein the multiple sub-data to be processed together constitute the data to be processed; configuring corresponding multiple business association relationships for the multiple sub-data to be processed respectively; Among them, the business association relationship describes the business association relationship between the third database and / or third data table to which the corresponding sub-data to be processed belongs, and the fourth database and / or fourth data table to which the target sub-data belongs. The target sub-data is obtained by dividing the target data with reference to the sharding interval.

11. A computer device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

Images