Method, apparatus, electronic device, and medium for data processing

Abstract

The present disclosure relates to a data processing method, device, electronic equipment and medium, which can be applied to the technical field of Internet and cloud service, and the data processing method comprises the following steps: obtaining a user's to-be-processed data structure and a corresponding user identifier; describing the to-be-processed data structure and the corresponding user identifier in the form of metadata to obtain a data structure logical table associated with the user identifier; configuring a mapping relationship between the data structure logical table and a pre-constructed physical table; and based on the mapping relationship, storing the to-be-processed data structure in the pre-constructed physical table to obtain a target physical table region associated with the user identifier.

Description

Technical Field

[0001] This disclosure relates to the fields of Internet technology and cloud service technology, and in particular to a data processing method, apparatus, electronic device and medium. Background Technology

[0002] With the development of internet technology, various organizations generate massive amounts of business data and internal operational data during their operations. Different organizations are introducing cloud services to store and manage these large amounts of data.

[0003] In realizing this disclosed concept, the inventors discovered at least the following technical problems in the related technology: The attribute information describing the same object differs between different institutions. For example, when describing an asset, Institution A might describe the asset's attribute information as: {User ID, Account, Deposit Amount}; Institution B might describe the asset's attribute information as: {User Name, Vehicle Ownership, Annual Income, Real Estate Status}. Therefore, when cloud service developers construct database models, to enable the cloud service to provide corresponding database services for different institutions, they must consider the object description methods and types of each institution. This requires developers to build different database models from scratch for different institutions to accommodate various new types of storage objects. This results in a huge workload and is detrimental to database operations such as adding, deleting, querying, and modifying data. Furthermore, using relatively independent code snippets for each institution leads to low database operation efficiency and code redundancy, resulting in high development and maintenance costs and increasingly slow technological iteration. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, embodiments of this disclosure provide a data processing method, apparatus, electronic device and medium.

[0005] In a first aspect, embodiments of this disclosure provide a data processing method. The method includes: obtaining a user's data structure to be processed and its corresponding user identifier; describing the data structure to be processed and its corresponding user identifier in metadata form to obtain a logical table of data structures associated with the user identifier; configuring a mapping relationship between the logical table of data structures and a pre-built physical table; and storing the data structure to be processed in the pre-built physical table based on the mapping relationship to obtain a target physical table region associated with the user identifier.

[0006] According to embodiments of this disclosure, the aforementioned data structure to be processed includes: a set of attribute information for describing the same object; the set of attribute information for describing the same object differs for different users. Configuring the mapping relationship between the aforementioned data structure logical table and the pre-built physical table includes: for a set of attribute information in the data structure logical table associated with the same user identifier, based on metadata descriptions, determining matching data columns from the pre-built physical table that match the data type of each attribute information; determining candidate data rows in the matching data columns that are not currently occupied; for matching data columns of all attribute information associated with the same user identifier, determining common candidate data rows that are not currently occupied; and generating a mapping relationship between the data structure logical table and the pre-built physical table based on the correspondence between the aforementioned set of attribute information, the common candidate data rows, and the matching data columns.

[0007] According to an embodiment of this disclosure, the above-mentioned storage of the data structure to be processed in the pre-constructed physical table based on the above-mentioned mapping relationship to obtain a target physical table region associated with a user identifier includes: based on the above-mentioned mapping relationship, storing a set of attribute information from the logical table of the data structure associated with the same user identifier into the corresponding common candidate data row and matching data column in the above-mentioned physical table, wherein the common candidate data row and the matching data column constitute the target physical table region, and the common candidate data row is associated with the user identifier.

[0008] According to embodiments of this disclosure, the method further includes: receiving an operation instruction from a user on the data structure to be processed; describing the operation instruction and the corresponding user identifier in the form of metadata to obtain an operation logic table associated with the user identifier; parsing the operation logic table into a target operation for the target physical table region associated with the user identifier based on the mapping relationship; and executing the corresponding target operation in the target physical table region.

[0009] According to embodiments of this disclosure, the above-mentioned operation instructions include at least one of the following: instructions to add a data structure to be processed, instructions to delete a data structure to be processed, instructions to modify a data structure to be processed, instructions to query a data structure to be processed, and instructions to display a data structure to be processed.

[0010] According to embodiments of this disclosure, the method further includes: constructing physical tables. The construction of physical tables includes: obtaining information about the data types of an existing data structure; generating a set of table parameters covering the data types of the existing data structure based on the data type information; and constructing one or more physical tables based on the set of table parameters.

[0011] According to embodiments of this disclosure, constructing one or more physical tables based on the aforementioned table parameter groups includes: statistically analyzing the frequency of occurrence of data types in the aforementioned table parameter groups based on the combination relationships of data types in existing data structures; dividing the aforementioned table parameter groups according to the frequency of occurrence of the aforementioned data types to obtain one or more sets of table parameter groups; and using the aforementioned one or more sets of table parameter groups as data type parameters for the data columns of the physical tables to construct one or more physical tables.

[0012] According to embodiments of this disclosure, dividing the table parameter group based on the frequency of occurrence of the data types includes: reorganizing data types located in the same range based on the frequency range of occurrence of the data types to obtain one or more sets of table parameter groups.

[0013] Secondly, embodiments of this disclosure provide a data processing apparatus. The apparatus includes: a data acquisition module, a logical table construction module, a mapping relationship configuration module, and a storage module. The data acquisition module acquires a user's data structure to be processed and its corresponding user identifier. The logical table construction module describes the data structure to be processed and its corresponding user identifier in metadata form, obtaining a logical table of data structures associated with the user identifier. The mapping relationship configuration module configures the mapping relationship between the logical table of data structures and a pre-built physical table. The storage module stores the data structure to be processed in the pre-built physical table based on the mapping relationship, obtaining a target physical table region associated with the user identifier.

[0014] Thirdly, embodiments of this disclosure provide an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus; the memory stores computer programs; and the processor, when executing the program stored in the memory, implements the data processing method described above.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the data processing method described above.

[0016] Compared with the prior art, the technical solutions provided in this disclosure have at least some or all of the following advantages:

[0017] By describing the data structure to be processed and its corresponding user identifier in the form of metadata, a logical table of data structures associated with user identifiers is obtained. By configuring the mapping relationship between the logical table of data structures and the pre-built physical table, the data structure to be processed described by the logical table can be stored in the target physical table area of ​​the pre-built physical table. Each target physical table area is associated with a user identifier, thus enabling the storage of different types of data structures to be processed in a single physical table. This processing method is universal for various types of data structures to be processed and facilitates subsequent CRUD operations. It overcomes the technical problems of related technologies that require building different database models from scratch for different organizations to access various new types of storage objects, resulting in low database operation efficiency, code redundancy, high development and maintenance costs, and increasingly slow technology iteration. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0020] Figure 1A The system architecture of the data processing method and apparatus applicable to embodiments of this disclosure is illustrated schematically;

[0021] Figure 1B The diagram illustrates the architecture of a metadata model for a data processing method applicable to embodiments of this disclosure;

[0022] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 3 A detailed implementation flowchart of operation S203 according to an embodiment of the present disclosure is illustrated schematically;

[0024] Figure 4 A flowchart illustrating a data processing method according to another embodiment of this disclosure is shown schematically;

[0025] Figure 5 A flowchart illustrating a data processing method according to yet another embodiment of the present disclosure is shown schematically;

[0026] Figure 6A detailed implementation flowchart of operation S501 according to an embodiment of the present disclosure is illustrated schematically;

[0027] Figure 7 A schematic block diagram of a data processing apparatus according to embodiments of the present disclosure is shown; and

[0028] Figure 8 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0029] In related technologies, the attribute information used to describe the same object varies between different institutions. For example, taking assets as an example, different institutions describe the attribute information of assets differently; different merchants, financial institutions, schools, etc., describe the attribute information of their own assets in vastly different ways.

[0030] The aforementioned attribute information describes an object and may include attribute parameters, attribute values, and data types. Different institutions may use different attribute information to describe assets. This difference may be reflected in at least one aspect, such as the dimension of the attribute parameters (e.g., the number of attribute parameters), the name of the attribute parameters (different ways of describing the same attribute parameter), and the data type of the attribute parameters.

[0031] For example, when describing an asset, Institution A's ABS securitization system uses the following attribute parameters: {User ID, Account, Deposit Amount}. The data types for User ID, Account, and Deposit Amount are respectively: numeric, numeric, and numeric, accurate to four decimal places. Institution B's ABS securitization system uses the following attribute parameters: {User Name, Vehicle Ownership, Annual Income, Real Estate}. The data types for User Name, Vehicle Ownership, Annual Income, and Real Estate are respectively: text, text, numeric, accurate to two decimal places, and text. User-generated data in each institution's ABS securitization system can be used as attribute values. For example, the asset data of user R11 in institution A is: {12304 (user R11's ID value), 6012...2507 (user R11's account), 100,000 yuan (user R11's deposit amount)}, and the asset data of user R12 in institution A is: {22369 (user R12's ID value), 6012...3066 (user R12's account), 200,000 yuan (user R12's deposit amount)}. The asset data of user R21 in Institution B is as follows: {Zhang San (user R21's name), owns 1 car (user R21's vehicle ownership), 500,000 yuan (user R21's annual income), and 1 property (user R21's property ownership)}. The asset data of user R22 in Institution B is as follows: {Li Si (user R22's name), owns 2 cars (user R22's vehicle ownership), 800,000 yuan (user R22's annual income), and 1 property (user R22's property ownership)}.

[0032] When developers building database models for cloud services, they must consider the different object description methods and types of each institution or individual in order to enable the cloud service to provide corresponding database services to different organizations or individuals. For example, when providing cloud services for an ABS (Asset-Backed Securitization) system, the differences in descriptions of the same object between different institutions must be taken into account. This requires developers to build different database models from scratch for different institutions to accommodate various new types of storage objects. This results in a huge workload and is not conducive to database operations such as adding, deleting, querying, and modifying data. At the same time, using relatively independent code snippets for each institution leads to low database operation efficiency and code redundancy, resulting in high development and maintenance costs and increasingly slow technology iteration.

[0033] Taking the cloud service (hereinafter referred to as ABS cloud service) corresponding to the ABS asset securitization system (an application on a terminal device) as an example, in related technologies, when accessing the database model corresponding to the ABS asset securitization system, the following operations need to be repeated each time a new asset is added.

[0034] First, we investigate new asset types, and then conduct database modeling based on these new asset types, designing the database asset table structure.

[0035] Then, the developers wrote Java classes for the relevant Java objects (POJOs, Plain Ordinary Java Objects) based on the database model.

[0036] Next, the R&D personnel follow up on the business process and receive asset data in JSON format in the service interface. At the same time, they convert it into a Java Bean (a reusable class in Java) and perform basic operations on the Bean according to the relevant business process.

[0037] Finally, the processed Bean object data is persisted to the new asset table in the database.

[0038] The process of accessing and storing new assets needs to be restarted each time. In addition, for the processing operations such as adding, deleting, querying, modifying and displaying assets, a set of corresponding logic needs to be written for the new asset type.

[0039] In the above-described approach, each time a new asset is added to the ABS asset securitization system, all the aforementioned processes need to be completed from scratch, resulting in repetitive workload in later development and wasting human resources. With repeated additions of new assets, each asset type is managed using relatively independent code snippets, which is not conducive to efficient changes. Multiple places need to be modified simultaneously during business changes, leading to extremely low code development efficiency in later stages. During the later code handover and management process, the redundancy and complexity of the code also lead to increasingly higher system maintenance costs, slower business support, higher labor costs, slower technology iteration, and repeated bug fixes can cause breaks in code readability and continuity.

[0040] In view of the above, embodiments of this disclosure provide a data processing method, apparatus, electronic device, and medium. The data processing method includes: obtaining a user's data structure to be processed and the corresponding user identifier; describing the data structure to be processed and the corresponding user identifier in the form of metadata to obtain a logical table of data structures associated with user identifiers; configuring a mapping relationship between the logical table of data structures and a pre-built physical table; and storing the data structure to be processed in the pre-built physical table based on the mapping relationship to obtain a target physical table region associated with user identifiers.

[0041] In the embodiments of this disclosure, the data processing methods and apparatus described above may include, but are not limited to, operations such as data storage, addition, deletion, modification, querying, and display.

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0043] Figure 1A The schematic illustration shows the system architecture of the data processing method and apparatus applicable to embodiments of this disclosure.

[0044] Reference Figure 1A As shown, the system architecture 100 of the data processing method and apparatus applicable to embodiments of this disclosure includes: terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0045] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as ABS asset securitization system applications, and may also include other client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0046] Terminal devices 101, 102, and 103 can be various electronic devices with displays that support web browsing, such as electronic devices including but not limited to smartphones, tablets, laptops, desktop computers, smartwatches, etc.

[0047] Server 105 can be a server that provides various services, such as a backend management server that provides support for data processing of application interfaces or websites browsed by users using terminal devices 101, 102, and 103 (this is just an example). The backend management server can analyze and process the received data processing requests and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0048] It should be noted that the data processing method provided in this embodiment can generally be executed by server 105 or a terminal device with a certain computing power. Server 105 can be a networked terminal device providing cloud services. Correspondingly, the data processing apparatus provided in this embodiment can generally be located in server 105 or the aforementioned terminal device with a certain computing power. The data processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the data processing apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0049] It should be understood that Figure 1A The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0050] Figure 1B The schematic illustration shows the architecture of a metadata model for a data processing method applicable to embodiments of this disclosure.

[0051] Reference Figure 1B As shown, the metadata model 110 of the data processing method applicable to this embodiment of the disclosure is mainly divided into three layers: the first layer is the physical model layer 111, which is the actual physical table structure and model of the database underlying layer; the second layer is the mapping relationship layer 112, which represents the correspondence between physical tables and logical tables, and is a one-to-one mapping relationship (1:1); the third layer is the logical model layer 113a and the logical metadata (Logic_Meta) layer 113b, which mainly describes the POJO objects of the business model design and can be used for various business operations. The dashed arrows indicate that the mapping relationship between each layer is a one-to-one (1:1) mapping or a one-to-many (1:N) mapping. For example, a user can be a tenant of the ABS cloud service, and the tenant has N types of newly accessed assets, N≥2, corresponding to N physical model layers, which is equal to the total number of attribute information in a set of attribute information in the subsequent operation S201.

[0052] Reference Figure 1B As shown, the metadata model 110 may also include a logical model relation (Logic_Relation) layer 114, which is used to represent the relationship and reference between logical models.

[0053] Reference Figure 1B As shown, the aforementioned metadata model 110 may further include a logic layer 115 corresponding to the user's operation instructions, used to implement operations such as adding, deleting, modifying, and querying newly accessed assets.

[0054] Based on Figure 1B The example metadata model 110 implements the data processing method of this disclosure. The system framework maintains and defines a logical model layer and a logical metadata layer (corresponding to implementation operations S201-S202) to support the expansion of traditional business and POJO objects in traditional code. Simultaneously, it is configured through the mapping relationship between physical tables and logical tables to locate the physical layer where the underlying data of the logical layer is actually stored (corresponding to operation S203). For example, the logical layer's user information UserInfo includes user age, user code user_id, and user address user_addr. Based on the above description, UserInfo can be defined as the logical layer, and age, user_id, and user_addr can be defined as metadata of the logical layer. This metadata can also be expressed as attribute information of Java code Beans; thus, the storage of newly accessed assets in physical tables can be realized (corresponding to operation S204).

[0055] The underlying physical database contains a physical table TableA, which contains any number of columns such as A1, A2, A3, A4, and A5. After defining the physical table model, the mapping relationship between the logical table UserInfo and TableA can be configured, mapping the age field to column A1, the user_id field to column A2, and the user_addr field to column A4. Through the model's mapping relationship, the read and write operations of the business method on UserInfo can be mapped and parsed to the operations on columns A1, A2, and A4 in the actual underlying physical table TableA (corresponding to subsequent operations S401 to S404).

[0056] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0057] The first exemplary embodiment of this disclosure provides a method for data processing.

[0058] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0059] Reference Figure 2 As shown, the data processing method provided in this embodiment includes the following operations: S201, S202, S203, and S204. Operations S201 to S204 can be performed by a networked terminal device or server providing cloud computing services.

[0060] In operation S201, the user's pending data structure and corresponding user identifier are obtained.

[0061] In operation S202, the above-mentioned data structure to be processed and the corresponding user identifier are described in the form of metadata, resulting in a logical table of data structure associated with the user identifier.

[0062] In operation S203, configure the mapping relationship between the above data structure logical table and the pre-built physical table.

[0063] In operation S204, based on the above mapping relationship, the above data structure to be processed is stored in the above pre-built physical table to obtain the target physical table region associated with the user identifier.

[0064] In the above operation S201, the method of acquisition can be to obtain the data structure to be processed from the dataset provided by the user, or to receive the data structure to be processed provided by the user.

[0065] Users can be various organizations or individuals with data processing needs, and the data structures to be processed differ for different users. According to embodiments of this disclosure, the aforementioned data structure to be processed can be a data structure from various users that needs to process the same object. This data structure can be a binary or multi-dimensional data group composed of multiple data points. The aforementioned object can be of various forms, such as assets, credit, income, etc. The data structure involved in this object can be described by a set of attribute information, and different users will have different attribute information describing the same object. This difference can be reflected in differences in attribute dimensions, attribute parameters, attribute values, and attribute value types, etc.

[0066] For example, if institution A provides a set of attribute information describing an asset as: attribute information A1, attribute information A2, attribute information A3, and attribute information A4, then the obtained data structure to be processed for institution A and the corresponding user identifier can be represented as: {User Identifier A: A1, A2, A3, A4}. Similarly, if institution B provides a set of attribute information describing an asset as: attribute information B1, attribute information B2, attribute information B3, attribute information B4, attribute information B5, and attribute information B6, then the obtained data structure to be processed for institution B and the corresponding user identifier can be represented as: {User Identifier B: B1, B2, B3, B4, B5, B6}.

[0067] In the above operation S202, metadata is data used to describe the data and its environment. It is mainly used to describe data property information to support functions such as indicating storage location, historical data, resource lookup, and file records. By describing the above-mentioned data structure to be processed and the corresponding user identifier in the form of metadata, a logical table of data structures associated with user identifiers can be obtained.

[0068] For example, the data structure to be processed of organization A and the corresponding user identifiers: {User Identifiers A: A1, A2, A3, A4} are described in the form of metadata. When describing, each attribute information A1, A2, A3, A4 is defined as a data column information of a data structure logical table. At the same time, the associated user identifier A is also included as a data column information of the data structure logical table, thus obtaining the data structure logical table of organization A (associated with user identifiers).

[0069] Similarly, the data structure to be processed for organization B and the corresponding user identifiers: {User Identifiers B: B1, B2, B3, B4, B5, B6} are described in the form of metadata. During the description, each attribute information B1, B2, B3, B4, B5, B6 is defined as a data column of the data structure logical table. At the same time, the associated user identifiers B are also included as a data column of the data structure logical table, thus obtaining the data structure logical table of organization B (associated with user identifiers).

[0070] When using metadata description, each attribute in the above attribute information A1-A4 and B1-B6 can include attribute parameters, attribute parameter values, and attribute parameter value types (e.g., integer, floating-point, string, custom, etc.). The internal implementation logic of the above metadata description can be... Figure 1B The example is implemented using the logical model layer 113a and the logical metadata (Logic_Meta) layer 113b. The aforementioned attribute information can further include: the source of the attribute information, the reference relationships between attribute information, etc. The metadata description regarding the source of the attribute information, the reference relationships between attribute information, etc., can be achieved through... Figure 1B The example is implemented using the logical model relation layer 114.

[0071] In the data architecture provided in this embodiment, the aforementioned data structure logical table serves as a means of describing the data structure to be processed, but is not the actual storage location of the data structure to be processed. The physical table is the actual storage location of the data structure to be processed. Therefore, after configuring the mapping relationship between the data structure logical table and the pre-built physical table in operation S203, the storage of the data structure to be processed in the physical table is realized in operation S204 based on the configured mapping relationship. For example, the storage of attribute information A1, A2, A3, and A4 of organization A in the target physical table area Area(A) (an area composed of specific columns and rows) is realized, and the storage of attribute information B1, B2, B3, B4, B5, and B6 of organization B in the target physical table area Area(B) is realized.

[0072] In operation S204 above, multiple data structures to be processed corresponding to different user identifiers can be stored in the pre-built physical table, thereby distinguishing target physical table areas belonging to different users based on user identifiers. Furthermore, based on the differences in user identifiers, when performing processing operations such as adding, deleting, querying, modifying, or displaying on each target physical table area, it will be associated with the target physical table area corresponding to the user identifier. This enables different users to share a single physical table to store their respective data structures to be processed, while the stored data is isolated from each other, and the operations performed on the stored data are also isolated from each other.

[0073] Based on the above operations S201-S204, by describing the data structure to be processed and the corresponding user identifier in the form of metadata, a logical table of data structures associated with user identifiers is obtained. By configuring the mapping relationship between the logical table of data structures and the pre-built physical table, the data structure to be processed described by the logical table of data structures can be stored in the target physical table area of ​​the pre-built physical table. Each target physical table area is associated with a user identifier, thereby enabling the storage of different types of data structures to be processed in a single physical table. This processing method is universal for various types of data structures to be processed and facilitates subsequent operations such as adding, deleting, querying, and modifying data. It overcomes the technical problems of related technologies that require building different database models from scratch for different organizations to access various new types of storage objects, resulting in low database operation efficiency, code redundancy, high development and maintenance costs, and increasingly slow technology iteration.

[0074] In the application scenario of integrating new assets into the ABS asset securitization system, the physical storage of real assets is isolated through the architectural design of ABS metadata. Abstracted logical tables (logical layer) are used to adapt to various asset changes and accommodate any asset type. A unified and universal basic service architecture design completes standard asset operation processes, ensuring that the code used in integrating any asset type is the same and does not grow. The two-layer design isolates the physical model (physical tables) and the logical model (logical tables), saving development resources and providing unified and convenient management for later operation and maintenance. When business adjustments are made to adapt to the market later, configurable support can be quickly provided, saving human resources.

[0075] Figure 3 A detailed implementation flowchart of operation S203 according to an embodiment of the present disclosure is illustrated.

[0076] According to embodiments of this disclosure, the aforementioned data structure to be processed includes: a set of attribute information for describing the same object; the set of attribute information for describing the same object differs for different users.

[0077] According to embodiments of this disclosure, referring to Figure 3 As shown, the operation S203, which configures the mapping relationship between the above data structure logical table and the pre-built physical table, includes the following sub-operations: S2031, S2032, S2033 and S2034.

[0078] In sub-operation S2031, for a set of attribute information in a data structure logical table associated with the same user identifier, a matching data column consistent with the data type of each attribute information is determined from the pre-constructed physical table based on the description in the form of metadata.

[0079] In sub-operation S2032, select the candidate data rows that are not occupied in the above-mentioned matching data columns.

[0080] In sub-operation S2033, for matching data columns that are associated with all attribute information of the same user identifier, determine the common alternative data rows that are in an unoccupied state.

[0081] In sub-operation S2034, based on the correspondence between the above set of attribute information and the above common candidate data rows and the above matching data columns, a mapping relationship between the data structure logical table and the pre-built physical table is generated.

[0082] The implementation process of operation S203 corresponding to the data structure logic table of organization A is used as an example. Similarly, the above sub-operations S2031 to S2034 can also be implemented for the data structure logic table of organization B, which will not be listed one by one here.

[0083] In sub-operation S2031, based on the data types of attribute information A1 to A4 described in metadata form in the data structure logical table of organization A, matching data columns that are consistent with the data types of each attribute information A1 to A4 can be determined in the pre-built physical table. For example, in the pre-built physical table, the data columns corresponding to A1, A2, A3, and A4 in sequence are P1, P3, P4, and P6.

[0084] In sub-operation S2032, select the candidate data rows that are not occupied in the above matching data columns P1, P3, P4, and P6. For example, the result is: the candidate data rows that are not occupied in P1, P3, P4, and P6 are the third data row C3, the third data row C3 to the fifth data row C5, the second data row C2 to the fourth data row C4, and the third data row C3, respectively.

[0085] In sub-operation S2033, for the matching data columns of all attribute information associated with the same user identifier, the common candidate data rows that are in an unoccupied state are determined. That is, by finding the intersection of the candidate data rows of each matching data column of all attributes, the common candidate data row is obtained by finding the intersection of the third data row C3, the third data row C3 to the fifth data row C5, the second data row C2 to the fourth data row C4, and the third data row C3.

[0086] In sub-operation S2034, a mapping relationship can be generated based on the correspondence between A1 to A4 and the common candidate data rows and the aforementioned matching data columns. Based on the correspondence between A1 and data column P1 and the third data row C3, a mapping relationship can be generated between the position of A1 in the data structure logical table and data column P1 and the third data row C3 in the physical table. Similarly, a mapping relationship can be generated between the position of A2 in the data structure logical table and data column P3 and the third data row C3 in the physical table, between the position of A3 in the data structure logical table and data column P4 and the third data row C3 in the physical table, and between the position of A4 in the data structure logical table and data column P6 and the third data row C3 in the physical table.

[0087] According to an embodiment of this disclosure, the operation S204 of storing the data structure to be processed in the pre-constructed physical table based on the mapping relationship to obtain a target physical table region associated with a user identifier includes: based on the mapping relationship, storing a set of attribute information from the logical table of the data structure associated with the same user identifier into the corresponding common candidate data row and matching data column in the physical table, wherein the common candidate data row and the matching data column constitute the target physical table region, and the common candidate data row is associated with the user identifier.

[0088] Figure 4 A flowchart illustrating a data processing method according to another embodiment of this disclosure is shown schematically.

[0089] Reference Figure 4 As shown, the data processing method provided in this embodiment includes not only the above-described operations S201 to S204, but also the following operations: S401, S402, S403 and S404.

[0090] In operation S401, the user's operation instructions for the aforementioned data structure to be processed are received.

[0091] In operation S402, the above operation instructions and corresponding user identifiers are described in the form of metadata to obtain an operation logic table associated with user identifiers.

[0092] In operation S403, based on the above mapping relationship, the above operation logic table is parsed into target operations to be performed on the target physical table region associated with the user identifier.

[0093] In operation S404, the corresponding target operation is performed in the aforementioned target physical table area.

[0094] According to embodiments of this disclosure, the above-mentioned operation instructions include at least one of the following: instructions to add a data structure to be processed, instructions to delete a data structure to be processed, instructions to modify a data structure to be processed, instructions to query a data structure to be processed, and instructions to display a data structure to be processed.

[0095] In the embodiments of this disclosure, operations S401 to S404 can be performed after operations S201 to S204, and the application scenario is: performing at least one operation such as adding, deleting, modifying, querying, or displaying the stored data structure to be processed. Operation S401 can be performed in parallel with operation S201 or within a single operation. For example, while obtaining the user's data structure to be processed and the corresponding user identifier, the user's operation instructions for the data structure to be processed can also be received. Operation S402 can be performed in parallel with operation S202 or within a single operation. Operation S403 needs to be executed after operation S203. Operation S404 can be executed after operation S204.

[0096] Each target physical table region is associated with a user identifier, enabling the storage of different types of data structures to be processed within a single physical table. The operation instructions executed for each type of data structure are also universal; simply locate the corresponding target physical table region using the user identifier associated with that region and perform the appropriate target operation.

[0097] For example, the operation instruction X implemented for the data structure to be processed in organization A. A Described in metadata form, the operation logic table of institution A (associated with user identifiers) is obtained; based on the mapping relationship, the target operation X implemented on the target physical table area of ​​the aforementioned institution A can be obtained. A The parsing is for the target operation Y of the target physical table region Area(A) (the physical table region that stores the data structure to be processed for organization A) of organization A. A The target operation Y A This represents the operation performed on the element information of organization A stored in a specific row and column of the physical table. In this way, based on the user identifier, it is possible to distinguish operations for different users and to distinguish the different physical table operation areas corresponding to each operation.

[0098] Figure 5A flowchart illustrating a data processing method according to yet another embodiment of the present disclosure is shown.

[0099] Reference Figure 5 As shown, the data processing method provided in this embodiment includes, in addition to the operations S201-S204 described above, the following operation S501: constructing a physical table. In other embodiments of this disclosure, the data processing method may include operations S201-S204, S401-S404, and operation S501.

[0100] The above operation S501 can be executed before operation S203, so that the physical table constructed using operation S501 can be called in operation S203. It should be understood that the operation of constructing the physical table in operation S501 can be performed only once in advance. After the physical table is constructed, it is only necessary to call the constructed physical table each time data processing is performed, without having to execute operation S501 every time data processing is performed.

[0101] Figure 6 A detailed implementation flowchart of operation S501 according to an embodiment of the present disclosure is illustrated.

[0102] According to embodiments of this disclosure, referring to Figure 6 As shown, the above operation S501 for constructing the physical table includes the following sub-operations: S5011, S5012 and S5013.

[0103] In suboperation S5011, information about the data type of the existing data structure is obtained.

[0104] For example, there are S existing data structures, and the data types of the obtained existing data structures include: data types T1, T2, ..., T. M There are M in total, where S≥2 and S is an integer, M≥2 and M is an integer, and M>S.

[0105] In sub-operation S5012, based on the information of the above data types, a table parameter group covering the data types of the above existing data structures is generated.

[0106] The table parameters need to cover the above M data types T1, T2, ... T M From this, we can obtain {T1, T2, ..., T} M} is a subset of the table parameter group.

[0107] In sub-operation S5013, one or more physical tables are constructed based on the above table parameter group.

[0108] Based on the table parameter group, the data columns of the physical table can be configured, thereby obtaining one or more physical tables.

[0109] According to an embodiment of this disclosure, the sub-operation S5013 of constructing one or more physical tables based on the above-mentioned table parameter group includes: based on the combination relationship of data types in the existing data structure, counting the frequency of occurrence of data types in the above-mentioned table parameter group; dividing the above-mentioned table parameter group according to the frequency of occurrence of the above-mentioned data types to obtain one or more sets of table parameter groups; and using the above-mentioned one or more sets of table parameter groups as data type parameters of data columns of physical tables to construct one or more physical tables.

[0110] For example, taking M=10 and S=5 as an example, the combination relationships of the data types of the five existing data structures are as follows: the combination relationship of the data types of data structure 1 is combination 1: {T1, T3, T7}; the combination relationship of the data types of data structure 2 is combination 2: {T1, T2, T7, T6}; the combination relationship of the data types of data structure 3 is combination 2: {T1, T4, T7, T8}; the combination relationship of the data types of data structure 4 is combination 4: {T2, T3, T7, T4}; and the combination relationship of the data types of data structure 5 is combination 2: {T9, T... 10 T7, T5, T8}.

[0111] Based on the above combination relationships, the frequency of occurrence of the data types in the parameter groups of the above table can be counted as follows: Frequency (T1) = 3 / 5; Frequency (T2) = 2 / 5; Frequency (T3) = 2 / 5; Frequency (T4) = 2 / 5; Frequency (T5) = 1 / 5; Frequency (T6) = 1 / 5; Frequency (T7) = 5 / 5; Frequency (T8) = 2 / 5; Frequency (T9) = 1 / 5; Frequency (T... 10 = 1 / 5.

[0112] According to embodiments of this disclosure, dividing the table parameter groups based on the frequency of occurrence of the aforementioned data types includes: reorganizing data types within the same frequency range to obtain one or more sets of table parameter groups. The frequency range can be set according to actual needs; for example, it can be divided into high-frequency and low-frequency ranges based on a threshold, or into three ranges based on two thresholds, and so on. In special cases, only a large frequency range can be set, which can cover all frequencies or 90%–95% of frequencies, and all data types falling within this range are reorganized.

[0113] For example, data types that appear more frequently than a set frequency can be reorganized to obtain a high-frequency table parameter group; based on the high-frequency table parameter group, a high-frequency physical table can be constructed; similarly, data types that appear less frequently than a set frequency can be reorganized to obtain a low-frequency table parameter group; based on the low-frequency table parameter group, a low-frequency physical table can be constructed.

[0114] A second exemplary embodiment of this disclosure provides an apparatus for data processing.

[0115] Figure 7 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0116] Reference Figure 7 As shown, the data processing apparatus 700 provided in this embodiment includes: a data acquisition module 701, a logical table construction module 702, a mapping relationship configuration module 703, and a storage module 704.

[0117] The aforementioned data acquisition module 701 is used to acquire the user's pending data structure and the corresponding user identifier.

[0118] The aforementioned logical table construction module 702 is used to describe the aforementioned data structure to be processed and the corresponding user identifier in the form of metadata, thereby obtaining a logical table of data structure associated with the user identifier.

[0119] The mapping configuration module 703 is used to configure the mapping relationship between the logical table of the data structure and the pre-built physical table. The mapping configuration module 703 includes various functional modules or sub-modules for implementing sub-operations S2031 to S2034.

[0120] The aforementioned storage module 704 is used to store the aforementioned data structure to be processed in the aforementioned pre-built physical table based on the aforementioned mapping relationship, thereby obtaining a target physical table region associated with a user identifier.

[0121] According to embodiments of this disclosure, in addition to the data acquisition module 701, logic table construction module 702, mapping relationship configuration module 703, and storage module 704, the data processing apparatus 700 may also include: an operation instruction receiving module, an operation logic table construction module, an operation parsing module, and an operation implementation module.

[0122] The aforementioned operation instruction receiving module is used to receive operation instructions from the user for the aforementioned data structure to be processed.

[0123] The aforementioned operation logic table construction module is used to describe the above operation instructions and their corresponding user identifiers in the form of metadata, thereby obtaining an operation logic table associated with the user identifier.

[0124] The operation parsing module described above is used to parse the operation logic table into target operations for the target physical table regions associated with user identifiers, based on the mapping relationship described above.

[0125] The above-mentioned operation implementation module is used to perform the corresponding target operation in the above-mentioned target physical table area.

[0126] In addition to the data acquisition module 701, logical table construction module 702, mapping relationship configuration module 703, and storage module 704, the data processing device 700 may also include a physical table construction module.

[0127] The physical table construction module described above is used to construct physical tables. This module includes functional modules or sub-modules for implementing sub-operations S5011 to S5013.

[0128] According to embodiments of this disclosure, the above-described apparatus 700 may also include a data acquisition module 701, a logical table construction module 702, a mapping relationship configuration module 703, a storage module 704, a physical table construction module, and a data acquisition module 701, a logical table construction module 702, a mapping relationship configuration module 703, a storage module 704, and a physical table construction module.

[0129] Any multiple of the aforementioned data acquisition module 701, logic table construction module 702, mapping relationship configuration module 703, storage module 704, operation instruction receiving module, operation logic table construction module, operation parsing module, and operation implementation module can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. At least one of the data acquisition module 701, logic table construction module 702, mapping relationship configuration module 703, storage module 704, operation instruction receiving module, operation logic table construction module, operation parsing module, and operation implementation module can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the data acquisition module 701, logical table construction module 702, mapping relationship configuration module 703, storage module 704, operation instruction receiving module, operation logical table construction module, operation parsing module, and operation implementation module can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0130] A third exemplary embodiment of this disclosure provides an electronic device.

[0131] Figure 8 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown.

[0132] Reference Figure 8 As shown, the electronic device 800 provided in this embodiment includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804. The memory 803 is used to store computer programs. When the processor 801 executes the program stored in the memory, it implements the data processing method described above.

[0133] A fourth exemplary embodiment of this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the data processing method described above.

[0134] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0135] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method, characterized in that, include: Obtain the user's pending data structure and corresponding user identifier; The data structure to be processed and the corresponding user identifier are described in the form of metadata to obtain a logical table of data structure associated with the user identifier. Configure the mapping relationship between the logical tables of the data structure and the pre-built physical tables; as well as Based on the mapping relationship, the data structure to be processed is stored in the pre-built physical table to obtain the target physical table region associated with the user identifier; This also includes: building physical tables; The construction of the physical table includes: To obtain information about the data type of an existing data structure; Based on the information of the data type, generate a table parameter group that covers the data types of the existing data structure; Based on the table parameter group, one or more physical tables are constructed; Based on the table parameter group, one or more physical tables are constructed, including: Based on the combination relationship of data types in the existing data structure, the frequency of occurrence of data types in the parameter group of the table is counted. The table parameter groups are divided according to the frequency of occurrence of the data types to obtain one or more sets of table parameter groups; and One or more sets of table parameters are used as data type parameters for the data columns of physical tables to construct one or more physical tables.

2. The method according to claim 1, characterized in that, The data structure to be processed includes: a set of attribute information for describing the same object; the set of attribute information for describing the same object differs for different users; The configuration of the mapping relationship between the data structure logical table and the pre-built physical table includes: For a set of attribute information in a logical table of data structures that are associated with the same user identifier, based on the description in the form of metadata, a matching data column consistent with the data type of each attribute information is determined from the pre-built physical table; Identify candidate data rows in the matching data column that are not currently occupied; For matching data columns containing all attribute information related to the same user identifier, identify common candidate data rows that are not currently occupied; and Based on the correspondence between the set of attribute information and the common candidate data rows and the matching data columns, a mapping relationship is generated between the data structure logical table and the pre-built physical table.

3. The method according to claim 2, characterized in that, Based on the mapping relationship, the data structure to be processed is stored in the pre-built physical table to obtain a target physical table region associated with a user identifier, including: Based on the mapping relationship, a set of attribute information from a logical table of data structures associated with the same user identifier is stored in the corresponding common candidate data row and matching data column of the physical table. The common candidate data row and the matching data column constitute the target physical table area, and the common candidate data row is associated with the user identifier.

4. The method according to any one of claims 1-3, characterized in that, Also includes: Receive operation instructions from the user for the data structure to be processed; The operation instructions and their corresponding user identifiers are described in the form of metadata to obtain an operation logic table associated with the user identifiers. Based on the mapping relationship, the operation logic table is parsed into target operations to be performed on the target physical table region associated with the user identifier; as well as Perform the corresponding target operation in the target physical table area.

5. The method according to claim 1, characterized in that, The step of dividing the table parameter group according to the frequency of occurrence of the data type includes: Based on the frequency range of the data types, data types within the same frequency range are reorganized to obtain one or more sets of table parameter groups.

6. A data processing apparatus, characterized in that, include: The data acquisition module is used to acquire the user's data structure to be processed and the corresponding user identifier; The logical table construction module is used to describe the data structure to be processed and the corresponding user identifier in the form of metadata, so as to obtain a logical table of data structure associated with the user identifier. The mapping relationship configuration module is used to configure the mapping relationship between the data structure logical table and the pre-built physical table; as well as A storage module is used to store the data structure to be processed in the pre-built physical table based on the mapping relationship, so as to obtain a target physical table area associated with a user identifier; The physical table construction module is used to construct physical tables; The construction of the physical table includes: To obtain information about the data type of an existing data structure; Based on the information of the data type, generate a table parameter group that covers the data types of the existing data structure; Based on the table parameter group, one or more physical tables are constructed; Based on the table parameter group, one or more physical tables are constructed, including: Based on the combination relationship of data types in the existing data structure, the frequency of occurrence of data types in the parameter group of the table is counted. The table parameter groups are divided according to the frequency of occurrence of the data types to obtain one or more sets of table parameter groups; and One or more sets of table parameters are used as data type parameters for the data columns of physical tables to construct one or more physical tables.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-5.

Images