List data processing method, device and equipment

Abstract

The invention provides a list data processing method, device and equipment, and the method comprises the steps: obtaining source table configuration information and wide table configuration information, the source table configuration information indicates a data mapping relation between a source table and original business data, and the wide table configuration information indicates a data processing rule for the source table; determining a source table based on the original business data according to the source table configuration information and the wide table configuration information, and assembling business data of a preset field in the source table to obtain a wide table; in response to an editing operation which is sent by the client and aims at a first preset field in the service table, obtaining target service data corresponding to the first preset field from the wide table, and obtaining a list operation result; according to the list header configuration of the first preset field, assembling the list operation result to obtain first list data; and returning the first list data to the client to update the table data in the service table. The diversity and personalized display of the list data are improved, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a list data processing method, apparatus, and device. Background Technology

[0002] Currently, client applications often display business information in list format. For example, when displaying customer-related business information, the client can show a customer list on the business information display page. The customer list includes multiple fields such as customer name, customer contact information, and customer follow-up person's name, with each field corresponding to different business information. However, in related list data processing solutions, the business information displayed in the list on the client is fixed, and users cannot adjust it according to their needs. Therefore, there are problems with the list data display format being relatively monotonous and the user experience being poor. Summary of the Invention

[0003] This application provides a method, apparatus, and device for processing list data, which can improve the diversity and personalized display of list data and enhance user experience.

[0004] In a first aspect, this application provides a list data processing method, which includes: obtaining source table configuration information and wide table configuration information, wherein the source table configuration information indicates the data mapping relationship between the source table and the original business data, and the wide table configuration information indicates the data processing rules for the source table; determining the source table based on the original business data according to the source table configuration information and the wide table configuration information, and assembling the business data of preset fields in the source table to obtain a wide table; responding to an edit operation sent by the client for a first preset field in the business table, obtaining the target business data corresponding to the first preset field from the wide table to obtain a list operation result; assembling the list operation result according to the list header configuration of the first preset field to obtain first list data; and returning the first list data to the client to update the table data in the business table.

[0005] Secondly, this application provides a list data processing apparatus, comprising: a first acquisition module, configured to acquire source table configuration information and wide table configuration information, wherein the source table configuration information indicates the data mapping relationship between the source table and the original business data, and the wide table configuration information indicates the data processing rules for the source table; a first assembly module, configured to determine the source table based on the original business data according to the source table configuration information and the wide table configuration information, and assemble the business data of preset fields in the source table to obtain a wide table; a second acquisition module, configured to respond to an edit operation sent by the client for a first preset field in the business table, acquire the target business data corresponding to the first preset field from the wide table, and obtain a list operation result; a second assembly module, configured to assemble the list operation result according to the list header configuration of the first preset field to obtain first list data; and a data return module, configured to return the first list data to the client to update the table data in the business table.

[0006] Thirdly, this application provides an electronic device, including: a processor and a memory, the memory for storing a computer program, and the processor for calling and running the computer program stored in the memory to perform the methods as described in the first aspect or its various implementations.

[0007] Fourthly, this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0008] Fifthly, this application provides a computer program product including computer program instructions that cause a computer to perform the methods as described in the first aspect or its various implementations.

[0009] Sixthly, this application provides a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0010] Other technical features and effects involved in the technical solution of this application will be described in subsequent embodiments, and will not be repeated here to avoid repetition. Attached Figure Description

[0011] The accompanying drawings used in the following description of the embodiments are introduced.

[0012] Figure 1 An application scenario diagram provided for an embodiment of this application;

[0013] Figure 2 A flowchart illustrating a list data processing method provided in this application embodiment;

[0014] Figure 3 A schematic diagram illustrating a list data processing method provided in an embodiment of this application;

[0015] Figure 4A A schematic diagram illustrating another list data processing method provided in an embodiment of this application;

[0016] Figure 4B A schematic diagram illustrating another list data processing method provided in an embodiment of this application;

[0017] Figure 4C A schematic diagram illustrating yet another list data processing method provided in an embodiment of this application;

[0018] Figure 4D A schematic diagram illustrating yet another list data processing method provided in an embodiment of this application;

[0019] Figure 4E A schematic diagram illustrating yet another list data processing method provided in an embodiment of this application;

[0020] Figure 5 A schematic diagram of a list data processing apparatus 500 provided in an embodiment of this application;

[0021] Figure 6 This is a schematic block diagram of an electronic device 600 provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0024] In one embodiment, the technical solution of this application can be used in a scenario of list data processing, specifically in scenarios such as displaying, creating, deleting, modifying, filtering or sorting list data, but is not limited thereto.

[0025] For example, Figure 1 An application scenario diagram provided for an embodiment of this application, such as... Figure 1 As shown, this application scenario may include a terminal device 110 with an installed client connected via a wired or wireless network and a server 120. The server may be a single server, a server cluster consisting of multiple servers, or a cloud platform control center, but is not limited to these. The terminal device 110 may be a mobile phone, tablet computer, laptop computer, or desktop computer, but is not limited to these.

[0026] It should be noted that, Figure 1 The terminal devices and servers mentioned are merely examples; in practice, other numbers or types of terminal devices or servers may also be included.

[0027] Specifically, the server can respond to operations sent by the client targeting the business table (specifically, editing operations on fields in the business table or data modification operations on the business table), obtaining source table configuration information, wide table configuration information, and list configuration information (described in detail in subsequent embodiments); and perform processing on the source table and wide table based on this configuration information, thereby enabling editing operations such as filtering or sorting fields in the business table, or, based on data modification operations, first modifying the source table configuration information and wide table configuration information, and then implementing changes to the business table based on the changes to the above configuration information, such as creating the business table, modifying specific business data in the business table, or adding or deleting fields in the business table; then, the business table or list data obtained after the above operations can be sent to the client; subsequently, the client can display the business table sent by the server or update the table data in the business table based on the list data, thereby enabling users to edit the information displayed in the business table according to their needs, improving the diversity and personalized display of list data, and enhancing the user experience.

[0028] The technical solution of this application will be described in detail below:

[0029] It should be noted that all technical solutions in this application can be combined in any way to form optional embodiments of this application, and will not be elaborated upon here to avoid repetition. Furthermore, in the specific implementations of this application, various configuration information (e.g., source table configuration information, wide table configuration information, or list configuration information), various fields, and various tables (e.g., source table, wide table, or business table) and related data, information, requests, or instructions are involved. When the embodiments of this application are applied to specific products or technologies, user permission, consent, or authorization is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0030] In the technical solution of this application, the processing method for list data can be achieved through steps such as editing fields in the business table and modifying data in the business table. These steps will be described below. This application does not restrict the order of the editing and data modification operations.

[0031] The steps involved in editing fields in a business table mainly include filtering and sorting of fields, which will be described in the following examples.

[0032] In one embodiment, Figure 2 A flowchart illustrating a list data processing method provided in this application embodiment, the method can be performed by, for example... Figure 1 The server shown performs the actions, but is not limited to this.

[0033] like Figure 2 As shown, the method may include the following steps:

[0034] S211: Obtain source table configuration information and wide table configuration information. The source table configuration information indicates the data mapping relationship between the source table and the original business data, and the wide table configuration information indicates the data processing rules for the source table.

[0035] S212: Based on the source table configuration information and the wide table configuration information, determine the source table based on the original business data, and assemble the business data of the preset fields in the source table to obtain the wide table;

[0036] S213: In response to the client's edit operation on the first preset field in the business table, retrieve the target business data corresponding to the first preset field from the wide table and obtain the list operation result;

[0037] S214: Assemble the list operation results according to the list header configuration of the first preset field to obtain the first list data;

[0038] S215: Return the first list data to the client to update the table data in the business table.

[0039] Before introducing the embodiments corresponding to editing and data change operations, we will first introduce the data lists such as source tables, wide tables, and business tables involved in the embodiments, as well as the configuration information such as source table configuration information, wide table configuration information, and list configuration information. It should be noted that in the accompanying drawings and embodiments, the source table configuration information, wide table configuration information, and list configuration information are referred to as source table configuration, wide table configuration, and list configuration.

[0040] In one embodiment, the server can store the raw business information in a database (DB). The data in the source tables can originate from the database and correspond one-to-one with each database table (specifically, a database table within the database). For example, the source tables may include a list of customer names, a list of customer genders, a table of customer contact information, customer follow-up information, etc. The server can retrieve the raw business data required to construct the source tables from the database based on the fields specified in the source table configuration information, and determine the various source tables accordingly.

[0041] A wide table can be a table obtained by processing multiple source tables according to at least one specific processing rule (which can be a random combination of multiple processing rules). Multiple source tables can correspond to one wide table, meaning a wide table can be obtained from multiple source tables. The source tables can be called physical tables, and the wide table can be called a logical table. The aforementioned specific processing rule can be the rule indicated in the wide table configuration information, specifically the rule specified for that wide table. For example, a wide table could be a customer information table, which could include three fields: customer name, customer gender, and customer phone number. Each field corresponds to a column in the customer information table, and each row of the customer information table could include a customer's name, gender, and phone number.

[0042] A business table refers to the table actually displayed on the client side; it's the table the user actually sees. It can be obtained by assembling corresponding business data retrieved from a wide table based on user-triggered editing or data change operations. Specifically, it can be obtained and assembled according to the business data to be retrieved from the wide table and / or the assembly method indicated by the list configuration information. One wide table can correspond to multiple business tables; that is, multiple business tables can be obtained from one wide table. For example, a business table can include a customer pool (e.g., an active customer information table corresponding to active customers in the customer information table), a lead pool (e.g., an information table corresponding to a specific follow-up person or a specific follow-up status in the customer information table), and an invalid lead pool (e.g., an information table of customers with invalid contact information in the customer information table).

[0043] Wide tables can be stored in Elasticsearch (ES). Specifically, one wide table can correspond to one ES index, and data recorded in the same wide table can be stored in the same ES index. One data record in a wide table can correspond to one ES record, and the same ES index can contain multiple ES records.

[0044] Similarly, the source table can also be stored in Elasticsearch.

[0045] Source table configuration information allows you to configure the mapping relationship between raw business data and source tables. This enables configuration of source tables, and further, it allows you to configure the mapping relationship from data in the database (DB) to data in Elasticsearch (ES), thus connecting the structures of the DB and ES. One record in the DB corresponds to one record in ES, and one source table configuration piece of information corresponds to one source table; that is, one source table configuration piece of information can configure one source table. Specifically, source table configuration information can indicate the mapping relationship between source tables and raw business data, specifically indicating the correspondence between fields in the source table and the raw business data. For example, raw business data may include customer names and customer genders. If the source table configuration information indicates that source table 1 corresponds to customer names and source table 2 corresponds to customer genders, then the server can retrieve customer names from the DB and construct source table 1 as a list of customer names, and retrieve customer genders and construct source table 2 as a table of customer genders.

[0046] Wide table configuration information specifies the data processing rules for the source table. The server can process the source table according to these rules to obtain the wide table. One wide table can correspond to one wide table configuration, or one wide table configuration can correspond to multiple wide tables. Specifically, wide table configuration information can include the main table corresponding to the wide table and its contained main table fields, the sub-table and its contained sub-table fields, and the relationships between the main table and the sub-table. The main table and the sub-table can be the source tables used to construct the wide table. For example, the wide table configuration information may include: main table 1 "Customer Information Table", sub-table 1 "Order Table", main table 1 contains the main table fields "Customer Name, Customer Gender and Customer Phone Number", sub-table 1 contains the sub-table fields "Customer Name and Transaction Amount", main table 1 and sub-table 1 are associated based on the field "Customer Name", and the association between source table 3 "Transaction Quantity" and source table 4 "Transaction Unit Price" is to multiply the corresponding data to calculate the table data corresponding to the field "Transaction Amount" based on the wide table configuration information. The server can obtain the source tables corresponding to "Customer Name, Customer Gender, Customer Phone Number, Transaction Quantity and Transaction Unit Price" according to the wide table configuration information, and calculate the above-mentioned main table 1 and sub-table 1 according to the association between these source tables and the indicated source tables, thereby determining the wide table including main table 1 and sub-table 1.

[0047] List configuration information is used to set the display, filtering, and sorting of business tables in the client. It can include list header configuration, field display configuration, filtering configuration, and sorting configuration. One list configuration file can correspond to one business table; that is, one list configuration file can configure one business table. Specifically, the list header configuration can configure the table header, such as whether the table header is displayed and its display format. The field display configuration can configure the display format of table data under various fields in the wide table in the business table. For example, it can configure the table data "pending / success" under the "processing status" field in the wide table to be displayed as "processing / completed" in the business table. The filtering configuration information can configure the filters displayed on the client, specifically indicating the fields to be filtered (i.e., the fields displayed on the client for user selection and used to filter the business table) and information on how to convert the field filtering operation into the corresponding query statement (i.e., indicating how to convert the filtering operation into a query statement). The sorting configuration information can configure the sorting buttons displayed on the client side. Specifically, it can indicate the field to be sorted (i.e., the field displayed on the client side for user selection and can be used for sorting in the business table) and the sorting method (e.g., full sort or default sort). Of course, any of the list header configuration, field display configuration, filter configuration information, and sorting configuration information can be information independent of the list configuration information, that is, it can be excluded from the list configuration information. This application does not impose any restrictions on this.

[0048] It is understandable that by configuring the source table, wide table, and list information, the database storing the original business data can be connected to the list ultimately displayed on the client. This links the original business data with the business tables, enabling diverse business scenarios for list data processing and flexible end-to-end control from original business data to final business display. For example, the configuration information, source table, and wide table can be pre-defined. Without affecting the wide table and list, various business tables that meet user needs can be generated through the list configuration information. For instance, it could be a business table after filtering and sorting an existing business table. Furthermore, when upgrading a business table (e.g., deleting / adding a field, modifying table data under a field) or creating a new business table—that is, when performing data change operations on a business table—the configuration information can be automatically changed based on the data change operation. Then, based on the changed configuration information, the source table and wide table are changed sequentially to achieve the final data change for the business table.

[0049] Furthermore, compared to related technologies that require the separate development of each business table, the technical solution of this application can reduce the development cost and difficulty of list data.

[0050] The process will be described below.

[0051] In one embodiment, combining the above, the user can determine various configuration information based on the client; the client can obtain this configuration information and send it to the server, thereby allowing the server to obtain the aforementioned configuration information. Then, the server can first retrieve the original business data required to generate the source table from the database storing the original business data, based on the source table configuration information, and generate the source table accordingly. Specifically, the source table can be obtained and generated based on the data mapping relationship between the source table and the original business data indicated by the source table configuration information. Then, the wide table is obtained by assembling the business data of preset fields in the source table according to the wide table configuration information. Specifically, the wide table can be assembled according to the data processing rules for the source table indicated by the wide table configuration information.

[0052] Here, a preset field refers to a field in the source table. Specifically, a field in the wide table can be the same as a preset field in the source table, and the field value of the field can be the field value of the corresponding preset field in the source table; a field in the wide table can also be different from a preset field in the source table, and the field value of the field can be calculated from at least one preset field in the source table, specifically according to the rules indicated by the wide table configuration information, which is not limited in this application.

[0053] In one embodiment, after identifying the wide table relevant to the business scenario, the wide table can be displayed as the business table on the client side. Furthermore, after the client displays the business table, it can receive editing operations from the user on a first preset field within the business table.

[0054] The first preset field refers to the field selected by the user in the preset fields of the displayed business table, which is the object of the editing operation. The editing operation includes, but is not limited to, filtering and sorting operations on various fields in the business table.

[0055] Taking the editing operation as a filtering operation as an example, after receiving a filtering operation for the first preset field in the business table, the business data corresponding to the first preset field in the wide table can be filtered to obtain the list operation result. Taking the business table as a customer list as an example, the wide table includes fields: customer name, customer gender, customer phone number, transaction quantity, and transaction price. The wide table is displayed as a business table on the client. When a user's filtering operation for the customer gender field in the business table is received, the business data corresponding to the customer gender in the wide table can be filtered to obtain the list operation result. It can be understood that the business table after the filtering operation includes the fields: customer name, customer phone number, transaction quantity, and transaction price.

[0056] Taking the editing operation as a sorting operation as an example, after receiving a sorting operation for the first preset field in the business table, the business data corresponding to the first preset field in the wide table can be sorted to obtain the list operation result. Taking the business table as a customer list as an example, the wide table includes the following fields, and the order of each field is: customer name, customer gender, customer phone number, transaction quantity, and transaction price. The wide table is displayed as a business table on the client. When a user's sorting operation for the transaction quantity and transaction price fields in the business table is received, such as sorting the transaction price first and the transaction quantity second, the transaction price and transaction quantity fields in the wide table can be sorted to obtain the list operation result. It can be understood that the field order of the business table after the sorting operation is: customer name, customer gender, customer phone number, transaction price, and transaction quantity.

[0057] In one embodiment, the first preset field is derived from the field to be filtered indicated by the filtering configuration information. For example, it may be a field selected by the user from the field to be filtered indicated by the filtering configuration information displayed on the client. Accordingly, S213 may include: in response to the filtering operation on the first preset field, obtaining the filtering configuration information; constructing a filtering query statement for the wide table based on the filtering configuration information and the first preset field; filtering the target business data corresponding to the first preset field in the wide table according to the filtering query statement to obtain the list operation result.

[0058] The server can store filtering configuration information and return it to the client via an interface. The client can display a filter based on the filtering configuration information. After the user triggers the filter, the client can display multiple filter items, which are the fields to be filtered indicated by the filtering configuration information. Then, the user can select a first preset field from the fields to be filtered. The client can then send the filtering operation for the first preset field to the server. The server can then respond to the filtering operation for the first preset field by filtering the preset fields in the business table, such as deleting the business data corresponding to the first preset field, and finally obtaining the list operation result.

[0059] The method for converting filter operations into query statements can be determined based on the database storing the wide table. For example, assuming the wide table is stored in Elasticsearch, the filter configuration information can instruct the filter operations to be converted into query statements suitable for Elasticsearch.

[0060] In another embodiment, the first preset field may be derived from the field to be sorted indicated by the sorting configuration information. For example, it may be a field selected by the user from the field to be sorted indicated by the sorting configuration information displayed on the client. Accordingly, S213 may include: in response to the sorting operation on the first preset field, constructing a sorting query statement for the wide table based on the first preset field and the sorting operation on the first preset field; sorting the target business data corresponding to the first preset field in the wide table according to the sorting query statement to obtain the list operation result.

[0061] The server can store sorting configuration information and return it to the client via an interface. The client can then display the sortable fields indicated by the sorting configuration information, i.e., the fields to be sorted. The user can then select a first preset field from the fields to be sorted. The client can then send a sorting operation for the first preset field to the server. The server can respond to the sorting operation for the first preset field by sorting the first preset field in the business table, or by sorting all business data based on the values ​​of each business data item in the first preset field, to confirm the display order of each field or business data item in the business table.

[0062] Furthermore, sorting configuration information can also indicate sorting rules and conversion methods for transforming sorting operations into sorting query statements. For example, the indicated sorting rule could be a default sorting rule, a sorting rule based on the first letter of the corresponding fields, etc. The client can display the sorting rules indicated by the sorting configuration information, and the user can select a target sorting rule from the displayed rules. The client can then send the target sorting rule and a first preset field to the server. Alternatively, the user can choose not to select a sorting rule, and the server can use the default sorting rule as the target sorting rule. The conversion method for transforming sorting operations into sorting query statements can be determined based on the database storing the wide table. For example, assuming the wide table is stored in Elasticsearch (ES), the sorting configuration information can indicate how to transform sorting operations into sorting query statements suitable for ES.

[0063] In one embodiment, for S214 and S215, such as Figure 3As shown, the server can retrieve the target business data corresponding to the first preset field from the wide table in Elasticsearch through data search and display in the data service, obtaining the list operation result. Then, it can retrieve the list header configuration for the first preset field from the list configuration. Next, the list header configuration and the list operation result can be assembled to obtain the operation result for the first preset field, i.e., the first list data. This first list data can be the result after filtering or sorting the first preset field in the business table. Finally, the server can send the first list data to the client, which can use the first list data to update the table data in the business table to display the first list data. The client can be a Customer Relationship Management (CRM) based client; the server can include a CRM front-end service, which can be used to receive editing operations on the first preset field.

[0064] In one embodiment, the first preset field may be a system-preset field or a user-defined field.

[0065] For user-defined fields, users can set the field's name, type (text, number, date, single / multiple selection, etc.), and whether it is required, based on the client. They can also set the display, filtering, and sorting settings for the field, and the server can automatically synchronize various configuration information accordingly.

[0066] By editing the first preset field in the business table, the filtering and sorting functions of all fields in the business table can be realized, enabling the customization of fields and sorting display capabilities of the business table.

[0067] Furthermore, in one embodiment, data modification operations can also be performed on the business data in the business table.

[0068] The steps for modifying data in a business table mainly involve adding or deleting fields, creating the business table, and modifying the data in the table below. The following example will describe these steps.

[0069] In one embodiment, changes to the business table can be implemented through the following steps:

[0070] S221: In response to a data change operation on a business table sent by the client, update the storage unit storing the original business data according to the data change operation, and generate a change record;

[0071] S222: In response to the detection of a change record, update the source table configuration information and the wide table configuration information according to the data change operation;

[0072] S223: Send the change record to the message queue;

[0073] S224: Consume message queues, update source tables according to source table configuration information based on change records in message queues, and update wide tables according to wide table configuration information and source tables.

[0074] S225: Determine the business table after data change operations based on the wide table.

[0075] The aforementioned source table can be a source table containing the fields involved in the business table, and the aforementioned wide table can be a wide table associated with the source table, that is, a wide table obtained by processing data from the source table. Correspondingly, the source table configuration information can be source table configuration information containing the fields involved in the business table, and the wide table configuration information can be wide table configuration information associated with the source table configuration information, that is, the corresponding wide table is associated with the source table corresponding to the source table configuration information.

[0076] It should be noted that data modification operations on business tables can include two types: data modification operations on existing business tables (e.g., adding or deleting a second preset field in a business table, or modifying business data under a third preset field in a business table) and data modification operations on non-existent business tables (e.g., creating a business table). Accordingly, data modification operations include any of the following: adding or deleting a second preset field, creating a list, or modifying business data under a third preset field. Both the second and third preset fields can be user-defined fields or system-preset fields; this application does not impose any restrictions on this.

[0077] The following diagram will provide an overall overview of the above steps. Specifically, as follows... Figure 3As shown, the server can first obtain data change operations for the business table, such as changes to the fourth preset field in the business table. If the fourth preset field is a user-defined field, the database can be modified according to the changes made by the user-defined field in the data service and the management based on the changes made by the data change operation. If the fourth preset field is a regular field, i.e., a system-preset field, the database can be modified according to the changes made by the data change operation through other services. The database is the storage unit for storing the original business data. Simultaneously, the server can generate change records based on changes to the database, such as binlog changes. Next, the server can listen for change records using Canal and update the source table configuration information and wide table configuration information according to the data change operations. Then, the server can send the change records to a message queue, such as Kafka. The server can then consume the message queue to synchronize and process data in the source table and wide table based on the change records in the message queue (e.g., changing the index data of the source table and wide table in Elasticsearch). Specifically, the source table can be updated according to the source table configuration information, and the wide table can be updated according to the wide table configuration information and the source table. Finally, the server can determine the business table after data modification operations based on the wide table.

[0078] The following section describes each of the above steps in detail.

[0079] In one embodiment, for S221, if the data change operation is to add or delete a second preset field in the business table, the corresponding update storage unit for storing the original business data can be to add or delete the second preset field and its corresponding original business data in the storage unit; if the data change operation is to change the business data under the third preset field in the business table, the corresponding update storage unit for storing the original business data can be to change the original business data under the third preset field in the storage unit; if the data change operation is to create a business table, the corresponding update storage unit for storing the original business data can be to add the original business data under each field of the business table in the storage unit.

[0080] In one embodiment, for S222, the update of the source table configuration information can be triggered first, and then the update of the wide table configuration information can be triggered after the source table configuration information is updated.

[0081] Similar to the content corresponding to S221 above, if the data change operation is to add or delete the second preset field in the business table, then the corresponding update of the source table configuration information or wide table configuration information can be to add or delete the information corresponding to the second preset field in the source table configuration information or wide table configuration information; if the data change operation is to change the business data under the third preset field in the business table, then the corresponding update of the source table configuration information or wide table configuration information can be to change the information corresponding to the original business data under the third preset field in the source table configuration information or wide table configuration information (if the source table configuration information or wide table configuration information does not involve the records of the original business data under the third preset field, then it does not need to be changed); if the data change operation is to create a business table, then the corresponding update of the source table configuration information or wide table configuration information can be to create the source table configuration information or wide table configuration information corresponding to the business table (specifically, it can be to first determine each field of the business table design; then, determine the original business data to be obtained and its data processing rules according to each field; finally, generate the source table configuration information or wide table configuration information according to the original business data to be obtained and its data processing rules).

[0082] Furthermore, before updating the source table configuration information and wide table configuration information according to the data change operation, the server can first validate the source table configuration information and wide table configuration information. Specifically, it can validate whether there are duplicate fields or duplicate field types in the source table configuration information; and it can validate the main table, relationships, fields, field types, and functions involved in the wide table configuration information. For example, if the data change operation is to add a second preset field or create a business table, the server can first validate whether the source table configuration information or wide table configuration information involves fields, field types, or functions that are duplicated with the second preset field or various fields in the business table. If duplicate fields, field types, or functions exist, then it is not necessary to create the duplicate field, field type, or function. If the data change operation involves deleting the second preset field or changing the business data under the third preset field in the business table, it is possible to first verify whether the main table or the sub-table involves the second preset field or the third preset field. If so, the business data under the fields related to the second preset field or the third preset field in the main table and the sub-table can be synchronously changed according to the relationship between the main table and the sub-table in the source table configuration information and the wide table configuration information.

[0083] For example, regarding updates to the source table configuration information, such as Figure 4AAs shown, the server can first read all the source table configurations. Then, it determines whether the source table configurations have been created (i.e., updated). If creation is complete, the wide table configuration can be processed, i.e., the wide table configuration information is updated. If creation is incomplete, the source table configuration to be upgraded is obtained, i.e., the source table configuration that needs to be updated; next, this source table configuration can be validated. Afterward, the source table configuration can be updated and saved based on the validation result, so as to create an Elasticsearch source table index based on the source table configuration, i.e., update the corresponding source table.

[0084] For example, regarding updates to wide table configuration information, such as Figure 4B As shown, the server can first read all the wide table configurations. Then, it determines whether the wide table configuration has been created (i.e., updated). If creation is complete, data synchronization can be performed, that is, synchronizing data between the source table and the wide table. If creation is incomplete, the server obtains the wide table configuration to be upgraded, i.e., the wide table configuration that needs to be updated; next, this wide table configuration can be validated. Afterward, the wide table configuration can be updated and saved based on the validation result, so as to create an Elasticsearch wide table index based on the wide table configuration, i.e., update the corresponding wide table.

[0085] In one embodiment, for S223, the server can concurrently send change records to the message queue to improve sending efficiency and avoid message blocking.

[0086] In one embodiment, for S224, when consuming a message queue, the amount of changed data in the change records of the message queue can be monitored, and the concurrency of consuming the message queue can be determined based on the amount of changed data; then, the concurrency of consuming messages in the message queue can be consumed concurrently based on the same data transmission connection.

[0087] Understandably, the server needs to process any list under all clients, which will generate a large number of change messages. For example, sometimes tens of millions of binlog change messages can be generated in a day. Therefore, the above implementation can be used to achieve data synchronization in scenarios with high concurrency and large message volume.

[0088] Specifically, the server can control the concurrency level by using Goroutine lightweight threads and the number of coroutine pools (worker pools), thus determining the amount of data being changed; it can also enable HyperText Transfer Protocol (HTTP) connection reuse in Elasticsearch to establish the same data transmission connection, ensuring that in high-concurrency scenarios, a large number of processes consuming message queues will not be without connections.

[0089] It is understandable that the above data change operation actually requires changes to the source table and the wide table, that is, an upgrade to the list version. This step is performed after the source table configuration information and the wide table configuration information have been changed, and it is based on the changed source table configuration information and the wide table configuration information. The process is explained below with a diagram.

[0090] In one embodiment, for changes to the source table, such as Figure 4C As shown, the server can first read all the source table configurations. Then, it determines whether data synchronization of the source tables based on the source table configuration is complete. If complete, it begins wide table index data synchronization, i.e., modifying the wide table based on the source table and wide table configuration. If not complete, it obtains the source table configuration to be synchronized, i.e., the source table configuration corresponding to the source table that has not been synchronized (i.e., modified), and iterates to obtain all records corresponding to the source table, i.e., all change records, a preset number can be obtained each time, such as 500 records. Next, it can determine whether the change records have been processed. If complete, it begins wide table index data synchronization. If not complete, it retrieves one record and processes it using a coroutine (i.e., the concurrent message queue consumption in the above embodiment). Then, it can organize and assemble the change data (changedata), and use the change data to modify the source table (create, update, or delete). Finally, it can save the ES source table index data, i.e., the source table.

[0091] In one embodiment, regarding changes to wide tables, such as Figure 4C As shown, after saving the modified source table, you can first retrieve the configurations of all associated wide tables for that source table. Then, determine whether the wide table configurations have been processed, i.e., the change is complete. If complete, proceed as follows: Figure 4D The wide table index data synchronization is shown; if incomplete, an incomplete wide table configuration is retrieved and processed by a coroutine, for example, by concurrently consuming a message queue to concurrently process changes to the incomplete wide table configuration and other processing processes. Then, as follows... Figure 4E The data synchronization of a single wide table configuration shown refers to the data synchronization changes made to the wide table based on the wide table configuration.

[0092] Specifically, such as Figure 4D As shown, you can first obtain all the wide table configurations. Then, determine whether the data synchronization changes for the wide tables have been completed based on the wide table configurations. If completed, end the list data; if not completed, obtain the wide table configurations to be synchronized, i.e., the wide table configurations for which data synchronization changes have not yet been performed, and then proceed as follows. Figure 4E The diagram shows the associated data synchronization of a single wide table configuration.

[0093] Specifically, such as Figure 4EAs shown, we can first determine the type of operation on the source table (which should be consistent with the data change type on the business table). If the source table is being updated (for example, modifying a field or business data under a field in the business table), then we can directly retrieve all associated records of the change record. The association relationships between the associated records include ID (identification) associations and conditional associations. In the process of retrieving all associated records of the change record, we can search for a record associated with its ID and / or at least one record associated with its condition based on the change record.

[0094] For example, taking a social networking business as an example, user information is recorded in user table A, and user posting information is recorded in post table B. Specifically, user table A includes: user id field, user account field, user nickname field, and gender field, while post table B includes: post id field, belonging user id field, post title field, post creation time, etc. For example, taking the post id field as an id association, a unique post record can be found through the post id field. As another example, taking the user id field as a condition association, all post records of that user can be found through the user id.

[0095] If the operation involves deleting a field from the source table (e.g., deleting a field from a business table), it checks if the source table is the parent table of the wide table. If so, it deletes the ES wide table index data, i.e., the wide table itself. Continuing with the example above, assuming the parent table of the wide table is post table B and the secondary table is user table A, and the fields of the wide table include: post ID, user account, title, content, and creation time, deleting a user record from user table A will not reduce the number of records in the wide table, and deleting a post record from post table B will also delete one record in the wide table. If not, it retrieves all associated records related to the change record. If the operation involves creating a field from the source table (e.g., adding a field to a business table or creating a business table), it checks if the source table is the parent table of the wide table. If so, it processes the fields to be added according to the data processing rules specified in the wide table configuration, obtaining the change data `allchangedata`, and saves it to the ES wide table index data. If not, it retrieves all associated records related to the change record. After retrieving all associated records related to the change record, all associated records can be iterated through. If the associated record processing is complete, the execution ends; otherwise, an incomplete associated record is retrieved and processed by a coroutine. Next, the fields requiring modification are processed according to the data processing rules specified in the wide table configuration, resulting in the modified data `allchangedata`, which is then saved to the ES wide table index data.

[0096] In one embodiment, for S225, the server can determine the business table after the data change operation based on the wide table and list configuration information. The corresponding content is similar to the process corresponding to the above-mentioned editing operation, and this application will not elaborate on it.

[0097] In one embodiment, when creating a business table, the server can determine the corresponding list configuration information based on the user's configuration of the business table, determine the list key (list_key) corresponding to the business table, save it in the list configuration information, and send it to the client. Later, when the client needs to display the business table, it can transmit the list_key to the server through an interface, allowing the server to retrieve the corresponding list configuration information based on the list_key and determine the business table accordingly.

[0098] Understandably, since the front-end interfaces are the same as the back-end interfaces, meaning the interfaces for transmitting list_keys or business tables between the client and server are unified and relatively fixed, and the integration cost is low, it is only necessary to configure different list_keys and list configuration information to realize the process of creating new business tables, thereby improving development efficiency and convenience and reducing development costs.

[0099] Figure 5 This is a schematic diagram of a list data processing device 500 provided in an embodiment of this application.

[0100] like Figure 5 As shown, the list data processing device 500 includes: a first acquisition module 501, a first assembly module 502, a second acquisition module 503, a second assembly module 504, a data return module 505, an update generation module 506, a change generation module 507, a listening and sending module 508, a consumption and update module 509, a list determination module 510, a first verification module 511, and a second verification module 512, wherein:

[0101] In one embodiment, the first acquisition module 501 is used to acquire source table configuration information and wide table configuration information. The source table configuration information indicates the data mapping relationship between the source table and the original business data, and the wide table configuration information indicates the data processing rules for the source table. The first assembly module 502 is used to determine the source table based on the original business data according to the source table configuration information and the wide table configuration information, and assemble the business data of preset fields in the source table to obtain the wide table. The second acquisition module 503 is used to respond to the client's edit operation on the first preset field in the business table, acquire the target business data corresponding to the first preset field from the wide table, and obtain the list operation result. The second assembly module 504 is used to assemble the list operation result according to the list header configuration of the first preset field to obtain the first list data. The data return module 505 is used to return the first list data to the client to update the table data in the business table.

[0102] Specifically, the first preset field originates from the field to be filtered indicated by the filtering configuration information; the second acquisition module 503 is specifically used for: in response to the filtering operation on the first preset field, acquiring the filtering configuration information, which also indicates that the filtering operation on the first preset field be converted into a corresponding query statement; constructing a filtering query statement for the wide table based on the filtering configuration information and the first preset field; filtering the target business data corresponding to the first preset field in the wide table based on the filtering query statement to obtain the list operation result.

[0103] Specifically, the first preset field is derived from the field to be sorted indicated by the sorting configuration information; the second acquisition module 503 is specifically used for: responding to the sorting operation for the first preset field, constructing a sorting query statement for the wide table based on the first preset field and the sorting operation for the first preset field; sorting the target business data corresponding to the first preset field in the wide table according to the sorting query statement to obtain the list operation result.

[0104] Specifically, the update generation module 506 is used to respond to data change operations on business tables sent by the client, update the storage unit storing the original business data according to the data change operation, and generate a change record; the change generation module 507 is used to respond to the detected change record, update the source table configuration information and wide table configuration information according to the data change operation; the listening and sending module 508 is used to send the change record to the message queue; the consumption update module 509 is used to consume the message queue, update the source table according to the source table configuration information based on the change record in the message queue, and update the wide table according to the wide table configuration information and the source table; the list determination module 510 is used to determine the business table after the data change operation based on the wide table.

[0105] Specifically, data change operations include any of the following: adding or deleting a second preset field, creating a list, or changing business data under a third preset field.

[0106] Specifically, the consumption update module 509 is used to monitor the amount of changed data in the message queue, determine the concurrency of consuming the message queue based on the amount of changed data, and consume the same number of messages concurrently in the message queue based on the same data transmission connection.

[0107] Specifically, the first verification module 511 is used to verify whether there are duplicate fields or duplicate field types in the source table configuration information; the second verification module 512 is used to verify the main table, relationships, fields, field types and functions involved in the wide table configuration information.

[0108] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 5 The apparatus 500 shown can execute the above-described method embodiments, and the aforementioned and other operations and / or functions of each module in the apparatus 500 are respectively for implementing the corresponding processes in the above-described methods. For the sake of brevity, they will not be described in detail here.

[0109] The apparatus 500 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0110] Figure 6 This is a schematic block diagram of an electronic device 600 provided in an embodiment of this application.

[0111] like Figure 6 As shown, the electronic device 600 may include:

[0112] The system includes a memory 610 and a processor 620. The memory 610 stores computer programs and transfers the program code to the processor 620. In other words, the processor 620 can retrieve and run the computer program from the memory 610 to implement the methods described in the embodiments of this application.

[0113] For example, the processor 620 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0114] In some embodiments of this application, the processor 620 may include, but is not limited to:

[0115] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0116] In some embodiments of this application, the memory 610 includes, but is not limited to:

[0117] Volatile memory and / or non-volatile memory. Non-volatile memory can be Read-Only Memory (ROM), Programmable ROM (PROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or Flash memory. Volatile memory can be Random Access Memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Memory Bus RAM (DR RAM).

[0118] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0119] like Figure 6 As shown, the electronic device may also include:

[0120] Transceiver 630, which can be connected to processor 620 or memory 610.

[0121] The processor 620 can control the transceiver 630 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

[0122] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0123] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0124] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, the computer can perform all or part of the corresponding processes in the methods of the embodiments of this application, producing the functions achievable by the methods of the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0125] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0127] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0128] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing list data, characterized in that, include: Obtain source table configuration information and wide table configuration information. The source table configuration information indicates the data mapping relationship between the source table and the original business data, and the wide table configuration information indicates the data processing rules for the source table. Based on the source table configuration information and the wide table configuration information, the source table is determined based on the original business data, and the business data of the preset fields in the source table are assembled to obtain the wide table; In response to an edit operation sent by the client targeting a first preset field in the business table, the target business data corresponding to the first preset field is retrieved from the wide table to obtain the list operation result; Based on the list header configuration of the first preset field, the list operation results are assembled to obtain the first list data; The first list data is returned to the client to update the table data in the business table.

2. The method according to claim 1, characterized in that, The first preset field is derived from the field to be filtered indicated by the filtering configuration information; The response to the client's edit operation on a first preset field in the business table involves retrieving the target business data corresponding to the first preset field from the wide table to obtain the list operation result, including: In response to a filtering operation on the first preset field, obtain the filtering configuration information; Based on the filtering configuration information and the first preset field, construct a filtering query statement for the wide table; The target business data corresponding to the first preset field in the wide table is filtered according to the filtering query statement to obtain the list operation result.

3. The method according to claim 1, characterized in that, The first preset field is derived from the field to be sorted indicated by the sorting configuration information; The response to the client's edit operation on a first preset field in the business table involves retrieving the target business data corresponding to the first preset field from the wide table to obtain the list operation result, including: In response to the sorting operation on the first preset field, a sorting query statement for the wide table is constructed based on the first preset field and the sorting operation on the first preset field. The target business data corresponding to the first preset field in the wide table is sorted according to the sorting query statement to obtain the list operation result.

4. The method according to any one of claims 1-3, characterized in that, Also includes: In response to the data change operation sent by the client for the business table, the storage unit storing the original business data is updated according to the data change operation, and a change record is generated; In response to the detection of the change record, update the source table configuration information and the wide table configuration information according to the data change operation; Send the change record to the message queue; Consume the message queue, update the source table according to the source table configuration information based on the change records in the message queue, and update the wide table according to the wide table configuration information and the source table. The business table after the data change operation is determined based on the wide table.

5. The method according to claim 4, characterized in that, The data change operation includes any of the following: adding or deleting a second preset field, creating a list, or changing business data under a third preset field.

6. The method according to claim 4, characterized in that, The consumption of the message queue includes: Monitor the amount of changed data in the message queue and determine the concurrency of consuming the message queue based on the amount of changed data. The concurrent consumption of the message queue is based on the same data transmission connection.

7. The method according to claim 4, characterized in that, Before updating the source table configuration information and the wide table configuration information according to the data change operation, the method further includes: Verify whether there are duplicate fields or duplicate field types in the source table configuration information; The main table, relationships, fields, field types, and functions involved in the wide table configuration information are validated.

8. A list data processing device, characterized in that, include: The first acquisition module is used to acquire source table configuration information and wide table configuration information. The source table configuration information indicates the data mapping relationship between the source table and the original business data, and the wide table configuration information indicates the data processing rules for the source table. The first assembly module is used to determine the source table based on the source table configuration information and the wide table configuration information, and to assemble the business data of preset fields in the source table to obtain the wide table. The second acquisition module is used to respond to the client's edit operation on the first preset field in the business table, acquire the target business data corresponding to the first preset field from the wide table, and obtain the list operation result; The second assembly module is used to assemble the list operation results according to the list header configuration of the first preset field to obtain the first list data; The data return module is used to return the first list data to the client to update the table data in the business table.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images