Data processing method and apparatus, device, and readable storage medium

HK40068125BActive Publication Date: 2026-10-30TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
HK42022057463
Authority / Receiving Office
HK · HK
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-07-28
Publication Date
2026-10-30
Estimated Expiration
2040-10-18

AI Technical Summary

Technical Problem

Existing technologies suffer from slow query speed and low efficiency when performing multi-dimensional cross-queries during user profile data indexing, especially when dealing with complex SQL logic and large amounts of data table queries.

Method used

By constructing a metadata management table and generating an index table, and using inverted index technology to map and merge data features, efficient indexing of the target data table can be achieved.

Benefits of technology

It improves the efficiency of cross-referencing multi-dimensional data, reduces query time, and can return hundreds of millions of query results in seconds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

The application discloses a data processing method and device, equipment and a readable storage medium, and relates to the field of data processing. The method comprises the following steps: obtaining a target data table, wherein the target data table comprises at least two index keywords corresponding to a target object; merging metadata information corresponding to the target data table to obtain a metadata management table; performing feature mapping on data in the target data table according to the metadata management table to obtain data features corresponding to the index keywords in the target data table, generating a data feature table; and merging the data feature table to obtain an index table. The metadata information of the target data table is used to construct the metadata management table, and the index table corresponding to the target data table is generated according to the metadata management table, so that the inverted index of the data is realized through the index table, the efficiency of cross query of different types of data is improved, and the time length required for indexing query of the data is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a data processing method, apparatus, device, and readable storage medium. Background Technology

[0002] In the field of data processing, taking the indexing process of user profile data as an example, user profile data usually corresponds to multiple different data tables, such as: user history browsing data table, user interest profile data table, user basic information table, etc. When indexing user profile data, if the keywords used in at least two indexes are included, there will be cross-queries.

[0003] In related technologies, the original table data is stored in a data warehouse tool, and then relevant analytical queries, such as Structured Query Language (SQL) statements, are executed to query the target profile fields, thereby obtaining the corresponding user profile data.

[0004] However, multi-dimensional intersections often involve complex SQL logic expressions and a large number of data table queries, resulting in long query times and low data indexing efficiency. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and readable storage medium that can improve the efficiency of data indexing. The technical solution is as follows:

[0006] On the one hand, a data processing method is provided, the method comprising:

[0007] Obtain the target data table, which includes at least two index keys corresponding to the target object;

[0008] The metadata information corresponding to the target data table is merged to obtain a metadata management table;

[0009] Based on the metadata management table, feature mapping is performed on the data in the target data table to obtain the data features in the target data table corresponding to the index key, and a data feature table is generated.

[0010] The data feature tables are merged to obtain an index table, which is used to store the target object and the corresponding data features of the target object.

[0011] On the other hand, a data processing apparatus is provided, the apparatus comprising:

[0012] The acquisition module is used to acquire a target data table, wherein the target data table includes at least two index keys corresponding to the target object;

[0013] The merging module is used to merge the metadata information corresponding to the target data table to obtain a metadata management table;

[0014] The mapping module is used to perform feature mapping on the data in the target data table according to the metadata management table, obtain the data features in the target data table corresponding to the index key, and generate a data feature table;

[0015] The merging module is further configured to merge the data feature table to obtain an index table, which is used to store the target object and the corresponding data features of the target object.

[0016] In an optional embodiment, the merging module includes:

[0017] A determining unit is used to determine the basic metadata in the target data table based on the index key, and obtain a basic field information table;

[0018] The generation unit is used to generate other metadata information tables based on the index keywords in the target data table, with the basic field information table as the core, to obtain the metadata management table.

[0019] In an optional embodiment, the generation unit is further configured to generate a data theme table based on the basic field information table; and to link the theme fields in the basic field information table to the data theme table, wherein the data theme table is used to indicate the theme relationship corresponding to the data.

[0020] The generation unit is further configured to generate a metadata information table based on the basic field information table; link the foreign keys of the table fields in the basic field information table to the metadata information table, wherein the metadata information table is used to indicate the metadata information in the target data table;

[0021] The generation unit is further configured to generate a metadata information management table based on the basic field information table; and to link the data type fields in the basic field information table to the metadata information management table via foreign keys, wherein the metadata information management table is used to indicate the data type and value range.

[0022] The determining unit is further configured to determine the metadata management table based on the foreign key link relationship between the basic field information table and other metadata information tables.

[0023] In an optional embodiment, the target data table includes category data corresponding to the index keywords;

[0024] The mapping module is further configured to map the categorized data in the target data table to a value range corresponding to the categorization type based on the metadata management table, thereby obtaining the feature value corresponding to the categorized data as the data feature.

[0025] In an optional embodiment, the target data table includes location service data corresponding to the index keywords;

[0026] The mapping module is further configured to divide the pre-defined map data into n grids, where n is a positive integer, according to the metadata management table; label the n grids using grid identifiers; and map the location service data to the grids to obtain the grid identifiers of the location service data as the data features.

[0027] In an optional embodiment, the merging module is further configured to, in response to the data feature table including a wide table feature, horizontally concatenate the wide table features to obtain a wide table; and in response to the data feature table including a vertical table feature, sequentially write the value range fields of the vertical table feature into the value range field storage unit, and sequentially write the field setting values ​​of the vertical table feature into the field storage unit to obtain a vertical table.

[0028] The merging module is also used to merge the wide table and the vertical table to obtain the index table.

[0029] In an optional embodiment, the merging module is further configured to horizontally concatenate the wide table and the vertical table in the form of features to obtain the index table.

[0030] In an optional embodiment, the apparatus further includes:

[0031] A receiving module is used to receive an index request, wherein the index request includes keyword requirements corresponding to at least two index keywords;

[0032] The acquisition module is further configured to acquire, from the index table, the target object whose data features meet the keyword requirements as an index result;

[0033] The sending module is used to return the index result.

[0034] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement any of the data processing methods described in the embodiments of this application above.

[0035] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the data processing method as described in any of the embodiments of this application above.

[0036] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the data processing methods described in the above embodiments.

[0037] The beneficial effects of the technical solutions provided in this application include at least the following:

[0038] By constructing a metadata management table from the metadata information of the target data table, and generating an index table corresponding to the target data table based on the metadata management table, an inverted index is created for the data through the index table. This improves the efficiency of cross-querying different types of data and reduces the time required for index queries. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of this application;

[0041] Figure 2 This is a schematic diagram of the framework of an inverted index retrieval system provided in an exemplary embodiment of this application;

[0042] Figure 3 This is a flowchart of a data processing method provided in an exemplary embodiment of this application;

[0043] Figure 4 Based on Figure 3 The illustrated embodiment provides a schematic diagram of the metadata management table structure;

[0044] Figure 5 Based on Figure 3 The illustrated embodiment provides a flowchart for aggregating metadata information to obtain a portrait metadata table;

[0045] Figure 6 This is a flowchart of a data processing method provided in another exemplary embodiment of this application;

[0046] Figure 7 Based on Figure 6 The illustrated embodiment provides a schematic diagram of the location service data mapping process;

[0047] Figure 8 This is a flowchart of a data processing method provided in another exemplary embodiment of this application;

[0048] Figure 9 Based on Figure 8 A schematic diagram of the index interface provided in the illustrated embodiment;

[0049] Figure 10 This is a schematic diagram of the overall technical framework provided by an exemplary embodiment of this application;

[0050] Figure 11 This is a structural block diagram of a data processing apparatus provided in an exemplary embodiment of this application;

[0051] Figure 12 This is a structural block diagram of a data processing apparatus provided in another exemplary embodiment of this application;

[0052] Figure 13 This is a structural block diagram of a server provided in an exemplary embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0054] First, a brief introduction to the terms used in the embodiments of this application:

[0055] Data indexing refers to indexing the data corresponding to the index key in a data table. It is divided into forward indexes and inverted indexes. This application embodiment mainly focuses on inverted indexes, which are indexing methods that locate records based on attribute values. Each entry in this type of index table includes an attribute value and the address of each record that has that attribute value. Because the attribute value determines the location of the record, rather than the record itself, it is called an inverted index.

[0056] It is worth noting that the above data can be stored on a physical server or on a cloud server.

[0057] In related technologies, profiling systems are typically constructed in the following ways:

[0058] 1. Store the original table data in a data warehouse tool, and then query the target profile fields by executing relevant query statements. Such systems can support simple queries such as SQL-like queries, but their query cross-dimensional support is only for a few dimensions. Multi-dimensional cross-dimensional support often involves complex SQL logic expression and a large number of table queries, resulting in long query times.

[0059] 2. First, process the data offline to conform to the input format of a key-value (KV) system before inputting it into the KV system. However, the KV system struggles to perform effective cross-referencing with latitude / longitude data, text data, and numerical data.

[0060] In this application embodiment, a retrieval system based on an inverted index is proposed, which constructs a profiling system that supports multimodal cross-queries and can simultaneously process cross-queries of different types of data such as latitude and longitude data, text data, and numerical data.

[0061] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. The backend services of a cloud computing network system require substantial computing and storage resources. For example, in the quality inspection process of dual-recorded video, it is necessary to store the dual-recorded video and perform artificial intelligence (AI) calculations on different key video segments. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0062] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0063] Figure 1 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of this application, such as... Figure 1As shown, the implementation environment includes a terminal 110, a server 120, and a communication network 130.

[0064] The terminal 110 displays a data query interface, which includes multiple index keywords. After the user sets requirements for at least two of the multiple index keywords through the terminal 110, the user sends an index request to the server 120 through the communication network 130. The index request includes the index requirements set for at least two keywords.

[0065] Server 120 includes an inverted index retrieval system 121 corresponding to the data tables. This inverted index retrieval system 121 includes an index table obtained by reorganizing the data tables based on their metadata. After querying the index system for objects that meet the index requirements, server 120 returns the query results to terminal 110 for display via communication network 130.

[0066] The terminal 110 includes at least one of the following: smartphone, tablet, laptop, desktop computer, smart speaker, smart wearable device, etc. The server 120 can be a physical server or a cloud server providing cloud computing services. The server can be a single server or a server cluster or distributed system composed of multiple servers. The terminal 110 and the server 120 can be directly or indirectly connected through wired or wireless communication, which is not limited in this embodiment.

[0067] Based on the above description, the application scenarios of the embodiments of this application are illustrated with examples:

[0068] In information promotion application scenarios, the backend server needs to query users who meet the recommendation requirements to push information. First, the administrator sets targeted user information in the administrator terminal, such as: female, master's degree, and work location within the first region. Based on the above user information, index conditions are generated and an index request is sent to the backend server. The index request includes the above index conditions. The backend server indexes the users who meet the conditions according to the index conditions and feeds back the user information that meets the index conditions to the administrator terminal. Thus, the administrator terminal pushes promotional messages to the user accounts corresponding to the user information.

[0069] It is worth noting that the above examples are based on information promotion scenarios. In some embodiments, the data processing method provided by this application can also be applied to other scenarios that require cross-data queries. This application does not limit the specific application scenarios.

[0070] Indicative Figure 2This is a schematic diagram of the framework of an inverted index retrieval system provided in an exemplary embodiment of this application, as shown below. Figure 2 As shown, taking the application of this system to user profile data processing as an example, the inverted index retrieval system 200 includes: application layer 220, business logic layer 230, data storage layer 240, data aggregation layer 250, and user profile layer 260.

[0071] Among them, the application layer 220 is mainly used to realize user targeting, which refers to targeted content delivery to users based on their historical behavior and basic data; LookAlike expansion is a technology that uses seed users and certain algorithmic evaluation models to find more potentially related similar groups of people; and user insight and feature management are also included.

[0072] The business logic layer 230 is mainly responsible for authentication, that is, verifying business permissions; converting the feature filtering logic determined by the application layer into specific query statements; managing user-uploaded and exported files; and monitoring the status of different indexing tasks, etc.

[0073] Data storage layer 240 is mainly used to store user profile data, user-uploaded data, user-targeted data, LookAlike data, and task-related information.

[0074] The main auxiliary function of the data aggregation layer 250 is to gather and aggregate features stored in different database tables into a single database table, and then store the aggregated database table in the database.

[0075] User profile layer 260 is mainly used to store user profile information. Different profiles are virtually stored in different database tables, and the granularity of updates is different.

[0076] Based on the above description, the data processing method provided in the embodiments of this application will be described in detail. Figure 3 This is a flowchart of a data processing method provided in an exemplary embodiment of this application. The method is illustrated using an example of its application in a server. Figure 3 As shown, the method includes:

[0077] Step 301: Obtain the target data table, which includes at least two index keys corresponding to the target object.

[0078] The methods for obtaining the target data table include: obtaining a data table related to the current system from the server's database as the target data table; or, receiving a data table related to the current system sent by another server as the target data table.

[0079] To illustrate, let's take a user profiling system as an example. The server's database itself stores user profile data tables related to users, and the server's database also stores multiple data tables related to user profile data. Then, the data tables related to user profile data are selected as the target data tables, such as: user basic information table, user behavior data table, user interest profile data table, etc.

[0080] In some embodiments, the data tables described above are stored on different servers, and the server receives the target data tables sent by each server.

[0081] In some embodiments, the server obtains at least two target data tables and then integrates the at least two target data tables.

[0082] The target data table includes at least two keywords corresponding to the target object. For example, if the target object is a user, the keywords corresponding to the user include: name, gender, age, geographical location, historical behavior data, interest tags, etc.

[0083] Step 302: Merge the metadata information corresponding to the target data table to obtain the metadata management table.

[0084] Optionally, the target data table also corresponds to metadata information. Metadata, also known as intermediary data or relay data, is data about data, mainly describing data properties to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0085] Optionally, when merging metadata information, the basic metadata of the target data table is first determined based on the index key to obtain the basic field information table; then, based on the basic field information table, other metadata information tables are generated according to the index key in the target data table to obtain the metadata management table.

[0086] In other words, for the target data table, based on the feature field information, other related metadata information management tables are derived.

[0087] In some embodiments, a data theme table is generated with the basic field information table as the core, and the theme fields in the basic field information table are linked to the data theme table via foreign keys. This data theme table is used to indicate the theme relationships corresponding to the data. A metadata information table is also generated with the basic field information table as the core, and the table fields in the basic field information table are linked to the metadata information table via foreign keys. This metadata information table is used to indicate the metadata information in the target data table. A metadata information management table is generated with the basic field information table as the core, and the data type fields in the basic field information table are linked to the metadata information management table via foreign keys. This metadata information management table is used to indicate the data type and value range. The metadata management table is determined based on the foreign key links between the basic field information table and other metadata information tables.

[0088] As an illustration, the basic information of the target data table includes the following:

[0089] Tag identifier (identity, id), globally unique (tag_id); tag field name (tag_field); Chinese name of tag (field_name); physical storage type of tag (field_physic_type), such as bigint or string storage type; responsible person (responser), used to indicate the person responsible for this metadata information.

[0090] Based on the above basic information, derived attributes are obtained, including:

[0091] Theme ID (theme_id): The theme corresponding to this feature field, which is linked to the theme table via a foreign key;

[0092] Table ID (table_id): The table information corresponding to this feature field, which is linked to the metadata information management table via a foreign key;

[0093] Label logical type (logic_type): This indicates whether the feature field is a category label or a continuous label. In some embodiments, if it is a category label, it is linked to the category value domain metadata information management table via a foreign key.

[0094] This is illustrative; please refer to it. Figure 4 This illustrates a schematic diagram of the structure of a metadata management table provided in an exemplary embodiment of this application, such as... Figure 4 As shown, this metadata management table includes a profile field information table 410 (corresponding to the aforementioned basic field information table). Based on the feature field information in the basic field information table, other related metadata information management tables are derived. For example... Figure 4The user profile classification identifier meaning mapping table 420, profile data table 430, profile task information table 440, profile theme information table 450, and profile theme view 460 derived from profile theme information table 450 are shown.

[0095] The profile field information table 410 includes the following fields: tag id, tag field name, tag name, storage type, logical type, topic id, calculation description, profile person in charge, usage instructions, accuracy rate, validity period, number of users covered, table id, tag type, remarks, validity status, original field, etc.

[0096] The derived user profile classification identifier meaning mapping table 420 includes the following fields: tag id, field, table name, database name, valid status, number of covered users, tag value, and tag translation.

[0097] The derived profile data table 430 includes the following fields: table id, table name, period type, database, task id, time partition field, time partition format, primary key, storage type, start data time, end data time, etc.

[0098] The derived profile task information table 440 includes the following fields: task id, task calculation platform type, task link, time period type, person in charge, update time, task start time - day, task start time - hour, task start time - minute, task start time - week, and status.

[0099] The derived portrait theme information table 450 includes the following fields: theme ID, parent theme ID, theme name, theme level, status, etc.

[0100] The portrait theme view 460, derived from the portrait theme information table 450, includes the following structure: category ID, category name, first-level theme ID, first-level theme name, second-level theme ID, second-level theme name, third-level theme ID, third-level theme name, fourth-level theme ID, fourth-level theme name, etc.

[0101] After aggregating the above metadata information, a user profile metadata table is finally obtained. This table is then applied to the subsequent application layer, business logic layer, and data aggregation layer.

[0102] This is illustrative; please refer to it. Figure 5 The portrait field information table 510, the user portrait classification identifier meaning mapping table 520, the portrait data table 530, the portrait task information table 540, the portrait theme information table 550, and the portrait correction table 560 are aggregated to obtain the portrait metadata table 570. The portrait metadata table 570 is then applied to the application layer 581, the business logic layer 582, and the data aggregation layer 583.

[0103] Step 303: Perform feature mapping on the data in the target data table based on the metadata management table to obtain the data features in the target data table corresponding to the index keywords, and generate a data feature table.

[0104] Optionally, the feature mapping method varies depending on the data type in the target data table. For example, for categorical data, mapping is performed using categorical value ranges.

[0105] Step 304: Merge the data feature tables to obtain the index table.

[0106] In some embodiments, if the data feature table includes wide table features and vertical table features, then the wide table features and vertical table features are merged respectively, and the resulting wide table and vertical table are merged to obtain the index table.

[0107] The index table is used to store the target object and its corresponding data features.

[0108] In summary, the data processing method provided in this application constructs a metadata management table from the metadata information of the target data table, and generates an index table corresponding to the target data table based on the metadata management table. This enables the inverted index of the data through the index table, improving the efficiency of cross-querying different types of data and reducing the time required for index querying the data.

[0109] In one optional embodiment, the feature mapping is different for different types of data. Figure 6 This is a flowchart of a data processing method provided in another exemplary embodiment of this application. The method is illustrated using an example of its application in a server. Figure 6 As shown, the method includes:

[0110] Step 601: Obtain the target data table, which includes at least two index keys corresponding to the target object.

[0111] The methods for obtaining the target data table include: obtaining a data table related to the current system from the server's database as the target data table; or, receiving a data table related to the current system sent by another server as the target data table.

[0112] The target data table includes at least two keywords corresponding to the target object. For example, if the target object is a user, the keywords corresponding to the user include: name, gender, age, geographical location, historical behavior data, interest tags, etc.

[0113] Step 602: Merge the metadata information corresponding to the target data table to obtain the metadata management table.

[0114] Optionally, when merging metadata information, the basic metadata of the target data table is first determined based on the index key to obtain the basic field information table; then, based on the basic field information table, other metadata information tables are generated according to the index key in the target data table to obtain the metadata management table.

[0115] In other words, for the target data table, based on the feature field information, other related metadata information management tables are derived.

[0116] In some embodiments, a data theme table is generated with the basic field information table as the core, and the theme fields in the basic field information table are linked to the data theme table via foreign keys. This data theme table is used to indicate the theme relationships corresponding to the data. A metadata information table is also generated with the basic field information table as the core, and the table fields in the basic field information table are linked to the metadata information table via foreign keys. This metadata information table is used to indicate the metadata information in the target data table. A metadata information management table is generated with the basic field information table as the core, and the data type fields in the basic field information table are linked to the metadata information management table via foreign keys. This metadata information management table is used to indicate the data type and value range. The metadata management table is determined based on the foreign key links between the basic field information table and other metadata information tables.

[0117] Step 603: Perform feature mapping on the data in the target data table based on the metadata management table to obtain the data features in the target data table corresponding to the index keywords, and generate a data feature table.

[0118] Optionally, the feature mapping method may vary depending on the data type in the target data table.

[0119] In some embodiments, if the target data table includes categorized data corresponding to the index keywords, then the categorized data in the target data table is mapped to the value range corresponding to the categorized type according to the metadata management table, and the feature values ​​corresponding to the categorized data are obtained as data features.

[0120] For illustration, the feature of tag_id 10000 is the user's education level, corresponding to a value range of 6: primary school, junior high school, high school, undergraduate, postgraduate, and doctoral. This is then mapped to six numbers from 0 to 5 (denoted as sort_id), and a unique encrypted feature hex_id is generated using the following method:

[0121] hex_id=concat('h',lower(hex(concat(tag_id,'000',sort_id))))

[0122] In some embodiments, the target data table includes location service data corresponding to the index key, i.e., location-based services (LBS) data. Then, according to the metadata management table, the pre-defined map data is divided into n grids, where n is a positive integer. The n grids are labeled with grid identifiers, and the location service data is mapped to the grids to obtain the grid identifiers of the location service data as data features.

[0123] Please refer to Figure 7 This illustrates a mapping process for location service data provided in an exemplary embodiment of this application, such as... Figure 7 As shown, firstly, the latitude and longitude data 710 of the table features are obtained. Then, the latitude and longitude data is mapped to a network to obtain grid data 720, which includes grid IDs. After aggregating the grid data, the data features 730 corresponding to the location service data are finally obtained.

[0124] To illustrate, the map is first divided into 200 million 100m x 100m grids, each grid identified by a unique ID. The method for dividing the 100m x 100m grid is as follows:

[0125] gridy = ceil(longitude × 1000), gridx = ceil(latitude × 1000)

[0126] Where gridy represents the y-axis coordinate of the grid, gridx represents the x-axis coordinate of the grid, and ceil represents the calculation function. Optionally, the coordinate system is a Cartesian coordinate system set for the grid division of the map.

[0127] Grid ID calculation method:

[0128] Grid ID = gridx - (73 × 1000) + (136 - 73) × 1000 × (gridy - 3 × 1000)

[0129] The user's latitude and longitude information is mapped to a grid. For each user, all their latitude and longitude information is aggregated into a grid ID string called grid_id_list, and the grid_id_list is stored in text format. Location retrieval is performed using the grid ID string. In some embodiments, the retrieval speed for 10,000 grids (100 square kilometers) is less than 3 seconds.

[0130] In some embodiments, the above data types include text data, LBS data, and numerical data.

[0131] 1. Text data is often used to store type information, such as user preferences and interest tags. For text data, spaces are used as delimiters for concatenation, and the resulting data is stored as a feature in text format.

[0132] 2. For the mapping method of LBS data, please refer to the mapping process of location service data mentioned above, which will not be repeated here.

[0133] 3. Numerical data is often used to store continuous features, such as age and frequency; or to store numerical categorical features, such as education level and gender. For these types of features, they are stored directly in numerical form.

[0134] Step 604: In response to the inclusion of wide table features in the data feature table, the wide table features are horizontally concatenated to obtain a wide table.

[0135] In some embodiments, the table type is first determined. For wide tables, the same wide table is concatenated in the concatenation method "key1:value1, key2:value2, ..., keyN:valueN", that is, the key-value pairs and horizontal concatenation are connected.

[0136] Step 605: In response to the inclusion of vertical table features in the data feature table, the value range fields of the vertical table features are written sequentially into the value range field storage unit, and the field setting values ​​of the vertical table features are written sequentially into the field storage unit to obtain the vertical table.

[0137] In some embodiments, the table type is first determined. For vertical table features, the result is concatenated as "key:value", where the key contains the corresponding value range field and the value contains the corresponding field setting value.

[0138] Step 606: Merge the wide table and the vertical table to obtain the index table.

[0139] Optionally, the wide table and the vertical table can be horizontally concatenated in the form of features to obtain an index table.

[0140] In summary, the data processing method provided in this application constructs a metadata management table from the metadata information of the target data table, and generates an index table corresponding to the target data table based on the metadata management table. This enables the inverted index of the data through the index table, improving the efficiency of cross-querying different types of data and reducing the time required for index querying the data.

[0141] The method provided in this embodiment can simultaneously process cross-queries of different types of data, such as latitude and longitude data, text data, and numerical data, thereby improving the feedback efficiency and speed in the data query process.

[0142] The method provided in this embodiment realizes a profiling system design method for automatically aggregating and storing data from wide tables and vertical tables, thereby improving data processing efficiency.

[0143] In some embodiments, after the index table is created, when indexing data, the index is performed based on the identifier and mapping value in the index table, and the corresponding result is returned. Figure 8 This is a flowchart of a data processing method provided in another exemplary embodiment of this application. The method is illustrated using an example of its application in a server. Figure 8 As shown, the method includes:

[0144] Step 801: Obtain the target data table, which includes at least two index keys corresponding to the target object.

[0145] The methods for obtaining the target data table include: obtaining a data table related to the current system from the server's database as the target data table; or, receiving a data table related to the current system sent by another server as the target data table.

[0146] Step 802: Merge the metadata information corresponding to the target data table to obtain the metadata management table.

[0147] Optionally, when merging metadata information, the basic metadata of the target data table is first determined based on the index key to obtain the basic field information table; then, based on the basic field information table, other metadata information tables are generated according to the index key in the target data table to obtain the metadata management table.

[0148] In other words, for the target data table, based on the feature field information, other related metadata information management tables are derived.

[0149] In some embodiments, a data theme table is generated with the basic field information table as the core, and the theme fields in the basic field information table are linked to the data theme table via foreign keys. This data theme table is used to indicate the theme relationships corresponding to the data. A metadata information table is also generated with the basic field information table as the core, and the table fields in the basic field information table are linked to the metadata information table via foreign keys. This metadata information table is used to indicate the metadata information in the target data table. A metadata information management table is generated with the basic field information table as the core, and the data type fields in the basic field information table are linked to the metadata information management table via foreign keys. This metadata information management table is used to indicate the data type and value range. The metadata management table is determined based on the foreign key links between the basic field information table and other metadata information tables.

[0150] Step 803: Perform feature mapping on the data in the target data table based on the metadata management table to obtain the data features in the target data table corresponding to the index keywords, and generate a data feature table.

[0151] Optionally, the feature mapping method varies depending on the data type in the target data table. For example, for categorical data, mapping is performed using categorical value ranges.

[0152] Step 804: Merge the data feature tables to obtain the index table.

[0153] In some embodiments, if the data feature table includes wide table features and vertical table features, then the wide table features and vertical table features are merged respectively, and the resulting wide table and vertical table are merged to obtain the index table.

[0154] The index table is used to store the target object and its corresponding data features.

[0155] Step 805: Receive an index request, which includes keyword requirements corresponding to at least two index keywords.

[0156] Optionally, the index request is sent from the terminal to the server. That is, after the user sets the index conditions in the terminal, the user sends an index request to the server. The keywords in the index request are required to correspond to the index conditions set by the user.

[0157] This is illustrative; please refer to it. Figure 9 The terminal display screen shows an index interface 900, which shows the user-defined index conditions 910, including index conditions 911, 912 and 913. Index condition 911 indicates that the work location is Times Square, index condition 912 indicates that the gender is female, and index condition 913 indicates that users with a master's degree are filtered out.

[0158] The terminal sends an index request to the server based on index conditions 911, 912 and 913. The index request includes keyword requirements to indicate the above index conditions.

[0159] Step 806: Obtain the target objects whose data characteristics meet the keyword requirements from the index table as the index results.

[0160] To illustrate, let's query users whose office is located in a specific building and whose age is greater than 18. Assuming hex_id = h1111 is the office location feature and hex_id = h2222 is the age feature, we only need to query h1111 containing the latitude and longitude of the building, and h2222 being greater than 18 to achieve a cross-reference between latitude / longitude and age. At this point, the data from the original data table to the inverted index system is complete. We can then perform cross-references of various data types with just a few clicks on the page.

[0161] Step 807: Return the index results.

[0162] In summary, the data processing method provided in this application constructs a metadata management table from the metadata information of the target data table, and generates an index table corresponding to the target data table based on the metadata management table. This enables the inverted index of the data through the index table, improving the efficiency of cross-querying different types of data and reducing the time required for index querying the data.

[0163] The method provided in this embodiment proposes a fast retrieval engine based on an inverted index, and constructs a profiling system that supports multimodal cross-queries. It can simultaneously process cross-queries of different types of data such as latitude and longitude data, text data, and numerical data, and achieves the return of results for hundreds of millions of profiling queries within seconds.

[0164] The method provided in this embodiment is a profiling system design method that can automatically aggregate and store data in wide tables and vertical tables.

[0165] The method provided in this embodiment solves the difficulties in the field of big data data analysis, such as the difficulty in cross-querying different types of data and the difficulty in quickly performing downstream analysis and returning results in seconds for billions of data points.

[0166] Figure 10 This is a schematic diagram of the overall technical framework provided by an exemplary embodiment of this application, such as... Figure 10 As shown, taking the application of this technical framework to the generation of index files for user profile data as an example, firstly, profile data 1010 is obtained, and profile metadata 1020 of profile data 1010 is determined. A definition file 1031 is generated based on the profile metadata 1020, and a profile data table 1032 is automatically generated based on the profile data 1010 and the profile metadata 1020. An index file 1040 is generated by combining the definition file 1031 and the profile data table 1032, and the index file is deployed to the index system 1050.

[0167] Figure 11 This is a structural block diagram of a data processing apparatus provided in an exemplary embodiment of this application, such as... Figure 11 As shown, the device includes:

[0168] The acquisition module 1110 is used to acquire a target data table, wherein the target data table includes at least two index keys corresponding to the target object;

[0169] The merging module 1120 is used to merge the metadata information corresponding to the target data table to obtain a metadata management table;

[0170] Mapping module 1130 is used to perform feature mapping on the data in the target data table according to the metadata management table, obtain the data features in the target data table corresponding to the index key, and generate a data feature table;

[0171] The merging module 1120 is further configured to merge the data feature table to obtain an index table, which is used to store the target object and the corresponding data features of the target object.

[0172] In an optional embodiment, such as Figure 12 As shown, the merging module 1120 includes:

[0173] The determining unit 1121 is used to determine the basic metadata in the target data table based on the index key, and obtain a basic field information table;

[0174] The generation unit 1122 is used to generate other metadata information tables based on the index keywords in the target data table, with the basic field information table as the core, to obtain the metadata management table.

[0175] In an optional embodiment, the generation unit 1122 is further configured to generate a data theme table with the basic field information table as the core; and link the theme field foreign keys in the basic field information table to the data theme table, wherein the data theme table is used to indicate the theme relationship corresponding to the data;

[0176] The generation unit 1122 is further configured to generate a metadata information table with the basic field information table as the core; link the foreign keys of the table fields in the basic field information table to the metadata information table, wherein the metadata information table is used to indicate the metadata information in the target data table;

[0177] The generation unit 1122 is further configured to generate a metadata information management table with the basic field information table as the core; and to link the data type fields in the basic field information table to the metadata information management table via foreign keys, wherein the metadata information management table is used to indicate the data type and value range;

[0178] The determining unit 1122 is further configured to determine the metadata management table based on the foreign key link relationship between the basic field information table and other metadata information tables.

[0179] In an optional embodiment, the target data table includes category data corresponding to the index keywords;

[0180] The mapping module 1130 is further configured to map the categorized data in the target data table to a value range corresponding to the categorization type according to the metadata management table, and obtain the feature value corresponding to the categorized data as the data feature.

[0181] In an optional embodiment, the target data table includes location service data corresponding to the index keywords;

[0182] The mapping module 1130 is further configured to divide the pre-defined map data into n grids, where n is a positive integer, according to the metadata management table; label the n grids with grid identifiers; and map the location service data to the grids to obtain the grid identifiers of the location service data as the data features.

[0183] In an optional embodiment, the merging module 1120 is further configured to, in response to the data feature table including a wide table feature, horizontally concatenate the wide table features to obtain a wide table; and in response to the data feature table including a vertical table feature, sequentially write the value range fields of the vertical table feature into the value range field storage unit, and sequentially write the field setting values ​​of the vertical table feature into the field storage unit to obtain a vertical table.

[0184] The merging module 1120 is further configured to merge the wide table and the vertical table to obtain the index table.

[0185] In an optional embodiment, the merging module 1120 is further configured to horizontally splice the wide table and the vertical table in the form of features to obtain the index table.

[0186] In an optional embodiment, the apparatus further includes:

[0187] The receiving module 1140 is used to receive an index request, wherein the index request includes keyword requirements corresponding to at least two index keywords;

[0188] The acquisition module 1110 is further configured to acquire, from the index table, the target object whose data features meet the keyword requirements as an index result;

[0189] The sending module 1150 is used to return the index result.

[0190] In summary, the data processing apparatus provided in this application embodiment constructs a metadata management table from the metadata information of the target data table, and generates an index table corresponding to the target data table based on the metadata management table. This enables the inverted index of the data through the index table, improving the efficiency of cross-querying different types of data and reducing the time required for index querying the data.

[0191] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0192] Figure 13 A schematic diagram of the structure of a server provided in an exemplary embodiment of this application is shown. Specifically:

[0193] Server 1300 includes a Central Processing Unit (CPU) 1301, a system memory 1304 including Random Access Memory (RAM) 1302 and Read Only Memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the CPU 1301. Server 1300 also includes a mass storage device 1306 for storing the operating system 1313, application programs 1314, and other program modules 1315.

[0194] Mass storage device 1306 is connected to central processing unit 1301 via a mass storage controller (not shown) connected to system bus 1305. Mass storage device 1306 and its associated computer-readable media provide non-volatile storage for server 1300. That is, mass storage device 1306 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0195] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1304 and mass storage device 1306 described above can be collectively referred to as memory.

[0196] According to various embodiments of this application, server 1300 can also be connected to a remote computer on a network, such as the Internet. That is, server 1300 can be connected to network 1312 via network interface unit 1311 connected to system bus 1305, or it can also use network interface unit 1311 to connect to other types of networks or remote computer systems (not shown).

[0197] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0198] Embodiments of this application also provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the data processing method provided in the above-described method embodiments.

[0199] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the data processing method provided in the above-described method embodiments.

[0200] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the data processing methods described in the above embodiments.

[0201] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0202] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0203] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the target data table, wherein the target data table includes at least two index keys corresponding to the target object; Based on the index key, the basic metadata in the target data table is determined to obtain the basic field information table; Using the aforementioned basic field information table as the core, other metadata information tables are generated based on the index keywords to obtain the metadata management table; Based on the metadata management table, feature mapping is performed on the data in the target data table to obtain the data features in the target data table corresponding to the index key, and a data feature table is generated. The data feature tables are merged to obtain an index table, which is used to store the target object and the corresponding data features of the target object to realize cross-query based on the at least two index keywords.

2. The method according to claim 1, characterized in that, The metadata management table is obtained by generating other metadata information tables based on the index keywords, using the basic field information table as the core, including: A data theme table is generated based on the basic field information table; the theme fields in the basic field information table are linked to the data theme table via foreign keys, and the data theme table is used to indicate the theme relationship corresponding to the data. A metadata information table is generated based on the basic field information table; foreign keys of table fields in the basic field information table are linked to the metadata information table, and the metadata information table is used to indicate the metadata information in the target data table; A metadata information management table is generated based on the basic field information table; the data type fields in the basic field information table are linked to the metadata information management table via foreign keys, and the metadata information management table is used to indicate the data type and value range. The metadata management table is determined based on the foreign key links between the basic field information table and other metadata information tables.

3. The method according to claim 1 or 2, characterized in that, The target data table includes category data corresponding to the index keywords; The step of performing feature mapping on the data in the target data table based on the metadata management table to obtain the data features in the target data table corresponding to the index key includes: The categorized data in the target data table is mapped to the value range corresponding to the categorization type according to the metadata management table, and the feature value corresponding to the categorized data is obtained as the data feature.

4. The method according to claim 1 or 2, characterized in that, The target data table includes location service data corresponding to the index keywords; The step of performing feature mapping on the data in the target data table based on the metadata management table to obtain the data features in the target data table corresponding to the index key includes: The pre-defined map data is divided into n grids based on the metadata management table, where n is a positive integer; Label the n grids using grid identifiers; The location service data is mapped to a grid, and the grid identifier of the location service data is used as the data feature.

5. The method according to claim 1 or 2, characterized in that, The process of merging the data feature tables to obtain an index table includes: In response to the data feature table including wide table features, the wide table features are horizontally concatenated to obtain a wide table; In response to the data feature table including vertical table features, the value range fields in the vertical table features are written sequentially into the value range field storage unit, and the field setting values ​​in the vertical table features are written sequentially into the field storage unit to obtain the vertical table; The wide table and the vertical table are merged to obtain the index table.

6. The method according to claim 5, characterized in that, The step of merging the wide table and the vertical table to obtain the index table includes: The wide table and the vertical table are horizontally joined together in a feature-based manner to obtain the index table.

7. The method according to claim 1 or 2, characterized in that, The method further includes: Receive an index request, the index request including keyword requirements corresponding to at least two index keywords; The target object whose data features match the keyword requirements is retrieved from the index table as the index result; Return the index result.

8. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire a target data table, wherein the target data table includes at least two index keys corresponding to the target object; The merging module is used to determine the basic metadata in the target data table based on the index key to obtain a basic field information table; and to generate other metadata information tables based on the index key using the basic field information table as the core to obtain a metadata management table. The mapping module is used to perform feature mapping on the data in the target data table according to the metadata management table, obtain the data features in the target data table corresponding to the index key, and generate a data feature table; The merging module is further configured to merge the data feature table to obtain an index table, which stores the target object and the corresponding data feature to enable cross-query based on the at least two index keywords.

9. The apparatus according to claim 8, characterized in that, The generation unit is also used to generate a data theme table based on the basic field information table; and to link the theme fields in the basic field information table to the data theme table, wherein the data theme table is used to indicate the theme relationship corresponding to the data. The generation unit is also used to generate a metadata information table with the basic field information table as the core; The table field foreign keys in the basic field information table are linked to the metadata information table, which is used to indicate the metadata information in the target data table; The generation unit is further configured to generate a metadata information management table based on the basic field information table; and to link the data type fields in the basic field information table to the metadata information management table via foreign keys, wherein the metadata information management table is used to indicate the data type and value range. The determining unit is also used to determine the metadata management table based on the foreign key link relationship between the basic field information table and other metadata information tables.

10. The apparatus according to claim 8 or 9, characterized in that, The target data table includes category data corresponding to the index keywords; The mapping module is further configured to map the categorized data in the target data table to a value range corresponding to the categorization type based on the metadata management table, thereby obtaining the feature value corresponding to the categorized data as the data feature.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the data processing method as described in any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1 to 7.

13. A computer program product comprising computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to cause the computer device to perform the data processing method as described in any one of claims 1 to 7.