Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product, and the data processing method comprises the steps: determining a target survival time which needs to be satisfied by a data table for storing to-be-processed data according to the survival time of the to-be-processed data; determining n data tables based on the target survival time; the n data tables have the same survival time which is the same as the target survival time; n is an integer greater than 1; the n data tables have different table name suffixes; generating a target table name suffix which needs to be met by a data table for storing the to-be-processed data; and searching a target data table from the n data tables according to the target table name suffix, and processing the to-be-processed data based on the target data table. According to the embodiment of the invention, the data processing efficiency can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method and device, electronic equipment, storage medium, and program product. Background Art

[0002] Data processing may include data storage and data query. Usually, data tables are used to store data. When data query is required, the data table corresponding to the data is first determined, and then the query is performed in the corresponding data table.

[0003] In most business scenarios, there is a lot of data to be processed and maintained. For the storage of large amounts of data, it is relatively complicated to decide which data table to store the data in and how to store the data in the data table, and the data storage efficiency is low. For the query of large amounts of data, it is relatively complicated to decide which data table to query the data from and how to query the data in the data table, and the data query efficiency is low. How to perform efficient data processing has become a hot topic in research. Summary of the invention

[0004] The present disclosure provides a data processing method and device, an electronic device, a storage medium, and a program product.

[0005] In a first aspect, the present disclosure provides a data processing method, the data processing method comprising:

[0006] Determine, according to the survival time of the data to be processed, a target survival time that a data table used to store the data to be processed must meet;

[0007] Based on the target survival time, n data tables are determined; the n data tables have the same survival time and are the same as the target survival time; n is an integer greater than 1; the n data tables have different table name suffixes;

[0008] Generate a target table name suffix that a data table used to store the data to be processed must satisfy;

[0009] According to the target table name suffix, a target data table is searched from the n data tables, and the to-be-processed data is processed based on the target data table.

[0010] In a second aspect, the present disclosure provides a data processing device, the data processing device comprising:

[0011] A processing unit, configured to determine a target survival time that a data table used to store the data to be processed needs to meet according to the survival time of the data to be processed;

[0012] The processing unit is further used to determine n data tables based on the target survival time; the n data tables have the same survival time and are the same as the target survival time; n is an integer greater than 1; the n data tables have different table name suffixes;

[0013] A generating unit, used for generating a target table name suffix that a data table used to store the data to be processed needs to satisfy;

[0014] A search unit is used to search for a target data table from the n data tables according to the target table name suffix, and process the data to be processed based on the target data table.

[0015] In a third aspect, the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned data processing method.

[0016] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data processing method when executed by a processor.

[0017] In a fifth aspect, the present disclosure provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method.

[0018] The data processing method provided by the embodiment of the present disclosure determines a target survival time that a data table used to store the data to be processed must meet based on the survival time of the data to be processed, determines n data tables based on the target survival time, and then generates a target table name suffix that the data table used to store the data to be processed must meet, searches for a target data table from the n data tables based on the target table name suffix, and processes the data to be processed based on the target data table.

[0019] In the above data processing process, the target data table used for data processing has the same survival time as the data to be processed. When the storage time of the data to be processed in the target data table is equal to the same survival time, the target data table will automatically delete the data to be processed, and there is no need to manually delete the expired data to be processed in the target data table, thereby improving data processing efficiency, reducing the occupation of the target data table by expired data, reducing the occurrence of querying expired data, and improving data processing efficiency. In addition, the selection of the target data table also refers to the target table name suffix generated according to the data to be processed, so that the target data table used for data processing can be accurately found according to the data to be processed, thereby improving data processing efficiency and accuracy.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of a configuration page provided for an embodiment of the present disclosure;

[0024] Figure 3 A flow chart of a data processing method when the data to be processed provided by the embodiment of the present disclosure includes data to be stored;

[0025] Figure 4 A flow chart of a data processing method when the data to be processed provided by the embodiment of the present disclosure includes data to be queried;

[0026] Figure 5 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0027] Figure 6 A block diagram of a data processing device provided in an embodiment of the present disclosure;

[0028] Figure 7 A block diagram of a data processing device provided in an embodiment of the present disclosure;

[0029] Figure 8A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0032] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0033] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0034] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0035] In the technical solution disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution complies with relevant national laws and regulations (for example, the "Information Security Technology Personal Information Security Specification", etc.). For example: corresponding prescribed measures are taken for personal information access control; prescribed restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; when using personal information, clear identity orientation is eliminated to avoid precise positioning of specific individuals.

[0036] In the process of realizing the present invention, the inventors analyzed the related technologies and found that: in the related technologies, when data is stored in a database, an archiving program is usually required. For example, a MySQL database starts to clear when it stores more than 1,000 data items. If the amount of data is large, such as 5,000 data items, an archiving program needs to be written. MySQL is a relational database management system. The archiving program needs to be developed separately, which makes data processing more complicated and the efficiency of data processing is low.

[0037] Therefore, the inventors have found through analysis that the related technology has the problem of low efficiency in data processing.

[0038] To solve the problem of low efficiency in data processing, the embodiments of the present disclosure provide a data processing method and device, an electronic device, a storage medium, and a program product.

[0039] The data processing method according to the embodiment of the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory.

[0040] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure. Figure 1 , the method comprising:

[0041] Step 101 : determining a target survival time that a data table used to store the data to be processed needs to meet according to the survival time of the data to be processed.

[0042] In an exemplary embodiment, step 101 includes: determining the survival time of the data to be processed as the target survival time that the data table used to store the data to be processed needs to meet, that is, the target data table used for data processing has the same survival time as the data to be processed.

[0043] Step 102: determine n data tables based on the target survival time.

[0044] Among them, the n data tables have the same survival time, which is the same as the target survival time; n is an integer greater than 1; and the n data tables have different table name suffixes.

[0045] In the exemplary embodiment, the data table refers to a data table in a distributed database. The distributed database refers to HBase, which is a distributed, column-oriented open source database. HBase's Time To Live (TTL) is a mechanism for specifying the survival time of data, allowing users to set a fixed survival time for data in HBase. After the specified survival time is reached, HBase will automatically delete the data.

[0046] Step 103: Generate a target table name suffix that a data table used to store the data to be processed must satisfy.

[0047] Step 104 , searching the target data table from the n data tables according to the target table name suffix, and processing the data to be processed based on the target data table.

[0048] The data processing method provided by the embodiment of the present disclosure determines a target survival time that a data table used to store the data to be processed must meet based on the survival time of the data to be processed, determines n data tables based on the target survival time, and then generates a target table name suffix that the data table used to store the data to be processed must meet, searches for a target data table from the n data tables based on the target table name suffix, and processes the data to be processed based on the target data table.

[0049] In the above data processing process, the target data table used for data processing has the same survival time as the data to be processed. When the storage time of the data to be processed in the target data table is equal to the same survival time, the target data table will automatically delete the data to be processed, and there is no need to manually delete the expired data to be processed in the target data table, thereby improving data processing efficiency, reducing the occupation of the target data table by expired data, reducing the occurrence of querying expired data, and improving data processing efficiency. In addition, the selection of the target data table also refers to the target table name suffix generated according to the data to be processed, so that the target data table used for data processing can be accurately found according to the data to be processed, thereby improving data processing efficiency and accuracy.

[0050] In some embodiments, the data processing method further includes: determining a data acquisition interface and query conditions; in step 104, processing the data to be processed based on the target data table, including: processing the data to be processed in the target data table based on the query conditions and the interface identifier of the data acquisition interface.

[0051] In some embodiments, the data to be processed includes data to be stored, and the data to be stored is acquired through a data acquisition interface based on query conditions.

[0052] Among them, the data to be stored is obtained based on the query conditions through the data acquisition interface after obtaining authorization and consent.

[0053] In an exemplary embodiment, the data to be stored is a Java object, which is an object-oriented programming language.

[0054] For example, the query condition can be name: Jia + ID number: xxx. Through the data acquisition interface A, based on the query condition (name: Jia + ID number: xxx), the data to be stored is obtained as follows: {name:'Jia', idCard:'xxx', isHit:'1', score:'93.456789'}, where name is the field name, idCard is the field ID number, isHit is whether the field is hit, score is the field score, 1 means hit, and the actual meaning of the data to be stored is name: Jia + ID number: xxx + whether it is hit: yes + score: 93.456789.

[0055] In an exemplary embodiment, the interface identifier of the data acquisition interface is used to uniquely identify the data acquisition interface. The interface identifier of the data acquisition interface may be an interface code or an interface name, as long as it can uniquely identify the data acquisition interface.

[0056] In some embodiments, the target lifetime is determined from the target table name prefix; the target table name prefix refers to the table name prefix that the data table used to store the data to be processed must satisfy; the target table name prefix is ​​determined in the following manner: according to the mapping relationship between the interface identifier and the table name prefix generation strategy, the target table name prefix generation strategy that is adapted to the interface identifier of the data acquisition interface is searched, wherein the target table name prefix generation strategy stores the mapping relationship between the input parameters and the table name prefix; the query condition is used as the input parameter, and based on the mapping relationship stored in the target table name prefix generation strategy, the table name prefix mapped to the query condition is used as the target table name prefix.

[0057] In an exemplary embodiment, the table name prefix generation strategy can be implemented by a script for generating a table name prefix. The script for generating a table name prefix can be an antrl script or other scripts. Antrl refers to an open source syntax analyzer that can automatically generate a syntax tree based on input and display it visually. Using the antrl script to generate a table name prefix is ​​simple and easy to operate.

[0058] For example, if the query condition includes product code, different products have different product codes, and different products may have different corresponding data storage life cycles, then the target table name prefix can be obtained based on the product code in the query condition through the target table name prefix generation strategy. For example, if the product code (busCode) is 5110, the data storage life cycle is 1 year, the target data table life cycle is 1 year, and the target data table name prefix is ​​DATA_CACHE_ONE_YEAR; if the product code is not 5110, the data storage life cycle is 2 years, the target data table life cycle is 2 years, and the target data table name prefix is ​​DATA_CACHE_TWO_YEAR.

[0059] According to the mapping relationship between the interface identifier and the table name prefix generation strategy, the target table name prefix generation strategy adapted to the interface identifier of the data acquisition interface is searched, wherein the target table name prefix generation strategy stores the mapping relationship between the input parameters and the table name prefix, takes the query condition as the input parameter, and based on the mapping relationship stored in the target table name prefix generation strategy, takes the table name prefix mapped to the query condition as the target table name prefix, and can generate the target table name prefix according to different interface identifiers and different query conditions, so that the data to be stored obtained according to different interface identifiers and different query conditions can be set with different life times, and storage is more convenient.

[0060] In some embodiments, according to the mapping relationship between the interface identifier and the table name prefix generation strategy, before searching for the target table name prefix generation strategy adapted by the interface identifier of the data acquisition interface, the data processing method further includes: displaying a configuration page; obtaining the interface identifier and the table name prefix generation strategy input to the configuration page; and establishing a mapping relationship between the interface identifier and the table name prefix generation strategy. Figure 2 As shown, it is a schematic diagram of a configuration page provided in an embodiment of the present disclosure. Figure 2 The interface identifier and table name prefix generation strategy are displayed on the configuration page.

[0061] The configuration page is a visual front-end page that allows users to view and modify the mapping relationship between the interface identifier and the table name prefix generation strategy.

[0062] In an exemplary embodiment, the configuration page may also include properties of custom serialization annotations such as encryption mode, enumeration type, decimal places, rounding mode, etc., and a drop-down menu may be set for each property to select. For example, the configuration page may also include encryption mode, and the drop-down menu may select: md5, sha256, aes, base64.

[0063] In an exemplary embodiment, the configuration page may also include properties of custom deserialization annotations such as decryption method, enumeration type, decimal places, rounding mode, etc., and a drop-down menu is set for each property to select. For example, the configuration page may also include a decryption method, and the drop-down selections are: des, base64.

[0064] In some embodiments, the table name of the data table includes a table name prefix and a table name suffix.

[0065] In an exemplary embodiment, the table name prefix of a data table includes the life span of the data table. The table name prefix of the data table may be the life span of the data table, or the table name prefix of the data table may include a fixed string + the life span of the data table. The disclosed embodiments do not limit this. For example, a data table life span of ONE_YEAR means that the data expires in one year, and a data table life span of ONE_MONTH means that the data expires in one month. For another example, the table name prefix of a data table is DATA_CACHE_ONE_YEAR, including a fixed string DATA_CACHE and a life span of ONE_YEAR, and the table name prefix of a data table is DATA_CACHE_TWO_YEAR, including a fixed string DATA_CACHE and a life span of TWO_YEAR.

[0066] In an exemplary embodiment, the table name of a data table includes a table name prefix and a table name suffix. The table name of a data table may include a table name prefix, a separator, and a table name suffix, and the table name prefix and the table name suffix are separated by a separator. For example, the separator may be "_", and the table name of the data table is "table name prefix_table name suffix". The separator may also be other symbols as long as they can play a role in separation. Using separators can avoid data duplication.

[0067] In some embodiments, in some embodiments, before step 102, the data processing method further includes: creating n data tables, wherein the table name of the data table is suffixed with m, m is an integer, m is greater than or equal to 0, and m is less than n.

[0068] For example, if the product code is 5110, the number of sub-tables n is 5, the table name prefix is ​​DATA_CACHE_ONE_YEAR, and the table name range is DATA_CACHE_ONE_YEAR_0 to DATA_CACHE_ONE_YEAR_4; if the product code is not 5110, the table name prefix is ​​DATA_CACHE_TWO_YEAR, and the table name range is DATA_CACHE_TWO_YEAR_0 to DATA_CACHE_TWO_YEAR_19.

[0069] In some embodiments, before creating n data tables, the data processing method further includes: obtaining the number n of sub-tables.

[0070] Obtaining the number of sub-tables n may be to obtain the data capacity and table capacity of the data acquisition interface, and determine the number of sub-tables n according to the data capacity and table capacity of the data acquisition interface. Obtaining the number of sub-tables n may also be to obtain the number of target data acquisition interfaces within a preset time period, the data capacity and table capacity of each target data acquisition interface, and determine the number of sub-tables n according to the number of target data acquisition interfaces within a preset time period, the data capacity and table capacity of each target data acquisition interface.

[0071] In an exemplary embodiment, the data capacity of the data acquisition interface may refer to the amount of data acquired by the data acquisition interface within a preset time period, for example, the amount of data acquired by the data acquisition interface A within 1 year.

[0072] In an exemplary embodiment, the data storage life cycle of the data acquired by the target data acquisition interface is the same as the data storage life cycle of the data acquired by the data acquisition interface.

[0073] In an exemplary embodiment, the table capacity may include the number of requests per second, the amount of writes per hour, and the amount of reads per hour of each table. For example, the number of requests per second of each table does not exceed 5,000, the amount of writes per hour does not exceed 4G, and the amount of reads per hour does not exceed 4G. If the number of requests per second is greater than 5,000, and / or the amount of writes per hour is greater than 4G, and / or the amount of reads per hour is greater than 4G, then the table needs to be split to ensure that the number of requests per second of each table does not exceed 5,000, the amount of writes per hour is not greater than 4G, and the amount of reads per hour is not greater than 4G.

[0074] Divided into multiple tables, each table is independent of each other and can store more data. If a table has a problem, it will not affect the use of other tables. In addition, the number of tables is controllable. There is no need to create a new table by connecting to a new interface with the same data storage life cycle as the existing interface. There is no need to write an archiving program, which reduces the maintenance and archiving costs of the table and improves the management efficiency of the table.

[0075] In some embodiments, step 103 includes: performing hash modulo operation on the query condition to obtain a target table name suffix that must be satisfied by the data table used to store the data to be processed.

[0076] Specifically, hash modulus refers to taking the remainder of the hash value of the data. Hash modulus is performed on the query condition to obtain the target table name suffix that the data table used to store the data to be processed must satisfy, which is target table name suffix = hash (query condition) % number of sub-tables n.

[0077] In some embodiments, the data to be processed may include data to be stored or data to be queried. The following embodiments respectively describe specific implementations of data processing in the two cases where the data to be processed includes data to be stored and the data to be processed includes data to be queried.

[0078] In some embodiments, the data to be processed includes data to be stored;

[0079] Based on the query condition and the interface identifier of the data acquisition interface, the data to be processed is processed in the target data table, including: according to the query condition, the interface identifier of the data acquisition interface and the storage timestamp, the data to be stored is stored in the target data table.

[0080] In an exemplary embodiment, the storage timestamp refers to the time when the data to be stored is stored in the target data table.

[0081] In some embodiments, the data to be stored is stored in a target data table according to the query conditions, the interface identifier of the data acquisition interface, and the storage timestamp, including: serializing the data to be stored to obtain the string to be stored; calculating the row key to be stored of the string to be stored according to the query conditions, the interface identifier, and the storage timestamp; and storing the string to be stored in the target data table according to the row key to be stored.

[0082] In the exemplary embodiment, the string to be stored is a json string. json (JavaScript Object Notation, JS object notation) is a lightweight data exchange format, also known as a data serialization method, which uses a text format that is completely independent of the programming language to store and represent data.

[0083] Specifically, objects in Java are wrapped in curly braces {}, and the data structure is a key-value pair structure of {key1: value1, key2: value2, ...}. In object-oriented languages, key is the attribute of the object, value is the corresponding value, key names can be represented by integers and strings, and value types can be of any type. JSON is a string representation of Java objects, which uses text to represent the information of a Java object, and is essentially a string.

[0084] For example, the data to be stored is a Java object {name: 'A', idCard: 'xxx', isHit: '1', score: '93.456789'}. The data to be stored includes multiple key-value pairs. The string to be stored obtained by serializing the data to be stored is: '{\"name\":\"A\",\"idCard\":\"xxx\",\"isHit\":\"1\",\"score\":\"93.456789\"}'. The string to be stored is a string.

[0085] In an exemplary embodiment, serializing the data to be stored to obtain a string to be stored can be implemented by Jackson. Jackson is an open-source Java framework for serializing and deserializing JSON.

[0086] In an exemplary embodiment, the row key refers to the rowkey, which is the primary key of the data table.

[0087] In some embodiments, the row key to be stored includes the hash value of the query condition, the interface identifier of the data acquisition interface, and the storage timestamp.

[0088] The row key to be stored includes the hash value of the query condition to facilitate maintaining the high-order hashing principle. The row key to be stored includes the interface identifier of the data acquisition interface, which can accurately distinguish from which data acquisition interface the object to be stored is obtained. The row key to be stored includes the storage timestamp, which is beneficial to maintaining the uniqueness of the row key to be stored. The row key to be stored including the storage timestamp can also support filtering and retrieving the latest one when there are multiple pieces of data later.

[0089] In an exemplary embodiment, the row key to be stored may further include a delimiter. The hash value of the query condition and the interface identifier of the data acquisition interface are separated by the delimiter, and the interface identifier of the data acquisition interface and the storage timestamp are separated by the delimiter. For example, the delimiter can be _, and the row key to be stored is hash(query condition)_interface identifier_storage timestamp. The delimiter can also be other symbols as long as it can play a separating role. Using the delimiter can avoid data duplication.

[0090] In the process of realizing the present invention, the inventors analyzed the related technologies and found that: in the related technologies, when the data acquired through the data acquisition interface is stored in the database, the data is usually stored in the form of a database vertical table. For example, the data acquired by the data acquisition interface A is stored in a vertical table, and the data acquired by the data acquisition interface B is stored in another vertical table. Moreover, since the fields in the data acquired by the data acquisition interface A and the data acquired by the data acquisition interface B are different, the table structure is different, and the table structure is not conducive to expansion, storage and query, and the storage efficiency and query efficiency are low. For another example, the credit data is stored in a vertical table, and 3 vertical tables will be stored. The first vertical table stores basic information, the second vertical table stores loan records, and the third vertical table stores overdue information. There are many vertical tables, and the table structure of each vertical table is different. The table structure is not conducive to expansion, storage and query, and the storage efficiency and query efficiency are low.

[0091] In the disclosed embodiment, the data to be stored is serialized to obtain a string to be stored, and the row key to be stored of the string to be stored is calculated according to the query condition, the interface identifier and the storage timestamp, and the string to be stored is stored in the target data table according to the row key to be stored, so that the table structure of the target data table corresponding to different data to be stored is the same, and the target data table includes only two fields, namely the row key to be stored and the string to be stored, which makes storage more convenient. Moreover, the row key to be stored of the string to be stored is calculated according to the query condition, the interface identifier and the storage timestamp, so that it can be accurately distinguished from which data acquisition interface the data to be stored is obtained, and the storage timestamp is conducive to maintaining the uniqueness of the row key to be stored.

[0092] In some embodiments, after serializing the data to be stored to obtain the string to be stored, the data processing method further includes: calculating the keywords to be stored of the string to be stored according to the query conditions and the interface identifier; and storing the string to be stored in a remote dictionary service according to the keywords to be stored.

[0093] In an exemplary embodiment, the Remote Dictionary Server (Redis) is an open source, network-supported, memory-based, persistent, log-type, Key-Value database written in ANSI C language and providing an application programming interface (API) in multiple languages.

[0094] In an exemplary embodiment, the keywords to be stored include a hash value of the query condition and an interface identifier of the data acquisition interface.

[0095] Specifically, the keyword to be stored may also include a separator, and the hash value of the query condition and the interface identifier of the data acquisition interface are separated by the separator. For example, the separator may be "_", and the keyword to be stored is "hash (query condition)_interface identifier". The separator may also be other symbols as long as they can play a role of separation. Using separators can avoid data duplication.

[0096] Specifically, in the remote dictionary service, the data storage time of the keywords and strings to be stored can be set. For example, if the data storage time is set to 30 seconds, the keywords and strings to be stored will be automatically deleted in the remote dictionary service when the storage time reaches 30 seconds.

[0097] Not only are the strings to be stored stored in the target data table according to the row keys to be stored, but the keywords to be stored and the strings to be stored are also stored in the remote dictionary service, which can alleviate the access pressure of the database and avoid frequent access to the database.

[0098] In some embodiments, the data to be stored is serialized to obtain a string to be stored, including: configuring a serialization annotation for the field to be annotated in the data to be stored; and serializing the data to be stored into a string to be stored through a serializer corresponding to the serialization annotation.

[0099] In some embodiments, the serialization annotation is a custom serialization annotation, and the custom serialization annotation includes at least one of an encryption serialization annotation, an enumeration serialization annotation, and a rounding serialization annotation. The properties of the encryption serialization annotation include encryption method, the properties of the enumeration serialization annotation include enumeration type, and the properties of the rounding serialization annotation include decimal places and rounding mode; the serializer corresponding to the encryption serialization annotation is an encryption serializer, the serializer corresponding to the enumeration serialization annotation is an enumeration serializer, and the serializer corresponding to the rounding serialization annotation is a rounding serializer.

[0100] Specifically, all fields of the data to be stored may need to be annotated, or only some fields may need to be annotated, and the embodiments of the present disclosure do not limit this.

[0101] In the process of realizing the present invention, the inventor analyzed the related technologies and found that the related technologies use database vertical tables to store data, each vertical table has different fields, and different fields need to be displayed to users with different effects. The related technologies need to write complex field processing programs, which is not conducive to maintenance.

[0102] In the disclosed embodiment, serialization is performed through Jackson custom annotations, which is convenient to write, and the field requirements are also set in the Jackson custom annotations. Encryption is implemented based on Jackson custom annotations, and the code is written concisely in the annotation method. At the JSON serialization level, field rounding, enumeration and other format conversions are implemented. The code conversion logic is concise and efficient, which reduces and simplifies the complexity of code writing for field processing and improves the efficiency of field conversion.

[0103] Specifically, the custom Jackson serialization annotations are as follows:

[0104] EncryptionData: encryption serialization annotation; properties of encryption serialization annotation: string type encryptionType, encryptionType refers to the encryption method, and multiple encryption methods can be set, for example, md5, sha256, aes, base64, the default value of the encryption method: md5, that is, if the encryption method is not selected, the default is md5;

[0105] EnumData: enumeration serialization annotation; properties of enumeration serialization annotation: string type enumType, enumType refers to the enumeration type, and the enumeration type is what data is converted into what data in advance according to the business definition;

[0106] RundingData: rounding serialization annotation; properties of rounding serialization annotation: numeric type newScale, newScale refers to the number of decimal places, which can be set to 2, 3, etc., and the default is two, that is, if the number of decimal places is not selected, the default is two; RoundingMode refers to the rounding mode, including multiple rounding modes, which can be selected;

[0107] EncryptionDataSerializer: encryption serializer, used with EncryptionData;

[0108] EnumDataSerializer: Enumeration serializer, used with EnumData;

[0109] RundingDataSerializer: Rounding serializer, used with RundingData.

[0110] For example, there are three fields to be annotated in the data to be stored: name, idCard, and isHit. For example, isHit can mean that product A has 30 million domestic users, and user a is a hit if it is among the 30 million users, and a miss if it is not. Another example is that a risk can be a hit, and a risk-free can be a miss, which can be defined according to the business.

[0111] The custom Jackson serialization annotations are as follows:

[0112] Configure encryption serialization annotation EncryptionData: Default encryption method: md5, support base64, aes, md5, sha256;

[0113] Enum serialization annotation EnumData: hit(1, yes), not_hit(0, no);

[0114] EncryptionDataSerializer: encryption serializer, used with EncryptionData;

[0115] EnumDataSerializer: Enumeration serializer, used with EnumData;

[0116] The name field and the idCard ID card number field are both configured with the encryption serialization annotation EncryptionData. The encryption method for name is configured as md5, and the encryption method for idCard ID card number is configured as base64.

[0117] isHit Whether the hit field configures the enumeration serialization annotation EnumData.

[0118] Enumeration serialization annotation EnumData: hit(1, yes), not_hit(0, no) means that a hit will convert 1 to yes, and a miss not_hit will convert 0 to no.

[0119] In some embodiments, Figure 3 A flowchart of a data processing method when the data to be processed provided by the embodiment of the present disclosure includes data to be stored. Figure 3 , the method comprising:

[0120] Step 301: Obtain the data to be stored based on the query conditions through the data acquisition interface.

[0121] Step 302: According to the mapping relationship between the interface identifier and the table name prefix generation strategy, find the target table name prefix generation strategy adapted by the interface identifier of the data acquisition interface.

[0122] The target table name prefix generation strategy stores a mapping relationship between input parameters and table name prefixes.

[0123] Step 303, configure a custom serialization annotation for the to-be-annotated field in the data to be stored, wherein the custom serialization annotation includes at least one of an encryption serialization annotation, an enumeration serialization annotation, and a rounding serialization annotation, and serialize the data to be stored into a string to be stored through a serializer corresponding to the custom serialization annotation.

[0124] Among them, the serializer corresponding to the encryption serialization annotation is the encryption serializer, the serializer corresponding to the enumeration serialization annotation is the enumeration serializer, and the serializer corresponding to the rounding serialization annotation is the rounding serializer.

[0125] Step 304 , calculating the to-be-stored keyword of the to-be-stored string according to the query condition and the interface identifier, and storing the to-be-stored string into the remote dictionary service according to the to-be-stored keyword.

[0126] Step 305 , taking the query condition as an input parameter, and based on the mapping relationship stored in the target table name prefix generation strategy, taking the table name prefix mapped to the query condition as the target table name prefix.

[0127] Step 306 , based on the target table name prefix, n data tables are determined, a target table name suffix that the data table for storing the data to be processed must satisfy is generated, and the target data table is searched from the n data tables according to the target table name suffix.

[0128] Among them, n is an integer greater than 1, the table name of the data table includes a table name prefix and a table name suffix, the n data tables have the same survival time and are the same as the target survival time that the data table used to store the data to be processed must meet, the target survival time is determined from the target table name prefix, the n data tables have the same table name prefix and are the same as the target table name prefix, and the n data tables have different table name suffixes.

[0129] Step 307, calculating the row key to be stored of the character string to be stored according to the query condition, the interface identifier and the storage timestamp.

[0130] Step 308: Store the character string to be stored in the target data table according to the row key to be stored.

[0131] In some embodiments, the data to be processed includes data to be queried;

[0132] Based on the query conditions and the interface identifier of the data acquisition interface, the data to be processed is processed in the target data table, including: calculating the query keywords of the data to be processed according to the query conditions and the interface identifier; if the query keywords do not exist in the remote dictionary service, the data to be processed is processed in the target data table.

[0133] In some embodiments, the keyword to be queried includes a hash value of the query condition and an interface identifier of the data acquisition interface.

[0134] Specifically, the keyword to be queried may also include a separator, and the hash value of the query condition and the interface identifier of the data acquisition interface are separated by a separator. For example, the separator may be "_", and the keyword to be queried is "hash (query condition)_interface identifier". The separator may also be other symbols as long as they can play a role of separation. Using a separator can avoid data duplication.

[0135] Specifically, the remote dictionary service stores the keywords and strings to be queried, and the data storage time of the keywords and strings to be queried can be set. For example, if the data storage time is set to 30 seconds, the keywords and strings to be queried will be automatically deleted in the remote dictionary service when the storage time reaches 30 seconds.

[0136] The remote dictionary service stores the keywords and strings to be queried, which can relieve the access pressure of the database and avoid frequent access to the database. When it is determined that the keywords to be queried do not exist in the remote dictionary service, the data to be processed is processed in the target data table instead of directly processing the data to be processed in the target data table, which can relieve the access pressure of the database and avoid frequent access to the database.

[0137] In some embodiments, after calculating the queried keyword of the data to be processed according to the query conditions and the interface identifier, the data processing method also includes: when the queried keyword exists in the remote dictionary service, searching for the queried character string corresponding to the queried keyword in the remote dictionary service, deserializing the queried character string, and obtaining the data to be queried.

[0138] When there are keywords to be queried in the remote dictionary service, the string to be queried is directly deserialized to obtain the data to be queried, and the data to be processed is no longer processed in the target data table, which can alleviate the access pressure of the database and avoid frequent access to the database.

[0139] In some embodiments, the data to be processed includes data to be queried;

[0140] Based on the query conditions and the interface identifier of the data acquisition interface, the data to be processed is processed in the target data table, including: according to the query conditions, the interface identifier of the data acquisition interface, the start timestamp to be queried and the end timestamp to be queried, the data to be queried is obtained from the target data table.

[0141] In some embodiments, the data to be queried is obtained from the target data table based on the query conditions, the interface identifier of the data acquisition interface, the start timestamp to be queried, and the end timestamp to be queried, including: calculating the start row key to be queried of the character string to be queried based on the query conditions, the interface identifier, and the start timestamp to be queried; calculating the end row key to be queried of the character string to be queried based on the query conditions, the interface identifier, and the end timestamp to be queried; scanning the target data table based on the start row key to be queried and the end row key to be queried to obtain the character string to be queried; and deserializing the character string to be queried to obtain the data to be queried.

[0142] In some embodiments, the start row key to be queried includes a hash value of the query condition, an interface identifier of the data acquisition interface, and a start timestamp of the query, and the end row key to be queried includes a hash value of the query condition, an interface identifier of the data acquisition interface, and a end timestamp of the query.

[0143] The start row key to be queried and the end row key to be queried include the hash value of the query condition to maintain the high-order hash principle. The start row key to be queried and the end row key to be queried include the interface identifier of the data acquisition interface, which can accurately distinguish which data acquisition interface the object to be stored is obtained from. The start row key to be queried includes the start timestamp to be queried, and the end row key to be queried includes the end timestamp to be queried, which is conducive to maintaining the uniqueness of the start row key to be queried and the end row key to be queried.

[0144] In an exemplary embodiment, the start row key to be queried may also include a separator, the hash value of the query condition and the interface identifier of the data acquisition interface are separated by a separator, and the interface identifier of the data acquisition interface and the start timestamp to be queried are separated by a separator. For example, the separator may be _, and the start row key to be queried is hash (query condition) _ interface identifier _ start timestamp to be queried. The separator may also be other symbols as long as they can play a role of separation. Using separators can avoid data duplication.

[0145] Specifically, the end row key to be queried may also include a separator, the hash value of the query condition and the interface identifier of the data acquisition interface are separated by a separator, and the interface identifier of the data acquisition interface and the end timestamp to be queried are separated by a separator. For example, the separator may be _, and the start row key to be queried is hash (query condition) _ interface identifier _ end timestamp to be queried. The separator may also be other symbols as long as they can play a role of separation. Using separators can avoid data duplication.

[0146] Specifically, you can query all data or part of the data. The query start timestamp can be the minimum stored timestamp, and the query end timestamp can be the maximum stored timestamp. In this way, you can query all data according to the query conditions and interface identifiers. You can also query part of the stored data according to the query start timestamp and query end timestamp. For example, if you want to query the data within 3 months before the current time, the query start timestamp is equal to the current time minus 3 months, and the query end timestamp is equal to the current time.

[0147] In some embodiments, data is scanned on the target data table according to the start row key to be queried and the end row key to be queried to obtain the character string to be queried. The data scan may be performed using the scan method of HBase.

[0148] According to the start row key to be queried and the end row key to be queried, the target data table is scanned to obtain the string to be queried. The string to be queried can be found by searching for the row key corresponding to the time range from the start timestamp to the end timestamp to be queried based on the hash value of the query condition and the interface identifier of the data acquisition interface.

[0149] In an exemplary embodiment, the data to be queried is a Java object, the string to be queried is a JSON string, and deserializing the string to be queried to obtain the data to be queried can be achieved through Jackson, which is a Java open source framework for serializing and deserializing JSON.

[0150] In some embodiments, the string to be queried is deserialized to obtain the data to be queried, including: configuring a deserialization annotation for the field to be annotated in the string to be queried; and deserializing the string to be queried into the data to be queried through a deserializer corresponding to the deserialization annotation.

[0151] In some embodiments, the deserialization annotation is a custom deserialization annotation, wherein the custom deserialization annotation includes at least one of a decryption deserialization annotation, an enumeration deserialization annotation, and a rounding deserialization annotation, the properties of the decryption deserialization annotation include a decryption method, the properties of the enumeration deserialization annotation include an enumeration type, and the properties of the rounding deserialization annotation include a number of decimal places and a rounding mode; the deserializer corresponding to the decryption deserialization annotation is a decryption deserializer, the deserializer corresponding to the enumeration deserialization annotation is an enumeration deserializer, and the deserializer corresponding to the rounding deserialization annotation is a rounding deserializer.

[0152] Specifically, all fields of the query string may need to be annotated, or only some fields may need to be annotated, and the embodiments of the present disclosure do not limit this.

[0153] In the disclosed embodiment, deserialization is performed through Jackson custom annotations, which is convenient to write, and the field requirements are also set in Jackson custom annotations. Decryption is realized based on Jackson custom annotations, and the code is written concisely in the annotation method. At the JSON deserialization level, field rounding, enumeration and other format conversions are realized. The code conversion logic is concise and efficient, which reduces and simplifies the complexity of code writing for field processing and improves the efficiency of field conversion.

[0154] Specifically, the custom Jackson deserialization annotations are as follows:

[0155] DecryptionDesData: decryption and deserialization annotation; Attribute: string type encryptionDesType, encryptionDesType refers to the decryption method, and multiple decryption methods can be set, for example, des, base64, the default value of the decryption method: des, that is, if the decryption method is not selected, the default is des;

[0156] EnumDesData: enumeration deserialization annotation; attribute: string type enumType, enumType refers to the enumeration type, and the enumeration type is what data is converted into what data in advance according to the business definition;

[0157] RundingDesData: rounding deserialization annotation; Attribute: numeric type newScale, newScale refers to the number of decimal places, which can be set to 2, 3, etc., and the default is two, that is, if the number of decimal places is not selected, the default is two; RoundingMode refers to the rounding mode, including multiple rounding modes, which can be selected;

[0158] DecryptionDesDataDeserializer: decryption deserializer, used with DecryptionDesData;

[0159] EnumDesDataDeserializer: enumeration deserializer, used with EnumDesData;

[0160] RundingDataSerializer: rounding deserializer, used with RundingDesData;

[0161] For example, there are three fields to be annotated in the query string: idCard ID number, isHit hit, and score. IsHit hit, for example, can mean that product A has 30 million domestic users, user a is a hit if it is among these 30 million users, and a miss if it is not; for another example, it can mean that a risk is a hit, and a risk is a miss, which can be defined according to the business.

[0162] The custom jackson deserialization annotations are as follows:

[0163] DecryptionDesData: decryption and deserialization annotation; attributes: string type encryptionType, decryption method, default value: des; supports des and base64;

[0164] EnumDesData: enumeration deserialization annotation, hit (yes, 1), not_hit (no, 0);

[0165] RundingDesData: rounding deserialization annotation; properties: numeric type newScale, decimal places, default two; RoundingMode rounding mode;

[0166] DecryptionDesDataDeserializer: decryption deserializer, used with DecryptionDesData;

[0167] EnumDesDataDeserializer: enumeration deserializer, used with EnumDesData;

[0168] RundingDataSerializer: rounding deserializer, used with RundingDesData;

[0169] idCard ID card number configuration decryption deserialization annotation DecryptionDesData, decryption method configuration base64;

[0170] isHit whether the configuration enumeration deserialization annotation EnumDesData is hit;

[0171] score configuration rounding deserialization annotation RundingDesData: decimal places newScale 4 digits; RoundingMode rounding mode: ROUND_HALF_UP: standard rounding, 0-4 are rounded down, 5-9 are carried up.

[0172] EnumDesData: Enumeration deserialization annotation, hit(yes, 1), not_hit(no, 0) means that a hit will be converted to 1, and a miss not_hit will be converted to 0.

[0173] In some embodiments, Figure 4 A flowchart of a data processing method when the data to be processed provided by the embodiment of the present disclosure includes data to be queried. Figure 4 , the method comprising:

[0174] Step 401, obtaining query conditions and an interface identifier of a data acquisition interface.

[0175] Step 402: Look up the target table name prefix generation strategy that is adapted to the interface identifier of the data acquisition interface.

[0176] The target table name prefix generation strategy stores a mapping relationship between input parameters and table name prefixes.

[0177] Step 403: Obtain the keyword to be searched.

[0178] Step 404 , determining whether the to-be-queried keyword exists in the remote dictionary service, if so, executing step 408 , otherwise, executing step 405 .

[0179] Step 405, obtaining the target table name prefix.

[0180] Specifically, the query condition is used as an input parameter, and based on the mapping relationship stored in the target table name prefix generation strategy, the table name prefix mapped to the query condition is used as the target table name prefix.

[0181] Step 406 , based on the target table name prefix, n data tables are determined, a target table name suffix that the data table for storing the data to be processed must satisfy is generated, and the target data table is searched from the n data tables according to the target table name suffix.

[0182] Among them, n is an integer greater than 1, the table name of the data table includes a table name prefix and a table name suffix, the n data tables have the same survival time and are the same as the target survival time that the data table used to store the data to be processed must meet, the target survival time is determined from the target table name prefix, the n data tables have the same table name prefix and are the same as the target table name prefix, and the n data tables have different table name suffixes.

[0183] Step 407, obtaining the start row key to be queried and the end row key to be queried, and performing data scanning on the target data table according to the start row key to be queried and the end row key to be queried to obtain the character string to be queried.

[0184] The start row key to be queried includes the hash value of the query condition, the interface identifier of the data acquisition interface, and the start timestamp of the query, and the end row key to be queried includes the hash value of the query condition, the interface identifier of the data acquisition interface, and the end timestamp of the query.

[0185] Step 408, configure a custom deserialization annotation for the field to be annotated in the string to be queried, wherein the custom deserialization annotation includes at least one of a decryption deserialization annotation, an enumeration deserialization annotation, and a rounding deserialization annotation, and deserialize the string to be queried into the data to be queried through the deserializer corresponding to the custom deserialization annotation.

[0186] In some embodiments, Figure 5 A flowchart of a data processing method provided by an embodiment of the present disclosure. Figure 5 , the method comprising:

[0187] Step 501 , displaying a configuration page, acquiring an interface identifier and a table name prefix generation strategy input into the configuration page, and establishing a mapping relationship between the interface identifier and the table name prefix generation strategy.

[0188] Step 502, according to the mapping relationship between the interface identifier and the table name prefix generation strategy, find the target table name prefix generation strategy that is adapted to the interface identifier of the data acquisition interface; use the query condition as an input parameter, and based on the mapping relationship stored in the target table name prefix generation strategy, use the table name prefix mapped to the query condition as the target table name prefix.

[0189] The target table name prefix generation strategy stores a mapping relationship between input parameters and table name prefixes.

[0190] Step 503 , based on the target table name prefix, n data tables are determined, a target table name suffix that the data table for storing the data to be stored needs to satisfy is generated, and the target data table is searched from the n data tables according to the target table name suffix.

[0191] Among them, n is an integer greater than 1, the table name of the data table includes a table name prefix and a table name suffix, the n data tables have the same survival time and are the same as the target survival time that the data table used to store the data to be processed must meet, the target survival time is determined from the target table name prefix, the n data tables have the same table name prefix and are the same as the target table name prefix, and the n data tables have different table name suffixes.

[0192] Step 504, serialize the data to be stored to obtain the string to be stored; calculate the row key to be stored of the string to be stored according to the query condition, interface identifier and storage timestamp; store the string to be stored in the target data table according to the row key to be stored.

[0193] Step 505, according to the mapping relationship between the interface identifier and the table name prefix generation strategy, find the target table name prefix generation strategy that is adapted to the interface identifier of the data acquisition interface; use the query condition as an input parameter, and based on the mapping relationship stored in the target table name prefix generation strategy, use the table name prefix mapped to the query condition as the target table name prefix.

[0194] The target table name prefix generation strategy stores a mapping relationship between input parameters and table name prefixes.

[0195] Step 506 , based on the target table name prefix, n data tables are determined, a target table name suffix that the data table storing the data to be queried must satisfy is generated, and the target data table is searched from the n data tables according to the target table name suffix.

[0196] Among them, n is an integer greater than 1, the table name of the data table includes a table name prefix and a table name suffix, the n data tables have the same survival time and are the same as the target survival time that the data table used to store the data to be processed must meet, the target survival time is determined from the target table name prefix, the n data tables have the same table name prefix and are the same as the target table name prefix, and the n data tables have different table name suffixes.

[0197] Step 507, obtain the start row key and the end row key to be queried, scan the target data table according to the start row key and the end row key to be queried, obtain the string to be queried, deserialize the string to be queried, and obtain the data to be queried.

[0198] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0199] In addition, the present disclosure also provides a data processing device, an electronic device, a storage medium, and a computer program product, all of which can be used to implement any data processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0200] Figure 6 A block diagram of a data processing device provided in an embodiment of the present disclosure.

[0201] Reference Figure 6 , an embodiment of the present disclosure provides a data processing device, the data processing device comprising:

[0202] The processing unit 601 is used to determine a target survival time that a data table used to store the data to be processed needs to meet according to the survival time of the data to be processed;

[0203] The processing unit 601 is further configured to determine n data tables based on the target survival time; the n data tables have the same survival time and are the same as the target survival time; n is an integer greater than 1; the n data tables have different table name suffixes;

[0204] A generating unit 602 is used to generate a target table name suffix that a data table used to store the data to be processed needs to satisfy;

[0205] The searching unit 603 is configured to search for a target data table from the n data tables according to the target table name suffix, and process the to-be-processed data based on the target data table.

[0206] Each unit in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each unit can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each unit above.

[0207] Figure 7 A block diagram of a data processing device provided in an embodiment of the present disclosure.

[0208] Reference Figure 7 , an embodiment of the present disclosure provides a data processing device, the data processing device comprising:

[0209] The configuration module 701 is used to display a configuration page, obtain an interface identifier and a table name prefix generation strategy input into the configuration page, and establish a mapping relationship between the interface identifier and the table name prefix generation strategy.

[0210] The data writing module 702 is used to find the target table name prefix generation strategy that is adapted to the interface identifier of the data acquisition interface according to the mapping relationship between the interface identifier and the table name prefix generation strategy; take the query condition as the input parameter, and based on the mapping relationship stored in the target table name prefix generation strategy, take the table name prefix mapped to the query condition as the target table name prefix; based on the target table name prefix, determine n data tables, generate the target table name suffix that the data table for storing the data to be stored must satisfy, and according to the target table name suffix, find the target data table from the n data tables; serialize the data to be stored to obtain the string to be stored; calculate the row key to be stored of the string to be stored according to the query condition, the interface identifier and the storage timestamp; store the string to be stored in the target data table according to the row key to be stored.

[0211] The target table name prefix generation strategy stores a mapping relationship between input parameters and table name prefixes.

[0212] Among them, n is an integer greater than 1, the table name of the data table includes a table name prefix and a table name suffix, the n data tables have the same survival time and are the same as the target survival time that the data table used to store the data to be processed must meet, the target survival time is determined from the target table name prefix, the n data tables have the same table name prefix and are the same as the target table name prefix, and the n data tables have different table name suffixes.

[0213] The data query module 703 is used to find the target table name prefix generation strategy that is adapted to the interface identifier of the data acquisition interface according to the mapping relationship between the interface identifier and the table name prefix generation strategy; take the query condition as the input parameter, and based on the mapping relationship stored in the target table name prefix generation strategy, take the table name prefix mapped to the query condition as the target table name prefix; based on the target table name prefix, determine n data tables, generate a target table name suffix that the data table used to store the data to be queried must satisfy, and according to the target table name suffix, search for the target data table from the n data tables; obtain the start row key to be queried and the end row key to be queried, and scan the target data table according to the start row key to be queried and the end row key to be queried to obtain the character string to be queried, and deserialize the character string to be queried to obtain the data to be queried.

[0214] The target table name prefix generation strategy stores a mapping relationship between input parameters and table name prefixes.

[0215] Among them, n is an integer greater than 1, the table name of the data table includes a table name prefix and a table name suffix, the n data tables have the same survival time and are the same as the target survival time that the data table used to store the data to be processed must meet, the target survival time is determined from the target table name prefix, the n data tables have the same table name prefix and are the same as the target table name prefix, and the n data tables have different table name suffixes.

[0216] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0217] Figure 8 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0218] Reference Figure 8 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 801; at least one memory 802, and one or more I / O interfaces 803, connected between the processor 801 and the memory 802; wherein the memory 802 stores one or more computer programs that can be executed by the at least one processor 801, and the one or more computer programs are executed by the at least one processor 801 so that the at least one processor 801 can execute the above-mentioned data processing method.

[0219] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0220] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned data processing method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0221] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method.

[0222] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0223] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0224] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0225] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0226] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0227] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0228] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0229] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0230] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0231] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A data processing method, characterized in that: include: Determine, according to the survival time of the data to be processed, a target survival time that a data table used to store the data to be processed must meet; Based on the target survival time, n data tables are determined; the n data tables have the same survival time and are the same as the target survival time; n is an integer greater than 1; the n data tables have different table name suffixes; Generate a target table name suffix that a data table used to store the data to be processed must satisfy; According to the target table name suffix, a target data table is searched from the n data tables, and the to-be-processed data is processed based on the target data table.

2. The data processing method according to claim 1, characterized in that: The method further comprises: Determine the data acquisition interface and query conditions; The processing of the data to be processed based on the target data table includes: Based on the query condition and the interface identifier of the data acquisition interface, the data to be processed is processed in the target data table.

3. The data processing method according to claim 2, characterized in that: The data to be processed includes data to be stored, and the processing of the data to be processed in the target data table based on the query condition and the interface identifier of the data acquisition interface includes: According to the query condition, the interface identifier of the data acquisition interface and the storage timestamp, the data to be stored is stored in the target data table.

4. The data processing method according to claim 2, characterized in that: The data to be processed includes data to be queried, and the processing of the data to be processed in the target data table based on the query condition and the interface identifier of the data acquisition interface includes: The data to be queried is obtained from the target data table according to the query condition, the interface identifier of the data acquisition interface, the start timestamp to be queried, and the end timestamp to be queried.

5. The data processing method according to claim 3, characterized in that: The storing the data to be stored in the target data table according to the query condition, the interface identifier of the data acquisition interface and the storage timestamp comprises: Serializing the data to be stored to obtain a character string to be stored; Calculate the row key to be stored of the character string to be stored according to the query condition, the interface identifier and the storage timestamp; The character string to be stored is stored in the target data table according to the row key to be stored.

6. The data processing method according to claim 2, characterized in that: The target survival time is determined from the target table name prefix; the target table name prefix refers to the table name prefix that the data table used to store the data to be processed must satisfy; the target table name prefix is ​​determined in the following manner: According to the mapping relationship between the interface identifier and the table name prefix generation strategy, searching for the target table name prefix generation strategy adapted to the interface identifier of the data acquisition interface, wherein the target table name prefix generation strategy stores the mapping relationship between the input parameters and the table name prefix; The query condition is used as an input parameter, and based on the mapping relationship stored in the target table name prefix generation strategy, the table name prefix mapped to the query condition is used as the target table name prefix.

7. The data processing method according to claim 2, characterized in that: The target table name suffix that needs to be satisfied by the generated data table for storing the data to be processed includes: The query condition is hashed and modulo-ed to obtain a target table name suffix that a data table used to store the data to be processed must satisfy.

8. The data processing method according to claim 5, characterized in that: After the data to be stored is serialized to obtain the character string to be stored, the method further includes: Calculating the to-be-stored keyword of the to-be-stored character string according to the query condition and the interface identifier; The character string to be stored is stored in a remote dictionary service according to the keyword to be stored.

9. The data processing method according to claim 5 or 8, characterized in that: The serializing the data to be stored to obtain a character string to be stored includes: Configuring serialization annotations for the fields to be annotated in the data to be stored; The data to be stored is serialized into a string to be stored through the serializer corresponding to the serialization annotation.

10. The data processing method according to claim 4, characterized in that: The querying and obtaining the data to be queried from the target data table according to the query condition, the interface identifier of the data acquisition interface, the start timestamp to be queried, and the end timestamp to be queried includes: Calculate the start row key of the character string to be queried according to the query condition, the interface identifier and the start timestamp to be queried; Calculate the end row key of the character string to be queried according to the query condition, the interface identifier and the end timestamp to be queried; Scan the target data table according to the start row key to be queried and the end row key to be queried to obtain the character string to be queried; The character string to be queried is deserialized to obtain the data to be queried.

11. The data processing method according to claim 2, characterized in that: The data to be processed includes data to be queried, and the processing of the data to be processed in the target data table based on the query condition and the interface identifier of the data acquisition interface includes: Calculating a keyword to be queried for the data to be processed according to the query condition and the interface identifier; If the keyword to be queried does not exist in the remote dictionary service, the data to be processed is processed in the target data table.

12. The data processing method according to claim 10, characterized in that: The deserializing the string to be queried to obtain the data to be queried includes: Configure deserialization annotations for the fields to be annotated in the string to be queried; The string to be queried is deserialized into data to be queried through the deserializer corresponding to the deserialization annotation.

13. A data processing device, characterized in that: include: A processing unit, configured to determine a target survival time that a data table used to store the data to be processed needs to meet according to the survival time of the data to be processed; The processing unit is further used to determine n data tables based on the target survival time; the n data tables have the same survival time and are the same as the target survival time; n is an integer greater than 1; the n data tables have different table name suffixes; A generating unit, used for generating a target table name suffix that a data table used to store the data to be processed needs to satisfy; A search unit is used to search for a target data table from the n data tables according to the target table name suffix, and process the data to be processed based on the target data table.

14. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the data processing method as described in any one of claims 1 to 12.

Images