Data processing method, electronic device, and storage medium

Abstract

The present application provides a data processing method, an electronic device, and a storage medium. The method comprises: on the basis of preset mapping relationship information and a plurality of user-defined preset data processing statements, respectively constructing a plurality of alternative semantic trees corresponding to the preset data processing statements, wherein the preset mapping relationship information is information representing the mapping relationship between the preset data processing statements and execution statements in a database to be used; matching the obtained format of data to be processed with the plurality of alternative semantic trees, and determining a target semantic tree corresponding to the data to be processed; on the basis of the target semantic tree corresponding to the data to be processed, determining a query statement corresponding to the data to be processed; and on the basis of the query statement corresponding to the data to be processed, querying the database to be used to obtain a data query result.

Description

Data processing method, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202310751978.0 filed on June 25, 2023, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present application relates to the field of data processing technology, and in particular to a data processing method, electronic device and storage medium. Background Art

[0004] In the process of statistical analysis of big data, offline database-based query analysis methods are usually used to process large amounts of data. This not only takes a long time (for example, several hours or days), but also fails to reflect real-time changes in the data.

[0005] For example, technical personnel usually need to use technologies such as Flink and Stream to perform statistics on streaming data, which requires technical personnel to be proficient in tools such as Flink. Moreover, for a new statistical business, technical personnel need to use operators in tools such as Flink to redeploy or write related business code based on the needs of the statistical business, which increases labor costs and prolongs the processing time of business data, making it impossible for users to quickly perform data statistics based on real-time changes in business data.

[0006] Summary of the Invention

[0007] The present application provides a data processing method, an electronic device, and a storage medium.

[0008] An embodiment of the present application provides a data processing method, which includes: based on preset mapping relationship information and multiple customized preset data processing statements, respectively constructing multiple alternative semantic trees corresponding to the preset data processing statements, wherein the preset mapping relationship information is information characterizing the mapping relationship between the preset data processing statements and the execution statements in the database to be used; matching the obtained format of the data to be processed with the multiple alternative semantic trees to determine the target semantic tree corresponding to the data to be processed; determining the query statement corresponding to the data to be processed based on the target semantic tree corresponding to the data to be processed; and querying the database to be used based on the query statement corresponding to the data to be processed to obtain a data query result.

[0009] An embodiment of the present application provides a data processing method, which includes: generating data to be processed in a corresponding format in response to a configuration instruction; wherein the configuration instruction includes at least one customized processing rule and multiple data processing objects, and sending the data to be processed to a server.

[0010] An embodiment of the present application provides an electronic device, comprising: one or more processors; a memory on which one or more programs are stored. When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the data processing methods in the embodiments of the present application.

[0011] An embodiment of the present application provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, any one of the data processing methods in the embodiments of the present application is implemented.

[0012] With respect to the above embodiments and other aspects of the present application and their implementation, further description is provided in the accompanying drawings, detailed description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG1 is a flow chart showing a data processing method according to an embodiment of the present application.

[0014] FIG2 shows a schematic diagram of an alternative semantic tree provided in an embodiment of the present application.

[0015] FIG3 shows a schematic diagram of an alternative semantic tree provided in an embodiment of the present application.

[0016] FIG4 is a flow chart showing a data processing method according to an embodiment of the present application.

[0017] FIG5 shows a schematic diagram of a rule interface provided in an embodiment of the present application.

[0018] FIG6 shows a block diagram of the data processing system provided in an embodiment of the present application.

[0019] FIG7 is a flow chart showing a working method of the data processing system provided in an embodiment of the present application.

[0020] FIG8 shows a schematic diagram of a data change filter chain provided in an embodiment of the present application.

[0021] FIG9 shows a block diagram of the composition of a first data processing device provided in an embodiment of the present application.

[0022] FIG10 shows a block diagram of the composition of a second data processing device provided in an embodiment of the present application.

[0023] FIG11 shows a structural diagram of an exemplary hardware architecture of a computing device capable of implementing the data processing method and apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the purpose, technical solutions and advantages of this application more clear, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other in any way.

[0025] In the process of statistical analysis of big data, offline database-based query analysis is often used. Processing large amounts of data not only takes a long time (e.g., hours or days), but also fails to reflect real-time data changes. This offline data statistical method, which relies entirely on databases, can be particularly disadvantageous when the data required for business statistics changes rapidly.

[0026] For example, first, you need to rely entirely on offline data statistics in the database, and then use scheduled tasks to collect statistics on business data again to obtain real-time changes in business data. However, using scheduled tasks to collect data statistics will increase the data processing pressure on the server.

[0027] Technical personnel typically use technologies such as Flink and Stream to perform statistics on streaming data, which requires technical personnel to be proficient in tools such as Flink and Stream. Moreover, to meet new business statistics requirements, technical personnel need to use operators in tools such as Flink and redeploy or write related business code based on the statistical business needs. This increases labor costs, prolongs business data processing time, and prevents users from quickly performing data statistics based on real-time changes in business data.

[0028] In order to solve the above problems, the present application provides a data processing method, electronic device and storage medium, so that users can quickly and accurately count business data, and determine multiple alternative semantic trees based on preset mapping relationship information and at least one customized preset data processing statement. The alternative semantic trees can be based on multiple different hierarchical structures and reflect the logical relationship between different statements. There is no need for R&D personnel to use special processing software to parse the data to be processed, and the data structure can be dynamically changed based on at least one customized preset data processing statement. Furthermore, by matching the obtained format of the data to be processed with multiple alternative semantic trees, the target semantic tree corresponding to the data to be processed is determined to meet the processing requirements of the data to be processed.

[0029] FIG1 is a flow chart illustrating a data processing method according to an embodiment of the present application. The data processing method can be applied to a server, such as a server for processing business data. As shown in FIG1 , the data processing method according to an embodiment of the present application includes but is not limited to the following steps S101 to S104.

[0030] In step S101 , based on the preset mapping relationship information and a plurality of user-defined preset data processing statements, a plurality of candidate semantic trees corresponding to the preset data processing statements are constructed respectively.

[0031] The preset mapping relationship information is information representing the mapping relationship between the preset data processing statement and the execution statement in the database to be used.

[0032] In step S102, the acquired format of the data to be processed is matched with a plurality of candidate semantic trees to determine a target semantic tree corresponding to the data to be processed.

[0033] It should be noted that the alternative semantic tree is a tree with multiple different hierarchical structures, each level corresponds to a data processing logic. By matching the format of the acquired data to be processed with each level in multiple alternative semantic trees, the target semantic tree corresponding to the data to be processed can be obtained. The hierarchical structure in the target semantic tree can meet the analysis requirements of the logical relationship in the data to be processed.

[0034] In step S103, the query statement corresponding to the data to be processed is determined according to the target semantic tree corresponding to the data to be processed.

[0035] By matching each processing logic in the data to be processed with each layer structure in the target semantic tree, a query statement corresponding to the data to be processed is obtained, so that the query statement conforms to the statement definition in the database to be used.

[0036] In step S104, a query is performed on the database to be used based on the query statement corresponding to the data to be processed to obtain a data query result.

[0037] The data query result includes data that matches the query statement corresponding to the data to be processed. For example, if the query statement corresponding to the data to be processed is "device identification = 1", the data query result includes multiple target devices, and the device identification of each target device is 1, but the device type of the target device may be different.

[0038] The above data query results are only examples. Other data query results not described are also within the scope of protection of this application and will not be described in detail here.

[0039] In this embodiment, based on preset mapping relationship information and multiple customized preset data processing statements, multiple alternative semantic trees corresponding to the preset data processing statements are constructed respectively. The alternative semantic trees can reflect the logical relationship between different statements based on multiple different hierarchical structures. There is no need for R&D personnel to use special processing software to parse the data to be processed, and the data structure can be dynamically changed based on at least one customized preset data processing statement. Among them, the preset mapping relationship information is information that characterizes the mapping relationship between the preset data processing statement and the execution statement in the database to be used, which can match the execution statement in the database to be used with the customized data processing statement, flexibly process the data in the database, and improve the data processing efficiency. According to the format of the acquired data to be processed, it is matched with multiple alternative semantic trees to determine the target semantic tree corresponding to the data to be processed to meet the processing requirements of the data to be processed. Further, based on the target semantic tree corresponding to the data to be processed, the query statement corresponding to the data to be processed is determined, so that the query statement corresponding to the data to be processed can be applied to the database to be used, thereby speeding up the query of the data. Based on the query statement corresponding to the data to be processed, the database to be used is queried to obtain a data query result, so that the data query result reflects the changes in the business data.

[0040] In some exemplary embodiments, the preset mapping relationship information includes: first mapping relationship information that matches the execution statement in the database to be used, and / or second mapping relationship information determined based on a configuration file, wherein the second mapping relationship information includes at least one of operation rule difference information, character difference information, and string difference information.

[0041] It should be noted that the execution statement in the database to be used does not necessarily match the format of the preset data processing statement (for example, a customized MySQL-based execution statement). The preset data processing statement needs to be converted into a statement that can be recognized by the database to be used (that is, the execution statement in the database to be used) based on the preset mapping relationship information, so that the preset data processing statement can be applicable to a variety of different databases.

[0042] Among them, the second mapping relationship information can be determined by means of a configuration file, so as to use at least one of the operation rule difference information, character difference information and string difference information included in the second mapping relationship information to convert the preset data processing statement to obtain a statement that can be recognized by the database to be used, so that the database to be used can perform faster data query based on the converted statement.

[0043] For example, the first mapping relationship information is mapping relationship information that does not contain difference information. In other words, the preset data processing statement can be directly converted into an execution statement in the database to be used through the first mapping relationship information, that is, there is a one-to-one mapping relationship between the preset data processing statement and the execution statement in the database to be used, which can map the preset data processing statement more quickly and facilitate querying the data in the database to be used.

[0044] In some exemplary embodiments, in step S101, based on the preset mapping relationship information and multiple customized preset data processing statements, multiple alternative semantic trees corresponding to the preset data processing statements are constructed respectively, including: obtaining multiple preset data processing statements; analyzing each preset data processing statement respectively to determine the preset data processing unit corresponding to each preset data processing statement; and constructing an alternative semantic tree corresponding to each preset data processing statement based on the preset data processing unit corresponding to each preset data processing statement.

[0045] The processing rules include: operation rules and / or connection rules; the data processing objects include at least one of the data type of business data, the data name of business data, and the value range of business data.

[0046] For example, if the preset data processing statement is set to "{location=Xi'an or location like Hai}and productInfo not contain LTE TDD, group by moc", by analyzing the preset data processing statement, four preset data processing units can be obtained, namely "location=Xi'an", "location like Hai", "productInfo not contain LTE TDD", and "group by moc".

[0047] Figure 2 shows a schematic diagram of an alternative semantic tree provided by an embodiment of the present application. As shown in Figure 2, the alternative semantic tree shown in Figure 2 can be constructed through the logical relationship between each preset data processing unit.

[0048] Among them, the logical relationship between the preset data processing unit "location=Xi'an" and the preset data processing unit "location like Hai" is "or" (i.e., an or relationship); the logical relationship between the preset data processing unit "productInfo not contain LTE TDD" and the preset data processing unit "{location=Xi'an or location like Hai}" is "and" (i.e., an and relationship); the preset data processing unit "group by moc" and the preset data processing unit "{location=Xi'an or location like Hai}and productInfo not contain LTE TDD" are in a parallel logical relationship.

[0049] Through the above analysis, the logical relationship between each preset data processing unit can be clarified, so that the constructed alternative semantic tree can display the relationship between each data processing object (for example, "sea", "Xi'an", etc.) and each preset data processing unit in a hierarchical manner, thereby facilitating the subsequent matching of the processed data based on the alternative semantic tree and improving the data processing speed.

[0050] In some exemplary embodiments, a preset data processing unit includes at least one processing rule and multiple data processing objects. Analyzing each preset data processing statement separately to determine the preset data processing unit corresponding to each preset data processing statement includes: segmenting the multiple data processing objects according to the at least one processing rule corresponding to each preset data processing statement to obtain multiple data objects to be matched; and matching the at least one processing rule with the multiple data objects to be matched to generate a preset data processing unit corresponding to each preset data processing statement.

[0051] For example, if the default data processing statement is "{nbiId=1 or nbiId=2}and location like a group by mimVersion," where nbiId represents the device identifier, segmenting the default data processing statement yields the following processing rules: "=," "or," "and," "like," "group by," and multiple data processing objects: "nbiId," "1," "2," "location," "a," and "mimVersion."

[0052] Figure 3 shows a schematic diagram of an alternative semantic tree provided by an embodiment of the present application. As shown in Figure 3, by matching the multiple processing rules obtained by the above segmentation with multiple data processing objects, multiple preset data processing units corresponding to the preset data processing statements can be obtained, namely "nbiId=1", "nbiId=2", "location like a", and "group by mimVersion" shown in Figure 3.

[0053] In some exemplary embodiments, the preset data processing unit may include at least one of a data type, a data name, and a data value range. For example, the preset data processing unit may be represented in the form of "keyword-operator (or connector)-value range" to clarify the processing logic for each data processing object.

[0054] The preset data processing unit obtained through the above analysis can analyze different data processing objects in sections, improve the processing efficiency of the preset data processing statements, and speed up the construction of the candidate semantic tree.

[0055] In some exemplary embodiments, based on the preset data processing unit corresponding to each preset data processing statement, an alternative semantic tree corresponding to each preset data processing statement is constructed, including: for the preset data processing unit corresponding to each preset data processing statement, based on the priority of the processing rule, determining the level to which each processing rule belongs step by step; and based on the level to which each processing rule belongs, hierarchically processing multiple preset data processing units corresponding to the processing rules to construct an alternative semantic tree corresponding to the preset data processing statement.

[0056] In some exemplary embodiments, the priority of the processing rules may be determined based on the priority of the processing rules in the database to be used. For example, brackets (e.g., "{}") have the highest priority; when both "and" and "or" connectors are present, "and" has a higher priority than "or", etc.

[0057] By using the priority of the processing rules in the database to be used, the level to which each processing rule belongs is determined step by step. If the priority of the processing rule is higher, the level to which it belongs is higher; conversely, if the priority of the processing rule is lower, the level to which it belongs is lower (such as being at the end of the alternative semantic tree, etc.), thereby enabling hierarchical processing of multiple preset data processing units corresponding to the processing rules, facilitating the hierarchical positioning of each preset data processing unit, and accelerating the construction of the alternative semantic tree.

[0058] In some exemplary embodiments, after querying the to-be-used database based on the query statement corresponding to the to-be-processed data in step S104 to obtain a data query result, the data processing method further includes: updating the target semantic tree according to the data query result.

[0059] In some exemplary embodiments, when the data query result is a result determined based on the changed data, the target semantic tree can be updated correspondingly according to the levels in the target semantic tree corresponding to the changed data.

[0060] For example, if it is determined that the alternative semantic tree shown in Figure 2 is the target semantic tree, when the value of the "location" field in the to-be-processed data changes from "Shanghai" to "Hangzhou", the rule in the target semantic tree that may match it can be obtained as "location like 海". Then, after finding the corresponding data query result, update "location like 海" in the alternative semantic tree shown in Figure 2 to "location like Hangzhou", so as to update the target semantic tree and facilitate subsequent matching of other data.

[0061] In some exemplary embodiments, the to-be-processed data includes historical business data. Querying the to-be-used database based on the query statement corresponding to the to-be-processed data in step S104 to obtain a data query result can be implemented in the following manner:

[0062] Without changing the semantics of the historical query statement corresponding to the historical business data, disassemble the historical query statement corresponding to the historical business data to obtain multiple disassembled statements; reorganize the multiple disassembled statements to obtain multiple to-be-used query statements that match the to-be-used database; respectively use the multiple to-be-used query statements to query the to-be-used database to obtain a data query result.

[0063] For example, the alternative semantic tree can be scanned from the bottom up to obtain all operators supported by the database to be used, and the historical query statements corresponding to the historical business data can be disassembled to obtain multiple disassembled statements (for example, disassembled into "location = Xi'an or location like Hai", "productInfo not contain LTE TDD", and "group by moc", etc.). Then, based on the priority information of the multiple disassembled statements, the multiple disassembled statements are reorganized to obtain query statements to be used that match the database to be used (for example, "{location = Xi'an or location like Hai}group by moc"). Further, "{location = Xi'an or location like Hai}" and "group by moc" are respectively input into the database to be used for query to obtain multiple query results to be matched. The multiple query results to be matched are then comprehensively analyzed to obtain data query results.

[0064] It should be noted that not all disassembled statements can match the execution statements in the database to be used. Therefore, the statements that do not match the execution statements in the database to be used (for example, "productInfo not contain LTE TDD") can be eliminated; then, the post-filtering method is used to filter the data query results obtained based on "{location=Xi'an or location like Hai} group by moc") to obtain more accurate data query results that meet the requirements of "{location=Xi'an or location like Hai} and productInfo not contain LTE TDD group by moc".

[0065] Through the above operations, historical query statements corresponding to historical business data can be analyzed in parallel, and multiple query statements that match the database to be used can be used to query the database to be used, which speeds up the query speed, enables users to quickly and accurately obtain data query results, and improves data processing efficiency.

[0066] In some exemplary embodiments, a plurality of query statements to be used are used to query the database to be used to obtain data query results, including: using a plurality of query statements to be used to query the database to be used to obtain a plurality of data sets to be processed; performing intersection processing on the plurality of data sets to be processed, and / or performing union processing on the plurality of data sets to be processed to obtain data query results.

[0067] By performing intersection processing on multiple data sets to be processed and / or performing union processing on multiple data sets to be processed, it is possible to ensure that the data query results finally obtained meet the query requirements of multiple query statements to be used, making the data query results more accurate.

[0068] In some exemplary embodiments, the data to be processed includes incremental change data in real-time data. The incremental change data in real-time data represents changes in real-time business data relative to historical business data. In other words, if the historical business data has changed, the incremental change data in the real-time data represents the data after the change.

[0069] In some exemplary embodiments, the above-mentioned step S103 of determining the query statement corresponding to the data to be processed based on the target semantic tree corresponding to the data to be processed can be implemented in the following manner: when it is determined that the target semantic tree does not include the query statement corresponding to the incremental change data in the real-time data, parse the incremental change data in the real-time data to obtain business difference information, which includes business names and / or business values; match the preset index information and the business difference information to determine the target query rules; and determine the query statement corresponding to the data to be processed based on the target query rules and multiple query statements to be used.

[0070] Preset index information is obtained by sorting cached service information in reverse order based on the time the service information was acquired. The smaller the difference between the time the service information was acquired and the current time, the higher the index number of the service information is, making it easier to search for updated service data. For example, the preset index information may include preset service names and / or preset service values.

[0071] Based on the chronological order, the business name in the business difference information is matched with the preset business name, and / or the business value in the business difference information is matched with the preset business value in the preset index information, so as to clarify which preset business the business difference information specifically corresponds to, thereby obtaining the target query rule corresponding to the preset business.

[0072] By using target query rules to supplement multiple query statements to be used and determine the query statements corresponding to the data to be processed, even if the target semantic tree does not include query statements corresponding to incremental change data in real-time data, comprehensive query matching can still be performed on the data to be processed, thereby meeting the query requirements of the data to be processed to the greatest extent and improving the accuracy of data queries.

[0073] In some exemplary embodiments, before determining the query statement corresponding to the data to be processed based on the target query rule and multiple query statements to be used, the data processing method also includes: based on multiple preset filtering rules in the preset rule filtering set, verifying the filtering rules that match the business difference information to determine the target query rule.

[0074] The preset rule filtering set includes multiple preset filtering rules, for example, at least one of an equal filtering rule, a like filtering rule, a between filtering rule, a contain filtering rule, and a not contain filtering rule.

[0075] By using multiple preset filtering rules to verify the filtering rules that match the business difference information, the target query rules can be more in line with the query requirements of the data to be processed, the query requirements of the data to be processed can be met to the greatest extent, and the accuracy of data query can be improved.

[0076] In some exemplary embodiments, after querying the database to be used based on the query statement corresponding to the data to be processed in step S104 and obtaining the data query result, the data processing method further includes: determining the update frequency of the database to be used based on the preset interval time and the incremental change data in the real-time data received within the preset interval time; and updating the database to be used using the incremental change data in the real-time data based on the update frequency of the database to be used.

[0077] The preset interval time can be 1 minute, 10 minutes, or any other set time.

[0078] For example, if the preset interval is set to 1 minute and 500 incremental changes are received in real-time data within 1 minute, the database to be used needs to be updated immediately. If the preset interval is set to 10 minutes and only 1 incremental change is received in real-time data within 10 minutes, the database can be updated after 10 minutes.

[0079] The above preset interval durations are only examples and can be set specifically according to actual needs. Other preset interval durations not described are also within the scope of protection of this application and will not be described in detail here.

[0080] By setting a preset interval time, obtaining incremental change data in the real-time data received within the preset time, and adaptively modifying the update frequency of the database to be used, the number of write operations on the database to be used can be reduced, reducing the data processing pressure on the database to be used.

[0081] Figure 4 shows a flow chart of a data processing method according to an embodiment of the present application. This data processing method can be applied to a terminal, such as a mobile phone for processing business data. As shown in Figure 4, the data processing method according to an embodiment of the present application includes but is not limited to the following steps S401 and S402.

[0082] In step S401, in response to a configuration instruction, data to be processed in a corresponding format is generated. The configuration instruction includes at least one self-defined processing rule and multiple data processing objects.

[0083] In some optional embodiments, the configuration instruction further includes: application scenario configuration information and / or device type configuration information, wherein at least one processing rule and multiple data processing objects all match the device type configuration information.

[0084] For example, Figure 5 shows a schematic diagram of a rule interface provided by an embodiment of the present application. As shown in Figure 5, in the rule interface, the operation components included are of three types, namely "Rule Name", "Member Type", and "Rule Definition".

[0085] When users operate on different operation components, they will select different member types accordingly. The fields in the "Rule Definition" can be dynamically adjusted to the data fields of the relevant business. The "And / Or" field can represent the logical "and" (corresponding to the "and" operator in the database) or the logical "or" (corresponding to the "or" operator in the database).

[0086] In the "Operator" field, you can select an operator supported by the current system. In the "Value" column, users can dynamically adjust the data type (e.g., string, number, enumeration, etc.) in the input box based on the "Field" and "Operator" fields.

[0087] The "Rule Name" field can be filled in based on the service data processing requirements. For example, "Custom Scenario" corresponds to "test." The "Member Type" field represents the device type configuration information, that is, the type of the network element device that carries the data to be processed. For example, it can be a "network element" device or a "cell" (i.e., a communication cell).

[0088] In the "rule definition", multiple configuration rules can be added and / or deleted based on the user's usage requirements, and each configuration rule can be selected based on "field", "operator", and "value" respectively. For example, 4 rules are shown in Figure 5. The first rule is "nbiId (network element ID) = 1", the second rule is "location like 海", the third rule is "subType (model type) not contain A", and the fourth rule is "ipAddress (access IP address) between (\"1.1.1.1\" and \"1.2.3.4\") group by moc (network element type)".

[0089] Among them, there is a logical "AND" relationship between the first rule and the second rule; there is a logical "OR" relationship between the third rule and (the results of the first rule and the second rule); there is a logical "AND" relationship between the fourth rule and (the results of the first rule, the second rule, and the third rule).

[0090] Through the above operations, the obtained data to be processed can be characterized as "{nbiId = 1 and location like 海} or subType not contain A and ipAddress between (\"1.1.1.1\", \"1.2.3.4\") group by moc".

[0091] Using a graphical interface to perform corresponding configuration on the configuration information facilitates user use and realizes a way for users to customize the data processing rules. There is no need for R & D personnel to use special processing software to parse the data to be processed, and it can realize dynamic changes to the data structure based on at least one preset data processing statement defined by the user, accelerating the data processing speed.

[0092] In step S402, send the data to be processed to the server.

[0093] When the server receives the data to be processed, it will match the format of the data to be processed with multiple alternative semantic trees to determine the target semantic tree corresponding to the data to be processed; based on the target semantic tree corresponding to the data to be processed, determine the query statement corresponding to the data to be processed; query the database to be used based on the query statement corresponding to the data to be processed to obtain the data query result.

[0094] Among them, the alternative semantic tree is a tree determined based on preset mapping relationship information and at least one preset data processing statement defined by the user. The preset mapping relationship information is information representing the mapping relationship between the preset data processing statement and the execution statement in the database to be used.

[0095] In this embodiment, by responding to the configuration instruction, data to be processed with a corresponding format is generated. The data to be processed is determined based on the user's customization of the data processing rules. There is no need for R&D personnel to use special processing software to parse the data to be processed. The data structure can be dynamically changed based on at least one customized processing rule and preset data processing statements generated by multiple data processing objects, thereby accelerating the processing speed of the data; then, the data to be processed is sent to the server for the server to match with multiple alternative semantic trees according to the format of the data to be processed, and determine the target semantic tree corresponding to the data to be processed to adapt to the processing requirements of the data to be processed; based on the target semantic tree corresponding to the data to be processed, the query statement corresponding to the data to be processed is determined, so that the query statement corresponding to the data to be processed can be applied to the database to be used, thereby accelerating the query speed of the data; based on the query statement corresponding to the data to be processed, the database to be used is queried to obtain data query results, so that the data query results reflect the changes in business data.

[0096] Figure 6 shows a block diagram of a data processing system according to an embodiment of the present invention. As shown in Figure 6 , the data processing system includes but is not limited to the following devices: a front-end device 610 , a business layer server 620 , and a database 630 .

[0097] This data processing system can be applied to big data statistics and business scenarios with high data timeliness requirements, such as data statistics scenarios for smart home cloud services, cloud platform products, traffic management platforms, IoT devices, and e-commerce systems. The above business scenarios are only examples and can be specifically configured according to actual needs. Other business scenarios not described are also within the scope of protection of this application and will not be detailed here.

[0098] The front-end device 610 includes a rule management module 611. The business layer server 620 includes a rule parsing module 621, a model management module 622, an offline data statistics module 623, and a near-real-time data statistics module 624. The database 630 includes a first-type database 631, a second-type database 632, and a third-type database 633.

[0099] For example, the first type of database 631 can be an MS SQL database; the second type of database 632 can be an Elastic Search database; the third type of database 633 can be an Oracle database, that is, different types of databases are used to support data for the business layer server 620 to be suitable for processing a variety of different types of data and improve data processing efficiency.

[0100] The rule management module 611 is configured to obtain customized configuration information based on a human-computer interaction interface (eg, the rule interface shown in FIG. 5 ), and to splice and assemble the obtained configuration information to construct customized preset data processing statements.

[0101] The rule parsing module 621 is configured to analyze the received preset data processing statements, identify key information such as connectors, operators, fields and values ​​therein, generate preset data processing units based on the identified multiple different types of information, and then construct an alternative semantic tree based on multiple preset data processing units.

[0102] The model management module 622 is configured to cache multiple candidate semantic trees and / or update the candidate semantic trees using the data query results output by the offline data statistics module 623 (or the data query results output by the near real-time data statistics module 624).

[0103] The offline data statistics module 623 is configured to map the query statements corresponding to the to-be-processed data into statements that can be recognized by different types of databases according to the second mapping relationship information determined in the configuration file.

[0104] The second mapping relationship information includes difference mapping relationship information between the execution statement in the database to be used and at least one customized preset data processing statement.

[0105] For example, the difference mapping relationship information includes: operators not supported by the database to be used (for example, the not contain operator, etc.), priority difference information between the database to be used and the custom characters (for example, the priority of the custom character "{}" matches the priority of "()" used in the database to be used, and the above difference information needs to be mapped and reflected in the configuration file), and string difference information (for example, double quotes are needed to represent strings in the first type of database, while single quotes are needed to represent strings in the second type of database, and the above difference information needs to be mapped and reflected in the configuration file).

[0106] The offline data statistics module 623 is also used to match the acquired format of the data to be processed with multiple candidate semantic trees to determine the target semantic tree corresponding to the data to be processed, wherein the target semantic tree is a semantic tree obtained by screening from the multiple candidate semantic trees constructed by the rule parsing module 621.

[0107] Furthermore, the offline data statistics module 623 maps the target semantic tree to the execution statements in the database to be used based on the second mapping relationship information, converting the custom characters in the data to be processed into statements supported by the database to be used, thereby generating a query statement corresponding to the data to be processed. When the query statement corresponding to the data to be processed is used to perform an offline data query on the database to be used, data query results matching the data to be processed can be quickly found, thereby improving the scalability of the data processing system.

[0108] It should be noted that for customized characters (or operators, etc.) not supported by the database to be used, the data query results matching the data to be processed need to be post-filtered until the final data query results meet all conditions in the data to be processed.

[0109] Post-filtering can be achieved by setting a preset rule filter set, where the preset rule filter set includes multiple preset filter rules, such as at least one of an equals filter rule, a like filter rule, a between filter rule, a contain filter rule, and a not contain filter rule. Multiple different filter rules are used to further process the data query results, ensuring that the final data query results meet all conditions in the data to be processed, thereby improving data accuracy.

[0110] The near real-time data statistics module 624 is configured to receive the changed data sent by the terminal and send the changed data to the message queue when it is determined that the business data sending has changed; the near real-time data statistics module 624 obtains the changed data by subscribing to the change message of the message queue, uses the cached multiple alternative semantic trees to match the changed data, and based on the multiple preset filtering rules in the preset rule filtering set, performs layer-by-layer judgment on the changed data to determine the rule level corresponding to the changed data in the target semantic tree, and then determines the query statement corresponding to the changed data based on the target semantic tree; based on the query statement corresponding to the changed data, queries the database to be used to obtain data query results.

[0111] After completing the above data query, the target semantic tree can be updated according to the data query result, so that the target semantic tree can be used to process the subsequently received data, thereby improving the data processing efficiency.

[0112] Figure 7 shows a flowchart of the working method of the data processing system provided by an embodiment of the present application. As shown in Figure 7, the working method of the data processing system includes but is not limited to the following steps S701 to S706.

[0113] In step S701, the terminal generates data to be processed and sends the data to be processed to the server.

[0114] The data to be processed may be data generated by the terminal during business processing.

[0115] In step S702 , the server matches the format of the data to be processed with a plurality of candidate semantic trees to determine a target semantic tree corresponding to the data to be processed.

[0116] The server may first use metadata of each network element to perform modeling to form multiple candidate semantic trees. The metadata of the network element may include field name, field type, field length, and field Chinese information.

[0117] For example, metadata information in the network element data includes: English name (eg, location), Chinese name (eg, location), field type (eg, String), and field length (eg, 200).

[0118] It should be noted that metadata is data that describes other data (for example, metadata is data that describes objects such as information resources or data), or structured data used to provide information about a certain resource.

[0119] The purpose of using metadata is to identify resources, evaluate resources, and track changes in resources during use; to achieve simple and efficient management of large amounts of networked data; to achieve effective discovery, search, integrated organization of information resources and effective management of used resources.

[0120] Once created, metadata can be shared. The structure and integrity of metadata depend on the value of the information resource and the context in which it is used; the environment in which metadata is developed and utilized is often a dynamic, distributed one. Metadata can be a coding system. Metadata is a coding system used to describe digital information resources, particularly online ones. Its most important characteristic and function is to establish a machine-understandable framework for digital information resources. Metadata can be stored in databases for easy access.

[0121] For example, the rule parsing module 621 is used to analyze the received preset data processing statements to identify key information such as connectors, operators, fields and values; a preset data processing unit is generated based on the identified multiple different types of information; and then an alternative semantic tree is constructed based on multiple preset data processing units.

[0122] The preset data processing statement can be a statement configured through the rule interface shown in Figure 5. For example, the preset data processing statement is a custom SQL statement, and the custom SQL statement supports the statement execution rules in the conventional SQL database, which also includes connectors such as "and" and "or", and priority calculation operators such as "{}", and supports operators such as "=, like, between, in, contain, not contain, group by", etc.

[0123] For example, the default data processing statement is: {location = Xi'an or location like Hai} and productInfo does not contain LTE TDD group by moc. When parsing this default data processing statement, you can use the following method:

[0124] First, the spaces, keywords, and line breaks in the preset data processing statements are segmented to obtain multiple data processing objects (for example, single words: location, Xi'an, Hai, productInfo, LTE TDD, moc, etc.) and multiple processing rules (for example, processing rules include: =, or, like, and, not contain, group by, etc.).

[0125] Then, based on the priority information in the SQL statement, at least one processing rule is matched with multiple data objects to be matched, and a preset data processing unit corresponding to the preset data processing statement is generated. For example, based on a custom SQL statement, the brackets (such as "{}") have the highest priority; when both "and" and "or" connectors exist, the priority of "and" is higher than the priority of "or", etc. Therefore, the above preset data processing statement can be split into the following preset data processing units, namely "location = Xi'an or location like Hai", "productInfo not contain LTE TDD", and "group by moc".

[0126] Furthermore, an alternative semantic tree is constructed based on the above-mentioned multiple preset data processing units. For example, with the operator (or connector) as the center, the preset data processing unit is represented as a "keyword-operator (or connector)-value range" format. Then, based on the priority of each operator and connector, an alternative semantic tree is constructed, for example, the alternative semantic tree shown in Figure 2.

[0127] It should be noted that the server also caches the candidate semantic trees obtained from the above analysis in the database to facilitate subsequent data analysis. Since different databases have different execution statements and corresponding execution rules, it is necessary to map the preset data processing statements based on preset mapping relationship information that reflects the mapping relationship between the preset data processing statements and the execution statements in the database to be used, in order to facilitate compatibility with different databases.

[0128] In step S703, when it is determined to perform offline data query, the query statement corresponding to the data to be processed is determined according to the target semantic tree corresponding to the data to be processed, and the database to be used is queried based on the query statement corresponding to the data to be processed to obtain the data query result.

[0129] When the preset data processing statement obtained is relatively complex and cannot be directly adapted to the database to be used, the preset data processing statement can be disassembled based on the target semantic tree while keeping the semantics of the preset data processing statement unchanged to obtain multiple disassembled statements; the multiple disassembled statements are reorganized to obtain multiple query statements to be used that match the database to be used; then, the multiple query statements to be used are used to query the database to be used respectively, and the multiple results obtained from the query are processed with intersection and / or union to obtain the final data query result.

[0130] For example, when the database to be used is a My SQL database, since the "not contain" operator in the preset data processing statement is an operator not supported by the My SQL database, it is necessary to first delete "productInfo not contain LTE TDD" and obtain the preliminary query statement corresponding to the data to be processed as "(location=Xi'an or location like Hai) group by moc". Then, the first query statement is input into the database to be used for query to obtain the first query result; then, multiple preset filtering rules in the preset rule filtering set are used to filter out the query results in the first query result that meet the "productInfo not contain LTE TDD" condition, thereby obtaining the final data query result.

[0131] For another example, if the database to be used is a PgSql database and the obtained preset data processing statement is "nbiId=1and location not contain d," the "not contain" operator is also an operator not supported by the PgSql database. Therefore, it is necessary to first enter "nbiId=1" into the database to be used for querying to obtain the second query result. Then, multiple preset filtering rules in the preset rule filtering set are used to filter out the query results in the second query result that match "location not contain d," thereby obtaining the final data query result.

[0132] Through customized preset data processing statements, more operators can be expanded to provide more data query methods and improve the accuracy of data query.

[0133] In some exemplary embodiments, a difference list can be established based on the difference information between the customized preset data processing statement and the statement supported by the database to be used, and the difference list can be cached on the server so that the customized preset data processing statement can be identified by querying the difference list, thereby converting the customized preset data processing statement into a statement supported by the database to be used.

[0134] In specific use, each alternative semantic tree is scanned from bottom to top to obtain the processing rules in the preset data processing unit in the alternative semantic tree, and the processing rules are compared with the operators in the difference list to obtain the statements that can be supported by the database to be used. For example, the priority of the custom character "{}" matches the priority of "()" used in the database to be used, and the above difference information needs to be written into the difference list. The difference list can also be cached in the server in the form of a configuration file.

[0135] In step S704 , the server receives the changed data.

[0136] The modified data may be data obtained through a packet capture tool or a data monitoring tool, and the modified data is data with certain identifiers changed based on the data to be processed. For example, if the data to be processed is business data corresponding to the identifier "Shanghai", the modified data may be business data corresponding to the identifier "Xi'an".

[0137] In step S705, when it is determined that the business data has changed, the multiple cached alternative semantic trees are used to match the changed data, determine the rule level corresponding to the changed data in the target semantic tree, and determine the query statement corresponding to the changed data based on the target semantic tree; based on the query statement corresponding to the changed data, query the database to be used to obtain the data query result.

[0138] When the server receives the changed data sent by the terminal, it sends the changed data to the message queue. The near-real-time data statistics module 624 obtains the changed data by subscribing to the change message in the message queue, matches the changed data using multiple cached alternative semantic trees, and performs layer-by-layer judgment on the changed data based on multiple preset filtering rules in the preset rule filtering set to determine the rule level corresponding to the changed data in the target semantic tree. Then, based on the target semantic tree, the query statement corresponding to the changed data is determined. The query statement corresponding to the changed data is used to query the database to obtain the data query result.

[0139] For example, in the alternative semantic tree shown in Figure 3, if the "nbiId" field changes from 1 to 0 and the values ​​of other fields remain unchanged, thereby obtaining the changed data, the server will send the changed data to the message queue; the near real-time data statistics module 624 obtains the changed data by subscribing to the change message of the message queue, and identifies that the "nbiId" field has changed, and uses the cached multiple alternative semantic trees to match the changed data to determine the rule level corresponding to the changed data in the target semantic tree.

[0140] Furthermore, based on multiple preset filtering rules in the preset rule filtering set, each operator (such as the operator "=", "like" and "group by") in the statement "{nbiId=1 or nbiId=2}and location like a group by mimVersion" corresponding to the alternative semantic tree shown in Figure 3 is checked in turn to obtain the query statement corresponding to the changed data; based on the query statement corresponding to the changed data, the database to be used is queried to obtain the data query result.

[0141] For example, when business data changes, for example, the value of the "location" field in the data to be processed changes from "Beijing" to "Shanghai," the near-real-time data statistics module 624 receives the changed data through the message queue. Since the initial value of the "location" field is "Beijing," when the original target semantic tree is used to process the changed data, no query statement corresponding to the changed data can be found.

[0142] By obtaining the possible matching rule "location like sea" in the alternative semantic tree, it is known that the changed data with the value of the "location" field as "Shanghai" meets the rule of "location like sea". When it is determined that other fields also meet the rule, it can be determined that the rule match for the changed data is successful, thereby determining that the changed data should belong to the rule, and at the same time removing the associated information of the unmatched rule.

[0143] It should be noted that in addition to multiple alternative semantic trees, the rule information cached in the data processing system can also cache preset index information. The preset index information is obtained by arranging multiple cached business information in reverse order based on the acquisition time of the business information. For example, the preset index information may include a preset business name and / or a preset business value, etc.

[0144] In some exemplary embodiments, the data to be processed further includes incremental change data in real-time data. The incremental change data in real-time data represents changes in the real-time business data relative to the historical business data. In other words, the business data has changed, and the incremental change data in the real-time data represents the data after the change.

[0145] When it is determined that the target semantic tree does not include query statements corresponding to the incremental change data in the real-time data, the incremental change data in the real-time data is parsed to obtain business difference information, which includes business names and / or business values; the preset index information and the business difference information are matched to determine the target query rules; based on the target query rules and multiple query statements to be used, the query statements corresponding to the data to be processed are determined.

[0146] In some exemplary embodiments, multiple operator filters can be established based on operators supported by the data processing system (e.g., multiple preset filter rules in a preset rule filter set) and connected in the form of a responsibility chain to form a data change filter chain. For example, FIG8 shows a schematic diagram of a data change filter chain provided in an embodiment of the present application. As shown in FIG8, the data change filter chain 800 includes: an equals filter 801, a like filter 802, a between filter 803, a contain filter 804, and a not contain filter 805.

[0147] By using multiple filters in the data change filter chain 800, the data query results obtained based on the target semantic tree are verified to facilitate supplementary queries on the statements in the target semantic tree, fully meeting the query requirements of the data to be processed, and making the final data query results more accurate.

[0148] For example, if the value of the "location" field in the above-mentioned data to be processed changes from "Beijing" to "Shanghai", the rule that may match it in the alternative semantic tree is "location like sea". Since this rule does not contain other operators (only the "like" operator), the equal filter 801 can be skipped during the data processing process, and only the value of the data to be processed is matched in the like filter 802, thereby simplifying the data processing steps. After the above processing, it can be determined that the data to be processed whose value of the "location" field is "Shanghai" meets the "location like sea" rule.

[0149] For another example, if the preset data processing statement obtained is more complex, for example, the preset data processing statement is "{location=Xi'an or location like Hai}and productInfo not contain LTE TDD", the rules can be checked from bottom to top on the target semantic tree corresponding to the preset data processing statement, that is, first execute "location=Xi'an" and "location like Hai" respectively to query the database to be used, and take the union of the execution results corresponding to the two rules to obtain the first processing result; then, execute "productInfo not contain LTE TDD" to query the database to be used to obtain the second processing result; further, intersect the first processing result and the second processing result to obtain the processing result to be confirmed, and determine whether the processing result to be confirmed is the final data query result required by the user by the degree of matching between the processing result to be confirmed and the preset data processing statement "{location=Xi'an or location like Hai}and productInfo not contain LTE TDD".

[0150] In step S706, persistence processing is performed on the target semantic tree and the database to be used.

[0151] For example, the model management module 622 may be used to perform persistence operations on the database to be used.

[0152] Among them, the persistence operation can be to update the database to be used by means of a sliding time window. In other words, the update frequency of the database to be used is determined based on the preset interval time and the incremental change data in the real-time data received within the preset interval time; based on the update frequency of the database to be used, the incremental change data in the real-time data is used to update the database to be used, thereby reducing the number of write operations on the database to be used and reducing the data processing pressure on the database to be used.

[0153] In this embodiment, the server constructs multiple alternative semantic trees corresponding to the preset data processing statements based on preset mapping relationship information and multiple customized preset data processing statements. The alternative semantic trees can reflect the logical relationship between different statements based on multiple different hierarchical structures. There is no need for R&D personnel to use special processing software to parse the data to be processed, and the data structure can be dynamically changed based on at least one customized preset data processing statement. The server can also make attribution judgments on the data to be processed and the changed data based on multiple alternative semantic trees, and determine the target semantic tree corresponding to the data to be processed to adapt to the processing requirements of the data to be processed. Furthermore, based on the target semantic tree corresponding to the data to be processed, the query statement corresponding to the data to be processed is determined, so that the query statement corresponding to the data to be processed can be applied to the database to be used, thereby speeding up the query of the data. The database to be used is queried based on the query statement corresponding to the data to be processed to obtain data query results, so that the data query results reflect the real-time changes in business data. There is no need to redeploy or write related business codes based on the needs of statistical business, which reduces development costs and improves the processing efficiency of business data.

[0154] Figure 9 shows a block diagram of the first data processing device provided by an embodiment of the present application. As shown in Figure 9, the first data processing device 900 includes but is not limited to the following modules 901 to 904.

[0155] The construction module 901 is configured to construct a plurality of candidate semantic trees corresponding to the preset data processing statements based on the preset mapping relationship information and a plurality of user-defined preset data processing statements.

[0156] The preset mapping relationship information is information representing the mapping relationship between the preset data processing statement and the execution statement in the database to be used.

[0157] The matching module 902 is configured to match the acquired format of the data to be processed with a plurality of candidate semantic trees to determine a target semantic tree corresponding to the data to be processed.

[0158] The determination module 903 is configured to determine the query statement corresponding to the data to be processed according to the target semantic tree corresponding to the data to be processed.

[0159] The query module 904 is configured to query the database to be used based on the query statement corresponding to the data to be processed to obtain a data query result.

[0160] It should be noted that the first data processing device 900 in this embodiment can implement any data processing method applied to the server in the embodiments of the present application.

[0161] According to the first data processing device of the embodiment of the present application, a plurality of alternative semantic trees corresponding to the preset data processing statements are respectively constructed based on the preset mapping relationship information and a plurality of customized preset data processing statements through the construction module. The alternative semantic trees can reflect the logical relationship between different statements based on a plurality of different hierarchical structures, and there is no need for R&D personnel to use special processing software to parse the data to be processed. The data structure can be dynamically changed based on at least one customized preset data processing statement; wherein the preset mapping relationship information is information that characterizes the mapping relationship between the preset data processing statement and the execution statement in the database to be used, which can make the execution statement in the database to be used and the customized data processing statement The system can match the sentences and flexibly process the data in the database to improve the data processing efficiency; use the matching module to match the acquired format of the data to be processed with multiple alternative semantic trees to determine the target semantic tree corresponding to the data to be processed to adapt to the processing requirements of the data to be processed; further, use the determination module to determine the query statement corresponding to the data to be processed based on the target semantic tree corresponding to the data to be processed, so that the query statement corresponding to the data to be processed can be applied to the database to be used, thereby speeding up the query speed of the data; use the query module to query the database to be used based on the query statement corresponding to the data to be processed to obtain data query results, so that the data query results can reflect the changes in real-time business data.

[0162] Figure 10 shows a block diagram of the second data processing device provided in an embodiment of the present application. As shown in Figure 10 , the second data processing device 1000 includes but is not limited to the following modules 1001 and 1002 .

[0163] The generating module 1001 is configured to generate data to be processed in a corresponding format in response to a configuration instruction, wherein the configuration instruction includes at least one self-defined processing rule and multiple data processing objects.

[0164] In some optional embodiments, the configuration instruction further includes: application scenario configuration information and / or device type configuration information, wherein at least one processing rule and multiple data processing objects all match the device type configuration information.

[0165] The sending module 1002 is configured to send the data to be processed to the server.

[0166] When the server receives the data to be processed, it will match the format of the data to be processed with multiple alternative semantic trees to determine the target semantic tree corresponding to the data to be processed; based on the target semantic tree corresponding to the data to be processed, it will determine the query statement corresponding to the data to be processed; based on the query statement corresponding to the data to be processed, it will query the database to be used to obtain the data query results.

[0167] It should be noted that the second data processing device 1000 in this embodiment can implement any data processing method applied to a terminal in the embodiments of the present application.

[0168] According to the second data processing device of the embodiment of the present application, a generation module is used to respond to a configuration instruction to generate data to be processed in a corresponding format. The data to be processed is determined based on a user's customization of data processing rules. There is no need for R&D personnel to use special processing software to parse the data to be processed. The data structure can be dynamically changed based on at least one customized processing rule and preset data processing statements generated by multiple data processing objects, thereby accelerating the processing speed of the data; then, a sending module is used to send the data to be processed to the server, so that the server matches the data to be processed with multiple alternative semantic trees according to the format of the data to be processed, and determines the target semantic tree corresponding to the data to be processed to meet the processing requirements of the data to be processed; based on the target semantic tree corresponding to the data to be processed, a query statement corresponding to the data to be processed is determined, so that the query statement corresponding to the data to be processed can be applied to the database to be used, thereby accelerating the query speed of the data; based on the query statement corresponding to the data to be processed, the database to be used is queried to obtain a data query result, so that the data query result reflects the real-time changes in business data.

[0169] It should be understood that the present application is not limited to the specific configurations and processes described in the above embodiments and illustrated in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here. The specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] FIG11 shows a structural diagram of an exemplary hardware architecture of a computing device capable of implementing the data processing method and apparatus according to an embodiment of the present application.

[0171] As shown in Figure 11, computing device 1100 includes an input device 1101, an input interface 1102, a central processing unit 1103, a memory 1104, an output interface 1105, and an output device 1106. The input interface 1102, the central processing unit 1103, the memory 1104, and the output interface 1105 are interconnected via a bus 1107. The input device 1101 and the output device 1106 are connected to the bus 1107 via the input interface 1102 and the output interface 1105, respectively, and are further connected to other components of the computing device 1100.

[0172] Specifically, the input device 1101 receives input information from the outside and transmits the input information to the central processing unit 1103 through the input interface 1102; the central processing unit 1103 processes the input information based on the computer-executable instructions stored in the memory 1104 to generate output information, stores the output information temporarily or permanently in the memory 1104, and then transmits the output information to the output device 1106 through the output interface 1105; the output device 1106 outputs the output information to the outside of the computing device 1100 for user use.

[0173] In one embodiment, the computing device shown in FIG11 may be implemented as an electronic device, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to execute the data processing method described in the above embodiment.

[0174] In one embodiment, the computing device shown in Figure 11 can be implemented as a data processing system, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to execute the data processing method described in the above embodiment.

[0175] The foregoing is merely an exemplary embodiment of the present application and is not intended to limit the scope of protection of the present application. Generally speaking, various embodiments of the present application can be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, although the present application is not limited thereto.

[0176] Embodiments of the present application may be implemented by a processor executing computer program instructions, for example, in a processor entity, or by hardware, or by a combination of software and hardware. The computer program instructions may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.

[0177] The block diagram of any logical flow in the accompanying drawings of the present application can represent program steps, or can represent interconnected logical circuits, modules and functions, or can represent a combination of program steps and logical circuits, modules and functions. The computer program can be stored on a memory. The memory can have any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as but not limited to read-only memory (ROM), random access memory (RAM), optical storage device and system (digital versatile disc DVD or CD optical disc) etc. The computer readable medium may include non-transient storage media. The processor can be any type suitable for the local technical environment, such as but not limited to a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (FPGA) and a processor based on a multi-core processor architecture.

[0178] The above description of exemplary embodiments of the present application has been provided by way of exemplary and non-limiting examples. However, various modifications and adaptations of the above embodiments will be apparent to those skilled in the art, when considered in conjunction with the accompanying drawings and the appended claims, without departing from the scope of the present application. Therefore, the proper scope of the present application will be determined by reference to the appended claims.

Claims

1. A data processing method, comprising: Based on the preset mapping relationship information and the multiple customized preset data processing statements, construct multiple candidate semantic trees corresponding to the preset data processing statements respectively, wherein the preset mapping relationship information is information representing the mapping relationship between the preset data processing statements and the execution statements in the database to be used; According to the format of the acquired data to be processed, the target semantic tree corresponding to the data to be processed is matched with a plurality of candidate semantic trees to determine; Determine the query statement corresponding to the data to be processed according to the target semantic tree corresponding to the data to be processed; The database to be used is queried based on the query statement corresponding to the data to be processed to obtain a data query result.

2. The method according to claim 1, wherein: The data to be processed includes: incremental change data in real-time data, wherein the incremental change data in real-time data represents data in which the business data acquired in real time has changed relative to the historical business data; The step of determining the query statement corresponding to the data to be processed according to the target semantic tree corresponding to the data to be processed includes: When it is determined that the target semantic tree does not include a query statement corresponding to the incremental change data in the real-time data, parsing the incremental change data in the real-time data to obtain service difference information, where the service difference information includes a service name and / or a service value; Matching the preset index information with the business difference information to determine a target query rule, wherein the preset index information is information obtained by arranging the cached multiple business information in reverse order based on the acquisition time of the business information; Based on the target query rule and a plurality of query statements to be used, a query statement corresponding to the data to be processed is determined.

3. The method according to claim 2, wherein: Before determining the query statement corresponding to the data to be processed based on the target query rule and the plurality of query statements to be used, the data processing method further includes: Based on a plurality of preset query rules in a preset rule filtering set, query rules matching the business difference information are verified to determine the target query rule.

4. The method according to claim 2, wherein: After querying the database to be used based on the query statement corresponding to the data to be processed and obtaining the data query result, the data processing method further includes: Determining the update frequency of the database to be used according to a preset interval time and incremental change data in the real-time data received within the preset interval time; According to the update frequency of the database to be used, the database to be used is updated using the incremental change data in the real-time data.

5. The method according to claim 1 or 2, wherein: The data to be processed also includes: historical business data; The querying the database to be used based on the query statement corresponding to the data to be processed to obtain the data query result includes: While keeping the semantics of the historical query statement corresponding to the historical business data unchanged, Decompose the corresponding historical query statements to obtain multiple decomposed statements; Reorganize the plurality of disassembled statements to obtain a plurality of query statements to be used that match the database to be used; The database to be used is queried using the plurality of query statements to be used respectively to obtain the data query result.

6. The method according to claim 5, wherein: The step of respectively using the plurality of query statements to be used to query the database to be used to obtain the data query result includes: Using the plurality of query statements to be used respectively to query the database to be used, to obtain a plurality of data sets to be processed; Intersection processing is performed on the multiple data sets to be processed, and / or union processing is performed on the multiple data sets to be processed to obtain the data query result.

7. The method according to any one of claims 1 to 6, wherein: The method of constructing a plurality of candidate semantic trees corresponding to the preset data processing statements based on the preset mapping relationship information and the plurality of customized preset data processing statements respectively includes: Obtaining a plurality of the preset data processing statements; Analyze each of the preset data processing statements respectively to determine a preset data processing unit corresponding to each of the preset data processing statements; According to the preset data processing unit corresponding to each of the preset data processing statements, an alternative semantic tree corresponding to each of the preset data processing statements is constructed.

8. The method according to claim 7, wherein: The preset data processing unit includes at least one processing rule and a plurality of data processing objects, and the analyzing each of the preset data processing statements to determine the preset data processing unit corresponding to each of the preset data processing statements includes: According to at least one processing rule corresponding to each of the preset data processing statements, the plurality of data processing objects are segmented to obtain a plurality of data objects to be matched; The at least one processing rule is matched with the multiple data objects to be matched respectively, and a preset data processing unit corresponding to each preset data processing statement is generated.

9. The method according to claim 8, wherein: The processing rules include: operation rules and / or connection rules; the data processing object includes at least one of the data type of the business data, the data name of the business data, and the value range of the business data; The preset mapping relationship information includes: first mapping relationship information matching the execution statement in the database to be used, and / or second mapping relationship information determined based on a configuration file, wherein the second mapping relationship information includes at least one of operation rule difference information, character difference information and string difference information.

10. The method according to claim 8, wherein: The step of constructing a candidate semantic tree corresponding to each of the preset data processing statements according to the preset data processing unit corresponding to each of the preset data processing statements includes: For the preset data processing unit corresponding to each of the preset data processing statements, based on the priority of the processing rules, the level to which each of the processing rules belongs is determined step by step; and according to the level to which each of the processing rules belongs, the multiple preset data processing units corresponding to the processing rules are hierarchically processed to construct an alternative semantic tree corresponding to the preset data processing statements.

11. The method according to any one of claims 1 to 6, wherein: After querying the database to be used based on the query statement corresponding to the data to be processed and obtaining the data query result, the data processing method further includes: The target semantic tree is updated according to the data query result.

12. A data processing method, the method comprising: In response to the configuration instruction, generating data to be processed in a corresponding format; wherein the configuration instruction includes at least one customized processing rule and a plurality of data processing objects; The data to be processed is sent to the server.

13. The method according to claim 12, wherein: The configuration instruction further includes: application scenario configuration information and / or device type configuration information, and the at least one processing rule and the multiple data processing objects all match the device type configuration information.

14. An electronic device comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 13.

15. A readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method according to any one of claims 1 to 13.

Images