Data processing method, electronic equipment and storage medium

Abstract

The invention provides a data processing method, electronic equipment and a storage medium, and relates to the technical field of data processing. The method comprises the steps that on the basis of preset mapping relation information and multiple self-defined preset data processing statements, multiple alternative semantic trees corresponding to the preset data processing statements are constructed, and the preset mapping relation information is information representing the mapping relation between the preset data processing statements and execution statements in a database to be used; according to the obtained format of the to-be-processed data, matching with a plurality of alternative semantic trees, and determining a target semantic tree corresponding to the to-be-processed data; determining a query statement corresponding to the to-be-processed data according to a target semantic tree corresponding to the to-be-processed data; and querying the to-be-used database based on the query statement corresponding to the to-be-processed data to obtain a data query result. And the processing steps of the business data are simplified, so that the data query result reflects the change condition of the business data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, an electronic device and a storage medium. Background Art

[0002] In the process of statistical analysis of big data, an offline database-based query analysis method is usually used to process large amounts of data. This not only takes a long time (for example, several hours or days), but also fails to reflect the real-time changes in the data.

[0003] For example, technicians usually need to use technologies such as flink and stream to perform statistics on streaming data, which requires technicians to be proficient in tools such as flink. Moreover, for a new statistical business, technicians need to use operators in tools such as flink and redeploy or write related business codes based on the needs of the statistical business, which increases labor costs and prolongs the processing time of business data, making it impossible for users to quickly perform data statistics based on real-time changes in business data. Summary of the invention

[0004] The present application provides a data processing method, an electronic device and a storage medium.

[0005] An embodiment of the present application provides a data processing method, which includes: based on preset mapping relationship information and multiple customized preset data processing statements, respectively constructing multiple candidate semantic trees corresponding to the preset data processing statements, wherein the preset mapping relationship information is information characterizing the mapping relationship between the preset data processing statements and the execution statements in the database to be used; matching the acquired format of the data to be processed with the multiple candidate semantic trees to determine the target semantic tree corresponding to the data to be processed; determining the query statement corresponding to the data to be processed based on the target semantic tree corresponding to the data to be processed; and querying the database to be used based on the query statement corresponding to the data to be processed to obtain a data query result.

[0006] An embodiment of the present application provides a data processing method, which includes: generating to-be-processed data having a corresponding format in response to a configuration instruction; wherein the configuration instruction includes at least one customized processing rule and multiple data processing objects, and sending the to-be-processed data to a server.

[0007] An embodiment of the present application provides an electronic device, including: one or more processors; a memory, on which one or more programs are stored. When the one or more programs are executed by one or more processors, the one or more processors implement any one of the data processing methods in the embodiments of the present application.

[0008] An embodiment of the present application provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, any one of the data processing methods in the embodiments of the present application is implemented.

[0009] According to the data processing method, electronic device and storage medium of the embodiment of the present application, multiple alternative semantic trees corresponding to the preset data processing statements are respectively constructed based on the preset mapping relationship information and multiple customized preset data processing statements. The alternative semantic trees can reflect the logical relationship between different statements based on multiple different hierarchical structures, and there is no need for R&D personnel to use special processing software to parse the data to be processed. The data structure can be dynamically changed based on at least one customized preset data processing statement; wherein the preset mapping relationship information is information representing the mapping relationship between the preset data processing statement and the execution statement in the database to be used, which can enable the execution statement in the database to be used to be dynamically changed. The statement is matched with the customized data processing statement, and the data in the database is flexibly processed to improve the data processing efficiency; according to the format of the acquired data to be processed, it is matched with multiple alternative semantic trees to determine the target semantic tree corresponding to the data to be processed to adapt to the processing requirements of the data to be processed; further, according to the target semantic tree corresponding to the data to be processed, the query statement corresponding to the data to be processed is determined, so that the query statement corresponding to the data to be processed can be applied to the database to be used, thereby speeding up the query speed of the data; based on the query statement corresponding to the data to be processed, the database to be used is queried to obtain the data query result, so that the data query result reflects the changes in the business data.

[0010] With regard to the above embodiments and other aspects of the present application and their implementation, further description is provided in the accompanying drawings, detailed description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flow chart showing a data processing method provided in an embodiment of the present application is shown.

[0012] Figure 2 A schematic diagram showing an alternative semantic tree provided in an embodiment of the present application.

[0013] Figure 3 A schematic diagram showing an alternative semantic tree provided in an embodiment of the present application.

[0014] Figure 4 A flow chart showing a data processing method provided in an embodiment of the present application is shown.

[0015] Figure 5 A schematic diagram of a rule interface provided in an embodiment of the present application is shown.

[0016] Figure 6A block diagram showing the composition of a data processing system provided in an embodiment of the present application is shown.

[0017] Figure 7 A flowchart showing a working method of a data processing system provided in an embodiment of the present application is shown.

[0018] Figure 8 A schematic diagram of a data change filter chain provided in an embodiment of the present application is shown.

[0019] Fig. 9 A block diagram showing the composition of a first data processing device provided in an embodiment of the present application is shown.

[0020] Fig.10 A block diagram showing the composition of a second data processing device provided in an embodiment of the present application is shown.

[0021] Fig.11 A structural diagram showing an exemplary hardware architecture of a computing device capable of implementing the data processing method and apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0023] In the process of statistical analysis of big data, offline database-based query analysis is usually used to process large amounts of data, which not only takes a long time (for example, several hours or days), but also fails to reflect the real-time changes in the data. Especially when the data that needs to be statistically analyzed changes rapidly, this offline data statistics method that relies entirely on the database will reveal its drawbacks.

[0024] For example, first you need to rely entirely on the offline data statistics method of the database, and then use the scheduled task to count the business data again to obtain the real-time changes in the business data. However, the method of counting data through scheduled tasks will increase the data processing pressure of the server.

[0025] Technical personnel usually use technologies such as flink and stream to perform statistics on streaming data, which requires technical personnel to be proficient in tools such as flink and stream. Moreover, for a new business statistics requirement, technical personnel need to use operators in tools such as flink and redeploy or write related business codes based on the needs of statistical business, which increases labor costs and prolongs the processing time of business data, making it impossible for users to quickly perform data statistics based on real-time changes in business data.

[0026] In order to solve the above problems, the present application provides a data processing method, an electronic device and a storage medium, so that users can quickly and accurately count business data, and determine multiple alternative semantic trees based on preset mapping relationship information and at least one customized preset data processing statement. The alternative semantic trees can be based on multiple different hierarchical structures, reflecting the logical relationship between different statements, without the need for R&D personnel to use special processing software to parse the data to be processed, and can achieve dynamic changes to the data structure based on at least one customized preset data processing statement. Furthermore, by matching the format of the acquired data to be processed with multiple alternative semantic trees, the target semantic tree corresponding to the data to be processed is determined to meet the processing requirements of the data to be processed.

[0027] Figure 1 FIG. 1 is a flow chart of a data processing method provided by an embodiment of the present application. The data processing method can be applied to a server, for example, a server for processing business data. Figure 1 As shown, the data processing method in the embodiment of the present application includes but is not limited to the following steps.

[0028] Step S101 : constructing a plurality of candidate semantic trees corresponding to the preset data processing statements based on the preset mapping relationship information and a plurality of user-defined preset data processing statements.

[0029] The preset mapping relationship information is information representing the mapping relationship between the preset data processing statements and the execution statements in the database to be used.

[0030] Step S102 : matching the acquired format of the data to be processed with a plurality of candidate semantic trees to determine a target semantic tree corresponding to the data to be processed.

[0031] It should be noted that the alternative semantic tree is a tree with multiple different hierarchical structures, each level corresponds to a data processing logic. By matching the format of the acquired data to be processed with each level in the multiple alternative semantic trees, the target semantic tree corresponding to the data to be processed can be obtained. The hierarchical structure in the target semantic tree can meet the analysis requirements of the logical relationship in the data to be processed.

[0032] Step S103: determining the query statement corresponding to the data to be processed according to the target semantic tree corresponding to the data to be processed.

[0033] By matching each processing logic in the data to be processed with each layer structure in the target semantic tree, a query statement corresponding to the data to be processed is obtained, so that the query statement conforms to the statement definition in the database to be used.

[0034] Step S104: query the database to be used based on the query statement corresponding to the data to be processed to obtain a data query result.

[0035] Among them, the data query result includes data that matches the query statement corresponding to the data to be processed. For example, if the query statement corresponding to the data to be processed is "device identification = 1", the data query result includes multiple target devices, and the device identification of each target device is 1, but the device type of the target device may be different.

[0036] The above data query results are only examples. Other data query results not described are also within the protection scope of this application and will not be described in detail here.

[0037] In this embodiment, by constructing multiple alternative semantic trees corresponding to the preset data processing statements based on preset mapping relationship information and multiple customized preset data processing statements, the alternative semantic trees can reflect the logical relationship between different statements based on multiple different hierarchical structures, and the R&D personnel no longer need to use special processing software to parse the data to be processed, and the data structure can be dynamically changed based on at least one customized preset data processing statement; wherein the preset mapping relationship information is information that characterizes the mapping relationship between the preset data processing statement and the execution statement in the database to be used, so that the execution statement in the database to be used can be matched with the customized data processing statement, and the data in the database can be flexibly processed to improve the processing efficiency of the data; according to the format of the acquired data to be processed, it is matched with multiple alternative semantic trees to determine the target semantic tree corresponding to the data to be processed to meet the processing requirements of the data to be processed; further, according to the target semantic tree corresponding to the data to be processed, the query statement corresponding to the data to be processed is determined, so that the query statement corresponding to the data to be processed can be applicable to the database to be used, thereby speeding up the query speed of the data; based on the query statement corresponding to the data to be processed, the database to be used is queried to obtain the data query result, so that the data query result reflects the change of the business data.

[0038] In some exemplary embodiments, the preset mapping relationship information includes: first mapping relationship information that matches the execution statement in the database to be used, and / or second mapping relationship information determined based on a configuration file, wherein the second mapping relationship information includes at least one of operation rule difference information, character difference information, and string difference information.

[0039] It should be noted that the execution statements to be used in the database may not necessarily match the format of the preset data processing statements (for example, custom execution statements based on MySQL). It is necessary to convert the preset data processing statements into statements recognizable by the database to be used (i.e., the execution statements in the database to be used) based on the preset mapping relationship information, so that the preset data processing statements can be applicable to multiple different databases.

[0040] Among them, the second mapping relationship information can be determined in the form of a configuration file, and at least one of the operation rule differences information, character differences information, and string differences information included in the second mapping relationship information can be used to convert the preset data processing statements to obtain statements recognizable by the database to be used, so that the database to be used can perform faster data queries based on the converted statements.

[0041] For example, the first mapping relationship information is mapping relationship information that does not contain difference information. In other words, the preset data processing statements can be directly converted into execution statements in the database to be used through the first mapping relationship information. That is, there is a one-to-one mapping relationship between the preset data processing statements and the execution statements in the database to be used, which can map the preset data processing statements more quickly and facilitate querying the data in the database to be used.

[0042] In some exemplary embodiments, based on the preset mapping relationship information and multiple custom preset data processing statements in step S101, multiple alternative semantic trees corresponding to the preset data processing statements are respectively constructed, including: obtaining multiple preset data processing statements; analyzing each preset data processing statement respectively to determine the preset data processing unit corresponding to each preset data processing statement; and constructing an alternative semantic tree corresponding to each preset data processing statement based on the preset data processing unit corresponding to each preset data processing statement.

[0043] Among them, the processing rules include: operation rules and / or connection rules; the data processing objects include at least one of the data type of the business data, the data name of the business data, and the value range of the business data.

[0044] For example, if the preset data processing statement is set as "{location = Xi'an or location like 海} and productInfo not contain LTE TDD, group by moc", analyzing this preset data processing statement can obtain 4 preset data processing units, namely "location = Xi'an", "location like 海", "productInfo not contain LTE TDD", and "group by moc".

[0045] Figure 2 A schematic diagram of an alternative semantic tree provided by an embodiment of the present application is shown. Figure 2 As shown, through the logical relationship between each preset data processing unit, a Figure 2 Alternative semantic trees shown.

[0046] Among them, the logical relationship between the preset data processing unit "location=Xi'an" and the preset data processing unit "location like Hai" is "or" (i.e., the relationship of "or"); the logical relationship between the preset data processing unit "productInfo not containLTE TDD" and the preset data processing unit "{location=Xi'an or location like Hai}" is "and" (i.e., the relationship of "and"); the preset data processing unit "group by moc" and the preset data processing unit "{location=Xi'an or location like Hai}and productInfo not contain LTE TDD" are in a parallel logical relationship.

[0047] Through the above analysis, the logical relationship between each preset data processing unit can be clarified, so that the constructed alternative semantic tree can display the various data processing objects (for example, "sea", "Xi'an", etc.) and the relationship between each preset data processing unit in a hierarchical manner, thereby facilitating the subsequent matching of the processed data based on the alternative semantic tree and improving the data processing speed.

[0048] In some exemplary embodiments, the preset data processing unit includes at least one processing rule and multiple data processing objects. Analyzing each preset data processing statement separately to determine the preset data processing unit corresponding to each preset data processing statement includes: segmenting multiple data processing objects according to at least one processing rule corresponding to each preset data processing statement to obtain multiple data objects to be matched; matching at least one processing rule with multiple data objects to be matched respectively to generate a preset data processing unit corresponding to each preset data processing statement.

[0049] For example, if the preset data processing statement is set to "{nbiId=1or nbiId=2}and location likea group by mimVersion", where nbiId represents a device identifier. If the preset data processing statement is segmented, the following processing rules "'=', 'or', 'and', 'like', 'group by'" and multiple data processing objects, namely "'nbiId', '1', '2', 'location', 'a', 'mimVersion'" can be obtained.

[0050] Figure 3 A schematic diagram of an alternative semantic tree provided by an embodiment of the present application is shown. Figure 3 As shown, by matching the multiple processing rules obtained by the above segmentation with the multiple data processing objects, multiple preset data processing units corresponding to the preset data processing statements can be obtained, that is, Figure 3 "nbiId=1", "nbiId=2", "location like a", and "group by mimVersion" shown in FIG.

[0051] The preset data processing unit may include at least one of a data type, a data name, and a data value range. For example, the preset data processing unit is characterized as a "keyword-operator (or connector)-value range" form to clarify the processing logic of each data processing object.

[0052] The preset data processing unit obtained through the above analysis can analyze different data processing objects in segments, improve the processing efficiency of the preset data processing statements, and speed up the construction of the candidate semantic tree.

[0053] In some exemplary embodiments, based on the preset data processing units corresponding to each preset data processing statement, an alternative semantic tree corresponding to each preset data processing statement is constructed, including: for the preset data processing units corresponding to each preset data processing statement, based on the priority of the processing rules, the level to which each processing rule belongs is determined step by step; and according to the level to which each processing rule belongs, multiple preset data processing units corresponding to the processing rules are layered processed to construct an alternative semantic tree corresponding to the preset data processing statement.

[0054] The priority of the processing rule may be determined based on the priority of the processing rule in the database to be used. For example, brackets (such as "{}") have the highest priority; when both "and" and "or" connectors exist, the priority of "and" is higher than the priority of "or", etc.

[0055] Determine the level to which each processing rule belongs step by step according to the priority of the processing rules in the database to be used. If the priority of a processing rule is higher, its level is higher; conversely, if the priority of a processing rule is lower, its level is lower (for example, at the end of the alternative semantic tree, etc.), so that multiple preset data processing units corresponding to the processing rules can be hierarchically processed, facilitating the positioning of the levels of each preset data processing unit and accelerating the construction of the alternative semantic tree.

[0056] In some exemplary embodiments, after querying the database to be used based on the query statement corresponding to the data to be processed in step S104 and obtaining a data query result, the data processing method further includes: updating the target semantic tree according to the data query result.

[0057] Among them, when the data query result is a result determined based on the changed data, the target semantic tree can be updated correspondingly according to the level in the target semantic tree corresponding to the changed data.

[0058] For example, if it is determined that Figure 2 the shown alternative semantic tree is the target semantic tree, when the value of the "location" field in the data to be processed changes from "Shanghai" to "Hangzhou", the rule that may match it in the target semantic tree can be obtained as "location like 海". Then, after finding the corresponding data query result, update Figure 2 "location like 海" in the shown alternative semantic tree to "location like Hangzhou" in the target semantic tree, thereby realizing the update of the target semantic tree and facilitating the subsequent matching of other data.

[0059] In some exemplary embodiments, the data to be processed includes historical business data; querying the database to be used based on the query statement corresponding to the data to be processed in step S104 and obtaining a data query result can be implemented in the following manner:

[0060] Under the condition of keeping the semantics of the historical query statement corresponding to the historical business data unchanged, disassemble the historical query statement corresponding to the historical business data to obtain multiple disassembled statements; reorganize the multiple disassembled statements to obtain multiple query statements to be used that match the database to be used; respectively use the multiple query statements to be used to query the database to be used to obtain a data query result.

[0061] Among them, by scanning the alternative semantic tree from bottom to top, all operators supported by the database to be used are obtained, and the historical query statements corresponding to the historical business data are disassembled to obtain multiple disassembled statements (for example, disassembled into "location = Xi'an or location like Sea", "productInfo not contain LTE TDD", and "group by moc", etc.). Then, based on the priority information of the multiple disassembled statements, the multiple disassembled statements are reorganized to obtain the query statements to be used that match the database to be used (for example, "{location = Xi'an or location like Sea} group by moc"). Further, "{location = Xi'an or locationlike Sea}" and "group by moc" are respectively input into the database to be used for query, and multiple query results to be matched are obtained. Then, the multiple query results to be matched are comprehensively analyzed to obtain data query results.

[0062] It should be noted that not all decomposed statements can match the execution statements in the database to be used. Therefore, the statements that do not match the execution statements in the database to be used (for example, "productInfo not contain LTE TDD") can be eliminated; then, the data query results obtained based on "{location=Xi'an or location like Hai} group by moc") are filtered again using post-filtering to obtain more accurate data query results that meet the requirements of "{location=Xi'an or location like Hai} and productInfo notcontain LTE TDD group by moc".

[0063] Through the above operations, the historical query statements corresponding to the historical business data can be analyzed in parallel, and the database to be used can be queried using multiple query statements to be used that match the database to be used, which speeds up the query speed, enables users to quickly and accurately obtain data query results, and improves data processing efficiency.

[0064] In some exemplary embodiments, a plurality of query statements to be used are used to query the database to be used respectively to obtain data query results, including: using a plurality of query statements to be used to query the database to be used respectively to obtain a plurality of data sets to be processed; performing intersection processing on the plurality of data sets to be processed, and / or performing union processing on the plurality of data sets to be processed to obtain data query results.

[0065] By performing intersection processing on multiple data sets to be processed and / or performing union processing on multiple data sets to be processed, it is possible to ensure that the data query results finally obtained meet the query requirements of multiple query statements to be used, making the data query results more accurate.

[0066] In some exemplary embodiments, the data to be processed includes: incremental change data in real-time data, where the incremental change data in real-time data represents data that has changed from the business data acquired in real time to the historical business data. In other words, the historical business data has changed, and the incremental change data in the real-time data is the changed data.

[0067] Determining the query statement corresponding to the data to be processed based on the target semantic tree corresponding to the data to be processed in step S103 can be implemented in the following manner: when it is determined that the target semantic tree does not include the query statement corresponding to the incremental change data in the real-time data, parse the incremental change data in the real-time data to obtain business difference information, where the business difference information includes the business name and / or business value; match the preset index information with the business difference information to determine the target query rule; and determine the query statement corresponding to the data to be processed based on the target query rule and multiple query statements to be used.

[0068] The preset index information is information obtained by arranging the cached multiple business information in reverse order based on the acquisition time of the business information. The smaller the difference between the acquisition time of the business information and the current time, the higher the sequence number of the arrangement index of the business information, which is convenient for querying the updated business data. For example, the preset index information may include a preset business name and / or a preset business value, etc.

[0069] Based on the chronological order, the service name in the service difference information is matched with the preset service name, and / or the service value in the service difference information is matched with the preset service value in the preset index information, so as to clarify which preset service the service difference information specifically corresponds to, thereby obtaining the target query rule corresponding to the preset service.

[0070] By using target query rules to supplement multiple query statements to be used and determine the query statements corresponding to the data to be processed, it is possible to perform comprehensive query matching on the data to be processed even when the target semantic tree does not include the query statements corresponding to the incremental change data in the real-time data, thereby meeting the query requirements of the data to be processed to the greatest extent and improving the accuracy of data query.

[0071] In some exemplary embodiments, before determining the query statement corresponding to the data to be processed based on the target query rule and multiple query statements to be used, the data processing method also includes: based on multiple preset filtering rules in the preset rule filtering set, verifying the filtering rules that match the business difference information to determine the target query rule.

[0072] The preset rule filter set includes multiple preset filter rules, for example, at least one of an equal filter rule, a like filter rule, a between filter rule, a contain filter rule, and a not contain filter rule.

[0073] By using multiple preset filtering rules to verify the filtering rules that match the business difference information, the target query rules can be more in line with the query requirements of the data to be processed, the query requirements of the data to be processed can be met to the greatest extent, and the accuracy of data query can be improved.

[0074] In some exemplary embodiments, after the database to be used is queried based on the query statement corresponding to the data to be processed in step S104 and the data query result is obtained, the data processing method also includes: determining the update frequency of the database to be used based on a preset interval time and the incremental change data in the real-time data received within the preset interval time; and updating the database to be used using the incremental change data in the real-time data based on the update frequency of the database to be used.

[0075] The preset interval time may be 1 minute, 10 minutes, or any other set time.

[0076] For example, if the preset interval is set to 1 minute, and 500 incremental changes in real-time data are received within 1 minute, the database to be used needs to be updated immediately. If the preset interval is set to 10 minutes, and only 1 incremental change in real-time data is received within 10 minutes, the database can be updated after 10 minutes.

[0077] The above preset interval duration is only an example and can be specifically set according to actual needs. Other preset interval durations not described are also within the protection scope of this application and will not be described here.

[0078] By setting a preset interval time, obtaining incremental change data in real-time data received within the preset time, and adaptively modifying the update frequency of the database to be used, the number of write operations on the database to be used can be reduced, thereby reducing the data processing pressure on the database to be used.

[0079] Figure 4The flowchart of the data processing method provided by the embodiment of the present application is shown. The data processing method can be applied to a terminal, for example, a mobile phone for processing business data. Figure 4 As shown, the data processing method in the embodiment of the present application includes but is not limited to the following steps.

[0080] Step S401, in response to the configuration instruction, generating data to be processed in a corresponding format.

[0081] The configuration instruction includes at least one customized processing rule and multiple data processing objects.

[0082] In some optional embodiments, the configuration instruction further includes: application scenario configuration information and / or device type configuration information, wherein at least one processing rule and multiple data processing objects all match the device type configuration information.

[0083] For example, Figure 5 Schematic diagram of the rule interface provided by the embodiment of the present application is shown. Figure 5 As shown, in the rule interface, there are three types of operation components, namely "rule name", "member type", and "rule definition".

[0084] When users operate different operation components, they will select different member types accordingly. The fields in the "Rule Definition" can be dynamically adjusted to the data fields of the relevant business. The "and / or" field can represent the logical "and" (corresponding to the "and" operator in the database), or the logical "or" (corresponding to the "or" operator in the database), etc.

[0085] In "Operator", you can select an operator supported by the current system. In the "Value" column, users can dynamically adjust the data type in the input box (such as string, numeric type, enumeration type, etc.) according to "Field" and "Operator".

[0086] Among them, "Rule Name" can be filled in according to the processing requirements of business data, for example, "Custom Scenario" corresponds to "test", etc. "Member Type" is used to characterize the device type configuration information, that is, the device type of the network element device that carries the data to be processed, for example, it can be a "network element" device or a "cell" (that is, a communication cell, etc.).

[0087] In "Rule Definition", you can add and / or delete multiple configuration rules based on user needs, and select each configuration rule based on "Field", "Operator", and "Value". For example, Figure 5Four rules are shown below. The first rule is "nbiId (network element ID) = 1", the second rule is "location like 'Hai'", the third rule is "subType (model type) does not contain 'A'", and the fourth rule is "ipAddress (access IP address) between ('1.1.1.1' and '1.2.3.4') group by moc (network element type)".

[0088] Among them, there is a logical "AND" relationship between the first rule and the second rule; there is a logical "OR" relationship between the third rule and (the results of the first rule and the second rule); there is a logical "AND" relationship between the fourth rule and (the results of the first rule, the second rule, and the third rule).

[0089] Through the above operations, the obtained data to be processed can be characterized as "{nbiId = 1 and location like 'Hai'} or subType does not contain 'A' and ipAddress between ('1.1.1.1', '1.2.3.4') group by moc".

[0090] Use a graphical interface to perform corresponding configuration on the configuration information, which is convenient for users to use, and realizes the way for users to customize the data processing rules. There is no need for R & D personnel to use special processing software to parse the data to be processed, and it can realize the dynamic change of the data structure based on at least one preset data processing statement defined by the user, and speed up the data processing speed.

[0091] Step S402: Send the data to be processed to the server.

[0092] When the server receives the data to be processed, it will match the format of the data to be processed with multiple alternative semantic trees to determine the target semantic tree corresponding to the data to be processed; based on the target semantic tree corresponding to the data to be processed, determine the query statement corresponding to the data to be processed; query the database to be used based on the query statement corresponding to the data to be processed to obtain the data query result.

[0093] Among them, the alternative semantic tree is a tree determined based on the preset mapping relationship information and at least one preset data processing statement defined by the user. The preset mapping relationship information is the information representing the mapping relationship between the preset data processing statement and the execution statement in the database to be used.

[0094] In this embodiment, by responding to the configuration instruction, the data to be processed with the corresponding format is generated. The data to be processed is determined based on the user's customization of the data processing rules. There is no need for R&D personnel to use special processing software to parse the data to be processed. The data structure can be dynamically changed based on at least one customized processing rule and preset data processing statements generated by multiple data processing objects, thereby accelerating the data processing speed; then, the data to be processed is sent to the server, so that the server can match the data to be processed with multiple alternative semantic trees according to the format of the data to be processed, and determine the target semantic tree corresponding to the data to be processed to meet the processing requirements of the data to be processed; based on the target semantic tree corresponding to the data to be processed, determine the query statement corresponding to the data to be processed, so that the query statement corresponding to the data to be processed can be applicable to the database to be used, thereby accelerating the data query speed; based on the query statement corresponding to the data to be processed, the database to be used is queried to obtain data query results, so that the data query results reflect the changes in business data.

[0095] Figure 6 FIG. 1 is a block diagram showing the composition of a data processing system provided by an embodiment of the present application. Figure 6 As shown, the data processing system includes but is not limited to the following devices: a front-end device 610 , a business layer server 620 and a database 630 .

[0096] The data processing system can be applied to big data statistics and business scenarios with high requirements for data timeliness, such as smart home cloud services, data statistics scenarios of cloud platform products, data statistics scenarios of traffic management platforms, data statistics scenarios of IoT devices, and data statistics scenarios of e-commerce systems. The above business scenarios are only examples, which can be specifically set according to actual needs. Other unspecified business scenarios are also within the scope of protection of this application and will not be repeated here.

[0097] The front-end device 610 includes a rule management module 611. The business layer server 620 includes a rule parsing module 621, a model management module 622, an offline data statistics module 623 and a near real-time data statistics module 624. The database 630 includes a first type database 631, a second type database 632 and a third type database 633.

[0098] For example, the first type of database 631 can be a MS SQL database; the second type of database 632 can be an Elastic Search database; the third type of database 633 can be an Oracle database, that is, different types of databases are used to support data for the business layer server 620 to adapt to the processing of various different types of data and improve data processing efficiency.

[0099] The rule management module 611 is used to manage the rules based on the human-computer interaction interface (for example, Figure 5 The rule interface shown in the figure, etc.) obtains customized configuration information, and splices and assembles the obtained configuration information to build a customized preset data processing statement.

[0100] The rule parsing module 621 is used to analyze the received preset data processing statements, identify key information such as connectors, operators, fields and values ​​therein, generate preset data processing units based on the identified multiple different types of information, and then construct an alternative semantic tree based on multiple preset data processing units.

[0101] The model management module 622 is used to cache multiple candidate semantic trees and / or update the candidate semantic trees using the data query results output by the offline data statistics module 623 (or the data query results output by the near real-time data statistics module 624).

[0102] The offline data statistics module 623 is used to map the query statements corresponding to the data to be processed into statements that can be recognized by different types of databases through the second mapping relationship information determined by the configuration file.

[0103] The second mapping relationship information includes the difference mapping relationship information between the execution statement in the database to be used and at least one customized preset data processing statement.

[0104] For example, the difference mapping relationship information includes: operators not supported by the database to be used (for example, the not contain operator, etc.), priority difference information between the database to be used and the custom characters (for example, the priority of the custom character "{}" matches the priority of "()" used in the database to be used, and the above difference information needs to be mapped and reflected in the configuration file), and string difference information (for example, double quotes are needed to represent strings in the first type of database, while single quotes are needed to represent strings in the second type of database, and the above difference information needs to be mapped and reflected in the configuration file).

[0105] The offline data statistics module 623 is also used to match the acquired format of the data to be processed with multiple candidate semantic trees to determine the target semantic tree corresponding to the data to be processed, wherein the target semantic tree is a semantic tree obtained by screening from multiple candidate semantic trees constructed by the rule parsing module 621.

[0106] Furthermore, the offline data statistics module 623 maps the target semantic tree with the execution statement in the database to be used based on the second mapping relationship information, converts the custom characters in the data to be processed into statements that can be supported by the database to be used, thereby generating a query statement corresponding to the data to be processed. When the query statement corresponding to the data to be processed is used to perform an offline data query on the database to be used, the data query result matching the data to be processed can be quickly found, thereby improving the scalability of the data processing system.

[0107] It should be noted that for customized characters (or operators, etc.) not supported by the database to be used, the data query results matching the data to be processed need to be post-filtered until the final data query results meet all conditions in the data to be processed.

[0108] The post-filtering can be implemented by setting a preset rule filter set, wherein the preset rule filter set includes multiple preset filter rules, for example, at least one of an equal filter rule, a like filter rule, a between filter rule, a contain filter rule, and a not contain filter rule. A plurality of different filter rules are used to further process the data query result, so that the data query result finally obtained satisfies all conditions in the data to be processed, thereby improving the accuracy of the data.

[0109] The near real-time data statistics module 624 is used to receive the changed data sent by the terminal and send the changed data to the message queue when it is determined that the business data sending is changed; the near real-time data statistics module 624 obtains the changed data by subscribing to the change message of the message queue, uses the cached multiple alternative semantic trees to match the changed data, and based on the multiple preset filtering rules in the preset rule filtering set, the changed data is judged layer by layer to determine the rule level corresponding to the changed data in the target semantic tree, and then, based on the target semantic tree, the query statement corresponding to the changed data is determined; based on the query statement corresponding to the changed data, the database to be used is queried to obtain the data query result.

[0110] After completing the above data query, the target semantic tree can be updated according to the data query result, so that the target semantic tree can be used to process the subsequently received data, thereby improving the data processing efficiency.

[0111] Figure 7 The following is a flow chart showing the working method of the data processing system provided by the embodiment of the present application. Figure 7 As shown, the working method of the data processing system includes but is not limited to the following steps.

[0112] Step S701: the terminal generates data to be processed and sends the data to be processed to the server.

[0113] The data to be processed may be data generated by the terminal during business processing.

[0114] Step S702: the server matches the format of the data to be processed with a plurality of candidate semantic trees to determine a target semantic tree corresponding to the data to be processed.

[0115] The server may first use metadata of each network element to perform modeling to form multiple candidate semantic trees. The metadata of the network element may include field name, field type, field length, and field Chinese information.

[0116] For example, the metadata information in the network element data includes: English name (eg, location), Chinese name (eg, location), field type (eg, String), field length (eg, 200), and the like.

[0117] It should be noted that metadata is data that describes other data (for example, metadata is data that describes objects such as information resources or data), or structured data used to provide relevant information about a certain resource.

[0118] The purpose of using metadata is to identify resources, evaluate resources, and track changes in resources during use; to achieve simple and efficient management of large amounts of networked data; to achieve effective discovery, search, integrated organization of information resources and effective management of used resources.

[0119] Once metadata is created, it can be shared. The structure and integrity of metadata depend on the value of information resources and the environment in which they are used; the environment in which metadata is developed and used is often a changing distributed environment. Metadata can be a coding system. Metadata is a coding system used to describe digital information resources, especially network information resources; the most important feature and function of metadata is to establish a machine-understandable framework for digital information resources. Metadata can be stored in a database to facilitate reading of metadata.

[0120] For example, the rule parsing module 621 is used to analyze the received preset data processing statements to identify key information such as connectors, operators, fields and values; a preset data processing unit is generated based on the identified multiple different types of information; and then an alternative semantic tree is constructed based on multiple preset data processing units.

[0121] The preset data processing statement can be Figure 5The statement configured in the rule interface shown. For example, the preset data processing statement is a custom SQL statement, and the custom SQL statement supports the statement execution rules in the conventional SQL database, which also includes connectors such as "and" and "or", and priority calculation operators such as "{}", and supports operators such as "=, like, between, in, contain, not contain, group by".

[0122] For example, the preset data processing statement is: {location = Xi'an or location like Hai} and productInfo not contain LTE TDD group by moc. When parsing the preset data processing statement, the following method can be used:

[0123] First, the spaces, keywords, line breaks, etc. in the preset data processing statements are segmented to obtain multiple data processing objects (for example, single words: location, Xi'an, Hai, productInfo, LTE TDD, moc, etc.) and multiple processing rules (for example, processing rules include: =, or, ike, and, not contain, group by, etc.)

[0124] Then, based on the priority information in the SQL statement, at least one processing rule is matched with multiple data objects to be matched, and a preset data processing unit corresponding to the preset data processing statement is generated. For example, based on a custom SQL statement, the brackets (such as "{}") have the highest priority; when there are both "and" and "or" connectors, the priority of "and" is higher than the priority of "or", etc. Therefore, the above preset data processing statement can be split into the following preset data processing units, namely "location = Xi'an or location like Hai", "productInfo not containLTE TDD", and "group by moc".

[0125] Further, based on the above-mentioned multiple preset data processing units, an alternative semantic tree is constructed. For example, with the operator (or connector) as the center, the preset data processing unit is characterized as a "keyword-operator (or connector)-value range" form, and then, based on the priority of each operator and connector, an alternative semantic tree is constructed, for example, Figure 2 Alternative semantic trees shown.

[0126] It should be noted that the server will also cache the candidate semantic trees obtained by the above analysis into the database to facilitate subsequent data analysis. Since different databases have different execution statements and corresponding different execution rules, it is necessary to map the preset data processing statements based on the preset mapping relationship information that can reflect the mapping relationship between the preset data processing statements and the execution statements in the database to be used, so as to facilitate compatibility with different databases.

[0127] Step S703, when it is determined to perform offline data query, the query statement corresponding to the data to be processed is determined according to the target semantic tree corresponding to the data to be processed, and the database to be used is queried based on the query statement corresponding to the data to be processed to obtain the data query result.

[0128] When the preset data processing statement obtained is relatively complex and cannot be directly adapted to the database to be used, the preset data processing statement can be disassembled based on the target semantic tree while keeping the semantics of the preset data processing statement unchanged to obtain multiple disassembled statements; the multiple disassembled statements are reorganized to obtain multiple query statements to be used that match the database to be used; then, the multiple query statements to be used are used respectively to query the database to be used, and the multiple results obtained from the query are processed by intersection and / or union, so as to obtain the final data query result.

[0129] For example, when the database to be used is a MySQL database, since the "notcontain" operator in the preset data processing statement is an operator not supported by the MySQL database, it is necessary to first delete "productInfo not containLTE TDD" and obtain the preliminary query statement corresponding to the data to be processed as "(location=Xi'an or locationlikeHai)group by moc". Then, the first query statement is input into the database to be used for query to obtain the first query result; then, multiple preset filtering rules in the preset rule filtering set are used to filter out the query results in the first query result that meet the "productInfo not contain LTE TDD" to obtain the final data query result.

[0130] For another example, when the database to be used is a PgSql database, if the obtained preset data processing statement is "nbiId=1and location not contain d". The "not contain" operator is also an operator not supported by the PgSql database, and it is necessary to first input "nbiId=1" into the database to be used for query to obtain the second query result; then, use multiple preset filtering rules in the preset rule filtering set to filter out the query results that meet "location not contain d" in the second query result, so as to obtain the final data query result.

[0131] Through customized preset data processing statements, more operators can be expanded to provide more data query methods and improve the accuracy of data query.

[0132] In some exemplary embodiments, a difference list may be established based on the difference information between the customized preset data processing statements and the statements supported by the database to be used, and the difference list may be cached on the server so that the customized preset data processing statements can be identified by querying the difference list, thereby converting the customized preset data processing statements into statements supported by the database to be used.

[0133] When used specifically, each candidate semantic tree is scanned from bottom to top to obtain the processing rules in the preset data processing unit in the candidate semantic tree, and the processing rules are compared with the operators in the difference list to obtain the statements that can be supported by the database to be used. For example, the priority of the custom character "{}" matches the priority of "()" used in the database to be used, and the above difference information needs to be written into the difference list. The difference list can also be cached in the server in the form of a configuration file.

[0134] Step S704: the server receives the changed data.

[0135] The changed data may be data obtained through a packet capture tool or a data monitoring tool, and the changed data is data with certain identifiers changed based on the data to be processed. For example, if the data to be processed is business data corresponding to the identifier "Shanghai", the changed data is business data corresponding to the identifier "Xi'an".

[0136] Step S705, when it is determined that the business data has changed, use the cached multiple alternative semantic trees to match the changed data, determine the rule level corresponding to the changed data in the target semantic tree, and determine the query statement corresponding to the changed data based on the target semantic tree; query the database to be used based on the query statement corresponding to the changed data to obtain the data query result.

[0137] Among them, when the server receives the changed data sent by the terminal, the server will send the changed data to the message queue; the near real-time data statistics module 624 obtains the changed data by subscribing to the change message of the message queue, matches the changed data using the cached multiple candidate semantic trees, and judges the changed data layer by layer based on the multiple preset filtering rules in the preset rule filtering set to determine the rule level corresponding to the changed data in the target semantic tree. Then, the query statement corresponding to the changed data is determined based on the target semantic tree; the database to be used is queried based on the query statement corresponding to the changed data to obtain the data query result.

[0138] For example, in Figure 3 In the alternative semantic tree shown, if the "nbiId" field changes from 1 to 0 and the values ​​of other fields remain unchanged, thereby obtaining the changed data, the server will send the changed data to the message queue; the near real-time data statistics module 624 obtains the changed data by subscribing to the change message of the message queue, and recognizes that the "nbiId" field has changed, and uses the cached multiple alternative semantic trees to match the changed data to determine the rule level corresponding to the changed data in the target semantic tree.

[0139] Further, based on multiple preset filtering rules in the preset rule filtering set, Figure 3 Each operator (such as operators "=", "like" and "group by") in the statement "{nbiId=1or nbiId=2}and location like a group by mimVersion" corresponding to the alternative semantic tree shown is checked in turn to obtain the query statement corresponding to the changed data; based on the query statement corresponding to the changed data, the database to be used is queried to obtain the data query result.

[0140] For another example, when business data changes, for example, the value of the "location" field in the data to be processed changes from "Beijing" to "Shanghai", the near real-time data statistics module 624 will receive the changed data through the message queue. Since the initial value of the "location" field is "Beijing", when the original target semantic tree is used to process the changed data, the query statement corresponding to the changed data cannot be found.

[0141] By obtaining the rule in the alternative semantic tree that may match it, which is "location like 海", it is known that the changed data with the value of the "location" field being "上海" satisfies the rule of "location like 海". When it is determined that other fields also satisfy this rule, it can be determined that the rule matching for the changed data is successful, and thus it is determined that the changed data should belong to this rule, and at the same time, the associated information of the unmatched rule is removed.

[0142] It should be noted that in addition to multiple alternative semantic trees, the rule information cached in the data processing system can also cache preset index information. The preset index information is the information obtained by arranging the cached multiple business information in reverse order based on the acquisition time of the business information. For example, the preset index information may include a preset business name and / or a preset business value, etc.

[0143] In some exemplary embodiments, the data to be processed further includes: incremental change data in real-time data. The incremental change data in real-time data represents the data in the business data obtained in real-time that has changed relative to the historical business data. In other words, the business data has changed, and the incremental change data in the real-time data among them is the changed data.

[0144] In the case where it is determined that the target semantic tree does not include the query statement corresponding to the incremental change data in the real-time data, parse the incremental change data in the real-time data to obtain business difference information. The business difference information includes a business name and / or a business value; match the preset index information with the business difference information to determine the target query rule; based on the target query rule and multiple query statements to be used, determine the query statement corresponding to the data to be processed.

[0145] In some exemplary embodiments, multiple operator filters can be established based on the operators supported by the data processing system (for example, multiple preset filtering rules in the preset rule filtering set), and they are connected in the form of a responsibility chain to form a data change filtering chain. For example, Figure 8 Show a schematic diagram of the data change filtering chain provided by the embodiments of the present application. As Figure 8 shown, the data change filtering chain 800 includes: an equal filter 801, a like filter 802, a between filter 803, a contain filter 804, and a Not contain filter 805.

[0146] By using multiple filters in the data change filtering chain 800 to verify the data query result obtained based on the target semantic tree, so as to perform a supplementary query on the statements in the target semantic tree, fully meet the query requirements of the data to be processed, and make the finally obtained data query result more accurate.

[0147] For example, if the value of the "location" field in the data to be processed changes from "Beijing" to "Shanghai", the rule that may match it in the alternative semantic tree is "location like 海". Since this rule does not contain other operators (only includes the "like" operator), during the data processing, the equal filter 801 can be skipped, and only the value of the data to be processed is matched in the like filter 802, thus simplifying the data processing steps. After the above processing, it can be known that the data to be processed with the value of the "location" field being "Shanghai" satisfies the rule of "location like 海".

[0148] For another example, if the obtained preset data processing statement is complex. For example, the preset data processing statement is "{location = Xi'an or location like 海} and productInfo not contain LTE TDD", then the rule verification can be performed on the target semantic tree corresponding to this preset data processing statement from bottom to top. That is, first execute "location = Xi'an" and "location like 海" respectively to query the database to be used, and take the union of the execution results corresponding to the two rules to obtain the first processing result; then, execute "productInfo not contain LTETDD" to query the database to be used to obtain the second processing result; further, take the intersection of the first processing result and the second processing result to obtain the processing result to be confirmed. By comparing the matching degree between the processing result to be confirmed and the preset data processing statement "{location = Xi'an or location like 海} and productInfo not contain LTE TDD", it is determined whether the processing result to be confirmed is the final data query result required by the user.

[0149] Step S706, perform persistent processing on the target semantic tree and the database to be used.

[0150] Among them, the model management module 622 can perform persistent operations on the database to be used.

[0151] Among them, the persistent operation can update the database to be used by means of a sliding time window. In other words, according to the preset interval duration and the incremental change data in the real-time data received within the preset interval duration, determine the update frequency of the database to be used; according to the update frequency of the database to be used, use the incremental change data in the real-time data to update the database to be used, thereby reducing the number of write operations on the database to be used and reducing the data processing pressure on the database to be used.

[0152] In this embodiment, the server constructs multiple alternative semantic trees corresponding to the preset data processing statements based on preset mapping relationship information and multiple customized preset data processing statements. The alternative semantic trees can reflect the logical relationship between different statements based on multiple different hierarchical structures. There is no need for R&D personnel to use special processing software to parse the data to be processed, and the data structure can be dynamically changed based on at least one customized preset data processing statement. The server can also make attribution judgments on the data to be processed and the changed data based on multiple alternative semantic trees, and determine the target semantic tree corresponding to the data to be processed to meet the processing requirements of the data to be processed. Further, based on the target semantic tree corresponding to the data to be processed, the query statement corresponding to the data to be processed is determined, so that the query statement corresponding to the data to be processed can be applied to the database to be used, thereby speeding up the query speed of the data. The database to be used is queried based on the query statement corresponding to the data to be processed to obtain a data query result, so that the data query result reflects the real-time changes in business data, and there is no need to redeploy or write related business codes based on the needs of statistical business, thereby reducing development costs and improving business data processing efficiency.

[0153] Fig. 9 FIG. 1 is a block diagram showing the composition of a first data processing device provided in an embodiment of the present application. Fig. 9 As shown, the first data processing device 900 includes but is not limited to the following modules.

[0154] The construction module 901 is configured to construct a plurality of candidate semantic trees corresponding to the preset data processing statements based on the preset mapping relationship information and a plurality of user-defined preset data processing statements.

[0155] The preset mapping relationship information is information representing the mapping relationship between the preset data processing statements and the execution statements in the database to be used.

[0156] The matching module 902 is configured to match the acquired format of the data to be processed with a plurality of candidate semantic trees to determine a target semantic tree corresponding to the data to be processed.

[0157] The determination module 903 is configured to determine the query statement corresponding to the data to be processed according to the target semantic tree corresponding to the data to be processed.

[0158] The query module 904 is configured to query the database to be used based on the query statement corresponding to the data to be processed to obtain the data query result.

[0159] It should be noted that the first data processing device 900 in this embodiment can implement any data processing method applied to the server in the embodiments of the present application.

[0160] According to the first data processing device of the embodiment of the present application, a plurality of alternative semantic trees corresponding to the preset data processing statements are respectively constructed through a construction module based on preset mapping relationship information and a plurality of customized preset data processing statements. The alternative semantic trees can reflect the logical relationship between different statements based on a plurality of different hierarchical structures, and there is no need for R&D personnel to use special processing software to parse the data to be processed, and dynamic changes to the data structure can be achieved based on at least one customized preset data processing statement; wherein the preset mapping relationship information is information characterizing the mapping relationship between the preset data processing statement and the execution statement in the database to be used, which can make the execution statement in the database to be used correspond to the customized data processing statement. sentences, flexibly process the data in the database, and improve the data processing efficiency; use the matching module to match the acquired format of the data to be processed with multiple candidate semantic trees, and determine the target semantic tree corresponding to the data to be processed to adapt to the processing requirements of the data to be processed; further, use the determination module to determine the query statement corresponding to the data to be processed based on the target semantic tree corresponding to the data to be processed, so that the query statement corresponding to the data to be processed can be applied to the database to be used, thereby speeding up the query speed of the data; use the query module to query the database to be used based on the query statement corresponding to the data to be processed, and obtain the data query result, so that the data query result reflects the real-time changes in business data.

[0161] Fig.10 FIG. 2 shows a block diagram of the second data processing device provided in an embodiment of the present application. Fig.10 As shown, the second data processing device 1000 includes but is not limited to the following modules.

[0162] The generating module 1001 is configured to generate to-be-processed data in a corresponding format in response to a configuration instruction.

[0163] The configuration instruction includes at least one customized processing rule and multiple data processing objects.

[0164] In some optional embodiments, the configuration instruction further includes: application scenario configuration information and / or device type configuration information, wherein at least one processing rule and multiple data processing objects all match the device type configuration information.

[0165] The sending module 1002 is configured to send the data to be processed to the server.

[0166] When the server receives the data to be processed, it will match the format of the data to be processed with multiple alternative semantic trees to determine the target semantic tree corresponding to the data to be processed; based on the target semantic tree corresponding to the data to be processed, it will determine the query statement corresponding to the data to be processed; based on the query statement corresponding to the data to be processed, it will query the database to be used to obtain the data query result.

[0167] It should be noted that the second data processing device 1000 in this embodiment can implement any data processing method applied to a terminal in the embodiments of the present application.

[0168] According to the second data processing device of the embodiment of the present application, a generation module is used to respond to a configuration instruction to generate data to be processed in a corresponding format. The data to be processed is determined based on a user's customization of data processing rules. There is no need for R&D personnel to use special processing software to parse the data to be processed. The data structure can be dynamically changed based on at least one customized processing rule and preset data processing statements generated by multiple data processing objects, thereby accelerating the processing speed of the data; then, the data to be processed is sent to the server using a sending module, so that the server matches the data to be processed with multiple candidate semantic trees according to the format of the data to be processed, and determines a target semantic tree corresponding to the data to be processed to meet the processing requirements of the data to be processed; based on the target semantic tree corresponding to the data to be processed, a query statement corresponding to the data to be processed is determined, so that the query statement corresponding to the data to be processed can be applied to the database to be used, thereby accelerating the query speed of the data; based on the query statement corresponding to the data to be processed, the database to be used is queried to obtain a data query result, so that the data query result reflects the real-time changes in business data.

[0169] It should be clear that the present application is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the convenience and brevity of description, a detailed description of the known methods is omitted here, and the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here.

[0170] Fig.11 A structural diagram showing an exemplary hardware architecture of a computing device capable of implementing the data processing method and apparatus according to an embodiment of the present application.

[0171] like Fig.11As shown, the computing device 1100 includes an input device 1101, an input interface 1102, a central processing unit 1103, a memory 1104, an output interface 1105, and an output device 1106. The input interface 1102, the central processing unit 1103, the memory 1104, and the output interface 1105 are connected to each other via a bus 1107, and the input device 1101 and the output device 1106 are connected to the bus 1107 via the input interface 1102 and the output interface 1105, respectively, and are further connected to other components of the computing device 1100.

[0172] Specifically, the input device 1101 receives input information from the outside and transmits the input information to the central processing unit 1103 through the input interface 1102; the central processing unit 1103 processes the input information based on the computer executable instructions stored in the memory 1104 to generate output information, temporarily or permanently stores the output information in the memory 1104, and then transmits the output information to the output device 1106 through the output interface 1105; the output device 1106 outputs the output information to the outside of the computing device 1100 for user use.

[0173] In one embodiment, Fig.11 The computing device shown may be implemented as an electronic device, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to execute the data processing method described in the above embodiment.

[0174] In one embodiment, Fig.11 The computing device shown may be implemented as a data processing system, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to execute the data processing method described in the above embodiment.

[0175] The above is only an exemplary embodiment of the present application and is not intended to limit the scope of protection of the present application. In general, various embodiments of the present application can be implemented in hardware or special circuits, software, logic or any combination thereof. For example, some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, a microprocessor or other computing device, although the present application is not limited thereto.

[0176] Embodiments of the present application may be implemented by a processor executing computer program instructions, such as in a processor entity, or by hardware, or by a combination of software and hardware. The computer program instructions may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.

[0177] The block diagram of any logic flow in the accompanying drawings of the present application can represent program steps, or can represent interconnected logic circuits, modules and functions, or can represent a combination of program steps and logic circuits, modules and functions. The computer program can be stored on a memory. The memory can have any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as but not limited to read-only memory (ROM), random access memory (RAM), optical memory device and system (digital versatile disc DVD or CD disc), etc. Computer-readable media may include non-transient storage media. The processor can be any type suitable for the local technical environment, such as but not limited to a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (FGPA) and a processor based on a multi-core processor architecture.

[0178] By way of exemplary and non-limiting examples, a detailed description of exemplary embodiments of the present application has been provided above. However, various modifications and adjustments to the above embodiments will be apparent to those skilled in the art when considered in conjunction with the accompanying drawings and claims, but will not depart from the scope of the present application. Therefore, the proper scope of the present application will be determined according to the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Based on the preset mapping relationship information and the multiple customized preset data processing statements, construct multiple candidate semantic trees corresponding to the preset data processing statements respectively, wherein the preset mapping relationship information is information representing the mapping relationship between the preset data processing statements and the execution statements in the database to be used; According to the format of the acquired data to be processed, the target semantic tree corresponding to the data to be processed is matched with a plurality of candidate semantic trees to determine; Determine the query statement corresponding to the data to be processed according to the target semantic tree corresponding to the data to be processed; The database to be used is queried based on the query statement corresponding to the data to be processed to obtain a data query result.

2. The method according to claim 1, characterized in that The data to be processed includes: incremental change data in real-time data, wherein the incremental change data in real-time data represents data in which the business data acquired in real time has changed relative to the historical business data; The step of determining the query statement corresponding to the data to be processed according to the target semantic tree corresponding to the data to be processed includes: When it is determined that the target semantic tree does not include a query statement corresponding to the incremental change data in the real-time data, parsing the incremental change data in the real-time data to obtain service difference information, where the service difference information includes a service name and / or a service value; Matching the preset index information with the business difference information to determine a target query rule, wherein the preset index information is information obtained by arranging the cached multiple business information in reverse order based on the acquisition time of the business information; Based on the target query rule and the plurality of query statements to be used, a query statement corresponding to the data to be processed is determined.

3. The method according to claim 2, characterized in that Before determining the query statement corresponding to the data to be processed based on the target query rule and the plurality of query statements to be used, the data processing method further includes: Based on a plurality of preset query rules in a preset rule filtering set, query rules matching the business difference information are verified to determine the target query rule.

4. The method according to claim 2, characterized in that: After querying the database to be used based on the query statement corresponding to the data to be processed and obtaining the data query result, the data processing method further includes: Determining the update frequency of the database to be used according to a preset interval time and incremental change data in the real-time data received within the preset interval time; According to the update frequency of the database to be used, the database to be used is updated using the incremental change data in the real-time data.

5. The method according to claim 1, characterized in that The data to be processed also includes: historical business data; The querying the database to be used based on the query statement corresponding to the data to be processed to obtain the data query result includes: Under the condition that the semantics of the historical query statements corresponding to the historical business data remain unchanged, the historical query statements corresponding to the historical business data are disassembled to obtain a plurality of disassembled statements; Reorganize the plurality of disassembled statements to obtain a plurality of query statements to be used that match the database to be used; The database to be used is queried using the plurality of query statements to be used respectively to obtain the data query result.

6. The method according to claim 5, characterized in that The step of respectively using the plurality of query statements to be used to query the database to be used to obtain the data query result includes: Using the plurality of query statements to be used respectively to query the database to be used, to obtain a plurality of data sets to be processed; Intersection processing is performed on the multiple data sets to be processed, and / or union processing is performed on the multiple data sets to be processed to obtain the data query result.

7. The method according to any one of claims 1 to 6, characterized in that The method of constructing a plurality of candidate semantic trees corresponding to the preset data processing statements based on the preset mapping relationship information and the plurality of customized preset data processing statements respectively includes: Obtaining a plurality of the preset data processing statements; Analyze each of the preset data processing statements respectively to determine a preset data processing unit corresponding to each of the preset data processing statements; According to the preset data processing unit corresponding to each of the preset data processing statements, an alternative semantic tree corresponding to each of the preset data processing statements is constructed.

8. The method according to claim 7, characterized in that The preset data processing unit includes at least one processing rule and a plurality of data processing objects, and the analyzing each of the preset data processing statements to determine the preset data processing unit corresponding to each of the preset data processing statements includes: According to at least one processing rule corresponding to each of the preset data processing statements, the plurality of data processing objects are segmented to obtain a plurality of data objects to be matched; The at least one processing rule is matched with the multiple data objects to be matched respectively, and a preset data processing unit corresponding to each preset data processing statement is generated.

9. The method according to claim 8, characterized in that The processing rules include: operation rules and / or connection rules; the data processing object includes at least one of the data type of the business data, the data name of the business data, and the value range of the business data; The preset mapping relationship information includes: first mapping relationship information matching the execution statement in the database to be used, and / or second mapping relationship information determined based on a configuration file, wherein the second mapping relationship information includes at least one of operation rule difference information, character difference information and string difference information.

10. The method according to claim 7, characterized in that The step of constructing a candidate semantic tree corresponding to each of the preset data processing statements according to the preset data processing unit corresponding to each of the preset data processing statements includes: For the preset data processing unit corresponding to each of the preset data processing statements, based on the priority of the processing rules, the level to which each of the processing rules belongs is determined step by step; and according to the level to which each of the processing rules belongs, the multiple preset data processing units corresponding to the processing rules are hierarchically processed to construct an alternative semantic tree corresponding to the preset data processing statements.

11. The method according to any one of claims 1 to 6, characterized in that After querying the database to be used based on the query statement corresponding to the data to be processed and obtaining the data query result, the data processing method further includes: The target semantic tree is updated according to the data query result.

12. A data processing method, characterized in that: The method comprises: In response to the configuration instruction, generating data to be processed in a corresponding format; wherein the configuration instruction includes at least one customized processing rule and a plurality of data processing objects; The data to be processed is sent to the server.

13. The method according to claim 12, characterized in that The configuration instruction further includes: application scenario configuration information and / or device type configuration information, and the at least one processing rule and the multiple data processing objects all match the device type configuration information.

14. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 13.

15. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 13 is implemented.

Images