Data processing methods, apparatus, storage media and electronic devices

Abstract

This application discloses a data processing method, apparatus, storage medium, and electronic device, relating to the field of smart home technology. The data processing method includes: acquiring data to be processed, wherein the data to be processed is object JSON format data; parsing target data from the data to be processed using a first structured query language (SQL) statement in an SDK engine, wherein the SDK engine is a plug-in device in the open-source stream processing framework Flink engine; determining the processing method corresponding to the target data using a second SQL statement; and processing the target data according to the processing method.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the massive influx of data, and to meet the industry's demands for rapid development and iteration, the data sources used by various interfaces are becoming increasingly diverse and unpredictable, leading to hybrid data processing models. For the same business requirement, multiple data formats need to be integrated. Furthermore, these data formats are constantly changing, and multiple business needs, even multiple projects, require diverse filtering of the same data format. For example, data in the JavaScript Object Notation (JSON) format from IoT devices is a lightweight data exchange format and near real-time data, requiring businesses to quickly obtain target data and instructions. In the bidirectional flow process, a hybrid processing approach is used. During development, due to the diverse changes in data before and after the data flow, and the differences in data structure between the two systems caused by minor changes in data format on one side during multi-department collaboration, the computing engine adaptation layer needs to identify different data sources and correspondingly generate different code logic, resulting in high code maintenance costs and poor readability. Summary of the Invention

[0003] This invention provides a data processing method and apparatus, a storage medium, and an electronic device to at least solve the problem of low data processing efficiency in related technologies.

[0004] According to an embodiment of the present invention, a data processing method is provided, comprising: acquiring data to be processed, wherein the data to be processed is data in JSON object format; parsing target data from the data to be processed using a first structured query language (SQL) statement in the aforementioned SDK engine, wherein the aforementioned SDK engine is a plug-in device in the Flink engine of the open-source stream processing framework; determining a method corresponding to the aforementioned target data using a second SQL statement; and processing the aforementioned target data according to the aforementioned processing method.

[0005] According to another embodiment of the present invention, a data processing apparatus is provided, comprising: a first acquisition module for acquiring data to be processed, wherein the data to be processed is object JSON format data; a first parsing module for parsing target data from the data to be processed using a first structured query language (SQL) statement in the SDK engine, wherein the SDK engine is a plug-in device in the open-source stream processing framework Flink engine; and a first determination module for determining a processing method corresponding to the target data using a second SQL statement; and processing the target data according to the processing method.

[0006] In one exemplary embodiment, the first acquisition module includes: a first invocation unit, configured to invoke the application programming interface (API) in the SDK engine to receive JSON-formatted data from the data stream transmitted by the Flink engine, thereby obtaining the data to be processed; and a first caching unit, configured to cache the data to be processed.

[0007] In one exemplary embodiment, the apparatus further includes a second determining module, configured to determine a parsing tool before parsing target data from the data to be processed using a first Structured Query Language (SQL) statement in the SDK engine, wherein the parsing tool is determined based on regular expressions in object-oriented Java, and the parsing tool includes information about keywords in the SQL statement.

[0008] In an exemplary embodiment, the first parsing module includes: a first conversion unit, configured to convert the first SQL statement into an object of first SQL rules using the parsing tool, wherein the object of first SQL rules has the ability to read data in JSON format; and a first parsing unit, configured to parse the target data from the data to be processed using the object of first SQL rules.

[0009] In an exemplary embodiment, the first determining module includes: a second conversion unit, configured to convert the second SQL statement into an object of second SQL rules using the parsing tool, wherein the object of the second SQL rules has the ability to read data in JSON format; and a first search unit, configured to search for the processed data from the JSON format data using the object of the second SQL rules.

[0010] In one exemplary embodiment, the apparatus further includes: a first processing module, configured to determine the processing data corresponding to the target data using a second SQL statement, and then return the processing data to the Flink engine to instruct the Flink engine to send the processing data to the device.

[0011] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0012] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0013] This invention involves acquiring data to be processed; parsing target data from the data using a first structured query language (SQL) statement in an SDK engine (a plugin device within the open-source stream processing framework Flink); and determining the processing method corresponding to the target data using a second SQL statement, followed by processing the target data according to the specified method. Because this method uses the SDK engine as a communication bridge between the business side and the data source, it ensures that the business side can query data in different JSON formats using SQL statements without needing to lay out different code logic for different JSON formats. Processing JSON data can be achieved simply by using SQL statements, without being limited by the data format, thus improving data processing efficiency. Therefore, it solves the problem of low data processing efficiency in related technologies. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the hardware environment for a data processing method according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of a data processing method according to an embodiment of the present invention;

[0018] Figure 3 This is a timing diagram according to an embodiment of the present invention;

[0019] Figure 4 This is a structural block diagram of a data processing apparatus according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] According to one aspect of the embodiments of this application, a data processing method is provided. This data processing method is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligencehouse ecosystems. Optionally, in this embodiment, the above-mentioned data processing method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0023] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0024] This embodiment provides a data processing method. Figure 2 This is a flowchart of a data processing method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0025] Step S202: Obtain the data to be processed, wherein the data to be processed is data in JSON object format;

[0026] Step S204: Parse the target data from the data to be processed using the first structured query language SQL statement in the SDK engine. The SDK engine is a plug-in device in the Flink engine of the open-source stream processing framework.

[0027] Step S206: Use the second SQL statement to determine the processing method corresponding to the target data;

[0028] Step S208: Process the target data according to the processing method.

[0029] Optionally, the above data processing methods can be applied to the processing of data sources in various formats, but are not limited to these methods. The data to be processed in JSON format can be a large volume of real-time data transmitted from multiple parties. This JSON format data includes various formats, such as JSONArray, JSONObject, multi-layered JSON, and single-layered JSON.

[0030] Optionally, the SDK engine is a separate engine used internally within Flink as a plugin, distinct from the Flink compute engine. Flink application business logic needs to depend on the SDK engine within the project. The Flink compute engine integrates and consumes data; the Flink business logic inputs the corresponding SQL and the JSON data to be queried according to the interface conventions provided by the SDK engine. After receiving the data, the SDK engine executes the SQL conditions and returns the processed data to the Flink compute stream for further data processing and to obtain the requested result.

[0031] Optionally, this embodiment does not require different code logic for different JSON formats. It only needs to use SQL statements to identify JSON format data and query the data processing method. SQL statements are not limited by the data format, which can improve the efficiency of data processing.

[0032] The entity performing the above steps may be a terminal, a server, a specific processor set in the terminal or server, or a processor or processing device set up relatively independently of the terminal or server, but is not limited to these.

[0033] Through the above steps, the process involves: acquiring the data to be processed; parsing the target data from the data using the first structured query language (SQL) statement in the SDK engine (a plugin device within the open-source stream processing framework Flink); and determining the processing method corresponding to the target data using a second SQL statement, followed by processing the target data according to that method. Because this method uses the SDK engine as a communication bridge between the business side and the data source, it ensures that the business side can query data in different JSON formats using SQL statements without needing to lay out different code logic for different JSON formats. Processing JSON data can be achieved simply by using SQL statements, without being limited by the data format, thus improving data processing efficiency. Therefore, it can solve the problem of low data processing efficiency in related technologies.

[0034] In one exemplary embodiment, obtaining the data to be processed in JSON format includes:

[0035] S1 calls the application programming interface (API) in the SDK engine to receive JSON-formatted data from the data stream transmitted by the Flink engine and obtain the data to be processed;

[0036] S1 caches data to be processed.

[0037] Optionally, the Flink engine integrates an SDK engine. After successful project startup, it begins consuming upstream JSON data, calling the SDK engine's Application Programming Interface (API). When the API is triggered, it automatically parses SQL statements, stores the results in a cache, and returns the parsed values. For example, the Flink engine can acquire real-time JSON data streams from home appliances, transmit the JSON data stream to the SDK engine via the API, and the SDK engine uses SQL statements to parse the JSON data stream. The parsed content includes retrieving necessary data from the JSON data stream, such as address data. The JSON data stream also includes device data transmitted by the home appliances, such as fault data and operational data. This allows for parsing of JSON data streams without complex code, requiring only SQL statements.

[0038] In one exemplary embodiment, before parsing the target data from the data to be processed using a first Structured Query Language (SQL) statement in the SDK engine, the method further includes:

[0039] S1, determine the parsing tool, which is determined based on regular expressions in object-oriented Java, and includes information about keywords in the SQL statement.

[0040] Optionally, the SDK engine includes a parsing tool that parses SQL statements into objects capable of reading JSON data. The parsing tool uses underlying Java regular expressions, including keyword information. For example, the parsing tool's code might include:

[0041] static final String matchPr=" / ^(select)\\s+([a-z0-9_\\,\\.\\s\\*]+)\\s+from\\s+([a-z0-9_\\.]+)(?:where\\s+\\((.+)\\))?\\ s*(?:order\\sby\\s+([a-z0-9_\\,]+))? \\s*(asc|desc|ascnum|descnum)? \\s*(?:limit\\s+([0-9_\\,]+))?

[0042] static final String matchPr="^(select)\\s+([a-zA-Z0-9_\\'\\(\\)\\+\\-\\,\\[\\]\\.\\s\\*]+)\\s+from\\s+([a-zA-Z0-9_\\.\\-]+)(?:where\\s+ \\((.+)\\))? \\s*(?:order\\sby\\s+([a-zA-Z0-9_\\,]+))? \\s*(?:limit\\s+([0-9_\\,]+))?

[0043] static final Stringfield="([a-zA-Z0-9_\\'\\(\\)\\+\\-\\*\\ / \\[\\]\\.\\*]+)\\s+[as|AS|aS|As]+\\s+([a-zA-Z0-9_\\[\\]\\.\\*]+)";

[0044] Optionally, the information of keywords in the SQL statement mentioned above is only used to identify the scenario in which the regular expression is applied, that is, the scenario in which the SQL statement is parsed, and does not mean that it has the function of an SQL statement.

[0045] In one exemplary embodiment, parsing target data from the data to be processed using a first structured query language (SQL) statement in the SDK engine includes:

[0046] S1, use a parsing tool to convert the first SQL statement into an object of the first SQL rule, wherein the object of the first SQL rule has the ability to read data in JSON format;

[0047] S2 uses the object of the first SQL rule to parse the target data from the data to be processed.

[0048] Optionally, the object of the first SQL rule is defined based on regular expressions and has the ability to read data in JSON format. For example, SQL statements can be used to find address data URLs from fault information of home appliances, and then the corresponding processing data can be found based on the address data URLs.

[0049] In one exemplary embodiment, the second SQL statement is used to determine the processing data corresponding to the target data, including:

[0050] S1, use a parsing tool to convert the second SQL statement into an object of the second SQL rule, wherein the object of the second SQL rule has the ability to read data in JSON format;

[0051] S2 uses the object of the second SQL rule to find the data to be processed from the JSON format data.

[0052] Optionally, the second SQL statement can be a statement that searches for and processes data, such as searching for troubleshooting methods for home appliances.

[0053] In one exemplary embodiment, after determining the processing data corresponding to the target data using the second SQL statement, the method further includes:

[0054] S1 returns the processed data to the Flink engine to instruct the Flink engine to send the processed data to the device.

[0055] Optionally, the Flink engine distributes data to the final output based on business logic, and further distributes it to users or devices according to the business requirements. Users or devices can quickly obtain the processing method, thus accelerating data processing and improving the user experience.

[0056] The present invention will now be described in conjunction with specific embodiments:

[0057] This embodiment is applied to scenarios involving the processing of real-time and near-real-time JSON data reported by IoT devices. In this scenario, various real-time stream processing business projects are rapidly developing, with constantly changing business requirements and data input formats. This has resulted in a large number of data access format projects. Even slight changes in data format can lead to inconsistencies between the data format and configuration format. Cross-departmental, cross-team, and even cross-company data access has become commonplace. Furthermore, even with the same data source, IoT-reported data can still deviate, posing a significant challenge to fixed data source formats and structures. This also presents a major challenge for multi-terminal data usage, resulting in extremely complex adaptation processes, bloated code, and reduced maintainability and readability. This embodiment addresses this by adding an SDK engine between the business code and the data source, acting as a communication bridge between the business and the source data. For the business side, the SDK engine ensures that the data format received downstream is consistent across different data sources. The underlying logic processing side does not need to worry about these differences, ensuring that the maintainability and readability of the business code remain at a good level.

[0058] like Figure 3 The diagram shown is a sequence diagram in this embodiment, including the position of the SDK engine in the entire business logic and the sequence diagram of the entire business logic. The Flink application business side needs to rely on the SDK engine in the project. The SDK engine is separate from the Flink computing engine and is used internally within Flink as a plugin. The Flink engine integrates data consumption; the Flink business side passes in the corresponding SQL and the JSON data to be queried according to the interface conventions provided by the SDK engine. After receiving the JSON data, the SDK engine executes the SQL conditions and can return the operation data to the Flink computing stream, thereby enabling further data processing and obtaining the requested result. Specifically, this includes the following steps:

[0059] S1. The Flink business side first introduces the SDK engine. After the project starts successfully, it begins to consume upstream JSON data and calls the API interface of the SDK engine. After the API interface of the SDK engine is triggered, it will automatically parse the SQL language. After successful parsing, the data is stored in the cache and the parsing result is returned.

[0060] After the S2 SDK engine successfully parses the data, it directly queries and matches the JSON data in the data stream according to the parsing specification. After data filtering, querying, and transformation are completed, the result data is sent to the next stage for further processing. This process is repeated until the business requirements are met, and the processed data is returned to the interface caller.

[0061] S3, after the data stream is processed, is sent to the final output end according to the business logic. Depending on the business, it may be sent to users or devices.

[0062] In summary, this embodiment, through the parsing layer of the SDK engine, eliminates the differences in Schema metadata structures among multiple data streams for Flink's streaming JSON data processing business. The SDK engine is responsible for smoothing out these differences and exposing a consistent set of API interfaces. The Flink business side introduces the SDK engine and directly uses the API interfaces exposed by the SDK engine, without the need to design the Schema metadata structure in advance. This greatly shortens the development cycle, allowing the Flink business side code to focus on the business requirements themselves, which is conducive to rapid project iteration.

[0063] This embodiment also provides a data processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0064] Figure 4 This is a structural block diagram of a data processing apparatus according to an embodiment of the present invention, such as... Figure 4 As shown, the device includes:

[0065] The first acquisition module 42 is used to acquire data to be processed, wherein the data to be processed is data in JSON object format;

[0066] The first parsing module 44 is used to parse target data from the data to be processed using the first structured query language SQL statement in the SDK engine. The SDK engine is a plug-in device in the Flink engine of the open-source stream processing framework.

[0067] The first determining module 46 is used to determine the processing data corresponding to the target data using a second SQL statement, wherein the processing data is used to represent the processing method of the target event in the target data.

[0068] The first processing module 48 is used to process the target data according to the processing method.

[0069] In one exemplary embodiment, the first acquisition module described above includes:

[0070] The first calling unit is used to call the application programming interface (API) in the SDK engine to receive JSON formatted data from the data stream transmitted by the Flink engine and obtain the data to be processed.

[0071] The first cache unit is used to cache the aforementioned data to be processed.

[0072] In one exemplary embodiment, the above-described apparatus further includes:

[0073] The second determining module is used to determine a parsing tool before parsing the target data from the data to be processed using the first structured query language (SQL) statement in the SDK engine. The parsing tool is determined based on regular expressions in object-oriented Java and includes information about keywords in the SQL statement.

[0074] In one exemplary embodiment, the first parsing module described above includes:

[0075] The first conversion unit is used to convert the first SQL statement into an object of the first SQL rule using the parsing tool, wherein the object of the first SQL rule has the ability to read data in JSON format.

[0076] The first parsing unit is used to parse the target data from the data to be processed using the object of the first SQL rule.

[0077] In one exemplary embodiment, the first determining module described above includes:

[0078] The second conversion unit is used to convert the second SQL statement into an object of second SQL rules using the parsing tool, wherein the object of the second SQL rules has the ability to read data in JSON format;

[0079] The first search unit is used to find the processed data from the JSON format data using the object of the second SQL rule.

[0080] In one exemplary embodiment, the above-described apparatus further includes:

[0081] The first processing module is used to determine the processing data corresponding to the target data using the second SQL statement, and then return the processing data to the Flink engine to instruct the Flink engine to send the processing data to the device.

[0082] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, Applications to software development kit (SDK) engines include: Obtain the data to be processed, wherein the data to be processed is data in JSON object format; The target data is parsed from the data to be processed using the first SQL statement in the SDK engine, wherein the SDK engine is a plug-in device in the Flink engine of the open-source stream processing framework; The second SQL statement is used to determine the processing method corresponding to the target data; The target data is processed according to the described processing method; Before parsing the target data from the data to be processed using the first SQL statement in the SDK engine, the method further includes: determining a parsing tool, wherein the parsing tool is determined based on regular expressions in object-oriented Java, and the parsing tool includes information about keywords in the first SQL statement; The step of parsing target data from the data to be processed using the first SQL statement in the SDK engine includes: using the parsing tool to convert the first SQL statement into an object of first SQL rules, wherein the object of the first SQL rules has the ability to read data in JSON format; and using the object of the first SQL rules to parse the target data from the data to be processed. Determining the processing data corresponding to the target data using a second SQL statement includes: using the parsing tool to convert the second SQL statement into an object of a second SQL rule, wherein the object of the second SQL rule has the ability to read data in JSON format; and using the object of the second SQL rule to find the processing data from the JSON format data.

2. The method according to claim 1, characterized in that, Obtain the data to be processed, including: The application programming interface (API) in the SDK engine is called to receive JSON-formatted data from the data stream transmitted by the Flink engine, and the data to be processed is obtained. The data to be processed is cached.

3. The method according to claim 1, characterized in that, After determining the processing data corresponding to the target data using a second SQL statement, the method further includes: The processed data is returned to the Flink engine to instruct the Flink engine to send the processed data to the device.

4. A data processing apparatus, characterized in that, include: The first acquisition module is used to acquire data to be processed, wherein the data to be processed is data in JSON object format; The first parsing module is used to parse target data from the data to be processed using the first SQL statement in the SDK engine, wherein the SDK engine is a plug-in device in the Flink engine of the open-source stream processing framework. The first determining module is used to determine the processing method corresponding to the target data using a second SQL statement; The first processing module is used to process the target data according to the processing method described above; Before the device uses the first SQL statement in the SDK engine to parse the target data from the data to be processed, it is also used to determine a parsing tool, wherein the parsing tool is determined based on regular expressions in object-oriented Java, and the parsing tool includes information about the keywords in the first SQL statement; The first parsing module is further configured to use the parsing tool to convert the first SQL statement into an object of first SQL rules, wherein the object of first SQL rules has the ability to read data in JSON format; and to use the object of first SQL rules to parse the target data from the data to be processed. The apparatus is further configured to use the parsing tool to convert the second SQL statement into an object of second SQL rules, wherein the object of the second SQL rules has the ability to read data in JSON format; and to use the object of the second SQL rules to find the processed data from the JSON format data.

5. The apparatus according to claim 4, characterized in that, The first acquisition module includes: The first calling unit is used to call the application programming interface (API) in the SDK engine to receive JSON formatted data from the data stream transmitted by the Flink engine, and obtain the data to be processed. The first cache unit is used to cache the data to be processed.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 3.

7. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 3 through the computer program.

Images