Data processing method and device, equipment, medium and program product

By parsing configuration files and mapping unstructured data into memory tables that can be queried by standard query languages, the problem of low utilization of unstructured data is solved and data analysis efficiency of data lakes and data warehouses is improved.

CN119066111APending Publication Date: 2024-12-03BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310636441.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, unstructured data has low utilization and efficiency in use, especially in data lakes. There is a lack of effective methods to convert it into memory tables that can be directly queried in standard query languages, resulting in inefficient data analysis.

Method used

By parsing the configuration file, obtaining the abstract syntax tree of metadata and data processing logic, determining the target deserialization method, deserializing the source data, obtaining the target memory object, and mapping it into a memory table available in the standard query language .

Benefits of technology

Improves the utilization and efficiency of unstructured data, simplifies data analysis between data lakes and data warehouses, and reduces data development cycles and storage costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066111A_ABST
    Figure CN119066111A_ABST
Patent Text Reader

Abstract

The invention provides a data processing method and device, equipment, a medium and a program product. Relates to the technical field of computers. In some embodiments of the disclosure, an obtained configuration file is analyzed to obtain metadata for describing source data and an abstract syntax tree of data processing logic corresponding to the source data, and a rule for processing the source data is configured through the configuration file; determining a target deserialization mode corresponding to the configuration file; adopting a target deserialization mode to perform deserialization operation on the source data to obtain a target memory object; according to the abstract syntax tree, mapping the target memory object into a memory table capable of performing data query by adopting a standard query language; unstructured source data can be converted into a memory table directly queried by a standard query language, and the data utilization rate and the data use efficiency are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, device, equipment, medium and program product. Background Art

[0002] Data plays an increasingly important role in all types of enterprises, and has become a key factor in the development of enterprises. The utilization rate and efficiency of data use determine the height of data value mining.

[0003] Currently, for unstructured data, data utilization and data usage efficiency are low. Summary of the invention

[0004] The present disclosure provides a data processing method, apparatus, device, medium and program product to at least solve the existing technical problems of low data utilization and data use efficiency for unstructured data.

[0005] The technical solution of the present disclosure is as follows:

[0006] The present disclosure provides a data processing method, including:

[0007] Parsing the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data;

[0008] Determine a target deserialization mode corresponding to the configuration file;

[0009] Using the target deserialization method, deserialize the source data to obtain a target memory object;

[0010] According to the abstract syntax tree, the target memory object is mapped into a memory table capable of performing data query using a standard query language.

[0011] Optionally, the obtained configuration file is parsed to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, including:

[0012] Performing basic configuration analysis on the configuration file to obtain the metadata; and

[0013] The configuration file is parsed using data processing logic to obtain the abstract syntax tree.

[0014] Optionally, determining a target deserialization mode corresponding to the configuration file includes:

[0015] According to the type of the configuration file, a mapping relationship table between the file type and the deserialization method is queried to obtain the target deserialization method.

[0016] Optionally, the adopting the target deserialization mode to perform a deserialization operation on the source data to obtain a target memory object includes:

[0017] Using the target deserialization method, deserialize the source data to obtain a candidate memory object;

[0018] A filtering operation is performed on the candidate memory objects to obtain the target memory object.

[0019] Optionally, the filtering operation on the candidate memory objects to obtain the target memory object includes:

[0020] Performing a predicate push-down operation on the candidate memory object to obtain a candidate memory object after the predicate is pushed down;

[0021] A pruning operation is performed on the candidate memory objects after the predicate is pushed down to obtain the target memory object.

[0022] Optionally, mapping the target memory object to a memory table capable of performing data query using a standard query language according to the abstract syntax tree includes:

[0023] Parsing mapping rules from the abstract syntax tree;

[0024] According to the mapping rule, the target memory object is mapped to the memory table.

[0025] The present disclosure also provides a data processing device, including:

[0026] A parsing module, used to parse the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data;

[0027] A determination module, used to determine a target deserialization mode corresponding to the configuration file;

[0028] A deserialization module, used to perform a deserialization operation on the source data using the target deserialization method to obtain a target memory object;

[0029] A mapping module is used to map the target memory object into a memory table capable of performing data query using a standard query language according to the abstract syntax tree.

[0030] Optionally, when the parsing module parses the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, it is used to:

[0031] Performing basic configuration analysis on the configuration file to obtain the metadata; and

[0032] The configuration file is parsed using data processing logic to obtain the abstract syntax tree.

[0033] Optionally, when determining the target deserialization mode corresponding to the configuration file, the determination module is used to:

[0034] According to the type of the configuration file, a mapping relationship table between the file type and the deserialization method is queried to obtain the target deserialization method.

[0035] Optionally, when the deserialization module adopts the target deserialization mode to perform a deserialization operation on the source data to obtain a target memory object, it is used to:

[0036] Using the target deserialization method, deserialize the source data to obtain a candidate memory object;

[0037] A filtering operation is performed on the candidate memory objects to obtain the target memory object.

[0038] Optionally, when the deserialization module performs a filtering operation on the candidate memory object to obtain the target memory object, it is used to:

[0039] Performing a predicate push-down operation on the candidate memory object to obtain a candidate memory object after the predicate is pushed down;

[0040] A pruning operation is performed on the candidate memory objects after the predicate is pushed down to obtain the target memory object.

[0041] Optionally, when mapping the target memory object to a memory table capable of performing data query using a standard query language according to the abstract syntax tree, the mapping module is configured to:

[0042] Parsing mapping rules from the abstract syntax tree;

[0043] According to the mapping rule, the target memory object is mapped to the memory table.

[0044] The present disclosure also provides an electronic device, including:

[0045] processor;

[0046] a memory for storing instructions executable by the processor;

[0047] The processor is configured to execute the instructions to implement each step in the above method.

[0048] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer instructions are used to enable the computer to execute each step in the above method.

[0049] The embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, characterized in that the computer program / instruction implements each step of the above method when executed by a processor.

[0050] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0051] In some embodiments of the present disclosure, an acquired configuration file is parsed to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, and rules for processing the source data are configured through the configuration file; a target deserialization method corresponding to the configuration file is determined; the source data is deserialized using the target deserialization method to obtain a target memory object; based on the abstract syntax tree, the target memory object is mapped to a memory table that can perform data query using a standard query language; unstructured source data can be converted into a memory table that can be directly queried using a standard query language, thereby improving data utilization and data usage efficiency.

[0052] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0054] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0055] Figure 2 A schematic diagram of a parsing module provided in an embodiment of the present disclosure;

[0056] Figure 3 A schematic diagram of predicate pushdown provided in an embodiment of the present disclosure;

[0057] Figure 4 A schematic diagram of a mapping module provided in an embodiment of the present disclosure;

[0058] Figure 5 A schematic diagram of the structure of a data processing device provided for an exemplary embodiment of the present application;

[0059] Figure 6 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0062] It should be noted that the user information involved in this disclosure includes but is not limited to: user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0063] There are two main coexisting data architecture models: data warehouse and data lake.

[0064] As a typical data storage and analysis system, data warehouse is widely used in enterprises. It takes data analysis as the starting point, standardizes data in a certain mode, such as star mode, and requires the format object of the table to be determined before data is written. Data warehouse supports data query and analysis based on standard query language. Data lake provides more flexibility, directly operates on raw data, and better copes with requirements in data scale and data format diversity. Analyzing data in data lake requires more data processing steps.

[0065] Data warehouses store structured data, while data lakes generally store semi-structured data, such as JSON, and unstructured data, such as documents, binary data, etc. In most cases, data analysis and mining in enterprises require the combination of data in both the data warehouse and the data lake. Currently, standardized query languages ​​are used to analyze data in data warehouses, but it is impossible to analyze and process raw data in data lakes. In this case, the current solution is to entrust data engineers to develop data in the data lake, and load the data in the data lake into the data warehouse table through multiple steps such as "business analysis-data quality analysis-data modeling-extraction-conversion-loading". This solution not only requires a long development cycle and increases communication costs, but also increases data storage costs, and has low data utilization and data usage efficiency.

[0066] In response to the above-mentioned technical problems, in some embodiments of the present disclosure, the acquired configuration file is parsed to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, and the rules for processing the source data are configured through the configuration file; the target deserialization method corresponding to the configuration file is determined; the source data is deserialized using the target deserialization method to obtain a target memory object; according to the abstract syntax tree, the target memory object is mapped to a memory table that can be used for data query using a standard query language; the unstructured source data can be converted into a memory table that can be directly queried using a standard query language, thereby improving data utilization and data usage efficiency.

[0067] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0068] Figure 1 A flow chart of a data processing method provided by an embodiment of the present disclosure. Figure 1 As shown, the data processing method comprises:

[0069] S101: Parse the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data;

[0070] S102: Determine a target deserialization method corresponding to the configuration file;

[0071] S103: Deserialize the source data using a target deserialization method to obtain a target memory object;

[0072] S104: According to the abstract syntax tree, the target memory object is mapped into a memory table capable of performing data query using a standard query language.

[0073] In this embodiment, the execution subject of the above method is a server or other types of computer devices.

[0074] In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server devices. The server mainly includes a processor, a hard disk, a memory, a system bus, etc., and a general computer architecture type.

[0075] In this embodiment, the acquired configuration file is parsed to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, and the rules for processing the source data are configured through the configuration file; the target deserialization method corresponding to the configuration file is determined; the source data is deserialized using the target deserialization method to obtain a target memory object; according to the abstract syntax tree, the target memory object is mapped to a memory table that can be used for data query using a standard query language; the unstructured source data can be converted into a memory table that can be directly queried using a standard query language, thereby improving data utilization and data usage efficiency.

[0076] In some embodiments of the present disclosure, the acquired configuration file is parsed to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data. One achievable method is to perform basic configuration parsing on the configuration file to obtain metadata; and perform data processing logic parsing on the configuration file to obtain an abstract syntax tree. For the input configuration file, the configuration file is parsed into two parts. One part is a description of the accessed source data, including data path, data format, etc. This part constitutes metadata after parsing; the other part is to parse the data processing logic into an abstract syntax tree.

[0077] Figure 2 A schematic diagram of a parsing module provided in an embodiment of the present disclosure. Figure 2 As shown in the figure, the parsing module performs basic configuration parsing and data processing logic parsing on the configuration file. The basic configuration information is parsed to obtain basic information including input and output paths, file formats, serialization types, etc., namely metadata. The parsing of data processing logic is based on parsing rules. After syntax verification and analysis, an abstract syntax tree is generated. The syntax tree covers the fields, filtering conditions, aggregated fields, etc. required for data analysis.

[0078] In some embodiments of the present disclosure, a target deserialization method corresponding to the configuration file is determined. One achievable method is to query a mapping table between file type and deserialization method according to the type of configuration file to obtain the target deserialization method. The formats of the accessed configuration files are various, such as Sequence File, Text, etc., and the corresponding deserialization methods should also be set to multiple methods, such as Protobuf, JSON, Avro, Parquet, etc. By pre-establishing a mapping table between file type and deserialization method, the target deserialization method corresponding to the configuration file can be quickly queried.

[0079] In some embodiments of the present disclosure, a target deserialization method is used to perform a deserialization operation on the source data to obtain a target memory object. One achievable method is to use a target deserialization method to perform a deserialization operation on the source data to obtain a candidate memory object; and filter the candidate memory object to obtain the target memory object. Among them, filtering the candidate memory object reduces resource usage and improves data processing efficiency. Among them, filtering methods include but are not limited to: predicate pushdown and pruning.

[0080] Figure 3 A schematic diagram of predicate pushdown provided for an embodiment of the present disclosure. In an optional embodiment, a filtering operation is performed on the candidate memory objects to obtain a target memory object. One achievable method is to perform a predicate pushdown operation on the candidate memory objects to obtain the candidate memory objects after the predicate pushdown; and perform a pruning operation on the candidate memory objects after the predicate pushdown to obtain the target memory object. The abstract syntax tree is analyzed, and for the filtering conditions that can be executed in advance, they are executed as close to the data source as possible, thereby reducing the amount of data entering the subsequent process. This is predicate pushdown; in addition, for some specific statements, such as a simple COUNT, unnecessary fields will be removed to reduce resource usage. This is pruning.

[0081] In some embodiments of the present disclosure, according to the abstract syntax tree, the target memory object is mapped to a memory table that can perform data query using a standard query language. One achievable method is to parse a mapping rule from the abstract syntax tree; according to the mapping rule, the target memory object is mapped to a memory table.

[0082] Figure 4 A schematic diagram of a mapping module provided in an embodiment of the present disclosure. Figure 4As shown in the figure, this module is responsible for normalizing the data and completing the mapping from memory objects to memory tables to facilitate subsequent data operations. For complex and diverse data structures, this module uses the reflection mechanism to achieve universality and can accurately draw the data structure of the object. Among them, the processing of generic data is strengthened. For generic data with complex inheritance links, the real type parameters are obtained through the backtracking mechanism. After accurately drawing the data structure of the object, combined with the rules specified in the metadata in the configuration file, the memory table module maps the object into a standardized memory table.

[0083] Currently, mainstream computing engines such as Spark and Flink can support the ANSI SQL standard, so in most cases, user SQL is universal on these engines. However, outside the ANSI SQL standard, different engines have their own function implementations, some of which may be slightly different (such as function names). The engine adaptation module adapts common functions outside the ANSI standard so that they can still run normally when switching to another computing engine.

[0084] Combined with the description of the above embodiments, the data in the data lake is mapped into a memory table through the cooperation of the above modules. In this process, optimization operations such as predicate pushdown and pruning are realized. Thus, the data analysis of the data lake and the data warehouse is uniformly converted into table operations, shielding the underlying complex big data processing implementation, and using standard query language, complex data analysis can be performed.

[0085] In addition, since each module is not coupled with any computing engine, users can freely choose computing engines such as Spark and FlinkBatch to perform data analysis tasks, avoiding dependence on a specific computing platform and increasing deployment flexibility.

[0086] The disclosed embodiments shield the complexity of big data processing, support user profiles to intuitively and conveniently describe and execute unified analysis and processing of data lakes and data warehouses, realize the connection of data between data lakes and data warehouses, fully tap the value of data, avoid the long cycle of data development and waste of storage resources in traditional solutions, and improve data utilization and low data use efficiency.

[0087] In the above method embodiment of the present disclosure, the acquired configuration file is parsed to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, and the rules for processing the source data are configured through the configuration file; the target deserialization method corresponding to the configuration file is determined; the source data is deserialized using the target deserialization method to obtain a target memory object; according to the abstract syntax tree, the target memory object is mapped to a memory table that can perform data query using a standard query language; the unstructured source data can be converted into a memory table that can be directly queried using a standard query language, thereby improving data utilization and data usage efficiency.

[0088] Figure 5 FIG. 5 is a schematic diagram of a data processing device 50 provided in an exemplary embodiment of the present application. Figure 5 As shown, the data processing device 50 includes: a parsing module 51 , a determining module 52 , a deserialization module 53 and a mapping module 54 .

[0089] The parsing module 51 is used to parse the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data;

[0090] A determination module 52, used to determine a target deserialization method corresponding to the configuration file;

[0091] The deserialization module 53 is used to perform a deserialization operation on the source data in a target deserialization manner to obtain a target memory object;

[0092] The mapping module 54 is used to map the target memory object into a memory table capable of performing data query using a standard query language according to the abstract syntax tree.

[0093] Optionally, when parsing the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, the parsing module 51 is used to:

[0094] Perform basic configuration parsing on the configuration file to obtain metadata; and

[0095] Perform data processing logic analysis on the configuration file to obtain an abstract syntax tree.

[0096] Optionally, when determining the target deserialization mode corresponding to the configuration file, the determination module 52 is used to:

[0097] According to the type of configuration file, query the mapping relationship table between file type and deserialization method to obtain the target deserialization method.

[0098] Optionally, when the deserialization module 53 adopts the target deserialization mode to perform a deserialization operation on the source data to obtain the target memory object, it is used to:

[0099] Use the target deserialization method to deserialize the source data and obtain the candidate memory object;

[0100] Perform a filtering operation on the candidate memory objects to obtain the target memory object.

[0101] Optionally, when the deserialization module 53 performs a filtering operation on the candidate memory objects to obtain the target memory object, it is used to:

[0102] Perform predicate push-down operation on the candidate memory object to obtain the candidate memory object after predicate push-down;

[0103] Perform pruning operations on the candidate memory objects after the predicate is pushed down to obtain the target memory object.

[0104] Optionally, when mapping the target memory object to a memory table capable of performing data query using a standard query language according to the abstract syntax tree, the mapping module 54 is configured to:

[0105] Parse mapping rules from the abstract syntax tree;

[0106] According to the mapping rules, the target memory object is mapped to a memory table.

[0107] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0108] Figure 6 The following is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Figure 6 As shown, the electronic device includes: a memory 61 and a processor 62. In addition, the electronic device also includes a power supply component 63 and a communication component 64.

[0109] The memory 61 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.

[0110] Memory 61 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0111] The communication component 64 is used for data transmission with other devices.

[0112] The processor 62 can execute computer instructions stored in the memory 61 to: parse the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data; determine the target deserialization method corresponding to the configuration file; use the target deserialization method to deserialize the source data to obtain a target memory object; and map the target memory object to a memory table capable of performing data query using a standard query language according to the abstract syntax tree.

[0113] Optionally, when the processor 62 parses the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, it is configured to:

[0114] Perform basic configuration parsing on the configuration file to obtain metadata; and

[0115] Perform data processing logic analysis on the configuration file to obtain an abstract syntax tree.

[0116] Optionally, when determining the target deserialization mode corresponding to the configuration file, the processor 62 is configured to:

[0117] According to the type of configuration file, query the mapping relationship table between file type and deserialization method to obtain the target deserialization method.

[0118] Optionally, when the processor 62 adopts the target deserialization mode to perform a deserialization operation on the source data to obtain the target memory object, it is configured to:

[0119] Use the target deserialization method to deserialize the source data and obtain the candidate memory object;

[0120] Perform a filtering operation on the candidate memory objects to obtain the target memory object.

[0121] Optionally, when the processor 62 performs a filtering operation on the candidate memory objects to obtain the target memory object, it is configured to:

[0122] Perform predicate push-down operation on the candidate memory object to obtain the candidate memory object after predicate push-down;

[0123] Perform pruning operations on the candidate memory objects after the predicate is pushed down to obtain the target memory object.

[0124] Optionally, when mapping the target memory object to a memory table capable of performing data query using a standard query language according to the abstract syntax tree, the processor 62 is configured to:

[0125] Parse mapping rules from the abstract syntax tree;

[0126] According to the mapping rules, the target memory object is mapped to a memory table.

[0127] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 1 Each step in the method embodiment.

[0128] Accordingly, the embodiment of the present application also provides a computer program product, the computer program product includes a computer program / instruction, and the computer program / instruction is executed by a processor. Figure 1 Each step in the method embodiment.

[0129] Above Figure 6 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 6G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0130] Above Figure 6 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0131] Above Figure 6 The display screen in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0132] The electronic device mentioned above also includes an audio component.

[0133] In the above-mentioned apparatus, device, storage medium and computer program product embodiments of the present disclosure, the acquired configuration file is parsed to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, and the rules for processing the source data are configured through the configuration file; the target deserialization method corresponding to the configuration file is determined; the source data is deserialized using the target deserialization method to obtain a target memory object; according to the abstract syntax tree, the target memory object is mapped to a memory table that can perform data query using a standard query language; the unstructured source data can be converted into a memory table that can be directly queried using a standard query language, thereby improving data utilization and data usage efficiency.

[0134] The audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0135] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0140] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0141] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0142] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0143] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that: include: Parsing the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data; Determine a target deserialization mode corresponding to the configuration file; Using the target deserialization method, deserialize the source data to obtain a target memory object; According to the abstract syntax tree, the target memory object is mapped into a memory table capable of performing data query using a standard query language.

2. The method according to claim 1, characterized in that The obtained configuration file is parsed to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, including: Performing basic configuration analysis on the configuration file to obtain the metadata; and The configuration file is parsed using data processing logic to obtain the abstract syntax tree.

3. The method according to claim 1, characterized in that The determining of the target deserialization mode corresponding to the configuration file includes: According to the type of the configuration file, a mapping relationship table between the file type and the deserialization method is queried to obtain the target deserialization method.

4. The method according to claim 1, characterized in that: The adopting the target deserialization mode to perform a deserialization operation on the source data to obtain a target memory object includes: Using the target deserialization method, deserialize the source data to obtain a candidate memory object; A filtering operation is performed on the candidate memory objects to obtain the target memory object.

5. The method according to claim 4, characterized in that The filtering operation on the candidate memory objects to obtain the target memory object includes: Performing a predicate push-down operation on the candidate memory object to obtain a candidate memory object after the predicate is pushed down; A pruning operation is performed on the candidate memory objects after the predicate is pushed down to obtain the target memory object.

6. The method according to claim 1, characterized in that Mapping the target memory object to a memory table capable of performing data query using a standard query language according to the abstract syntax tree includes: Parsing mapping rules from the abstract syntax tree; According to the mapping rule, the target memory object is mapped to the memory table.

7. A data processing device, characterized in that: include: A parsing module, used to parse the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data; A determination module, used to determine a target deserialization mode corresponding to the configuration file; A deserialization module, used to perform a deserialization operation on the source data using the target deserialization method to obtain a target memory object; A mapping module is used to map the target memory object into a memory table capable of performing data query using a standard query language according to the abstract syntax tree.

8. The device according to claim 7, characterized in that When the parsing module parses the acquired configuration file to obtain metadata describing the source data and an abstract syntax tree of the data processing logic corresponding to the source data, it is used to: Performing basic configuration analysis on the configuration file to obtain the metadata; as well as The configuration file is parsed using data processing logic to obtain the abstract syntax tree.

9. The device according to claim 7, characterized in that When determining the target deserialization mode corresponding to the configuration file, the determination module is used to: According to the type of the configuration file, a mapping relationship table between the file type and the deserialization method is queried to obtain the target deserialization method.

10. The device according to claim 7, characterized in that When the deserialization module adopts the target deserialization mode to perform a deserialization operation on the source data to obtain a target memory object, it is used to: Using the target deserialization method, deserialize the source data to obtain a candidate memory object; A filtering operation is performed on the candidate memory objects to obtain the target memory object.

11. The device according to claim 10, characterized in that When the deserialization module performs a filtering operation on the candidate memory objects to obtain the target memory object, it is used to: Performing a predicate push-down operation on the candidate memory object to obtain a candidate memory object after the predicate is pushed down; A pruning operation is performed on the candidate memory objects after the predicate is pushed down to obtain the target memory object.

12. The device according to claim 7, characterized in that When the mapping module maps the target memory object to a memory table capable of performing data query using a standard query language according to the abstract syntax tree, the mapping module is used to: Parsing mapping rules from the abstract syntax tree; According to the mapping rule, the target memory object is mapped to the memory table.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement each step in the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The computer instructions are used to enable the computer to execute each step in the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Method and device for processing input data and output data

    CN108153896A

  • Method and device for generating real-time calculation logic data

    CN113515285A

  • Data query method and device

    CN114860997A

  • Event processing method, device and equipment and computer readable storage medium

    CN115168070A

  • Target data obtaining method and apparatus

    US20210191934A1