Data processing method and device, electronic equipment, storage medium and program

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment, a storage medium and a program, and the method comprises the steps: obtaining a structured data file stored by an upstream data storage engine, and building a structured data table by taking a data directory of the structured data file as table building information; class information is obtained from the table building information, and a code package containing the class information is generated; loading the code package, and reading table metadata of the target storage data from the structured data table; and obtaining a file data source of the target storage data according to the table metadata of the target storage data. Storage data in a specified structured data format is used as a readable file source of a downstream data use engine, and a file data source of the storage data is read by loading a code package, so that troubleshooting of original data is facilitated, table metadata are automatically updated on the basis of recording of the code package, reusability of the code package is realized, and the efficiency is improved. And the development and use difficulty of a downstream engine on specified format data is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, a data processing device, an electronic device, a computer-readable storage medium and a computer program product. Background Art

[0002] Advertising has shifted from a content-intensive industry to a data-intensive industry. The business process of digital advertising mainly revolves around data, that is, data affects the transformation of advertising operation processes at the meso-level. In the advertising industry ecosystem dominated by digital advertising, in order to deeply understand the relationship between data and advertising, there is a need to effectively and quickly analyze useful information from a relatively large amount of data.

[0003] However, as the amount of data increases, the resource consumption of computers also increases linearly. In order to optimize the resource loss during data transmission, the upstream advertising engine system usually uses the Protocol Buffer format (protobuf for short, which is a data storage format for structured data and an extended serialized structured data format) for data storage, so as to reduce the hard disk space based on the aforementioned file format. However, in terms of downstream data usage, the reading of the aforementioned protobuf data needs to be implemented by developing Java code. Every time the protobuf data to be read is modified or updated, the corresponding Java code needs to be redeveloped for reading, which increases the difficulty of downstream engines in developing the use of protobuf data. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data processing method, a data processing device, an electronic device, a computer-readable storage medium and a computer program to achieve the reusability of code packages and reduce the difficulty of downstream engines in developing and using data in a specified format. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a data processing method is provided, which is applied to a downstream data usage engine. The method comprises:

[0006] Acquire a structured data file stored in an upstream data storage engine, and use the data directory of the structured data file as table building information to construct a structured data table;

[0007] Acquire class information from the table creation information, and generate a code package containing the class information;

[0008] Load the code package and read table metadata of target storage data from the structured data table;

[0009] According to the table metadata of the target storage data, the file data source of the target storage data is obtained.

[0010] In a second aspect of the present invention, there is also provided a data processing device, which is applied to a downstream data usage engine, and the device comprises:

[0011] A data table construction module, used to obtain the structured data file stored in the upstream data storage engine, and use the data directory of the structured data file as table construction information to construct a structured data table;

[0012] A code package generation module, used to obtain class information from the table creation information and generate a code package containing the class information;

[0013] A code package loading module, used to load the code package and read table metadata of target storage data from the structured data table;

[0014] The data reading module is used to obtain the file data source of the target storage data according to the table metadata of the target storage data.

[0015] In another aspect of the implementation of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the data processing methods when executing the programs stored in the memory.

[0016] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned data processing methods.

[0017] In yet another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned data processing methods.

[0018] The data processing method, device, electronic device, storage medium and program provided by the embodiments of the present invention construct a structured data table by taking the data directory of the structured data file stored by the upstream data storage engine as the table building information, and generate a code package containing class information based on the table building information, so that the storage data of the specified structured data format is used as the readable file source of the downstream data usage engine. On this basis, by loading the code package, the table metadata based on the target storage data to be read is realized, so as to obtain the file data source of the storage data. Since the code package is generated by the class information of the structured data file, the table metadata in the data file and the structured data table can be independently updated in a decoupled manner based on the consistency of the serialization rules. The table metadata of the updated content can be automatically updated directly by loading the generated code package, so as to realize the reusability of the code package, reduce the difficulty of the downstream data usage engine to develop and use the data in the specified structured data format, and can also facilitate the troubleshooting of the original data through the directly read file data source. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0020] Figure 1 A flowchart of a data processing method according to an embodiment of the present invention;

[0021] Figure 2 is a flowchart of another data processing method embodiment in an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a data reading process provided by an embodiment of the present invention;

[0023] Figure 4 is a structural block diagram of an embodiment of a data processing device in an embodiment of the present invention;

[0024] Figure 5 The figure is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0026] To facilitate those skilled in the art to understand the present invention, the following explains the terms or nouns involved in the following embodiments of the present invention:

[0027] Protocol Buffer: Protobuf for short, is a data storage format for structured data, similar to XML (Extensible Markup Language), Json (JavaScript Object Notation, a text-based data exchange format), etc. It is mainly a language-neutral, platform-neutral, extensible serialized structured data format that can be used in communication protocols, data storage and other fields.

[0028] Hive: is a data warehouse that provides a mechanism for processing structured data and allows users to use the SQL-like query language HiveQL to operate data.

[0029] SQL: Structured Query Language, is a standard programming language for managing relational databases, allowing users to perform various tasks such as querying, inserting, updating, and deleting data in a database, as well as creating, modifying, and deleting database structures.

[0030] In the related art, when the downstream data usage engine reads the proto data stored in the upstream advertising engine system, each time a new field is added or modified in the proto data, it is necessary to redevelop the corresponding Java code to implement the reading and remodify the table structure to go online. In the case of new data sources in the proto data, it is even more necessary to redevelop a set of reading codes, which causes the downstream data usage engine to have a high code development time cost but low ease of use when reading data, increasing the difficulty of the downstream engine in using and developing protobuf data.

[0031] In an embodiment of the present invention, in order to reduce the difficulty of using and developing protobuf data in the big data field, the downstream data usage engine can be developed in the form of SQL to analyze the protobuf data, thereby reducing data analysis costs, development difficulty and ensuring the reusability of the code language.

[0032] Reference Figure 1 , shows a flow chart of steps of a data processing method embodiment of the present invention, which may specifically include the following steps:

[0033] Step 101, obtaining a structured data file stored in an upstream data storage engine, and using the data directory of the structured data file as table building information to construct a structured data table;

[0034] In an embodiment of the present invention, for data reading and data analysis of a relatively large amount of data (for example, TB level), an upstream data storage engine and a downstream data usage engine are involved. The data storage engine is the upstream of the downstream data usage engine, and the data usage engine is the downstream of the upstream data storage engine. The upstream data storage engine is responsible for data collection and storage, and the downstream data usage engine is responsible for batch processing and analysis of data. The downstream data usage engine can perform batch processing and analysis on the data stored in the upstream data storage engine.

[0035] Among them, the upstream data storage engine can store data in a specified structured data format, such as protobuf file format, similar to Excel file storage by row, and divide information by head+body, so as to reduce the hard disk space based on the aforementioned file format; and the downstream data usage engine can be an engine with a mechanism for processing structured data, that is, the downstream data engine can be an engine that can query proto data files in SQL form.

[0036] For example, in order to meet the requirements of querying proto data files in SQL form, when selecting the execution engine, it can meet the requirements of supporting big data processing capabilities, supporting secondary development, and being able to read data files. In the embodiment of the present invention, the Hive engine can be selected as the final execution engine. Optionally, in the application scenario of data advertising, the upstream data storage engine can be the upstream advertising engine, and the downstream data usage engine can be the Hive engine, which can realize the reading of data stored by the upstream engine on the corresponding client.

[0037] Specifically, Hive can be used as the SQL execution engine, that is, Hive can be integrated into the downstream data usage engine. When reading data, Hive can be used as the SQL execution engine to read the protobuf data stored in the upstream advertising engine. That is, Hive's customized reading method can be used, such as the SQL form to query the protobuf data file. By using the protobuf data file type as a readable file source of Hive, the downstream data usage engine integrated with Hive can realize the reading of protobuf data.

[0038] Among them, in the field of big data, the Hive engine itself cannot read the Protocol Buffer file normally. The embodiment of the present invention can use the InputFormat custom reading method of the Hive engine for secondary development to achieve normal reading of the Protocol Buffer file. InputFormat is a mechanism used by the Hive engine to read input data. The embodiment of the present invention can enable the Hive engine to read the Protocol Buffer file by customizing the InputFormat.

[0039] In one embodiment of the present invention, the customization of the reading method of the Hive engine can be manifested as using the data directory of the structured data files stored by the upstream data storage engine as table building information to construct a structured data table, thereby realizing subsequent SQL table lookup operations when reading data.

[0040] In practical applications, the data source may be determined first, such as the type and location of the upstream data storage engine. In an embodiment of the present invention, the data source may be obtained via a data directory. At this time, appropriate tools or APIs may be used to extract structured data files from the upstream data storage engine. For the obtained structured data files, the file contents may be parsed to understand their structure and meaning, thereby obtaining table creation information based on the pattern of the parsed structured data files, so as to use the table creation information to create structured data tables in the target database or data warehouse.

[0041] Step 102, obtaining class information from the table building information, and generating a code package containing the class information;

[0042] In order to use the stored data in a specified structured data format as a readable file source, the class information used by the structured data files stored in the upstream data storage engine can be obtained from the table creation information. The class information can be used to indicate the class information defined in Protocol Buffers. At this time, a code package containing the aforementioned class information can be generated, thereby packaging the secondary development of the data reading method, that is, the data processing process, into a code package.

[0043] Specifically, class information can be obtained from the table creation information, and the data content of the structured data file can be read based on the class information, and then the data content can be encapsulated to obtain a code package, wherein the class information can be used to define an interface for reading input data from the structured data file. In a specific implementation, the data content can be read through the indicated interface for reading input data.

[0044] Optionally, the embodiment of the present invention uses the storage data in the specified structured data format as a readable file source, and in the process of encapsulating to obtain the code package, the data format of the read data content can be converted into a data object that can be processed by the downstream data usage engine. Among them, since the downstream data usage engine can use SQL form to query the proto data file, at this time, the data format of the read data content can be converted into a relational data form, that is, SQL form.

[0045] It should be noted that since the code package is generated through the class information of the structured data file, it can be decoupled and independently updated based on the consistency of serialization rules. The table metadata of the updated content can be automatically updated directly by loading the generated code package, thereby achieving the reusability of the code package and reducing the difficulty of downstream data usage engines in developing and using data in a specified structured data format.

[0046] Exemplary, decoupled independent updates, as an example, when the proto metadata file first adds a field, it will not affect the table metadata in the structured data table; as another example, when the data file first adds a field and the proto metadata file does not change, it will not affect the table metadata in the structured data table; as another example, when the field name is modified but the field position remains unchanged and the field type remains unchanged, it will not affect the table metadata in the structured data table; as another example, when the proto metadata file deletes the middle field but the remaining field sequence number remains unchanged, it will not affect the table metadata in the structured data table. This embodiment of the present invention is not limited to this.

[0047] Step 103, loading the code package, and reading the table metadata of the target storage data from the structured data table;

[0048] In actual applications, the packaged code package can be put online as a type similar to an installation package, that is, promoted to the market, so that it can be provided to the downstream data usage engine for application and data reading.

[0049] When the downstream data engine uses it, it can respond to data query instructions through the downstream usage engine. The data query instructions can be used to indicate the target storage data to be read currently. At this time, the code package can be loaded, and the downstream data usage engine can use SQL to query the proto data file. In the specific implementation, the reading of the target storage data can be specifically expressed as a query operation on the structured data table. When the code package is loaded and executed, the target storage data is mainly read from the structured data table.

[0050] Step 104: Obtain the file data source of the target storage data according to the table metadata of the target storage data.

[0051] In an embodiment of the present invention, when reading data, the table metadata of the target storage data may be read from the structured data table first, and then the target storage data may be successfully read based on the table metadata.

[0052] Among them, table metadata mainly refers to Proto metadata, which is mainly class information in the generated code package. The class information can be a proto class file, and the code package to be generated can be a jar package (belonging to a special compressed folder). The jar package usually contains the class file compiled from the proto metadata. The table metadata can be loaded through the add jar command.

[0053] Optionally, the data file storing the data may be proto data, specifically a structured data table, that is, a directory corresponding to a Hive table partition. For example, the metadata may be a student's name, academic performance, personality, etc., and the data may be the actual content corresponding to each field of the metadata, such as Zhang San, Chinese 96, Mathematics 99, outgoing and cheerful, etc.

[0054] When the target storage data is read based on the table metadata, since the table metadata can be used to indicate the data directory location of the target storage data, the file data source of the target storage data can be obtained at this time, thereby facilitating troubleshooting of the original data.

[0055] In an embodiment of the present invention, a structured data table is constructed by taking the data directory of the structured data file stored by the upstream data storage engine as the table construction information, and generating a code package containing class information based on the table construction information, so that the storage data in the specified structured data format can be used as a readable file source for the downstream data usage engine. On this basis, by loading the code package, the table metadata based on the target storage data to be read is realized, thereby obtaining the file data source of the storage data. Since the code package is generated by the class information of the structured data file, the table metadata in the data file and the structured data table can be independently updated in a decoupled manner based on the consistency of the serialization rules. By directly loading the generated code package, the table metadata of the updated content can be automatically updated, thereby realizing the reusability of the code package, reducing the difficulty of the downstream data usage engine in developing and using the data in the specified structured data format, and also being able to directly read the file data source to facilitate the troubleshooting of the original data.

[0056] Reference Figure 2 , shows a step flow chart of another data processing method embodiment in an embodiment of the present invention, focusing on the development process of the code package, which may specifically include the following steps:

[0057] Step 201, obtaining class information from table creation information of a structured data table, and reading data content of a structured data file based on the class information;

[0058] In an embodiment of the present invention, the downstream data engine may select the Hive engine and use SQL to query the proto data file.

[0059] In order to meet the requirement of querying proto data files in SQL form, the Hive engine itself cannot read the Protocol Buffer file normally. The embodiment of the present invention can perform secondary development with the help of the InputFormat custom reading method of the Hive engine to achieve normal reading of the Protocol Buffer file.

[0060] The customization of the reading method can be manifested as constructing a structured data table using the data directory of the structured data files stored in the upstream data storage engine as table creation information, thereby realizing the subsequent SQL table lookup operation when reading the data; and at this time, the class information used by the structured data files stored in the upstream data storage engine can be obtained from the table creation information, and the class information can be used to indicate the class information defined in Protocol Buffers. The embodiment of the present invention can generate a code package containing the aforementioned class information, thereby packaging the secondary development of the data reading method, that is, the data processing process into a code package.

[0061] Specifically, the data content of the structured data file may be read based on the class information first, so that the data content may be subsequently encapsulated to obtain a code package.

[0062] The data content reading process can be mainly implemented by determining the calling entity of the data content based on the class information, and then reading the data content of the structured data file according to the calling entity.

[0063] Among them, the class information can be used to define an interface for reading input data from a structured data file. At this time, the calling entity of the data content can be determined based on the interface for reading input data from the structured data file. The class information can also be used to define a reading method for a structured data file. At this time, the calling entity can be decompressed according to the reading method to obtain the data content of the structured data file.

[0064] For example, some interfaces provided by the open source project Hive can be used to secondary develop a custom reading method to read the proto file.

[0065] As an example, there is a class FileInputFormat, which can be used to define the type of data file, compression method, whether to slice and file filtering, etc., where the data file type can refer to the Hive file format, for example, including but not limited to TEXTFILE, SEQUENCEFILE, RCFILE, ORCFILE, PARQUET, etc., and the embodiment of the present invention can be a custom project class stored as INPUTFORMAT'com.qiyi.serde.protobuf.ProtobufMessageInputFormat'; the compression method can be automatically selected by the Hive project according to the file format suffix, for example, the .gz ending is generally GZIP compression, which can be mainly identified and constructed via Compress ionCodecFactory.getCodec; whether to slice can be used to indicate whether to split a file into multiple files when Hive reads the data file, or whether multiple small files are merged into one. The slicing result can also be associated with the compression method, for example, gz compression does not support slicing; file filtering refers to using the setInputPathFilter method to define the filtering of files with a file name prefix of proto.

[0066] As another example, there is also a RecordReader class, which can be used to define the way to read the data file content, such as whether to split it by rows and columns or by head+body. The proto file in the embodiment of the present invention is mainly split by head+body.

[0067] In the specific implementation, at this time, the FileInputFormat abstract class can be inherited first, and the class information, that is, the proto class information, can be obtained from the Hive table creation information. At this time, the class file of the proto metadata can be verified. If the corresponding class file does not exist, the user can be reminded to correct it. After the verification, the calling entity of the data content can be determined based on the obtained class information. The interface for reading input data from the structured data file, such as the com.google.protobuf.Parser object, can be obtained mainly through class information reflection to pass it to the downstream data usage engine for use. For example, based on the interface for reading input data from the structured data file, the calling entity of the data content is determined; then the RecordReader interface can be inherited, and the calling entity is decompressed according to the reading method to read the data content of the structured data file, for example, the proto compressed file stream API com.google.protobuf.CodedInputStream.readMessage (a method for reading messages from an input stream) is used to decompress and read the data content.

[0068] Among them, the calling entity mainly refers to the object or method that actually calls and operates the data in the code. Optionally, the calling entity may at least include an input stream, which is a source for reading compressed data. The decompression performed on the calling entity may be specifically performed by parsing the compressed data in the input stream according to the reading method defined by the class information, and then parsing the read data to obtain the data content. Exemplarily, the reading method defines how to read data from the input stream, which usually involves using related classes to read data. The related class used in the embodiment of the present invention may be the above-mentioned custom class. The decompression performed may be performed by parsing the read data to obtain a message object, thereby realizing reading and decompressing a structured data file from the input stream, thereby realizing reading of the protobuf message in the structured data file, and realizing reading of the data content of the structured data file.

[0069] Step 202, encapsulating the data content to obtain a code package;

[0070] In order to use the stored data in the specified structured data format as a readable file source, a code package containing the aforementioned class information can be generated, thereby packaging the secondary development of the data reading method, that is, the data processing process, into a code package. In one embodiment of the present invention, the above-mentioned read data content can be encapsulated to obtain a code package.

[0071] Specifically, class information can also be used to define a data carrier. At this time, the data content can be deserialized to obtain deserialized data, and then the deserialized data can be placed in the data carrier. The deserialized data in the data carrier is converted to encapsulate data objects in relational data form, and then the class information and the data objects in relational data form are packaged to obtain a code package.

[0072] As another example, there is also a class Write table, which can be used to define the carrier of data output, such as basic type string, complex type map and composite type; as another example, there is also a class AbstractSerDe, which can be used to define data serialization and deserialization methods, for example, protobuf message data can be converted into hive struct type data.

[0073] In a specific implementation, after the above-mentioned inherited RecordReader interface reads the data content, the data content can be deserialized, for example, deserialized into a proto single message through a com.google.protobuf.Parser object, and then when the deserialized data is put into the data carrier, the aforementioned proto single message can be inserted into the data carrier that inherits the Write interface; then AbstractSerDe can be inherited to perform data conversion on the deserialized data in the data carrier, mainly to realize the conversion of the data in the Write carrier into Hive processable data (i.e., hive struct type data), and convert it into a data object in the form of relational data, that is, encapsulated into a data object that can be processed by the SQL language.

[0074] In a preferred embodiment of the present invention, when a field change occurs in a structured data file stored in an upstream data storage engine, the table structure of the structured data table can be changed based on the code package. The changed table structure can be used to point to the data directory location of the changed field, and the data directory location can be used to read the file data source of the changed field. When a field is added or modified in the stored data, for example, when there is a need to update the field of the structured data table, the table structure change and automatic completion of the online process can be achieved by simply updating the reading code. Specifically, the new data source only needs to provide the original metadata file, and the new table points to the data directory location.

[0075] Step 203, loading the code package;

[0076] In the embodiment of the present invention, the generated code package is mainly obtained by packaging the data processing process performed in the above steps. In actual application, the packaged code package can be put online as a type similar to an installation package, that is, promoted to the market, so that it can be provided to the downstream data usage engine for application and data reading.

[0077] When the downstream data engine uses it, the downstream use engine can respond to the data query instruction, and the data query instruction can be used to indicate the target storage data to be read currently, and the code package can be loaded at this time.

[0078] Specifically, when loading a code package, the structured data table may be loaded in a partitioned and timed manner.

[0079] Partitioned scheduled loading involves a partitioning process and a scheduled loading process. In one embodiment of the present invention, in the partitioning process, the structured data table can be partitioned to obtain at least one partition table. Specifically, the structured data table can be partitioned according to different directories, such as date, region, user ID, etc. For each partition, a script or program can be written to perform data loading operations at a specified time, that is, when loading a code package, each partition table can be loaded regularly so that the table metadata of the target storage data can be read from the target partition table later.

[0080] Partition is a term used by Hive to distinguish different directories. For example, assume that the directory used for partitioning is a data file directory uploaded at different times. For example, data uploaded at 11 o'clock on 2023-12-11 is stored in a directory, and its partition can be dt='2023-12-11'and hour='11'. This partition can also be used as a query filter condition for Hive SQL, indicating the behavior data at this point in time. At this time, the behavior trajectory of users of a certain software at various time points can be analyzed and partitioned by time to facilitate real-time upload of user behavior data and scheduled analysis.

[0081] An example of partition timing loading after the code package is launched is as follows:

[0082] SET PART_YEAR=2023; / / define variable PART_YEAR, its value is 2023;

[0083] SET PART_MONTH=05; / / define variable PART_MONTH, its value is 05, representing May;

[0084] SET PART_DAY=05; / / define variable PART_DAY, its value is 05, representing May 5th;

[0085] SET PART_HOUR=15; / / define variable PART_HOUR, whose value is 15 (representing 3:00 p.m.);

[0086] SET PART_MINUTE=00; / / define variable PART_MINUTE, its value is 00 (representing minutes);

[0087] set HOUR='15'; / / define variable HOUR, its value is 15;

[0088] set DATE = '2023-05-05'; / / define variable DATE, its value is 2023-05-05;

[0089] set LOCATION = "test / ${hiveconf:PART_YEAR} / ${hiveconf:PART_MONTH} / ${hivec onf:PART_DAY} / ${hiveconf:PART_HOUR}"; / / Define the variable LOCATION, which is a string containing a path template. Use the previously defined variables in the path (referenced by ${hiveconf:...}), so that the actual path can be generated according to the values ​​of these variables;

[0090] alter table cupid_test.ods_test add PARTITION(dt=${hiveconf:DATE},hour=${hiveconf:HOUR})

[0091] location${hiveconf:LOCATION};

[0092] / / Modify or add a Hive table named cupid_test.ods_test. Specifically, add a new partition to the table. This partition has two fields: dt and hour. Their values ​​are defined by the variables DATE and HOUR respectively. The partition location can be defined by the variable LOCATION.

[0093] It should be noted that the purpose of the above code is to add a new partition to a Hive table so that data can be organized according to date and time fields, so that when processing large-scale data, query performance and data management can be improved based on partitions.

[0094] Step 204: obtaining a file data source of target storage data based on the code package.

[0095] Since the downstream data usage engine can use SQL to query the proto data file, the reading of the target storage data can be specifically expressed as a query operation on the structured data table. When the code package is loaded and executed, the target storage data is mainly read from the structured data table.

[0096] When the target storage data is read based on the table metadata, in a specific performance, since the table metadata can be used to indicate the data directory location of the target storage data, the file data source of the target storage data can be obtained based on the data directory location of the target storage data, thereby facilitating troubleshooting of the original data.

[0097] For example, you can temporarily add a jar file in the Hive client, create a table with protoclass information, and then query the table with SQL. The example of creating a table can be as follows:

[0098]

[0099]

[0100] The above example mainly creates an external table that uses a specific serializer and input / output format to store and read data in protobuf format. This configuration allows Hive to interact with non-standard data formats (in this case, protobuf), enabling querying and manipulation of raw data stored on Hadoop.

[0101] It should be noted that the user usage process is mainly manifested in that after the user defines the proto metadata according to the proto official guide, the user can compile and obtain the jar package file containing the proto class file. If there is a need to change the proto.class information, all fields can be found by executing sql show create table. Subsequent updates to the fields only require updating the proto jar package, which enables the reusability of the code package.

[0102] After taking the structured data file, that is, the Protocol Buffer file, as the data source and obtaining the Hive table creation information and constructing the structured data table, Figure 3As shown, when the code package is loaded, the process substantially executed by the code package is mainly the relevant processing process in the code package generation process of the aforementioned steps, that is, when the downstream data usage engine responds to data query instructions, such as query SQL instructions, the code package can execute the steps of inheriting the abstract class for defining the file format, inheriting the interface for defining the reading method, inheriting the interface for defining the data output carrier, and inheriting the abstract class A for defining the serialization and deserialization logic. Exemplarily, the abstract class for defining the file format can be the FileInputFormat abstract class, the interface for defining the reading method can be the RecordReader interface, the interface for defining the data output carrier can be the Writable interface, and the abstract class for defining the serialization and deserialization logic can be the bstractSerDe abstract class, so that according to the type of data file defined by the FileInputFormat abstract class, the compression method, whether to slice and file filtering, and according to the data content reading method defined by the RecordReader interface, and the data output carrier defined by the Writable interface, and the data serialization and deserialization method defined by the AbstractSerDe abstract class, the data result of the data content to be read is output based on the aforementioned processing.

[0103] The data processing method provided by the embodiment of the present invention is suitable for large data volumes. For upstream and downstream interactive data, proto format storage can not only save storage resources, but also align metadata with each other. In addition, the embodiment of the present invention uses the Hive engine as a downstream data usage engine to realize the reading of proto data, so that SQL can be used to directly operate proto data. This method is simple and easy to use, and can repeatedly read and change the query and calculation logic of the data based on the generated reusable code package.

[0104] In an embodiment of the present invention, a structured data table is constructed by taking the data directory of the structured data file stored by the upstream data storage engine as the table construction information, and generating a code package containing class information based on the table construction information, so that the storage data in the specified structured data format can be used as a readable file source for the downstream data usage engine. On this basis, by loading the code package, the table metadata based on the target storage data to be read is realized, thereby obtaining the file data source of the storage data. Since the code package is generated by the class information of the structured data file, the table metadata in the data file and the structured data table can be independently updated in a decoupled manner based on the consistency of the serialization rules. By directly loading the generated code package, the table metadata of the updated content can be automatically updated, thereby realizing the reusability of the code package, reducing the difficulty of the downstream data usage engine in developing and using the data in the specified structured data format, and also being able to directly read the file data source to facilitate the troubleshooting of the original data.

[0105] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0106] Reference Figure 4 , shows a structural block diagram of a data processing device embodiment of the present invention, which is applied to a downstream data usage engine and may specifically include the following modules:

[0107] The data table construction module 401 is used to obtain the structured data file stored in the upstream data storage engine, and use the data directory of the structured data file as table construction information to construct a structured data table;

[0108] A code package generating module 402, configured to obtain class information from the table building information and generate a code package containing the class information;

[0109] A code package loading module 403 is used to load the code package and read table metadata of target storage data from the structured data table;

[0110] The data reading module 404 is used to obtain the file data source of the target storage data according to the table metadata of the target storage data.

[0111] In one embodiment of the present invention, the code package generation module 402 may include the following submodules:

[0112] The code package generation submodule is used to obtain class information from the table building information, read the data content of the structured data file based on the class information, and encapsulate the data content to obtain a code package.

[0113] In one embodiment of the present invention, the code package generation submodule may include the following units:

[0114] The data content reading unit is used to determine the calling entity of the data content based on the class information; and read the data content of the structured data file according to the calling entity.

[0115] In one embodiment of the present invention, the class information is used to define an interface for reading input data from a structured data file; the data content reading unit may include the following subunits:

[0116] A calling entity determining subunit, configured to determine a calling entity of the data content based on the interface for reading input data from the structured data file;

[0117] The calling entity at least includes an input stream, which is a source for reading compressed data. The class information is also used to define a reading method for the structured data file. The data content reading unit may include the following subunits:

[0118] The data content reading subunit is used to parse the compressed data in the input stream according to the reading method, and read the data content of the structured data file.

[0119] In an embodiment of the present invention, the class information is also used to define a data carrier, and the code package generation submodule may include the following units:

[0120] A code package generation unit is used to deserialize the data content to obtain deserialized data; put the deserialized data into the data carrier, perform data conversion on the deserialized data in the data carrier, and encapsulate to obtain a data object in the form of relational data; and package the class information and the data object in the form of relational data to obtain a code package.

[0121] In an embodiment of the present invention, the device provided by the embodiment of the present invention may further include the following modules:

[0122] An update module is used to change the table structure of the structured data table based on the code package when a field change occurs in the structured data file stored in the upstream data storage engine; wherein the changed table structure is used to point to the data directory location of the changed field, and the data directory location is used to read the file data source of the changed field.

[0123] In one embodiment of the present invention, the code package loading module 403 may include the following submodules:

[0124] The table metadata acquisition submodule is used to partition the structured data table to obtain at least one partition table; when loading the code package, each partition table is loaded regularly, and the table metadata of the target storage data is read from the target partition table.

[0125] In an embodiment of the present invention, the table metadata is used to point to the data directory location of the target storage data; the data reading module 404 may include the following submodules:

[0126] The data reading submodule is used to read the file data source of the target storage data based on the data directory location of the target storage data.

[0127] In an embodiment of the present invention, a data processing device provided by an embodiment of the present invention constructs a structured data table by taking the data directory of the structured data file stored by the upstream data storage engine as the table building information, and generates a code package containing class information based on the table building information, thereby using the storage data of the specified structured data format as a readable file source of the downstream data usage engine. On this basis, by loading the code package, the table metadata based on the target storage data to be read is realized, thereby obtaining the file data source of the storage data. Since the code package is generated by the class information of the structured data file, when the table metadata in the data file and the structured data table are independently updated in a decoupled manner based on the consistency of the serialization rules, the table metadata of the updated content can be automatically updated directly by loading the generated code package, thereby realizing the reusability of the code package, reducing the difficulty of the downstream data usage engine in developing and using the data in the specified structured data format, and also being able to directly read the file data source to facilitate the troubleshooting of the original data.

[0128] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0129] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0130] Memory 503, used for storing computer programs;

[0131] The processor 501 is used to execute the program stored in the memory 503, and implements the following steps:

[0132] Acquire a structured data file stored in an upstream data storage engine, and use the data directory of the structured data file as table building information to construct a structured data table;

[0133] Acquire class information from the table creation information, and generate a code package containing the class information;

[0134] Load the code package and read table metadata of target storage data from the structured data table;

[0135] According to the table metadata of the target storage data, the file data source of the target storage data is obtained.

[0136] In an embodiment of the present invention, an electronic device provided by an embodiment of the present invention constructs a structured data table by taking the data directory of the structured data file stored by the upstream data storage engine as the table building information, and generates a code package containing class information based on the table building information, thereby using the storage data of the specified structured data format as a readable file source of the downstream data usage engine. On this basis, by loading the code package, the table metadata based on the target storage data to be read is realized, thereby obtaining the file data source of the storage data. Since the code package is generated by the class information of the structured data file, when the table metadata in the data file and the structured data table are independently updated in a decoupled manner based on the consistency of the serialization rules, the table metadata of the updated content can be automatically updated directly by loading the generated code package, thereby realizing the reusability of the code package, reducing the difficulty of the downstream data usage engine in developing and using the data in the specified structured data format, and also being able to directly read the file data source to facilitate the troubleshooting of the original data.

[0137] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0138] The communication interface is used for communication between the above terminal and other devices.

[0139] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0140] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0141] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the data processing method described in any one of the above embodiments.

[0142] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the data processing method described in any one of the above embodiments.

[0143] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0144] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0145] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: Applied to a downstream data usage engine, the method comprises: Acquire a structured data file stored in an upstream data storage engine, and use the data directory of the structured data file as table building information to construct a structured data table; Acquire class information from the table creation information, and generate a code package containing the class information; Load the code package and read table metadata of target storage data from the structured data table; According to the table metadata of the target storage data, the file data source of the target storage data is obtained.

2. The method according to claim 1, characterized in that The acquiring class information from the table building information and generating a code package containing the class information includes: Acquire class information from the table creation information, and read data content of the structured data file based on the class information; The data content is encapsulated to obtain a code package.

3. The method according to claim 2, characterized in that The reading of the data content of the structured data file based on the class information includes: Determining a calling entity of the data content based on the class information; According to the calling entity, data content of the structured data file is read.

4. The method according to claim 3, characterized in that The class information is used to define an interface for reading input data from a structured data file; the calling entity for determining the data content based on the class information includes: Determine a calling entity of the data content based on the interface for reading input data from the structured data file; The calling entity at least includes an input stream, the input stream is a source for reading compressed data, and the class information is also used to define a reading method of the structured data file; reading the data content of the structured data file according to the calling entity includes: According to the reading method, the compressed data in the input stream is parsed to obtain the data content of the structured data file.

5. The method according to claim 2, characterized in that: The class information is also used to define a data carrier, and the data content is encapsulated to obtain a code package, including: Deserialize the data content to obtain deserialized data; Putting the deserialized data into the data carrier, performing data conversion on the deserialized data in the data carrier, and encapsulating to obtain a data object in the form of relational data; The class information and the data objects in the relational data form are packaged to obtain a code package.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: When a field change occurs in the structured data file stored in the upstream data storage engine, the table structure of the structured data table is changed based on the code package; wherein the changed table structure is used to point to the data directory location of the changed field, and the data directory location is used to read the file data source of the changed field.

7. The method according to claim 1, characterized in that The loading of the code package and reading table metadata of target storage data from the structured data table include: Partitioning the structured data table to obtain at least one partition table; When the code package is loaded, each partition table is loaded periodically, and table metadata of target storage data is read from the target partition table.

8. The method according to claim 1, characterized in that The table metadata is used to point to the data directory location of the target storage data; The step of obtaining the file data source of the target storage data according to the table metadata of the target storage data includes: Based on the data directory location of the target storage data, the file data source of the target storage data is read.

9. A data processing device, characterized in that: Applied to a downstream data usage engine, the device comprises: A data table construction module is used to obtain a structured data file stored in an upstream data storage engine, and use the data directory of the structured data file as table construction information to construct a structured data table; A code package generation module, used to obtain class information from the table creation information and generate a code package containing the class information; A code package loading module, used to load the code package and read table metadata of target storage data from the structured data table; The data reading module is used to obtain the file data source of the target storage data according to the table metadata of the target storage data.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the data processing method according to any one of claims 1 to 8 when executing a program stored in a memory.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method as described in any one of claims 1 to 8 is implemented.

12. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the data processing method according to any one of claims 1 to 8.

Images