Processing method, processor, system and storage medium for semi-structured data

Abstract

The application relates to the computer technical field, in particular to a semi-structured data processing method, a processor, a system and a storage medium. The method comprises the following steps: obtaining business data updated by a business system; pushing the business data to a kafka message queue; consuming the business data in the kafka message queue through a flink engine; converting the data format of the serialized business data into standard format business data of a data lake; analyzing the standard format business data to generate first object data corresponding to the business data; adding a newly-added field to the end of second object data of the business system when determining that a field in the first object data is newly added; storing the updated second object data to a distributed file system to update a business file stored in the distributed file system; and updating a file directory of a metadata management center according to the updated business file.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to a semi-structured data processing method, processor, system and storage medium. BACKGROUND

[0002] In the current technology, when structured data is written to a file, the object data corresponding to the structured data, that is, the schema, needs to be saved. According to the object data, the structured data is stored to facilitate user query in the data management system. However, business data is mostly semi-structured data, and semi-structured data does not have fixed object data, which cannot be directly imported into the data management system, so it is mostly imported by manual operation, which is labor-intensive and prone to errors.

[0003] In the prior art, there are difficulties in importing semi-structured data into the data management system, because the semi-structured data may contain object data that does not exist in the data management system, at which time the data management system needs to be rewritten and repaired to re-import or obtain the file in the data relationship system, which is labor-intensive and time-consuming. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a semi-structured data processing method, processor, system and storage medium.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a semi-structured data processing method, comprising:

[0006] Obtaining updated business data of a business system, the business data being semi-structured data;

[0007] Pushing the business data to a kafka message queue;

[0008] Consuming the business data in the kafka message queue by a flink engine to perform serialization processing on the business data in the kafka message queue;

[0009] Converting the data format of the serialized business data into a preset format;

[0010] Converting the business data in the preset format into business data in the standard format of a data lake according to a preset specification table;

[0011] Parsing the business data in the standard format to generate first object data corresponding to the business data;

[0012] In a case where it is determined that a field in the first object data is newly added relative to second object data of the business system, adding the newly added field to the end of the second object data;

[0013] store the updated second object data to the distributed file system to update the business file stored in the distributed file system;

[0014] update the file directory of the metadata management center according to the updated business file.

[0015] In the embodiments of the present application, the processing method further comprises: after updating the file directory of the metadata management center according to the updated business file, sending an update notification to the demand side by the metadata management center, the update notification carrying partition information of the updated business data; in the case that the demand side calls the corresponding data interface in the metadata management center according to the partition information, querying the updated business file corresponding to the partition information in the distributed file system according to the partition information.

[0016] In the embodiments of the present application, the serialization processing of the business data in the kafka message queue by the flink engine comprises: creating a JAVA object, the JAVA object including a schema field and a data field; running the JAVA object to extract the schema in the business data and store it in the schema field, and extract the business fields in the business data and the field values of each business field and store them in the data field.

[0017] In the embodiments of the present application, the preset format is rowdata format, wherein the order of the business fields contained in the data field in the rowdata format is determined according to the serial number of each business field.

[0018] In the embodiments of the present application, in the case that it is determined that the fields in the first object data are newly added with respect to the second object data of the business system, adding the newly added fields to the end of the second object data comprises: obtaining the collection data of the business system from the memory; parsing the collection data to obtain the second object data of the collection data; comparing the fields contained in the first object data and the second object data; in the case that it is determined that the fields in the first object data are newly added with respect to the second object data, adding the newly added fields to the end of the second object data to generate third object data; updating the collection data according to the third object data, so that the third object data corresponds to the position of the collection data.

[0019] In the embodiments of the present application, updating the file directory of the metadata management center according to the updated business file comprises: obtaining fourth object data corresponding to the business file data every interval of a preset time period; in the case that the fourth object data has newly added fields compared with the fifth object data corresponding to the file directory of the metadata management center, modifying the file directory of the metadata management center according to the newly added fields.

[0020] The second aspect of the present application provides a processor configured to execute the semi-structured data processing method described above.

[0021] The third aspect of the present application provides a semi-structured data processing system, comprising: a kafka message queue for temporarily storing business data; a flink engine for consuming the business data in the kafka message queue to perform serialization processing on the business data in the kafka message queue; a data lake for storing business data in a standard format; a distributed file system for storing business files; a metadata management center for storing a file directory of the business files; and the processor described above.

[0022] In the embodiments of the present application, the metadata management center is further configured to call a corresponding data interface and send an update notification to the demand side after updating the file directory of the metadata management center according to the updated business file.

[0023] The fourth aspect of the present application provides a programmable storage medium having instructions stored thereon, which, when executed by a processor, cause the processor to be configured to execute the semi-structured data processing method described above.

[0024] Through the above technical solution, the processor can obtain business data updated by a business system, the business data being semi-structured data; push the business data to a kafka message queue; consume the business data in the kafka message queue by a flink engine to perform serialization processing on the business data in the kafka message queue; convert the data format of the serialized business data into a preset format; convert the business data in the preset format into business data in a standard format of a data lake according to a preset specification table; parse the business data in the standard format to generate first object data corresponding to the business data; add a newly added field to the end of second object data of the business system when it is determined that the field in the first object data is newly added; store the updated second object data to a distributed file system to update the business files stored in the distributed file system; and update the file directory of the metadata management center according to the updated business file. The semi-structured business data can be converted into business files in a specific format and stored in the distributed file system, so that users can obtain the business files in the distributed file system. The efficiency of data processing is improved, and errors that may occur during manual data input are avoided. In the above process, the processor can import the business data into the distributed file system in real time, without the need to update and repair the data management system, thereby ensuring the smooth operation of the data management system.

[0025] Other features and advantages of the embodiments of the present application will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings are included to provide a further understanding of embodiments of the application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the application, and together with the description serve to explain the principles of the application. In the drawings:

[0027] Figure 1 A flowchart of a processing method of semi-structured data according to an embodiment of the application is schematically shown;

[0028] Figure 2 A structural block diagram of a processing system of semi-structured data according to an embodiment of the application is schematically shown;

[0029] Figure 3 An internal structural diagram of a computer device according to an embodiment of the application is schematically shown. DETAILED DESCRIPTION

[0030] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the specific implementation manners described herein are only used to explain and explain the embodiments of the present application and do not limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0031] Figure 1 A flowchart of a processing method of semi-structured data according to an embodiment of the application is schematically shown. As shown in Figure 1 In an embodiment of the present application, a processing method of semi-structured data is provided, comprising the following steps:

[0032] S202, obtaining business data updated by a business system, the business data being semi-structured data.

[0033] S204, pushing the business data to a kafka message queue.

[0034] S206, consuming the business data in the kafka message queue through a flink engine to perform serialization processing on the business data in the kafka message queue.

[0035] S208, converting the data format of the serialized business data into a preset format.

[0036] S210, converting the business data in the preset format into business data in a standard format of a data lake according to a preset specification table.

[0037] S112, parse the service data in the standard format to generate first object data corresponding to the service data.

[0038] S114, in a case where it is determined that a field in the first object data is newly added with respect to second object data of the service system, add the newly added field to the end of the second object data.

[0039] S116, store the updated second object data to the distributed file system to update the service file stored in the distributed file system.

[0040] S118, update the file directory of the metadata management center according to the updated service file.

[0041] Semi-structured data is data between completely structured data (such as relational data) and completely unstructured data (such as audio and video). It does not conform to the data model structure of the relational database or other data table form association, but contains relevant tags to separate semantic elements and layer records and fields. Common semi-structured data includes HTML, XML, and JSON documents. The service data in this application is also a semi-structured data, which is the data recorded in the work process and can include object data that the structured data in the data management system does not have. It is necessary to parse and confirm the semi-structured data and convert it into service data in the data lake format before importing it into the database. Therefore, the semi-structured data can be serialized and format-converted, and the object data and content data in the service data can be parsed and then updated to the distributed file system. After the semi-structured data conversion, it is imported into the distributed file system. The distributed file system can store service files and can be accessed by users through a specific way to obtain files in the system. The data lake can store service data in a standard format. The flink engine is a real-time computing engine that can analyze and calculate data in real time. The metadata management center can generate a file directory of the service file in the distributed file system according to the object data corresponding to the distributed file system, so that users can obtain the service file in the distributed system through a specific interface.

[0042] Firstly, the processor can acquire business data in the business system, wherein the business data exported from the business system is semi-structured data and cannot be directly imported into the data lake or the distributed file system. The processor pushes the business data to the kafka message queue to temporarily store the business data. Then the processor consumes the business data in the kafka message queue through the flink engine to serialize the business data in the kafka message queue, and classifies and saves the object data and the content data in the business data. After obtaining the serialized business data, the processor can convert the serialized business data into a preset format to facilitate further processing of the business data by the processor. The processor can convert the business data in the preset format into the standard format of the data lake according to the preset specification table corresponding to the data lake, so that the processor can import the business data into the data lake. The processor can parse the business data in the standard format to obtain first object data corresponding to the business data, wherein the first object data can include a plurality of object fields. The processor compares the fields in the first object data with second object data of the business system, and determines that the fields in the first object data are newly added compared with the second object data of the business system. The processor can add the newly added fields to the end of the second object data. The second object data of the business data refers to the object data determined by importing the business system into the distributed file system before acquiring the business system data this time, which contains the object data corresponding to all previous business data. If the semi-structured business data has not been written into the distributed file system before, the second object data is blank, and the processor can add the first object data to the second object data to generate updated second object data. Finally, the processor stores the updated second object data in the distributed file system, and the processor can update the data in the distributed file system according to the updated second object data, and update the business file in the distributed object system. The processor can also update the file directory of the metadata management center according to the updated business file, so that the file directory stored in the metadata management center is the latest business file directory.

[0043] By the technical solution, the processor can acquire the service data updated by the service system, the service data being semi-structured data; push the service data to the kafka message queue; consume the service data in the kafka message queue through the flink engine to perform serialization processing on the service data in the kafka message queue; convert the data format of the service data after the serialization processing into a preset format; convert the service data in the preset format into service data in a standard format of the data lake according to a preset specification table; parse the service data in the standard format to generate first object data corresponding to the service data; in a case where it is determined that a field in the first object data is newly added relative to second object data of the service system, add the newly added field to the end of the second object data; store the updated second object data to the distributed file system to update the service file stored in the distributed file system; and update the file directory of the metadata management center according to the updated service file. The semi-structured service data can be converted into a service file in a specific format and stored in the distributed file system, so that the user can acquire the service file in the distributed file system. The efficiency of data processing is improved, and errors that can occur in manual data input are avoided. In the above process, the processor can import the service data into the distributed file system in real time, without the need to re-update and repair the data management system, thereby ensuring smooth operation of the data management system.

[0044] In one embodiment, the processing method further includes: after updating the file directory of the metadata management center according to the updated service file, sending an update notification to the demand side through the metadata management center, the update notification carrying partition information of the updated service data; and in a case where the demand side calls a corresponding data interface in the metadata management center according to the partition information, querying the updated service file corresponding to the partition information in the distributed file system according to the partition information. After updating the file directory of the metadata management center, the processor can send an update notification to the demand side through the metadata management center, and the update notification contains partition information of the updated service data. The demand side can query the corresponding service file according to the acquired partition information.

[0045] In an embodiment, the processing of the business data in the kafka message queue by the flink engine includes: creating a JAVA object, the JAVA object including a schema field and a data field; and running the JAVA object to extract the schema in the business data and store the schema in the schema field, and to extract the business fields in the business data and the field values of each business field and store the business fields and the field values in the data field. The processor can parse the data in the kafka message queue by the flink engine, serialize the data in the kafka queue, and create a JAVA object including a schema field and a data field in the serialization process. After the processor parses the business data in the kafka message queue, the processor extracts the business fields and the schema field in the business data, stores the schema data in the business data in the schema field of the JAVA object, and stores the business fields in the business data in the data field of the JAVA object, to complete the serialization of the business data.

[0046] In an embodiment, the preset format is a rowdata format, and the order of the business fields contained in the data field in the rowdata format is determined according to the serial numbers of the business fields. The processor can convert the serialized business data into the rowdata format, and number the business fields in the serialized business data. In the present application, the order of the business fields is determined according to the serial numbers of the business fields in the data field, and the business fields with smaller serial numbers are arranged in front and the business fields with larger serial numbers are arranged in back. The obtained rowdata data can make the flink engine process data more efficiently and faster.

[0047] In one embodiment, when it is determined that a field in the first object data is newly added with respect to the second object data of the business system, adding the newly added field to the end of the second object data comprises: obtaining the collection data of the business system from the memory; parsing the collection data to obtain the second object data of the collection data; comparing the fields contained in the first object data and the second object data; when it is determined that a field in the first object data is newly added with respect to the second object data, adding the newly added field to the end of the second object data to generate third object data; and updating the collection data according to the third object data so that the third object data corresponds to the position of the collection data. After obtaining the first object data, the processor can compare the first object data with the second object data of the collection data in the distributed file system. When there is a field in the first object data that does not exist in the second object data, the newly added field is added to the end of the second object data to generate third object data, which corresponds to the latest business data. The processor can update the collection data in the distributed file system according to the third object data, and update the latest standard format business data to the business file in the distributed object system.

[0048] In one embodiment, updating the file directory of the metadata management center according to the updated business file comprises: obtaining fourth object data corresponding to the business file data at intervals of a preset time period; and when there is a newly added field in the fourth object data compared with the fifth object data corresponding to the file directory of the metadata management center, modifying the file directory of the metadata management center according to the newly added field. The processor can obtain the fourth object data corresponding to the business file data in the distributed file system and the fifth object data corresponding to the file directory of the metadata management center at intervals of a preset time period. When there is a newly added field in the fourth object data compared with the fifth object data, the processor can modify the file directory of the metadata management center according to the newly added field, so that the user can obtain the latest file directory through a specific interface and further obtain the latest business data.

[0049] In one specific embodiment, the processor acquires business data in a business system, and the acquired business data is semi-structured data. The processor pushes the acquired business data to a kafka message queue, and consumes the business data in the kafka message queue through a flink engine to perform serialization processing on the business data in the kafka message queue. The serialization processing on the business data refers to creating a JAVA object, and the JAVA object includes a schema field and a data field. The processor runs the JAVA object to extract the schema in the business data and store it in the schema field, and extract the business fields in the business data and the field values of each business field and store them in the data field. The processor converts the data format of the serialized business data into a rowdata format, and sorts the business fields in the business data according to the serial numbers of the business fields in the business data in the rowdata format. The processor converts the business data in the preset format into business data in the standard format of the data lake according to the preset specification table, and parses the business data in the standard format to generate first object data corresponding to the business data. Subsequently, the processor acquires collection data of a distributed file system from the memory, and parses the collection data to obtain second object data corresponding to the collection data. The processor compares the fields contained in the first object data and the second object data, and in a case where it is determined that a field in the first object data is newly added with respect to the second object data, adds the newly added field to the end of the second object data to generate third object data. The processor updates the collection data according to the third object data, so that the third object data corresponds to the position of the collection data. The processor acquires fourth object data corresponding to the business file data every interval of a preset time period; in a case where the fourth object data has a newly added field compared to fifth object data corresponding to a file directory of a metadata management center, modifies the file directory of the metadata management center according to the newly added field. After updating the file directory of the metadata management center according to the updated business file, the processor sends an update notification to the demand side through the metadata management center, and the update notification carries partition information of the updated business data. In a case where the demand side calls a corresponding data interface in the metadata management center according to the partition information, the updated business file corresponding to the partition information can be queried in the distributed file system according to the partition information.

[0050] Through the above method, semi-structured business data can be converted into business files in a specific format and stored in a distributed file system, so that users can acquire business files in the distributed file system. The efficiency of data processing is improved, and errors that may occur during manual data input are avoided. In the above process, the processor can import business data into the distributed file system in real time, without the need to update and repair the data management system, ensuring smooth operation of the data management system.

[0051] Figure 1 This is a flowchart illustrating a method for processing semi-structured data in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0052] In one embodiment, such as Figure 2 As shown, a semi-structured data processing system is provided, including a Kafka message queue 201, a Flink engine 202, a data lake 203, a distributed file system 204, a metadata management center 205, and a processor 206, wherein:

[0053] Kafka message queue 201 is used to temporarily store business data.

[0054] The Flink engine 202 is used to consume business data from the Kafka message queue for serialization processing.

[0055] Data Lake 203 is used to store business data in a standard format.

[0056] Distributed file system 204 is used to store business files.

[0057] Metadata Management Center 205 is a file directory used to store business files.

[0058] Processor 206 is used to execute the above-described semi-structured data processing method.

[0059] In one embodiment, the metadata management center is also used to call the corresponding data interface and send an update notification to the requester after updating the file directory of the metadata management center according to the updated business file.

[0060] The processing system of the semi-structured data comprises a processor and a memory, and the above-mentioned kafka message queue 201, flink engine 202, data lake 203, distributed file system 204, metadata management center 205 and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the above-mentioned program modules stored in the memory.

[0061] The processor comprises a core, and the core retrieves the corresponding program unit in the memory. The core can be one or more, and the processing method of the semi-structured data is realized by adjusting the core parameters.

[0062] The memory can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0063] The embodiment of the application provides a storage medium, which stores a program, and the program is executed by a processor to realize the processing method of the semi-structured data.

[0064] The embodiment of the application provides a processor, which is used for running a program, and the program is executed to realize the processing method of the semi-structured data.

[0065] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 3 The computer device comprises a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. The processor A01 of the computer device is used for providing computing and control capabilities. The memory of the computer device comprises an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The network interface A02 of the computer device is used for communicating with an external terminal through a network connection. The computer program B02 is executed by the processor A01 to realize a processing method of semi-structured data.

[0066] Those skilled in the art can understand that Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the application, and does not constitute a limitation on the computer device to which the scheme of the application is applied. The specific computer device can comprise more or fewer components than those shown in the figure, or some components can be combined, or have a different component arrangement.

[0067] An embodiment of the present application provides a device, which comprises a processor, a memory, and a program stored on the memory and executable on the processor, and the processor implements the following steps when executing the program: obtaining service data updated by a service system, the service data being semi-structured data; pushing the service data to a kafka message queue; consuming the service data in the kafka message queue through a flink engine to perform serialization processing on the service data in the kafka message queue; converting a data format of the service data after the serialization processing into a preset format; converting the service data in the preset format into service data in a standard format of a data lake according to a preset specification table; parsing the service data in the standard format to generate first object data corresponding to the service data; in a case where it is determined that a field in the first object data is newly added relative to second object data of the service system, adding the newly added field to an end of the second object data; storing the updated second object data to a distributed file system to update a service file stored in the distributed file system; and updating a file directory of a metadata management center according to the updated service file.

[0068] In one embodiment, the processing method further comprises: after updating the file directory of the metadata management center according to the updated service file, sending an update notification to a demand side through the metadata management center, the update notification carrying partition information of the updated service data; and in a case where the demand side calls a corresponding data interface in the metadata management center according to the partition information, querying the updated service file corresponding to the partition information in the distributed file system according to the partition information.

[0069] In one embodiment, the processing method further comprises: creating a JAVA object, the JAVA object comprising a schema field and a data field; and running the JAVA object to extract the schema in the service data and store the schema in the schema field, and to extract service fields in the service data and field values of each service field and store the service fields and the field values in the data field.

[0070] In one embodiment, the preset format is a rowdata format, and an order of the service fields contained in the data field in the rowdata format is determined according to serial numbers of the service fields.

[0071] In one embodiment, when it is determined that the field in the first object data is newly added with respect to the second object data of the business system, adding the newly added field to the end of the second object data comprises: obtaining collection data of the business system from the memory; parsing the collection data to obtain the second object data of the collection data; comparing the fields contained in the first object data and the second object data; when it is determined that the field in the first object data is newly added with respect to the second object data, adding the newly added field to the end of the second object data to generate third object data; and updating the collection data according to the third object data, so that the third object data corresponds to the collection data in position.

[0072] In one embodiment, updating the file directory of the metadata management center according to the updated business file comprises: obtaining fourth object data corresponding to the business file data at every interval of a preset time period; and when there is a newly added field in the fourth object data compared with fifth object data corresponding to the file directory of the metadata management center, modifying the file directory of the metadata management center according to the newly added field.

[0073] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0074] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce an apparatus for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0075] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1the function(s) specified in the block or blocks.

[0076] These computer program instructions can also be loaded into computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the flowchart block(s) or flowchart flow(s) and / or portions thereof. Figure 1 the flowchart block(s) or flowchart flow(s) and / or portions thereof. Figure 1 the function(s) specified in the block or blocks.

[0077] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0078] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores information such as computer program instructions. Memory is an example of computer readable media.

[0079] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0080] It should also be noted that the terms "comprising", "comprises", "including", "includes" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article or apparatus that comprises the element.

[0081] The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. A method for processing semi-structured data, characterized in that, The processing method includes: Obtain updated business data from the business system; the business data is semi-structured data. The business data is pushed to the Kafka message queue; The business data in the Kafka message queue is consumed through the Flink engine to perform serialization processing on the business data in the Kafka message queue; Convert the serialized business data into a preset format; The business data in the preset format is converted into the standard format of the data lake according to the preset specification table; Parse the standard format business data to generate first object data corresponding to the business data; If it is determined that a field in the first object data has been added relative to the second object data of the business system, the newly added field is added to the end of the second object data; The updated second object data is stored in a distributed file system to update the business files stored in the distributed file system; Update the file directory of the metadata management center according to the updated business documents; Wherein, when it is determined that a field in the first object data has been added relative to the second object data of the business system, adding the newly added field to the end of the second object data includes: Retrieve the collection data of the business system from memory; The set of data is parsed to obtain the second object data of the set of data; Compare the fields contained in the first object data with those contained in the second object data; If it is determined that a field in the first object data has been added relative to the second object data, the newly added field is added to the end of the second object data to generate the third object data; The set data is updated based on the third object data so that the third object data corresponds to the position of the set data.

2. The method for processing semi-structured data according to claim 1, characterized in that, The processing method further includes: After updating the file directory of the metadata management center according to the updated business files, the metadata management center sends an update notification to the requester, and the update notification carries the partition information of the updated business data. When the requester calls the corresponding data interface in the metadata management center based on the partition information, the updated business file corresponding to the partition information is queried in the distributed file system based on the partition information.

3. The method for processing semi-structured data according to claim 1, characterized in that, The step of consuming business data from the Kafka message queue using the Flink engine to perform serialization processing on the business data in the Kafka message queue includes: Create a Java object, which includes a schema field and a data field; Run the JAVA object to extract the schema from the business data and store it in the schema field, and extract the business fields and the field value of each business field from the business data and store them in the data field.

4. The method for processing semi-structured data according to claim 3, characterized in that, The preset format is rowdata format, wherein the order of the business fields contained in the data field of the rowdata format is determined according to the ordinal number of each business field.

5. The method for processing semi-structured data according to claim 1, characterized in that, The process of updating the file directory of the metadata management center based on the updated business files includes: The fourth object data corresponding to the business file data is retrieved at preset time intervals; If the fourth object data has a new field compared to the fifth object data corresponding to the file directory of the metadata management center, the file directory of the metadata management center is modified according to the new field.

6. A processor, characterized in that, It is configured to perform the semi-structured data processing method according to any one of claims 1 to 5.

7. A semi-structured data processing system, characterized in that, include: Kafka message queues are used to temporarily store business data. The Flink engine is used to consume business data in the Kafka message queue and perform serialization processing on the business data in the Kafka message queue. A data lake is used to store business data in a standard format. A distributed file system used to store business files; Metadata management center, a file directory used to store business files; as well as The processor as described in claim 6.

8. The semi-structured data processing system according to claim 7, characterized in that, The metadata management center is also used to call the corresponding data interface, and to send an update notification to the requesting party after updating the file directory of the metadata management center according to the updated business file.

9. A programmable storage medium storing instructions, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the semi-structured data processing method according to any one of claims 1 to 5.

Images