A data synchronization method and device for unstructured data

Abstract

The application relates to a data synchronization method and device for unstructured data, and the method comprises the following steps: writing unstructured data into MongoDB and Elasticsearch; creating a second data table corresponding to a first data table in MongoDB or Elasticsearch in Hive or Doris; parsing a field needing to be structured and storing the field into a first list; reading target data in the first data table from MongoDB or Elasticsearch, storing a target field in the target data into a second list, and storing a field that must exist in the target data into a third list; writing data in the second list into a specified file; and executing the specified file to synchronize the data to Hive or Doris. Through the application, the problem that the data volume of unstructured data needing to be synchronized is large, and structured synchronization code is manually written, thereby leading to low data synchronization efficiency in the prior art is solved, and the effect of improving the data synchronization efficiency is achieved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, computer device, and computer-readable storage medium for synchronizing unstructured data. Background Technology

[0002] The unstructured data synchronization process in related technologies may include: writing the acquired unstructured data into MongoDB and Elasticsearch; then the developers review the requirements, determine the fields to be parsed, parse each field one by one, and manually write code using Python; then the manually written code is executed to write the data that needs to be analyzed in a timely manner into Doris, and the rest into Hive.

[0003] The above-mentioned unstructured data synchronization process has the following problems:

[0004] 1. There are a large number of tables that need to be synchronized from MongoDB and Elasticsearch, reaching more than 500 tables. The structure of each table is different, and the required structured fields are different. Manually writing structured synchronization code is a huge workload, very inefficient, and takes a very long time.

[0005] 2. The fields that need to be parsed in each table should be manually checked and determined. Manual processing is prone to errors in field writing or logical problems. If these are not discovered, there may be unpredictable problems or economic losses.

[0006] Currently, no effective solution has been proposed to address the problem of low data synchronization efficiency caused by manually writing structured synchronization code due to the large volume of unstructured data that needs to be synchronized in related technologies. Summary of the Invention

[0007] The purpose of this application is to address the shortcomings of the prior art by providing a data synchronization method, apparatus, computer device, and computer-readable storage medium for unstructured data, so as to at least solve the problem that the large amount of unstructured data to be synchronized and the need for manually writing structured synchronization code lead to low data synchronization efficiency in the related art.

[0008] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0009] In a first aspect, embodiments of this application provide a method for synchronizing unstructured data, including:

[0010] Unstructured data is written to MongoDB and Elasticsearch, wherein the MongoDB or Elasticsearch includes a first data table;

[0011] Create a second data table in Hive or Doris that corresponds to the first data table, wherein the second data table includes fields that need to be structured;

[0012] Parse the fields that need to be structured and store them in the first list;

[0013] Read the target data from the first data table from MongoDB or Elasticsearch, store the target fields in the target data in the second list, and store the fields that must exist in the target data in the third list, wherein the target fields are data fields whose key values ​​exist in the first list;

[0014] Write the data from the second list to the specified file;

[0015] Execute the specified file to synchronize the data to Hive or Doris.

[0016] In some embodiments, parsing the fields that need to be structured and storing them in a first list includes:

[0017] Use the pyhive or pymysql packages provided by Python to read the schema of the second data table from Hive or Doris to obtain the fields that need to be structured;

[0018] The fields that need to be structured are stored in the first list.

[0019] In some embodiments, after storing the fields that must exist in the target data into a third list, the method further includes:

[0020] Determine whether the required fields can be retrieved;

[0021] If the required fields can be obtained, then the step of writing the data in the second list to the specified file is executed;

[0022] If the required fields cannot be obtained, the data synchronization operation will end.

[0023] In some of these embodiments, writing unstructured data to MongoDB and Elasticsearch includes:

[0024] Write system execution logs to Elasticsearch in an unstructured format;

[0025] Write unstructured data, excluding the system execution logs, to MongoDB.

[0026] Secondly, embodiments of this application provide a data synchronization device for unstructured data, comprising:

[0027] The first writing unit is used to write unstructured data into MongoDB and Elasticsearch, wherein the MongoDB or Elasticsearch includes a first data table;

[0028] A creation unit is used to create a second data table in Hive or Doris that corresponds to the first data table, wherein the second data table includes fields that need to be structured;

[0029] A parsing unit is used to parse the fields that need to be structured and store them in a first list;

[0030] The processing unit is used to read target data from the first data table from MongoDB or Elasticsearch, store the target fields in the target data into a second list, and store the fields that must exist in the target data into a third list, wherein the target fields are data fields whose key values ​​exist in the first list;

[0031] The second writing unit is used to write the data in the second list to a specified file;

[0032] The synchronization unit is used to execute the specified file and synchronize the data to Hive or Doris.

[0033] In some embodiments, the parsing unit includes:

[0034] The read module is used to read the schema of the second data table from Hive or Doris using the pyhive or pymysql packages provided by Python, so as to obtain the fields that need to be structured;

[0035] A storage module is used to store the fields that need to be structured into the first list.

[0036] In some embodiments, it also includes:

[0037] The judgment unit is used to determine whether the required fields in the target data can be obtained after storing them in the third list;

[0038] An execution unit is configured to execute the step of writing the data in the second list to a specified file if the required field can be obtained;

[0039] The termination unit is used to terminate the data synchronization operation if the required field cannot be obtained.

[0040] In some embodiments, the first writing unit includes:

[0041] The first write module is used to write system execution logs to Elasticsearch in an unstructured form;

[0042] The second write module is used to write unstructured data other than the system execution log into MongoDB.

[0043] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data synchronization method for unstructured data as described in the first aspect above.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data synchronization method for unstructured data as described in the first aspect above.

[0045] This application adopts the above technical solution. Compared with the prior art, the data synchronization method for unstructured data provided in this application embodiment writes unstructured data into MongoDB and Elasticsearch, wherein the MongoDB or Elasticsearch includes a first data table; creates a second data table in Hive or Doris corresponding to the first data table, wherein the second data table includes fields that need to be structured; parses the fields that need to be structured and stores them in a first list; reads target data from the first data table from MongoDB or Elasticsearch, stores the target fields in the target data in a second list, and stores the fields that must exist in the target data in a third list, wherein the target fields are data fields whose key values ​​exist in the first list; writes the data in the second list into a specified file; executes the specified file to synchronize the data to Hive or Doris. This solves the problem in related technologies where the amount of unstructured data to be synchronized is large and the manual writing of structured synchronization code leads to low data synchronization efficiency, thus achieving the effect of improving data synchronization efficiency.

[0046] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0048] Figure 1 This is a structural block diagram of a mobile terminal according to an embodiment of this application;

[0049] Figure 2 This is a flowchart of a data synchronization method for unstructured data according to an embodiment of this application;

[0050] Figure 3 This is a flowchart of a data synchronization method for unstructured data according to a preferred embodiment of this application;

[0051] Figure 4 This is a flowchart of the automated code generation process according to a preferred embodiment of this application;

[0052] Figure 5 This is a structural block diagram of a data synchronization device for unstructured data according to an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0055] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0056] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0057] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0058] This embodiment provides a mobile terminal. Figure 1 This is a structural block diagram of a mobile terminal according to an embodiment of this application. For example... Figure 1 As shown, the mobile terminal includes: a radio frequency (RF) circuit 110, a memory 120, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a wireless fidelity (WiFi) module 170, a processor 180, and a power supply 190, among other components. Those skilled in the art will understand that... Figure 1 The mobile terminal structure shown does not constitute a limitation on the mobile terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0059] The following is combined Figure 1A detailed introduction to each component of a mobile terminal:

[0060] RF circuit 110 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 180; additionally, it transmits uplink data to the base station. Typically, RF circuitry includes, but is not limited to, antennas, at least one amplifier, transceiver, coupler, low-noise amplifier (LNA), duplexer, etc. Furthermore, RF circuit 110 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0061] The memory 120 can be used to store software programs and modules. The processor 180 executes various functional applications and data processing of the mobile terminal by running the software programs and modules stored in the memory 120. The memory 120 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile terminal (such as audio data, phone book, etc.). In addition, the memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0062] The input unit 130 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile terminal. Specifically, the input unit 130 may include a touch panel 131 and other input devices 132. The touch panel 131, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 131), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 131 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 180, and can receive and execute commands sent by the processor 180. In addition, the touch panel 131 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 131, the input unit 130 may also include other input devices 132. Specifically, other input devices 132 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0063] Display unit 140 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile terminal. Display unit 140 may include a display panel 141, optionally configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 141. Further, touch panel 131 may cover display panel 141. When touch panel 131 detects a touch operation on or near it, it transmits the information to processor 180 to determine the type of touch event. Subsequently, processor 180 provides corresponding visual output on display panel 141 based on the type of touch event. Although in Figure 1 In this embodiment, the touch panel 131 and the display panel 141 are two separate components to realize the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 131 and the display panel 141 can be integrated to realize the input and output functions of the mobile terminal.

[0064] The mobile terminal may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 141 according to the ambient light level, and the proximity sensor can turn off the display panel 141 and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that identify the mobile terminal's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that may be configured in the mobile terminal, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0065] The speaker 161 and microphone 162 in the audio circuit 160 provide an audio interface between the user and the mobile terminal. The audio circuit 160 can convert the received audio data into electrical signals and transmit them to the speaker 161, where the speaker 161 converts them into sound signals for output. On the other hand, the microphone 162 converts the collected sound signals into electrical signals, which are received by the audio circuit 160, converted into audio data, and then output to the processor 180 for processing. After processing, the audio data is transmitted via the RF circuit 110 to, for example, another mobile terminal, or the audio data is output to the memory 120 for further processing.

[0066] WiFi is a short-range wireless transmission technology. Mobile terminals using a WiFi module 170 can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 1 WiFi module 170 is shown, but it is understood that it is not a necessary component of a mobile terminal and can be omitted or replaced with other short-range wireless transmission modules, such as Zigbee modules or WAPI modules, as needed without changing the nature of the invention.

[0067] The processor 180 is the control center of the mobile terminal. It connects various parts of the mobile terminal via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 120, and by calling data stored in the memory 120, it performs various functions and processes data of the mobile terminal, thereby providing overall monitoring of the mobile terminal. Optionally, the processor 180 may include one or more processing units; preferably, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 180.

[0068] The mobile terminal also includes a power supply 190 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 180 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0069] Although not shown, mobile terminals may also include cameras, Bluetooth modules, etc., which will not be elaborated here.

[0070] In this embodiment, the processor 180 is configured as follows:

[0071] Unstructured data is written to MongoDB and Elasticsearch, wherein the MongoDB or Elasticsearch includes a first data table;

[0072] Create a second data table in Hive or Doris that corresponds to the first data table, wherein the second data table includes fields that need to be structured;

[0073] Parse the fields that need to be structured and store them in the first list;

[0074] Read the target data from the first data table from MongoDB or Elasticsearch, store the target fields in the target data in the second list, and store the fields that must exist in the target data in the third list, wherein the target fields are data fields whose key values ​​exist in the first list;

[0075] Write the data from the second list to the specified file;

[0076] Execute the specified file to synchronize the data to Hive or Doris.

[0077] In some embodiments, the processor 180 is further configured to:

[0078] Use the pyhive or pymysql packages provided by Python to read the schema of the second data table from Hive or Doris to obtain the fields that need to be structured;

[0079] The fields that need to be structured are stored in the first list.

[0080] In some embodiments, the processor 180 is further configured to:

[0081] Determine whether the required fields can be retrieved;

[0082] If the required fields can be obtained, then the step of writing the data in the second list to the specified file is executed;

[0083] If the required fields cannot be obtained, the data synchronization operation will end.

[0084] In some embodiments, the processor 180 is further configured to:

[0085] Write system execution logs to Elasticsearch in an unstructured format;

[0086] Write unstructured data, excluding the system execution logs, to MongoDB.

[0087] This embodiment provides a method for synchronizing unstructured data. Figure 2 This is a flowchart of a data synchronization method for unstructured data according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0088] Step S201: Write unstructured data into MongoDB and Elasticsearch, wherein the MongoDB or Elasticsearch includes a first data table;

[0089] Step S202: Create a second data table in Hive or Doris that corresponds to the first data table, wherein the second data table includes fields that need to be structured;

[0090] Step S203: Parse the fields that need to be structured and store them in the first list;

[0091] Step S204: Read the target data from the first data table from MongoDB or Elasticsearch, store the target fields in the target data in the second list, and store the fields that must exist in the target data in the third list, wherein the target fields are data fields whose key values ​​exist in the first list;

[0092] Step S205: Write the data in the second list to the specified file;

[0093] Step S206: Execute the specified file to synchronize the data to Hive or Doris.

[0094] The first data table in the above steps can be any table in MongoDB or Elasticsearch, for example, the first data table could be `event_model`. The second data table is the corresponding data table in Hive or Doris, and it can include the fields that need to be structured. The first, second, and third lists are lists, used to store the fields that need to be structured, the target fields, and the fields that must exist, respectively. The target fields are the data fields whose keys exist in the `f_list`. The specified file can be a `.py` file, such as `event_model.py`.

[0095] By following the steps above, the problem of low data synchronization efficiency caused by the large volume of unstructured data that needs to be synchronized and the need for manually writing structured synchronization code can be solved, thereby improving the technical effect of data synchronization efficiency.

[0096] In some embodiments, parsing the fields that need to be structured and storing them in a first list includes:

[0097] Use the pyhive or pymysql packages provided by Python to read the schema of the second data table from Hive or Doris to obtain the fields that need to be structured;

[0098] The fields that need to be structured are stored in the first list.

[0099] In some embodiments, after storing the fields that must exist in the target data into a third list, the method further includes:

[0100] Determine whether the required fields can be retrieved;

[0101] If the required fields can be obtained, then the step of writing the data in the second list to the specified file is executed;

[0102] If the required fields cannot be obtained, the data synchronization operation will end.

[0103] In some of these embodiments, writing unstructured data to MongoDB and Elasticsearch includes:

[0104] Write system execution logs to Elasticsearch in an unstructured format;

[0105] Write unstructured data, excluding the system execution logs, to MongoDB.

[0106] The embodiments of this application will be described and illustrated below through preferred embodiments.

[0107] This preferred embodiment, inspired by low-code principles, aims to minimize code writing by developers for large volumes of unstructured data requiring synchronization. It features an automated code generation system. The system automatically generates code based on the MongoDB table, Elasticsearch index name, required fields (unstructured data is all JSON; some fields require a parent table that doesn't exist, rendering the parsed data meaningless), and the fields to be parsed. Once the required fields are identified, the system reads the table schema from Hive or Doris to retrieve the necessary fields. Since the final data is written to Hive and Doris, the corresponding tables must be created and verified beforehand to prevent errors in parsing fields.

[0108] Figure 3 This is a flowchart of a data synchronization method for unstructured data according to a preferred embodiment of this application, such as... Figure 3 As shown, it specifically includes:

[0109] Step S301: Unstructured data is written to MongoDB and Elasticsearch;

[0110] Step S302: Automatically generate code;

[0111] Step S303, execute the code;

[0112] Step S304: Synchronize the structured data to Hive and Doris.

[0113] Among them, such as Figure 4 As shown, step S302 specifically includes:

[0114] Step S3021: Create the Hive and Doris tables;

[0115] Step S3022: Automatically obtain the table schema and extract the parsed fields from it;

[0116] Step S3023: Automatically obtain Python code.

[0117] The specific steps described above are as follows:

[0118] 1. Writing unstructured data to MongoDB and Elasticsearch: This part involves the microservice system writing logs to Elasticsearch in an unstructured form, while the rest is written to MongoDB. All data structures are of JSON type.

[0119] 2. Automatic code generation:

[0120] 1) First, create the fields that need to be structured in each table in Hive or Doris. For example, the event_model in the Elasticsearch database (i.e., the first data table in the above example) is named event_model in Doris (i.e., the second data table in the above example). The table names are the same, which makes it convenient to look up the mapping and does not require writing the mapping relationship between Elasticsearch and Doris.

[0121] 2) The code reads the data that needs to be structured and stores it into a list (f_list, i.e., the first list in the above example), thus ensuring the correctness of the structured data. For example, if the fields of the event_model table in Doris are a and b, the result of f_list is [a, b].

[0122] 3) Next, read a record of data to be synchronized from MongoDB and Elasticsearch to determine the structure of the unstructured data, which is JSON type. Then, iterate through this record. If the key is in f_list, it will be stored in code_list (the second list in the above example), which is of type list. For example, the structure of event_model in Elasticsearch is...

[0123]

[0124] So, if `code_list` only contains `a` and `b`, the code for processing the `channel` field will not be retained. Simultaneously, it's necessary to read fields that must exist in the configuration file and store them in a `c_list` (the third list in the above example) of type `list` (this field is not in the final Hive or Doris). If the key in the JSON is in `c_list`, it needs to be checked whether it can be retrieved; if not, this processing step needs to be skipped. For example, in the `event_model` structure, if the `header` does not exist, the `a` and `b` fields cannot be retrieved. Not skipping this processing step will cause an error when writing to Doris because `a` and `b` were set to be non-empty when the table was created. Even without an error, storing empty values ​​is meaningless from a business perspective.

[0125] 4) Finally, the data in code_list is written to the event_model.py file.

[0126] 3. Execute the event_model.py file generated in step 2.

[0127] 4. Synchronize structured data to Hive and Doris: Synchronize the data processed in step 3 to Hive or Doris. Data for real-time analysis is written to Doris, and other data is written to Hive.

[0128] It should be noted that the above process of automatically obtaining and parsing fields may specifically include:

[0129] 1) First, determine the fields that need to be parsed and whether to write them to Hive or Doris based on the requirements. For example, the event_model table needs to be analyzed in a timely manner, so write this table to Doris. According to the requirements, the a and b fields in the event_model table need to be parsed, so the event_model table in Doris only has these two fields.

[0130] 2) Use Python packages pyhive or pymysql to read the schema from Hive or Doris and obtain the corresponding fields. In this example, since the data is written to Doris, pymysql is used to create a connection and retrieve all fields 'a' and 'b' from the event_model.

[0131] 3) Store the read fields into a list of type f_list, i.e., [a, b]. Obtaining the fields to be parsed from Hive or Doris in this way will not result in errors. If additional fields need to be parsed later, the automatic code generation system can be run directly, eliminating the need for manual code modification, saving time and effort.

[0132] It should also be noted that the above-mentioned automated code generation process may specifically include:

[0133] 1) The data to be processed is read from MongoDB and Elasticsearch, all in JSON format. For example, the unstructured data format of event_model is:

[0134]

[0135] 2) Iterate through this data. If the key is in f_list, it will be saved to code_list, which is of type list. For example, if both a and b are in f_list, the processing code for this field will be saved. If the channel is not in f_list, the processing code for this field will not be saved in code_list.

[0136] 3) Finally, write the code_list to a specified file, such as the event_model.py file.

[0137] The following is a specific example to illustrate this:

[0138] 1. System execution logs are written to Elasticsearch in unstructured form, while the rest are written to MongoDB. For example, the structure of `event_model` in Elasticsearch is:

[0139]

[0140] event_model is the model's execution log, and this data will be written to Elasticsearch.

[0141] 2. Create the table structure for Hive or Doris according to your needs. For example, extract the fields 'a' and 'b' from the 'event_model' table in Elasticsearch and store them in Doris. Then write the table creation statement to complete the creation of the 'event_model' table.

[0142]

[0143]

[0144] 3. Then, use the pymysql package provided by Python to obtain the fields that need to be parsed and store them in a list of type f_list, as shown in the following code:

[0145]

[0146] Adding fields later does not require modifying the code; the automatic code generation system will automatically generate the code. For example, if a channel needs to be added to event_model, there is no need to modify the event_model.py file; simply run the automatic code generation system to generate a new event_model.py file.

[0147] 4. Next, read a record of data that needs to be synchronized from MongoDB and Elasticsearch to understand its unstructured data structure. For example, the structure of event_model in Elasticsearch is as follows:

[0148]

[0149] 5. Iterate through the JSON data obtained in step 4. If the key is in `f_list`, it will be stored in `code_list`, a list of type `code_list`. Simultaneously, fields that must exist in the configuration file need to be read and stored in `c_list`, a list of type `c_list`. If a key in the JSON is in `c_list`, it needs to be checked whether it can be retrieved; if not, this step should be skipped. For example, in Elasticsearch's `event_model`, `code_list` only contains `a` and `b`, so the code for processing the `channel` field will not be retained. Similarly, in the `header` structure of `event_model`, if the header does not exist, the `a` and `b` fields cannot be retrieved, and this step should be skipped. Some core code is as follows:

[0150]

[0151] 6. After step 5 is completed, write code_list to the specified file:

[0152]

[0153] For example, event_model is eventually written to the event_model.py file.

[0154] 7. Execute the file generated in step 2, such as event_model.py.

[0155] 8. Synchronize the data processed in step 7 to Hive or Doris. Write the data for real-time analysis to Doris, and the rest to Hive.

[0156] The key technologies of this preferred embodiment include:

[0157] 1. When there are many tables that need to be synchronized in a structured manner, the key issue is that it is inefficient and time-consuming for developers to manually write the code. The key to solving this problem is to improve efficiency. The idea is to develop an automated code generation system. Users only need to call this system and pass in the table name or index name to automatically generate the corresponding code, avoiding the need to manually write a lot of code and greatly improving work efficiency.

[0158] 2. The previously manual processing of fields requiring parsing has been replaced by automated code retrieval. Before automatically generating code, the schema of the corresponding Hive or Doris table is read to further obtain the fields that need to be parsed, thus avoiding problems caused by manual processing.

[0159] The embodiments of this application can achieve the following technical effects:

[0160] There are over 500 structured data sheets required, which were estimated to take about two weeks to complete. Developing the automatic code generation system took two days, and it needed to be compatible with various scenarios and tests. After the system development was completed, code generation only took less than half a day. Including the system development time, the overall efficiency was improved by three times, greatly reducing development time and improving efficiency and accuracy.

[0161] Obtaining the required structured fields by reading configuration tables often results in inaccurate field values. Using automatic retrieval from Hive or Doris ensures 100% accuracy and avoids rework due to errors. If many new fields or tables are added, the automatic code generation system can be run again to overwrite the original files, eliminating the need for manual file modifications—making it both accurate and fast.

[0162] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0163] This embodiment provides a data synchronization device for unstructured data, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as described previously. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0164] Figure 5 This is a structural block diagram of a data synchronization device for unstructured data according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0165] The first writing unit 51 is used to write unstructured data into MongoDB and Elasticsearch, wherein the MongoDB or Elasticsearch includes a first data table;

[0166] Creation unit 52 is used to create a second data table in Hive or Doris that corresponds to the first data table, wherein the second data table includes fields that need to be structured;

[0167] Parsing unit 53 is used to parse the fields that need to be structured and store them in the first list;

[0168] Processing unit 54 is used to read target data from the first data table from MongoDB or Elasticsearch, store the target fields in the target data into a second list, and store the fields that must exist in the target data into a third list, wherein the target fields are data fields whose key values ​​exist in the first list;

[0169] The second writing unit 55 is used to write the data in the second list to a specified file;

[0170] Synchronization unit 56 is used to execute the specified file and synchronize the data to Hive or Doris.

[0171] In some embodiments, the parsing unit 53 includes:

[0172] The read module is used to read the schema of the second data table from Hive or Doris using the pyhive or pymysql packages provided by Python, so as to obtain the fields that need to be structured;

[0173] A storage module is used to store the fields that need to be structured into the first list.

[0174] In some embodiments, it also includes:

[0175] The judgment unit is used to determine whether the required fields in the target data can be obtained after storing them in the third list;

[0176] An execution unit is configured to execute the step of writing the data in the second list to a specified file if the required field can be obtained;

[0177] The termination unit is used to terminate the data synchronization operation if the required field cannot be obtained.

[0178] In some embodiments, the first writing unit 51 includes:

[0179] The first write module is used to write system execution logs to Elasticsearch in an unstructured form;

[0180] The second write module is used to write unstructured data other than the system execution log into MongoDB.

[0181] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0182] An embodiment provides a computer device. The data synchronization method for unstructured data, as described in this embodiment, can be implemented using a computer device. Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application.

[0183] The computer device may include a processor 61 and a memory 62 storing computer program instructions.

[0184] Specifically, the processor 61 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0185] The memory 62 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 62 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 62 may include removable or non-removable (or fixed) media. Where appropriate, the memory 62 may be internal or external to a data processing device. In a particular embodiment, the memory 62 is non-volatile memory. In a particular embodiment, the memory 62 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0186] The memory 62 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 61.

[0187] The processor 61 implements any of the unstructured data synchronization methods described in the above embodiments by reading and executing computer program instructions stored in the memory 62.

[0188] In some embodiments, the computer device may further include a communication interface 63 and a bus 60. For example, Figure 6 As shown, the processor 61, memory 62, and communication interface 63 are connected via bus 60 and communicate with each other.

[0189] The communication interface 63 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 63 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0190] Bus 60 includes hardware, software, or both, that couples components of a computer device together. Bus 60 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 60 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 60 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0191] Furthermore, in conjunction with the data synchronization methods for unstructured data in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the data synchronization methods for unstructured data in the above embodiments.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for synchronizing unstructured data, characterized in that, include: Unstructured data is written to MongoDB and Elasticsearch, wherein the MongoDB or Elasticsearch includes a first data table; Create a second data table in Hive or Doris that corresponds to the first data table, wherein the second data table includes fields that need to be structured; Parse the fields that need to be structured and store them in the first list; Read the target data from the first data table from MongoDB or Elasticsearch, store the target fields in the target data in the second list, and store the fields that must exist in the target data in the third list, wherein the target fields are data fields whose key values ​​exist in the first list; Write the data from the second list to the specified file; Execute the specified file to synchronize the data to Hive or Doris; Parsing the fields that need to be structured and storing them in the first list includes: using the pyhive or pymysql packages provided by Python to read the schema of the second data table from Hive or Doris to obtain the fields that need to be structured; and storing the fields that need to be structured in the first list. After storing the fields that must exist in the target data into the third list, the method further includes: determining whether the fields that must exist can be obtained; if the fields that must exist can be obtained, then executing the step of writing the data in the second list to the specified file; if the fields that must exist cannot be obtained, then ending the data synchronization operation. Writing unstructured data to MongoDB and Elasticsearch includes: writing system execution logs to Elasticsearch in an unstructured form; and writing unstructured data other than the system execution logs to MongoDB. The specified file is a .py file.

2. A data synchronization device for unstructured data, characterized in that, include: The first writing unit is used to write unstructured data into MongoDB and Elasticsearch, wherein the MongoDB or Elasticsearch includes a first data table; A creation unit is used to create a second data table in Hive or Doris that corresponds to the first data table, wherein the second data table includes fields that need to be structured; A parsing unit is used to parse the fields that need to be structured and store them in a first list; The processing unit is used to read target data from the first data table from MongoDB or Elasticsearch, store the target fields in the target data into a second list, and store the fields that must exist in the target data into a third list, wherein the target fields are data fields whose key values ​​exist in the first list; The second writing unit is used to write the data in the second list to a specified file; The synchronization unit is used to execute the specified file and synchronize the data to Hive or Doris; The parsing unit includes a reading module, used to read the schema of the second data table from Hive or Doris using the pyhive or pymysql packages provided by Python, in order to obtain the fields that need to be structured; A storage module is used to store the fields that need to be structured into the first list; It also includes: a judgment unit, used to determine whether the required fields in the target data can be obtained after storing them in the third list; an execution unit, used to execute the step of writing the data in the second list to the specified file if the required fields can be obtained; and a termination unit, used to terminate the data synchronization operation if the required fields cannot be obtained. The first writing unit includes: a first writing module, used to write system execution logs into Elasticsearch in an unstructured form; and a second writing module, used to write unstructured data other than the system execution logs into MongoDB. The specified file is a .py file.

3. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data synchronization method for unstructured data as described in claim 1.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the data synchronization method for unstructured data as described in claim 1.

Images