Data processing method, device and equipment, readable storage medium and program product
By obtaining the field information and content information of unstructured data, the structured data table is updated, and the complexity of converting unstructured data into structured data is solved, the data processing efficiency is improved and data synchronization is achieved.
Patent Information
- Application Number
- CN202410980667.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, the process of converting unstructured data into structured data is complex and inefficient, making it difficult to effectively synchronize unstructured data with relational databases.
By obtaining the field information and content information of unstructured data, structure updates and data updates are performed, matching structured data tables are generated, data processing flow is simplified and efficiency is improved.
It effectively reduces the complexity of data processing, improves data processing efficiency, realizes synchronization between unstructured data and structured data tables, and ensures data consistency.
Smart Images

Figure CN118939655A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, readable storage medium and program product. Background Art
[0002] Some databases (such as MongoDB) can store unstructured data and have highly flexible data models that can nest and contain different data types, effectively meeting application requirements. At the same time, these databases allow the data model to be adjusted and extended without the need for a complex migration process, so they are often used to store business data.
[0003] Since there is no fixed data model, unstructured data is not conducive to data analysis. Therefore, when performing data analysis, it is usually necessary to synchronize unstructured data to a relational database. Currently, the processing flow of the method for converting unstructured data into structured data is relatively complicated and the processing efficiency is low. Summary of the invention
[0004] The embodiments of the present application provide a data processing method, apparatus, device, readable storage medium and program product, which can update unstructured data into a structured data table, effectively reducing the complexity of data processing and improving the efficiency of data processing.
[0005] On the one hand, an embodiment of the present application provides a data processing method, the data processing method comprising:
[0006] Acquire target unstructured data, where the target unstructured data includes at least one first field and field content of each of the first fields;
[0007] Performing data analysis and processing on the target unstructured data to obtain field information and content information of the target unstructured data; wherein the field information includes the at least one first field and field attribute information of each of the first fields, and the content information includes data indicated by the field content of each of the first fields and the number of field contents of the at least one first field;
[0008] Performing structure update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure;
[0009] The structured data table with the structure updated is subjected to data updating processing according to the content information to obtain an updated structured data table, wherein the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information.
[0010] In one embodiment, performing structure update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain the structured data table with updated structure includes:
[0011] Acquire table structure information of the structured data table, where the table structure information includes each second field in the structured data table and field attribute information of each second field;
[0012] If the field information matches the table structure information, determining the structured data table as a structured data table with updated structure;
[0013] If the field information does not match the table structure information, the structured data table is locked to obtain a structured data table whose access status is locked; wherein the structured data table whose access status is locked cannot be updated;
[0014] updating the table structure of the structured data table in the locked state according to the field information to obtain a structured data table in the locked state with an updated structure;
[0015] The structured data table for structural update in the locked state is unlocked to obtain the structured data table for structural update.
[0016] In one embodiment, the method further comprises:
[0017] Determining an access status of the structured data table;
[0018] If the access state of the structured data table is locked, suspending data update processing on the structured data table;
[0019] After a preset time period has passed, determining the access status of the structured data table again;
[0020] If the access state of the structured data table is unlocked, the structured data table with the updated structure is obtained, and the step of performing data update processing on the structured data table with the updated structure according to the content information to obtain an updated structured data table is triggered.
[0021] In one embodiment, performing data updating processing on the structured data table whose structure is updated according to the content information to obtain an updated structured data table includes:
[0022] Obtain the number of columns contained in the structured data table for structural update;
[0023] If the number of field contents in the content information matches the number of columns, then, according to the content information, writing the data indicated by the field contents of each of the first fields into the structured data table with the structure updated, to obtain an updated structured data table;
[0024] If the number of field contents in the content information does not match the number of columns, wait for a target time period, and execute the step of obtaining the number of columns contained in the structured data table of the structure update and subsequent steps.
[0025] In one embodiment, the method further comprises:
[0026] Acquire supplementary information of the target unstructured data, where the supplementary information includes one or more of a table name, a library name, a field type, and a field identifier;
[0027] Performing attribute supplementation processing on the target unstructured data using the supplementary information to obtain the target unstructured data after attribute supplementation;
[0028] The performing data analysis on the target unstructured data to obtain field information and content information of the target unstructured data includes:
[0029] Data analysis and processing are performed on the target unstructured data after the attribute supplementation to obtain field information and content information of the target unstructured data.
[0030] In one embodiment, the method further comprises:
[0031] Receiving a data processing request for a target set in an unstructured database, the unstructured database includes one or more sets, each set includes at least one piece of unstructured data, and the target set is any set in the unstructured database;
[0032] Traversing at least one piece of unstructured data in the target set, and updating the structured data table obtained by the last traversal according to the currently traversed unstructured data;
[0033] After the traversal is completed, it is determined that the structured data table associated with the target set is the structured data table obtained after the traversal is completed.
[0034] In one embodiment, obtaining target unstructured data includes:
[0035] In response to the data processing request, obtaining the target set including at least one piece of unstructured data;
[0036] If it is detected that the target set is updated, obtaining the update increment information of the target set;
[0037] The target unstructured data is obtained according to at least one piece of unstructured data included in the target set and the update increment information.
[0038] In one embodiment, the performing data analysis on the target unstructured data to obtain field information and content information of the target unstructured data includes:
[0039] If the type of the field content of the target first field is a non-nested data type, determining the field content of the target first field as the data indicated by the field content of the target first field, the target first field being any first field in the target unstructured data;
[0040] If the type of the field content of the target first field is a nested data type, outer nested parsing processing is performed on the field content of the target first field to obtain parsed data of the field content of the target first field, and the parsed data of the field content of the target first field is determined as the data indicated by the field content of the target first field;
[0041] Determining content information of the target unstructured data according to data indicated by field contents of each of the first fields in the target unstructured data;
[0042] According to at least one first field included in the target unstructured data, field information of the target unstructured data is determined.
[0043] In one embodiment, the method further comprises:
[0044] Receiving a data query request for a structured data table corresponding to the target unstructured data, wherein the data query request carries a data query condition;
[0045] Perform query processing on the structured data table according to the data query condition to obtain query result data;
[0046] If the type of the query result data is a non-nested data type, outputting the query result data;
[0047] If the type of the query result data is a nested data type, the query result data is parsed using a nested parsing function to obtain parsed result data, and the parsed result data is output.
[0048] In one embodiment, obtaining target unstructured data includes:
[0049] Receive a data conversion request sent by a client, the data conversion request including authentication information of the client, an unstructured data file, and a data conversion requirement;
[0050] Perform device authentication processing according to the authentication information to obtain an authentication result;
[0051] If the authentication result indicates that the authentication is successful, the unstructured data file is processed according to the data conversion requirement to obtain the target unstructured data.
[0052] Accordingly, an embodiment of the present application provides a data processing device, the data processing device comprising:
[0053] An acquisition unit, configured to acquire target unstructured data, wherein the target unstructured data includes at least one first field and field contents of each of the first fields;
[0054] a processing unit, configured to perform data analysis on the target unstructured data to obtain field information and content information of the target unstructured data; wherein the field information includes the at least one first field and field attribute information of each of the first fields, and the content information includes data indicated by field contents of each of the first fields and the number of field contents of the at least one first field;
[0055] The processing unit is further used to perform structure update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure;
[0056] The processing unit is also used to perform data update processing on the structured data table with the structure updated according to the content information to obtain an updated structured data table, wherein the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information.
[0057] Accordingly, an embodiment of the present application provides a computer device, the computer device comprising:
[0058] a processor suitable for implementing a computer program;
[0059] A computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned data processing method.
[0060] Accordingly, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned data processing method.
[0061] Accordingly, an embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned data processing method.
[0062] In the present application, unstructured data can be acquired, and data analysis can be performed on the unstructured data to obtain field information and content information of the unstructured data; the corresponding structured data table can be structurally updated using the field information to obtain a structured data table with updated structure; the structured data table with updated structure can be updated using the content information to obtain an updated structured data table, which can effectively shorten the processing flow of converting unstructured data into structured data, thereby effectively improving data processing efficiency. At the same time, the method provided in the present application can realize table structure update and table data update of structured data tables, so that the structured data tables match the unstructured data, which is conducive to the synchronization of data in unstructured databases with structured data tables. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0064] Figure 1 is a schematic diagram of the system architecture of a data processing system provided in an embodiment of the present application;
[0065] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application;
[0066] Figure 3 is a schematic diagram of unstructured data provided by an embodiment of the present application;
[0067] Figure 4 is a schematic diagram of a structured data table structure update method provided in an embodiment of the present application;
[0068] Figure 5 is a schematic diagram of a data updating method provided in an embodiment of the present application;
[0069] Figure 6 is a schematic diagram of a data processing method provided in an embodiment of the present application;
[0070] Figure 7is a schematic diagram of another data processing method provided in an embodiment of the present application;
[0071] Figure 8 It is a flowchart of another data processing method provided in an embodiment of the present application;
[0072] Fig. 9 is a schematic diagram of another data processing method provided in an embodiment of the present application;
[0073] Fig.10 is a schematic diagram of a structured data table provided in an embodiment of the present application;
[0074] Fig.11 It is a flowchart of another data processing method provided in an embodiment of the present application;
[0075] Fig.12 is a schematic diagram of a data conversion method provided in an embodiment of the present application;
[0076] Fig.13 is a structural block diagram of a data processing device provided in an embodiment of the present application;
[0077] Fig.14 It is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0078] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0079] It should be noted that the descriptions of "first", "second", etc. involved in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" or "second" may explicitly or implicitly include at least one of the features.
[0080] In the specific implementation of this application, user-related data, such as unstructured data corresponding to the user, is involved. When the embodiment of this application is applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with local laws, regulations and standards.
[0081] To facilitate understanding, some nouns are briefly explained as follows:
[0082] MongoDB: A document-oriented database based on distributed file storage, which uses the BSON format to store data. The data model of this database is highly flexible, similar to the JSON data structure, allowing nesting and inclusion of different data types, and can adapt to different application requirements. MongoDB does not have a fixed data model, and different documents stored in it may have different fields and structures.
[0083] Unstructured data: refers to data with irregular or incomplete structure, no predefined data model, and data that is not convenient to represent using a two-dimensional logical table in a database. Files stored in the MongoDB database contain unstructured data.
[0084] The data processing method in the embodiment of the present application can be applied to a data processing device, which is arranged on a server, and the server is provided with one or more processors, a memory, and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the data processing method; wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.
[0085] The architecture of the data processing system provided in the embodiments of the present application will be introduced below in conjunction with the accompanying drawings.
[0086] See also Figure 1 , this figure is a system architecture diagram of a data processing system provided in an embodiment of the present application, the data processing system includes an unstructured database 101 (for example: MongoDB), a data processing device 102, and a client 103, wherein the unstructured database 101 is used to store business data sent by a business server, and the business data is unstructured data; the data processing device 102 can interact with the client 103 for data, and can also obtain data stored in the unstructured database 101, and the data processing device 102 can execute the data processing method provided in the present application; the client 103 can interact with objects and has display and communication functions.
[0087] The following will explain in detail Figure 1 The working principle of the data processing system shown:
[0088] The client 103 can interact with the object and send a data processing request to the data processing device 102. The data processing request is used to request that the unstructured data stored in the unstructured database be stored in the structured data table. After receiving the data processing request, the data processing device 102 can obtain the unstructured data stored in the unstructured database 101, and process the unstructured data to obtain the structured data table corresponding to the unstructured database 101.
[0089] At a certain moment, the business server stores new business data in the unstructured database 101, and data updates appear in the unstructured database 101. At this time, the data processing device 102 can obtain the target unstructured data, which includes at least one first field and the field content of each first field. The target unstructured data can be associated with the data update in the unstructured database 101. The data processing device 102 can perform data analysis and processing on the target unstructured data to obtain the field information and content information of the target unstructured data; wherein the field information includes at least one first field and field attribute information of each first field (for example: the data type of the field, whether the field is a primary key, etc.), and the content information includes the data indicated by the field content of each first field, and the number of field contents of at least one first field. The data processing device 102 can perform structural update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure; it can perform data update processing on the structured data table with updated structure according to the content information to obtain an updated structured data table, the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information, so that the unstructured data stored in the unstructured database 101 matches the structured data table, and the synchronization of upstream and downstream data is achieved. Through the data processing method provided in the embodiment of the present application, the unstructured data can be updated to the structured data table, which effectively shortens the data processing flow of converting unstructured data into structured data and improves the efficiency of data processing. At the same time, the method provided in the present application can also realize the synchronization between the data in the unstructured database and the data in the structured data table, which is conducive to the synchronization of upstream and downstream data in actual application scenarios.
[0090] It is understandable that the schematic diagram of the data processing system described in the embodiment of the present application is to more clearly illustrate the data processing method of the embodiment of the present application, and does not constitute a limitation on the data processing method provided in the embodiment of the present application. For example, the data processing method provided in the embodiment of the present application can be executed by the data processing device 102, and can also be executed by other devices that are different from the data processing device 102 and can communicate with the unstructured database 101 and the client 103. It is known to those skilled in the art that Figure 1 The number of the unstructured database 101, the data processing device 102 and the client 103 is only illustrative. Any number of devices can be configured according to business implementation needs. Moreover, with the evolution of the system architecture and the emergence of new business scenarios, the data processing method provided in the embodiment of the present application is also applicable to similar technical problems.
[0091] In actual application scenarios, the upstream database can receive and store data, and the downstream data processing device can obtain the data in the database and generate a data table to facilitate subsequent data analysis based on the data table. However, if the data stored in the upstream database changes (the data content changes, or the data structure changes), the downstream data table should also change accordingly, that is, to achieve synchronization of upstream and downstream data, thereby ensuring data consistency. For this data synchronization problem, the commonly used processing logic is: S1. After the data stored in the upstream database changes, the downstream data processing device perceives the change; S2. The downstream data processing device blocks the write process for the data table; S3. The downstream data processing device modifies the data table according to the change; S4. The downstream data processing device restarts the write process for the data table.
[0092] Based on the above processing logic, this application provides a data processing method. Figure 2 , Figure 2 A flow chart of a data processing method provided in an embodiment of the present application. The data processing method can be implemented by the above-mentioned data processing device 102, or by other devices that can implement the data processing method. The following is an example of the data processing method being implemented by the above-mentioned data processing device 102. The flow of the data processing method provided in the embodiment of the present application includes but is not limited to:
[0093] S201. Acquire target unstructured data, where the target unstructured data includes at least one first field and field content of each first field.
[0094] In an embodiment of the present application, a data processing device can obtain target unstructured data, which may include at least one first field and the field content of each first field. For example: the first field is "name", and the field content of the first field is "zhang". The target unstructured data can be data already stored in the unstructured database (that is, part of the full data in the unstructured database), or it can be updated data in the unstructured database (that is, part or all of the incremental data in the unstructured database). The step in which the data processing device obtains the target unstructured data can correspond to S1 in the above processing logic to a certain extent, and this step is conducive to the subsequent update processing of the structured data table based on the target unstructured data. Please refer to Figure 3 , which is a schematic diagram of unstructured data provided in an embodiment of the present application. Figure 3 In the unstructured data shown, the first field is "_id", and the field content of the first field is "ObjectId(epiweufhnqpeuf135)"; the fields in the unstructured data also include "PWDDf", "UIMgN", "pkdVk", etc. Figure 3 It can be seen that unstructured data does not have a fixed data model and is usually stored in the form of "key-value pairs", with high data flexibility.
[0095] It should be noted that the unstructured data may be unstructured data stored in a MongoDB database, may be unstructured data of character-separated values (CSV) type, JavaScript Object Notation (JSON) type, or other types of unstructured data.
[0096] S202. Perform data analysis and processing on the target unstructured data to obtain field information and content information of the target unstructured data; wherein the field information includes the at least one first field and field attribute information of each of the first fields, and the content information includes the data indicated by the field content of each of the first fields, and the number of field contents of the at least one first field.
[0097] In an embodiment of the present application, a data processing device can perform data analysis and processing on the target unstructured data to obtain field information and content information of the target unstructured data. Among them, the field information can include each first field in the target unstructured data and the field attribute information of each first field, and the field attribute information of the first field can include a field type (for example: the field type is a string type), a field identifier (for example: the first field is a primary key), etc. The content information includes the data indicated by the field content of each first field in the target unstructured data, and the number of field contents of each first field in the target unstructured data. Through the method provided in the embodiment of the present application, the unstructured data can be divided into field information and content information, and the field information is used to subsequently indicate the adjustment of the table structure of the structured data table, which is equivalent to a structure flow (or schema flow), and the content information is used to subsequently indicate the adjustment of the table content of the structured data table, which is equivalent to a data flow, which is conducive to the subsequent data processing device to use such two flows to process the structured data table, thereby shortening the data processing flow and improving the efficiency of data processing.
[0098] In one embodiment, the data processing method provided by the embodiment of the present application may also include the following steps: obtaining supplementary information of the target unstructured data, the supplementary information including one or more of the table name, library name, field type, and field identifier; using the supplementary information to perform attribute supplementation processing on the target unstructured data to obtain the target unstructured data after attribute supplementation; performing data analysis processing on the target unstructured data to obtain the field information and content information of the target unstructured data, including: performing data analysis processing on the target unstructured data after attribute supplementation to obtain the field information and content information of the target unstructured data. Figure 3 It can be seen that the data in the target unstructured data is usually stored in the form of "key-value pairs" and does not contain information such as field type and identification. Therefore, after the data processing device obtains the target unstructured data, in order to ensure that the target unstructured data can be correctly added to the structured data table later, the target unstructured data can be supplemented with attributes.
[0099] Specifically, the data processing device can obtain the supplementary information of the target unstructured data according to the structured data table corresponding to the target unstructured data, the data processing request for the target unstructured data, etc. The supplementary information may include one or more of the table name, library name, field type, and field identifier. The data processing device can use the supplementary information to supplement the target unstructured data, and subsequently perform data analysis and processing on the target unstructured data after attribute supplementation to obtain the field information and content information of the target unstructured data.
[0100] In some cases, when the target unstructured data is data stored in a MongoDB database, the target unstructured data can be obtained based on the Change Streams function in MongoDB. Specifically, the format of the real-time stream data obtained by Change Streams is as follows:<BSON Object>},"operationType":" <operation>","fullDocument":{ <document>},"ns":{"db":" <database>","coll":" <collection>"},"to":{"db":" <database>","coll":" <collection>"},"documentKey":{"_id": <value>},"updateDescr iption":{"updatedFields":{ <document>},"removedFields":[" <field>",...]}"clusterTime": <timestamp>,"txnNumber": <numberlong>,"lsid":{"id": <uuid>,"uid":<Bin Data>}}. It can be seen that if the real-time streaming data does not contain the definition of the data type of the field, then the target unstructured data does not contain any data type definition. Since MongoDB is a document-oriented database, one of its core features is the dynamic mode. Documents in MongoDB can contain different fields, and the data types of the fields can be flexible. Therefore, the data provided by Change Streams lacks data type definitions to maintain this flexibility and scalability. Based on this feature, the method provided in the present application usually sets the field type of all first fields to string type when supplementing attributes for unstructured data stored in a MongoDB database, thereby solving the problem of being unable to obtain the field type. Correspondingly, in the structured data table corresponding to the unstructured data, the field type of each table field is also string type.
[0101] In addition, in actual applications, since MongoDB's change event records are not updated before the message, Flink's upsert stream is usually used to change the log stream, and the upsert stream requires a unique key, so it is usually necessary to define the "_id" field in the unstructured data as the primary key, and other columns cannot be declared as the primary key, and the value of the "_id" field cannot be deleted. Correspondingly, in the structured data table of the unstructured data object, the "_id" field is also the primary key.
[0102] In one embodiment, a specific implementation method of performing data analysis processing on the target unstructured data to obtain field information and content information of the target unstructured data may be as follows: if the type of the field content of the target first field is a non-nested data type, the field content of the target first field is determined to be the data indicated by the field content of the target first field, and the target first field is any first field in the target unstructured data; if the type of the field content of the target first field is a nested data type, the field content of the target first field is subjected to outer nested parsing processing to obtain parsed data of the field content of the target first field, and the parsed data of the field content of the target first field is determined to be the data indicated by the field content of the target first field; the content information of the target unstructured data is determined according to the data indicated by the field content of each first field in the target unstructured data; the field information of the target unstructured data is determined according to at least one first field included in the target unstructured data.
[0103] The type of the field content of any first field in the target unstructured data can be a non-nested data type or a nested data type. If the type of the field content of the first field is a non-nested data type, the data indicated by the field content of the first field is the field content itself. For example, if the field content of the first field "name" is "li", the data indicated by the field content of the first field is "li".
[0104] If the type of the field content of the first field is a nested data type, the field content can be subjected to an outer nested parsing process to obtain parsed data after the outermost nested parsing, and the parsed data is the data indicated by the field content of the first field, for example: the field content of the first field "name" is address A, and the data indicated by address A is "wang", then the data indicated by the field content of the first field is "wang". It should be noted that in some cases, the field content of the first field is multi-layer nested data, and the method provided in the present application can perform an outer nested parsing process on the multi-layer nested data to obtain nested parsed data after the outermost nested parsing (that is, the data indicated by the field content of the first field), and the type of the nested parsed data is still a nested data type. When the table data of the structured data table is subsequently updated according to the content information containing the nested parsed data, the nested parsed data can be directly filled into the structured data table in the form of JSON. The method provided by the embodiment of the present application can better adapt to the flexibility of unstructured data, can be compatible with different types of field content, is conducive to the conversion of unstructured data to structured data tables, and improves the efficiency of data processing.
[0105] S203: Perform structure update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure.
[0106] In an embodiment of the present application, the data processing device can perform a structural update process on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with an updated structure. For example, if the structured data table corresponding to the target unstructured data contains two fields, namely "name" and "age", and the field information contains three first fields, namely "name", "age", and "time", then the structured data table is structurally updated according to the field information to obtain a structured data table with an updated structure, and the structured data table with an updated structure contains three fields, namely "name", "age", and "time".
[0107] See also Figure 4 , which is a schematic diagram of a structured data table structure update method provided by an embodiment of the present application. Figure 4 It can be seen that a structured data table A can be generated based on the unstructured data 1 and the unstructured data 2, and the structured data table A contains two fields, namely "name" and "age". The data processing device obtains the target unstructured data, and the target unstructured data contains three first fields, namely "name", "age", and "address". The structured data table A can be structurally updated according to the field information of the target unstructured data to obtain a structured data table A with an updated structure. At this time, the data table contains three fields, namely "name", "age", and "address". Through the method provided in the embodiment of the present application, the table structure of the structured data table can be updated based on the unstructured data, thereby achieving synchronization between the unstructured data and the structured data table and ensuring data consistency.
[0108] It should be noted that in some cases, the structured data table can be a hive table or a paimon table. Hive is a data warehouse tool based on Hadoop that can map structured data files to a database table and provide SQL-like query functions. Hive has a variety of ways to operate and manage data, and can provide efficient, flexible and scalable data processing solutions. Paimon is a streaming data lake platform derived from hive, which has the capabilities of high-speed data ingestion, change log monitoring and real-time analysis.
[0109] In one embodiment, a structured data table corresponding to the target unstructured data is structurally updated according to the field information to obtain a structured data table with an updated structure. The specific implementation method may be: obtaining table structure information of the structured data table, the table structure information including each second field in the structured data table and field attribute information of each second field; if the field information matches the table structure information, the structured data table is determined to be a structured data table with an updated structure; if the field information does not match the table structure information, the structured data table is locked to obtain a structured data table with an access status in a locked state; wherein, a structured data table with an access status in a locked state cannot be updated; updating the table structure of the structured data table in a locked state according to the field information to obtain a structured data table with an updated structure in a locked state; unlocking the structured data table with an updated structure in a locked state to obtain a structured data table with an updated structure.
[0110] Specifically, the data processing device can obtain the table structure information corresponding to the target structured data, and the table structure information can include each second field in the structured data table and the field attribute information of each second field. The data processing device can match the field information of the target unstructured data with the table structure information, that is, for any first field in the field information, determine whether there is a matching second field in the table structure information whose field and field attribute information match. If so, it is determined that the field information matches the table structure information, and there is no need to update the table structure of the structured data table, and it can be directly used as a structured data table with updated structure. For example: the field information includes three first fields, namely "name", "age", and "address", and the field types of the three first fields are all string, and the table structure information includes three second fields, namely "name", "age", and "address", and the field types of the three second fields are all string; it can be determined that the field information matches the table structure information.
[0111] If there is no second field matching a first field and its field attribute information in the table structure information, it means that the table structure of the structured data table needs to be updated. For example: the field information includes three first fields, namely "name", "age", and "address", and the field types of the three first fields are all string, and the table structure information includes two second fields, namely "name" and "age", and the field types of the two second fields are both string; it can be determined that the field information does not match the table structure information. It should be noted that in actual application, the data processing device can maintain a data map (LinkedHashMap) in memory. The key of the Map represents the field name, and the value represents the field attribute information (such as field type, etc.). After the data processing device determines the field information corresponding to the unstructured data, it will use the "map.get" operation to check whether each first field in the field information exists in the Map and whether its field attribute information matches the field attribute information of the current field. If there is no match, the first field and its field attribute information are added to the Map, and the changed record data is sent to the output stream so that the output stream can be further processed through the updateSchema operation, thereby realizing the table structure update of the structured data table.
[0112] When using field information to update the structure of a structured data table, the data processing device can first lock the structured data table to obtain structured data with an access status in a locked state. The structured data table with an access status in a locked state cannot be updated. This ensures that when the structure of the structured data table is subsequently updated, no other process will modify the structured data table, thereby effectively avoiding possible exceptions and ensuring that the table structure of the structured data table can be successfully updated.
[0113] The data processing device can update the table structure of the structured data table in a locked state according to the field information to obtain the structured data table with the structure updated in the locked state. After the structure update is completed, the structured data table with the structure updated in the locked state can be unlocked to ensure that other subsequent processes can update the structured data table with the structure updated. In some cases, the data processing device uploads the structured data table (hive table) (i.e., locks the table structure) based on the matching structure of the field information and the table structure data, and performs atomic table structure (schema) changes on the locked structured data table, thereby ensuring that the schema update operation of the hive table is successful. Through the method provided in the embodiment of the present application, it is possible to determine whether to update the table structure according to the field information and the table structure information, and it is also possible to use the locking and unlocking operations to ensure the atomicity of the table structure adjustment of the structured data table, thereby ensuring the data security of the structured data table.
[0114] In one embodiment, the data processing method provided by the present application may also include the following steps: determining the access status of the structured data table; if the access status of the structured data table is a locked state, suspending the data update processing of the structured data table; after a preset period of time, determining the access status of the structured data table again; if the access status of the structured data table is an unlocked state, obtaining the structured data table with a structure update, and triggering the execution of data update processing of the structured data table with a structure update according to the content information to obtain the updated structured data table.
[0115] Specifically, before the data processing device processes the structured data table using the content information, it can first determine the access status of the structured data table. If the access status of the structured data table is locked, it means that the structured data table is currently being updated. The data update processing of the structured data table can be suspended, and a preset duration (for example, 3 seconds) is waited. After the preset duration, the access status of the structured data table can be determined again. If the access status of the structured data table is unlocked, the current structured data table (the structured data table may be a structured data table with a structured update) can be obtained, and the structured data table can be updated according to the content information. The method provided by the embodiment of the present application can determine whether to perform data update processing according to the access status of the structured data table, and can ensure that the structure update and data update of the structured data table can be performed normally, which is conducive to realizing the synchronization of unstructured data to the structured data table.
[0116] It should be noted that the data processing method provided by the present application can be applied not only to the case of increasing the number of columns of a structured data table, but also to the case of reducing the number of columns of a structured data table. In addition, the method for determining the structured data table corresponding to the target unstructured data can be based on the data processing requirements and the relevant information of the target unstructured data. For example: if the data processing requirement is "synchronize the unstructured data stored in collection 1 in the MongoDB database to the structured data table A in the database DB1", the data processing device can obtain the unstructured data stored in collection 1 in the MongoDB database, and update the structured data table A according to the obtained unstructured data. Subsequently, if the data processing device obtains the unstructured data for collection 1, the steps shown in S201-S204 can also be executed.
[0117] S204, performing data update processing on the structured data table with the updated structure according to the content information to obtain an updated structured data table, wherein the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information.
[0118] In an embodiment of the present application, a data processing device can perform data update processing on a structured data table with a structure update according to content information to obtain an updated structured data table, wherein the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information. Through the method provided in an embodiment of the present application, the conversion of unstructured data to structured data can be achieved, which effectively shortens the data processing process, improves the efficiency of data processing, and ensures the synchronization between unstructured data and structured data. When the unstructured data is upstream data and the structured data table is downstream data, the method provided in the present application can achieve consistency between upstream and downstream data.
[0119] In one embodiment, a specific implementation method of performing data update processing on a structured data table with a structure updated according to content information to obtain an updated structured data table may be: obtaining the number of columns contained in the structured data table with a structure updated; if the number of field contents in the content information matches the number of columns, then writing the data indicated by the field content of each first field into the structured data table with a structure updated according to the content information to obtain an updated structured data table; if the number of field contents in the content information does not match the number of columns, then waiting for a target duration, and executing the step of obtaining the number of columns contained in the structured data table with a structure updated and subsequent steps.
[0120] Specifically, the data processing device can obtain the number of columns contained in the structured data table with updated structure. If the number of field contents in the content information is the same as the number of columns, it means that the number of field contents in the content information matches the number of columns. Since the field information matches the table structure information of the structured data table with updated structure (for example, the first field in the field information corresponds one-to-one to the second field in the table structure information), the data indicated by the field content of each first field can be directly written into the structured data table with updated structure according to the content data to obtain an updated structured data table.
[0121] If the number of field contents in the content information is the same as the number of columns, it means that the number of field contents in the content information does not match the number of columns, which means that the table structure of the structured data table needs to be adjusted before the content information can be written. Therefore, you can wait for the target time (for example: 10 seconds) and then re-execute the steps of obtaining the number of columns contained in the structured data table with updated structure and subsequent steps. The method provided in the embodiment of the present application can realize data update of the structured data table, and at the same time, it does not require complex judgment or processing logic, which reduces the complexity of the data processing method and is conducive to improving the efficiency of data processing.
[0122] See also Figure 5 , which is a schematic diagram of a data updating method provided by an embodiment of the present application. After the data processing device determines the content information of the target unstructured data, it can determine whether the data conversion is successful, that is, whether the number of field contents in the content information matches the number of columns in the structured data table. If so (that is, the number of field contents in the content information matches the number of columns in the structured data table), the data indicated by the field content of each first field in the content information is written into the structured data table; if not (that is, the number of field contents in the content information does not match the number of columns in the structured data table), wait for the structure of the structured data table to be updated, and after waiting for the target time, the data conversion can be retried. Through the method provided by the embodiment of the present application, the data update of the structured data table can be effectively realized, and at the same time, the anomalies caused by modifying the data and structure of the structured data table at the same time are avoided, thereby ensuring the stability of the system and the efficiency of data processing.
[0123] It should be noted that in the data processing method provided by the present application, the execution order of step S203 and step S204 is not fixed, and the embodiment of the present application only provides a feasible implementation method. In the actual application process, step S203 and step S204 can be started at the same time, or step S204 can be started before step S203. For example: for the case where step S203 and step S204 can be started at the same time: the data processing device uses the field information to update the structure of the structured data table, and uses the content information to update the data of the structured data table. If it is determined according to the field information that the structured data table needs to be updated, and at this time, the content information has obtained the structured data table, it can be determined that the number of field contents in the content information does not match the number of columns of the structured data table, and the data processing device will wait for the target time length before trying to update the data of the structured data table, and the data processing device can lock the structured data table according to the field information, and update the structure of the structured data table. The same is true for the case where step S204 starts to execute before step S203. It can be seen that, based on the data processing method provided in the embodiment of the present application, the execution order between step S203 and step S204 is variable.
[0124] See also Figure 6 , which is a schematic diagram of a data processing method provided in an embodiment of the present application. Figure 6 In the example, the structured data table is the paimon table. The data processing device can use the data acquisition (CdcSource) function to obtain unstructured data, and the unstructured data is a string. After that, the record parsing (RecordParser) function can be used to perform attribute supplementation processing on the unstructured data. Specifically, the attribute supplementation (RichCdcMultiplexRecc) function can be used to add supplementary information to the unstructured data. The supplementary information may include one or more of the table name (tableName), database name (databaseName), field type (fieldTypes), field identifier (primaryKeys), and data record (cdcRecord). The data analysis function (CdcParsingProcessFunction) can be used to analyze the unstructured data after attribute supplementation to obtain the field information and content information corresponding to the unstructured data. The data analysis function can also match the field information with the table structure information of the structured data table corresponding to the unstructured data. If the two do not match, the field information and content information are sent. If the two match, only the content information can be sent to the subsequent node or module to perform the subsequent process. The data processing device can perform diversion processing on the field information and content information: for the field information, the database field update process (UpdatedDatabaseFieldsProcess) can be used to modify the table structure of the unstructured data table, that is, the table structure stored in the hive hms (hive hms is used to store the table structure and table partition components of each structured data table) is modified. At this time, the access status of the structured data table is locked. After the table structure modification is completed, the modification can be submitted to HDFS (i.e., distributed file system, Hadoop Distributed File System), thereby realizing the modification of the table structure (schema) of the structured data table. The specific implementation method for updating the table structure of the structured data table is shown in the above step S203.
[0125] For content information, the data processing device can use the record storage and writing operation module (CdcRecordStoreWriteOperator), which includes a storage and writing interface (StoreSinkWriteImpl), a table writing interface (TableWriteImpl), an abstract file reading and writing module (AbstractFileStoreWrite) and a merge tree filling module (MergeTreeWriter) to implement the above Figure 5 The processing flow shown is used to modify the table data of the structured data table. After the data modification is completed, the operation submission module (committerOperator) can submit the modification to HDFS, thereby realizing the modification of the table data (bucket) of the structured data table. The specific implementation method for updating the table data of the structured data table is shown in the above step S204. Through the method provided in the embodiment of the present application, the structure modification and data modification of the structured data table can be performed respectively by the diversion processing method, thereby realizing the conversion of unstructured data to structured data, effectively shortening the data processing flow and improving the data processing efficiency.
[0126] In one embodiment, the data processing method provided by the embodiment of the present application may also include the following steps: receiving a data query request for a structured data table corresponding to the target unstructured data, the data query request carrying a data query condition; querying the structured data table according to the data query condition to obtain query result data; if the type of the query result data is a non-nested data type, outputting the query result data; if the type of the query result data is a nested data type, parsing the query result data using a nested parsing function to obtain parsed result data, and outputting the parsed result data.
[0127] Specifically, the data processing method provided by the present application can synchronize unstructured data into a structured data table, and the structured data table has a data query function. The data processing device can receive a data query request for a structured data table corresponding to the target unstructured data, and the data query request carries a query condition, for example: the query condition can be "all data in the query table", "data in the query table that meets condition X", etc. The data processing device can query and process the structured data table according to the data query condition to obtain query result data. If the type of the query result data is a non-nested data type, the query result data can be directly output; if the type of the query result data is a nested data type, the nested parsing function can be used to parse the query result data to obtain parsed result data. For example: for multi-layer nested data in unstructured data, the method provided by the present application can parse the outermost nesting of the multi-layer nested data, and store the nested parsed data in the form of JSON in a structured data table, and the structured data table can be a hive table. When performing data query, if the nested parsed data (whose type is a nested data type) is obtained, the parsing function provided by hive can be used to parse the nested parsed data to obtain the final parsing result (whose type is a non-nested data type). Through the method provided in the embodiment of the present application, the query operation of the structured data table can be implemented, and the nested data can also be compatible and parsed, which ensures the readability of the query results, improves the user experience, and is conducive to data analysis based on the data in the structured data table to obtain accurate data analysis results.
[0128] See also Figure 7 , which is a schematic diagram of another data processing method provided by an embodiment of the present application. Figure 7 As shown, the MongoDB database contains 4 collections, each of which stores multiple files. Each file contains the following Figure 3 The unstructured data shown. The client can send a data processing request to the data processing device, which can indicate the synchronization of the unstructured data stored in MongoDB to the structured data table, and indicate the data demand information in the synchronization process. The data processing device can use the data processing method provided by the present application to create an initial structured data table according to the unstructured data, and update the initial structured data table to obtain a structured data table, which can contain data of non-nested data types and data of nested data types. When the client needs to obtain a certain field or data that meets certain requirements, a data query request for the structured data table can be generated, and the data query request can be sent to the data processing device. The data processing device can query the structured data table according to the data query request and output the data query result. Through the method provided in the embodiment of the present application, the synchronization of unstructured data to the structured data table can be achieved, and the query of the data in the structured data table can also be achieved, so that the application requirements can be better met and the user experience can be improved.
[0129] In an embodiment of the present application, attributes of unstructured data can be supplemented to ensure that the structured data table can be successfully updated based on the unstructured data later; the unstructured data can be diverted to obtain field information (i.e., structure flow or schema flow) and content information (i.e., data flow or write flow), and different processing flows can be used to process these two pieces of information respectively, thereby effectively shortening the data processing flow and improving data processing efficiency. At the same time, the introduction of the field and column number judgment flow and the structured data table access status adjustment flow (i.e., locking and unlocking flow) can effectively avoid data modification anomalies and effectively ensure the security of data in the structured data table; in addition, when the data processing method provided by the present application is applied to a scenario where the upstream is an unstructured database and the downstream is a structured database, the synchronization of upstream and downstream data can also be achieved to ensure data consistency, which is conducive to meeting application requirements and improving user experience.
[0130] See also Figure 8 , Figure 8 This is a flow chart of another data processing method provided in an embodiment of the present application. The data processing method can be implemented by the above-mentioned data processing device 102. Figure 8 The application scenario corresponding to the data processing method shown is to synchronize a large amount of data in an unstructured database into a structured data table. The process of the data processing method provided in the embodiment of the present application includes but is not limited to:
[0131] S801. In response to the data processing request, obtain a target set in the unstructured database including at least one unstructured data, where the target set is any set included in the unstructured database.
[0132] In an embodiment of the present application, an unstructured database (e.g., MongoDB) contains one or more collections, each collection contains one or more files, and each file stores unstructured data. A data processing device can receive a data processing request sent by a client, and the data processing request is used to request that the unstructured data contained in the target collection of the unstructured database be synchronized to a structured data table. After receiving the data processing request, the data processing device can perform full synchronization and incremental synchronization on the unstructured data in the target collection, wherein full synchronization refers to that after the data processing device receives the data processing request, the data processing device immediately processes the unstructured data stored in the current target collection to obtain a structured data table; incremental synchronization refers to that after the data processing device receives the data processing request for a period of time, if the unstructured data stored in the target collection changes, the data processing device can synchronize and update the structured data table corresponding to the target collection according to the updated unstructured data. The process described in steps S801-S804 in the embodiment of the present application is an incremental synchronization process. Specifically, the data processing device can respond to the data processing request and obtain the unstructured data contained in the target set some time after receiving the data processing request, so that after subsequently detecting that the target set has been updated, the updated target unstructured data can be determined based on the update incremental information and the acquired unstructured data.
[0133] It should be noted that the data processing request may be for a certain set in the unstructured database or for the entire unstructured database.
[0134] In one embodiment, a specific implementation method for a data processing device to perform full synchronization may be: receiving a data processing request for a target set in an unstructured database, the unstructured database including one or more sets, each set including at least one unstructured data, and the target set being any set in the unstructured database; traversing at least one unstructured data in the target set, and updating a structured data table obtained in a previous traversal according to the currently traversed unstructured data; after the traversal is completed, determining that the structured data table associated with the target set is the structured data table obtained after the traversal is completed.
[0135] Specifically, the data processing device can receive a data processing request for a target set, and traverse each unstructured data in the target set, and update the structured data table obtained in the previous traversal according to the currently traversed unstructured data, and the update process can be shown in the above steps S201-S204. It should be noted that if there is no structured data table corresponding to the target set when the data processing request is received, the data processing device can read the first record (or unstructured data) from the target set, and create a structured data table based on the record, and then read subsequent records in turn, and update the structured data table. After the traversal is completed, it can be determined that the structured data table associated with the target set is the structured data table obtained after the traversal is completed.
[0136] In some cases, the data processing method provided by this application can be used in actual applications to synchronize one or more MongoDB tables to a paimon table by using MongoDBSyncTableAction in a Flink DataStream job or directly running "flink run". To use this function through "flink run", you can execute the following shell command: Line 1:<FLINK_HOME> / bin / flink run\
[0137] Line 2: / path / to / paimon-flink-action-0.5-SNAPSHOT.jar\
[0138] Line 3: mongodb-sync-table
[0139] Line 4: --warehouse <warehouse-path>\
[0140] Line 5: --database <database-name>\
[0141] Line 6: --table <table-name>\
[0142] Line 7: --partition-keys <partition-keys>\
[0143] Line 8: --primary-keys <primary-keys>\
[0144] Line 9: --mongodb-conf hosts=localhost:27017\
[0145] Line 10: --mongodb-conf database=test\
[0146] Line 11: --mongodb-conf collection=temp\
[0147] Line 12: [--catalog-conf <paimon-catalog-conf>[--catalog-conf <paimon-catalog-conf>...]]\
[0148] Line 13: [--table-conf <paimon-table-sink-conf>[--table-conf <paimon-table-sink-conf>...]]
[0149] In the above code, lines 4-8 indicate the relevant parameters of the structured data table (i.e., warehouse path, database name, table name, partition, primary key), lines 9-11 indicate the relevant parameters of the unstructured database (i.e., communication interface, unstructured database name, collection name), and lines 12-13 indicate the connection parameters related to the structured data table. Through the above code, the unstructured data stored in the temp collection in the test database in MongoDB can be synchronized to the structured data table.
[0150] It should be noted that in the full synchronization and incremental synchronization stages, the data processing device can reuse the RichCdcMultiplexR ecordEventParser#parseSchemaChange() method to update the structured data table. Through the method provided in the embodiment of the present application, the conversion of library-level and collection-level unstructured data to structured data tables can be realized, which effectively improves the data processing efficiency and is also conducive to subsequent other processing based on the data in the structured data table.
[0151] See also Fig. 9 , which is a schematic diagram of another data processing method provided in an embodiment of the present application. In the figure, the unstructured database is MongoDB, and MongoDB contains four collections, each collection contains multiple files, and each file stores unstructured data. The data processing device receives a data processing request for the unstructured database sent by the client, and the data processing request is used to request that the unstructured data in MongoDB's collection 1 be synchronized to the structured data table. The data processing device can then obtain the first unstructured data in MongoDB's collection 1, and build an initial structured data table based on the unstructured data. The data processing device can traverse each unstructured data in collection 1 in turn, and adjust the table structure and table content of the structured data table based on these unstructured data ( Fig. 9 The "pending columns" in the table refer to the columns updated during the data processing process). Through the method provided in the embodiment of the present application, unstructured data can be synchronized to the structured data table, shortening the data processing process and improving the data processing efficiency.
[0152] S802: If it is detected that the target set is updated, the update increment information of the target set is obtained.
[0153] In an embodiment of the present application, the data processing device can detect the unstructured database, and if it is detected that the target set has been updated, the update increment information of the target set can be obtained. In some cases, when the unstructured database is a MongoDB database, since MongoDB is a distributed database, when the data in the master database changes, the master database can send the update information to the slave database to ensure the consistency of the data between the master and slave databases, and the data processing device can use the cdcSource function to monitor the communication between the master and slave databases and obtain the update information sent by the master database.
[0154] The update incremental information of the target collection can indicate the changes in the unstructured data in the target collection. For example, the update incremental information can be "Add a new column.db.getCollection('temp').updateOne({_id:ObjectId("643e9943fa4dfe102cfcfa61")},{$set:{name:"zhangsan"}})", and the update incremental information indicates that the data numbered (_id) as "643e9943fa4dfe102cfcfa61" in the collection "temp" adds the information "name:"zhangsan"". Through the method provided in the embodiment of the present application, the update incremental information of the target collection can be obtained, which is conducive to the subsequent update processing of the structured data table according to the update incremental information, thereby ensuring data synchronization between the unstructured database and the structured data table.
[0155] It should be noted that, in some cases, if the update incremental information directly includes updated unstructured data, there is no need to execute steps S801 and S803, and step S804 is directly executed according to the updated unstructured data included in the update incremental information.
[0156] S803. Obtain target unstructured data according to at least one unstructured data included in the target set and the update increment information, where the target unstructured data includes at least one first field and field content of each of the first fields.
[0157] In the embodiment of the present application, the data processing device can obtain the target unstructured data according to the unstructured data and the update increment information included in the target set, and the target unstructured data may include at least one first field and the field content of each first field. That is, the data processing device can determine the updated unstructured data in the target set according to the unstructured data before the update and the update increment information in the target set, and determine the updated unstructured data as the target unstructured data, which is conducive to the subsequent update processing of the structured data table according to the target unstructured data.
[0158] S804. Perform data analysis on the target unstructured data to obtain field information and content information of the target unstructured data, perform structural update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure, and perform data update processing on the structured data table with updated structure according to the content information to obtain an updated structured data table.
[0159] In the embodiment of the present application, the data processing device can perform data analysis and processing on the target unstructured data to obtain field information and content information of the target unstructured data, where the field information may include at least one first field and field attribute information of each first field, and the content information includes data indicated by the field content of each first field and the number of field contents of at least one first field. The specific implementation method of this step can be as shown in the above step S202.
[0160] The data processing device can perform structure update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure, that is, to implement schema update of the structured data table. The specific implementation method of this step can be as shown in the above step S203.
[0161] The data processing device can perform data update processing on the structured data table with updated structure according to the content information to obtain an updated structured data table, the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information. It should be noted that when updating the structured data table with updated structure according to the content information, the relevant rules of the structured data table must also be followed. For example: each data in the structured data table should have a value of the primary key field, and the value of the primary key field of each data is different. When the structured data table is updated according to the content data, if matching data with the same value of the primary key field already exists in the structured data table, the data processing device can replace the matching data according to the content data, or delete the matching data, and refill the content data into the structured data table. In some cases, if the business data generated by the business system is stored in an unstructured database, the unstructured database is the upstream data, the data analysis system is associated with the data processing system provided by this application, the data analysis system needs to analyze the structured business data, and the data processing system is the downstream data, then the method provided by this application can synchronize the upstream unstructured data to the downstream structured data table, and can ensure the consistency of the upstream and downstream data, thereby ensuring the analysis accuracy of the data analysis system.
[0162] See also Fig.10 , which is a schematic diagram of a structured data table provided in an embodiment of the present application. Fig.10 As shown, the unstructured database can receive data update instructions sent by the client and process the unstructured data stored in it. The data processing device includes a structured data table (here, the structured data table is a hive table), and the device can query the stored structured data table (for example, hive's Beeline can be used for table query). If the data processing device can query the structured data table corresponding to the unstructured database, the first query statement is "select * from temp". It can be seen from the first query result that the current temp table contains a field temp.id, and the table contains three data.
[0163] The client sends a first data update instruction to the unstructured database, and the first data update instruction is "Add anew column.db.getCollection('temp').updateOne({_id:ObjectId("01")},{$set:{email:"1@example.com"}})", and the unstructured database adjusts the stored unstructured data according to the first data update instruction. The data processing method provided in the embodiment of the present application can sense the changes in the unstructured data of the unstructured database, and adjust the corresponding structured data table, thereby realizing data synchronization. At this time, the structured data table corresponding to the unstructured database is queried again, and the second query statement is "select * from temp". It can be seen from the second query result that the current temp table contains two fields temp.id and temp.email, and the table contains three data. If the data in the unstructured database changes subsequently, the method provided in the present application can also realize the synchronous changes of the data in the structured data table, which can effectively ensure the consistency of the data.
[0164] In an embodiment of the present application, attributes of unstructured data can be supplemented to ensure that the structured data table can be successfully updated based on the unstructured data later; unstructured data can be diverted to obtain field information (i.e., structure flow or schema flow) and content information (i.e., data flow or write flow), and different processing flows are used to process these two pieces of information respectively, thereby effectively shortening the data processing flow and improving data processing efficiency. At the same time, the introduction of the field and column number judgment flow and the structured data table access status adjustment flow (i.e., locking and unlocking flow) can effectively avoid data modification anomalies and effectively ensure the security of data in the structured data table; in addition, when the data processing method provided by the present application is applied to a scenario where the upstream is an unstructured database and the downstream is a structured database, the synchronization of upstream and downstream data can also be achieved, ensuring data consistency, which is conducive to meeting application requirements and improving the user experience; in a scenario where a large amount of unstructured data needs to be converted into structured data, the method provided by the present application can achieve data synchronization, effectively improve data processing efficiency, save storage resources, and reduce storage costs.
[0165] See also Fig.11 , Fig.11 A flowchart of another data processing method provided in an embodiment of the present application is provided. The data processing method can be implemented by the above-mentioned data processing device 102. Fig.11 The application scenario corresponding to the data processing method shown is to add a small amount of unstructured data to a structured data table. The process of the data processing method provided in the embodiment of the present application includes but is not limited to:
[0166] S1101. Receive a data conversion request sent by a client, and perform device authentication processing according to authentication information to obtain an authentication result. The data conversion request includes the authentication information of the client, an unstructured data file, and a data conversion requirement.
[0167] In an embodiment of the present application, a data processing device may receive a data conversion request sent by a client, and the data conversion request includes the client's authentication information (e.g., the client's name, password, etc.), an unstructured data file, and a data conversion requirement. The unstructured data file contains a small amount of unstructured data, and the data conversion requirement may be a conversion requirement (e.g., synchronizing only certain fields in the unstructured data to a structured data table), a format requirement (e.g., the meaning represented by a special character), etc. The data processing device may verify the identity of the client and the authority to import data according to the authentication information of the client, and obtain an authentication result. In some cases, there is a small amount of unstructured data that needs to be synchronized to a structured data table. In order to shorten the processing flow and improve processing efficiency, the present application provides a LoadAction service, which is used to execute the data processing method shown in steps S201-S204. The client can directly provide a file containing a small amount of unstructured data to the LoadAction service without performing complex data import operations, thereby completing the conversion of unstructured data to a structured data table without relying on an additional data processing engine.
[0168] Specifically, the data conversion request can be an http request. For example, the data conversion request is "curl -uuser:pass wd[-H""...]-T data.file-XPUT http: / / load_host:http_port / api / {db} / {table} / _load", where "user:passwd" refers to the client's authentication data, "data.file" refers to the unstructured data file, and "http: / / load_host:http_port / api / {db} / {table} / _load" refers to the access address of the LoadAction service. Through this data conversion request, the client can send the unstructured data file to the data processing service, realizing convenient and fast data submission (or data import), without the need for additional table creation or data import operations, greatly improving the efficiency of data processing.
[0169] In some cases, the http request can also carry data conversion requirements. You can add SQL parameters in the http request, which can replace previous parameters, such as column_separator, line_delimiter, where, column, etc. For example, the http request is "curl -u user:passwd [-H sql:${load_sql}...] -T data.file -XPUT http: / / load_host:http_port / api / http_load
[0170] #--load_sql
[0171] #CREATE TABLE db.table(col BIGINT PRIMARY KEY NOT ENFORCED,...)WIT H('sequence.field'='dt');
[0172] #INSERT INTO db.table(col,...)SELECT col,...FROM http_load(file: / / / / <path>)WITH("column_separator"=",","format"="csv");"--load_sql" indicates the data conversion requirement (i.e., SQL parameter), tag: the identifier of the import task, column_separator is used to specify the column separator in the import file, which defaults to \t; line_delimiter is used to specify the line break in the import file, which defaults to \n; where is the filter condition specified by the import task; columns indicates the name of the column field of the data to be imported; enclose is a closing character. When the data field of the unstructured data file (CSV) contains a row separator or a column separator, When using CSV as a separator, single-byte characters can be specified as closed characters to prevent accidental truncation. For example, if the column separator is "," and the attachment character is "'", then in the data "a,'b,c'", "b,c" will be interpreted as a single field. Escape indicates an escape character, which is used to escape characters that are the same as the shell characters in the CSV field. For example, if the data is "a,'b,'c", and the closing character is "'", and you want to interpret "b,'c" as a single field, you need to specify a single-byte escape character, such as a space character, then the data should be modified to "a,'b,'c'". format is used to specify the format of the data to be imported. This application supports csv format and json format, and the default is cs v format. Through the method provided in the embodiment of the present application, the data conversion requirements can be carried in the data conversion request, so that the determined unstructured data table is more in line with the application requirements, the processing efficiency and operation convenience can be improved, thereby improving the user experience.
[0173] S1102: If the authentication result indicates that the authentication is successful, the unstructured data file is processed according to the data conversion requirement to obtain the target unstructured data.
[0174] In an embodiment of the present application, the data processing device can indicate that the authentication is passed based on the authentication result, and the data processing device can process the unstructured data file according to the data conversion requirements to obtain the target unstructured data, so as to facilitate importing the target unstructured data into the structured data table.
[0175] S1103. Perform data analysis on the target unstructured data to obtain field information and content information of the target unstructured data, perform structural update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure, and perform data update processing on the structured data table with updated structure according to the content information to obtain an updated structured data table.
[0176] In the embodiment of the present application, the data processing device can perform data analysis and processing on the target unstructured data to obtain field information and content information of the target unstructured data, where the field information may include at least one first field and field attribute information of each first field, and the content information includes data indicated by the field content of each first field and the number of field contents of at least one first field. The specific implementation method of this step can be as shown in the above step S202.
[0177] The data processing device can perform structure update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure, that is, to implement schema update of the structured data table. The specific implementation method of this step can be as shown in the above step S203.
[0178] The data processing device can perform data update processing on the structured data table with updated structure according to the content information to obtain an updated structured data table, wherein the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information. The specific implementation method of this step can be as shown in the above step S204.
[0179] S1104: Generate data conversion response information according to the updated structured data table, and send the data conversion response information to the client.
[0180] In an embodiment of the present application, the data processing device can generate data conversion response information based on the updated structured data table, and the data conversion response information can indicate that the data conversion is successful. The data processing device (or LoadAction service) can send the data conversion response information to the client. In some cases, if the data conversion fails (for example: the authentication result indicates that the authentication fails or the process of synchronizing the target unstructured data to the structured data table fails), the data processing device can generate data conversion response information based on the reason for the data conversion failure, and send the data conversion response information to the client. Through the method provided in the embodiment of the present application, fast and efficient synchronization of a small amount of unstructured data can be achieved, and flexible data synchronization operations can be completed without the help of other complex data processing engines, and the operation is convenient and fast, which can improve the user experience.
[0181] See also Fig.12 , which is a schematic diagram of a data conversion method provided by an embodiment of the present application. The client can generate a data conversion request, which is "curl-T Table.csv-XPUT http: / / 127.0.0.1:8888 / api / {db} / {table} / _load", where the unstructured data file indicated by "Table.csv" is as follows Fig.12 As shown. The client can send the data conversion request to the LoadAction service (the network address of the LoadAction service is 127.0.0.1:8888). After the Load Action service receives the unstructured data file in the data conversion request, it can process the data to obtain a structured data table paimon Table. The LoadAction service can generate data conversion response information and send the data conversion response information to the client. The data analysis system can obtain the structured data table generated by the LoadAction service, and perform data analysis and processing based on these structured data tables. Through the method provided in the embodiment of the present application, it is possible to achieve rapid conversion of a small amount of unstructured data to a structured data table, thereby improving data processing efficiency and enhancing the user experience.
[0182] In an embodiment of the present application, attributes of unstructured data can be supplemented to ensure that the structured data table can be successfully updated based on the unstructured data later; the unstructured data can be diverted to obtain field information (i.e., structure flow or schema flow) and content information (i.e., data flow or write flow), and different processing flows are used to process these two pieces of information respectively, thereby effectively shortening the data processing flow and improving data processing efficiency. At the same time, the introduction of the field and column number judgment flow and the structured data table access status adjustment flow (i.e., locking and unlocking flow) can effectively avoid data modification anomalies and effectively ensure the security of data in the structured data table; in addition, fast and efficient synchronization of small amounts of unstructured data can be achieved, and flexible data synchronization operations can be completed without the aid of other complex data processing engines, and the operation is convenient and fast, which can improve the user experience.
[0183] See also Fig.13 , Fig.13 A structural block diagram of a data processing device provided in an embodiment of the present application.
[0184] In one implementation of the data processing device of an embodiment of the present application, the data processing device includes the following structure.
[0185] An acquiring unit 1301 is configured to acquire target unstructured data, where the target unstructured data includes at least one first field and field contents of each of the first fields;
[0186] The processing unit 1302 is configured to perform data analysis processing on the target unstructured data to obtain field information and content information of the target unstructured data; wherein the field information includes the at least one first field and field attribute information of each of the first fields, and the content information includes data indicated by the field content of each of the first fields and the number of field contents of the at least one first field;
[0187] The processing unit 1302 is further configured to perform a structure update process on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with an updated structure;
[0188] The processing unit 1302 is also used to perform data update processing on the structured data table with the structure updated according to the content information to obtain an updated structured data table, wherein the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information.
[0189] In one embodiment, the processing unit 1302 is used to perform a structure update process on the structured data table corresponding to the target unstructured data according to the field information, and when obtaining the structured data table with updated structure, specifically to perform the following steps:
[0190] Acquire table structure information of the structured data table, where the table structure information includes each second field in the structured data table and field attribute information of each second field;
[0191] If the field information matches the table structure information, determining the structured data table as a structured data table with updated structure;
[0192] If the field information does not match the table structure information, the structured data table is locked to obtain a structured data table whose access status is locked; wherein the structured data table whose access status is locked cannot be updated;
[0193] updating the table structure of the structured data table in the locked state according to the field information to obtain a structured data table in the locked state with an updated structure;
[0194] The structured data table for structural update in the locked state is unlocked to obtain the structured data table for structural update.
[0195] In one embodiment, the processing unit 1302 is further configured to perform the following steps:
[0196] Determining an access status of the structured data table;
[0197] If the access state of the structured data table is locked, suspending data update processing on the structured data table;
[0198] After a preset time period has passed, determining the access status of the structured data table again;
[0199] If the access state of the structured data table is unlocked, the structured data table with the updated structure is obtained, and the step of performing data update processing on the structured data table with the updated structure according to the content information to obtain an updated structured data table is triggered.
[0200] In one embodiment, the processing unit 1302 is used to perform data update processing on the structured data table with updated structure according to the content information, and when obtaining the updated structured data table, specifically to perform the following steps:
[0201] Obtain the number of columns contained in the structured data table for structural update;
[0202] If the number of field contents in the content information matches the number of columns, then, according to the content information, writing the data indicated by the field contents of each of the first fields into the structured data table with the structure updated, to obtain an updated structured data table;
[0203] If the number of field contents in the content information does not match the number of columns, wait for a target time period, and execute the step of obtaining the number of columns contained in the structured data table of the structure update and subsequent steps.
[0204] In one embodiment, the processing unit 1302 is further configured to perform the following steps:
[0205] Acquire supplementary information of the target unstructured data, where the supplementary information includes one or more of a table name, a library name, a field type, and a field identifier;
[0206] Performing attribute supplementation processing on the target unstructured data using the supplementary information to obtain the target unstructured data after attribute supplementation;
[0207] The processing unit 1302 is used to perform data analysis on the target unstructured data to obtain field information and content information of the target unstructured data, and is specifically used to perform the following steps:
[0208] Data analysis and processing are performed on the target unstructured data after the attribute supplementation to obtain field information and content information of the target unstructured data.
[0209] In one embodiment, the apparatus further includes a receiving unit 1303. The receiving unit 1303 is further configured to perform the following steps:
[0210] Receiving a data processing request for a target set in an unstructured database, the unstructured database includes one or more sets, each set includes at least one piece of unstructured data, and the target set is any set in the unstructured database;
[0211] Traversing at least one piece of unstructured data in the target set, and updating the structured data table obtained by the last traversal according to the currently traversed unstructured data;
[0212] After the traversal is completed, it is determined that the structured data table associated with the target set is the structured data table obtained after the traversal is completed.
[0213] In one embodiment, when the acquisition unit 1301 is used to acquire the target unstructured data, it is specifically used to perform the following steps:
[0214] In response to the data processing request, obtaining the target set including at least one piece of unstructured data;
[0215] If it is detected that the target set is updated, obtaining the update increment information of the target set;
[0216] The target unstructured data is obtained according to at least one piece of unstructured data included in the target set and the update increment information.
[0217] In one embodiment, the processing unit 1302 is used to perform data analysis on the target unstructured data to obtain the field information and content information of the target unstructured data, and is specifically used to perform the following steps:
[0218] If the type of the field content of the target first field is a non-nested data type, determining the field content of the target first field as the data indicated by the field content of the target first field, the target first field being any first field in the target unstructured data;
[0219] If the type of the field content of the target first field is a nested data type, outer nested parsing processing is performed on the field content of the target first field to obtain parsed data of the field content of the target first field, and the parsed data of the field content of the target first field is determined as the data indicated by the field content of the target first field;
[0220] Determining content information of the target unstructured data according to data indicated by field contents of each of the first fields in the target unstructured data;
[0221] According to at least one first field included in the target unstructured data, field information of the target unstructured data is determined.
[0222] In one embodiment, the processing unit 1302 is further configured to perform the following steps:
[0223] Receiving a data query request for a structured data table corresponding to the target unstructured data, wherein the data query request carries a data query condition;
[0224] Perform query processing on the structured data table according to the data query condition to obtain query result data;
[0225] If the type of the query result data is a non-nested data type, outputting the query result data;
[0226] If the type of the query result data is a nested data type, the query result data is parsed using a nested parsing function to obtain parsed result data, and the parsed result data is output.
[0227] In one embodiment, when the acquisition unit 1301 is used to acquire the target unstructured data, it is specifically used to perform the following steps:
[0228] Receive a data conversion request sent by a client, the data conversion request including authentication information of the client, an unstructured data file, and a data conversion requirement;
[0229] Perform device authentication processing according to the authentication information to obtain an authentication result;
[0230] If the authentication result indicates that the authentication is successful, the unstructured data file is processed according to the data conversion requirement to obtain the target unstructured data.
[0231] Through the data processing device provided in the embodiment of the present application, the attributes of unstructured data can be supplemented, thereby ensuring that the structured data table can be successfully updated based on the unstructured data in the future; the unstructured data can be diverted to obtain field information (i.e., structure flow or schema flow) and content information (i.e., data flow or write flow), and different processing flows are used to process these two information respectively, thereby effectively shortening the data processing flow and improving data processing efficiency. At the same time, the introduction of the field and column number judgment flow and the access status adjustment flow of the structured data table (i.e., the locking and unlocking flow) can effectively avoid data modification anomalies and effectively ensure the security of data in the structured data table; in addition, when the data processing method provided by the present application is applied to a scenario where the upstream is an unstructured database and the downstream is a structured database, the synchronization of upstream and downstream data can also be achieved to ensure data consistency, which is conducive to meeting application requirements and improving user experience.
[0232] Based on the above method and device embodiments, the present application embodiment provides a computer device. Fig.14 , Fig.14 A structural block diagram of a computer device provided in an embodiment of the present application. Fig.14 The computer device shown can be the above Figure 1 The data processing device 102 in FIG. Fig.14 The computer device shown includes at least a processor 1401, an input interface 1402, an output interface 1403, and a computer-readable storage medium 1404. The processor 1401, the input interface 1402, the output interface 1403, and the computer-readable storage medium 1404 may be connected via a bus or other means.
[0233] The computer-readable storage medium 1404 may be stored in a memory of a computer device. The computer-readable storage medium 1404 is used to store a computer program. The computer program includes program instructions (such as Fig.14 The processor 1401 is used to execute the computer program stored in the computer-readable storage medium 1404. The processor 1401 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing the computer program, specifically suitable for loading and executing the computer program to implement the corresponding method flow or corresponding function.
[0234] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the computer device. In addition, a computer program suitable for being loaded and executed by a processor is also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.
[0235] In a specific implementation, the processor 1401 may load and execute a computer program stored in the computer-readable storage medium 1404 to implement the corresponding steps of the above data processing method. In a specific implementation, the computer program in the computer-readable storage medium 1404 is loaded by the processor 1401 and executes the following steps:
[0236] Acquire target unstructured data, where the target unstructured data includes at least one first field and field content of each of the first fields;
[0237] Performing data analysis and processing on the target unstructured data to obtain field information and content information of the target unstructured data; wherein the field information includes the at least one first field and field attribute information of each of the first fields, and the content information includes data indicated by the field content of each of the first fields and the number of field contents of the at least one first field;
[0238] Performing structure update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure;
[0239] The structured data table with the structure updated is subjected to data updating processing according to the content information to obtain an updated structured data table, wherein the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information.
[0240] In one embodiment, the computer program in the computer-readable storage medium 1404 is loaded and executed by the processor 1401 to perform a structure update process on the structured data table corresponding to the target unstructured data according to the field information, and when the structured data table with updated structure is obtained, it is specifically used to perform the following steps:
[0241] Acquire table structure information of the structured data table, where the table structure information includes each second field in the structured data table and field attribute information of each second field;
[0242] If the field information matches the table structure information, determining the structured data table as a structured data table with updated structure;
[0243] If the field information does not match the table structure information, the structured data table is locked to obtain a structured data table whose access status is locked; wherein the structured data table whose access status is locked cannot be updated;
[0244] updating the table structure of the structured data table in the locked state according to the field information to obtain a structured data table in the locked state with an updated structure;
[0245] The structured data table for structural update in the locked state is unlocked to obtain the structured data table for structural update.
[0246] In one embodiment, when the computer program in the computer-readable storage medium 1404 is loaded and executed by the processor 1401, it is also used to perform the following steps:
[0247] Determining an access status of the structured data table;
[0248] If the access state of the structured data table is locked, suspending data update processing on the structured data table;
[0249] After a preset time period has passed, determining the access status of the structured data table again;
[0250] If the access state of the structured data table is unlocked, the structured data table with the updated structure is obtained, and the step of performing data update processing on the structured data table with the updated structure according to the content information to obtain an updated structured data table is triggered.
[0251] In one embodiment, the computer program in the computer-readable storage medium 1404 is loaded and executed by the processor 1401 to perform data update processing on the structured data table with the structure updated according to the content information, and when the updated structured data table is obtained, it is specifically used to perform the following steps:
[0252] Obtain the number of columns contained in the structured data table for structural update;
[0253] If the number of field contents in the content information matches the number of columns, then, according to the content information, writing the data indicated by the field contents of each of the first fields into the structured data table with the structure updated, to obtain an updated structured data table;
[0254] If the number of field contents in the content information does not match the number of columns, wait for a target time period, and execute the step of obtaining the number of columns contained in the structured data table of the structure update and subsequent steps.
[0255] In one embodiment, when the computer program in the computer-readable storage medium 1404 is loaded and executed by the processor 1401, it is also used to perform the following steps:
[0256] Acquire supplementary information of the target unstructured data, where the supplementary information includes one or more of a table name, a library name, a field type, and a field identifier;
[0257] Performing attribute supplementation processing on the target unstructured data using the supplementary information to obtain the target unstructured data after attribute supplementation;
[0258] When performing data analysis processing on the target unstructured data to obtain field information and content information of the target unstructured data, it is specifically used to perform the following steps:
[0259] Data analysis and processing are performed on the target unstructured data after the attribute supplementation to obtain field information and content information of the target unstructured data.
[0260] In one embodiment, when the computer program in the computer-readable storage medium 1404 is loaded and executed by the processor 1401, it is also used to perform the following steps:
[0261] Receiving a data processing request for a target set in an unstructured database, the unstructured database includes one or more sets, each set includes at least one piece of unstructured data, and the target set is any set in the unstructured database;
[0262] Traversing at least one piece of unstructured data in the target set, and updating the structured data table obtained by the last traversal according to the currently traversed unstructured data;
[0263] After the traversal is completed, it is determined that the structured data table associated with the target set is the structured data table obtained after the traversal is completed.
[0264] In one embodiment, when the computer program in the computer-readable storage medium 1404 is loaded and executed by the processor 1401 to obtain the target unstructured data, it is specifically used to perform the following steps:
[0265] In response to the data processing request, obtaining the target set including at least one piece of unstructured data;
[0266] If it is detected that the target set is updated, obtaining the update increment information of the target set;
[0267] The target unstructured data is obtained according to at least one piece of unstructured data included in the target set and the update increment information.
[0268] In one embodiment, the computer program in the computer-readable storage medium 1404 is loaded and executed by the processor 1401 to perform data analysis on the target unstructured data to obtain the field information and content information of the target unstructured data, specifically for performing the following steps:
[0269] If the type of the field content of the target first field is a non-nested data type, determining the field content of the target first field as the data indicated by the field content of the target first field, the target first field being any first field in the target unstructured data;
[0270] If the type of the field content of the target first field is a nested data type, outer nested parsing processing is performed on the field content of the target first field to obtain parsed data of the field content of the target first field, and the parsed data of the field content of the target first field is determined as the data indicated by the field content of the target first field;
[0271] Determining content information of the target unstructured data according to data indicated by field contents of each of the first fields in the target unstructured data;
[0272] According to at least one first field included in the target unstructured data, field information of the target unstructured data is determined.
[0273] In one embodiment, when the computer program in the computer-readable storage medium 1404 is loaded and executed by the processor 1401, it is also used to perform the following steps:
[0274] Receiving a data query request for a structured data table corresponding to the target unstructured data, wherein the data query request carries a data query condition;
[0275] Perform query processing on the structured data table according to the data query condition to obtain query result data;
[0276] If the type of the query result data is a non-nested data type, outputting the query result data;
[0277] If the type of the query result data is a nested data type, the query result data is parsed using a nested parsing function to obtain parsed result data, and the parsed result data is output.
[0278] In one embodiment, when the computer program in the computer-readable storage medium 1404 is loaded and executed by the processor 1401 to obtain the target unstructured data, it is specifically used to perform the following steps:
[0279] Receive a data conversion request sent by a client, the data conversion request including authentication information of the client, an unstructured data file, and a data conversion requirement;
[0280] Perform device authentication processing according to the authentication information to obtain an authentication result;
[0281] If the authentication result indicates that the authentication is successful, the unstructured data file is processed according to the data conversion requirement to obtain the target unstructured data.
[0282] Through the computer device provided in the embodiment of the present application, the attributes of unstructured data can be supplemented, so as to ensure that the structured data table can be successfully updated based on the unstructured data in the future; the unstructured data can be diverted to obtain field information (i.e., structure flow or schema flow) and content information (i.e., data flow or write flow), and different processing flows are used to process these two information respectively, so as to effectively shorten the data processing flow and improve the data processing efficiency. At the same time, the introduction of the field and column number judgment flow and the access status adjustment flow of the structured data table (i.e., the locking and unlocking flow) can effectively avoid data modification anomalies and effectively ensure the security of data in the structured data table; in addition, when the data processing method provided by the present application is applied to a scenario where the upstream is an unstructured database and the downstream is a structured database, the synchronization of upstream and downstream data can also be achieved to ensure the consistency of the data, which is conducive to meeting application requirements and improving the user experience.
[0283] The embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored, and when the computer instructions are executed on a computer device, the computer device executes the steps in each method embodiment of the present application to implement the data processing method provided by the embodiment of the present application. The specific implementation method can be referred to the above description, which will not be repeated here.
[0284] The embodiment of the present application also provides a computer program product, which includes a computer program or a computer instruction, and the computer program or the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer program or the computer instruction from the computer-readable storage medium, and the processor executes the computer program or the computer instruction, so that the computer device executes the steps in the embodiments of the present application to implement the data processing method provided in the embodiment of the present application. The specific implementation method can be referred to the above description, which will not be repeated here.
[0285] It is appreciated by those skilled in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or in combination with computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical objects can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or memory) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit containing the function of the module or unit.
[0286] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
[0287] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technical object familiar with the technical field within the technical scope disclosed in the present application can be easily thought of for changes or replacements, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.< / path> < / uuid> < / numberlong> < / timestamp> < / field> < / document> < / value> < / collection> < / database> < / collection> < / database> < / document> < / operation>
Claims
1. A data processing method, characterized in that: The method comprises: Acquire target unstructured data, where the target unstructured data includes at least one first field and field content of each of the first fields; Performing data analysis and processing on the target unstructured data to obtain field information and content information of the target unstructured data; wherein the field information includes the at least one first field and field attribute information of each of the first fields, and the content information includes data indicated by the field content of each of the first fields and the number of field contents of the at least one first field; Performing structure update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure; The structured data table with the structure updated is subjected to data updating processing according to the content information to obtain an updated structured data table, wherein the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information.
2. The method according to claim 1, characterized in that The step of performing a structure update process on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with an updated structure includes: Acquire table structure information of the structured data table, where the table structure information includes each second field in the structured data table and field attribute information of each second field; If the field information matches the table structure information, determining the structured data table as a structured data table with updated structure; If the field information does not match the table structure information, the structured data table is locked to obtain a structured data table whose access status is locked; wherein the structured data table whose access status is locked cannot be updated; updating the table structure of the structured data table in the locked state according to the field information to obtain a structured data table in the locked state with an updated structure; The structured data table for structural update in the locked state is unlocked to obtain the structured data table for structural update.
3. The method according to claim 2, characterized in that The method further comprises: Determining an access status of the structured data table; If the access state of the structured data table is locked, suspending data update processing on the structured data table; After a preset time period has passed, determining the access status of the structured data table again; If the access state of the structured data table is unlocked, the structured data table with the updated structure is obtained, and the step of performing data update processing on the structured data table with the updated structure according to the content information to obtain an updated structured data table is triggered.
4. The method according to claim 1, characterized in that The step of performing data updating processing on the structured data table of the structure update according to the content information to obtain an updated structured data table includes: Obtain the number of columns contained in the structured data table for structural update; If the number of field contents in the content information matches the number of columns, then, according to the content information, writing the data indicated by the field contents of each of the first fields into the structured data table with the structure updated, to obtain an updated structured data table; If the number of field contents in the content information does not match the number of columns, wait for a target time period, and execute the step of obtaining the number of columns contained in the structured data table of the structure update and subsequent steps.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire supplementary information of the target unstructured data, where the supplementary information includes one or more of a table name, a library name, a field type, and a field identifier; Performing attribute supplementation processing on the target unstructured data using the supplementary information to obtain the target unstructured data after attribute supplementation; The performing data analysis on the target unstructured data to obtain field information and content information of the target unstructured data includes: Data analysis and processing are performed on the target unstructured data after the attribute supplementation to obtain field information and content information of the target unstructured data.
6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Receiving a data processing request for a target set in an unstructured database, the unstructured database includes one or more sets, each set includes at least one piece of unstructured data, and the target set is any set in the unstructured database; Traversing at least one piece of unstructured data in the target set, and updating the structured data table obtained by the last traversal according to the currently traversed unstructured data; After the traversal is completed, it is determined that the structured data table associated with the target set is the structured data table obtained after the traversal is completed.
7. The method according to claim 6, characterized in that The obtaining of target unstructured data includes: In response to the data processing request, obtaining the target set including at least one piece of unstructured data; If it is detected that the target set is updated, obtaining the update increment information of the target set; The target unstructured data is obtained according to at least one piece of unstructured data included in the target set and the update increment information.
8. The method according to any one of claims 1 to 4, characterized in that: The performing data analysis on the target unstructured data to obtain field information and content information of the target unstructured data includes: If the type of the field content of the target first field is a non-nested data type, determining the field content of the target first field as the data indicated by the field content of the target first field, the target first field being any first field in the target unstructured data; If the type of the field content of the target first field is a nested data type, outer nested parsing processing is performed on the field content of the target first field to obtain parsed data of the field content of the target first field, and the parsed data of the field content of the target first field is determined as the data indicated by the field content of the target first field; Determining content information of the target unstructured data according to data indicated by field contents of each of the first fields in the target unstructured data; According to at least one first field included in the target unstructured data, field information of the target unstructured data is determined.
9. The method according to claim 8, characterized in that The method further comprises: Receiving a data query request for a structured data table corresponding to the target unstructured data, wherein the data query request carries a data query condition; Perform query processing on the structured data table according to the data query condition to obtain query result data; If the type of the query result data is a non-nested data type, outputting the query result data; If the type of the query result data is a nested data type, the query result data is parsed using a nested parsing function to obtain parsed result data, and the parsed result data is output.
10. The method according to any one of claims 1 to 4, characterized in that: The obtaining of target unstructured data includes: Receive a data conversion request sent by a client, the data conversion request including authentication information of the client, an unstructured data file, and a data conversion requirement; Perform device authentication processing according to the authentication information to obtain an authentication result; If the authentication result indicates that the authentication is successful, the unstructured data file is processed according to the data conversion requirement to obtain the target unstructured data.
11. A data processing device, characterized in that: include: An acquisition unit, configured to acquire target unstructured data, wherein the target unstructured data includes at least one first field and field contents of each of the first fields; a processing unit, configured to perform data analysis on the target unstructured data to obtain field information and content information of the target unstructured data; wherein the field information includes the at least one first field and field attribute information of each of the first fields, and the content information includes data indicated by field contents of each of the first fields and the number of field contents of the at least one first field; The processing unit is further used to perform structure update processing on the structured data table corresponding to the target unstructured data according to the field information to obtain a structured data table with updated structure; The processing unit is also used to perform data update processing on the structured data table with the structure updated according to the content information to obtain an updated structured data table, wherein the table structure of the updated structured data table matches the field information, and the table data of the updated structured data table matches the content information.
12. A computer device, characterized in that: The computer device comprises: a processor suitable for implementing a computer program; A computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by the processor and implementing the data processing method according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and implementing the data processing method according to any one of claims 1 to 10.
14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.
Citation Information
Cited By
Text analysis method and device, electronic terminal and computer readable storage medium
CN121478783A
Systems and methods for data integration
US12524451B1
