Data processing method, system and device, electronic equipment and storage medium

Abstract

The invention relates to a data processing method, system and device, electronic equipment and a storage medium. The data processing method comprises the steps of obtaining a to-be-processed data set, wherein the to-be-processed data set comprises data of multiple data types with different data structures; obtaining an analysis rule corresponding to each data type in the to-be-processed data set; based on an analysis rule, analyzing the data of each data type in the to-be-processed data set into data with the same data structure; and fusing the data with the same data structure to obtain a data processing result. According to the method, when a new data type appears, only a corresponding analysis rule needs to be configured for the new data type, a data processing program does not need to be iteratively updated, and the iteration updating cost can be reduced while the data processing flow is simplified.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, system, device, electronic device and storage medium. Background Art

[0002] In actual production operations, it is a common requirement to perform fusion analysis on data with different data structures. For example, for some equipment with functions such as taking pictures, testing, and processing, the data generated by the equipment includes pictures, XML (Extensible Markup Language) files, test logs, processing logs, etc. When identifying the processing and testing performance of the equipment, it is very important to perform fusion analysis on the data generated by the equipment. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a data processing method, system, device, electronic device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, the method comprising:

[0005] Acquire a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures;

[0006] Obtaining parsing rules corresponding to each data type in the data set to be processed;

[0007] Based on the parsing rule, the data of each data type in the to-be-processed data set is parsed into data having the same data structure;

[0008] The data with the same data structure are merged to obtain a data processing result.

[0009] In an exemplary embodiment, the method further comprises:

[0010] Determine the parsing rules corresponding to different data types according to the data structure of data of different data types;

[0011] The parsing rules and the data types are stored in a database in correspondence.

[0012] In an exemplary embodiment, the obtaining of the parsing rules corresponding to each data type in the to-be-processed data set includes:

[0013] Obtain from the database the parsing rules corresponding to each data type in the data set to be processed.

[0014] In an exemplary embodiment, the merging of the data having the same data structure includes:

[0015] Obtaining associations between different data types in the data set to be processed;

[0016] Based on the association relationship, the data with the same data structure are merged.

[0017] In an exemplary embodiment, the method further comprises:

[0018] Store the associations between different data types in the database.

[0019] In an exemplary embodiment, the obtaining of association relationships between different data types in the data set to be processed includes:

[0020] The association relationship between each data type and other data types in the data set to be processed is obtained from the database.

[0021] In an exemplary embodiment, after parsing the data of each data type in the to-be-processed data set into data having the same data structure, the method further includes:

[0022] The data having the same data structure is stored in the message queue.

[0023] In an exemplary embodiment, the merging of the data having the same data structure includes:

[0024] The data with the same data structure is obtained from the message queue, and the data with the same data structure is merged using a preset computing framework.

[0025] In an exemplary embodiment, the method further comprises:

[0026] When the data set to be processed includes image data, parsing the image data into binary stream data,

[0027] The data having the same data structure is merged with the binary stream data to obtain a data processing result.

[0028] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing system, the system comprising:

[0029] A data acquisition module is configured to obtain a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures;

[0030] The data parsing module is configured to obtain the parsing rules corresponding to each data type in the data set to be processed from the first registration center module; based on the parsing rules, parse the data of each data type in the data set to be processed into data with the same data structure;

[0031] The computing module is configured to fuse the data having the same data structure to obtain a data processing result.

[0032] In an exemplary embodiment, the first registry module is configured to:

[0033] Determine the parsing rules corresponding to different data types according to the data structure of data of different data types;

[0034] The parsing rules and the data types are stored in a database in correspondence.

[0035] In an exemplary embodiment, the system further comprises:

[0036] The data storage module is configured to store the data having the same data structure in a message queue.

[0037] In an exemplary embodiment, the system further comprises:

[0038] The data extraction module is configured to obtain the data with the same data structure from the data storage module.

[0039] In an exemplary embodiment, the system further comprises:

[0040] The second registration center module is configured to store the association relationship between different data types in a database.

[0041] In an exemplary embodiment, the computing module is further configured to:

[0042] Acquire, from the second registration center module, association relationships between different data types in the to-be-processed data set;

[0043] Based on the association relationship, the data with the same data structure are merged.

[0044] According to a third aspect of an embodiment of the present disclosure, a data processing device is provided, the device comprising:

[0045] A first acquisition unit is configured to acquire a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures;

[0046] A second acquisition unit is configured to acquire a parsing rule corresponding to each data type in the data set to be processed;

[0047] A parsing unit, configured to parse data of each data type in the to-be-processed data set into data having the same data structure based on the parsing rule;

[0048] The computing unit is configured to fuse the data having the same data structure to obtain a data processing result.

[0049] According to a fourth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0050] processor;

[0051] a memory for storing processor-executable instructions;

[0052] The processor is configured to execute the method as described in the first aspect of the embodiment of the present disclosure.

[0053] According to a fifth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect of the embodiment of the present disclosure.

[0054] The above method disclosed in the present invention has the following beneficial effects: when a new data type appears, it is only necessary to configure the corresponding parsing rules for it, without the need to iteratively update the data processing program, which can simplify the data processing process while reducing the iterative update cost.

[0055] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0057] Figure 1 is a flow chart of a data processing method according to an exemplary embodiment;

[0058] Figure 2 is a schematic diagram of a data processing flow according to an exemplary embodiment;

[0059] Figure 3 is a flow chart of a data processing method according to an exemplary embodiment;

[0060] Figure 4 is a flow chart of a data processing method according to an exemplary embodiment;

[0061] Figure 5is a block diagram of a data processing system according to an exemplary embodiment;

[0062] Figure 6 is a schematic diagram showing a data processing flow according to an exemplary embodiment;

[0063] Fig. 7A is a schematic diagram of a first registration center module according to an exemplary embodiment;

[0064] Figure 7B is a schematic diagram of a second registration center module according to an exemplary embodiment;

[0065] Figure 8 is a block diagram of a data processing device according to an exemplary embodiment;

[0066] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0068] In some embodiments, heterogeneous data is a general term for data of multiple data types with very different data structures, and homogeneous data is data with basically the same data structure. Therefore, data with different data structures are fused and analyzed, that is, heterogeneous data are converted into homogeneous data for data analysis. In the related art, when heterogeneous data is converted into homogeneous data, a corresponding conversion program is developed according to the data structure of each data type contained in the heterogeneous data to convert it into homogeneous data. However, since the data types contained in the heterogeneous data are variable, the data structure is not fixed, and the data structures of the log files generated by the equipment of various manufacturers are also different, it is impossible to exhaustively enumerate all heterogeneous data types. Whenever a new data type appears in the heterogeneous data, it is necessary to iteratively update the conversion program to make it compatible with the new data type and convert the new data type into homogeneous data. This process is cumbersome and requires high manpower and time costs.

[0069] In an exemplary embodiment of the present disclosure, in order to overcome the problem of cumbersome and high cost of conversion program iteration process caused by variable data types in heterogeneous data in related technologies, a data processing method is provided, including: obtaining a data set to be processed, the data set to be processed includes data of multiple data types with different data structures; obtaining parsing rules corresponding to each data type in the data set to be processed; based on the parsing rules, parsing the data of each data type in the data set to be processed into data with the same data structure; fusing the data with the same data structure to obtain a data processing result. When a new data type appears, the method only needs to configure the corresponding parsing rules for it, without iteratively updating the data processing program, which can simplify the data processing process while reducing the iterative update cost.

[0070] In an exemplary embodiment of the present disclosure, a data processing method is provided. Figure 1 is a flow chart of a data processing method according to an exemplary embodiment. Figure 1 As shown, the following steps are included:

[0071] Step S101, obtaining a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures;

[0072] Step S102, obtaining the parsing rules corresponding to each data type in the data set to be processed;

[0073] Step S103, parsing data of each data type in the data set to be processed into data with the same data structure based on the parsing rule;

[0074] Step S104, merging data with the same data structure to obtain a data processing result.

[0075] The method in the embodiments of the present disclosure is applied to electronic devices, including edge servers and other electronic devices with computing resources or computing capabilities.

[0076] In step S101, the data set to be processed includes data of multiple data types with different data structures, that is, the data in the data set to be processed is heterogeneous data. The data set to be processed is collected by the production equipment and sent to the edge server by the production equipment, or the production equipment is connected to the edge server and collected by the edge server, wherein the production equipment is the equipment for producing products, such as the equipment for dispensing glue to the frame of the mobile phone during the production of the mobile phone. The data types in the data set to be processed include at least two of the following: pictures generated by the equipment, XML data, CSV (Comma-Separated Values, character separated value) data, JSON (JavaScript Object Notation, JavaScript Object Notation) data, detection logs and processing logs of different data structures, and data stored in the equipment database. Data of different data types may also have different data structures. In this embodiment, the data type is divided according to the data structure as the standard, and the data with the same data structure is divided into one data type. For example, for log data, logs with different data structures belong to different data types. The number of data types in the data set to be processed is determined according to the actual collected data.

[0077] In one example, images, REC files, and XML files in a production device are read through an edge server, and the read files are classified using a data classification tool. Data with the same data structure are divided into a data type, and the data set to be processed consists of the data type and the data corresponding to each data type.

[0078] In step S102, a mapping relationship table between data types and parsing rules is stored in a database. After obtaining a data set to be processed, the data types contained in the data set to be processed are determined, and the parsing rules corresponding to each data type in the data set to be processed are queried from the mapping relationship table. The mapping relationship table is a preset table, and some parsing rules corresponding to data types can be preset in the mapping relationship table in advance. When a new data type appears, the parsing rules corresponding to the new data type are added to the mapping relationship table. The parsing rules are used to parse data of data type A into data of data type B.

[0079] In one example, a mapping table between data types and parsing rules is stored in a registration center, and parsing rules corresponding to various data types are registered in the registration center. When new data types appear in actual production business, the parsing rules corresponding to the new data types can be registered in the registration center without updating the entire data processing program.

[0080] In step S103, the data of each data type in the data set to be processed is parsed into data with the same data structure, that is, the heterogeneous data is converted into homogeneous data. When converting to homogeneous data, the data with the same data structure can be any data with a high readability. The data structure of the converted homogeneous data can be determined according to actual needs. For example, the data structure of the homogeneous data is a JSON data structure, that is, the data of each data type in the data set to be processed is parsed into JSON data. Based on the parsing rules corresponding to each data type in the data set to be processed, the heterogeneous data is converted into homogeneous data.

[0081] In step S104, a real-time computing framework, such as the Flink framework, can be used to perform fusion computing on data with the same data structure by selecting the Flink SQL program, the Flink Jar program, or the Flink Scala program to obtain data processing results, and output the computing structure and display it on the production equipment. The data processing results are determined according to actual business needs, such as offset and uneven glue dispensing in the middle frame, or the yield and non-yield of the product.

[0082] In one example, Figure 2 is a schematic diagram of a data processing flow according to an exemplary embodiment. Figure 2 As shown, data is collected from the smart factory to obtain a data set to be processed, which includes heterogeneous data such as various logs, CSV files, pictures, XML files, etc. The parsing rules corresponding to each data type are obtained from the registration center. According to the parsing rules, the data in the data set to be processed is converted into homogeneous data with the same data structure. The computing resources in the edge server are used to perform data fusion operations to obtain data processing results, and the processing results are displayed on the electronic device.

[0083] In an exemplary embodiment of the present disclosure, a data set to be processed of multiple data types with different data structures and parsing rules corresponding to each data type in the data set to be processed are obtained. Based on the parsing rules, data of each data type in the data set to be processed is parsed into data with the same data structure, and then the data with the same data structure are fused to obtain a data processing result, thereby realizing the configurability of data types. When a new data type appears, it is only necessary to configure the corresponding parsing rules for it, without the need to iteratively update the data processing program, thereby simplifying the data processing process and reducing the iterative update cost.

[0084] In an exemplary embodiment of the present disclosure, a data processing method is provided. Figure 3 is a flow chart of a data processing method according to an exemplary embodiment. Figure 3 As shown, the following steps are included:

[0085] Step S301, obtaining a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures;

[0086] Step S302, obtaining the parsing rules corresponding to each data type in the data set to be processed;

[0087] Step S303, parsing data of each data type in the data set to be processed into data with the same data structure based on the parsing rule;

[0088] Step S304, obtaining association relationships between different data types in the data set to be processed;

[0089] Step S305: Based on the association relationship, data with the same data structure are merged to obtain a data processing result.

[0090] The specific implementation of steps S301 to S303 refers to steps S101 to S103 and will not be described in detail here.

[0091] In step S304, a mapping relationship table between data types and association relationships is stored in a database, and the association relationship corresponding to each data type in the data set to be processed is queried from the mapping relationship table according to the data type contained in the data set to be processed. The association relationship corresponding to the same data type may include association relationships with multiple data types. When searching for the association relationship corresponding to each data type in the data set to be processed in the mapping relationship table, it is only necessary to obtain the association relationship with other data types in the data set to be processed. For example, the association relationship of data type 1 includes: the association relationship between data type 1 and data type 2 is ralationship1, and the association relationship between data type 1 and data type 3 is ralationship2, wherein data type 1 and data type 2 are data types in the data set to be processed, then when determining the association relationship of data type 1, it is only necessary to obtain ralationship1. The mapping relationship table is a preset table, and some association relationships between different data types can be preset in the mapping relationship table in advance. When a new data type appears, the association relationship corresponding to the new data type is added to the mapping relationship table.

[0092] In one example, a mapping relationship table between data types and association relationships is stored in a registration center, and association relationships between different data types are registered in the registration center. When new data types appear in actual production business, or association relationships change, the association relationships between the new data types and other data types are registered in the registration center, or the changed association relationships are modified, without updating the entire data processing program.

[0093] In step S305, the homogeneous data is fused and calculated based on the association relationship to avoid data islands. The association relationship is simultaneously input into the real-time calculation framework, and the data of different data types with the same data structure are fused and calculated according to the association relationship between different data types to obtain data processing results.

[0094] In this embodiment, the configurable association relationship between data types is achieved. When a new data type appears or the association relationship between data types changes, it is only necessary to configure the corresponding association relationship for the new data type or update the association relationship. There is no need to iteratively update the data processing program, which simplifies the data processing process and reduces the iterative update cost.

[0095] In an exemplary embodiment of the present disclosure, a data processing method is provided. Figure 4 is a flow chart of a data processing method according to an exemplary embodiment. Figure 4 As shown, the following steps are included:

[0096] Step S401, obtaining a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures;

[0097] Step S402, determining parsing rules corresponding to different data types according to the data structures of data of different data types;

[0098] Step S403, storing the parsing rules and data types in a database accordingly;

[0099] Step S404, obtaining the parsing rules corresponding to each data type in the data set to be processed from the database;

[0100] Step S405, parsing the data of each data type in the data set to be processed into data with the same data structure based on the parsing rule;

[0101] Step S406, storing data having the same data structure into a message queue;

[0102] Step S407, storing the association relationship between different data types in a database;

[0103] Step S408, obtaining from the database the association relationship between each data type and other data types in the data set to be processed;

[0104] Step S409, based on the association relationship, obtain data with the same data structure from the message queue, use the preset computing framework to merge the data with the same data structure, and obtain the data processing result.

[0105] The specific implementation of step S401 and step S405 refer to step S301 and step S303, which will not be repeated here.

[0106] In steps S402-S404, the parsing rules corresponding to the various data types are determined based on the data structure of the common data types, and the parsing rules and the data types are stored in a database accordingly. After determining the data types contained in the data set to be processed, the parsing rules corresponding to each data type in the data set to be processed are searched from the database.

[0107] In one example, for the convenience of searching, when storing data types in the database, each data type corresponds to a unique identifier, and the data type in the data set to be processed is determined, and the parsing rules are searched according to the identifier corresponding to the data type. When storing parsing rules in the database, in order to avoid confusion, the data structure before parsing and the data structure after parsing are clearly defined at the same time. For example, the data structure before parsing is CSV, and the data structure after parsing is JSON. The parsing rules are stored as CSV data structure parsed as JSON data structure.

[0108] In step S406, data with the same data structure is stored in the message queue instead of in the database in order to achieve streaming computing, ensure the timeliness of fusion computing, realize instant analysis, improve computing efficiency, and help staff to find problems in production in a timely manner based on computing results. Kafka can be used as the message queue.

[0109] In step S407-step S408, the association relationship between multiple data types is determined based on the data structure of common data types, and the association relationship and data type are stored in the database accordingly. After the data type contained in the data set to be processed is determined, the association relationship between each data type and other data types in the data set to be processed is searched from the database.

[0110] In step S409, the preset computing framework is, for example, the Flink real-time computing framework, which obtains data with the same data structure from the message queue, inputs the data and the association relationship into the Flink jar program for fusion operation, and outputs the data processing result.

[0111] In some implementations, when the data set to be processed includes image data, the image data is parsed into binary stream data, and data having the same data structure is merged with the binary stream data to obtain a data processing result.

[0112] When the data set to be processed includes image data, the image data is stored in HDFS (Hadoop Distributed File System). After obtaining the association relationship, the data with the same data structure is obtained from the message queue, the image data is obtained from HDFS, and it is parsed into binary stream data. The data with the same data structure and the binary stream data are input into the preset computing framework for fusion computing.

[0113] In an exemplary embodiment of the present disclosure, a data processing system is provided. Figure 5 is a block diagram of a data processing system according to an exemplary embodiment. Figure 5 As shown, the data processing system includes:

[0114] The data acquisition module 501 is configured to obtain a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures;

[0115] The data parsing module 502 is configured to obtain the parsing rules corresponding to each data type in the data set to be processed from the first registration center module 503; based on the parsing rules, parse the data of each data type in the data set to be processed into data with the same data structure;

[0116] The calculation module 504 is configured to merge data with the same data structure to obtain a data processing result.

[0117] In an exemplary embodiment, the first registry module 503 is configured to:

[0118] Determine the parsing rules corresponding to different data types according to the data structure of data of different data types;

[0119] The parsing rules and data types are stored in the database accordingly.

[0120] In an exemplary embodiment, the system further comprises:

[0121] The data storage module 505 is configured to store data having the same data structure into the message queue.

[0122] In an exemplary embodiment, the system further comprises:

[0123] The data extraction module 506 is configured to obtain data with the same data structure from the data storage module.

[0124] In an exemplary embodiment, the system further comprises:

[0125] The second registration center module 507 is configured to store the association relationship between different data types in a database.

[0126] In an exemplary embodiment, the calculation module 504 is further configured to:

[0127] Obtaining associations between different data types in the data set to be processed from the second registration center module 507;

[0128] Based on the association relationship, data with the same data structure are merged.

[0129] In an exemplary embodiment of the present disclosure, based on the above data processing system, Figure 6 is a schematic diagram showing a data processing flow according to an exemplary embodiment. Figure 6 As shown, including:

[0130] S1, the data collection module 501 collects the data set to be processed from the smart factory through the Python data collection program;

[0131] S2, the data parsing module 502 obtains the parsing rules corresponding to each data type in the to-be-processed data set from the first registration center module 503, and parses the heterogeneous data into homogeneous Json data through the Python data parsing program based on the parsing rules;

[0132] The first registration center module is implemented by a web-side microservice. In one example, Fig. 7A is a schematic diagram of a first registration center module according to an exemplary embodiment. Fig. 7A As shown, the first registration center module 503 is a data type registration center, where users register data types and parsing rules, data types file_type and parsing rules analyze_type are stored correspondingly, data type file_type includes field id and field file_type, field id represents data type identification, field file_type represents data type name; parsing rule analyze_type includes field file_type, field data_type, field json_key and field index, field file_type represents data type name before parsing, field data_type represents data type name after parsing, field json_key represents keywords of data before parsing, field index represents index, wherein field json_key is set according to data content before parsing, for example, when the data type before parsing is fault log, json_key includes fault location, fault identification, fault time, etc., and field index is used to find corresponding parsing rules. After registration is completed, data parsing module 502 queries corresponding parsing rules from data type registration center according to data type.

[0133] S3, the data storage module 505 includes HDFS and Kafka, stores the homogeneous Json data in the message queue Kafka, and stores source files such as pictures, CSV files, and XML files in HDFS;

[0134] S4, the data extraction module 506 extracts data from Kafka in real time through the Java / Sacla program, extracts pictures from HDFS, and converts the pictures into binary stream data. In addition, the data extraction module 506 queries the association relationship between data types from the second registration center module 507;

[0135] The second registration center module is implemented by a web-side microservice. In one example, Figure 7B is a schematic diagram of a second registration center module according to an exemplary embodiment. Figure 7B As shown, the second registration center module 507 is an association registration center, in which users register associations between data types. The data type file_type and the parsing rule associate_ralationship are stored correspondingly. The data type file_type includes fields id and file_type. The field id represents the data type identifier, and the field file_type represents the data type name; the parsing rule associate_ralationship includes fields file_type1, file_type2, key1, key2, ralationship1, and ralationship2. The field file_type1 represents the name of the data type 1, and the field file_type2 represents the name of the data type 2. The field key1 represents the keyword 1 shared by the data type 1 and the data type 2, and the field key2 represents the keyword 2 shared by the data type 1 and the data type 2. The field ralationship1 represents the association relationship between the keyword 1 in the data type 1 and the keyword 1 in the data type 2, and the field ralationship2 represents the association relationship between the keyword 2 in the data type 1 and the keyword 2 in the data type 2. For example, if the keyword 1 in the data type 1 and the keyword 1 in the data type 2 are equal, the field ralationship1 is "equal". After the registration is completed, the data extraction module 506 queries the corresponding association relationship from the association relationship registration center according to the data type.

[0136] S5, the computing module 504 performs fusion computing through the Flink real-time computing framework and outputs the data processing results.

[0137] In an exemplary embodiment of the present disclosure, a data processing device is provided. Figure 8is a block diagram of a data processing device according to an exemplary embodiment. Figure 8 As shown, the data processing device includes:

[0138] A first acquisition unit 801 is configured to acquire a data set to be processed, where the data set to be processed includes data of multiple data types with different data structures;

[0139] The second acquisition unit 802 is configured to acquire the parsing rules corresponding to each data type in the data set to be processed;

[0140] The parsing unit 803 is configured to parse the data of each data type in the to-be-processed data set into data with the same data structure based on the parsing rule;

[0141] The computing unit 804 is configured to fuse data with the same data structure to obtain a data processing result.

[0142] In an exemplary embodiment, the data processing device further includes a storage unit 805 configured to:

[0143] Determine the parsing rules corresponding to different data types according to the data structure of data of different data types;

[0144] The parsing rules and data types are stored in the database accordingly.

[0145] In an exemplary embodiment, the second acquiring unit 802 is further configured to:

[0146] Obtain the parsing rules corresponding to each data type in the data set to be processed from the database.

[0147] In an exemplary embodiment, the computing unit 804 is further configured to:

[0148] Obtain the association relationship between different data types in the data set to be processed;

[0149] Based on the association relationship, data with the same data structure are merged.

[0150] In an exemplary embodiment, the storage unit 805 is further configured to:

[0151] Store the associations between different data types in the database.

[0152] In an exemplary embodiment, the computing unit 804 is further configured to:

[0153] Obtain the association relationship between each data type and other data types in the data set to be processed from the database.

[0154] In an exemplary embodiment, the storage unit 805 is further configured to:

[0155] Store data with the same data structure into the message queue.

[0156] In an exemplary embodiment, the computing unit 804 is further configured to:

[0157] Get data with the same data structure from the message queue, and use the preset computing framework to merge the data with the same data structure.

[0158] In an exemplary embodiment, the computing unit 804 is further configured to:

[0159] When the data set to be processed includes image data, the image data is parsed into binary stream data.

[0160] The data with the same data structure are merged with the binary stream data to obtain the data processing result.

[0161] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0162] Fig. 9 is a block diagram of an electronic device 900 according to an exemplary embodiment.

[0163] Reference Fig. 9 , the electronic device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0164] The processing component 902 generally controls the overall operation of the electronic device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0165] The memory 904 is configured to store various types of data to support operations on the electronic device 900. Examples of such data include instructions for any application or method operating on the electronic device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0166] The power supply component 906 provides power to the various components of the electronic device 900. The power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 900.

[0167] The multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0168] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the electronic device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0169] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0170] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the electronic device 900. For example, the sensor assembly 914 can detect the open / closed state of the electronic device 900, the relative positioning of components, such as the display and keypad of the electronic device 900, and the sensor assembly 914 can also detect the position change of the electronic device 900 or a component of the electronic device 900, the presence or absence of user contact with the electronic device 900, the orientation or acceleration / deceleration of the electronic device 900, and the temperature change of the electronic device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0171] The communication component 916 is configured to facilitate wired or wireless communication between the electronic device 900 and other devices. The electronic device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0172] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0173] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by a processor 920 of an electronic device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0174] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform a data processing method, the method including any of the above methods.

[0175] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0176] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures; Obtaining parsing rules corresponding to each data type in the data set to be processed; Based on the parsing rule, the data of each data type in the to-be-processed data set is parsed into data having the same data structure; The data with the same data structure are merged to obtain a data processing result.

2. The data processing method according to claim 1, characterized in that: The method further comprises: Determine the parsing rules corresponding to different data types according to the data structure of data of different data types; The parsing rules and the data types are stored in a database in correspondence.

3. The data processing method according to claim 2, characterized in that: The obtaining of the parsing rules corresponding to each data type in the to-be-processed data set includes: Obtain from the database the parsing rules corresponding to each data type in the data set to be processed.

4. The data processing method according to claim 1, characterized in that: The fusing the data having the same data structure comprises: Obtaining associations between different data types in the data set to be processed; Based on the association relationship, the data with the same data structure are merged.

5. The data processing method according to claim 4, characterized in that: The method further comprises: Store the associations between different data types in the database.

6. The data processing method according to claim 5, characterized in that: The obtaining of associations between different data types in the data set to be processed includes: The association relationship between each data type and other data types in the data set to be processed is obtained from the database.

7. The data processing method according to claim 1, characterized in that: After parsing the data of each data type in the to-be-processed data set into data having the same data structure, the method further comprises: The data having the same data structure is stored in a message queue.

8. The data processing method according to claim 7, characterized in that: The fusing the data having the same data structure comprises: The data with the same data structure is obtained from the message queue, and the data with the same data structure is merged using a preset computing framework.

9. The data processing method according to claim 1, characterized in that: The method further comprises: When the data set to be processed includes image data, parsing the image data into binary stream data, The data having the same data structure is merged with the binary stream data to obtain a data processing result.

10. A data processing system, characterized in that: The system comprises: A data acquisition module is configured to obtain a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures; The data parsing module is configured to obtain the parsing rules corresponding to each data type in the data set to be processed from the first registration center module; based on the parsing rules, parse the data of each data type in the data set to be processed into data with the same data structure; The computing module is configured to fuse the data having the same data structure to obtain a data processing result.

11. The data processing system according to claim 10, characterized in that: The first registration center module is configured as follows: Determine the parsing rules corresponding to different data types according to the data structure of data of different data types; The parsing rules and the data types are stored in a database in correspondence.

12. The data processing system according to claim 10, characterized in that: The system further comprises: The data storage module is configured to store the data having the same data structure in a message queue.

13. The data processing system according to claim 12, characterized in that: The system further comprises: The data extraction module is configured to obtain the data with the same data structure from the data storage module.

14. The data processing system according to claim 10, characterized in that: The system further comprises: The second registration center module is configured to store the association relationship between different data types in a database.

15. The data processing system according to claim 14, characterized in that: The computing module is further configured to: Acquire, from the second registration center module, association relationships between different data types in the to-be-processed data set; Based on the association relationship, the data with the same data structure are merged.

16. A data processing device, characterized in that: The device comprises: A first acquisition unit is configured to acquire a data set to be processed, wherein the data set to be processed includes data of multiple data types with different data structures; A second acquisition unit is configured to acquire a parsing rule corresponding to each data type in the data set to be processed; A parsing unit, configured to parse data of each data type in the to-be-processed data set into data having the same data structure based on the parsing rule; The computing unit is configured to fuse the data having the same data structure to obtain a data processing result.

17. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method as claimed in any one of claims 1 to 9.

18. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 9.

Images