A data analysis method, device, equipment and medium

Abstract

The embodiment of the present invention discloses a data parsing method, device, equipment and medium. The method comprises: determining a target parsing file from candidate parsing files according to a user's selection operation; parsing the data to be parsed according to the candidate structure in the target parsing file and the attribute information of the candidate data field in the candidate structure; wherein the attribute information comprises data length and interpretation information. The embodiment of the present invention achieves the effect of improving the efficiency of data parsing and increasing the flexibility of data parsing.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a data analysis method, device, equipment and medium. Background Art

[0002] CTCS (Chinese Train Control System) is a train control system developed by my country to realize the digitalization, networking, intelligence and integration of railway signal systems and achieve the high-speed rail strategy. CTCS controls trains by exchanging data with trains.

[0003] Before CTCS is officially delivered, it is necessary to test CTCS, that is, the technical staff will judge the running status of CTCS based on the parsed interactive data. Currently, the parsing of interactive data is mostly implemented through the logical structure in the code, which has low parsing efficiency and poor flexibility. Summary of the invention

[0004] The embodiments of the present application disclose a data parsing method, device, equipment and medium to solve the problem of low parsing efficiency when currently parsing data.

[0005] In a first aspect, an embodiment of the present invention provides a data parsing method, the method comprising:

[0006] According to the user's selection operation, determine the target parsing file from the candidate parsing files;

[0007] The data to be parsed is parsed according to the candidate structures in the target parsing file and the attribute information of the candidate data fields in the candidate structures; wherein the attribute information includes data length and interpretation information.

[0008] In a second aspect, an embodiment of the present invention provides a data parsing device, the device comprising:

[0009] A target parsing file determination module is used to determine a target parsing file from candidate parsing files according to a user's selection operation;

[0010] The data parsing module is used to parse the data to be parsed according to the candidate structures in the target parsing file and the attribute information of the candidate data fields in the candidate structures; wherein the attribute information includes data length and interpretation information.

[0011] In a third aspect, an embodiment of the present invention provides a device, the device comprising:

[0012] one or more processors;

[0013] a storage device for storing one or more programs,

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the data parsing method as described in any one of the embodiments of the present invention.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements a data parsing method as described in any one of the embodiments of the present invention.

[0016] The embodiments of the present invention achieve the effect of improving the efficiency of data analysis and increasing the flexibility of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0018] Figure 1A A flowchart of a data analysis method provided in Embodiment 1 of the present invention;

[0019] Figure 1B A schematic diagram of data transmission provided in Embodiment 1 of the present invention;

[0020] Figure 1C A schematic diagram of obtaining data to be parsed provided in the first embodiment of the present invention;

[0021] Figure 2A A flowchart of a data analysis method provided in Embodiment 2 of the present invention;

[0022] Figure 2B A schematic diagram of a structure orientation provided in Embodiment 2 of the present invention;

[0023] Figure 2C A schematic diagram of data analysis provided in the second embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of a data analysis device provided in Embodiment 3 of the present invention;

[0025] Figure 4 A schematic diagram of the structure of a device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0026] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only structures related to the embodiments of the present invention are shown in the accompanying drawings, rather than all structures.

[0027] During the research and development process, the applicant discovered that CTCS can be divided into 5 levels according to functions in the technical specifications, among which the CTCS-3 level train control system is a train control system standard with independent intellectual property rights that my country has developed by introducing, digesting and absorbing ETCS (European Train Control System) and combining its own national conditions. RBC (Radio Block Center) is the most critical equipment in the ground equipment of the CTCS-3 level train control system. It generates safe train control commands based on the line information and train position information obtained through interaction with other external ground equipment systems, calculates MA (Movement Authority), and transmits it to the train through the GSM-R (GSM For Railway) wireless network, so that the RBC controls the train to run safely and reliably within its coverage area.

[0028] As a safety device with SIL (Safety Integrity Level) of SIL4, RBC needs to undergo rigorous testing before the official product is delivered for normal use. During the testing process, a large number of test cases will be used to cover all usage scenarios. For the result of each test case, the tester will make a judgment based on the DMI (Driver Machine Interface) display and the actual operation of the train. If the test result is inconsistent with the expectation, the tester will further analyze the phenomenon.

[0029] DMI displays the analysis data of the interaction data between RBC and train ATP (Automatic Train Protection) and the tester judges the operation status of RBC based on the analysis data. Currently, the analysis of interaction data is mostly implemented through the logical structure in the code, which has low analysis efficiency and poor flexibility, which indirectly affects the test efficiency of the entire test process.

[0030] Embodiment 1

[0031] Figure 1AThis is a flow chart of a data analysis method provided in the first embodiment of the present invention. This embodiment is applicable to the case of analyzing the interactive data between RBC and ATP. The method can be executed by the data analysis device provided in the embodiment of the present invention, and the device can be implemented by software and / or hardware. Figure 1A As shown, the method may include:

[0032] S101. Determine a target parsing file from candidate parsing files according to a user's selection operation.

[0033] Among them, the candidate parsing file is written by relevant technical personnel according to a special format, and is used as a configuration file for parsing the data to be parsed, which is usually an XML format file. In addition, for different communication protocols used in different scenarios, corresponding candidate parsing files are written according to the data transmission format specified by each communication protocol. Users can determine the target parsing file that conforms to the communication protocol used in the current scenario from the candidate parsing file.

[0034] In one embodiment, the user selects at least one candidate parsing file included in the parsing file folder through the front-end interface according to the communication protocol adopted by the data to be parsed, wherein each candidate parsing file is distinguished by the file name as an identifier. According to the user's selection operation on any candidate parsing file, the candidate parsing file is used as the target parsing file. In this embodiment, the data to be parsed can be a data frame in any scenario, and this embodiment does not impose any restrictions on the data content of the data to be parsed. Optionally, the data to be parsed includes the interaction data between RBC and ATP. To facilitate the understanding of this solution, the data to be parsed will be regarded as the interaction data between RBC and ATP by default in the future.

[0035] A communication module, such as a GSM-R unit, is provided in the RBC device for wireless communication with the ATP in the train. The communication unit is connected and communicated with the communication unit in the RBC host of the RBC device, such as the C machine, and then the communication unit transmits the received data to the operation unit in the RBC host, such as the A / B machine, for logical processing based on the received data.

[0036] Figure 1B A schematic diagram of data transmission provided in Embodiment 1 of the present invention is shown in FIG. Figure 1BAs shown, data is exchanged between the ATP 101 in the train 100 and the GSM-R unit 102 in the RBC device, and the GSM-R unit 102 sends the received data to the C machine 104 in the RBC host 103, and the C machine 104 then passes the data to the A / B machine 105 for logical processing. In this implementation, only the GSM-R unit 102 is used as a communication module, the C machine 104 is used as a communication unit, and the A / B machine 105 is used as an operation unit for explanation, and the specific types of the communication module, communication unit and operation unit are not limited. All modules / units that can play the same role are within the protection scope of this embodiment.

[0037] Since the data processing is only performed by the operation unit in the RBC device, and the communication unit and the communication module are only responsible for the transmission of data, and the data does not change during the period, the data transmitted between the communication module and the operation unit is used as the data to be parsed in this embodiment. In other words, the data to be parsed is the train operation data sent by the ATP to the RBC device, or the train control data sent by the RBC device to the ATP.

[0038] S102, parsing the data to be parsed according to the candidate structures in the target parsing file and the attribute information of the candidate data fields in the candidate structures; wherein the attribute information includes data length and interpretation information.

[0039] Among them, any candidate parsing file includes at least one candidate structure, and each candidate structure includes at least one candidate data field. The role of the candidate structure is to carry the candidate data field, and to make an aggregated organizational structure for each candidate data field, which can increase the readability, logic and reusability of the candidate parsing file. Each candidate structure has a unique identifier for indexing it. For example, the candidate structure identified as "intrinsic variable in the message header" can include two candidate data fields, "vehicle timestamp" and "vehicle equipment identification number". The role of the candidate data field is data interpretation, and the candidate data field is pre-assigned with two attribute information, "data length" and "interpretation information". "Data length" is used to calibrate the data length of the corresponding candidate data field in the data to be parsed, so as to intercept the data according to the data length, and "interpretation information" is used to mark the intercepted data to ensure that the user knows the meaning of each intercepted data.

[0040] In one embodiment, a predetermined starting structure is determined from a target parsing file, and the data to be parsed is parsed according to the arrangement order of the candidate data fields in the starting structure and the pointing relationship between each candidate data field and other candidate structures.

[0041] Specifically, determine whether the first-order candidate data field in the starting structure points to any other candidate structure. If it does not point to any other candidate structure, data is intercepted from the header of the data to be parsed according to the data length of the first-order candidate data field, and the intercepted data within the range is marked according to the interpretation information of the first-order candidate data field to generate parsed data corresponding to the first-order candidate data field.

[0042] If it points to any other candidate structure, the data to be parsed is parsed according to the arrangement order of the candidate data fields in the other candidate structure, and the data length and interpretation information corresponding to each candidate data field.

[0043] When the parsing is completed, the starting structure is returned to determine whether the second-order candidate data field in the starting structure points to any other candidate structure. If it does not point to any other candidate structure, data is intercepted from the head of the remaining data to be parsed according to the data length of the second-order candidate data field, and the intercepted data within the range is marked according to the interpretation information of the second-order candidate data field to generate the parsed data corresponding to the second-order candidate data field.

[0044] If it points to any other candidate structure, the data to be parsed is parsed according to the arrangement order of the candidate data fields in the other candidate structure, and the data length and interpretation information corresponding to each candidate data field.

[0045] By analogy, according to the above implementation, all candidate data fields in the starting structure are traversed in sequence, thereby completing the parsing of the data to be parsed.

[0046] Exemplarily, it is assumed that the starting structure includes candidate data field 1, candidate data field 2 and candidate data field 3, candidate data field 2 points to other structure 1, and other structure 1 includes candidate data field 4 and candidate data field 5. The "data length" of candidate data field 1 is "1 bit", and the "interpretation information" is "timestamp"; the "data length" of candidate data field 3 is "2 bits", and the "interpretation information" is "line description"; the "data length" of candidate data field 4 is "1 bit", and the "interpretation information" is "device identification number"; the "data length" of candidate data field 5 is "2 bits", and the "interpretation information" is "message type".

[0047] The parsing process of the data to be parsed is as follows: if candidate data field 1 does not point to any other structure, then 0 to 1 bits of the data to be parsed are intercepted, and "timestamp" is used as the label of the intercepted data to generate the parsed data corresponding to candidate data field 1. If candidate data field 2 points to other structure 1, then 1 to 2 bits of the data to be parsed are intercepted, and "device identification number" is used as the label of the intercepted data to generate the parsed data corresponding to candidate data field 4, and then 2 to 4 bits of the data to be parsed are intercepted, and "message type" is used as the label of the intercepted data to generate the parsed data corresponding to candidate data field 5. If candidate data field 3 does not point to any other structure, then 4 to 6 bits of the data to be parsed are intercepted, and "line description" is used as the label of the intercepted data to generate the parsed data corresponding to candidate data field 3. The above completes the data parsing of the data to be parsed.

[0048] The interactive data between the RBC device and the ATP is grouped according to a proprietary communication protocol. The communication protocols used in the direction from ground to vehicle (ATP sends data to the RBC device) and the direction from vehicle to ground (RBC device sends data to ATP) are different. The existing technology performs data parsing through the logical structure in the code, and the parsing efficiency is low and the flexibility is poor. The technical solution provided by the embodiment of the present invention determines the target parsing file from the candidate parsing file according to the user's selection operation, and parses the data to be parsed according to the candidate structure in the target parsing file and the attribute information of the candidate data field in the candidate structure; wherein the attribute information includes data length and interpretation information. Since the data length and interpretation information have been configured in advance for the candidate data field of each candidate structure in the candidate parsing file, when the user selects the target parsing file according to the needs, the file to be parsed is directly parsed according to the data length and interpretation information of the candidate data field of each candidate structure in the target parsing file, which greatly improves the efficiency of data parsing; and the user can select different target parsing files to parse the file to be parsed according to the needs, which increases the flexibility of data parsing.

[0049] This embodiment also provides an implementation method for obtaining the data to be parsed. The data transmission between the communication module and the communication unit in the RBC device is usually implemented based on a switch. Take the communication module as a GSM-R unit, the communication unit as a C machine, and the computing unit as an A / B machine as an example. Since the A / B machine and the C machine are board structures, they are connected by the backplane inside the cage and have no external interface. The hardware interface between the GSM-R unit and the C machine is an RJ-45 Ethernet interface. Since the C machine is a communication unit, the data that communicates with the host in the entire RBC device is transmitted to the A / B machine through it. Therefore, the GSM-R unit and the C machine are not directly connected, but communicate data with the C machine through the switch like other modules.

[0050] Based on the above connection relationship, this embodiment uses the switch mirroring method to capture the data to be parsed between the GSM-R unit and the C machine. Figure 1C A schematic diagram of obtaining data to be parsed provided in the first embodiment of the present invention, such as Figure 1C As shown, the GSM-R unit 106 sends data to the C machine 108 and the remaining units 109 in the RBC device through the switch 107. In this embodiment, the GSM-R unit 106 is used as the mirror source, and the data of the GSM-R unit 106 is sent to the mirror destination target interface 110 through the switch 107 to obtain the data to be parsed.

[0051] By adopting the switch mirroring method to obtain the data to be parsed, the hardware environment of the existing RBC device is not damaged, and no external information is injected, which will not affect the function of the RBC device.

[0052] Embodiment 2

[0053] Figure 2A This is a flow chart of a data analysis method provided in Embodiment 2 of the present invention. This embodiment is optimized based on the above optional embodiments, such as Figure 2A As shown, the method may include:

[0054] S201. Remove network layer data and link layer data from original data to obtain secure data.

[0055] The original data is the collection of all data transmitted during the data transmission process.

[0056] In one embodiment, since the data to be parsed is the application layer data in the original data, it is necessary to remove the non-application data in the original data that has no practical meaning before parsing the data to be parsed. After receiving the original data, it is determined whether the original data contains application layer data. If the original data does not contain application layer data, the original data is directly discarded. For example, if the original data is ACK (Acknowledgement, confirmation message), the original data is directly discarded; if the original data contains application layer data, the network layer data and link layer data in the original data are removed, and the remaining data is used as security data.

[0057] S202: When the security data complies with a preset security protocol, remove the security layer data in the security data and use the remaining data as data to be parsed.

[0058] Among them, the preset security protocol is a protocol for ensuring the security of communication. Taking the communication between RBC equipment and ATP as an example, the preset security protocol may optionally include RSSP-II (Railway Security Communication Protocol-II) protocol.

[0059] In one implementation, a protocol check is performed on the security data to determine whether the security data complies with a preset security protocol. If it is determined that the security data does not comply with the preset security protocol, the security data is directly discarded. Since the security protocol ensures communication security by overwriting the application layer data with the security layer data, if it is determined that the security data complies with the preset security protocol, the security layer data in the security data is removed, and the remaining application layer data is used as the data to be parsed.

[0060] S203: Determine a target parsing file from candidate parsing files according to a selection operation of the user.

[0061] S204: Determine a first structure from the candidate structures in the target parsing file, and use each candidate data field in the first structure as a first data field, and parse the data to be parsed in sequence according to the order of each first data field and the attribute information of each first data field.

[0062] The first structure represents a predetermined initial structure.

[0063] In one implementation, the data to be parsed is intercepted in sequence according to the data length of each first data field and the order of each first data field, and the intercepted data is marked according to the interpretation information of each first data field, so as to obtain the parsed data corresponding to each first data field.

[0064] Exemplarily, it is assumed that the first structure includes the first data field A, the first data field B and the first data field C, and the "data length" of the first data field A is "1 bit" and the "interpretation information" is "timestamp"; the "data length" of the first data field B is "2 bits" and the "interpretation information" is "line description"; the "data length" of the first data field C is "1 bit" and the "interpretation information" is "device identification number". Intercept 0 to 1 bits of data to be parsed, and use "timestamp" as the annotation of the intercepted data to generate the parsed data corresponding to the first data field A. Intercept 1 to 3 bits of data to be parsed, and use "line description" as the annotation of the intercepted data to generate the parsed data corresponding to the first data field B. Intercept 3 to 4 bits of data to be parsed, and use "device identification number" as the annotation of the intercepted data to generate the parsed data corresponding to the first data field C.

[0065] Optionally, the step of “parse the to-be-parsed data in sequence according to the order of the first data fields and the attribute information of the first data fields” in S204 includes the following steps:

[0066] Determine whether the current first data field points to a second structure; wherein the second structure is any candidate structure other than the first structure; when not pointing to the second structure, intercept the remaining data of the data to be parsed according to the data length of the current first data field to obtain a target data field; and mark the target data field according to the interpretation information as parsed data corresponding to the current first data field.

[0067] In one embodiment, the first data field may point to the second structure. When the current first data field does not point to the second structure, the remaining data to be parsed is intercepted according to the data length of the current first data field, and the intercepted data is annotated according to the interpretation information to generate the parsed data corresponding to the current first data field.

[0068] Optionally, after determining whether the current first data field points to the second structure, the method further includes:

[0069] When pointing to the second structure, each candidate data field in the second structure is used as the second data field; and the remaining data of the data to be parsed are parsed in sequence according to the order of each second data field and the attribute information of each second data field.

[0070] In one embodiment, if the current first data field points to the second structure, the remaining data to be parsed is intercepted in sequence according to the data length of each second data field and the order of each second data field, and the intercepted data is marked according to the interpretation information of each second data field, so as to obtain the parsed data corresponding to each second data field.

[0071] Optionally, according to the order of each of the second data fields and the attribute information of each of the second data fields, the remaining data of the data to be parsed are parsed in sequence, including the following steps:

[0072] 1) When the current second data field is the target second data field, determine whether the target second data field points to a third structure; wherein the target second data field is the second data field in the last order, and the third structure is any candidate structure except the first structure and the second structure.

[0073] 2) Without pointing to the third structure, the remaining data of the data to be parsed is parsed according to the attribute information of the target second data field, and the first data field that is located after the current first data field is used as the updated current first data field.

[0074] 3) When pointing to the third structure, each candidate data field in the third structure is used as a third data field, and the remaining data of the data to be parsed is parsed in sequence according to the order of each third data field and the attribute information of each third data field.

[0075] In one embodiment, when the target second data field does not point to the third structure, the remaining data to be parsed is intercepted according to the data length of the target second data field, and the intercepted data is marked according to the interpretation information to generate parsed data corresponding to the target second data field.

[0076] If the target second data field points to the third structure, the remaining data to be parsed is intercepted in sequence according to the data length of each third data field and the order of each third data field, and the intercepted data is marked according to the interpretation information of each third data field, so as to obtain the parsed data corresponding to each third data field.

[0077] Exemplarily, this embodiment illustrates the specific process of data parsing:

[0078] Figure 2B A schematic diagram of a structure pointing provided in the second embodiment of the present invention is shown as follows: Figure 2B As shown, the first structure 200 includes a first data field 201, a first data field 202, a first data field 203 and a first data field 204, the first data field 202 points to the second structure 205, the first data field 203 points to the second structure 206, the second structure 205 includes a second data field 207 and a second data field 208, the second structure 206 includes a second data field 209 and a second data field 210, the second data field 208 points to the third structure 211, and the third structure 211 includes a third data field 212 and a third data field 213.

[0079] according to Figure 2B The structure pointing form in the table, the corresponding data parsing process is as follows:

[0080] Figure 2C A schematic diagram of data analysis provided in the second embodiment of the present invention is shown in FIG. Figure 2C As shown, 214 represents the data to be parsed, 215 represents the corresponding data domain, and 216 represents the structure to which the data domain belongs.

[0081] If the first data field 201 in the first structure 200 does not point to the second structure, the data to be parsed 214 is intercepted according to the data length of the first data field 201, and the intercepted data is marked according to the interpretation information of the first data field 201 to obtain the parsed data 217 corresponding to the first data field 201.

[0082] The first data field 202 points to the second structure 205, then the remaining data of the data to be parsed 214 is intercepted according to the data length of the second data field 207, and the intercepted data is annotated according to the interpretation information of the second data field 207 to obtain the parsed data 218 corresponding to the second data field 207. The second data field 208 is the second data field in the last order, that is, the target second data field, and the second data field 208 points to the third structure 211, then the remaining data of the data to be parsed 214 is intercepted according to the data length of the third data field 212, and the intercepted data is annotated according to the interpretation information of the third data field 212 to obtain the parsed data 219 corresponding to the third data field 212. In order, the remaining data of the data to be parsed 214 is intercepted according to the data length of the third data field 213, and the intercepted data is annotated according to the interpretation information of the third data field 213 to obtain the parsed data 220 corresponding to the third data field 213.

[0083] Next, returning to the first structure 200, the first data field 203 points to the second structure 206, then the remaining data of the data to be parsed 214 is intercepted according to the data length of the second data field 209, and the intercepted data is annotated according to the interpretation information of the second data field 209 to obtain the parsed data 221 corresponding to the second data field 209. If the second data field 210 does not point to the third structure, then the remaining data of the data to be parsed 214 is intercepted according to the data length of the second data field 210, and the intercepted data is annotated according to the interpretation information of the second data field 210 to obtain the parsed data 222 corresponding to the second data field 210.

[0084] Next, the first structure 200 is returned. If the first data field 204 does not point to the second structure, the remaining data of the data to be parsed 214 is intercepted according to the data length of the first data field 204, and the intercepted data is annotated according to the interpretation information of the first data field 204 to obtain the parsed data 223 corresponding to the first data field 204.

[0085] The above completes the entire data parsing process. It is worth noting that each third data field in the third structure can continue to point to other structures, and the process is the same as above, which will not be elaborated in this embodiment.

[0086] S205. Determine parsed data according to the parsing result, and store the parsed data and the original data according to a preset storage path; wherein the preset storage path includes at least one of a local terminal and a database.

[0087] In one implementation, the set of parsed data corresponding to each data field is used as parsed data corresponding to the data to be parsed. After the parsed data is obtained, the parsed data and the original data are stored in a local terminal or a database.

[0088] Specifically, if the preset storage path is the local end, the parsed data and the original data will be recorded in the TXT log file, and each log file also includes additional content for easy screening, such as the log file generation time, the original data acquisition time, and the IP address from which the original data was acquired. The log file can be configured as a time mode or a capacity mode. In the time mode, a new log file will be generated when the set time is reached. In the capacity mode, a new log file will be generated for recording when the log text storage size reaches the set storage capacity.

[0089] If the preset storage path is a database, the parsed data and the original data are converted into json format and stored in the corresponding database. The database is based on the RBC device, that is, the parsed data and the original data of each RBC device are stored in a corresponding database. Among them, the database can be a mongodb database, and the information storage mode is pre-enabled.

[0090] S206: Visually display the parsed data and the original data according to the user's query instruction.

[0091] In one implementation, a user generates a query instruction according to actual business needs to view stored data.

[0092] Specifically, the query includes but is not limited to the following three forms:

[0093] A. Real-time query

[0094] The raw data acquisition and data analysis have a front-end interface to display data records. Testers can refresh and view the latest raw data and analyzed data through the list display on the interface. At the same time, the list interface has a certain cache capacity, and testers can scroll back to view a certain number of recorded messages.

[0095] B. Log query

[0096] Query and edit the corresponding log files on the local side.

[0097] C. Database query

[0098] This mode is implemented through springboot. The backend service obtains the data to be displayed by querying the database. It can display the latest raw data and parsed data in real time according to the update of the database. At the same time, the data can be further filtered or displayed graphically through human-computer interaction on the front end of the web page. This mode is more traversal than local query for data query and filtering. It can query all the saved data in the database according to time, feature value, etc. In addition, this mode can be viewed and used by multiple people in multiple locations at the same time, which is convenient for collaborative problem analysis.

[0099] The technical solution provided by the embodiment of the present invention obtains security data by removing network layer data and link layer data from original data, and removes security layer data from security data when the security data complies with a preset security protocol, and uses the remaining data as data to be parsed, thereby achieving the effect of data filtering and security verification of original data; by determining a first structure from candidate structures, and using each candidate data field in the first structure as a first data field, and parsing the data to be parsed in sequence according to the order of each first data field and the attribute information of each first data field, thereby achieving the effect of improving the efficiency of data parsing and increasing the flexibility of data parsing; by determining parsed data according to the parsing result, and storing the parsed data and original data according to a preset storage path, thereby achieving the effect of data storage, and laying a foundation for subsequent data query; by visually displaying the parsed data and original data according to the user's query instruction, the user's data query needs are met and the user experience is improved.

[0100] Embodiment 3

[0101] Figure 3 This is a schematic diagram of the structure of a data analysis device provided in the third embodiment of the present invention, which can execute a data analysis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 3 As shown, the device may include:

[0102] A target parsing file determining module 31 is used to determine a target parsing file from candidate parsing files according to a user's selection operation;

[0103] The data parsing module 32 is used to parse the data to be parsed according to the candidate structures in the target parsing file and the attribute information of the candidate data fields in the candidate structures; wherein the attribute information includes data length and interpretation information.

[0104] Based on the above embodiment, the data parsing module 32 is specifically used for:

[0105] Determine a first structure from the candidate structures, and use each candidate data field in the first structure as a first data field;

[0106] The data to be parsed are parsed in sequence according to the order of the first data fields and the attribute information of the first data fields.

[0107] Based on the above embodiment, the data parsing module 32 is further used for:

[0108] Determine whether the current first data field points to a second structure; wherein the second structure is any candidate structure except the first structure;

[0109] In the case of not pointing to the second structure, according to the data length of the current first data field, data interception is performed on the remaining data of the data to be parsed to obtain a target data field;

[0110] The target data field is annotated according to the interpretation information as parsed data corresponding to the current first data field.

[0111] Based on the above embodiment, the device further includes a second data domain determination module, which is specifically used to:

[0112] In the case of pointing to the second structure, taking each candidate data field in the second structure as the second data field;

[0113] According to the order of each of the second data fields and the attribute information of each of the second data fields, the remaining data of the data to be parsed are parsed in sequence.

[0114] Based on the above embodiment, the second data domain determination module is further configured to:

[0115] In the case where the current second data field is the target second data field, determining whether the target second data field points to a third structure; wherein the target second data field is the second data field in the last order, and the third structure is any candidate structure except the first structure and the second structure;

[0116] In the case of not pointing to the third structure, the remaining data of the data to be parsed is parsed according to the attribute information of the target second data field, and the first data field located after the current first data field is used as the updated current first data field;

[0117] When pointing to the third structure, each candidate data field in the third structure is used as a third data field, and the remaining data of the data to be parsed is parsed in sequence according to the order of each third data field and the attribute information of each third data field.

[0118] On the basis of the above embodiment, the device further includes a module for obtaining data to be analyzed, which is specifically used for:

[0119] Remove the network layer data and link layer data from the original data to obtain secure data;

[0120] In the case where the security data complies with the preset security protocol, the security layer data in the security data is removed, and the remaining data is used as the data to be parsed.

[0121] On the basis of the above embodiment, the device further includes a data storage module, which is specifically used for:

[0122] The parsed data is determined according to the parsing result, and the parsed data and the original data are stored according to a preset storage path; wherein the preset storage path includes at least one of a local end and a database.

[0123] On the basis of the above embodiment, the device further includes a data query module, which is specifically used to:

[0124] The parsed data and the original data are visualized according to the user's query instruction.

[0125] A data parsing device provided in an embodiment of the present invention can execute a data parsing method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the instruction validity verification method provided in any embodiment of the present invention.

[0126] Embodiment 4

[0127] Figure 4 A schematic diagram of the structure of a device provided in Embodiment 4 of the present invention. Figure 4 A block diagram of an exemplary device 400 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The device 400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0128] like Figure 4 As shown, the device 400 is in the form of a general-purpose computing device. The components of the device 400 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).

[0129] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0130] Device 400 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by device 400, including volatile and non-volatile media, removable and non-removable media.

[0131] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Device 400 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 403 via one or more data medium interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

[0132] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 407 generally perform the functions and / or methods of the embodiments described herein.

[0133] Device 400 can also communicate with one or more external devices 409 (e.g., keyboard, pointing device, display 410, etc.), can also communicate with one or more devices that enable users to interact with the device 400, and / or communicate with any device that enables the device 400 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed through input / output (I / O) interface 411. In addition, device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through network adapter 412. As shown in the figure, network adapter 412 communicates with other modules of device 400 through bus 403. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with device 400, including but not limited to: microcode, device driver, redundant processing unit, external disk drive array, RAID system, tape drive and data backup storage system, etc.

[0134] The processing unit 401 executes various functional applications and data processing by running the program stored in the system memory 402, for example, implementing the data parsing method provided in the embodiment of the present invention, including:

[0135] According to the user's selection operation, determine the target parsing file from the candidate parsing files;

[0136] The data to be parsed is parsed according to the candidate structures in the target parsing file and the attribute information of the candidate data fields in the candidate structures; wherein the attribute information includes data length and interpretation information.

[0137] Embodiment 5

[0138] Embodiment 5 of the present invention further provides a computer-readable storage medium, wherein the computer-executable instructions are used to execute a data parsing method when executed by a computer processor, the method comprising:

[0139] According to the user's selection operation, determine the target parsing file from the candidate parsing files;

[0140] The data to be parsed is parsed according to the candidate structures in the target parsing file and the attribute information of the candidate data fields in the candidate structures; wherein the attribute information includes data length and interpretation information.

[0141] Of course, a storage medium containing computer executable instructions provided in an embodiment of the present invention, whose computer executable instructions are not limited to the method operations described above, can also perform related operations in a data parsing method provided in any embodiment of the present invention. The computer-readable storage medium of an embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0142] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0143] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0144] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data analysis method, characterized in that: include: According to the user's selection operation, determine the target parsing file from the candidate parsing files; Parsing the data to be parsed according to the candidate structure in the target parsing file and the attribute information of the candidate data field in the candidate structure; wherein the attribute information includes data length and interpretation information; The step of parsing the data to be parsed according to the structure in the target parsing file and the attribute information of the data field in the structure includes: Determine a first structure from the candidate structures, and use each candidate data field in the first structure as a first data field; According to the order of each of the first data fields and the attribute information of each of the first data fields, the data to be parsed are parsed in sequence; The step of sequentially parsing the data to be parsed according to the order of the first data fields and the attribute information of the first data fields includes: Determine whether the current first data field points to a second structure; wherein the second structure is any candidate structure except the first structure; In the case of not pointing to the second structure, according to the data length of the current first data field, data interception is performed on the remaining data of the data to be parsed to obtain a target data field; Annotate the target data field according to the interpretation information as parsed data corresponding to the current first data field; After determining whether the current first data field points to the second structure, the method further includes: In the case of pointing to the second structure, taking each candidate data field in the second structure as the second data field; According to the order of each of the second data fields and the attribute information of each of the second data fields, the remaining data of the data to be parsed are parsed in sequence; The step of sequentially parsing the remaining data of the data to be parsed according to the order of the second data fields and the attribute information of the second data fields includes: In the case where the current second data field is the target second data field, determining whether the target second data field points to a third structure; wherein the target second data field is the second data field in the last order, and the third structure is any candidate structure except the first structure and the second structure; In the case of not pointing to the third structure, the remaining data of the data to be parsed is parsed according to the attribute information of the target second data field, and the first data field located after the current first data field is used as the updated current first data field; When pointing to the third structure, each candidate data field in the third structure is used as a third data field, and the remaining data of the data to be parsed is parsed in sequence according to the order of each third data field and the attribute information of each third data field.

2. The method according to claim 1, before parsing the data to be parsed, further comprises: Remove the network layer data and link layer data from the original data to obtain secure data; In the case where the security data complies with the preset security protocol, the security layer data in the security data is removed, and the remaining data is used as the data to be parsed.

3. The method according to claim 2, after parsing the data to be parsed, further comprises: The parsed data is determined according to the parsing result, and the parsed data and the original data are stored according to a preset storage path; wherein the preset storage path includes at least one of a local end and a database.

4. The method according to claim 3, after storing the parsed data and the original data according to the preset storage path, further comprises: The parsed data and the original data are visualized according to the user's query instruction.

5. A data analysis device, characterized in that: include: A target parsing file determination module is used to determine a target parsing file from candidate parsing files according to a user's selection operation; A data parsing module, used for parsing the data to be parsed according to the candidate structures in the target parsing file and the attribute information of the candidate data fields in the candidate structures; wherein the attribute information includes data length and interpretation information; The data parsing module is specifically used to determine a first structure from the candidate structures, and use each candidate data field in the first structure as a first data field; and parse the data to be parsed in sequence according to the order of each first data field and the attribute information of each first data field; The data parsing module is further used to determine whether the current first data field points to a second structure; wherein the second structure is any candidate structure other than the first structure; when the second structure is not pointed to, the remaining data of the data to be parsed is intercepted according to the data length of the current first data field to obtain a target data field; the target data field is marked according to the interpretation information as parsed data corresponding to the current first data field; The device further includes a second data domain determination module, wherein the second data domain determination module is specifically configured to, when pointing to the second structure, use each candidate data domain in the second structure as the second data domain; and parse the remaining data of the data to be parsed in sequence according to the order of each second data domain and the attribute information of each second data domain; Among them, the second data domain determination module is specifically used to determine whether the target second data domain points to a third structure when the current second data domain is the target second data domain; wherein the target second data domain is the second data domain in the last order, and the third structure is any candidate structure except the first structure and the second structure; when not pointing to the third structure, the remaining data of the data to be parsed is parsed according to the attribute information of the target second data domain, and the first data domain whose order is after the current first data domain is used as the updated current first data domain; when pointing to the third structure, each candidate data domain in the third structure is used as the third data domain, and the remaining data of the data to be parsed is parsed in sequence according to the order of each third data domain and the attribute information of each third data domain.

6. An electronic device, characterized in that: The electronic device further comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data parsing method as described in any one of claims 1-4.

7. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data parsing method as described in any one of claims 1 to 4 is implemented.

Images