Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining to-be-transmitted first JSON data, the first JSON data comprises at least one key value pair, the key value pair comprises a key field name and a key field value of the first JSON data, through the key value pair, obtaining first template data of the first JSON data, storing the first template data to a message queue, generating a first template identifier for the first template data, obtaining a storage sequence of key field names in the first template data, obtaining key field values from the key value pairs according to the storage sequence, carrying out compression conversion on the key field values, obtaining a binary byte stream, and sending the binary byte stream to the message queue; and transmitting the first template identifier and the binary byte stream as target transmission data. According to the invention, the occupied space during message transmission is reduced, and the network transmission efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] When large amounts of data are continuously transmitted between systems, the middleware MQ (Message Queue) is usually used. As an asynchronous communication mechanism, MQ message queue can effectively decouple the dependencies between systems and improve the scalability and reliability of the system. When using MQ message queues to transmit messages, readable message formats such as JSON are usually used. JSON (JavaScript Object Notation) is a lightweight data exchange format that is easy for people to read and write, and is also easy for machines to parse and generate.

[0003] However, the JSON format takes up a lot of space, affecting the efficiency of data storage and network transmission. Summary of the invention

[0004] In view of this, the present application aims to propose a data processing method, device, electronic device and storage medium to solve the problem that the JSON format used in current message transmission occupies a large space and affects the efficiency of data storage and network transmission. The specific technical solution is as follows:

[0005] According to a first aspect of the present application, a data processing method is provided, the method comprising:

[0006] Acquire first JSON data to be transmitted, where the first JSON data includes at least one key-value pair, where the key-value pair includes a key field name and a key field value of the first JSON data;

[0007] Obtain first template data of the first JSON data through the key-value pair;

[0008] Saving the first template data to a message queue, and generating a first template identifier for the first template data;

[0009] Obtaining a storage order of key field names in the first template data, and obtaining key field values ​​from the key-value pairs according to the storage order;

[0010] Compress and convert the key field value to obtain a binary byte stream;

[0011] The first template identifier and the binary byte stream are used as target transmission data for transmission.

[0012] Optionally, obtaining the first template data of the first JSON data through the key-value pair includes:

[0013] Determine the field type, field value type, precision value, offset and length value corresponding to the key field value in the key-value pair through the key field value, wherein the field type includes at least one of an array type, an object type and a regular field type, and the field value type includes at least one of a numeric type, a date type, a string type and a Boolean value type;

[0014] The first template data of the first JSON data is obtained through the key field name, the field type corresponding to the key field value, the field value type, the precision value, the offset and the length value.

[0015] Optionally, determining the field type, field value type, precision value, offset and length value corresponding to the key field value in the key-value pair by using the key field value includes:

[0016] Get the numerical representation of the key field value;

[0017] Determine the field type and field value type corresponding to the key field value in the key-value pair by using the numerical representation method;

[0018] In the case where it is determined that the numerical representation of the key field value is a digital type, determining the precision value and the offset corresponding to the key field value in the key-value pair by using the key field value;

[0019] The length value corresponding to the key field value in the key-value pair is determined through the key field value, the precision value, the offset, and the numerical representation of the key field value.

[0020] Optionally, determining the field type and field value type corresponding to the key field value in the key-value pair by using the numerical representation method includes:

[0021] Determine the field value type corresponding to the key field value in the key-value pair by using the numerical representation method;

[0022] When it is determined that the numerical representation of the key field value includes square brackets, determining that the field type corresponding to the key field value in the key-value pair is an array type;

[0023] When it is determined that the numerical representation of the key field value includes curly brackets, determining that the field type corresponding to the key field value in the key-value pair is an object type;

[0024] When it is determined that the numerical representation of the key field value does not include square brackets and curly brackets, it is determined that the field type corresponding to the key field value in the key-value pair is a regular field type.

[0025] Optionally, compressing and converting the key field value to obtain a binary byte stream includes:

[0026] When it is determined that the numerical representation of the key field value is a digital type, compressing and converting the key field value according to the precision value and the offset corresponding to the key field value to obtain a binary byte stream;

[0027] When it is determined that the numerical representation of the key field value is a date type and a string type, converting the key field value into a binary byte stream by using a first preset encoding strategy;

[0028] When it is determined that the numerical representation of the key field value is a Boolean value type, the key field value is converted into a binary byte stream using a second preset encoding strategy.

[0029] Optionally, after transmitting the first template identifier and the binary byte stream as target transmission data, the method further includes:

[0030] Obtain second JSON data to be transmitted and a key-value pair of the second JSON data;

[0031] Obtain second template data of the second JSON data through the key-value pairs of the second JSON data;

[0032] Comparing the first template data with the second template data;

[0033] If they are completely consistent, setting the first template identifier as the template identifier of the second template data;

[0034] If they are not completely consistent, a second template identifier is generated for the second template data.

[0035] Optionally, after transmitting the first template identifier and the binary byte stream as target transmission data, the method further includes:

[0036] When it is determined that the target transmission data is received, obtaining first template data of the first JSON data through a first template identifier carried by the target transmission data;

[0037] Obtain the key field name of the first JSON data through the first template data;

[0038] Extracting bytes in the binary byte stream carried by the target transmission data through the first template data to obtain a target byte combination;

[0039] Decode the target byte combination according to a preset decoding strategy to obtain a key field value of the first JSON data;

[0040] The key field name and the key field value are combined to obtain first JSON data.

[0041] According to a second aspect of the present application, a data processing device is provided, the device comprising:

[0042] A first acquisition module, configured to acquire first JSON data to be transmitted, wherein the first JSON data includes at least one key-value pair, wherein the key-value pair includes a key field name and a key field value of the first JSON data;

[0043] A second acquisition module, used for acquiring first template data of the first JSON data through the key-value pair;

[0044] A first generating module, used for saving the first template data to a message queue and generating a first template identifier for the first template data;

[0045] A third acquisition module, used to acquire the storage order of the key field names in the first template data, and acquire the key field values ​​from the key-value pairs according to the storage order;

[0046] A fourth acquisition module, used for compressing and converting the key field value to obtain a binary byte stream;

[0047] The transmission module is used to transmit the first template identifier and the binary byte stream as target transmission data.

[0048] According to another aspect of the present application, there is also provided an electronic device, a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0049] The memory is used to store computer programs;

[0050] The processor is used to implement the data processing method according to any one of claims 1 to 7 when executing the program stored in the memory.

[0051] According to another aspect of the present application, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0052] According to another aspect of the present application, a vehicle is provided, comprising: the above-mentioned data processing device.

[0053] The data processing method provided by the present application obtains the first JSON data to be transmitted, the first JSON data includes at least one key-value pair, the key-value pair includes the key field name and the key field value of the first JSON data, obtains the first template data of the first JSON data through the key-value pair, saves the first template data to the message queue, and generates a first template identifier for the first template data, extracts the template data in the first JSON data, and reduces the space occupied by data storage during subsequent transmission. At the same time, the data to be transmitted can also be extracted according to the template data, and the storage order of the key field names in the first template data is obtained, and the key field values ​​are obtained from the key-value pair according to the storage order, and the key field values ​​are compressed and converted to obtain a binary byte stream. By compressing and converting the key field values, the space occupied by data storage is further reduced, and the network transmission efficiency is improved. The first template identifier and the binary byte stream are used as the target transmission data for transmission. The present application extracts the first template data in the first JSON data and does not transmit it, but transmits the first template identifier, which reduces the space occupied during message transmission and improves the network transmission efficiency. At the same time, the key field values ​​in the first JSON data are compressed and converted, which further reduces the space occupied during message transmission and improves the network transmission efficiency.

[0054] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0056] Figure 1 It is a flow chart of the steps of a data processing method provided by the present application;

[0057] Figure 2 yes Figure 1 A flowchart of step 102 in a data processing method provided by the present application;

[0058] Figure 3 yes Figure 1 A flowchart of step 105 in a data processing method provided by the present application;

[0059] Figure 4 It is a structural schematic diagram of a data processing device provided by the present application;

[0060] Figure 5 It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, each embodiment of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in each embodiment of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined with each other and referenced to each other without contradiction.

[0062] At present, when a large amount of data is continuously transmitted between systems, the middleware MQ message queue is usually used. When using the MQ message queue to transmit messages, a message format with good readability such as json is usually used, but this format itself takes up a large space, has high requirements for network bandwidth and storage, and increases the cost of message transmission. In order to solve this problem, this application proposes a data processing method.

[0063] Reference Figure 1 , shows a flow chart of the steps of a data processing method provided by the present application, the method may include:

[0064] Step 101: Acquire first JSON data to be transmitted, where the first JSON data includes at least one key-value pair, where the key-value pair includes a key field name and a key field value of the first JSON data.

[0065] When transmitting messages through the message middleware, the format of the message can be various, such as JSON format, XML format, binary format, plain text format, or user-defined format. This application mainly processes messages in JSON format, that is, the first JSON data to be transmitted. The format of the first JSON data is relatively large and requires more space. The following is an example of the first JSON data:

[0066] {

[0067] "field1":2.5,

[0068] "field2":"2024-10-01 13:33:24",

[0069] "field3":[

[0070] 0.8, 0.9

[0072] ],

[0073] "field4":{

[0074] "f1":"KEY8912934894wl",

[0075] "f2":false

[0076] }

[0077] }

[0078] It can be seen that the first JSON data includes multiple key-value pairs, each of which includes a key field name (key) and a key field value (value), where key can also be called an attribute name, and the corresponding value is called an attribute value. The key field names and key field values ​​in the key-value pairs in the above example are as follows: the key field name is "field1", the key field value is 2.5, the key field name is "field2", the key field value is "2024-10-01 13:33:24", the key field name is "field3", the key field value is [0.8,0.9], the key field name is "field4", the key field value is {"f1":"KEY8912934894wl","f2":false}, where the key field value of the key field name "field4" also includes two key-value pairs, the key field value is "field4.f1", the key field value is "KEY8912934894wl", the key field value is "field4.f2", and the key field value is false.

[0079] Step 102: Obtain first template data of first JSON data through key-value pairs.

[0080] Normally, when a client publishes a certain type of JSON message, the data template will not change, only the data in it will change. Therefore, this application will extract the template data of the first JSON data, save it to the message queue, and then only transmit the key field value of the first JSON data. At the same time, when facing the subsequent data of the same type, only the key field value needs to be extracted for transmission, so as to reduce the space occupied during transmission.

[0081] The first template data obtained by the present application is determined based on the key-value pair (key field name and key field value) in the first JSON data. Through the key field value, the field type to which the key field value belongs can be determined. For key field values ​​of digital type, the precision value of the numerical value is determined by the specific expression of the key field value. Through the specific numerical value of the key field value, the offset can also be pre-set so that the offset can convert the key field value into a numerical value with less space. By converting the key field value, the final transmitted numerical value is determined. According to the final transmitted numerical value, the data length value of the key field value during transmission can also be determined. The first template data of the first JSON data can be obtained through the key field name and various information obtained through the key field value.

[0082] Further, step 102, such as Figure 2 As shown:

[0083] Step 1021, determine the field type, field value type, precision value, offset and length value corresponding to the key field value in the key-value pair through the key field value, wherein the field type includes at least one of an array type, an object type and a regular field type, and the field value type includes at least one of a numeric type, a date type, a string type and a Boolean value type.

[0084] It should be noted that when the field type, field value type, precision value, offset and length value corresponding to the key field value in the key-value pair are used, the field type and field value type are first determined by the numerical representation of the key field value, where the numerical representation refers to the specific representation of the key field value in JSON, such as string, number, Boolean value, array, object, etc. When the numerical representation of the key field value is a numeric type, its precision value can be determined according to the specific numeric value of the key field value. For example, if the key field name is "field1" and the key field value is 2.5, because there are no quotation marks in the key field value 2.5, it is considered that 2.5 is a numeric type, and the precision value is determined according to the number of digits after the decimal point, so the precision value of 2.5 is 0.1; if the key field value is 0.25, the precision value is 0.01. The precision value can reduce the number of digits occupied by the number. Similarly, the role of the offset is to reduce the number of digits occupied by the number after subtracting the offset from the value, so that the transmitted number is smaller and can occupy a smaller space, so the size of the offset needs to be determined according to the specific numeric value of the key field value. After compressing the key field value by precision value and offset value, the occupied length value can be determined according to the obtained value. Because different messages with the same template may have different corresponding key field values, the precision value, offset value and length value are set according to the value range of the data of the message type (with the same template). The precision value, offset value and length value of the same template are fixed values. The specific steps include:

[0085] Get the numerical representation of the key field value;

[0086] Determine the field type and field value type corresponding to the key field value in the key-value pair through numerical representation;

[0087] When it is determined that the numerical representation of the key field value is a numeric type, the precision value and offset corresponding to the key field value in the key-value pair are determined by the key field value;

[0088] Determine the length value corresponding to the key field value in the key-value pair through the key field value, precision value, offset, and the numerical representation of the key field value.

[0089] Through the above steps, the required content of the first template data can be accurately obtained, and by setting the parameters therein, a solid foundation is provided for subsequent data compression.

[0090] It should be noted that when determining the field type and field value type corresponding to the key field value in the key-value pair by numerical expression, the type of brackets included in the key field value can be used to determine the field type. When square brackets are included, the field type is determined to be an array type. When curly brackets are included, the field type is determined to be an object type. When nothing is included, it is considered to be a regular field type. Regular field types can be simple types such as strings, numbers, or Boolean values. The field value type is determined by the numerical expression of the key field value. For example, if the key field value is 2.5, the field value type is a numeric type. If the key field value is "2024-10-01 13:33:24", the field value type is a date type. If the key field value is "KEY8912934894wl", the field value type is a string type. If the key field value is false, the field value type is a Boolean type. In addition to numeric types, date types, string types, and Boolean types, other field value types can also be obtained according to the numerical expression of the data. This application does not make specific restrictions here. The specific steps include:

[0091] Determine the field value type corresponding to the key field value in the key-value pair through the numerical representation;

[0092] When the numeric representation of the key field value includes square brackets, the field type corresponding to the key field value in the key-value pair is determined to be an array type;

[0093] When the numeric representation of the key field value includes curly brackets, the field type corresponding to the key field value in the key-value pair is determined to be an object type;

[0094] When the numeric representation of the key field value is determined without including square brackets and curly brackets, the field type corresponding to the key field value in the key-value pair is determined to be a general field type.

[0095] Through the above steps, different types of JSON data can be accurately identified and processed, ensuring that no type errors occur during the parsing process, and being able to handle complex data structures (such as nested arrays and nested objects), improving the accuracy, maintainability and scalability of data parsing, making it more efficient and reliable when processing JSON data.

[0096] Step 1022, obtain the first template data of the first JSON data through the key field name, the field type corresponding to the key field value, the field value type, the precision value, the offset and the length value.

[0097] This application obtains the key field name, the field type corresponding to the key field value, the field value type, the precision value, the offset and the length value, and then determines the first template data of the first JSON data based on these data. For example, if the key field name of the first JSON data is "field1", the key field value is 2.5, the key field name is "field2", the key field value is "2024-10-01 13:33:24", the key field name is "field3", the key field value is [0.8], the key field name is "field4", and the key field value is {"f1":"KEY8912934894wl","f2":false}, then in the first template data, for the key field name "field1", the field type corresponding to the key field value is a regular field type, the field value type is a numeric type, the precision value is 0.1, the offset is 20, and the length value is 3; for the key field name "field2", the field type corresponding to the key field value is a regular field type, the field value type is a date type, the precision value is 1, the offset is 0, and the length value is 32; for the key field name "f ield3", the field type corresponding to the key field value is array type, the field value type is numeric type, the precision value is 0.1, the offset is 5, and the length value is 3; for the key field name "field4", the field type corresponding to the key field value is object type. Because there are two key-value pairs in this field value, the key-value pairs are described in detail, but not the whole field. Therefore, the key field name is "field4.f1", the field type corresponding to the key field value is regular field type, the field value type is string type, and the length value is 120. The key field name is "field4.f2", the field type corresponding to the key field value is regular field type, the field value type is Boolean type, and the length value is 1. The length value is determined according to the value after conversion. For example, the value of 2.5 after precision and offset conversion is: 2.5 / 0.1-20=5, which is 101 in binary, so the length is 3 bits. The length value of other numeric types is determined in the same way. For a key field value of string type, one character occupies 8 bits, so the length value is: 15×8=120.

[0098] It should be noted that the offset and precision values ​​are preset, so they are set according to the value range of the data. For example, if the value range of the transmission data of the key field named "field1" of a certain type of message is 0.15-0.56, then the precision value can be set to 0.01 and the offset can be set to 10. If the value range of the transmission data of the key field named "field1" of a certain type of message is 1000-1005, then the precision value can be set to 1 and the offset can be set to 1000. The specific settings are determined according to the actual situation. According to the value range, precision value and offset of the transmission data, the maximum length value of the data that can be obtained is obtained and saved as template data.

[0099] Step 103: save the first template data to a message queue, and generate a first template identifier for the first template data.

[0100] After obtaining the first template data, the application stores it in a message queue, and then sets an API or user interface (UI) to allow the client to register and publish a data template according to the first template data stored in the message queue, and generate a first template identifier that uniquely identifies the first template data. Typically, the first template identifier is a number that occupies 4 bytes, and the template identifiers of the same data template are the same.

[0101] Step 104, obtain the storage order of the key field names in the first template data, and obtain the key field values ​​from the key-value pairs according to the storage order.

[0102] When transmitting a message, the present application only needs to transmit the key field value in the first JSON data. In order to facilitate the subsequent restoration of the message, it is necessary to follow certain rules, that is, to obtain the key field value in the order in which the key field name is saved, so as to facilitate the subsequent extraction of the correct data from the merged data according to this saved order.

[0103] Step 105: compress and convert the key field value to obtain a binary byte stream.

[0104] After obtaining the key field value, this application will not transmit it directly, but will compress the data according to the precision value and offset to further reduce the occupied space.

[0105] Further, step 105, such as Figure 3 As shown:

[0106] Step 1051: When it is determined that the numerical representation of the key field value is a digital type, the key field value is compressed and converted using the precision value and offset corresponding to the key field value to obtain a binary byte stream.

[0107] Step 1052: When it is determined that the numerical representation of the key field value is a date type or a character string type, the key field value is converted into a binary byte stream using a first preset encoding strategy.

[0108] Step 1053: When it is determined that the numerical representation of the key field value is a Boolean value type, the key field value is converted into a binary byte stream using a second preset encoding strategy.

[0109] For example, when the key field value is 2.5, the key field value at this time contains 3 characters, and one character occupies 8 bits. The precision value and the offset are calculated to obtain: 2.5 / 0.1-20=5, which is converted into binary as 101, which only occupies 3 bits. When the numerical representation of the key field value is a date type and a string type, the first preset encoding strategy can be to convert the key field value of the date type into a Unix timestamp (second level) first, and then convert it into binary. For example, the key field value is "2024-10-01 13:33:24", which originally contains 18 characters and occupies 18×8=144 bits. 13:33:24" is converted to the Unix timestamp 1727842404. The binary value of 1727842404 is 1100101011010101000101000100100, which occupies 31 bits. The first preset encoding strategy can also be to use UTF-8 encoding to convert the string into a binary byte stream. For example, the key field value is "KEY8912934894wl", which contains 15 characters. After UTF-8 encoding, it is determined to occupy 15×8=120 bits. The second preset encoding strategy is to encode the Boolean value true as 1 and the Boolean value false as 0.

[0110] Through the above steps, the key field value can be compressed and converted under different numerical expressions, thereby reducing the space occupied by data transmission.

[0111] Step 106: Transmit the first template identifier and the binary byte stream as target transmission data.

[0112] In this application, the key field values ​​are taken out according to the storage order of the key field names in the first template data, and after compressing to obtain a binary byte stream, the first template identifier and the binary byte stream are used as the target transmission data for transmission. During transmission, the binary byte stream is also converted into a byte array, and 4 bytes are added before the byte array to store the first template identifier.

[0113] Among them, when transmitting, for strings of variable length (length not defined in the data template field), this application will use 16-bit space to record the string length when writing the binary byte stream, and then write it into the byte stream. For other fixed-length values, their length will be fixed when writing.

[0114] For example, the first JSON data is:

[0115]

[0116] According to the above content, the first template data and the binary byte stream obtained by compressing the key field value are obtained, and the obtained content is formed into a field definition list, as shown in Table 1:

[0117] Table 1: Field definition list of the first JSON data

[0118]

[0119]

[0120] Among them, in the field type, 0 represents a regular field, 1 represents an array, and 2 represents an object; for an array, the following field value type, precision value, offset, length value and other attributes define the value of the array element, and in the field value type, 0 represents a number, 1 represents a date, 2 represents a string, and 3 represents a Boolean value; for dates and Boolean values, their values ​​are converted to numbers during transmission to save space, and the length value indicates the number of bits occupied by the field value in the data. For a string type value, if the length is not defined or the length is 0, it indicates that the string is a variable-length string, and when writing a binary byte stream, the length is recorded in the data using 16 bits of space. The first template data includes attributes such as field name, field type, field value type, precision value, offset, and length value. The key field value is compressed to obtain the transmission value, but these transmission values ​​are also converted into binary byte streams during actual transmission. For example, the transmission value is 5, which is converted into a binary byte stream of 101, and the transmission value is 1727760804, which is converted into a binary byte stream of 1100110101010101010101010101010100.

[0121] It should be noted that when the template data is the same, the corresponding template identifier is the same. Therefore, after obtaining the new second JSON data, first determine whether the template data contained therein is the same as the first JSON data. If they are the same, directly use the first template identifier for transmission. If they are different, it is necessary to re-store the obtained second template data into the message queue, and then generate a new second template identifier for subsequent calls. The specific steps include:

[0122] Obtain the second JSON data to be transmitted and the key-value pair of the second JSON data;

[0123] Obtain second template data of the second JSON data through the key-value pair of the second JSON data;

[0124] Comparing the first template data with the second template data;

[0125] If they are completely consistent, the first template identifier is set as the template identifier of the second template data;

[0126] If they are not completely consistent, a second template identifier is generated for the second template data.

[0127] The above steps ensure the consistency and uniqueness of the data template.

[0128] Furthermore, as the receiver of the message, i.e., the consumer, after receiving the message, i.e., the target transmission data, firstly needs to obtain the first template identifier according to the first 4 bytes of the message, obtain the first template data corresponding to the message from MQ through the first template identifier, and then parse the data according to the data definition in the template data to restore the original message. The specific steps include:

[0129] When it is determined that the target transmission data is received, first template data of the first JSON data is acquired through a first template identifier carried in the target transmission data;

[0130] Obtain the key field name of the first JSON data through the first template data;

[0131] Extracting bytes in a binary byte stream carried by the target transmission data through the first template data to obtain a target byte combination;

[0132] Decode the target byte combination according to a preset decoding strategy to obtain the key field value of the first JSON data;

[0133] Combine the key field name and the key field value to obtain the first JSON data.

[0134] The above steps ensure that the received binary data can be correctly parsed and restored to the original JSON data format.

[0135] The data processing method provided by the present application obtains the first JSON data to be transmitted, the first JSON data includes at least one key-value pair, the key-value pair includes the key field name and the key field value of the first JSON data, obtains the first template data of the first JSON data through the key-value pair, saves the first template data to the message queue, and generates a first template identifier for the first template data, extracts the template data in the first JSON data, and reduces the space occupied by data storage during subsequent transmission. At the same time, the data to be transmitted can also be extracted according to the template data, and the storage order of the key field names in the first template data is obtained, and the key field values ​​are obtained from the key-value pair according to the storage order, and the key field values ​​are compressed and converted to obtain a binary byte stream. By compressing and converting the key field values, the space occupied by data storage is further reduced, and the network transmission efficiency is improved. The first template identifier and the binary byte stream are used as the target transmission data for transmission. The present application extracts the first template data in the first JSON data and does not transmit it, but transmits the first template identifier, which reduces the space occupied during message transmission and improves the network transmission efficiency. At the same time, the key field values ​​in the first JSON data are compressed and converted, which further reduces the space occupied during message transmission and improves the network transmission efficiency.

[0136] Reference Figure 4 , shows a schematic diagram of the structure of a data processing device provided by the present application, the device comprising:

[0137] The first acquisition module 201 is used to acquire first JSON data to be transmitted, where the first JSON data includes at least one key-value pair, where the key-value pair includes a key field name and a key field value of the first JSON data.

[0138] The second acquisition module 202 is used to acquire first template data of the first JSON data through a key-value pair.

[0139] The first generating module 203 is used to save the first template data to the message queue and generate a first template identifier for the first template data.

[0140] The third acquisition module 204 is used to acquire the storage order of the key field names in the first template data, and acquire the key field values ​​from the key-value pairs according to the storage order.

[0141] The fourth acquisition module 205 is used to compress and convert the key field value to obtain a binary byte stream.

[0142] The transmission module 206 is used to transmit the first template identifier and the binary byte stream as target transmission data.

[0143] Optionally, the second acquisition module 202 specifically includes:

[0144] The first determination submodule is used to determine the field type, field value type, precision value, offset and length value corresponding to the key field value in the key-value pair through the key field value, wherein the field type includes at least one of an array type, an object type and a regular field type, and the field value type includes at least one of a numeric type, a date type, a string type and a Boolean value type.

[0145] The first acquisition submodule is used to acquire the first template data of the first JSON data through the key field name, the field type corresponding to the key field value, the field value type, the precision value, the offset and the length value.

[0146] Optionally, the first determining submodule specifically includes:

[0147] The first acquisition unit is used to acquire a numerical representation of the key field value.

[0148] The first determining unit is used to determine the field type and field value type corresponding to the key field value in the key-value pair by numerical expression.

[0149] The second determining unit is configured to determine a precision value and an offset corresponding to the key field value in the key-value pair through the key field value when it is determined that the numerical representation of the key field value is a digital type.

[0150] The third determining unit is used to determine the length value corresponding to the key field value in the key-value pair through the key field value, the precision value, the offset, and the numerical representation of the key field value.

[0151] Optionally, the first determining unit specifically includes:

[0152] The first determination subunit is used to determine the field value type corresponding to the key field value in the key-value pair through a numerical expression.

[0153] The second determining subunit is used to determine that the field type corresponding to the key field value in the key-value pair is an array type when the numerical representation of the key field value includes square brackets.

[0154] The third determining subunit is used to determine that the field type corresponding to the key field value in the key-value pair is an object type when the numerical expression of the key field value includes curly brackets.

[0155] The fourth determining subunit is used to determine that the field type corresponding to the key field value in the key-value pair is a regular field type when it is determined that the numerical representation of the key field value does not include square brackets and curly brackets.

[0156] Optionally, the fourth acquisition module 205 specifically includes:

[0157] The second acquisition submodule is used to compress and convert the key field value according to the precision value and offset corresponding to the key field value to obtain a binary byte stream when it is determined that the numerical representation of the key field value is a digital type.

[0158] The first conversion submodule is used to convert the key field value into a binary byte stream by using a first preset encoding strategy when it is determined that the numerical representation of the key field value is a date type and a character string type.

[0159] The second conversion submodule is used to convert the key field value into a binary byte stream by using a second preset encoding strategy when it is determined that the numerical representation of the key field value is a Boolean value type.

[0160] Optionally, the data processing device further includes:

[0161] The fifth acquisition module is used to acquire the second JSON data to be transmitted and the key-value pair of the second JSON data.

[0162] The sixth acquisition module is used to acquire the second template data of the second JSON data through the key-value pairs of the second JSON data.

[0163] The comparison module is used to compare the first template data with the second template data.

[0164] The identifier setting module is used to set the first template identifier as the template identifier of the second template data if they are completely consistent.

[0165] The second generating module is used to generate a second template identifier for the second template data if they are not completely consistent.

[0166] The seventh acquisition module is used to acquire the first template data of the first JSON data through the first template identifier carried by the target transmission data when it is determined that the target transmission data is received.

[0167] An eighth acquisition module is used to acquire the key field name of the first JSON data through the first template data.

[0168] The extraction module is used to extract bytes in the binary byte stream carried by the target transmission data through the first template data to obtain a target byte combination.

[0169] The ninth acquisition module is used to decode the target byte combination according to a preset decoding strategy to obtain the key field value of the first JSON data.

[0170] The tenth acquisition module is used to combine the key field name and the key field value to obtain the first JSON data.

[0171] The data processing method provided by the present application obtains the first JSON data to be transmitted, the first JSON data includes at least one key-value pair, the key-value pair includes the key field name and the key field value of the first JSON data, obtains the first template data of the first JSON data through the key-value pair, saves the first template data to the message queue, and generates a first template identifier for the first template data, extracts the template data in the first JSON data, and reduces the space occupied by data storage during subsequent transmission. At the same time, the data to be transmitted can also be extracted according to the template data, and the storage order of the key field names in the first template data is obtained, and the key field values ​​are obtained from the key-value pair according to the storage order, and the key field values ​​are compressed and converted to obtain a binary byte stream. By compressing and converting the key field values, the space occupied by data storage is further reduced, and the network transmission efficiency is improved. The first template identifier and the binary byte stream are used as the target transmission data for transmission. The present application extracts the first template data in the first JSON data and does not transmit it, but transmits the first template identifier, which reduces the space occupied during message transmission and improves the network transmission efficiency. At the same time, the key field values ​​in the first JSON data are compressed and converted, which further reduces the space occupied during message transmission and improves the network transmission efficiency.

[0172] Reference Figure 5 , the present application also provides an electronic device, such as Figure 5 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0173] A processor 301, a memory 303 for storing processor executable instructions;

[0174] The processor 301 is configured to execute the instructions to implement the data processing method as described above:

[0175] Acquire first JSON data to be transmitted, where the first JSON data includes at least one key-value pair, where the key-value pair includes a key field name and a key field value of the first JSON data;

[0176] Obtain first template data of the first JSON data through the key-value pair;

[0177] Saving the first template data to a message queue, and generating a first template identifier for the first template data;

[0178] Obtaining a storage order of key field names in the first template data, and obtaining key field values ​​from the key-value pairs according to the storage order;

[0179] Compress and convert the key field value to obtain a binary byte stream;

[0180] The first template identifier and the binary byte stream are used as target transmission data for transmission.

[0181] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0182] The communication interface is used for communication between the above terminal and other devices.

[0183] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0184] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0185] In another embodiment provided in the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method described in any one of the above embodiments is implemented.

[0186] In another embodiment provided in the present application, a vehicle is also provided, which may specifically include: the above-mentioned data processing device.

[0187] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0188] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0189] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0190] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire first JSON data to be transmitted, where the first JSON data includes at least one key-value pair, where the key-value pair includes a key field name and a key field value of the first JSON data; Obtain first template data of the first JSON data through the key-value pair; Saving the first template data to a message queue, and generating a first template identifier for the first template data; Obtaining a storage order of key field names in the first template data, and obtaining key field values ​​from the key-value pairs according to the storage order; Compress and convert the key field value to obtain a binary byte stream; The first template identifier and the binary byte stream are used as target transmission data for transmission.

2. The method according to claim 1, characterized in that: The step of obtaining the first template data of the first JSON data through the key-value pair includes: Determine the field type, field value type, precision value, offset and length value corresponding to the key field value in the key-value pair through the key field value, wherein the field type includes at least one of an array type, an object type and a regular field type, and the field value type includes at least one of a numeric type, a date type, a string type and a Boolean value type; The first template data of the first JSON data is obtained through the key field name, the field type corresponding to the key field value, the field value type, the precision value, the offset and the length value.

3. The method according to claim 2, characterized in that Determining the field type, field value type, precision value, offset and length value corresponding to the key field value in the key-value pair by using the key field value includes: Get the numerical representation of the key field value; Determine the field type and field value type corresponding to the key field value in the key-value pair by using the numerical representation method; In the case where it is determined that the numerical representation of the key field value is a digital type, determining the precision value and the offset corresponding to the key field value in the key-value pair by using the key field value; The length value corresponding to the key field value in the key-value pair is determined through the key field value, the precision value, the offset, and the numerical representation of the key field value.

4. The method according to claim 3, characterized in that The determining, by the numerical representation, the field type and the field value type corresponding to the key field value in the key-value pair includes: Determine the field value type corresponding to the key field value in the key-value pair by using the numerical representation method; When it is determined that the numerical representation of the key field value includes square brackets, determining that the field type corresponding to the key field value in the key-value pair is an array type; When it is determined that the numerical representation of the key field value includes curly brackets, determining that the field type corresponding to the key field value in the key-value pair is an object type; When it is determined that the numerical representation of the key field value does not include square brackets and curly brackets, it is determined that the field type corresponding to the key field value in the key-value pair is a regular field type.

5. The method according to claim 2, characterized in that: The compressing and converting the key field value to obtain a binary byte stream includes: When it is determined that the numerical representation of the key field value is a digital type, compressing and converting the key field value according to the precision value and the offset corresponding to the key field value to obtain a binary byte stream; When it is determined that the numerical representation of the key field value is a date type and a string type, converting the key field value into a binary byte stream by using a first preset encoding strategy; When it is determined that the numerical representation of the key field value is a Boolean value type, the key field value is converted into a binary byte stream using a second preset encoding strategy.

6. The method according to claim 1, characterized in that After transmitting the first template identifier and the binary byte stream as target transmission data, the method includes: Obtain second JSON data to be transmitted and a key-value pair of the second JSON data; Obtain second template data of the second JSON data through the key-value pairs of the second JSON data; Comparing the first template data with the second template data; If they are completely consistent, setting the first template identifier as the template identifier of the second template data; If they are not completely consistent, a second template identifier is generated for the second template data.

7. The method according to claim 1, characterized in that After transmitting the first template identifier and the binary byte stream as target transmission data, the method further includes: When it is determined that the target transmission data is received, obtaining first template data of the first JSON data through a first template identifier carried by the target transmission data; Obtain the key field name of the first JSON data through the first template data; Extracting bytes in the binary byte stream carried by the target transmission data through the first template data to obtain a target byte combination; Decode the target byte combination according to a preset decoding strategy to obtain a key field value of the first JSON data; The key field name and the key field value are combined to obtain first JSON data.

8. A data processing device, characterized in that: The device comprises: A first acquisition module, configured to acquire first JSON data to be transmitted, wherein the first JSON data includes at least one key-value pair, wherein the key-value pair includes a key field name and a key field value of the first JSON data; A second acquisition module, used for acquiring first template data of the first JSON data through the key-value pair; A first generating module, used for saving the first template data to a message queue and generating a first template identifier for the first template data; A third acquisition module, used to acquire the storage order of the key field names in the first template data, and acquire the key field values ​​from the key-value pairs according to the storage order; A fourth acquisition module, used for compressing and converting the key field value to obtain a binary byte stream; The transmission module is used to transmit the first template identifier and the binary byte stream as target transmission data.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the data processing method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the data processing method according to any one of claims 1 to 7.

Images