Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and apparatus, an electronic device and a storage medium, a message queue server caches original data sent by a sending client into at least one data packet, extracts fields of the original data and field values under the fields from the data packet, generates a field dictionary of each field according to the field values under the fields, and sends the field dictionary to the sending client; encoding the field value of each field in the data packet according to the field dictionary of the field to obtain a byte array of each field in the data packet, when the message queue server receives a first data request sent by a receiving client, obtaining a byte array of a target field corresponding to the first data request, and sending the byte array to the message queue server; and sending the byte array of the target field to a receiving client, wherein the receiving client is used for restoring the byte array of the target field into original data for processing. According to the invention, the message queue server converts a large number of fields and field values in the data packet into the binary byte array, so that the data processing and transmission efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, in particular to a data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] Message Queue (MQ) is a method of inter-process communication or communication between different threads in the same process. Currently, products such as Kafka, RabbitMQ, RocketMQ, etc. using MQ technology are widely used for data transmission between different systems. The data is usually transmitted in the form of key-value pairs. The data format for transmission is mainly JSON object. JSON object (JavaScript Object Notation) is a lightweight data exchange format. A JSON object is represented by a pair of braces. The properties of the object are described by key-value pairs in the braces. The key of the key-value pair is enclosed in quotation marks. The value of the key-value pair is any type of data in JS (JavaScript).

[0003] Currently, the MQ system stores and transmits the JSON format data sent by the producer as it is. When a large amount of data is transmitted, the amount of data is large and there is a lot of redundancy. The cached and transmitted data occupies a large amount of space and bandwidth, affecting the throughput performance of the MQ system and further affecting the data transmission efficiency. SUMMARY

[0004] Therefore, the present application aims to provide a data processing method and device, electronic equipment and storage medium to solve the problem that the JSON format data transmitted by the MQ system is large in amount and has a lot of redundancy, occupies a large amount of space and bandwidth, affects the throughput performance of the MQ system, and further affects the data transmission efficiency.

[0005] According to a first aspect of the present application, a data processing method is provided, applied to a message queue server, the message queue server being in communication with a sending client and a receiving client respectively, the method comprising: The message queue server caches the original data sent by the sending client as at least one data packet; From the data packet, the fields of the original data and the field values under the fields are extracted; According to the field values under the fields, a field dictionary of each field is generated; wherein the field dictionary includes a field code matching the field value; The field values of each field in the data packet are respectively encoded according to the field dictionary of the field to obtain a byte array of each field in the data packet; the byte array is used to restore the original data based on the field dictionary; When the message queue server receives the first data request sent by the receiving client, the byte array of the target field corresponding to the first data request is obtained, and the byte array of the target field is sent to the receiving client, and the receiving client is used to restore the byte array of the target field into original data for processing.

[0006] Optionally, the message queue server buffers the original data sent by the sending client as at least one data packet, comprising: The message queue server receives the original data sent by the sending client; The original data is divided into at least one data packet by a preset data amount or a preset period; The original data in the data packet is buffered to a preset message queue according to a field list.

[0007] Optionally, the field of the original data and the field value under the field are extracted from the data packet, comprising: Identifying the table header of the original data buffered according to the field list in the data packet, and extracting the field of the original data from the table header; Traverse the field column where the field is located to obtain the field value under each field respectively.

[0008] Optionally, the field dictionary of each field is generated according to the field value under the field, comprising: De-duplicate the field value under the field to obtain the effective field value under the field after de-duplication; Generating the field encoding corresponding to the effective field value according to the effective field value and the number of the effective field value; Matching the field value under the field according to the field encoding corresponding to the effective field value to generate the field dictionary of each field.

[0009] Optionally, the field value of each field in the data packet is encoded according to the field dictionary of the field respectively to obtain the byte array of each field in the data packet, comprising: Determine the characteristic information of each field in the data packet, wherein the characteristic information comprises at least one of the field number, the field length and the field value position; According to the field dictionary of the field, the characteristic information of the field is encoded to convert the field encoding of the field dictionary, and the byte array of each field in the data packet is obtained.

[0010] Optionally, the message queue server receives a first data request sent by the receiving client, acquires a byte array of a target field corresponding to the first data request, and sends the byte array of the target field to the receiving client, including: When the message queue server receives a first data request sent by the receiving client, the target field corresponding to the first data request is determined. According to the target field, the data packet is queried, and a byte array of the target field is acquired from the data packet. The message queue server sends the byte array of the target field to the receiving client.

[0011] Optionally, before the message queue server receives a first data request sent by the receiving client, acquires a byte array of a target field corresponding to the first data request, and sends the byte array of the target field to the receiving client, it further includes: When the message queue server receives a second data request sent by the receiving client, the target field quantity corresponding to the second data request is determined. A field of the target field quantity and a byte array of the field of the target field quantity are acquired from the data packet, and the byte array of the field of the target field quantity is sent to the receiving client.

[0012] According to a second aspect of the present application, a data processing device is provided, which is applied to a message queue server, the message queue server is in communication with a sending client and a receiving client respectively, and the device includes: A data cache module is configured to cache original data sent by the sending client as at least one data packet by the message queue server. An extraction information module is configured to extract a field of the original data and a field value under the field from the data packet. A dictionary generation module is configured to generate a field dictionary of each field according to the field value under the field, wherein the field dictionary includes a field code matched with a feature of the field value. A data conversion module is configured to encode the field value of each field in the data packet according to the field dictionary of the field, to obtain a byte array of each field in the data packet. A first processing module is configured to acquire a byte array of a target field corresponding to a first data request sent by the receiving client when the message queue server receives the first data request, and send the byte array of the target field to the receiving client, the receiving client being configured to restore the byte array of the target field into original data for processing.

[0013] According to another aspect of the present application, an electronic device is also provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the data processing method as described above.

[0014] According to another aspect of the present application, a readable storage medium is also provided, on which a computer program is stored, the computer program, when executed by a processor, implements the steps of the data processing method as described above.

[0015] The data processing method provided by the embodiments of the present application, the message queue server caches the original data sent by the sending client as at least one data packet, extracts the fields of the original data and the field values under the fields from the data packet, generates a field dictionary of each field according to the field values under the fields, wherein the field dictionary includes the field code matched with the field value, encodes the field values of each field in the data packet according to the field dictionary of the field respectively to obtain the byte array of each field in the data packet, when the message queue server receives the first data request sent by the receiving client, obtains the byte array of the target field corresponding to the first data request, and sends the byte array of the target field to the receiving client, which is used to restore the byte array of the target field into the original data for processing. The message queue server of the embodiments of the present application stores a large amount of original data in packets, encodes the fields and field values into field dictionaries according to the data packet, and converts the large number of fields and field values in the data packet into non-redundant and concise binary byte arrays by using the field dictionaries. The data volume after encoding conversion is small, which avoids occupying a large amount of space and bandwidth, improves the efficiency of data processing and transmission, and guarantees the data throughput performance of the message queue server.

[0016] The above description is only a summary of the technical solutions of the present application. In order to enable one of ordinary skill in the art to better understand the technical means of the present application and implement it according to the content of the description, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to depict only preferred embodiments of the application and therefore should not be considered to limit the scope of the application in any way. Furthermore, the drawings are merely intended to show one of the preferred embodiments of the application and are not intended to show all possible embodiments of the application. In the drawings: Figure 1 is a step flow chart of a data processing method provided by the embodiments of the present application; Figure 2 isFigure 1 Figure 101 is a flow chart of step 101 of the data processing method provided by the embodiments of the present application; Figure 3 is Figure 1 Figure 103 is a flow chart of step 103 of the data processing method provided by the embodiments of the present application; Figure 4 is Figure 1 Figure 104 is a flow chart of step 104 of the data processing method provided by the embodiments of the present application; Figure 5 is Figure 1 Figure 105 is a flow chart of step 105 of the data processing method provided by the embodiments of the present application; Figure 6 Figure 1 is a structural schematic diagram of a byte array in a data processing method provided by the embodiments of the present application; Figure 7 Figure 2 is a structural schematic diagram of a data processing device provided by the embodiments of the present application; Figure 8 Figure 3 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that, in the embodiments of the present application, many technical details are presented in order to make the readers better understand the present application. However, the technical solutions claimed by the present application can be implemented even without these technical details and based on various changes and modifications of the following embodiments. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present application. The embodiments can be combined and referenced with each other without contradiction.

[0019] Referring to Figure 1 Figure 1 is a flow chart of a step process of a data processing method provided by the embodiments of the present application, applied to a message queue server, the message queue server being in communication with a sending client and a receiving client respectively, and the method can include: Step 101: The message queue server caches original data sent by the sending client as at least one data packet.

[0020] In the vehicle production and use scenarios, the sending client, i.e., the producer client, can be a vehicle manufacturing terminal, a vehicle sensor, etc., the message queue server can be a product using MQ technology such as Kafka, RabbitMQ, RocketMQ, etc., and the receiving client, i.e., the consumer client, can be a data analysis system, an operation and maintenance platform, etc., wherein the message queue server communicates with the sending client and the receiving client, and the message queue server, as a kind of middleware, is used to deliver messages in a distributed system to realize data delivery between the sending client and the receiving client.

[0021] It should be noted that the original data of the sending client refers to the data generated by the sending client without processing, including production data, energy consumption data, device data, etc. For example, the production data includes assembly torque value, welding quality, station number, production timestamp, etc., the energy consumption data includes vehicle speed, throttle pedal opening, mileage, actual torque, etc. in production or use, and the device data includes device fault alarm, battery temperature, etc., which are not described one by one here.

[0022] In this embodiment, the sending client collects the original data, encapsulates it into JSON format, and sends it to the message queue server. The message queue server caches the original data sent by the sending client as at least one data packet. The data packet can be that the message queue server divides the original data into multiple data packets according to the preset data storage amount of the data packet, or the message queue server divides the original data into multiple time period data packets according to the preset period and caches them, which is not limited here.

[0023] In step 102, the fields of the original data and the field values under the fields are extracted from the data packet.

[0024] In this embodiment, the message queue server processes the original data in the data packet in units of data packets, extracts the fields of the original data and the field values under the fields from the data packet, wherein in the original data in JSON format, the fields and field values are the core components of structured data. The field is also called "key" and is used to represent the attribute name of the data to describe the meaning of the data, for example, the column name in the data or the table header of the table. The field value is also called "key value", which is the specific data content corresponding to the field and can be various data types, such as string, number, boolean, object, nested, array, null (null value), etc.

[0025] Specifically, the original data in the data packet can be stored in the form of a list, i.e., one piece of data at a time, divided into columns according to fields, and the header of each column is a different field, and the content in each column is a field value corresponding to the field. The message queue server extracts the fields and field values from the original data in the data packet, can traverse the original data in the JSON format in the data packet, eliminate the empty field values in the original data, and extract the fields in the list and the field values under each field.

[0026] In step 103, a field dictionary of each field is generated according to the field value under the field; wherein the field dictionary includes a field code matched with the field value.

[0027] In the embodiment of the application, in order to reduce redundant data and improve the efficiency of data processing, the field data is converted into a field dictionary in the embodiment, which does not lose the data content of the field and reduces the data volume. The field dictionary is a mapping table used to map the field value to a unique field code. The field dictionary is usually stored in the form of a key-value pair, with the key being the field value and the value being the corresponding field code.

[0028] Specifically, the field values under the field are de-duplicated to obtain effective field values after de-duplication. According to the effective field values and the number of effective field values, a field code corresponding to the effective field values is generated. All field values under the field are matched according to the field code corresponding to the effective field values, and all field values under the field are converted into field codes, thereby obtaining a field dictionary of each field. Specifically, duplicate values are removed from the field values, and non-duplicate field values are retained to obtain effective field values after de-duplication. According to the effective field values and the number of effective field values, a field code corresponding to the effective field values is generated. Each effective field value is assigned a unique field code to replace the original field value. The field code is a simple incremental binary representation, i.e., a binary field code.

[0029] In step 104, the field values of each field in the data packet are respectively encoded according to the field dictionary of the field to obtain a byte array of each field in the data packet.

[0030] In the embodiment of the application, the field dictionary reflects the field and the field code of the field value. The field values of each field in the data packet are respectively encoded according to the field dictionary of the field to obtain a byte array of each field in the data packet. According to the field dictionary of the field, the characteristic information of the field is encoded and converted into the field code of the field dictionary to obtain the byte array of the field. That is, based on the field dictionary of the field, the characteristic information of the field is encoded to obtain the complete byte array of the field.

[0031] The field characteristic information includes at least one of a field quantity, a field length and a field value position, the field quantity is a total number of fields contained in the data packet, the field length is a length of each field, the field value position is position information of a field value in the data packet, such as a row number and a column number, and the byte array refers to a byte sequence obtained by converting field encoding, the byte is binary data, and the obtained byte array is a Byte array, so that field data is converted into byte data.

[0032] In step 105, when the message queue server receives the first data request sent by the receiving client, the byte array of the target field corresponding to the first data request is obtained, and the byte array of the target field is sent to the receiving client, which is used to restore the byte array of the target field into original data for processing.

[0033] In the embodiment, when the message queue server receives the first data request sent by the receiving client, the byte array of the target field corresponding to the first data request is obtained, and the byte array of the target field is sent to the receiving client, which is used to restore the byte array of the target field into original data for processing. Specifically, the first data request refers to a request message sent by the receiving client for requesting specified field data, the target field refers to a field specified by the receiving client in the first data request and needed to be obtained, the receiving client synchronizes the encoding conditions of the fields and the bytes with the message queue server in advance, and the receiving client can restore the byte array into original data for processing, that is, restore the binary Byte array into original data in the json format. The embodiment does not limit the process of restoring data by the receiving client.

[0034] The data processing method provided in this application embodiment involves a message queue server caching raw data sent by a sending client into at least one data packet. From the data packet, fields and their values ​​are extracted from the raw data. Based on the field values, a field dictionary is generated for each field, where the field dictionary includes field codes matching the field values. The field values ​​of each field in the data packet are encoded according to the field dictionary to obtain byte arrays for each field in the data packet. When the message queue server receives a first data request from a receiving client, it obtains the byte array of the target field corresponding to the first data request and sends the target field's byte array to the receiving client. The receiving client then uses this target field's byte array to reconstruct the raw data for processing. This application embodiment of the message queue server stores large amounts of raw data in packets, encoding fields and their values ​​into a field dictionary on a packet-by-packet basis. The field dictionary converts the numerous fields and their values ​​in the data packet into concise, non-redundant binary byte arrays. The encoded data is small in size, avoiding excessive space and bandwidth usage, improving data processing and transmission efficiency, and ensuring the data throughput performance of the message queue server.

[0035] Reference Figure 2 , showed Figure 1 A flowchart of step 101 in a data processing method is provided. This method is basically the same as the data processing method provided in the first embodiment of this application. Step 101 may include: Step 1011: The message queue server receives the raw data sent by the sending client; Step 1012: Divide the original data into at least one data packet using a preset data volume or a preset period; Step 1013: Cache the original data in the data packet to a preset message queue according to the field list.

[0036] In this embodiment of the application, the sending client sends raw data to the message queue server. The sending client and the message queue server communicate with each other. The message queue server receives the raw data sent by the sending client. Since the amount of raw data is large and the sending time is irregular, in order to facilitate data processing, the message queue server divides the raw data into at least one data packet using a preset data volume or a preset period, and caches the raw data in the data packet into a preset message queue according to the field list.

[0037] Specifically, the preset data amount refers to dividing the original data into a plurality of data packets according to the data amount in the data packet, and the data in the data packet can be less than the preset data amount, for example, 1 MB of original data is divided into a data packet, the preset period refers to the time period of timing the buffered data packet, and the original data received in the time period is divided into a data packet according to the preset period to obtain a plurality of data packets, for example, the received original data is divided into a data packet every 5 seconds, and the data packet refers to a set of original data, which is usually stored in a JSON structured format.

[0038] In this embodiment, the message queue server divides the received original data into data packets according to a preset rule, and the original data in the data packet is cached in the form of a list after processing, specifically, the original data in the data packet is cached to a preset message queue according to a field list, the field list refers to a set of field names contained in the data packet, for example, the field list in the production data can include "assembly torque", "part batch number", etc., and each type of field is taken as a column, and the field value under the field is taken as the data content of the specific list, the processed data packet is temporarily stored in the message queue to respond to the data pulling request of the receiving client in time, wherein the preset message queue refers to a queue defined in advance in the message queue server and used for storing data packets, and each queue can correspond to different receiving clients.

[0039] The message queue server in the embodiment divides a large amount of original data into a plurality of data packets, and caches the original data in the data packet according to the field list, so as to realize the classified storage of data and facilitate the subsequent classified processing and analysis.

[0040] In some embodiments of the application, step 102, extracting the fields of the original data and the field values under the fields from the data packet, can specifically include: Identifying the table header of the original data cached according to the field list in the data packet, and extracting the fields of the original data from the table header; Traversing the field column where the fields are located to obtain the field values under each field respectively.

[0041] In the embodiment of the application, the data packet is a set of processed original data and is stored in the form of a list, wherein the original data is in JSON format, in order to facilitate the analysis of the original data, the message queue server divides the original data in the data packet into a field list according to the fields in advance, therefore, when the original data in the data packet is processed, the table header of the original data cached according to the field list in the data packet is identified, the fields of the original data are extracted from the table header, and the field column where the fields are located is traversed, so as to obtain the field values under each field respectively.

[0042] It should be noted that the field list refers to a set of field names contained in the data packet. For example, the field list in the production data can include table headers such as "assembly torque", "component batch number", etc. The table headers list the names of all fields in the data packet. The message queue server first identifies the table headers in the data packet and extracts the fields of the original data from the table headers. The field column where the field is located refers to the column in the data packet corresponding to the field. The field value refers to the specific value of the field in the field column. For example, the column where the "assembly torque" field is located contains all "assembly torque" field values or specific assembly torque values. The message queue server traverses each field column where the field is located to extract the field values and obtain the field values under each field, respectively.

[0043] In the embodiment of the present application, the original data in the data packet is stored in the form of a field list. The message queue server can comprehensively extract all field values under each field. The extracted field values are used for subsequent operations such as generating a field dictionary and encoding, thereby improving the efficiency of data processing.

[0044] Referring to Figure 3 , a flowchart of step 103 in a data processing method provided by the present application is shown. Figure 1 The method is basically the same as the data processing method provided by the first embodiment of the present application. Step 103 can include: Step 1031, de-duplicating the field values under the field to obtain valid field values under the field after de-duplication; Step 1032, generating a field code corresponding to the valid field values according to the valid field values and the number of valid field values; Step 1033, matching the field values under the field according to the field code corresponding to the valid field values to generate a field dictionary of each field.

[0045] In the embodiment of the present application, the field value refers to the specific value of a field in the data packet. For example, the "fault alarm" field in the energy consumption data can include multiple fault alarm types. To reduce redundant data and improve the efficiency of data processing, the field data is converted into a field dictionary in the embodiment, which not only does not lose the data content of the field, but also reduces the data volume. The field dictionary is a kind of mapping table used to map the field value to a unique field code. The field dictionary is usually stored in the form of key-value pairs, where the key is the field value and the value is the corresponding field code. Before generating the field dictionary, the field values under the field are first de-duplicated to obtain valid field values under the field after de-duplication, that is, the repeated values are removed from the field values and the non-repeated field values are retained. The valid field value refers to the field value after de-duplication.

[0046] Specifically, based on the valid field values ​​and their quantity, a corresponding field code is generated for each valid field value. The quantity of valid field values ​​refers to the number of field values ​​among the valid field values, used to determine the code for each field value. A unique field code is assigned to each valid field value to replace the original field value, thereby reducing data storage and transmission overhead. The message queue server generates a corresponding field code for each valid field value based on its quantity. The field code can be a simple incremental number converted into a binary expression, i.e., the field code is in binary form. For example, if the field is a fault alarm, the corresponding field value is the specific fault alarm type. According to the quantity of field values ​​in the fault alarm, the original fault alarm type is converted into a field code. For example, if the first fault alarm's field value is "battery overheating alarm," it is converted to code 0001; if the second fault alarm's field value is "engine fault," it is converted to code 0010, and so on. Of course, the above is only an example. In a specific implementation, the code can be any field code that can distinguish and reflect the field value. This embodiment does not impose any specific limitations.

[0047] In this embodiment, the message queue server matches the field values ​​under a field with the field codes corresponding to the valid field values ​​to generate a field dictionary for each field. That is, according to the field codes corresponding to the determined valid field values, each field value under a field is matched and mapped to the corresponding field code to generate a field dictionary containing all field values. The field dictionary can be stored on the message queue server for subsequent data encoding and client decoding.

[0048] This application embodiment significantly reduces data storage space by replacing longer field values ​​with shorter codes. The encoded data is smaller in size, and the generation and use of the field dictionary can reduce the complexity of data processing.

[0049] Reference Figure 4 , showed Figure 1 A flowchart of step 104 in a data processing method is provided. This method is basically the same as the data processing method provided in the first embodiment of this application. Step 104 may include: Step 1041: Determine the feature information of each field in the data packet, wherein the feature information includes at least one of the following: number of fields, field length, and position of field value; Step 1042: Encode the feature information of the field according to the field dictionary, convert it into the field code of the field dictionary, and obtain the byte array of each field in the data packet.

[0050] In the embodiment of the present application, after obtaining the field dictionary, the field dictionary reflects the field encoding of the field and the field value. In order to accurately express the data content of the original data in the data packet, the embodiment encodes and converts the characteristic information of the field of the original data in the data packet, and obtains the byte array of each field in the data packet.

[0051] Specifically, the characteristic information of each field in the data packet is determined, wherein the characteristic information includes at least one of the field quantity, the field length, and the field value position. The field quantity is the total number of fields contained in the data packet, the field length is the length of each field, and the field value position is the position information of the field value in the data packet, such as row number and column number.

[0052] In the embodiment, the characteristic information of the field is encoded according to the field dictionary of the field, and is converted into the field encoding of the field dictionary, and the byte array of the field is obtained. That is, based on the field dictionary of the field, the characteristic information of the field is encoded into the complete byte array of the field. The encoding operation refers to converting the field encoding into a byte sequence. The byte is binary data, and the obtained is the byte array, which realizes storing the field value in the form of binary bytes. The message queue server encodes the characteristic information of the field according to the field dictionary of the field, and converts it into a byte array, and obtains the byte array of each field in the data packet.

[0053] For example, referring to Figure 6 , Figure 6 The structure of the byte array of the data processing method provided by the embodiment of the present application is shown. According to the number of fields in the field dictionary, the length of the byte required to store the encoding is determined. The field value under each field is converted into the encoding of the field value in the field dictionary according to the position of the field list in the data packet, and the encoding of the field value in the field dictionary is written into the byte array. The field number is the number of fields in the data packet, occupying 2 bytes, and supporting up to 65535 fields. The number of data rows reflects the number of data packets in the data packet, and the number of bytes occupied is calculated by the maximum number of data packets. The field name is the field itself, the field position is the position of the field list in the field dictionary, the field number is the encoding number of the field in the field dictionary, and the field dictionary length is the total number of bytes occupied by the field encoding of the field dictionary. Each field information and the field dictionary are serialized into a byte array, that is, the byte array of the field.

[0054] The field dictionary of the field encodes the field and the field value in the data packet, converts the large field data into a binary byte array, and the size of the encoded data is small, which is convenient for transmission and significantly improves the efficiency of data processing and transmission.

[0055] For example, referring to Figure 5 , it is shown that Figure 1A flowchart of step 105 of the provided data processing method is provided, which is basically the same as the data processing method provided by the first embodiment of the present application. Step 105 can include: Step 1051, when the message queue server receives the first data request sent by the receiving client, determining the target field corresponding to the first data request; Step 1052, querying the data packet according to the target field to obtain the byte array of the target field from the data packet; Step 1053, the message queue server sends the byte array of the target field to the receiving client.

[0056] In the embodiment of the present application, when the message queue server receives the first data request sent by the receiving client, the target field corresponding to the first data request is determined. The first data request refers to the request message sent by the receiving client to request the data of the specified field. The target field refers to the field that the receiving client specifies to obtain in the first data request, for example, the receiving client requests to obtain the data of the "assembly torque" field or the "vehicle speed" field. The embodiment does not limit the target field obtained at one time, which can be at least one field.

[0057] Specifically, the message queue server queries the cached data packet according to the target field to query the target field needed in the data packet, directly obtains the byte array of the target field from the data packet, and sends the byte array of the target field to the receiving client. After the receiving client receives the byte array of the target field, the original field and field value are restored by decoding the byte array, and the original data is obtained.

[0058] In the embodiment of the present application, the message queue server explicitly receives the target field specified by the receiving client to obtain, only processes the target field, and returns the byte array of the target field. The byte array is encoded binary data, which is small in size and convenient for transmission, and significantly improves the efficiency of data processing and transmission.

[0059] In some embodiments of the present application, when the message queue server receives the first data request sent by the receiving client, the byte array of the target field corresponding to the first data request is obtained, and the byte array of the target field is sent to the receiving client. Before the receiving client restores the byte array of the target field into the original data for processing, it can further include: When the message queue server receives the second data request sent by the receiving client, the target field quantity corresponding to the second data request is determined; Obtaining the fields of the target field quantity and the byte array of the fields of the target field quantity from the data packet, and sending the byte array of the fields of the target field quantity to the receiving client.

[0060] In the embodiment of the present application, when the message queue server receives a second data request sent by a receiving client, the target field quantity corresponding to the second data request is determined, the second data request refers to another type of data request sent by the receiving client, which is used to request a preset data quantity of fields, and the field category is not limited, and the target field quantity refers to the field quantity required to be obtained by the receiving client in the second data request. By analyzing the second data request, the message queue server can determine the field quantity required to be obtained by the receiving client, and can randomly obtain the target field quantity of fields and the byte array of the target field quantity of fields from the data packet in the data packet caching order, and send the target field quantity of fields and the byte array of the target field quantity of fields to the receiving client.

[0061] In the embodiment of the present application, the message queue server determines the quantity of fields specified to be obtained by the receiving client, and randomly returns the byte array of the required quantity of fields. The byte array is encoded binary data, which has a small volume and is convenient for transmission, and significantly improves the efficiency of data processing and transmission.

[0062] Referring to Figure 7 , a structure schematic diagram of a data processing apparatus provided by an embodiment of the present application is shown, which is applied to a message queue server, the message queue server communicates with a sending client and a receiving client respectively, and the apparatus comprises: A data caching module 201, configured to cache original data sent by the sending client into at least one data packet by the message queue server; An extraction information module 202, configured to extract fields of the original data and field values under the fields from the data packet; A dictionary generation module 203, configured to generate a field dictionary of each field according to the field values under the field; wherein the field dictionary comprises field encoding matched with the characteristics of the field values; A data conversion module 204, configured to encode the field values of each field in the data packet according to the field dictionary of the field respectively, to obtain byte arrays of each field in the data packet; A first processing module 205, configured to obtain a byte array of a target field corresponding to a first data request sent by the receiving client when the message queue server receives the first data request, and send the byte array of the target field to the receiving client, the receiving client being configured to restore the byte array of the target field into original data for processing.

[0063] Optionally, the data caching module 201 comprises: A receiving sub-module, configured to receive original data sent by the sending client by the message queue server; a dividing sub-module, configured to divide the original data into at least one data packet by using a preset data amount or a preset period; a caching sub-module, configured to cache the original data in the data packet to a preset message queue according to a field list.

[0064] Optionally, the obtaining information module 202 comprises: an extracting sub-module, configured to identify a table header of the original data cached according to the field list in the data packet, and extract a field of the original data from the table header; a traversing sub-module, configured to traverse a field column where the field is located, and obtain a field value under each field.

[0065] Optionally, the dictionary generating module 203 comprises: a deduplication sub-module, configured to deduplicate the field value under the field, and obtain an effective field value under the field after deduplication; a first generating sub-module, configured to generate a field code corresponding to the effective field value according to the effective field value and a quantity of the effective field values; a second generating sub-module, configured to match the field value under the field according to the field code corresponding to the effective field value, and generate a field dictionary of each field.

[0066] Optionally, the data converting module 204 comprises: a first determining sub-module, configured to determine characteristic information of each field in the data packet, wherein the characteristic information comprises at least one of a field quantity, a field length and a field value position; a coding sub-module, configured to code the characteristic information of the field according to the field dictionary of the field, convert the characteristic information into a field code of the field dictionary, and obtain a byte array of each field in the data packet.

[0067] Optionally, the first processing module 205 comprises: a first determining sub-module, configured to determine a target field corresponding to a first data request when the message queue server receives the first data request sent by the receiving client; a screening sub-module, configured to query the data packet according to the target field, and obtain a byte array of the target field from the data packet; a sending sub-module, configured to send the byte array of the target field to the receiving client by the message queue server.

[0068] Optionally, the apparatus further comprises: The request receiving module is used to determine the target field quantity corresponding to the second data request when the message queue server receives the second data request sent by the receiving client; The second processing module is used to obtain the target field quantity and the byte array of the target field quantity from the data packet, and send the byte array of the target field quantity to the receiving client.

[0069] The data processing apparatus provided in this application embodiment caches raw data sent by a sending client into at least one data packet. From the data packet, it extracts fields of the raw data and field values ​​under each field. Based on the field values, it generates a field dictionary for each field, where the field dictionary includes field codes matching the field values. The field values ​​of each field in the data packet are encoded according to the field dictionary to obtain byte arrays of each field in the data packet. When the message queue server receives a first data request from a receiving client, it obtains the byte array of the target field corresponding to the first data request and sends the target field byte array to the receiving client. The receiving client uses this target field byte array to restore the raw data for processing. This application embodiment of the message queue server stores large amounts of raw data in packets, encoding fields and field values ​​into a field dictionary on a packet-by-packet basis. The field dictionary converts the numerous fields and field values ​​in the data packet into concise, non-redundant binary byte arrays. The encoded data is small in size, avoiding excessive space and bandwidth usage, improving data processing and transmission efficiency, and ensuring the data throughput performance of the message queue server.

[0070] Reference Figure 8 This application also provides an electronic device, such as... Figure 8 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. Processor 301; Memory 303 is used to store processor-executable instructions; The processor 301 is configured to execute the instructions to implement the data processing method described above.

[0071] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0072] The communication interface is used for communication between the terminal and other devices.

[0073] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the processor.

[0074] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0075] In another embodiment provided in the application, a computer readable storage medium is also provided, and the readable storage medium stores a computer program. When the computer program is executed by a processor, the data processing method in any of the above embodiments is implemented.

[0076] In the embodiments described above, all or some of the steps can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, all or some of the steps can be implemented in the form of one or more computer programs or program elements. The computer programs reside (at least temporarily) in a memory of a computer during execution. The memory can be a RAM memory, a flash memory, a ROM memory, an EPROM memory, or any other suitable memory. The memory can be integral to or separate from the computer. The computer programs can be written in any suitable programming language, such as C, C++, Java, Visual Basic, etc. The computer programs can be written in assembly or machine language, if desired. The computer programs can be distributed over network coupled file servers, or can be distributed by any other suitable means.

[0077] It is to be understood that the terminology "first", "second", etc. used herein merely identifies one entity or action from another, but does not necessarily imply, or require, any actual such relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0078] Each of the embodiments described in the specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the system embodiments are described in a relatively simple manner, since they are substantially similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments.

[0079] The above merely provides the preferred embodiments of the application, and not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, and the like made within the principle and technical scope of the application shall fall into the protection scope of the application.

Claims

1. A data processing method, characterized in that, Applied to a message queue server, which communicates with both a sending client and a receiving client, the method includes: The message queue server caches the raw data sent by the sending client as at least one data packet; Extract the fields of the original data and the field values ​​under those fields from the data packet; Based on the field values ​​under the fields, a field dictionary is generated for each field; wherein, the field dictionary includes field codes that match the field values; The field values ​​of each field in the data packet are encoded according to the field dictionary of the field to obtain the byte array of each field in the data packet; When the message queue server receives a first data request sent by the receiving client, it obtains the byte array of the target field corresponding to the first data request and sends the byte array of the target field to the receiving client. The receiving client is used to restore the byte array of the target field to the original data for processing.

2. The method according to claim 1, characterized in that, The message queue server caches the raw data sent by the sending client into at least one data packet, including: The message queue server receives the raw data sent by the sending client; The original data is divided into at least one data packet using a preset data volume or a preset period. The original data in the data packet is cached in a preset message queue according to the field list.

3. The method according to claim 2, characterized in that, The step of extracting the fields of the original data and the field values ​​under the fields from the data packet includes: Identify the header of the original data cached by field list in the data packet, and extract the fields of the original data from the header; Iterate through the column containing the field to obtain the field value for each field.

4. The method according to claim 1, characterized in that, The step of generating a field dictionary for each field based on the field values ​​under the field includes: The field values ​​under the aforementioned field are deduplicated to obtain the valid field values ​​under the aforementioned field after deduplication. Based on the valid field values ​​and the number of valid field values, generate the field codes corresponding to the valid field values; The field values ​​under the field are matched according to the field codes corresponding to the valid field values ​​to generate a field dictionary for each field.

5. The method according to claim 1, characterized in that, The step of encoding the field values ​​of each field in the data packet according to the field dictionary of the field to obtain the byte array of each field in the data packet includes: Determine the feature information of each field in the data packet, wherein the feature information includes at least one of the following: number of fields, field length, and position of field value; Based on the field dictionary of the field, the feature information of the field is encoded and converted into the field code of the field dictionary to obtain the byte array of each field in the data packet.

6. The method according to claim 1, characterized in that, When the message queue server receives a first data request sent by the receiving client, it obtains a byte array of the target field corresponding to the first data request and sends the byte array of the target field to the receiving client, including: When the message queue server receives the first data request sent by the receiving client, it determines the target field corresponding to the first data request; Query the data packet based on the target field, and obtain the byte array of the target field from the data packet; The message queue server sends the byte array of the target field to the receiving client.

7. The method according to claim 1, characterized in that, Before the step of obtaining the byte array of the target field corresponding to the first data request and sending the byte array of the target field to the receiving client when the message queue server receives the first data request sent by the receiving client, the method further includes: When the message queue server receives the second data request sent by the receiving client, it determines the target field quantity corresponding to the second data request; The target field quantity and its byte array are obtained from the data packet, and the byte array of the target field quantity is sent to the receiving client.

8. A data processing apparatus, characterized in that, The device is applied to a message queue server, which communicates with both a sending client and a receiving client. The device includes: A data caching module is used by the message queue server to cache the raw data sent by the sending client into at least one data packet; The information extraction module is used to extract fields of the original data and the field values ​​under the fields from the data packet; A dictionary generation module is used to generate a field dictionary for each field based on the field values ​​under the field; wherein, the field dictionary includes field codes that match the features of the field values; The data conversion module is used to encode the field values ​​of each field in the data packet according to the field dictionary of the field, so as to obtain the byte array of each field in the data packet. The first processing module is configured to, when the message queue server receives a first data request sent by the receiving client, obtain a byte array of the target field corresponding to the first data request, and send the byte array of the target field to the receiving client, wherein the receiving client is configured to restore the byte array of the target field to the original data for processing.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the instructions to implement the data processing method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, A computer program is stored on the readable storage medium, which, when executed by a processor, implements the data processing method as described in any one of claims 1 to 7.

Images